Ceph » rgw

the teuthology.log for two-zonegroup jobs tend to be much larger (2GB+) than for three-zones jobs (~7MB)

a two-zonegroup example:
http://qa-proxy.ceph.com/teuthology/cbodley-2024-03-19_01:20:35-rgw-wip-rgw-account-topic-distro-default-smithi/7610238/teuthology.log

2024-03-19T04:57:12.894 DEBUG:boto:path=/vhubgc-23/
2024-03-19T04:57:12.894 DEBUG:boto:auth_path=/vhubgc-23/
2024-03-19T04:57:12.894 DEBUG:boto:path=/vhubgc-23/?versioning
2024-03-19T04:57:12.894 DEBUG:boto:auth_path=/vhubgc-23/?versioning
2024-03-19T04:57:12.894 DEBUG:boto:Method: PUT
2024-03-19T04:57:12.895 DEBUG:boto:Path: /vhubgc-23/?versioning
2024-03-19T04:57:12.895 DEBUG:boto:Data: <?xml version="1.0" encoding="UTF-8"?>
       <VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
         <Status>Enabled</Status>
         <MfaDelete>Disabled</MfaDelete>
       </VersioningConfiguration>
2024-03-19T04:57:12.895 DEBUG:boto:Headers: {}
2024-03-19T04:57:12.895 DEBUG:boto:Host: smithi170.front.sepia.ceph.com:8000
2024-03-19T04:57:12.895 DEBUG:boto:Port: 8000
2024-03-19T04:57:12.895 DEBUG:boto:Params: {}
2024-03-19T04:57:12.895 DEBUG:boto:Token: None
2024-03-19T04:57:12.895 DEBUG:boto:StringToSign:
PUT

Tue, 19 Mar 2024 04:57:12 GMT
/vhubgc-23/?versioning
2024-03-19T04:57:12.895 DEBUG:boto:Signature:
AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=
2024-03-19T04:57:12.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 19 Mar 2024 04:57:12 GMT', 'Authorization': 'AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=', 'Content-Length': '228'}
2024-03-19T04:57:12.897 DEBUG:boto:Response headers: [('Location', 'http://smithi170.front.sepia.ceph.com:8000/vhubgc-23/?versioning'), ('x-amz-request-id', 'tx00000500571966ab431ae-0065f91b28-4346-b1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 19 Mar 2024 04:57:12 GMT'), ('Connection', 'Keep-Alive')]
2024-03-19T04:57:12.897 DEBUG:boto:Redirecting: http://smithi170.front.sepia.ceph.com/vhubgc-23/?versioning
2024-03-19T04:57:12.897 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'}
2024-03-19T04:57:12.898 DEBUG:boto:Token: None
2024-03-19T04:57:12.898 DEBUG:boto:StringToSign:
PUT

Tue, 19 Mar 2024 04:57:12 GMT
/vhubgc-23/?versioning
2024-03-19T04:57:12.898 DEBUG:boto:Signature:
AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=
2024-03-19T04:57:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 19 Mar 2024 04:57:12 GMT', 'Authorization': 'AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=', 'Content-Length': '228'}
2024-03-19T04:57:12.901 DEBUG:boto:Response headers: [('Location', 'http://smithi170.front.sepia.ceph.com:8000/vhubgc-23/?versioning'), ('x-amz-request-id', 'tx000003dba6c02bd29847b-0065f91b28-4346-b1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 19 Mar 2024 04:57:12 GMT'), ('Connection', 'Keep-Alive')]
2024-03-19T04:57:12.902 DEBUG:boto:Redirecting: http://smithi170.front.sepia.ceph.com/vhubgc-23/?versioning
2024-03-19T04:57:12.902 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'}
2024-03-19T04:57:12.902 DEBUG:boto:Token: None
2024-03-19T04:57:12.902 DEBUG:boto:StringToSign:
PUT

Tue, 19 Mar 2024 04:57:12 GMT
/vhubgc-23/?versioning
2024-03-19T04:57:12.902 DEBUG:boto:Signature:
AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=
2024-03-19T04:57:12.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 19 Mar 2024 04:57:12 GMT', 'Authorization': 'AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=', 'Content-Length': '228'}
2024-03-19T04:57:12.905 DEBUG:boto:Response headers: [('Location', 'http://smithi170.front.sepia.ceph.com:8000/vhubgc-23/?versioning'), ('x-amz-request-id', 'tx000005b39dc3b7a6d7fde-0065f91b28-4346-b1'), ('Server', 'Ceph Object Gateway (squid)'), ('Content-Length', '0'), ('Date', 'Tue, 19 Mar 2024 04:57:12 GMT'), ('Connection', 'Keep-Alive')]
2024-03-19T04:57:12.905 DEBUG:boto:Redirecting: http://smithi170.front.sepia.ceph.com/vhubgc-23/?versioning

this request gets redirected to the same endpoint it was sent to, so the client keeps resending in a tight loop for over 3 hours. eventually it fails with a RequestTimeTooSkewed error

2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Redirecting: http://smithi170.front.sepia.ceph.com/vhubgc-23/?versioning
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'}
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign:
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:PUT
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:
2024-03-19T07:42:50.240 INFO:tasks.rgw_multisite_tests:
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:Tue, 19 Mar 2024 04:57:12 GMT
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:/vhubgc-23/?versioning
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature:
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 19 Mar 2024 04:57:12 GMT', 'Authorization': 'AWS XLA09NNEANTX08LSOFFK:BdbJMnebcPTX4V5AOSyAz2xQzKA=', 'Content-Length': '228'}
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '196'), ('x-amz-request-id', 'tx000000ae8b8f7bdec6ae1-0065f91eac-4346-b1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Server', 'Ceph Object Gateway (squid)'), ('Date', 'Tue, 19 Mar 2024 05:12:12 GMT'), ('Connection', 'Keep-Alive')]
2024-03-19T07:42:50.241 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << ---------------------
2024-03-19T07:42:50.509 INFO:tasks.rgw_multisite_tests:
2024-03-19T07:42:50.509 INFO:tasks.rgw_multisite_tests:======================================================================
2024-03-19T07:42:50.510 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_acl
2024-03-19T07:42:50.510 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2024-03-19T07:42:51.082 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2024-03-19T07:42:51.082 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2024-03-19T07:42:51.082 INFO:tasks.rgw_multisite_tests:    self.test(*self.arg)
2024-03-19T07:42:51.082 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_ceph-c_edd0fb3d563d7896f87f99934a4f5216dae07a0e/qa/../src/test/rgw/rgw_multi/tests.py", line 1045, in test_bucket_acl
2024-03-19T07:42:51.082 INFO:tasks.rgw_multisite_tests:    assert(len(bucket.get_acl().acl.grants) == 1) # single grant on owner
2024-03-19T07:42:51.083 INFO:tasks.rgw_multisite_tests:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/boto/s3/bucket.py", line 963, in get_acl
2024-03-19T07:42:51.083 INFO:tasks.rgw_multisite_tests:    raise self.connection.provider.storage_response_error(
2024-03-19T07:42:51.083 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
2024-03-19T07:42:51.083 INFO:tasks.rgw_multisite_tests:<?xml version="1.0" encoding="UTF-8"?><Error><Code>RequestTimeTooSkewed</Code><Message></Message><RequestId>tx00000cdf101aa5b881e84-0065f92230-4346-b1</RequestId><HostId>4346-b1-b</HostId></Error>
2024-03-19T07:42:51.083 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << --------------------