Ceph » rgw

from http://qa-proxy.ceph.com/teuthology/cbodley-2022-09-01_15:25:15-rgw-wip-cbodley-testing-distro-default-smithi/7005410/teuthology.log

2022-09-02T04:58:29.067 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket upvuik-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0
2022-09-02T04:58:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:ignoring --setuser ceph since I am not root
2022-09-02T04:58:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:ignoring --setgroup ceph since I am not root
2022-09-02T04:58:29.195 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.196+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0
2022-09-02T04:58:29.200 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.201+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: waiting on bucket lease
2022-09-02T04:58:29.201 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.202+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: status after acquiring the lock is: init
2022-09-02T04:58:29.225 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.225+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: listing bucket for full sync
2022-09-02T04:58:29.272 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj11
2022-09-02T04:58:29.273 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj11]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj11 op=0 op_state=1
2022-09-02T04:58:29.273 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj11]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj11[0]
2022-09-02T04:58:29.273 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj111
2022-09-02T04:58:29.273 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj111]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj111 op=0 op_state=1
2022-09-02T04:58:29.274 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj111]: : injecting data sync error on key=obj111
2022-09-02T04:58:29.274 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error)
2022-09-02T04:58:29.274 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj111]: ERROR: failed to sync object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj111
2022-09-02T04:58:29.275 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj115
2022-09-02T04:58:29.275 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj115]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj115 op=0 op_state=1
2022-09-02T04:58:29.275 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj115]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj115[0]
2022-09-02T04:58:29.276 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj116
2022-09-02T04:58:29.276 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj116]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj116 op=0 op_state=1
2022-09-02T04:58:29.276 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj116]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj116[0]
2022-09-02T04:58:29.276 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj124
2022-09-02T04:58:29.277 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj124]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj124 op=0 op_state=1
2022-09-02T04:58:29.277 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj124]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj124[0]
2022-09-02T04:58:29.277 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj145
2022-09-02T04:58:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj145]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj145 op=0 op_state=1
2022-09-02T04:58:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj145]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj145[0]
2022-09-02T04:58:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj148
2022-09-02T04:58:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj148]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj148 op=0 op_state=1
2022-09-02T04:58:29.279 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj148]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj148[0]
2022-09-02T04:58:29.279 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj155
2022-09-02T04:58:29.279 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj155]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj155 op=0 op_state=1
2022-09-02T04:58:29.280 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj155]: : injecting data sync error on key=obj155
2022-09-02T04:58:29.280 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error)
2022-09-02T04:58:29.280 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj155]: ERROR: failed to sync object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj155
2022-09-02T04:58:29.280 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj17
2022-09-02T04:58:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj17]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj17 op=0 op_state=1
2022-09-02T04:58:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj17]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj17[0]
2022-09-02T04:58:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.271+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj185
2022-09-02T04:58:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj185]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj185 op=0 op_state=1
2022-09-02T04:58:29.282 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj185]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj185[0]
2022-09-02T04:58:29.282 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj19
2022-09-02T04:58:29.282 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj19]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj19 op=0 op_state=1
2022-09-02T04:58:29.283 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj19]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj19[0]
2022-09-02T04:58:29.283 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj22
2022-09-02T04:58:29.283 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj22]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj22 op=0 op_state=1
2022-09-02T04:58:29.283 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj22]: : injecting data sync error on key=obj22
2022-09-02T04:58:29.284 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error)
2022-09-02T04:58:29.284 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj22]: ERROR: failed to sync object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj22
2022-09-02T04:58:29.284 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj23
2022-09-02T04:58:29.285 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj23]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj23 op=0 op_state=1
2022-09-02T04:58:29.285 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj23]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj23[0]
2022-09-02T04:58:29.285 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj29
2022-09-02T04:58:29.285 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj29]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj29 op=0 op_state=1
2022-09-02T04:58:29.286 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj29]: : injecting data sync error on key=obj29
2022-09-02T04:58:29.286 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error)
2022-09-02T04:58:29.286 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj29]: ERROR: failed to sync object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj29
2022-09-02T04:58:29.287 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj31
2022-09-02T04:58:29.287 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj31]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj31 op=0 op_state=1
2022-09-02T04:58:29.287 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj31]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj31[0]
2022-09-02T04:58:29.287 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj33
2022-09-02T04:58:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj33]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj33 op=0 op_state=1
2022-09-02T04:58:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj33]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj33[0]
2022-09-02T04:58:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj34
2022-09-02T04:58:29.289 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj34]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj34 op=0 op_state=1
2022-09-02T04:58:29.289 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj34]: : injecting data sync error on key=obj34
2022-09-02T04:58:29.289 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error)
2022-09-02T04:58:29.289 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  0 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj34]: ERROR: failed to sync object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj34
2022-09-02T04:58:29.290 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj37
2022-09-02T04:58:29.290 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj37]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj37 op=0 op_state=1
2022-09-02T04:58:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj37]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj37[0]
2022-09-02T04:58:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj38
2022-09-02T04:58:29.292 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj38]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj38 op=0 op_state=1
2022-09-02T04:58:29.292 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj38]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj38[0]
2022-09-02T04:58:29.293 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj44
2022-09-02T04:58:29.293 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj44]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj44 op=0 op_state=1
2022-09-02T04:58:29.294 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj44]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj44[0]
2022-09-02T04:58:29.294 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: [full sync] syncing object: upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0/obj46
2022-09-02T04:58:29.295 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj46]: bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b=bucket sync single entry (source_zone=64f865d2-01f6-4fea-8292-de0e93e33fec) b= log_entry=obj46 op=0 op_state=1
2022-09-02T04:58:29.295 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.272+0000 7f272a9c5600  5 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj46]: bucket sync: sync obj: 64f865d2-01f6-4fea-8292-de0e93e33fec/:upvuik-59[64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24])/obj46[0]
2022-09-02T04:58:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.279+0000 7f272a9c5600 20 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]: ERROR: full sync failed. error: -5
2022-09-02T04:58:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.295+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj148]: success
2022-09-02T04:58:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj17]: success
2022-09-02T04:58:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj116]: success
2022-09-02T04:58:29.299 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj44]: success
2022-09-02T04:58:29.299 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj46]: success
2022-09-02T04:58:29.300 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj185]: success
2022-09-02T04:58:29.300 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj31]: success
2022-09-02T04:58:29.300 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj33]: success
2022-09-02T04:58:29.300 INFO:teuthology.orchestra.run.smithi040.stderr:2022-09-02T04:58:29.296+0000 7f272a9c5600 10 RGW-SYNC:bucket[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24<-upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:full_sync[upvuik-59:64f865d2-01f6-4fea-8292-de0e93e33fec.5327.24:0]:entry[obj115]: success
2022-09-02T04:58:29.301 INFO:teuthology.orchestra.run.smithi040.stderr:*** Caught signal (Segmentation fault) **
2022-09-02T04:58:29.301 INFO:teuthology.orchestra.run.smithi040.stderr: in thread 7f272a9c5600 thread_name:radosgw-admin
2022-09-02T04:58:29.316 INFO:teuthology.orchestra.run.smithi040.stderr: ceph version 17.0.0-14694-g08bea909 (08bea90915abc98fc303c5e8ead6bd46479bcfcb) quincy (dev)
2022-09-02T04:58:29.317 INFO:teuthology.orchestra.run.smithi040.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f273218bcf0]
2022-09-02T04:58:29.317 INFO:teuthology.orchestra.run.smithi040.stderr: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f27331d9523]
2022-09-02T04:58:29.318 INFO:teuthology.orchestra.run.smithi040.stderr: 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f27331d986e]
2022-09-02T04:58:29.318 INFO:teuthology.orchestra.run.smithi040.stderr: 4: (std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >*)+0x69) [0x56082525e929]
2022-09-02T04:58:29.319 INFO:teuthology.orchestra.run.smithi040.stderr: 5: (RGWFetchObjFilter_Sync::~RGWFetchObjFilter_Sync()+0x96) [0x5608253dbd96]
2022-09-02T04:58:29.319 INFO:teuthology.orchestra.run.smithi040.stderr: 6: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x3a) [0x56082519904a]
2022-09-02T04:58:29.320 INFO:teuthology.orchestra.run.smithi040.stderr: 7: (RGWFetchRemoteObjCR::~RGWFetchRemoteObjCR()+0x43) [0x5608253d6c03]
2022-09-02T04:58:29.320 INFO:teuthology.orchestra.run.smithi040.stderr: 8: (ceph::common::RefCountedObject::put() const+0x1ba) [0x7f27315b748a]
2022-09-02T04:58:29.321 INFO:teuthology.orchestra.run.smithi040.stderr: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x360) [0x560825515470]
2022-09-02T04:58:29.321 INFO:teuthology.orchestra.run.smithi040.stderr: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x521) [0x560825515a71]
2022-09-02T04:58:29.322 INFO:teuthology.orchestra.run.smithi040.stderr: 11: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x5608253c5eca]
2022-09-02T04:58:29.322 INFO:teuthology.orchestra.run.smithi040.stderr: 12: main()
2022-09-02T04:58:29.322 INFO:teuthology.orchestra.run.smithi040.stderr: 13: __libc_start_main()
2022-09-02T04:58:29.323 INFO:teuthology.orchestra.run.smithi040.stderr: 14: _start()