Ceph » rgw

ceph version 17.0.0-5278-g79eb0c85 (79eb0c853ca1ee491410e0c6c6796675a7449ef9) quincy (dev)

Steps:
- On a versioned bucket start IO from both directions on objects 1 to 300k (same objects from both directions)
- around 250k written from primary and 225k from secondary
- Try a bi list from any site , fails with IO error , and subsequent bucket reshard also fails with the same .
- I dont know if this makes a difference , but rgw_max_objs_per_shard is set to 500 and 300 on primary and secondary respectively.

/]# radosgw-admin bi list --bucket ver1 >o1
ERROR: bi_list(): (5) Input/output error

/]# radosgw-admin bucket stats --bucket ver1 {
"bucket": "ver1",
"num_shards": 11,
"tenant": "",
"zonegroup": "1785a4fa-f7d6-4081-8c72-74f9cc441d3a",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "b76458b4-55bc-41b4-8610-b4aa5df49661.25021.1",
"marker": "b76458b4-55bc-41b4-8610-b4aa5df49661.25021.1",
"index_type": "Normal",
"owner": "test1",
"ver": "0#86569,1#86697,2#86661,3#86303,4#86693,5#86707,6#86397,7#86947,8#86609,9#86757,10#86708",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2021-06-28T14:27:35.268133Z",
"max_marker": "0#00000086568.173215.11,1#00000086696.173471.11,2#00000086660.173399.11,3#00000086302.172683.11,4#00000086692.173463.11,5#00000086706.173491.11,6#00000086396.172871.11,7#00000086946.173971.11,8#00000086608.173295.11,9#00000086756.173591.11,10#00000086707.173493.11",
"usage": {
"rgw.main": {
"size": 5510454152,
"size_actual": 5855453184,
"size_utilized": 5510454152,
"size_kb": 5381303,
"size_kb_actual": 5718216,
"size_kb_utilized": 5381303,
"num_objects": 476518
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

Command with debug rgw:

/]# radosgw-admin bi list --bucket ver1 >o1 --debug_rgw=20
2021-06-29T13:14:27.795+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.799+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.799+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.799+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=105
2021-06-29T13:14:27.799+0000 7f4dbb042340 20 realm  india 6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7
2021-06-29T13:14:27.799+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.800+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=105
2021-06-29T13:14:27.800+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.801+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=10
2021-06-29T13:14:27.801+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.801+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=1093
2021-06-29T13:14:27.801+0000 7f4dbb042340 20 current period 253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6
2021-06-29T13:14:27.801+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.803+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.803+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.803+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.803+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.804+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=105
2021-06-29T13:14:27.804+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.805+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6.2
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got zonegroups_names.south
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got realms_names.india
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got period_config.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got realms.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6.latest_epoch
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got realms.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7.control
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got default.realm
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7:staging
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got zone_info.b76458b4-55bc-41b4-8610-b4aa5df49661
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got default.zone.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got zone_names.blr
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got default.zonegroup.6d34d4f8-671c-4cf0-a6b7-48e1fa21fde7
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6.1
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.bf3e33fa-e935-4a68-8234-2912c6719db1.1
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got zonegroup_info.1785a4fa-f7d6-4081-8c72-74f9cc441d3a
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6.3
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.bf3e33fa-e935-4a68-8234-2912c6719db1.latest_epoch
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 RGWRados::pool_iterate: got periods.253b44c3-aa4d-441e-bbb9-0e6c9e6f73a6.4
2021-06-29T13:14:27.844+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.845+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.845+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.845+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.845+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=105
2021-06-29T13:14:27.846+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=610
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 period zonegroup init ret 0
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 period zonegroup name south
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 using current period zonegroup south
2021-06-29T13:14:27.847+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=105
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=46
2021-06-29T13:14:27.848+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.849+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=829
2021-06-29T13:14:27.849+0000 7f4dbb042340 10 Cannot find current period zone using local zone
2021-06-29T13:14:27.849+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.849+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=829
2021-06-29T13:14:27.849+0000 7f4dbb042340 20 zone blr found
2021-06-29T13:14:27.849+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 init(): pipe_map (zone=b6caae11-7796-4265-8133-374bf1992f5a bucket=*): adding potential pipe: {id=all,s={b=*,z=},d={b=*,z=}}
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0},d={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0}}
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0},d={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0}}
2021-06-29T13:14:27.850+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 rados_obj.operate() r=-2 bl.length=0
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 init(): pipe_map (zone=b76458b4-55bc-41b4-8610-b4aa5df49661 bucket=*): adding potential pipe: {id=all,s={b=*,z=},d={b=*,z=}}
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0},d={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0}}
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0},d={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0}}
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0},d={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0}}
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=b76458b4-55bc-41b4-8610-b4aa5df49661,az=0},d={b=*,z=b6caae11-7796-4265-8133-374bf1992f5a,az=0}}
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 started sync module instance, tier type = 
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 generating connection object for zone pune id b6caae11-7796-4265-8133-374bf1992f5a
2021-06-29T13:14:27.851+0000 7f4dbb042340 20 started zone id=b76458b4-55bc-41b4-8610-b4aa5df49661 (name=blr) with tier type = 
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=0
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=1
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=2
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=3
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=4
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=5
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=6
2021-06-29T13:14:27.900+0000 7f4dbb042340 20 add_watcher() i=7
2021-06-29T13:14:27.900+0000 7f4dbb042340  2 all 8 watchers are set, enabling cache
2021-06-29T13:14:27.901+0000 7f4d857e2700  2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start
2021-06-29T13:14:27.902+0000 7f4dbb042340 20 check_secure_mon_conn(): auth registy supported: methods=[2,1] modes=[2,1]
2021-06-29T13:14:27.902+0000 7f4dbb042340 20 check_secure_mon_conn(): method 1 is insecure
2021-06-29T13:14:27.910+0000 7f4dbb042340  5 note: GC not initialized
2021-06-29T13:14:27.913+0000 7f4cf3fff700 20 reqs_thread_entry: start
2021-06-29T13:14:27.913+0000 7f4cf1ffb700 20 reqs_thread_entry: start
2021-06-29T13:14:27.913+0000 7f4dbb042340 20 init_complete bucket index max shards: 11
2021-06-29T13:14:27.914+0000 7f4dbb042340 10 rgw notify: Started notification manager with: 1 workers
2021-06-29T13:14:27.914+0000 7f4cdffff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jun 29 13:14:58 2021

2021-06-29T13:14:27.914+0000 7f4cdf7fe700 20 reqs_thread_entry: start
2021-06-29T13:14:27.914+0000 7f4dbb042340 20 get_system_obj_state: rctx=0x7fffc5ef8430 obj=blr.rgw.meta:root:ver1 state=0x55768941cec0 s->prefetch_data=0
2021-06-29T13:14:27.914+0000 7f4dbb042340 20 get_system_obj_state: s->obj_tag was set empty
2021-06-29T13:14:27.914+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=170
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 get_system_obj_state: rctx=0x7fffc5ef8430 obj=blr.rgw.meta:root:.bucket.meta.ver1:b76458b4-55bc-41b4-8610-b4aa5df49661.25021.1 state=0x5576893f29f0 s->prefetch_data=0
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 get_system_obj_state: s->obj_tag was set empty
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 Read xattr: user.rgw.acl
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 Read xattr: user.rgw.idtag
2021-06-29T13:14:27.915+0000 7f4dbb042340 20 rados->read ofs=0 len=0
2021-06-29T13:14:27.916+0000 7f4dbb042340 20 rados_obj.operate() r=0 bl.length=355
2021-06-29T13:14:27.916+0000 7f4dbb042340 20 couldn't put binfo cache entry, might have raced with data changes
2021-06-29T13:14:27.916+0000 7f4dbb042340 20 bucket index object: blr.rgw.buckets.index:.dir.b76458b4-55bc-41b4-8610-b4aa5df49661.25021.1.0
ERROR: bi_list(): (5) Input/output error
2021-06-29T13:14:32.468+0000 7f4dbb042340 20 remove_watcher() i=0
2021-06-29T13:14:32.468+0000 7f4dbb042340 2 removed watcher, disabling cache
2021-06-29T13:14:32.470+0000 7f4dbb042340 20 remove_watcher() i=1
2021-06-29T13:14:32.471+0000 7f4dbb042340 20 remove_watcher() i=2
2021-06-29T13:14:32.473+0000 7f4dbb042340 20 remove_watcher() i=3
2021-06-29T13:14:32.475+0000 7f4dbb042340 20 remove_watcher() i=4
2021-06-29T13:14:32.476+0000 7f4dbb042340 20 remove_watcher() i=5
2021-06-29T13:14:32.478+0000 7f4dbb042340 20 remove_watcher() i=6
2021-06-29T13:14:32.480+0000 7f4dbb042340 20 remove_watcher() i=7
2021-06-29T13:14:32.481+0000 7f4d857e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start