Actions
Bug #23628
opendeleting zonegroup doesn't remove it from the period
Added by Orit Wasserman about 6 years ago. Updated about 6 years ago.
% Done:
0%
Source:
Tags:
Backport:
luminous, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
radosgw-admin realm create --rgw-realm=myrealm --default
$ radosgw-admin realm get --rgw-realm=my_realm
$ radosgw-admin zonegroup create --rgw-zonegroup=my_zg --rgw-realm=my_realm —master --default
$ radosgw-admin zone create --rgw-zone=myzone --zonegroup=my_zg --master --default
$ radosgw-admin period update --commit
$ radosgw-admin zonegroup add --rgw-zonegroup=my_zg --rgw-zone=myzone
$ radosgw-admin period update --commit
Deletion procedure:
$ radosgw-admin zonegroup remove --rgw-zonegroup=my_zg --rgw-zone=myzone
$ radosgw-admin period update --commit
$ radosgw-admin zone delete --rgw-zone=myzone
$ radosgw-admin period update --commit
$ radosgw-admin zonegroup delete --rgw-zonegroup=my_zg
$ radosgw-admin period update --commit
when you do radsogw-admin period get you still see my_zg
trying to change default zonegroup fails:
radosgw-admin zonegroup default --rgw-zonegroup=default
radosgw-admin period update --commit --rgw-zonegroup=default
2018-03-22 16:04:03.144716 7f900f53cc40 0 zonegroup my_zg missing zone for master_zone=794b5a26-5984-45be-a782-c17798fcf6dc
couldn't init storage provider
The zonegroup and zone are not in .rgw.root or in the output of zone list and zonegroup commands
Updated by Jérôme Poulin about 6 years ago
Interestingly, I hit the same issue today. Cannot commit period anymore and the whole realm is broken since I cannot even list users now.
radosgw-admin --cluster dev period commit --debug-rgw 20 2018-04-10 17:45:03.837759 7f4a437fe700 2 RGWDataChangesLog::ChangesRenewThread: start 2018-04-10 17:45:03.837767 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e96b0 obj=.rgw.root:default.realm state=0x559334324c80 s->prefetch_data=0 2018-04-10 17:45:03.842142 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.842152 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.843117 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.843147 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9730 obj=.rgw.root:realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x559334324c80 s->prefetch_data=0 2018-04-10 17:45:03.846267 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.846277 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.847396 7f4a5f384cc0 20 rados->read r=0 bl.length=109 2018-04-10 17:45:03.847415 7f4a5f384cc0 20 realm cauca-dev 57ee8784-e9f2-4c4a-914d-5ca1689cf042 2018-04-10 17:45:03.847436 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9520 obj=.rgw.root:realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x559334324c80 s->prefetch_data=0 2018-04-10 17:45:03.848548 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.848555 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.849633 7f4a5f384cc0 20 rados->read r=0 bl.length=109 2018-04-10 17:45:03.849652 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9690 obj=.rgw.root:periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch state=0x559334324c40 s->prefetch_data=0 2018-04-10 17:45:03.850703 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.850711 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.851656 7f4a5f384cc0 20 rados->read r=0 bl.length=10 2018-04-10 17:45:03.851682 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e96b0 obj=.rgw.root:periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.3 state=0x5593343309e0 s->prefetch_data=0 2018-04-10 17:45:03.852738 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.852746 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.853860 7f4a5f384cc0 20 rados->read r=0 bl.length=2023 2018-04-10 17:45:03.853911 7f4a5f384cc0 20 current period 2bd78f26-d8c7-4797-9dd3-38edecaa4ac3 2018-04-10 17:45:03.853921 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e8e00 obj=.rgw.root:converted state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.855157 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e8380 obj=.rgw.root:default.realm state=0x559334333af0 s->prefetch_data=0 2018-04-10 17:45:03.856176 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.856183 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.857198 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.857215 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e8400 obj=.rgw.root:realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x559334333a70 s->prefetch_data=0 2018-04-10 17:45:03.858316 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.858324 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.859385 7f4a5f384cc0 20 rados->read r=0 bl.length=109 2018-04-10 17:45:03.859407 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e8510 obj=.rgw.root:default.region state=0x559334333880 s->prefetch_data=0 2018-04-10 17:45:03.873756 7f4a5f384cc0 20 RGWRados::pool_iterate: got default.zone.57ee8784-e9f2-4c4a-914d-5ca1689cf042 2018-04-10 17:45:03.873783 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.5 2018-04-10 17:45:03.873786 7f4a5f384cc0 20 RGWRados::pool_iterate: got realms_names.cauca-dev 2018-04-10 17:45:03.873788 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.2 2018-04-10 17:45:03.873789 7f4a5f384cc0 20 RGWRados::pool_iterate: got zone_info.3afc016f-6b54-47d8-9bea-fec00bb32441 2018-04-10 17:45:03.873791 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.6 2018-04-10 17:45:03.873792 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.526fd047-49ea-40e3-9eef-9fe281436977.1 2018-04-10 17:45:03.873802 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch 2018-04-10 17:45:03.873806 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.526fd047-49ea-40e3-9eef-9fe281436977.latest_epoch 2018-04-10 17:45:03.873808 7f4a5f384cc0 20 RGWRados::pool_iterate: got period_config.57ee8784-e9f2-4c4a-914d-5ca1689cf042 2018-04-10 17:45:03.873809 7f4a5f384cc0 20 RGWRados::pool_iterate: got default.realm 2018-04-10 17:45:03.873811 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.latest_epoch 2018-04-10 17:45:03.873813 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.57ee8784-e9f2-4c4a-914d-5ca1689cf042:staging 2018-04-10 17:45:03.873814 7f4a5f384cc0 20 RGWRados::pool_iterate: got realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 2018-04-10 17:45:03.873816 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.1 2018-04-10 17:45:03.873817 7f4a5f384cc0 20 RGWRados::pool_iterate: got zone_info.22169160-8020-4d03-8054-ea09b590108f 2018-04-10 17:45:03.873819 7f4a5f384cc0 20 RGWRados::pool_iterate: got zonegroup_info.f7adb718-61e6-434e-b870-f46275823205 2018-04-10 17:45:03.873822 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.1 2018-04-10 17:45:03.873824 7f4a5f384cc0 20 RGWRados::pool_iterate: got zonegroups_names.dev 2018-04-10 17:45:03.873826 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.2 2018-04-10 17:45:03.873827 7f4a5f384cc0 20 RGWRados::pool_iterate: got zone_names.dev2 2018-04-10 17:45:03.873829 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.3 2018-04-10 17:45:03.873830 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.3 2018-04-10 17:45:03.873834 7f4a5f384cc0 20 RGWRados::pool_iterate: got realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042.control 2018-04-10 17:45:03.873836 7f4a5f384cc0 20 RGWRados::pool_iterate: got default.zonegroup.57ee8784-e9f2-4c4a-914d-5ca1689cf042 2018-04-10 17:45:03.873838 7f4a5f384cc0 20 RGWRados::pool_iterate: got periods.382bf99e-a8c4-4485-b17e-e98089bf9db5.4 2018-04-10 17:45:03.873840 7f4a5f384cc0 20 RGWRados::pool_iterate: got zone_names.dev1 2018-04-10 17:45:03.873868 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e84d0 obj=.rgw.root:zone_names.default state=0x55933433c450 s->prefetch_data=0 2018-04-10 17:45:03.875546 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e84d0 obj=.rgw.root:zonegroups_names.default state=0x55933433c450 s->prefetch_data=0 2018-04-10 17:45:03.876553 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e98c0 obj=.rgw.root:region_map state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.877722 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9080 obj=.rgw.root:default.realm state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.878772 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.878791 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.879845 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.879894 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9100 obj=.rgw.root:realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.881143 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.881163 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.882337 7f4a5f384cc0 20 rados->read r=0 bl.length=109 2018-04-10 17:45:03.882388 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9210 obj=.rgw.root:default.zonegroup.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.883351 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.883374 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.884327 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.884379 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9360 obj=.rgw.root:zonegroup_info.f7adb718-61e6-434e-b870-f46275823205 state=0x559334332ef0 s->prefetch_data=0 2018-04-10 17:45:03.885563 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.885583 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.886580 7f4a5f384cc0 20 rados->read r=0 bl.length=553 2018-04-10 17:45:03.886635 7f4a5f384cc0 20 period zonegroup init ret 0 2018-04-10 17:45:03.886638 7f4a5f384cc0 20 period zonegroup name dev 2018-04-10 17:45:03.886640 7f4a5f384cc0 20 using current period zonegroup dev 2018-04-10 17:45:03.886658 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9080 obj=.rgw.root:default.realm state=0x55933433c9e0 s->prefetch_data=0 2018-04-10 17:45:03.887773 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.887799 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.889007 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.889086 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9100 obj=.rgw.root:realms.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x55933433c9e0 s->prefetch_data=0 2018-04-10 17:45:03.890447 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.890470 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.891690 7f4a5f384cc0 20 rados->read r=0 bl.length=109 2018-04-10 17:45:03.891743 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9210 obj=.rgw.root:default.zone.57ee8784-e9f2-4c4a-914d-5ca1689cf042 state=0x55933433c9e0 s->prefetch_data=0 2018-04-10 17:45:03.892830 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.892849 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.893789 7f4a5f384cc0 20 rados->read r=0 bl.length=46 2018-04-10 17:45:03.893835 7f4a5f384cc0 20 get_system_obj_state: rctx=0x7ffd941e9360 obj=.rgw.root:zone_info.3afc016f-6b54-47d8-9bea-fec00bb32441 state=0x55933433c9e0 s->prefetch_data=0 2018-04-10 17:45:03.894946 7f4a5f384cc0 20 get_system_obj_state: s->obj_tag was set empty 2018-04-10 17:45:03.894966 7f4a5f384cc0 20 rados->read ofs=0 len=524288 2018-04-10 17:45:03.896231 7f4a5f384cc0 20 rados->read r=0 bl.length=733 2018-04-10 17:45:03.896294 7f4a5f384cc0 0 zonegroup devB missing zone for master_zone=b5f63d95-4890-498d-b0dd-b6e9aa2bac63 couldn't init storage provider
Updated by Jérôme Poulin about 6 years ago
I absolutely couldn't find a way to rollback a period, but I was able to "fix" the problem by manually rollbacking my period using:
Stopping all radosgw instance.
Getting a copy of current period.
rados --cluster dev -p .rgw.root get periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.1 periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.1 rados --cluster dev -p .rgw.root get periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.2 periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.2 rados --cluster dev -p .rgw.root get periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.3 periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.3 rados --cluster dev -p .rgw.root get periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch
Using ghex periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch to modify the epoch from 0x03 to 0x01 at offset 0x6.
Removing all newer epochs and swapping the latest epoch to my modified one.
rados --cluster dev -p .rgw.root rm periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.3 rados --cluster dev -p .rgw.root rm periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.2 rados --cluster dev -p .rgw.root put periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch periods.2bd78f26-d8c7-4797-9dd3-38edecaa4ac3.latest_epoch
Starting radosgw and everything is back up. Of course, if I had other modifications pending, that would be unacceptable.
Actions