https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2017-04-26T11:42:56ZCeph rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=903612017-04-26T11:42:56ZDenis Horbunov
<ul></ul><p>Hi there. I've also stambled upon this issue. My ceph cluster was migrated from 10.2.5 to 11.2.0 (ceph --version<br />ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)) last week. I was aware about this issue in advance and all snapshots were deleted prior to the migration as a precaution meassure but one image has a undeleteable shapshot taken after migration to kraken.<br />- # rbd info oneimages-pool/one-73<br />rbd image 'one-73':<br /> size 200 GB in 51200 objects<br /> order 22 (4096 kB objects)<br /> block_name_prefix: rbd_data.10425036238e1f29<br /> format: 2<br /> features: layering, exclusive-lock<br /> flags:</p>
<p>- # rbd snap ls oneimages-pool/one-73<br />SNAPID NAME SIZE <br /> 35637 snapmirror.1 200 GB <br /> 35704 snapmirror.2 200 GB</p>
<p>- # rbd snap unprotect <a class="email" href="mailto:oneimages-pool/one-73@snapmirror.1">oneimages-pool/one-73@snapmirror.1</a><br />rbd: snap is already unprotected</p>
<p>- # rbd snap rm <a class="email" href="mailto:oneimages-pool/one-73@snapmirror.1">oneimages-pool/one-73@snapmirror.1</a><br />Removing snap: 0% complete...failed.<br />rbd: failed to remove snapshot: (22) Invalid argument</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=903622017-04-26T11:56:35ZDenis Horbunov
<ul></ul><p>May be this would be more helpfull.</p>
<p>rbd snap rm <a class="email" href="mailto:oneimages-pool/one-73@snapmirror.1">oneimages-pool/one-73@snapmirror.1</a> --debug-rbd 50<br />2017-04-26 13:55:07.388358 7f89d2b88000 5 librbd::AioImageRequestWQ: 0x55887fef8a70 : ictx=0x55887ff677b0<br />2017-04-26 13:55:07.388485 7f89d2b88000 20 librbd::ImageState: 0x55887ff66e80 open<br />2017-04-26 13:55:07.388500 7f89d2b88000 10 librbd::ImageState: 0x55887ff66e80 0x55887ff66e80 send_open_unlock<br />2017-04-26 13:55:07.388519 7f89d2b88000 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_detect_header<br />2017-04-26 13:55:07.390219 7f89ab7fe700 10 librbd::image::OpenRequest: handle_v2_detect_header: r=0<br />2017-04-26 13:55:07.390237 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_get_id<br />2017-04-26 13:55:07.390986 7f89ab7fe700 10 librbd::image::OpenRequest: handle_v2_get_id: r=0<br />2017-04-26 13:55:07.391005 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_get_immutable_metadata<br />2017-04-26 13:55:07.392548 7f89ab7fe700 10 librbd::image::OpenRequest: handle_v2_get_immutable_metadata: r=0<br />2017-04-26 13:55:07.392559 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_get_stripe_unit_count<br />2017-04-26 13:55:07.393214 7f89ab7fe700 10 librbd::image::OpenRequest: handle_v2_get_stripe_unit_count: r=-8<br />2017-04-26 13:55:07.393219 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_get_data_pool<br />2017-04-26 13:55:07.393855 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 handle_v2_get_data_pool: r=0<br />2017-04-26 13:55:07.393870 7f89ab7fe700 10 librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.10425036238e1f29 format rbd_data.10425036238e1f29.%016llx<br />2017-04-26 13:55:07.393877 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_v2_apply_metadata: start_key=conf_<br />2017-04-26 13:55:07.394675 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 handle_v2_apply_metadata: r=0<br />2017-04-26 13:55:07.394694 7f89ab7fe700 20 librbd::ImageCtx: apply_metadata<br />2017-04-26 13:55:07.395198 7f89ab7fe700 20 librbd::ImageCtx: enabling caching...<br />2017-04-26 13:55:07.395203 7f89ab7fe700 20 librbd::ImageCtx: Initial cache settings: size=1024000 num_objects=10 max_dirty=768000 target_dirty=512000 max_dirty_age=1<br />2017-04-26 13:55:07.395290 7f89ab7fe700 10 librbd::ImageCtx: cache bytes 1024000 -> about 32 objects<br />2017-04-26 13:55:07.395330 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_register_watch<br />2017-04-26 13:55:07.395433 7f89ab7fe700 10 librbd::ImageWatcher: 0x7f8998034c50 registering image watcher<br />2017-04-26 13:55:07.401575 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 handle_register_watch: r=0<br />2017-04-26 13:55:07.401586 7f89ab7fe700 10 librbd::image::OpenRequest: 0x55887ff68d70 send_refresh<br />2017-04-26 13:55:07.401589 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_get_mutable_metadata<br />2017-04-26 13:55:07.402798 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_get_mutable_metadata: r=0<br />2017-04-26 13:55:07.402821 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_get_flags<br />2017-04-26 13:55:07.403775 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_get_flags: r=0<br />2017-04-26 13:55:07.403788 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_get_group<br />2017-04-26 13:55:07.404642 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_get_group: r=0<br />2017-04-26 13:55:07.404652 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_get_snapshots<br />2017-04-26 13:55:07.406129 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_get_snapshots: r=0<br />2017-04-26 13:55:07.406143 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_get_snap_namespaces<br />2017-04-26 13:55:07.407092 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_get_snap_namespaces: r=0<br />2017-04-26 13:55:07.407115 7f89ab7fe700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_init_exclusive_lock<br />2017-04-26 13:55:07.407123 7f89ab7fe700 10 librbd::ExclusiveLock: 0x7f8998041710 init<br />2017-04-26 13:55:07.407126 7f89ab7fe700 5 librbd::AioImageRequestWQ: block_writes: 0x55887ff677b0, num=1<br />2017-04-26 13:55:07.407198 7f89aaffd700 10 librbd::ExclusiveLock: 0x7f8998041710 handle_init_complete<br />2017-04-26 13:55:07.407208 7f89aaffd700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_init_exclusive_lock: r=0<br />2017-04-26 13:55:07.407211 7f89aaffd700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_v2_apply<br />2017-04-26 13:55:07.407215 7f89aaffd700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_v2_apply<br />2017-04-26 13:55:07.407222 7f89aaffd700 20 librbd::image::RefreshRequest: 0x7f8998036ad0 apply<br />2017-04-26 13:55:07.407228 7f89aaffd700 20 librbd::image::RefreshRequest: new snapshot id=35704 name=snapmirror.2 size=214748364800<br />2017-04-26 13:55:07.407231 7f89aaffd700 20 librbd::image::RefreshRequest: new snapshot id=35637 name=snapmirror.1 size=214748364800<br />2017-04-26 13:55:07.407244 7f89aaffd700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 send_flush_aio<br />2017-04-26 13:55:07.407249 7f89aaffd700 10 librbd::image::RefreshRequest: 0x7f8998036ad0 handle_flush_aio: r=0<br />Removing snap: 2017-04-26 13:55:07.407254 7f89aaffd700 10 librbd::image::OpenRequest: handle_refresh: r=0<br />2017-04-26 13:55:07.407259 7f89aaffd700 10 librbd::ImageState: 0x55887ff66e80 0x55887ff66e80 handle_open: r=0<br />2017-04-26 13:55:07.407315 7f89d2b88000 20 librbd: snap_remove 0x55887ff677b0 snapmirror.1 flags: 0<br />2017-04-26 13:55:07.407327 7f89d2b88000 20 librbd: get_snap_namespace 0x55887ff677b0 snapmirror.10% complete...failed.</p>
<p>rbd: failed to remove snapshot: (22) Invalid argument<br />2017-04-26 13:55:07.407495 7f89d2b88000 20 librbd::ImageState: 0x55887ff66e80 close<br />2017-04-26 13:55:07.407502 7f89d2b88000 10 librbd::ImageState: 0x55887ff66e80 0x55887ff66e80 send_close_unlock<br />2017-04-26 13:55:07.407504 7f89d2b88000 10 librbd::image::CloseRequest: 0x55887ff68a50 send_shut_down_update_watchers<br />2017-04-26 13:55:07.407505 7f89d2b88000 20 librbd::ImageState: 0x55887ff66e80 shut_down_update_watchers<br />2017-04-26 13:55:07.407506 7f89d2b88000 20 librbd::ImageState: 0x55887ff66f60 ImageUpdateWatchers::shut_down<br />2017-04-26 13:55:07.407508 7f89d2b88000 20 librbd::ImageState: 0x55887ff66f60 ImageUpdateWatchers::shut_down: completing shut down<br />2017-04-26 13:55:07.407542 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_shut_down_update_watchers: r=0<br />2017-04-26 13:55:07.407550 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_unregister_image_watcher<br />2017-04-26 13:55:07.407554 7f89aaffd700 10 librbd::ImageWatcher: 0x7f8998034c50 unregistering image watcher<br />2017-04-26 13:55:07.412588 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_unregister_image_watcher: r=0<br />2017-04-26 13:55:07.412597 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_shut_down_aio_queue<br />2017-04-26 13:55:07.412600 7f89aaffd700 5 librbd::AioImageRequestWQ: shut_down: in_flight=0<br />2017-04-26 13:55:07.412604 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_shut_down_aio_queue: r=0<br />2017-04-26 13:55:07.412607 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_shut_down_exclusive_lock<br />2017-04-26 13:55:07.412608 7f89aaffd700 10 librbd::ExclusiveLock: 0x7f8998041710 shut_down<br />2017-04-26 13:55:07.412612 7f89aaffd700 10 librbd::ExclusiveLock: 0x7f8998041710 handle_shutdown: r=0<br />2017-04-26 13:55:07.412615 7f89aaffd700 20 librbd::AioImageRequestWQ: clear_require_lock_on_read<br />2017-04-26 13:55:07.412617 7f89aaffd700 5 librbd::AioImageRequestWQ: unblock_writes: 0x55887ff677b0, num=0<br />2017-04-26 13:55:07.412628 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_shut_down_exclusive_lock: r=0<br />2017-04-26 13:55:07.412634 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_flush_readahead<br />2017-04-26 13:55:07.412640 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_flush_readahead: r=0<br />2017-04-26 13:55:07.412644 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_shut_down_cache<br />2017-04-26 13:55:07.412700 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_shut_down_cache: r=0<br />2017-04-26 13:55:07.412706 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 send_flush_op_work_queue<br />2017-04-26 13:55:07.412710 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_flush_op_work_queue: r=0<br />2017-04-26 13:55:07.412713 7f89aaffd700 10 librbd::image::CloseRequest: 0x55887ff68a50 handle_flush_image_watcher: r=0<br />2017-04-26 13:55:07.412734 7f89aaffd700 10 librbd::ImageState: 0x55887ff66e80 0x55887ff66e80 handle_close: r=0</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=905962017-05-01T22:46:51ZJason Dillamandillaman@redhat.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>rbd</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>Jason Dillaman</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=905972017-05-01T22:47:01ZJason Dillamandillaman@redhat.com
<ul><li><strong>Backport</strong> set to <i>kraken</i></li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=905982017-05-01T22:56:27ZJason Dillamandillaman@redhat.com
<ul><li><strong>Backport</strong> deleted (<del><i>kraken</i></del>)</li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=905992017-05-02T00:27:30ZJason Dillamandillaman@redhat.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Need More Info</i></li></ul><p>@Denis: can you do me a favor, run the following, and attach the results?</p>
<pre>
# rados -p oneimages-pool listomapvals rbd_header.10425036238e1f29
</pre> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906002017-05-02T01:13:33ZJason Dillamandillaman@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Fix Under Review</i></li><li><strong>Backport</strong> set to <i>kraken</i></li></ul><p><strong>PR</strong>: <a class="external" href="https://github.com/ceph/ceph/pull/14903">https://github.com/ceph/ceph/pull/14903</a></p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906182017-05-02T11:17:42ZBenoit Loriot
<ul></ul><pre><code class="text syntaxhl"><span class="CodeRay"># rados -p qa-volumes listomapvals rbd_data.3d91f429d7ac52
error getting omap keys qa-volumes/rbd_data.3d91f429d7ac52: (2) No such file or directory
</span></code></pre> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906192017-05-02T11:52:52ZJason Dillamandillaman@redhat.com
<ul></ul><p>@Benoit: switch "rbd_data" for "rbd_header" in your command above.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906202017-05-02T12:03:14ZBenoit Loriot
<ul></ul><p>Here is the correct output</p>
<pre><code class="text syntaxhl"><span class="CodeRay"># rados -p qa-volumes listomapvals rbd_header.3d91f429d7ac52 --debug-rbd 50
features
value (8 bytes) :
00000000 3d 00 00 00 00 00 00 00 |=.......|
00000008
object_prefix
value (27 bytes) :
00000000 17 00 00 00 72 62 64 5f 64 61 74 61 2e 33 64 39 |....rbd_data.3d9|
00000010 31 66 34 32 39 64 37 61 63 35 32 |1f429d7ac52|
0000001b
order
value (1 bytes) :
00000000 16 |.|
00000001
size
value (8 bytes) :
00000000 00 00 00 80 02 00 00 00 |........|
00000008
snap_seq
value (8 bytes) :
00000000 e8 01 00 00 00 00 00 00 |........|
00000008
snapshot_00000000000001e5
value (132 bytes) :
00000000 05 01 7e 00 00 00 e5 01 00 00 00 00 00 00 2d 00 |..~...........-.|
00000010 00 00 73 6e 61 70 73 68 6f 74 2d 64 35 64 39 62 |..snapshot-d5d9b|
00000020 38 37 30 2d 32 34 31 36 2d 34 32 30 30 2d 62 64 |870-2416-4200-bd|
00000030 36 64 2d 64 39 61 34 30 62 65 38 39 32 64 36 00 |6d-d9a40be892d6.|
00000040 00 00 80 02 00 00 00 3d 00 00 00 00 00 00 00 01 |.......=........|
00000050 01 1c 00 00 00 ff ff ff ff ff ff ff ff 00 00 00 |................|
00000060 00 fe ff ff ff ff ff ff ff 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 00 00 00 00 01 01 04 00 00 00 |................|
00000080 ff ff ff ff |....|
00000084
</span></code></pre> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906212017-05-02T12:19:01ZJason Dillamandillaman@redhat.com
<ul></ul><p>@Benoit: Great, thanks. I was expecting the problem to be four 0xFF bytes at the end of your snapshot_XYZ value. The associated PR should prevent that issue from occurring in the future. In the meantime, the easiest way to resolve the issue would be to use a pre-Kraken rbd CLI to remove the snapshot. Alternatively, you can use the rados CLI to write the key value to a file, use a hex editor to change the last four bytes to all 0x00, and again use rados CLI to update the value using the contents of said file.</p>
<p>The issue occurred when the OSDs were upgraded to Kraken and a Jewel RBD client created a snapshot. The Kraken OSD incorrectly wrote the four 0xFF values (-1) instead of four 0x00 values since the Jewel RBD client doesn't understand snapshot namespaces. When the RBD client was later upgraded to Kraken, it interpreted that namespace value as an invalid namespace and returns a -EINVAL error code.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906242017-05-02T13:36:37ZBenoit Loriot
<ul></ul><p>Thanks Jason, snapshot deletion worked from a Jewel client.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906602017-05-03T12:25:45ZMykola Golubmgolub@suse.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=906632017-05-03T13:49:57ZJason Dillamandillaman@redhat.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/19833">Backport #19833</a>: kraken: Cannot delete some snapshots after upgrade from jewel to kraken</i> added</li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=907322017-05-04T08:00:16ZDenis Horbunov
<ul></ul><p>I'm sorry for being late. Thank you very much indeed Jason! This solved my problem too.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=979372017-08-30T09:09:20ZRoss Martyn
<ul></ul><p>Using an old client also fixed this issue for me. Glad this has been fixed in 11.2.1. Appreciate the info.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=979412017-08-30T12:19:06ZJason Dillamandillaman@redhat.com
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=985552017-09-07T13:52:46ZLionel BEARD
<ul></ul><p>Hi,</p>
<p>FYI, we just have exactly the same issue when upgrading from Kraken to Luminous (four OxFF).<br />It was fixed by using a jewel RBD client (as kraken RBD client crashed!) to remove snapshot.</p>
<p>I think I probably did something wrong when upgrading. <br />Do I have to mark out OSD before upgrading them? <br />What are the requirement when upgrading clients? After or before OSD? Client must be off?</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=985582017-09-07T14:03:05ZJason Dillamandillaman@redhat.com
<ul></ul><p>@Lionel: the bug was fixed in kraken 11.2.1 but if you had previously created any snapshots on kraken 11.2.0 using jewel or earlier clients, the snapshot header would have already been incorrectly written. If you can still repeat this issue on luminous when creating a new snapshot using a jewel client and attempting to delete the snapshot using a luminous client, let me know.</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=1162022018-07-04T12:38:21ZRoss Martyn
<ul></ul><p>Just seen this issue again, as above FFFF in header info. I was unable to remove with Luminous rbd client (12.2.5), but <b>was</b> able to remove with Mimic client (13.2.0).</p> rbd - Bug #19413: Cannot delete some snapshots after upgrade from jewel to krakenhttps://tracker.ceph.com/issues/19413?journal_id=1164612018-07-07T12:13:07ZJason Dillamandillaman@redhat.com
<ul></ul><p>@Ross: what version of Ceph is your cluster running and what version of Ceph is your client running when you created a snapshot w/ 0xFFFF at the end? Can you provide the hexdump for its data?</p>