Bug #21353: upgrade to luminous results in seemingly corrupt images in QEMU - rbd - Ceph

Actions

Copy link

Bug #21353

closed

upgrade to luminous results in seemingly corrupt images in QEMU

Added by Jason Dillaman over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

High

Assignee:

Jason Dillaman

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Three mailing list users have reported apparent filesystem corrupt post-upgrade to Luminous when using QEMU. Mapping the images via krbd and checking the filesystems showed no corruption and exporting/re-importing the images allowed QEMU to work again.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020648.html

Actions

Copy link

Updated by Jason Dillaman over 6 years ago

Description updated (diff)
Source set to Community (user)

Actions

Copy link

Updated by Jason Dillaman over 6 years ago

Status changed from Need More Info to In Progress
Priority changed from Urgent to High

Related to the Luminous cap enforcement fix for the blacklist op and users caps w/o the required caps for RBD exclusive-lock:

2017-09-11 22:26:38.944452 7efd67fff700 10 librbd::managed_lock::BreakRequest: 0x7efd58020e70 send_blacklist: local entity=client.1039597, locker entity=client.935876
2017-09-11 22:26:38.945761 7efd677fe700 10 librbd::managed_lock::BreakRequest: 0x7efd58020e70 handle_blacklist: r=-13
2017-09-11 22:26:38.945776 7efd677fe700 -1 librbd::managed_lock::BreakRequest: 0x7efd58020e70 handle_blacklist: failed to blacklist lock owner: (13) Permission denied
2017-09-11 22:26:38.945795 7efd677fe700 10 librbd::managed_lock::BreakRequest: 0x7efd58020e70 finish: r=-13
2017-09-11 22:26:38.945798 7efd677fe700 10 librbd::managed_lock::AcquireRequest: 0x7efd60017960 handle_break_lock: r=-13
2017-09-11 22:26:38.945800 7efd677fe700 -1 librbd::managed_lock::AcquireRequest: 0x7efd60017960 handle_break_lock: failed to break lock : (13) Permission denied
2017-09-11 22:26:38.945865 7efd677fe700 10 librbd::ManagedLock: 0x7efd580267d0 handle_acquire_lock: r=-13
2017-09-11 22:26:38.945873 7efd677fe700 -1 librbd::ManagedLock: 0x7efd580267d0 handle_acquire_lock: failed to acquire exclusive lock:(13) Permission denied
2017-09-11 22:26:38.945883 7efd677fe700 10 librbd::ExclusiveLock: 0x7efd580267d0 post_acquire_lock_handler: r=-13
2017-09-11 22:26:38.945887 7efd677fe700 10 librbd::ImageState: 0x55b55ace8dc0 handle_prepare_lock_complete
2017-09-11 22:26:38.945892 7efd677fe700 10 librbd::ManagedLock: 0x7efd580267d0 handle_post_acquire_lock: r=-13
2017-09-11 22:26:38.945895 7efd677fe700  5 librbd::io::ImageRequestWQ: 0x55b55ace9a20 handle_acquire_lock: r=-13, req=0x55b55add32a0
2017-09-11 22:26:38.945901 7efd677fe700 -1 librbd::io::AioCompletion: 0x55b55add46a0 fail: (13) Permission denied

Actions

Copy link

Updated by Jason Dillaman over 6 years ago

Prior to upgrading the monitors to the Luminous release, verify that
your RBD users have at least the following cap:

$ ceph auth get client.<RBD ID> 2>&1 | grep mon
caps mon = "allow r, allow command "osd blacklist""

Post upgrade to the Luminous release, you can future-proof your RBD
caps by updating to the following:

$ ceph auth get client.rbd_read_write
exported keyring for client.test
[client.rbd_read_write]
    key = AQCyzbdZ13EVARAAt7EpNOt4C911Q3CEtBiCyw==
    caps mon = "profile rbd" 
    caps osd = "profile rbd pool=xyz"

$ ceph auth get client.rbd_read_only
exported keyring for client.test
[client.rbd_read_only]
    key = AQCyzbdZ13EVARAAt7EpNOt4C911Q3CEtBiCyw==
    caps mon = "profile rbd" 
    caps osd = "profile rbd-read-only"

[1] http://docs.ceph.com/docs/master/rados/operations/user-management/#authorization-capabilities

Actions

Copy link