Project

General

Profile

Bug #41392

ceph-volume can't do a 'simple scan' on a running osd when dmcrypt is enabled

Added by Guillaume Abrioux 9 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

ceph-volume isn't able to do a 'simple scan' on a dmcrypt running osd:

-bash-4.2# docker exec -ti -e CEPH_VOLUME_DEBUG=1 ceph-osd-5 ceph-volume simple scan /dev/sda1 --stdout
Running command: /usr/sbin/cryptsetup status /dev/sda1
Running command: /usr/sbin/cryptsetup status 59239dbe-ca08-4ae4-b903-330d0b8c8361
Running command: /bin/mount -v /dev/sda5 /tmp/tmpLc9mZg
stdout: mount: /dev/sda5 mounted on /tmp/tmpLc9mZg.
Running command: /usr/sbin/cryptsetup status /dev/sda5
Running command: /bin/ceph --cluster ceph --name client.osd-lockbox.59239dbe-ca08-4ae4-b903-330d0b8c8361 --keyring /tmp/tmpLc9mZg/keyring config-key get dm-crypt/osd/59239dbe-ca08-4ae4-b903-330d0b8c8361/luks
Running command: /bin/umount -v /tmp/tmpLc9mZg
stderr: umount: /tmp/tmpLc9mZg (/dev/sda5) unmounted
Running command: /bin/mount -v /dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361 /tmp/tmpWfry1n
stdout: mount: /dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361 mounted on /tmp/tmpWfry1n.
Running command: /usr/sbin/cryptsetup status /tmp/tmpWfry1n/journal_dmcrypt
Running command: /usr/sbin/cryptsetup status /tmp/tmpWfry1n/journal
Running command: /usr/sbin/cryptsetup status /dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361
Running command: /bin/umount -v /tmp/tmpWfry1n
stderr: umount: /tmp/tmpWfry1n (/dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361) unmounted
Running command: /usr/sbin/cryptsetup remove /dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361
stderr: Device /dev/mapper/59239dbe-ca08-4ae4-b903-330d0b8c8361 is still in use.
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 9, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 38, in init
self.main(self.argv)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 206, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line 33, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 206, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 376, in main
self.scan(args)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 237, in scan
osd_metadata = self.scan_encrypted()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", line 203, in scan_encrypted
osd_metadata = self.scan_directory(device_path)
File "/usr/lib/python2.7/site-packages/ceph_volume/util/system.py", line 158, in exit
encryption.dmcrypt_close(self.device)
File "/usr/lib/python2.7/site-packages/ceph_volume/util/encryption.py", line 106, in dmcrypt_close
process.run(['cryptsetup', 'remove', mapping])
File "/usr/lib/python2.7/site-packages/ceph_volume/process.py", line 153, in run
raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: 5
-bash-4.2#

where it's possible on a non-dmcrypt running osd:

-bash-4.2# docker exec ceph-osd-14 ceph-volume simple scan /dev/sda1 --stdout                                                                                                                                                                                                                                                                               
Running command: /usr/sbin/cryptsetup status /dev/sda1 {
"active": "ok",
"ceph_fsid": "b295b2c9-241d-4399-a3b2-3dc00b5235f3",
"cluster_name": "ceph",
"data": {
"path": "/dev/sda1",
"uuid": "8df70b89-e1b5-4aa9-9968-4334cce7af2a"
},
"fsid": "8df70b89-e1b5-4aa9-9968-4334cce7af2a",
"journal": {
"path": "/dev/sda2",
"uuid": "5eb0e752-6d8f-4a20-bf2c-2828f2f42e28"
},
"journal_uuid": "5eb0e752-6d8f-4a20-bf2c-2828f2f42e28",
"keyring": "AQBPY1xdzqMDHhAAZPEOmhLgrIGh2qggqYbAJg==",
"magic": "ceph osd volume v026",
"none": "",
"ready": "ready",
"require_osd_release": 14,
"type": "filestore",
"whoami": 14
}
-bash-4.2#

Behavior should be the same regardless if dmcrypt is enabled.

seen on ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)

ceph-volume.log View - ceph-volume log (249 KB) Guillaume Abrioux, 09/20/2019 12:23 PM

History

#1 Updated by Alfredo Deza 8 months ago

  • Status changed from New to Need More Info

Looks like the call to cryptsetup is failing:

process.run(['cryptsetup', 'remove', mapping])

Could you add the logs from /var/log/ceph/ceph-volume.log when this happens? we might be able to get some extra information as to what is crypsetup complaining about

#2 Updated by Guillaume Abrioux 8 months ago

#3 Updated by Guillaume Abrioux 8 months ago

I guess we can close this since https://github.com/ceph/ceph/pull/30490 has been merged

#4 Updated by Alfredo Deza 8 months ago

  • Status changed from Need More Info to Resolved

Also available in: Atom PDF