Project

General

Profile

Bug #23874

failed ceph-osd --mkfs command doesn't halt the OSD creation process

Added by Alfredo Deza almost 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This happens because the two helper functions use `process.call` without checking the exit status. The usage of `process.call` is necessary because
`stdin` is being sent. Otherwise the recommended way for automatic capturing would be `process.run`.

To fix this, the returncode has to be caught and dealt with

root@node9:/home/vagrant# ceph-volume lvm create --bluestore --data ceph/partial
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1d8ad0ef-0ac0-4cd7-b6ad-54c66584b744
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2
Running command: chown -R ceph:ceph /dev/dm-4
Running command: ln -s /dev/ceph/partial /var/lib/ceph/osd/ceph-2/block
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap
 stderr: got monmap epoch 1
Running command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQDSlN9aAFd+DBAA5ExPXoqzqSsh/WtnQNws2w==
 stdout: creating /var/lib/ceph/osd/ceph-2/keyring
 stdout: added entity osd.2 auth auth(auid = 18446744073709551615 key=AQDSlN9aAFd+DBAA5ExPXoqzqSsh/WtnQNws2w== with 0 caps)
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 1d8ad0ef-0ac0-4cd7-b6ad-54c66584b744 --setuser ceph --setgroup ceph
 stderr: 2018-04-24 23:34:42.351770 7effca0d0e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid
 stderr: /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread 7effca0d0e00 time 2018-04-24 23:34:42.355853
 stderr: /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 172: FAILED assert(bdev[id]->get_size() >= offset + length)
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55c096400a02]
 stderr: 2: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 3: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 4: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 6: (main()+0x11a7) [0x55c095d34147]
 stderr: 7: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 8: (_start()+0x29) [0x55c095dc1659]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: 2018-04-24 23:34:42.365641 7effca0d0e00 -1 /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread 7effca0d0e00 time 2018-04-24 23:34:42.355853
 stderr: /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 172: FAILED assert(bdev[id]->get_size() >= offset + length)
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55c096400a02]
 stderr: 2: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 3: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 4: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 6: (main()+0x11a7) [0x55c095d34147]
 stderr: 7: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 8: (_start()+0x29) [0x55c095dc1659]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: -12> 2018-04-24 23:34:42.351770 7effca0d0e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid
 stderr: 0> 2018-04-24 23:34:42.365641 7effca0d0e00 -1 /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread 7effca0d0e00 time 2018-04-24 23:34:42.355853
 stderr: /build/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 172: FAILED assert(bdev[id]->get_size() >= offset + length)
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55c096400a02]
 stderr: 2: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 3: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 4: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 5: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 6: (main()+0x11a7) [0x55c095d34147]
 stderr: 7: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 8: (_start()+0x29) [0x55c095dc1659]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: *** Caught signal (Aborted) **
 stderr: in thread 7effca0d0e00 thread_name:ceph-osd
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (()+0xa7cab4) [0x55c0963bdab4]
 stderr: 2: (()+0x11390) [0x7effc85b2390]
 stderr: 3: (gsignal()+0x38) [0x7effc754d428]
 stderr: 4: (abort()+0x16a) [0x7effc754f02a]
 stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55c096400b8e]
 stderr: 6: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 7: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 8: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 10: (main()+0x11a7) [0x55c095d34147]
 stderr: 11: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 12: (_start()+0x29) [0x55c095dc1659]
 stderr: 2018-04-24 23:34:42.369278 7effca0d0e00 -1 *** Caught signal (Aborted) **
 stderr: in thread 7effca0d0e00 thread_name:ceph-osd
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (()+0xa7cab4) [0x55c0963bdab4]
 stderr: 2: (()+0x11390) [0x7effc85b2390]
 stderr: 3: (gsignal()+0x38) [0x7effc754d428]
 stderr: 4: (abort()+0x16a) [0x7effc754f02a]
 stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55c096400b8e]
 stderr: 6: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 7: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 8: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 10: (main()+0x11a7) [0x55c095d34147]
 stderr: 11: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 12: (_start()+0x29) [0x55c095dc1659]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 stderr: 0> 2018-04-24 23:34:42.369278 7effca0d0e00 -1 *** Caught signal (Aborted) **
 stderr: in thread 7effca0d0e00 thread_name:ceph-osd
 stderr: ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 stderr: 1: (()+0xa7cab4) [0x55c0963bdab4]
 stderr: 2: (()+0x11390) [0x7effc85b2390]
 stderr: 3: (gsignal()+0x38) [0x7effc754d428]
 stderr: 4: (abort()+0x16a) [0x7effc754f02a]
 stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55c096400b8e]
 stderr: 6: (BlueFS::add_block_extent(unsigned int, unsigned long, unsigned long)+0x4da) [0x55c0963824aa]
 stderr: 7: (BlueStore::_open_db(bool)+0x144d) [0x55c09628eecd]
 stderr: 8: (BlueStore::mkfs()+0x126d) [0x55c0962c440d]
 stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x164) [0x55c095e0f824]
 stderr: 10: (main()+0x11a7) [0x55c095d34147]
 stderr: 11: (__libc_start_main()+0xf0) [0x7effc7538830]
 stderr: 12: (_start()+0x29) [0x55c095dc1659]
 stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--> ceph-volume lvm prepare successful for: ceph/partial
Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph/partial --path /var/lib/ceph/osd/ceph-2
Running command: ln -snf /dev/ceph/partial /var/lib/ceph/osd/ceph-2/block
Running command: chown -R ceph:ceph /dev/dm-4
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: systemctl enable ceph-volume@lvm-2-1d8ad0ef-0ac0-4cd7-b6ad-54c66584b744
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-1d8ad0ef-0ac0-4cd7-b6ad-54c66584b744.service to /lib/systemd/system/ceph-volume@.service.
Running command: systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
--> ceph-volume lvm create successful for: ceph/partial

History

#1 Updated by Alfredo Deza almost 6 years ago

  • Status changed from New to Fix Under Review

#2 Updated by Alfredo Deza almost 6 years ago

  • Status changed from Fix Under Review to Resolved

merged commit 2f15a4f into ceph:master
merged commit 1f81f78 into ceph:luminous

Also available in: Atom PDF