Ceph » bluestore

ceph-osd luminous crashes on startup after emulating power loss (via magic sysrq reboot):

Aug 22 13:27:47 dpr-2a1702-061-crd systemd¹: Started Ceph object storage daemon osd.ceph.36.
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.352925 7fe7efb49d80 -1 WARNING: the following dangerous and experimental features are enabled: btrfs
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.353097 7fe7efb49d80 -1 WARNING: the following dangerous and experimental features are enabled: btrfs
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.353105 7fe7efb49d80 0 set uid:gid to 167:167 (ceph:ceph)
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.353114 7fe7efb49d80 0 ceph version 12.1.2-463-gbdaf63e (bdaf63e2951e0d23440c19c3ab075e48cf65132b) luminous (rc), process
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: starting osd.36 at - osd_data /var/lib/ceph/osd/ceph-36 /var/lib/ceph/osd/ceph-36/journal
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.373036 7fe7efb49d80 0 pidfile_write: ignore empty --pid-file
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.429496 7fe7efb49d80 -1 WARNING: the following dangerous and experimental features are enabled: btrfs
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.435158 7fe7efb49d80 0 load: jerasure load: lrc load: isa
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733362 7fe7efb49d80 0 set rocksdb option compaction_readahead_size = 2097152
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733380 7fe7efb49d80 0 set rocksdb option compression = kNoCompression
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733385 7fe7efb49d80 0 set rocksdb option max_write_buffer_number = 4
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733395 7fe7efb49d80 0 set rocksdb option min_write_buffer_number_to_merge = 1
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733399 7fe7efb49d80 0 set rocksdb option recycle_log_file_num = 4
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733402 7fe7efb49d80 0 set rocksdb option writable_file_max_buffer_size = 0
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733406 7fe7efb49d80 0 set rocksdb option write_buffer_size = 268435456
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733430 7fe7efb49d80 0 set rocksdb option compaction_readahead_size = 2097152
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733435 7fe7efb49d80 0 set rocksdb option compression = kNoCompression
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733438 7fe7efb49d80 0 set rocksdb option max_write_buffer_number = 4
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733441 7fe7efb49d80 0 set rocksdb option min_write_buffer_number_to_merge = 1
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733444 7fe7efb49d80 0 set rocksdb option recycle_log_file_num = 4
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733447 7fe7efb49d80 0 set rocksdb option writable_file_max_buffer_size = 0
Aug 22 13:27:47 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:27:47.733450 7fe7efb49d80 0 set rocksdb option write_buffer_size = 268435456
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:29:28.193934 7fe7efb49d80 -1 bluefs allocate failed to allocate 0x100000 on bdev 2, dne
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:29:28.193940 7fe7efb49d80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x1a19
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:29:28.389892 7fe7efb49d80 -1 rocksdb: Corruption: file is too short to be an sstable
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:29:28.389909 7fe7efb49d80 -1 bluestore(/var/lib/ceph/osd/ceph-36) _open_db erroring opening db:
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2017-08-22 13:29:28.390166 7fe7efb49d80 -1 bluefs _allocate failed to allocate 0x400000 on bdev 2, dne/root/rpmbuild/BUILD/ceph-12.1.2-463-gb
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: /root/rpmbuild/BUILD/ceph-12.1.2-463-gbdaf63e/src/os/bluestore/BlueFS.cc: 1198: FAILED assert(r == 0)
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: ceph version 12.1.2-463-gbdaf63e (bdaf63e2951e0d23440c19c3ab075e48cf65132b) luminous (rc)
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 1: (ceph::_ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x5604fdb166a0]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 2: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0x1831) [0x5604fda9d9c1]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 3: (BlueFS::sync_metadata()+0x545) [0x5604fda9e045]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 4: (BlueFS::umount()+0x74) [0x5604fda9e234]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 5: (BlueStore::_open_db(bool)+0x568) [0x5604fd9c4af8]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 6: (BlueStore::_mount(bool)+0x3fe) [0x5604fd9f6d7e]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 7: (OSD::init()+0x3ba) [0x5604fd5c202a]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 8: (main()+0x2def) [0x5604fd4c97ef]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 9: (__libc_start_main()+0xf5) [0x7fe7ec01baf5]
Aug 22 13:29:28 dpr-2a1702-061-crd ceph-custom-daemon¹³⁷⁹²: 10: (()+0x4abb29) [0x5604fd567b29]