2019-03-05 12:37:26.922 7fa4af337240 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-03-05 12:37:26.922 7fa4af337240 0 ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable), process ceph-osd, pid 71549 2019-03-05 12:37:26.922 7fa4af337240 0 pidfile_write: ignore empty --pid-file 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) set_cache_shards 1 2019-03-05 12:37:26.926 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs path /var/lib/ceph/osd/ceph-0/ 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label 2019-03-05 12:37:26.926 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label 2019-03-05 12:37:26.926 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label 2019-03-05 12:37:26.930 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding 2019-03-05 12:37:26.930 7fa4af337240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid 2019-03-05 12:37:26.930 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs using provided fsid 889f02eb-fe94-4829-a11c-b87ed81c5936 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block path size 10737418240 create=1 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block.wal path size 100663296 create=0 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block.db path size 0 create=0 2019-03-05 12:37:26.930 7fa4af337240 1 bdev create path /var/lib/ceph/osd/ceph-0//block type kernel 2019-03-05 12:37:26.930 7fa4af337240 1 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open path /var/lib/ceph/osd/ceph-0//block 2019-03-05 12:37:26.930 7fa4af337240 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open devname dm-0 2019-03-05 12:37:26.930 7fa4af337240 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _detect_vdo no VDO volume maps to dm-0 2019-03-05 12:37:26.930 7fa4af337240 10 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _aio_start 2019-03-05 12:37:26.930 7fa4af337240 1 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational 2019-03-05 12:37:26.930 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _write_bdev_label path /var/lib/ceph/osd/ceph-0//block label bdev(osd_uuid 889f02eb-fe94-4829-a11c-b87ed81c5936, size 0x3a381400000, btime 2019-03-05 12:37:26.933971, desc main, 0 meta) 2019-03-05 12:37:26.930 7fa4a0a89700 10 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _aio_thread start 2019-03-05 12:37:26.930 7fa49eaaf700 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _discard_thread sleep 2019-03-05 12:37:26.934 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db kv_backend = rocksdb 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db do_bluefs = 1 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db initializing bluefs 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0//block 2019-03-05 12:37:26.934 7fa4af337240 1 bdev create path /var/lib/ceph/osd/ceph-0//block type kernel 2019-03-05 12:37:26.934 7fa4af337240 1 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open path /var/lib/ceph/osd/ceph-0//block 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open devname dm-0 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _detect_vdo no VDO volume maps to dm-0 2019-03-05 12:37:26.934 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_start 2019-03-05 12:37:26.934 7fa4af337240 1 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0//block size 3.6 TiB 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs add_block_extent bdev 1 0x1bf1f500000~2542a00000 2019-03-05 12:37:26.934 7fa49e2ae700 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_thread start 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs add_block_extent done 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs mkfs osd_uuid 889f02eb-fe94-4829-a11c-b87ed81c5936 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _init_alloc 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 init_add_free 0x1bf1f500000~2542a00000 2019-03-05 12:37:26.934 7fa49daad700 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _discard_thread sleep 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs mkfs uuid 06421aba-924a-4dd8-8320-bde13e4b9ca4 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _allocate len 0x400000 from 0 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _allocate len 0x400000 from 1 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 reserve need 0x400000 num_free 0x2542a00000 num_reserved 0x0 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 allocate_int want_size 0x400000 alloc_unit 0x100000 hint 0x0 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs mkfs op_alloc_add 1 0x1bf1f500000~2542a00000 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush_and_sync_log txn(seq 1 len 0x13 crc 0x7ded50fd) 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _pad_bl padding with 0xfc7 zeros 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush 0x556d7e94a8c0 0x0~1000 to file(ino 1 size 0x0 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush_range 0x556d7e94a8c0 pos 0x0 0x0~1000 to file(ino 1 size 0x0 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range file now file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range in 1:0x1bf1f500000+400000 x_off 0x0 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write 0x1bf1f500000~1000 (direct) 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write rebuilding buffer to be aligned 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1bf1f500000~1000 2019-03-05 12:37:26.934 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write 0x1bf1f500000~1000 aio 0x556d7e94b2d0 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_submit ioc 0x556d7ed88490 pending 1 running 0 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range h 0x556d7e94a8c0 pos now 0x1000 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _claim_completed_aios got 1 aios 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs wait_for_aio 0x556d7e94a8c0 2019-03-05 12:37:26.934 7fa4af337240 10 bdev aio_wait 0x556d7ed88490 waiting for 1 aios to complete 2019-03-05 12:37:26.946 7fa49e2ae700 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_finish 1 0x1bf1f500000~1000 2019-03-05 12:37:26.946 7fa49e2ae700 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_thread finished aio 0x556d7e94b2d0 r 4096 ioc 0x556d7ed88490 with 0 aios left 2019-03-05 12:37:26.946 7fa4af337240 20 bdev aio_wait 0x556d7ed88490 done 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs wait_for_aio 0x556d7e94a8c0 done in 0.010733 2019-03-05 12:37:26.946 7fa4af337240 20 bluefs flush_bdev 2019-03-05 12:37:26.946 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush start 2019-03-05 12:37:26.946 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush in 0.000005 2019-03-05 12:37:26.946 7fa4af337240 20 bluefs _flush_and_sync_log log_seq_stable 1 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super super block length(encoded): 89 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super superblock 1 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.946 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) write 0x1000~1000 (direct) 2019-03-05 12:37:26.946 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) write rebuilding buffer to be aligned 2019-03-05 12:37:26.946 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _sync_write 0x1000~1000 buffered 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs _write_super v 1 crc 0xbf92795f offset 0x1000 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs flush_bdev 2019-03-05 12:37:26.970 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush start 2019-03-05 12:37:26.970 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush in 0.000006 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs _close_writer 0x556d7e94a8c0 type 0 2019-03-05 12:37:26.970 7fa4af337240 20 bdev aio_wait 0x556d7ed88490 done 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs _stop_alloc 2019-03-05 12:37:26.970 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) discard_drain 2019-03-05 12:37:26.970 7fa4af337240 1 stupidalloc 0x0x556d7e91baa0 shutdown 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs mkfs success 2019-03-05 12:37:26.970 7fa4af337240 1 bluefs mount 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs _open_super 2019-03-05 12:37:26.970 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) read 0x1000~1000 (direct) 2019-03-05 12:37:26.970 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1000~1000 2019-03-05 12:37:26.974 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_finish 1 0x1000~1000 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _open_super superblock 1 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _open_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.974 7fa4af337240 20 bluefs _init_alloc 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _replay 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _replay log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _read h 0x556d7e914f80 0x0~1000 from file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) 2019-03-05 12:37:26.974 7fa4af337240 20 bluefs _read fetching 0x0~100000 of 1:0x1bf1f500000+400000 2019-03-05 12:37:26.974 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) read 0x1bf1f500000~100000 (buffered) 2019-03-05 12:37:26.974 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1bf1f500000~100000 2019-03-05 12:37:26.986 7fa4af337240 -1 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7fa4af337240 time 2019-03-05 12:37:26.989832 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: FAILED assert((uint64_t)r == len) ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fa4a67573e2] 2: (()+0x26d5a7) [0x7fa4a67575a7] 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x556d7bd43817] 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x556d7bd035c5] 5: (BlueFS::_replay(bool, bool)+0x214) [0x556d7bd09434] 6: (BlueFS::mount()+0x1f1) [0x556d7bd0dc81] 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x556d7bc3404d] 8: (BlueStore::mkfs()+0x805) [0x556d7bc64fe5] 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x556d7b80d480] 10: (main()+0x4222) [0x556d7b6f4462] 11: (__libc_start_main()+0xe7) [0x7fa4a3d14b97] 12: (_start()+0x2a) [0x556d7b7bd95a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -191> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perfcounters_dump hook 0x556d7e8d4310 -190> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command 1 hook 0x556d7e8d4310 -189> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perf dump hook 0x556d7e8d4310 -188> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perfcounters_schema hook 0x556d7e8d4310 -187> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perf histogram dump hook 0x556d7e8d4310 -186> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command 2 hook 0x556d7e8d4310 -185> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perf schema hook 0x556d7e8d4310 -184> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perf histogram schema hook 0x556d7e8d4310 -183> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command perf reset hook 0x556d7e8d4310 -182> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config show hook 0x556d7e8d4310 -181> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config help hook 0x556d7e8d4310 -180> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config set hook 0x556d7e8d4310 -179> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config unset hook 0x556d7e8d4310 -178> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config get hook 0x556d7e8d4310 -177> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config diff hook 0x556d7e8d4310 -176> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command config diff get hook 0x556d7e8d4310 -175> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command log flush hook 0x556d7e8d4310 -174> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command log dump hook 0x556d7e8d4310 -173> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command log reopen hook 0x556d7e8d4310 -172> 2019-03-05 12:37:26.910 7fa4af337240 5 asok(0x556d7e8d6900) register_command dump_mempools hook 0x556d7ec98068 -171> 2019-03-05 12:37:26.914 7fa4af337240 10 monclient: get_monmap_and_config -170> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: build_initial_monmap -169> 2019-03-05 12:37:26.918 7fa4a0a89700 2 Event(0x556d7e8d6e00 nevent=5000 time_id=1).set_owner idx=0 owner=140345046767360 -168> 2019-03-05 12:37:26.918 7fa4a0288700 2 Event(0x556d7e8d7040 nevent=5000 time_id=1).set_owner idx=1 owner=140345038374656 -167> 2019-03-05 12:37:26.918 7fa49fa87700 2 Event(0x556d7e8d7280 nevent=5000 time_id=1).set_owner idx=2 owner=140345029981952 -166> 2019-03-05 12:37:26.918 7fa4af337240 1 Processor -- start -165> 2019-03-05 12:37:26.918 7fa4af337240 1 -- - start start -164> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: init -163> 2019-03-05 12:37:26.918 7fa4af337240 5 adding auth protocol: cephx -162> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: auth_supported 2 method cephx -161> 2019-03-05 12:37:26.918 7fa4af337240 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-0//keyring -160> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: _reopen_session rank -1 -159> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient(hunting): picked mon.ceph2 con 0x556d7ed6e700 addr 141.26.70.211:6789/0 -158> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient(hunting): picked mon.ceph1 con 0x556d7ed6ee00 addr 141.26.70.210:6789/0 -157> 2019-03-05 12:37:26.918 7fa4af337240 1 -- - --> 141.26.70.210:6789/0 -- auth(proto 0 26 bytes epoch 2) v1 -- 0x556d7ece0780 con 0 -156> 2019-03-05 12:37:26.918 7fa4af337240 1 -- - --> 141.26.70.211:6789/0 -- auth(proto 0 26 bytes epoch 2) v1 -- 0x556d7ece0a00 con 0 -155> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient(hunting): _renew_subs -154> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient(hunting): authenticate will time out at 2019-03-05 12:42:26.923126 -153> 2019-03-05 12:37:26.918 7fa49fa87700 1 -- 141.26.70.210:0/1189472781 learned_addr learned my addr 141.26.70.210:0/1189472781 -152> 2019-03-05 12:37:26.918 7fa49fa87700 2 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -151> 2019-03-05 12:37:26.918 7fa4a0288700 2 -- 141.26.70.210:0/1189472781 >> 141.26.70.211:6789/0 conn(0x556d7ed6e700 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -150> 2019-03-05 12:37:26.918 7fa49fa87700 5 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54 cs=1 l=1). rx mon.0 seq 1 0x556d7ece0780 auth_reply(proto 2 0 (0) Success) v1 -149> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 <== mon.0 141.26.70.210:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (606083615 0 0) 0x556d7ece0780 con 0x556d7ed6ee00 -148> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient(hunting): my global_id is 14115 -147> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 --> 141.26.70.210:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x556d7ece0f00 con 0 -146> 2019-03-05 12:37:26.918 7fa4a0288700 5 -- 141.26.70.210:0/1189472781 >> 141.26.70.211:6789/0 conn(0x556d7ed6e700 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4417 cs=1 l=1). rx mon.1 seq 1 0x556d7ece0a00 auth_reply(proto 2 0 (0) Success) v1 -145> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 <== mon.1 141.26.70.211:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (1859929057 0 0) 0x556d7ece0a00 con 0x556d7ed6e700 -144> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient(hunting): my global_id is 14137 -143> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 --> 141.26.70.211:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x556d7ece0780 con 0 -142> 2019-03-05 12:37:26.918 7fa49fa87700 5 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54 cs=1 l=1). rx mon.0 seq 2 0x556d7ece0f00 auth_reply(proto 2 0 (0) Success) v1 -141> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 <== mon.0 141.26.70.210:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3926786789 0 0) 0x556d7ece0f00 con 0x556d7ed6ee00 -140> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 >> 141.26.70.211:6789/0 conn(0x556d7ed6e700 :-1 s=STATE_OPEN pgs=4417 cs=1 l=1).mark_down -139> 2019-03-05 12:37:26.918 7fa49f286700 2 -- 141.26.70.210:0/1189472781 >> 141.26.70.211:6789/0 conn(0x556d7ed6e700 :-1 s=STATE_OPEN pgs=4417 cs=1 l=1)._stop -138> 2019-03-05 12:37:26.918 7fa49f286700 1 monclient: found mon.ceph1 -137> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient: _send_mon_message to mon.ceph1 at 141.26.70.210:6789/0 -136> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 --> 141.26.70.210:6789/0 -- mon_subscribe({config=0+,monmap=3+}) v3 -- 0x556d7e8d7440 con 0 -135> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-03-05 12:36:56.924395) -134> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient: _send_mon_message to mon.ceph1 at 141.26.70.210:6789/0 -133> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 --> 141.26.70.210:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x556d7ece0a00 con 0 -132> 2019-03-05 12:37:26.918 7fa4af337240 5 monclient: authenticate success, global_id 14115 -131> 2019-03-05 12:37:26.918 7fa49fa87700 5 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54 cs=1 l=1). rx mon.0 seq 3 0x556d7e8d7440 config(0 keys) v1 -130> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 <== mon.0 141.26.70.210:6789/0 3 ==== config(0 keys) v1 ==== 4+0+0 (0 0 0) 0x556d7e8d7440 con 0x556d7ed6ee00 -129> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient: handle_config config(0 keys) v1 -128> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: get_monmap_and_config success -127> 2019-03-05 12:37:26.918 7fa4af337240 10 monclient: shutdown -126> 2019-03-05 12:37:26.918 7fa4af337240 1 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN pgs=54 cs=1 l=1).mark_down -125> 2019-03-05 12:37:26.918 7fa49fa87700 5 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=54 cs=1 l=1). rx mon.0 seq 4 0x556d7ece0a00 auth_reply(proto 2 0 (0) Success) v1 -124> 2019-03-05 12:37:26.918 7fa4af337240 2 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:6789/0 conn(0x556d7ed6ee00 :-1 s=STATE_OPEN pgs=54 cs=1 l=1)._stop -123> 2019-03-05 12:37:26.918 7fa49da83700 4 set_mon_vals no callback set -122> 2019-03-05 12:37:26.918 7fa49f286700 1 -- 141.26.70.210:0/1189472781 <== mon.0 141.26.70.210:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3157836970 0 0) 0x556d7ece0a00 con 0x556d7ed6ee00 -121> 2019-03-05 12:37:26.918 7fa49f286700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 -120> 2019-03-05 12:37:26.922 7fa4af337240 1 -- 141.26.70.210:0/1189472781 shutdown_connections -119> 2019-03-05 12:37:26.922 7fa4af337240 5 -- 141.26.70.210:0/1189472781 shutdown_connections mark down 141.26.70.210:6789/0 0x556d7ed6ee00 -118> 2019-03-05 12:37:26.922 7fa4af337240 5 -- 141.26.70.210:0/1189472781 shutdown_connections mark down 141.26.70.211:6789/0 0x556d7ed6e700 -117> 2019-03-05 12:37:26.922 7fa4af337240 5 -- 141.26.70.210:0/1189472781 shutdown_connections delete 0x556d7ed6e700 -116> 2019-03-05 12:37:26.922 7fa4af337240 5 -- 141.26.70.210:0/1189472781 shutdown_connections delete 0x556d7ed6ee00 -115> 2019-03-05 12:37:26.922 7fa4af337240 1 -- 141.26.70.210:0/1189472781 shutdown_connections -114> 2019-03-05 12:37:26.922 7fa4af337240 1 -- 141.26.70.210:0/1189472781 wait complete. -113> 2019-03-05 12:37:26.922 7fa4af337240 1 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:0/1189472781 conn(0x556d7ed6e000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down -112> 2019-03-05 12:37:26.922 7fa4af337240 2 -- 141.26.70.210:0/1189472781 >> 141.26.70.210:0/1189472781 conn(0x556d7ed6e000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop -111> 2019-03-05 12:37:26.922 7fa4af337240 0 set uid:gid to 64045:64045 (ceph:ceph) -110> 2019-03-05 12:37:26.922 7fa4af337240 0 ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable), process ceph-osd, pid 71549 -109> 2019-03-05 12:37:26.922 7fa4af337240 0 pidfile_write: ignore empty --pid-file -108> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) init /var/run/ceph/ceph-osd.0.asok -107> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) bind_and_listen /var/run/ceph/ceph-osd.0.asok -106> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) register_command 0 hook 0x556d7e8d3550 -105> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) register_command version hook 0x556d7e8d3550 -104> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) register_command git_version hook 0x556d7e8d3550 -103> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) register_command help hook 0x556d7e8d44b0 -102> 2019-03-05 12:37:26.926 7fa4af337240 5 asok(0x556d7e8d6900) register_command get_command_descriptions hook 0x556d7e8d44c0 -101> 2019-03-05 12:37:26.926 7fa4a0288700 5 asok(0x556d7e8d6900) entry start -100> 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) set_cache_shards 1 -99> 2019-03-05 12:37:26.926 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs path /var/lib/ceph/osd/ceph-0/ -98> 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label -97> 2019-03-05 12:37:26.926 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding -96> 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label -95> 2019-03-05 12:37:26.926 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding -94> 2019-03-05 12:37:26.926 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label -93> 2019-03-05 12:37:26.930 7fa4af337240 2 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding -92> 2019-03-05 12:37:26.930 7fa4af337240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid -91> 2019-03-05 12:37:26.930 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs using provided fsid 889f02eb-fe94-4829-a11c-b87ed81c5936 -90> 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block path size 10737418240 create=1 -89> 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block.wal path size 100663296 create=0 -88> 2019-03-05 12:37:26.930 7fa4af337240 20 bluestore(/var/lib/ceph/osd/ceph-0/) _setup_block_symlink_or_file name block.db path size 0 create=0 -87> 2019-03-05 12:37:26.930 7fa4af337240 1 bdev create path /var/lib/ceph/osd/ceph-0//block type kernel -86> 2019-03-05 12:37:26.930 7fa4af337240 1 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open path /var/lib/ceph/osd/ceph-0//block -85> 2019-03-05 12:37:26.930 7fa4af337240 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open devname dm-0 -84> 2019-03-05 12:37:26.930 7fa4af337240 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _detect_vdo no VDO volume maps to dm-0 -83> 2019-03-05 12:37:26.930 7fa4af337240 10 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _aio_start -82> 2019-03-05 12:37:26.930 7fa4af337240 1 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational -81> 2019-03-05 12:37:26.930 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0//block) _write_bdev_label path /var/lib/ceph/osd/ceph-0//block label bdev(osd_uuid 889f02eb-fe94-4829-a11c-b87ed81c5936, size 0x3a381400000, btime 2019-03-05 12:37:26.933971, desc main, 0 meta) -80> 2019-03-05 12:37:26.930 7fa4a0a89700 10 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _aio_thread start -79> 2019-03-05 12:37:26.930 7fa49eaaf700 20 bdev(0x556d7ed7e700 /var/lib/ceph/osd/ceph-0//block) _discard_thread sleep -78> 2019-03-05 12:37:26.934 7fa4af337240 1 bluestore(/var/lib/ceph/osd/ceph-0/) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2 -77> 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db kv_backend = rocksdb -76> 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db do_bluefs = 1 -75> 2019-03-05 12:37:26.934 7fa4af337240 10 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db initializing bluefs -74> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0//block -73> 2019-03-05 12:37:26.934 7fa4af337240 1 bdev create path /var/lib/ceph/osd/ceph-0//block type kernel -72> 2019-03-05 12:37:26.934 7fa4af337240 1 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open path /var/lib/ceph/osd/ceph-0//block -71> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open devname dm-0 -70> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _detect_vdo no VDO volume maps to dm-0 -69> 2019-03-05 12:37:26.934 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_start -68> 2019-03-05 12:37:26.934 7fa4af337240 1 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational -67> 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0//block size 3.6 TiB -66> 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs add_block_extent bdev 1 0x1bf1f500000~2542a00000 -65> 2019-03-05 12:37:26.934 7fa49e2ae700 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_thread start -64> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs add_block_extent done -63> 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs mkfs osd_uuid 889f02eb-fe94-4829-a11c-b87ed81c5936 -62> 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _init_alloc -61> 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 init_add_free 0x1bf1f500000~2542a00000 -60> 2019-03-05 12:37:26.934 7fa49daad700 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _discard_thread sleep -59> 2019-03-05 12:37:26.934 7fa4af337240 1 bluefs mkfs uuid 06421aba-924a-4dd8-8320-bde13e4b9ca4 -58> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _allocate len 0x400000 from 0 -57> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _allocate len 0x400000 from 1 -56> 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 reserve need 0x400000 num_free 0x2542a00000 num_reserved 0x0 -55> 2019-03-05 12:37:26.934 7fa4af337240 10 stupidalloc 0x0x556d7e91baa0 allocate_int want_size 0x400000 alloc_unit 0x100000 hint 0x0 -54> 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs mkfs op_alloc_add 1 0x1bf1f500000~2542a00000 -53> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush_and_sync_log txn(seq 1 len 0x13 crc 0x7ded50fd) -52> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _pad_bl padding with 0xfc7 zeros -51> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush 0x556d7e94a8c0 0x0~1000 to file(ino 1 size 0x0 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -50> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _flush_range 0x556d7e94a8c0 pos 0x0 0x0~1000 to file(ino 1 size 0x0 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -49> 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range file now file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -48> 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range in 1:0x1bf1f500000+400000 x_off 0x0 -47> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write 0x1bf1f500000~1000 (direct) -46> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write rebuilding buffer to be aligned -45> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1bf1f500000~1000 -44> 2019-03-05 12:37:26.934 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_write 0x1bf1f500000~1000 aio 0x556d7e94b2d0 -43> 2019-03-05 12:37:26.934 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) aio_submit ioc 0x556d7ed88490 pending 1 running 0 -42> 2019-03-05 12:37:26.934 7fa4af337240 20 bluefs _flush_range h 0x556d7e94a8c0 pos now 0x1000 -41> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs _claim_completed_aios got 1 aios -40> 2019-03-05 12:37:26.934 7fa4af337240 10 bluefs wait_for_aio 0x556d7e94a8c0 -39> 2019-03-05 12:37:26.934 7fa4af337240 10 bdev aio_wait 0x556d7ed88490 waiting for 1 aios to complete -38> 2019-03-05 12:37:26.946 7fa49e2ae700 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_finish 1 0x1bf1f500000~1000 -37> 2019-03-05 12:37:26.946 7fa49e2ae700 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_thread finished aio 0x556d7e94b2d0 r 4096 ioc 0x556d7ed88490 with 0 aios left -36> 2019-03-05 12:37:26.946 7fa4af337240 20 bdev aio_wait 0x556d7ed88490 done -35> 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs wait_for_aio 0x556d7e94a8c0 done in 0.010733 -34> 2019-03-05 12:37:26.946 7fa4af337240 20 bluefs flush_bdev -33> 2019-03-05 12:37:26.946 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush start -32> 2019-03-05 12:37:26.946 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush in 0.000005 -31> 2019-03-05 12:37:26.946 7fa4af337240 20 bluefs _flush_and_sync_log log_seq_stable 1 -30> 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super super block length(encoded): 89 -29> 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super superblock 1 -28> 2019-03-05 12:37:26.946 7fa4af337240 10 bluefs _write_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -27> 2019-03-05 12:37:26.946 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) write 0x1000~1000 (direct) -26> 2019-03-05 12:37:26.946 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) write rebuilding buffer to be aligned -25> 2019-03-05 12:37:26.946 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _sync_write 0x1000~1000 buffered -24> 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs _write_super v 1 crc 0xbf92795f offset 0x1000 -23> 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs flush_bdev -22> 2019-03-05 12:37:26.970 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush start -21> 2019-03-05 12:37:26.970 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) flush in 0.000006 -20> 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs _close_writer 0x556d7e94a8c0 type 0 -19> 2019-03-05 12:37:26.970 7fa4af337240 20 bdev aio_wait 0x556d7ed88490 done -18> 2019-03-05 12:37:26.970 7fa4af337240 20 bluefs _stop_alloc -17> 2019-03-05 12:37:26.970 7fa4af337240 10 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) discard_drain -16> 2019-03-05 12:37:26.970 7fa4af337240 1 stupidalloc 0x0x556d7e91baa0 shutdown -15> 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs mkfs success -14> 2019-03-05 12:37:26.970 7fa4af337240 1 bluefs mount -13> 2019-03-05 12:37:26.970 7fa4af337240 10 bluefs _open_super -12> 2019-03-05 12:37:26.970 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) read 0x1000~1000 (direct) -11> 2019-03-05 12:37:26.970 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1000~1000 -10> 2019-03-05 12:37:26.974 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_finish 1 0x1000~1000 -9> 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _open_super superblock 1 -8> 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _open_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -7> 2019-03-05 12:37:26.974 7fa4af337240 20 bluefs _init_alloc -6> 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _replay -5> 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _replay log_fnode file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -4> 2019-03-05 12:37:26.974 7fa4af337240 10 bluefs _read h 0x556d7e914f80 0x0~1000 from file(ino 1 size 0x1000 mtime 0.000000 bdev 0 allocated 400000 extents [1:0x1bf1f500000+400000]) -3> 2019-03-05 12:37:26.974 7fa4af337240 20 bluefs _read fetching 0x0~100000 of 1:0x1bf1f500000+400000 -2> 2019-03-05 12:37:26.974 7fa4af337240 5 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) read 0x1bf1f500000~100000 (buffered) -1> 2019-03-05 12:37:26.974 7fa4af337240 20 bdev(0x556d7ed7ea80 /var/lib/ceph/osd/ceph-0//block) _aio_log_start 0x1bf1f500000~100000 0> 2019-03-05 12:37:26.986 7fa4af337240 -1 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7fa4af337240 time 2019-03-05 12:37:26.989832 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: FAILED assert((uint64_t)r == len) ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fa4a67573e2] 2: (()+0x26d5a7) [0x7fa4a67575a7] 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x556d7bd43817] 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x556d7bd035c5] 5: (BlueFS::_replay(bool, bool)+0x214) [0x556d7bd09434] 6: (BlueFS::mount()+0x1f1) [0x556d7bd0dc81] 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x556d7bc3404d] 8: (BlueStore::mkfs()+0x805) [0x556d7bc64fe5] 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x556d7b80d480] 10: (main()+0x4222) [0x556d7b6f4462] 11: (__libc_start_main()+0xe7) [0x7fa4a3d14b97] 12: (_start()+0x2a) [0x556d7b7bd95a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 20/20 bluestore 20/20 bluefs 20/20 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2019-03-05 12:37:26.990 7fa4af337240 -1 *** Caught signal (Aborted) ** in thread 7fa4af337240 thread_name:ceph-osd ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (()+0x92aa40) [0x556d7bd54a40] 2: (()+0x12890) [0x7fa4a4e5b890] 3: (gsignal()+0xc7) [0x7fa4a3d31e97] 4: (abort()+0x141) [0x7fa4a3d33801] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fa4a6757530] 6: (()+0x26d5a7) [0x7fa4a67575a7] 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x556d7bd43817] 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x556d7bd035c5] 9: (BlueFS::_replay(bool, bool)+0x214) [0x556d7bd09434] 10: (BlueFS::mount()+0x1f1) [0x556d7bd0dc81] 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x556d7bc3404d] 12: (BlueStore::mkfs()+0x805) [0x556d7bc64fe5] 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x556d7b80d480] 14: (main()+0x4222) [0x556d7b6f4462] 15: (__libc_start_main()+0xe7) [0x7fa4a3d14b97] 16: (_start()+0x2a) [0x556d7b7bd95a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2019-03-05 12:37:26.990 7fa4af337240 -1 *** Caught signal (Aborted) ** in thread 7fa4af337240 thread_name:ceph-osd ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (()+0x92aa40) [0x556d7bd54a40] 2: (()+0x12890) [0x7fa4a4e5b890] 3: (gsignal()+0xc7) [0x7fa4a3d31e97] 4: (abort()+0x141) [0x7fa4a3d33801] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7fa4a6757530] 6: (()+0x26d5a7) [0x7fa4a67575a7] 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x556d7bd43817] 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x556d7bd035c5] 9: (BlueFS::_replay(bool, bool)+0x214) [0x556d7bd09434] 10: (BlueFS::mount()+0x1f1) [0x556d7bd0dc81] 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x556d7bc3404d] 12: (BlueStore::mkfs()+0x805) [0x556d7bc64fe5] 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string, std::allocator > const&, uuid_d, int)+0x1b0) [0x556d7b80d480] 14: (main()+0x4222) [0x556d7b6f4462] 15: (__libc_start_main()+0xe7) [0x7fa4a3d14b97] 16: (_start()+0x2a) [0x556d7b7bd95a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 20/20 bluestore 20/20 bluefs 20/20 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events ---