Project

General

Profile

Bug #49110 » no-recovery-osd.108.txt

a try to recover fails - Andreas Buschmann, 02/02/2021 04:56 PM

 
2021-02-02 12:53:18.084995 7f5da959dec0 0 set uid:gid to 167:167 (ceph:ceph)
2021-02-02 12:53:18.085017 7f5da959dec0 0 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable), process ceph-osd, pid 374299
2021-02-02 12:53:18.094069 7f5da959dec0 0 pidfile_write: ignore empty --pid-file
2021-02-02 12:53:18.138962 7f5da959dec0 0 load: jerasure load: lrc load: isa
2021-02-02 12:53:18.139055 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
2021-02-02 12:53:18.139067 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
2021-02-02 12:53:18.139306 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
2021-02-02 12:53:18.139515 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _set_cache_sizes cache_size 3221225472 meta 0.4 kv 0.4 data 0.2
2021-02-02 12:53:18.139546 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) close
2021-02-02 12:53:18.399641 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _mount path /var/lib/ceph/osd/ceph-108
2021-02-02 12:53:18.399991 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
2021-02-02 12:53:18.400002 7f5da959dec0 1 bdev(0x55670aad8600 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
2021-02-02 12:53:18.400156 7f5da959dec0 1 bdev(0x55670aad8600 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
2021-02-02 12:53:18.400413 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _set_cache_sizes cache_size 3221225472 meta 0.4 kv 0.4 data 0.2
2021-02-02 12:53:18.400564 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
2021-02-02 12:53:18.400571 7f5da959dec0 1 bdev(0x55670aad6600 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
2021-02-02 12:53:18.400716 7f5da959dec0 1 bdev(0x55670aad6600 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
2021-02-02 12:53:18.400731 7f5da959dec0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-108/block size 932GiB
2021-02-02 12:53:18.400758 7f5da959dec0 1 bluefs mount
2021-02-02 12:53:18.401032 7f5da959dec0 1 bluefs _init_alloc id 1 alloc_size 0x10000 size 0xe8e0c00000
2021-02-02 13:21:09.304110 7f5da959dec0 -1 *** Caught signal (Segmentation fault) **
in thread 7f5da959dec0 thread_name:ceph-osd

ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (()+0xa76e51) [0x5566ff53ce51]
2: (()+0xf630) [0x7f5da6a40630]
3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x452) [0x5566ff4efcd2]
4: (BlueFS::_replay(bool)+0x48d) [0x5566ff50378d]
5: (BlueFS::mount()+0x1d4) [0x5566ff507374]
6: (BlueStore::_open_db(bool)+0x1857) [0x5566ff415827]
7: (BlueStore::_mount(bool)+0x2f6) [0x5566ff44acf6]
8: (OSD::init()+0x472) [0x5566fefe9762]
9: (main()+0x2f49) [0x5566feeea719]
10: (__libc_start_main()+0xf5) [0x7f5da5a4c555]
11: (()+0x4c4383) [0x5566fef8a383]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-75> 2021-02-02 12:53:18.077371 7f5da959dec0 5 asok(0x55670a7881c0) register_command perfcounters_dump hook 0x55670a74e180
-74> 2021-02-02 12:53:18.077391 7f5da959dec0 5 asok(0x55670a7881c0) register_command 1 hook 0x55670a74e180
-73> 2021-02-02 12:53:18.077394 7f5da959dec0 5 asok(0x55670a7881c0) register_command perf dump hook 0x55670a74e180
-72> 2021-02-02 12:53:18.077398 7f5da959dec0 5 asok(0x55670a7881c0) register_command perfcounters_schema hook 0x55670a74e180
-71> 2021-02-02 12:53:18.077400 7f5da959dec0 5 asok(0x55670a7881c0) register_command perf histogram dump hook 0x55670a74e180
-70> 2021-02-02 12:53:18.077402 7f5da959dec0 5 asok(0x55670a7881c0) register_command 2 hook 0x55670a74e180
-69> 2021-02-02 12:53:18.077404 7f5da959dec0 5 asok(0x55670a7881c0) register_command perf schema hook 0x55670a74e180
-68> 2021-02-02 12:53:18.077407 7f5da959dec0 5 asok(0x55670a7881c0) register_command perf histogram schema hook 0x55670a74e180
-67> 2021-02-02 12:53:18.077410 7f5da959dec0 5 asok(0x55670a7881c0) register_command perf reset hook 0x55670a74e180
-66> 2021-02-02 12:53:18.077417 7f5da959dec0 5 asok(0x55670a7881c0) register_command config show hook 0x55670a74e180
-65> 2021-02-02 12:53:18.077453 7f5da959dec0 5 asok(0x55670a7881c0) register_command config help hook 0x55670a74e180
-64> 2021-02-02 12:53:18.077458 7f5da959dec0 5 asok(0x55670a7881c0) register_command config set hook 0x55670a74e180
-63> 2021-02-02 12:53:18.077465 7f5da959dec0 5 asok(0x55670a7881c0) register_command config get hook 0x55670a74e180
-62> 2021-02-02 12:53:18.077468 7f5da959dec0 5 asok(0x55670a7881c0) register_command config diff hook 0x55670a74e180
-61> 2021-02-02 12:53:18.077474 7f5da959dec0 5 asok(0x55670a7881c0) register_command config diff get hook 0x55670a74e180
-60> 2021-02-02 12:53:18.077482 7f5da959dec0 5 asok(0x55670a7881c0) register_command log flush hook 0x55670a74e180
-59> 2021-02-02 12:53:18.077485 7f5da959dec0 5 asok(0x55670a7881c0) register_command log dump hook 0x55670a74e180
-58> 2021-02-02 12:53:18.077489 7f5da959dec0 5 asok(0x55670a7881c0) register_command log reopen hook 0x55670a74e180
-57> 2021-02-02 12:53:18.077524 7f5da959dec0 5 asok(0x55670a7881c0) register_command dump_mempools hook 0x55670a9e4158
-56> 2021-02-02 12:53:18.084995 7f5da959dec0 0 set uid:gid to 167:167 (ceph:ceph)
-55> 2021-02-02 12:53:18.085017 7f5da959dec0 0 ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable), process ceph-osd, pid 374299
-54> 2021-02-02 12:53:18.085066 7f5da959dec0 5 object store type is bluestore
-53> 2021-02-02 12:53:18.092759 7f5da3955700 2 Event(0x55670a9e8880 nevent=5000 time_id=1).set_owner idx=1 owner=140040153159424
-52> 2021-02-02 12:53:18.092786 7f5da3154700 2 Event(0x55670a9e8c80 nevent=5000 time_id=1).set_owner idx=2 owner=140040144766720
-51> 2021-02-02 12:53:18.092748 7f5da4156700 2 Event(0x55670a9e8480 nevent=5000 time_id=1).set_owner idx=0 owner=140040161552128
-50> 2021-02-02 12:53:18.093282 7f5da959dec0 1 -- 10.173.20.63:0/0 learned_addr learned my addr 10.173.20.63:0/0
-49> 2021-02-02 12:53:18.093298 7f5da959dec0 1 -- 10.173.20.63:6800/374299 _finish_bind bind my_inst.addr is 10.173.20.63:6800/374299
-48> 2021-02-02 12:53:18.093443 7f5da959dec0 1 -- 10.173.21.63:0/0 learned_addr learned my addr 10.173.21.63:0/0
-47> 2021-02-02 12:53:18.093457 7f5da959dec0 1 -- 10.173.21.63:6800/374299 _finish_bind bind my_inst.addr is 10.173.21.63:6800/374299
-46> 2021-02-02 12:53:18.093696 7f5da959dec0 1 -- 10.173.21.63:0/0 learned_addr learned my addr 10.173.21.63:0/0
-45> 2021-02-02 12:53:18.093714 7f5da959dec0 1 -- 10.173.21.63:6801/374299 _finish_bind bind my_inst.addr is 10.173.21.63:6801/374299
-44> 2021-02-02 12:53:18.094041 7f5da959dec0 1 -- 10.173.20.63:0/0 learned_addr learned my addr 10.173.20.63:0/0
-43> 2021-02-02 12:53:18.094056 7f5da959dec0 1 -- 10.173.20.63:6801/374299 _finish_bind bind my_inst.addr is 10.173.20.63:6801/374299
-42> 2021-02-02 12:53:18.094069 7f5da959dec0 0 pidfile_write: ignore empty --pid-file
-41> 2021-02-02 12:53:18.131080 7f5da959dec0 5 asok(0x55670a7881c0) init /var/run/ceph/ceph-osd.108.asok
-40> 2021-02-02 12:53:18.131092 7f5da959dec0 5 asok(0x55670a7881c0) bind_and_listen /var/run/ceph/ceph-osd.108.asok
-39> 2021-02-02 12:53:18.131169 7f5da959dec0 5 asok(0x55670a7881c0) register_command 0 hook 0x55670a74c1d8
-38> 2021-02-02 12:53:18.131175 7f5da959dec0 5 asok(0x55670a7881c0) register_command version hook 0x55670a74c1d8
-37> 2021-02-02 12:53:18.131178 7f5da959dec0 5 asok(0x55670a7881c0) register_command git_version hook 0x55670a74c1d8
-36> 2021-02-02 12:53:18.131181 7f5da959dec0 5 asok(0x55670a7881c0) register_command help hook 0x55670a74e580
-35> 2021-02-02 12:53:18.131183 7f5da959dec0 5 asok(0x55670a7881c0) register_command get_command_descriptions hook 0x55670a74e590
-34> 2021-02-02 12:53:18.131253 7f5da959dec0 10 monclient: build_initial_monmap
-33> 2021-02-02 12:53:18.131307 7f5da19d1700 5 asok(0x55670a7881c0) entry start
-32> 2021-02-02 12:53:18.138962 7f5da959dec0 0 load: jerasure load: lrc load: isa
-31> 2021-02-02 12:53:18.138991 7f5da959dec0 5 adding auth protocol: cephx
-30> 2021-02-02 12:53:18.138996 7f5da959dec0 5 adding auth protocol: cephx
-29> 2021-02-02 12:53:18.139055 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
-28> 2021-02-02 12:53:18.139067 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
-27> 2021-02-02 12:53:18.139306 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
-26> 2021-02-02 12:53:18.139515 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _set_cache_sizes cache_size 3221225472 meta 0.4 kv 0.4 data 0.2
-25> 2021-02-02 12:53:18.139546 7f5da959dec0 1 bdev(0x55670aad9200 /var/lib/ceph/osd/ceph-108/block) close
-24> 2021-02-02 12:53:18.399086 7f5da959dec0 5 asok(0x55670a7881c0) register_command objecter_requests hook 0x55670a74e610
-23> 2021-02-02 12:53:18.399142 7f5da959dec0 1 -- 10.173.20.63:6800/374299 start start
-22> 2021-02-02 12:53:18.399149 7f5da959dec0 1 -- - start start
-21> 2021-02-02 12:53:18.399152 7f5da959dec0 1 -- - start start
-20> 2021-02-02 12:53:18.399171 7f5da959dec0 1 -- 10.173.20.63:6801/374299 start start
-19> 2021-02-02 12:53:18.399175 7f5da959dec0 1 -- 10.173.21.63:6801/374299 start start
-18> 2021-02-02 12:53:18.399186 7f5da959dec0 1 -- 10.173.21.63:6800/374299 start start
-17> 2021-02-02 12:53:18.399189 7f5da959dec0 1 -- - start start
-16> 2021-02-02 12:53:18.399613 7f5da959dec0 2 osd.108 0 init /var/lib/ceph/osd/ceph-108 (looks like ssd)
-15> 2021-02-02 12:53:18.399631 7f5da959dec0 2 osd.108 0 journal /var/lib/ceph/osd/ceph-108/journal
-14> 2021-02-02 12:53:18.399641 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _mount path /var/lib/ceph/osd/ceph-108
-13> 2021-02-02 12:53:18.399991 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
-12> 2021-02-02 12:53:18.400002 7f5da959dec0 1 bdev(0x55670aad8600 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
-11> 2021-02-02 12:53:18.400156 7f5da959dec0 1 bdev(0x55670aad8600 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
-10> 2021-02-02 12:53:18.400413 7f5da959dec0 1 bluestore(/var/lib/ceph/osd/ceph-108) _set_cache_sizes cache_size 3221225472 meta 0.4 kv 0.4 data 0.2
-9> 2021-02-02 12:53:18.400477 7f5da959dec0 5 asok(0x55670a7881c0) register_command bluestore bluefs available hook 0x55670a74e620
-8> 2021-02-02 12:53:18.400564 7f5da959dec0 1 bdev create path /var/lib/ceph/osd/ceph-108/block type kernel
-7> 2021-02-02 12:53:18.400571 7f5da959dec0 1 bdev(0x55670aad6600 /var/lib/ceph/osd/ceph-108/block) open path /var/lib/ceph/osd/ceph-108/block
-6> 2021-02-02 12:53:18.400716 7f5da959dec0 1 bdev(0x55670aad6600 /var/lib/ceph/osd/ceph-108/block) open size 1000203091968 (0xe8e0c00000, 932GiB) block_size 4096 (4KiB) non-rotational
-5> 2021-02-02 12:53:18.400731 7f5da959dec0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-108/block size 932GiB
-4> 2021-02-02 12:53:18.400758 7f5da959dec0 1 bluefs mount
-3> 2021-02-02 12:53:18.401032 7f5da959dec0 1 bluefs _init_alloc id 1 alloc_size 0x10000 size 0xe8e0c00000
-2> 2021-02-02 12:53:18.401048 7f5da959dec0 5 asok(0x55670a7881c0) register_command bluestore allocator dump bluefs-db hook 0x55670aad3660
-1> 2021-02-02 12:53:18.401057 7f5da959dec0 5 asok(0x55670a7881c0) register_command bluestore allocator score bluefs-db hook 0x55670aad3660
0> 2021-02-02 13:21:09.304110 7f5da959dec0 -1 *** Caught signal (Segmentation fault) **
in thread 7f5da959dec0 thread_name:ceph-osd

ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)
1: (()+0xa76e51) [0x5566ff53ce51]
2: (()+0xf630) [0x7f5da6a40630]
3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x452) [0x5566ff4efcd2]
4: (BlueFS::_replay(bool)+0x48d) [0x5566ff50378d]
5: (BlueFS::mount()+0x1d4) [0x5566ff507374]
6: (BlueStore::_open_db(bool)+0x1857) [0x5566ff415827]
7: (BlueStore::_mount(bool)+0x2f6) [0x5566ff44acf6]
8: (OSD::init()+0x472) [0x5566fefe9762]
9: (main()+0x2f49) [0x5566feeea719]
10: (__libc_start_main()+0xf5) [0x7f5da5a4c555]
11: (()+0x4c4383) [0x5566fef8a383]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.108.log
--- end dump of recent events ---
(2-2/14)