Actions
Bug #55698
openosd: segfault at boot up
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
In the https://tracker.ceph.com/issues/55407#note-14 an OSD crash during early boot up is reported:
... -79> 2022-05-16T09:10:21.707+0000 7fb1f1728640 10 monclient: _renew_subs -78> 2022-05-16T09:10:21.707+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -77> 2022-05-16T09:10:21.707+0000 7fb1f5730640 3 osd.3 1739257 handle_osd_map epochs [1739258,1739297], i have 1739257, src has [1491297,1739460] -76> 2022-05-16T09:10:21.751+0000 7fb1f1728640 10 monclient: _renew_subs -75> 2022-05-16T09:10:21.751+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -74> 2022-05-16T09:10:21.751+0000 7fb1f5730640 3 osd.3 1739297 handle_osd_map epochs [1739298,1739337], i have 1739297, src has [1491297,1739460] -73> 2022-05-16T09:10:21.807+0000 7fb1f1728640 10 monclient: _renew_subs -72> 2022-05-16T09:10:21.807+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -71> 2022-05-16T09:10:21.811+0000 7fb1f5730640 3 osd.3 1739337 handle_osd_map epochs [1739338,1739377], i have 1739337, src has [1491297,1739460] -70> 2022-05-16T09:10:21.859+0000 7fb1f1728640 10 monclient: _renew_subs -69> 2022-05-16T09:10:21.859+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -68> 2022-05-16T09:10:21.859+0000 7fb1f5730640 3 osd.3 1739377 handle_osd_map epochs [1739378,1739417], i have 1739377, src has [1491297,1739460] -67> 2022-05-16T09:10:21.903+0000 7fb1f1728640 10 monclient: _renew_subs -66> 2022-05-16T09:10:21.903+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -65> 2022-05-16T09:10:21.915+0000 7fb1f5730640 3 osd.3 1739417 handle_osd_map epochs [1739418,1739457], i have 1739417, src has [1491297,1739460] -64> 2022-05-16T09:10:21.967+0000 7fb1f1728640 10 monclient: _renew_subs -63> 2022-05-16T09:10:21.967+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -62> 2022-05-16T09:10:21.967+0000 7fb1f5730640 3 osd.3 1739457 handle_osd_map epochs [1739458,1739460], i have 1739457, src has [1491297,1739460] -61> 2022-05-16T09:10:21.991+0000 7fb1f1728640 5 osd.3 1739460 heartbeat osd_stat(store_statfs(0xe7a54d6000/0x0/0xe8e0db6000, data 0xab32b8c9/0xf770e000, compress 0x0/0x0/0x0, omap 0x0, meta 0x441d0000), peers [] op hist []) -60> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -59> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _renew_subs -58> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -57> 2022-05-16T09:10:22.027+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -56> 2022-05-16T09:10:22.031+0000 7fb1f5730640 1 osd.3 1739460 start_boot -55> 2022-05-16T09:10:22.031+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -54> 2022-05-16T09:10:22.031+0000 7fb1f5730640 10 monclient: handle_get_version_reply finishing 2 version 1739460 -53> 2022-05-16T09:10:22.031+0000 7fb201194640 5 osd.3 1739460 heartbeat osd_stat(store_statfs(0xe7a54d6000/0x0/0xe8e0db6000, data 0xab32b8c9/0xf770e000, compress 0x0/0x0/0x0, omap 0x0, meta 0x441d0000), peers [] op hist []) -52> 2022-05-16T09:10:22.039+0000 7fb1f472e640 10 monclient: tick -51> 2022-05-16T09:10:22.039+0000 7fb1f472e640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-05-16T09:09:52.041726+0000) -50> 2022-05-16T09:10:22.039+0000 7fb1f9f39640 -1 osd.3 1739460 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory -49> 2022-05-16T09:10:22.039+0000 7fb1f9f39640 1 osd.3 1739460 set_numa_affinity setting numa affinity to node 0 cpus 0-19 -48> 2022-05-16T09:10:22.063+0000 7fb1f9f39640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -47> 2022-05-16T09:10:22.411+0000 7fb1f5f31640 5 prioritycache tune_memory target: 2147483648 mapped: 378167296 unmapped: 335536128 heap: 713703424 old mem: 1020054731 new mem: 1020054731 -46> 2022-05-16T09:10:22.499+0000 7fb1f3f2d640 4 mgrc reconnect Starting new session with [v2:172.16.99.10:6842/11647,v1:172.16.99.10:6843/11647] -45> 2022-05-16T09:10:22.499+0000 7fb202196640 10 monclient: get_auth_request con 0x561b0af07c00 auth_method 0 -44> 2022-05-16T09:10:22.503+0000 7fb1f5730640 4 mgrc ms_handle_reset ms_handle_reset con 0x561b0af07c00 -43> 2022-05-16T09:10:22.503+0000 7fb1f5730640 4 mgrc reconnect Terminating session with v2:172.16.99.10:6842/11647 -42> 2022-05-16T09:10:22.503+0000 7fb1f5730640 4 mgrc reconnect waiting to retry connect until 2713.683826s -41> 2022-05-16T09:10:22.847+0000 7fb1fe742640 1 osd.3 1739460 tick checking mon for new map -40> 2022-05-16T09:10:23.039+0000 7fb1f472e640 10 monclient: tick -39> 2022-05-16T09:10:23.039+0000 7fb1f472e640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-05-16T09:09:53.041842+0000) -38> 2022-05-16T09:10:23.059+0000 7fb1f5f31640 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.4 -37> 2022-05-16T09:10:23.059+0000 7fb1f5f31640 5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.168421 -36> 2022-05-16T09:10:23.059+0000 7fb1f5f31640 5 bluestore.MempoolThread(0x561b09d1cb38) _resize_shards cache_size: 1020054731 kv_alloc: 398458880 kv_used: 24928 kv_onode_alloc: 167772160 kv_onode_used: 8276112 meta_alloc: 339738624 meta_used: 13100328 data_alloc: 104857600 data_used: 0 -35> 2022-05-16T09:10:23.099+0000 7fb202196640 10 monclient: handle_auth_request added challenge on 0x561b0e47cc00 -34> 2022-05-16T09:10:23.099+0000 7fb203198640 10 monclient: handle_auth_request added challenge on 0x561b0e47d000 -33> 2022-05-16T09:10:23.131+0000 7fb1f5730640 10 monclient: _renew_subs -32> 2022-05-16T09:10:23.131+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -31> 2022-05-16T09:10:23.131+0000 7fb1f5730640 3 osd.3 1739460 handle_osd_map epochs [1739461,1739461], i have 1739460, src has [1491297,1739461] -30> 2022-05-16T09:10:23.151+0000 7fb1f1728640 1 osd.3 1739461 state: booting -> active -29> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d576400 auth_method 0 -28> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b298b0c00 auth_method 0 -27> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d577c00 auth_method 0 -26> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61a000 auth_method 0 -25> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61b800 auth_method 0 -24> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61ac00 auth_method 0 -23> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d577000 auth_method 0 -22> 2022-05-16T09:10:23.187+0000 7fb1f5730640 3 osd.3 1739461 handle_osd_map epochs [1739461,1739461], i have 1739461, src has [1491297,1739461] -21> 2022-05-16T09:10:23.411+0000 7fb1f5f31640 5 prioritycache tune_memory target: 2147483648 mapped: 378462208 unmapped: 335241216 heap: 713703424 old mem: 1020054731 new mem: 1020054731 -20> 2022-05-16T09:10:23.499+0000 7fb1f3f2d640 4 mgrc reconnect Starting new session with [v2:172.16.99.10:6842/11647,v1:172.16.99.10:6843/11647] -19> 2022-05-16T09:10:23.547+0000 7fb1fb73c640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0 -18> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1d576800 auth_method 0 -17> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d577800 auth_method 0 -16> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1d577400 auth_method 0 -15> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1a61a800 auth_method 0 -14> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b218fe000 auth_method 0 -13> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b0e47d800 auth_method 0 -12> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b0e47dc00 auth_method 0 -11> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61a400 auth_method 0 -10> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61bc00 auth_method 0 -9> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1a61b400 auth_method 0 -8> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d576c00 auth_method 0 -7> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d576000 auth_method 0 -6> 2022-05-16T09:10:23.599+0000 7fb203198640 10 monclient: get_auth_request con 0x561b0af07c00 auth_method 0 -5> 2022-05-16T09:10:23.599+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61b000 auth_method 0 -4> 2022-05-16T09:10:23.599+0000 7fb1f5730640 4 mgrc handle_mgr_configure stats_period=5 -3> 2022-05-16T09:10:23.699+0000 7fb203198640 10 monclient: handle_auth_request added challenge on 0x561b1fe40400 -2> 2022-05-16T09:10:23.699+0000 7fb202997640 10 monclient: handle_auth_request added challenge on 0x561b1fe40c00 -1> 2022-05-16T09:10:23.699+0000 7fb202196640 10 monclient: handle_auth_request added challenge on 0x561b1fe40800 0> 2022-05-16T09:10:23.711+0000 7fb202997640 -1 *** Caught signal (Segmentation fault) ** in thread 7fb202997640 thread_name:msgr-worker-1 ceph version 17.0.0-12154-g6f78f2f42ea (6f78f2f42eaa5f84d25bf175563df9717f8c4203) quincy (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb2044a8520] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_lba 0/ 5 seastore_lba_details 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7fb1e5f11640 / osd_srv_heartbt 7fb1f1728640 / cfin 7fb1f1f29640 / bstore_kv_sync 7fb1f3f2d640 / safe_timer 7fb1f472e640 / safe_timer 7fb1f5730640 / ms_dispatch 7fb1f5f31640 / bstore_mempool 7fb1f6732640 / rocksdb:high0 7fb1f6f33640 / rocksdb:low1 7fb1f7734640 / rocksdb:low0 7fb1f9f39640 / fn_anonymous 7fb1fb73c640 / safe_timer 7fb1fe742640 / safe_timer 7fb201194640 / io_context_pool 7fb202196640 / msgr-worker-2 7fb202997640 / msgr-worker-1 7fb203198640 / msgr-worker-0 max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2022-05-16T09:10:23.706211Z_777e42fd-25c5-4efc-a27e-b02c6eb3110a/log --- end dump of recent events --- ViolaciĆ³n de segmento (`core' generado)
This happens on master
:
Merge: 12d0955b5fc 89430b1c39d Author: Samuel Just <sjust@redhat.com> Date: Wed May 11 19:06:53 2022 -0700
The invocation was:
root@red-compute:/mnt/ceph-recovery# LD_LIBRARY_PATH=lib/ bin/ceph-osd -d --cluster ceph --id 3 --default-log-to-stderr=true --err-to-stderr=true --default-log-to-file=false --foreground --setuser ceph --setgroup ceph --version ceph version 17.0.0-12154-g6f78f2f42ea (6f78f2f42eaa5f84d25bf175563df9717f8c4203) quincy (dev) root@red-compute:/mnt/ceph-recovery# ls
No data to display
Actions