Ceph » bluestore

We are having 5 node cluster with 5 mons and 120 OSDs.

One of the OSD (osd.7) crashed with following logs:

    -4> 2018-03-14 22:14:01.748116 7f37b586a700  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_files.cc:307] [JOB 16] Delete db/006322.sst type=2 #6322 -- OK

    -3> 2018-03-14 22:14:01.748124 7f37b586a700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1521065641748121, "job": 16, "event": "table_file_deletion", "file_number": 6322}
    -2> 2018-03-14 22:14:01.748130 7f37b586a700  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_files.cc:307] [JOB 16] Delete db/006276.sst type=2 #6276 -- OK

    -1> 2018-03-14 22:14:01.748134 7f37b586a700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1521065641748133, "job": 16, "event": "table_file_deletion", "file_number": 6276}
     0> 2018-03-14 22:49:29.198238 7f37bf87e700 -1 *** Caught signal (Segmentation fault) **
in thread 7f37bf87e700 thread_name:safe_timer

ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
1: (()+0xa3c611) [0x5633ee9df611]
2: (()+0xf5e0) [0x7f37c6b3f5e0]
3: [0x563400080000]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   0/ 0 journal
   0/ 0 ms
   1/ 5 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   1/ 1 reserver
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 0 bluestore
   1/ 0 bluefs
   0/ 0 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.7.log

what all extra information is required to debug this issue?