Actions
Bug #18820
closedsegfault in ceph-osd --flush-journal
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
jewel,kraken
Regression:
No
Severity:
1 - critical
Reviewed:
Description
Got segfault while trying to flush osd journal.
ceph 10.2.5
Ubuntu 14.04.5, 3.19.0-79-generic x86_64
Debug output:
# ceph-osd -d --flush-journal --debug_filestore 20/20 --debug_journal 20/20 -i 6
2017-02-05 17:56:58.841612 7fe5719a8800 0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
2017-02-05 17:56:58.844862 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.844993 7fe5719a8800 0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
2017-02-05 17:56:58.845643 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-02-05 17:56:58.845656 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2017-02-05 17:56:58.845686 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
2017-02-05 17:56:58.865060 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2017-02-05 17:56:58.865222 7fe5719a8800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
2017-02-05 17:56:58.865980 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
2017-02-05 17:56:58.866594 7fe5719a8800 1 leveldb: Recovering log #3833784
2017-02-05 17:56:58.870358 7fe5719a8800 1 leveldb: Delete type=0 #3833784
2017-02-05 17:56:58.870412 7fe5719a8800 1 leveldb: Delete type=3 #3833783
2017-02-05 17:56:58.870477 7fe5719a8800 1 leveldb: Delete type=2 #3833785
2017-02-05 17:56:58.871053 7fe55de7d700 1 leveldb: Compacting 1@1 + 4@2 files
2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.874330 7fe5719a8800 0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
2017-02-05 17:56:58.876352 7fe5719a8800 2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.880252 7fe5719a8800 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
2017-02-05 17:56:58.880571 7fe5719a8800 10 journal write_pos 4096
2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.880817 7fe5719a8800 2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
2017-02-05 17:56:58.880852 7fe5719a8800 2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880854 7fe5719a8800 3 journal journal_replay: end of journal, done.
2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.885209 7fe5719a8800 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
2017-02-05 17:56:58.885528 7fe54ea32700 5 journal complete_write finished 0 ops and 0 bytes
2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6) ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6) temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
2017-02-05 17:56:58.886862 7fe5719a8800 1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.886916 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
2017-02-05 17:56:58.888015 7fe5719a8800 1 journal close /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
*** Caught signal (Segmentation fault) **
in thread 7fe55de7d700 thread_name:ceph-osd
ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
1: (()+0x8f2d32) [0x5596b8642d32]
2: (()+0x10330) [0x7fe57087e330]
3: [0x5596c41638a0]
2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
in thread 7fe55de7d700 thread_name:ceph-osd
ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
1: (()+0x8f2d32) [0x5596b8642d32]
2: (()+0x10330) [0x7fe57087e330]
3: [0x5596c41638a0]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-121> 2017-02-05 17:56:58.834469 7fe5719a8800 5 asok(0x5596c40ba280) register_command perfcounters_dump hook 0x5596c4096050
-120> 2017-02-05 17:56:58.834512 7fe5719a8800 5 asok(0x5596c40ba280) register_command 1 hook 0x5596c4096050
-119> 2017-02-05 17:56:58.834525 7fe5719a8800 5 asok(0x5596c40ba280) register_command perf dump hook 0x5596c4096050
-118> 2017-02-05 17:56:58.834530 7fe5719a8800 5 asok(0x5596c40ba280) register_command perfcounters_schema hook 0x5596c4096050
-117> 2017-02-05 17:56:58.834534 7fe5719a8800 5 asok(0x5596c40ba280) register_command 2 hook 0x5596c4096050
-116> 2017-02-05 17:56:58.834541 7fe5719a8800 5 asok(0x5596c40ba280) register_command perf schema hook 0x5596c4096050
-115> 2017-02-05 17:56:58.834549 7fe5719a8800 5 asok(0x5596c40ba280) register_command perf reset hook 0x5596c4096050
-114> 2017-02-05 17:56:58.834553 7fe5719a8800 5 asok(0x5596c40ba280) register_command config show hook 0x5596c4096050
-113> 2017-02-05 17:56:58.834558 7fe5719a8800 5 asok(0x5596c40ba280) register_command config set hook 0x5596c4096050
-112> 2017-02-05 17:56:58.834567 7fe5719a8800 5 asok(0x5596c40ba280) register_command config get hook 0x5596c4096050
-111> 2017-02-05 17:56:58.834571 7fe5719a8800 5 asok(0x5596c40ba280) register_command config diff hook 0x5596c4096050
-110> 2017-02-05 17:56:58.834575 7fe5719a8800 5 asok(0x5596c40ba280) register_command log flush hook 0x5596c4096050
-109> 2017-02-05 17:56:58.834578 7fe5719a8800 5 asok(0x5596c40ba280) register_command log dump hook 0x5596c4096050
-108> 2017-02-05 17:56:58.834585 7fe5719a8800 5 asok(0x5596c40ba280) register_command log reopen hook 0x5596c4096050
-107> 2017-02-05 17:56:58.841612 7fe5719a8800 0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
-106> 2017-02-05 17:56:58.841659 7fe5719a8800 5 object store type is filestore
-105> 2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
-104> 2017-02-05 17:56:58.844600 7fe5719a8800 5 asok(0x5596c40ba280) init /var/run/ceph/ceph-osd.6.asok
-103> 2017-02-05 17:56:58.844611 7fe5719a8800 5 asok(0x5596c40ba280) bind_and_listen /var/run/ceph/ceph-osd.6.asok
-102> 2017-02-05 17:56:58.844763 7fe5719a8800 5 asok(0x5596c40ba280) register_command 0 hook 0x5596c40920a0
-101> 2017-02-05 17:56:58.844778 7fe5719a8800 5 asok(0x5596c40ba280) register_command version hook 0x5596c40920a0
-100> 2017-02-05 17:56:58.844782 7fe5719a8800 5 asok(0x5596c40ba280) register_command git_version hook 0x5596c40920a0
-99> 2017-02-05 17:56:58.844786 7fe5719a8800 5 asok(0x5596c40ba280) register_command help hook 0x5596c4096140
-98> 2017-02-05 17:56:58.844790 7fe5719a8800 5 asok(0x5596c40ba280) register_command get_command_descriptions hook 0x5596c4096130
-97> 2017-02-05 17:56:58.844862 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
-96> 2017-02-05 17:56:58.844886 7fe562e7e700 5 asok(0x5596c40ba280) entry start
-95> 2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
-94> 2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
-93> 2017-02-05 17:56:58.844993 7fe5719a8800 0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
-92> 2017-02-05 17:56:58.845643 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-91> 2017-02-05 17:56:58.845656 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-90> 2017-02-05 17:56:58.845686 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
-89> 2017-02-05 17:56:58.865060 7fe5719a8800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-88> 2017-02-05 17:56:58.865222 7fe5719a8800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
-87> 2017-02-05 17:56:58.865980 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
-86> 2017-02-05 17:56:58.866594 7fe5719a8800 1 leveldb: Recovering log #3833784
-85> 2017-02-05 17:56:58.870358 7fe5719a8800 1 leveldb: Delete type=0 #3833784
-84> 2017-02-05 17:56:58.870412 7fe5719a8800 1 leveldb: Delete type=3 #3833783
-83> 2017-02-05 17:56:58.870477 7fe5719a8800 1 leveldb: Delete type=2 #3833785
-82> 2017-02-05 17:56:58.871053 7fe55de7d700 1 leveldb: Compacting 1@1 + 4@2 files
-81> 2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
-80> 2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
-79> 2017-02-05 17:56:58.874330 7fe5719a8800 0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-78> 2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
-77> 2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
-76> 2017-02-05 17:56:58.876352 7fe5719a8800 2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
-75> 2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
-74> 2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
-73> 2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
-72> 2017-02-05 17:56:58.880252 7fe5719a8800 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-71> 2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
-70> 2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
-69> 2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
-68> 2017-02-05 17:56:58.880571 7fe5719a8800 10 journal write_pos 4096
-67> 2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
-66> 2017-02-05 17:56:58.880817 7fe5719a8800 2 journal No further valid entries found, journal is most likely valid
-65> 2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
-64> 2017-02-05 17:56:58.880852 7fe5719a8800 2 journal No further valid entries found, journal is most likely valid
-63> 2017-02-05 17:56:58.880854 7fe5719a8800 3 journal journal_replay: end of journal, done.
-62> 2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
-61> 2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
-60> 2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
-59> 2017-02-05 17:56:58.885209 7fe5719a8800 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
-58> 2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
-57> 2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
-56> 2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
-55> 2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
-54> 2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
-53> 2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
-52> 2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
-51> 2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
-50> 2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
-49> 2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
-48> 2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
-47> 2017-02-05 17:56:58.885528 7fe54ea32700 5 journal complete_write finished 0 ops and 0 bytes
-46> 2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
-45> 2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
-44> 2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
-43> 2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
-42> 2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
-41> 2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
-40> 2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6) ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
-39> 2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6) temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
-38> 2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
-37> 2017-02-05 17:56:58.886862 7fe5719a8800 1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
-36> 2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
-35> 2017-02-05 17:56:58.886916 7fe5719a8800 5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
-34> 2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
-33> 2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
-32> 2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
-31> 2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
-30> 2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
-29> 2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
-28> 2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
-27> 2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
-26> 2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
-25> 2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
-24> 2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
-23> 2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
-22> 2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
-21> 2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
-20> 2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
-19> 2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
-18> 2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
-17> 2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
-16> 2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
-15> 2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
-14> 2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
-13> 2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
-12> 2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
-11> 2017-02-05 17:56:58.887601 7fe53fa2f700 1 FileStore::op_tp worker finish
-10> 2017-02-05 17:56:58.887669 7fe53aa2e700 1 FileStore::op_tp worker finish
-9> 2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
-8> 2017-02-05 17:56:58.888015 7fe5719a8800 1 journal close /var/lib/ceph/osd/ceph-6/journal
-7> 2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
-6> 2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
-5> 2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
-4> 2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
-3> 2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
-2> 2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
-1> 2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
0> 2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
in thread 7fe55de7d700 thread_name:ceph-osd
ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
1: (()+0x8f2d32) [0x5596b8642d32]
2: (()+0x10330) [0x7fe57087e330]
3: [0x5596c41638a0]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
20/20 filestore
20/20 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
Segmentation fault (core dumped)
strace -f output attached.
Files
Actions