Project

General

Profile

Actions
Copy link

Bug #18820

closed

segfault in ceph-osd --flush-journal

Added by Sergey Baukin over 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel,kraken
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
v10.2.5
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Got segfault while trying to flush osd journal.

ceph 10.2.5
Ubuntu 14.04.5, 3.19.0-79-generic x86_64

Debug output:

# ceph-osd -d --flush-journal --debug_filestore 20/20 --debug_journal 20/20 -i 6 
2017-02-05 17:56:58.841612 7fe5719a8800  0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
2017-02-05 17:56:58.844862 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.844993 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
2017-02-05 17:56:58.845643 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-02-05 17:56:58.845656 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2017-02-05 17:56:58.845686 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
2017-02-05 17:56:58.865060 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2017-02-05 17:56:58.865222 7fe5719a8800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
2017-02-05 17:56:58.865980 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
2017-02-05 17:56:58.866594 7fe5719a8800  1 leveldb: Recovering log #3833784
2017-02-05 17:56:58.870358 7fe5719a8800  1 leveldb: Delete type=0 #3833784

2017-02-05 17:56:58.870412 7fe5719a8800  1 leveldb: Delete type=3 #3833783

2017-02-05 17:56:58.870477 7fe5719a8800  1 leveldb: Delete type=2 #3833785

2017-02-05 17:56:58.871053 7fe55de7d700  1 leveldb: Compacting 1@1 + 4@2 files
2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.874330 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
2017-02-05 17:56:58.876352 7fe5719a8800  2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.880252 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
2017-02-05 17:56:58.880571 7fe5719a8800 10 journal  write_pos 4096
2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.880817 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
2017-02-05 17:56:58.880852 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880854 7fe5719a8800  3 journal journal_replay: end of journal, done.
2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.885209 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
2017-02-05 17:56:58.885528 7fe54ea32700  5 journal complete_write finished 0 ops and 0 bytes
2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
2017-02-05 17:56:58.886862 7fe5719a8800  1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.886916 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
2017-02-05 17:56:58.888015 7fe5719a8800  1 journal close /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
*** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd
 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -121> 2017-02-05 17:56:58.834469 7fe5719a8800  5 asok(0x5596c40ba280) register_command perfcounters_dump hook 0x5596c4096050
  -120> 2017-02-05 17:56:58.834512 7fe5719a8800  5 asok(0x5596c40ba280) register_command 1 hook 0x5596c4096050
  -119> 2017-02-05 17:56:58.834525 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf dump hook 0x5596c4096050
  -118> 2017-02-05 17:56:58.834530 7fe5719a8800  5 asok(0x5596c40ba280) register_command perfcounters_schema hook 0x5596c4096050
  -117> 2017-02-05 17:56:58.834534 7fe5719a8800  5 asok(0x5596c40ba280) register_command 2 hook 0x5596c4096050
  -116> 2017-02-05 17:56:58.834541 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf schema hook 0x5596c4096050
  -115> 2017-02-05 17:56:58.834549 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf reset hook 0x5596c4096050
  -114> 2017-02-05 17:56:58.834553 7fe5719a8800  5 asok(0x5596c40ba280) register_command config show hook 0x5596c4096050
  -113> 2017-02-05 17:56:58.834558 7fe5719a8800  5 asok(0x5596c40ba280) register_command config set hook 0x5596c4096050
  -112> 2017-02-05 17:56:58.834567 7fe5719a8800  5 asok(0x5596c40ba280) register_command config get hook 0x5596c4096050
  -111> 2017-02-05 17:56:58.834571 7fe5719a8800  5 asok(0x5596c40ba280) register_command config diff hook 0x5596c4096050
  -110> 2017-02-05 17:56:58.834575 7fe5719a8800  5 asok(0x5596c40ba280) register_command log flush hook 0x5596c4096050
  -109> 2017-02-05 17:56:58.834578 7fe5719a8800  5 asok(0x5596c40ba280) register_command log dump hook 0x5596c4096050
  -108> 2017-02-05 17:56:58.834585 7fe5719a8800  5 asok(0x5596c40ba280) register_command log reopen hook 0x5596c4096050
  -107> 2017-02-05 17:56:58.841612 7fe5719a8800  0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
  -106> 2017-02-05 17:56:58.841659 7fe5719a8800  5 object store type is filestore
  -105> 2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
  -104> 2017-02-05 17:56:58.844600 7fe5719a8800  5 asok(0x5596c40ba280) init /var/run/ceph/ceph-osd.6.asok
  -103> 2017-02-05 17:56:58.844611 7fe5719a8800  5 asok(0x5596c40ba280) bind_and_listen /var/run/ceph/ceph-osd.6.asok
  -102> 2017-02-05 17:56:58.844763 7fe5719a8800  5 asok(0x5596c40ba280) register_command 0 hook 0x5596c40920a0
  -101> 2017-02-05 17:56:58.844778 7fe5719a8800  5 asok(0x5596c40ba280) register_command version hook 0x5596c40920a0
  -100> 2017-02-05 17:56:58.844782 7fe5719a8800  5 asok(0x5596c40ba280) register_command git_version hook 0x5596c40920a0
   -99> 2017-02-05 17:56:58.844786 7fe5719a8800  5 asok(0x5596c40ba280) register_command help hook 0x5596c4096140
   -98> 2017-02-05 17:56:58.844790 7fe5719a8800  5 asok(0x5596c40ba280) register_command get_command_descriptions hook 0x5596c4096130
   -97> 2017-02-05 17:56:58.844862 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
   -96> 2017-02-05 17:56:58.844886 7fe562e7e700  5 asok(0x5596c40ba280) entry start
   -95> 2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
   -94> 2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
   -93> 2017-02-05 17:56:58.844993 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
   -92> 2017-02-05 17:56:58.845643 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
   -91> 2017-02-05 17:56:58.845656 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
   -90> 2017-02-05 17:56:58.845686 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
   -89> 2017-02-05 17:56:58.865060 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
   -88> 2017-02-05 17:56:58.865222 7fe5719a8800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
   -87> 2017-02-05 17:56:58.865980 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
   -86> 2017-02-05 17:56:58.866594 7fe5719a8800  1 leveldb: Recovering log #3833784
   -85> 2017-02-05 17:56:58.870358 7fe5719a8800  1 leveldb: Delete type=0 #3833784

   -84> 2017-02-05 17:56:58.870412 7fe5719a8800  1 leveldb: Delete type=3 #3833783

   -83> 2017-02-05 17:56:58.870477 7fe5719a8800  1 leveldb: Delete type=2 #3833785

   -82> 2017-02-05 17:56:58.871053 7fe55de7d700  1 leveldb: Compacting 1@1 + 4@2 files
   -81> 2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
   -80> 2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
   -79> 2017-02-05 17:56:58.874330 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
   -78> 2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
   -77> 2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
   -76> 2017-02-05 17:56:58.876352 7fe5719a8800  2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
   -75> 2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
   -74> 2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
   -73> 2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
   -72> 2017-02-05 17:56:58.880252 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
   -71> 2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
   -70> 2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
   -69> 2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
   -68> 2017-02-05 17:56:58.880571 7fe5719a8800 10 journal  write_pos 4096
   -67> 2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
   -66> 2017-02-05 17:56:58.880817 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
   -65> 2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
   -64> 2017-02-05 17:56:58.880852 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
   -63> 2017-02-05 17:56:58.880854 7fe5719a8800  3 journal journal_replay: end of journal, done.
   -62> 2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
   -61> 2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
   -60> 2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
   -59> 2017-02-05 17:56:58.885209 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
   -58> 2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
   -57> 2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
   -56> 2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
   -55> 2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
   -54> 2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
   -53> 2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
   -52> 2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
   -51> 2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
   -50> 2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
   -49> 2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
   -48> 2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
   -47> 2017-02-05 17:56:58.885528 7fe54ea32700  5 journal complete_write finished 0 ops and 0 bytes
   -46> 2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
   -45> 2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
   -44> 2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
   -43> 2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
   -42> 2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
   -41> 2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
   -40> 2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
   -39> 2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
   -38> 2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
   -37> 2017-02-05 17:56:58.886862 7fe5719a8800  1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
   -36> 2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
   -35> 2017-02-05 17:56:58.886916 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
   -34> 2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
   -33> 2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
   -32> 2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
   -31> 2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
   -30> 2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
   -29> 2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
   -28> 2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
   -27> 2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
   -26> 2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
   -25> 2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
   -24> 2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
   -23> 2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
   -22> 2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
   -21> 2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
   -20> 2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
   -19> 2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
   -18> 2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
   -17> 2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
   -16> 2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
   -15> 2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
   -14> 2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
   -13> 2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
   -12> 2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
   -11> 2017-02-05 17:56:58.887601 7fe53fa2f700  1 FileStore::op_tp worker finish
   -10> 2017-02-05 17:56:58.887669 7fe53aa2e700  1 FileStore::op_tp worker finish
    -9> 2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
    -8> 2017-02-05 17:56:58.888015 7fe5719a8800  1 journal close /var/lib/ceph/osd/ceph-6/journal
    -7> 2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
    -6> 2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
    -5> 2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
    -4> 2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
    -3> 2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
    -2> 2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
    -1> 2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
     0> 2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
  20/20 filestore
  20/20 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file 
--- end dump of recent events ---
Segmentation fault (core dumped)

strace -f output attached.

Files

ceph-osd.strace (195 KB) ceph-osd.strace strace -f ceph-osd --flush-journal Sergey Baukin, 02/05/2017 03:05 PM

Related issues 2 (0 open — 2 closed)

Copied to Ceph - Backport #18951: jewel: segfault in ceph-osd --flush-journalResolvedAlexey SheplyakovActions
Copied to Ceph - Backport #18952: kraken: segfault in ceph-osd --flush-journalResolvedShinobu KinjoActions
Actions
Copy link

Also available in: Atom PDF