Project

General

Profile

Actions
Copy link

Bug #18820

closed

segfault in ceph-osd --flush-journal

Added by Sergey Baukin about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel,kraken
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
v10.2.5
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Got segfault while trying to flush osd journal.

ceph 10.2.5
Ubuntu 14.04.5, 3.19.0-79-generic x86_64

Debug output:

# ceph-osd -d --flush-journal --debug_filestore 20/20 --debug_journal 20/20 -i 6 
2017-02-05 17:56:58.841612 7fe5719a8800  0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
2017-02-05 17:56:58.844862 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.844993 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
2017-02-05 17:56:58.845643 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-02-05 17:56:58.845656 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2017-02-05 17:56:58.845686 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
2017-02-05 17:56:58.865060 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2017-02-05 17:56:58.865222 7fe5719a8800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
2017-02-05 17:56:58.865980 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
2017-02-05 17:56:58.866594 7fe5719a8800  1 leveldb: Recovering log #3833784
2017-02-05 17:56:58.870358 7fe5719a8800  1 leveldb: Delete type=0 #3833784

2017-02-05 17:56:58.870412 7fe5719a8800  1 leveldb: Delete type=3 #3833783

2017-02-05 17:56:58.870477 7fe5719a8800  1 leveldb: Delete type=2 #3833785

2017-02-05 17:56:58.871053 7fe55de7d700  1 leveldb: Compacting 1@1 + 4@2 files
2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.874330 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
2017-02-05 17:56:58.876352 7fe5719a8800  2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.880252 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
2017-02-05 17:56:58.880571 7fe5719a8800 10 journal  write_pos 4096
2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
2017-02-05 17:56:58.880817 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
2017-02-05 17:56:58.880852 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
2017-02-05 17:56:58.880854 7fe5719a8800  3 journal journal_replay: end of journal, done.
2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
2017-02-05 17:56:58.885209 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
2017-02-05 17:56:58.885528 7fe54ea32700  5 journal complete_write finished 0 ops and 0 bytes
2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
2017-02-05 17:56:58.886862 7fe5719a8800  1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
2017-02-05 17:56:58.886916 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
2017-02-05 17:56:58.888015 7fe5719a8800  1 journal close /var/lib/ceph/osd/ceph-6/journal
2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
*** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd
 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -121> 2017-02-05 17:56:58.834469 7fe5719a8800  5 asok(0x5596c40ba280) register_command perfcounters_dump hook 0x5596c4096050
  -120> 2017-02-05 17:56:58.834512 7fe5719a8800  5 asok(0x5596c40ba280) register_command 1 hook 0x5596c4096050
  -119> 2017-02-05 17:56:58.834525 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf dump hook 0x5596c4096050
  -118> 2017-02-05 17:56:58.834530 7fe5719a8800  5 asok(0x5596c40ba280) register_command perfcounters_schema hook 0x5596c4096050
  -117> 2017-02-05 17:56:58.834534 7fe5719a8800  5 asok(0x5596c40ba280) register_command 2 hook 0x5596c4096050
  -116> 2017-02-05 17:56:58.834541 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf schema hook 0x5596c4096050
  -115> 2017-02-05 17:56:58.834549 7fe5719a8800  5 asok(0x5596c40ba280) register_command perf reset hook 0x5596c4096050
  -114> 2017-02-05 17:56:58.834553 7fe5719a8800  5 asok(0x5596c40ba280) register_command config show hook 0x5596c4096050
  -113> 2017-02-05 17:56:58.834558 7fe5719a8800  5 asok(0x5596c40ba280) register_command config set hook 0x5596c4096050
  -112> 2017-02-05 17:56:58.834567 7fe5719a8800  5 asok(0x5596c40ba280) register_command config get hook 0x5596c4096050
  -111> 2017-02-05 17:56:58.834571 7fe5719a8800  5 asok(0x5596c40ba280) register_command config diff hook 0x5596c4096050
  -110> 2017-02-05 17:56:58.834575 7fe5719a8800  5 asok(0x5596c40ba280) register_command log flush hook 0x5596c4096050
  -109> 2017-02-05 17:56:58.834578 7fe5719a8800  5 asok(0x5596c40ba280) register_command log dump hook 0x5596c4096050
  -108> 2017-02-05 17:56:58.834585 7fe5719a8800  5 asok(0x5596c40ba280) register_command log reopen hook 0x5596c4096050
  -107> 2017-02-05 17:56:58.841612 7fe5719a8800  0 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367), process ceph-osd, pid 104785
  -106> 2017-02-05 17:56:58.841659 7fe5719a8800  5 object store type is filestore
  -105> 2017-02-05 17:56:58.844580 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) dump_stop
  -104> 2017-02-05 17:56:58.844600 7fe5719a8800  5 asok(0x5596c40ba280) init /var/run/ceph/ceph-osd.6.asok
  -103> 2017-02-05 17:56:58.844611 7fe5719a8800  5 asok(0x5596c40ba280) bind_and_listen /var/run/ceph/ceph-osd.6.asok
  -102> 2017-02-05 17:56:58.844763 7fe5719a8800  5 asok(0x5596c40ba280) register_command 0 hook 0x5596c40920a0
  -101> 2017-02-05 17:56:58.844778 7fe5719a8800  5 asok(0x5596c40ba280) register_command version hook 0x5596c40920a0
  -100> 2017-02-05 17:56:58.844782 7fe5719a8800  5 asok(0x5596c40ba280) register_command git_version hook 0x5596c40920a0
   -99> 2017-02-05 17:56:58.844786 7fe5719a8800  5 asok(0x5596c40ba280) register_command help hook 0x5596c4096140
   -98> 2017-02-05 17:56:58.844790 7fe5719a8800  5 asok(0x5596c40ba280) register_command get_command_descriptions hook 0x5596c4096130
   -97> 2017-02-05 17:56:58.844862 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) basedir /var/lib/ceph/osd/ceph-6 journal /var/lib/ceph/osd/ceph-6/journal
   -96> 2017-02-05 17:56:58.844886 7fe562e7e700  5 asok(0x5596c40ba280) entry start
   -95> 2017-02-05 17:56:58.844906 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) mount fsid is 10d2e877-6a39-48f5-bdc5-3df6fb904542
   -94> 2017-02-05 17:56:58.844953 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
   -93> 2017-02-05 17:56:58.844993 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) backend xfs (magic 0x58465342)
   -92> 2017-02-05 17:56:58.845643 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
   -91> 2017-02-05 17:56:58.845656 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
   -90> 2017-02-05 17:56:58.845686 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: splice is supported
   -89> 2017-02-05 17:56:58.865060 7fe5719a8800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
   -88> 2017-02-05 17:56:58.865222 7fe5719a8800  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-6) detect_feature: extsize is disabled by conf
   -87> 2017-02-05 17:56:58.865980 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) mount op_seq is 21415527
   -86> 2017-02-05 17:56:58.866594 7fe5719a8800  1 leveldb: Recovering log #3833784
   -85> 2017-02-05 17:56:58.870358 7fe5719a8800  1 leveldb: Delete type=0 #3833784

   -84> 2017-02-05 17:56:58.870412 7fe5719a8800  1 leveldb: Delete type=3 #3833783

   -83> 2017-02-05 17:56:58.870477 7fe5719a8800  1 leveldb: Delete type=2 #3833785

   -82> 2017-02-05 17:56:58.871053 7fe55de7d700  1 leveldb: Compacting 1@1 + 4@2 files
   -81> 2017-02-05 17:56:58.874254 7fe5719a8800 20 filestore (init)dbobjectmap: seq is 926170
   -80> 2017-02-05 17:56:58.874286 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) open_journal at /var/lib/ceph/osd/ceph-6/journal
   -79> 2017-02-05 17:56:58.874330 7fe5719a8800  0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
   -78> 2017-02-05 17:56:58.874335 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
   -77> 2017-02-05 17:56:58.876340 7fe5719a8800 10 journal journal_replay fs op_seq 21415527
   -76> 2017-02-05 17:56:58.876352 7fe5719a8800  2 journal open /var/lib/ceph/osd/ceph-6/journal fsid 10d2e877-6a39-48f5-bdc5-3df6fb904542 fs_op_seq 21415527
   -75> 2017-02-05 17:56:58.876396 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
   -74> 2017-02-05 17:56:58.876387 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
   -73> 2017-02-05 17:56:58.879854 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
   -72> 2017-02-05 17:56:58.880252 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
   -71> 2017-02-05 17:56:58.880271 7fe5719a8800 10 journal read_header
   -70> 2017-02-05 17:56:58.880552 7fe5719a8800 10 journal header: block_size 4096 alignment 4096 max_size 5368709120
   -69> 2017-02-05 17:56:58.880569 7fe5719a8800 10 journal header: start 1938739200
   -68> 2017-02-05 17:56:58.880571 7fe5719a8800 10 journal  write_pos 4096
   -67> 2017-02-05 17:56:58.880575 7fe5719a8800 10 journal open header.fsid = 10d2e877-6a39-48f5-bdc5-3df6fb904542
   -66> 2017-02-05 17:56:58.880817 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
   -65> 2017-02-05 17:56:58.880832 7fe5719a8800 10 journal open reached end of journal.
   -64> 2017-02-05 17:56:58.880852 7fe5719a8800  2 journal No further valid entries found, journal is most likely valid
   -63> 2017-02-05 17:56:58.880854 7fe5719a8800  3 journal journal_replay: end of journal, done.
   -62> 2017-02-05 17:56:58.880857 7fe5719a8800 10 journal make_writeable
   -61> 2017-02-05 17:56:58.881151 7fe5719a8800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5368709120)
   -60> 2017-02-05 17:56:58.884734 7fe5719a8800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/var/lib/ceph/osd/ceph-6/journal)
   -59> 2017-02-05 17:56:58.885209 7fe5719a8800  1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 13: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 1
   -58> 2017-02-05 17:56:58.885360 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) init_temp_collections
   -57> 2017-02-05 17:56:58.885383 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) list_collections
   -56> 2017-02-05 17:56:58.885408 7fe54ea32700 10 journal write_thread_entry start
   -55> 2017-02-05 17:56:58.885423 7fe54ea32700 20 journal prepare_multi_write queue_pos now 1938739200
   -54> 2017-02-05 17:56:58.885422 7fe549a31700 10 journal write_finish_thread_entry enter
   -53> 2017-02-05 17:56:58.885455 7fe549a31700 20 journal write_finish_thread_entry sleeping
   -52> 2017-02-05 17:56:58.885462 7fe54ea32700 15 journal do_aio_write writing 1938739200~0 + header
   -51> 2017-02-05 17:56:58.885466 7fe54ea32700 20 journal write_aio_bl 0~4096 seq 0
   -50> 2017-02-05 17:56:58.885475 7fe54ea32700 20 journal write_aio_bl .. 0~4096 in 1
   -49> 2017-02-05 17:56:58.885514 7fe54ea32700 20 journal write_aio_bl io_submit return value: 1
   -48> 2017-02-05 17:56:58.885525 7fe54ea32700 20 journal write_aio_bl 1938739200~0 seq 0
   -47> 2017-02-05 17:56:58.885528 7fe54ea32700  5 journal complete_write finished 0 ops and 0 bytes
   -46> 2017-02-05 17:56:58.885531 7fe54ea32700 20 journal write_thread_entry going to sleep
   -45> 2017-02-05 17:56:58.885565 7fe549a31700 20 journal write_finish_thread_entry waiting for aio(s)
   -44> 2017-02-05 17:56:58.885849 7fe549a31700 10 journal write_finish_thread_entry aio 0~4096 done
   -43> 2017-02-05 17:56:58.885870 7fe549a31700 20 journal check_aio_completion
   -42> 2017-02-05 17:56:58.885873 7fe549a31700 20 journal check_aio_completion completed seq 0 0~4096
   -41> 2017-02-05 17:56:58.885879 7fe549a31700 20 journal write_finish_thread_entry sleeping
   -40> 2017-02-05 17:56:58.886400 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  ls [meta,29.41_TEMP,29.30_head,29.30_TEMP,29.42_head,29.42_TEMP,29.5e_head,29.14_TEMP,29.69_head,29.69_TEMP,29.5e_TEMP,29.2_head,29.60_head,29.60_TEMP,29.67_head,29.67_TEMP,29.4_head,29.4_TEMP,29.68_head,29.68_TEMP,29.76_head,29.76_TEMP,29.36_head,29.36_TEMP,29.23_head,29.23_TEMP,29.0_head,29.0_TEMP,29.6_head,29.6_TEMP,29.18_head,29.2_TEMP,29.5d_head,29.5d_TEMP,29.74_head,29.74_TEMP,29.4b_head,29.4b_TEMP,29.3_head,29.3_TEMP,29.46_head,29.46_TEMP,29.50_head,29.50_TEMP,29.d_head,29.d_TEMP,29.59_head,29.59_TEMP,29.34_head,29.34_TEMP,29.6f_head,29.6f_TEMP,29.49_head,29.49_TEMP,29.7c_head,29.7c_TEMP,29.4d_head,29.4d_TEMP,29.4e_head,29.1_head,29.44_head,29.18_TEMP,29.15_head,29.7a_head,29.7a_TEMP,29.45_head,29.45_TEMP,29.44_TEMP,29.35_head,29.15_TEMP,29.58_head,29.66_head,29.6e_head,29.38_head,29.38_TEMP,29.2e_head,29.47_TEMP,29.8_head,29.7b_TEMP,29.35_TEMP,29.1e_head,29.4e_TEMP,29.1_TEMP,29.4a_head,29.5b_head,29.1e_TEMP,29.43_head,29.43_TEMP,29.2d_head,29.2e_TEMP,29.6a_head,29.8_TEMP,29.71_head,29.5b_TEMP,29.25_head,29.31_head,29.4a_TEMP,29.7d_head,29.25_TEMP,29.7d_TEMP,29.57_head,29.3f_head,29.3f_TEMP,29.24_head,29.24_TEMP,29.6c_head,29.57_TEMP,29.5_head,29.5_TEMP,29.6b_head,29.6b_TEMP,29.54_head,29.54_TEMP,29.6c_TEMP,29.70_head,29.65_head,29.65_TEMP,29.a_head,29.a_TEMP,29.72_head,29.72_TEMP,29.41_head,29.10_head,29.70_TEMP,29.58_TEMP,29.66_TEMP,29.6e_TEMP,29.2d_TEMP,29.47_head,29.6a_TEMP,29.7b_head,29.71_TEMP,29.10_TEMP,29.b_head,29.31_TEMP,29.14_head,29.b_TEMP,29.1b_head,29.1b_TEMP,29.20_head,29.20_TEMP,29.37_head,29.37_TEMP,29.53_head,29.53_TEMP,29.55_head,29.55_TEMP,29.1c_head,29.1c_TEMP,29.7f_head,29.7f_TEMP,29.11_head,29.11_TEMP,29.3a_head,29.3a_TEMP,29.4c_head,29.4c_TEMP,29.75_head,29.75_TEMP,29.7_head,29.7_TEMP,29.61_head,29.61_TEMP,29.7e_head,29.7e_TEMP]
   -39> 2017-02-05 17:56:58.886494 7fe5719a8800 20 filestore(/var/lib/ceph/osd/ceph-6)  temps 29.0_TEMP,29.1_TEMP,29.2_TEMP,29.3_TEMP,29.4_TEMP,29.5_TEMP,29.6_TEMP,29.7_TEMP,29.8_TEMP,29.a_TEMP,29.b_TEMP,29.d_TEMP,29.10_TEMP,29.11_TEMP,29.14_TEMP,29.15_TEMP,29.18_TEMP,29.1b_TEMP,29.1c_TEMP,29.1e_TEMP,29.20_TEMP,29.23_TEMP,29.24_TEMP,29.25_TEMP,29.2d_TEMP,29.2e_TEMP,29.30_TEMP,29.31_TEMP,29.34_TEMP,29.35_TEMP,29.36_TEMP,29.37_TEMP,29.38_TEMP,29.3a_TEMP,29.3f_TEMP,29.41_TEMP,29.42_TEMP,29.43_TEMP,29.44_TEMP,29.45_TEMP,29.46_TEMP,29.47_TEMP,29.49_TEMP,29.4a_TEMP,29.4b_TEMP,29.4c_TEMP,29.4d_TEMP,29.4e_TEMP,29.50_TEMP,29.53_TEMP,29.54_TEMP,29.55_TEMP,29.57_TEMP,29.58_TEMP,29.59_TEMP,29.5b_TEMP,29.5d_TEMP,29.5e_TEMP,29.60_TEMP,29.61_TEMP,29.65_TEMP,29.66_TEMP,29.67_TEMP,29.68_TEMP,29.69_TEMP,29.6a_TEMP,29.6b_TEMP,29.6c_TEMP,29.6e_TEMP,29.6f_TEMP,29.70_TEMP,29.71_TEMP,29.72_TEMP,29.74_TEMP,29.75_TEMP,29.76_TEMP,29.7a_TEMP,29.7b_TEMP,29.7c_TEMP,29.7d_TEMP,29.7e_TEMP,29.7f_TEMP
   -38> 2017-02-05 17:56:58.886563 7fe5719a8800 10 journal journal_start
   -37> 2017-02-05 17:56:58.886862 7fe5719a8800  1 filestore(/var/lib/ceph/osd/ceph-6) upgrade
   -36> 2017-02-05 17:56:58.886905 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) version_stamp_is_valid was 4 vs target 4
   -35> 2017-02-05 17:56:58.886916 7fe5719a8800  5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6
   -34> 2017-02-05 17:56:58.886922 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush
   -33> 2017-02-05 17:56:58.886925 7fe5719a8800 10 journal waiting for completions to empty
   -32> 2017-02-05 17:56:58.886928 7fe5719a8800 10 journal flush waiting for finisher
   -31> 2017-02-05 17:56:58.886931 7fe5719a8800 10 journal flush done
   -30> 2017-02-05 17:56:58.886932 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher
   -29> 2017-02-05 17:56:58.886934 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp
   -28> 2017-02-05 17:56:58.886937 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher
   -27> 2017-02-05 17:56:58.886939 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete
   -26> 2017-02-05 17:56:58.886949 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync
   -25> 2017-02-05 17:56:58.886952 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting
   -24> 2017-02-05 17:56:58.887017 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
   -23> 2017-02-05 17:56:58.887033 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
   -22> 2017-02-05 17:56:58.887037 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
   -21> 2017-02-05 17:56:58.887039 7fe553a33700 10 journal commit_start nothing to do
   -20> 2017-02-05 17:56:58.887042 7fe553a33700 10 journal commit_start
   -19> 2017-02-05 17:56:58.887072 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000
   -18> 2017-02-05 17:56:58.887115 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) sync done
   -17> 2017-02-05 17:56:58.887127 7fe5719a8800 10 filestore(/var/lib/ceph/osd/ceph-6) do_force_sync
   -16> 2017-02-05 17:56:58.887194 7fe553a33700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set
   -15> 2017-02-05 17:56:58.887208 7fe553a33700 10 journal commit_start max_applied_seq 21415527, open_ops 0
   -14> 2017-02-05 17:56:58.887212 7fe553a33700 10 journal commit_start blocked, all open_ops have completed
   -13> 2017-02-05 17:56:58.887215 7fe553a33700 10 journal commit_start nothing to do
   -12> 2017-02-05 17:56:58.887217 7fe553a33700 10 journal commit_start
   -11> 2017-02-05 17:56:58.887601 7fe53fa2f700  1 FileStore::op_tp worker finish
   -10> 2017-02-05 17:56:58.887669 7fe53aa2e700  1 FileStore::op_tp worker finish
    -9> 2017-02-05 17:56:58.887815 7fe5719a8800 10 journal journal_stop
    -8> 2017-02-05 17:56:58.888015 7fe5719a8800  1 journal close /var/lib/ceph/osd/ceph-6/journal
    -7> 2017-02-05 17:56:58.888117 7fe54ea32700 20 journal write_thread_entry woke up
    -6> 2017-02-05 17:56:58.888128 7fe54ea32700 10 journal write_thread_entry finish
    -5> 2017-02-05 17:56:58.888259 7fe5719a8800 15 journal do_write writing 1938739200~0 + header
    -4> 2017-02-05 17:56:58.888688 7fe5719a8800 20 journal do_write latency 0.000408
    -3> 2017-02-05 17:56:58.888709 7fe5719a8800 20 journal do_write queueing finishers through seq 0
    -2> 2017-02-05 17:56:58.888716 7fe5719a8800 20 journal write_header_sync finish
    -1> 2017-02-05 17:56:58.888772 7fe549a31700 10 journal write_finish_thread_entry exit
     0> 2017-02-05 17:56:58.890384 7fe55de7d700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fe55de7d700 thread_name:ceph-osd

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x8f2d32) [0x5596b8642d32]
 2: (()+0x10330) [0x7fe57087e330]
 3: [0x5596c41638a0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
  20/20 filestore
  20/20 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file 
--- end dump of recent events ---
Segmentation fault (core dumped)

strace -f output attached.

Files

ceph-osd.strace (195 KB) ceph-osd.strace strace -f ceph-osd --flush-journal Sergey Baukin, 02/05/2017 03:05 PM

Related issues 2 (0 open — 2 closed)

Copied to Ceph - Backport #18951: jewel: segfault in ceph-osd --flush-journalResolvedAlexey SheplyakovActions
Copied to Ceph - Backport #18952: kraken: segfault in ceph-osd --flush-journalResolvedShinobu KinjoActions
Actions
Copy link
 #1

Updated by Sergey Baukin about 7 years ago 

# gdb ceph-osd
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ceph-osd...(no debugging symbols found)...done.
(gdb) run -i 6 --flush-journal
Starting program: /usr/bin/ceph-osd -i 6 --flush-journal
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3c37700 (LWP 108915)]
[New Thread 0x7fffee4b9700 (LWP 108916)]
[New Thread 0x7fffe94b8700 (LWP 108917)]
[New Thread 0x7fffe44b7700 (LWP 108919)]
[New Thread 0x7fffdee48700 (LWP 108920)]
[New Thread 0x7fffd9e47700 (LWP 108921)]
[New Thread 0x7fffd4e46700 (LWP 108926)]
[New Thread 0x7fffcfe45700 (LWP 108927)]
[New Thread 0x7fffcae44700 (LWP 108928)]
[New Thread 0x7fffc5e43700 (LWP 108929)]
[New Thread 0x7fffc0e42700 (LWP 108930)]
[New Thread 0x7fffbbe41700 (LWP 108931)]
[New Thread 0x7fffb6e40700 (LWP 108932)]
[New Thread 0x7fffb1e3f700 (LWP 108933)]
[Thread 0x7fffd9e47700 (LWP 108921) exited]
[Thread 0x7fffdee48700 (LWP 108920) exited]
[Thread 0x7fffc0e42700 (LWP 108930) exited]
[Thread 0x7fffc5e43700 (LWP 108929) exited]
[Thread 0x7fffcae44700 (LWP 108928) exited]
[Thread 0x7fffd4e46700 (LWP 108926) exited]
[Thread 0x7fffcfe45700 (LWP 108927) exited]
[Thread 0x7fffbbe41700 (LWP 108931) exited]
[Thread 0x7fffb6e40700 (LWP 108932) exited]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe44b7700 (LWP 108919)]
0x000055555fdcd8a0 in ?? ()
(gdb) bt
#0  0x000055555fdcd8a0 in ?? ()
#1  0x00007ffff753f2fd in leveldb::Log(leveldb::Logger*, char const*, ...) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#2  0x00007ffff752060c in leveldb::DBImpl::DoCompactionWork(leveldb::DBImpl::CompactionState*) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#3  0x00007ffff7520fd3 in leveldb::DBImpl::BackgroundCompaction() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#4  0x00007ffff75219b2 in leveldb::DBImpl::BackgroundCall() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#5  0x00007ffff753fb3b in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#6  0x00007ffff6eb0184 in start_thread (arg=0x7fffe44b7700) at pthread_create.c:312
#7  0x00007ffff4fdb37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Actions
Copy link
 #2

Updated by Shinobu Kinjo about 7 years ago

Can't reproduce with:
  1. uname -r
    3.10.0-514.2.2.el7.x86_64
  2. ceph -v
    ceph version 10.2.5-5663-g601fb96 (601fb96599ebcb07017951542ddc58b2ab2abfd4)
Actions
Copy link
 #3

Updated by Kjetil Joergensen about 7 years ago

Were this by chance a newly created journal ? (As in, not really used before you tried to flush). We had a similar problem where creating a new journal, immediately followed by flush ended up with segfault (It's entirely possible that the journal being a NVMe device helped as well). (I'm uncertain if we actually filed a bug for that). Our hacky workaround ended up running the flush inside of gdb. (gdb -ex run --args ceph-osd -d --setuser ceph --setgroup ceph --flush-journal -i "${osdid}" --osd-data "${OSDTMP}" --osd-journal "${JOURNAL}" --debug_osd 1).

Actions
Copy link
 #4

Updated by Kjetil Joergensen about 7 years ago

Please ignore me - you appear to have stumbled across a different problem, given that you managed to get it to SEGFAULT inside of gdb. We had a problem with flush beeing a bit too cheap, so threads normally would have finished, hadn't and gdb solved that for us.

Actions
Copy link
 #5

Updated by Alexey Sheplyakov about 7 years ago

@Sergey:
Could you please check if https://github.com/ceph/ceph/pull/13311 solves the problem for you?

Actions
Copy link
 #6

Updated by Alexey Sheplyakov about 7 years ago

  • Backport set to jewel,kraken
Actions
Copy link
 #7

Updated by Sergey Baukin about 7 years ago

Alexey Sheplyakov wrote:

@Sergey:
Could you please check if https://github.com/ceph/ceph/pull/13311 solves the problem for you?

Sorry, can't check now.
That was a part of crashed OSD reapairing. After a number of starts/stops and backfilling the journal flushes ok, but as for now I still have a PG which causes other fault. I have exported faulty PG 29.45 and trying to import it back to the cluster.

# ceph-objectstore-tool --debug --op import  --data-path /var/lib/ceph/osd/ceph-3/ --journal-path /var/lib/ceph/osd/ceph-3/journal --file 29.45.export

(a lot of Write #29...)

Write #29:a25ca113:::rbd_data.43777e2d089d3b.00000000000045ab:head#
    attrs: len 2
    omap header: 
Write #29:a25ca546:::rbd_data.6b3cb8283df995.0000000000000221:4008#
    data: offset 0 len 1048576
    data: offset 1048576 len 1048576
    data: offset 2097152 len 1048576
    data: offset 3145728 len 1048576
    attrs: len 1
object_info 29:a25ca546:::rbd_data.6b3cb8283df995.0000000000000221:4008(645519'19706180 client.52733560.0:659899 [4008] dirty|omap_digest s 4194304 uv 19705475 od ffffffff)
osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const hobject_t&, const std::set<snapid_t>&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7fcee1302940 time 2017-02-12 10:01:15.964239
osd/SnapMapper.cc: 228: FAILED assert(r == -2)
 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x55e9aa2c38db]
 2: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 3: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 6: (main()+0x2dce) [0x55e9a9bccc3e]
 7: (__libc_start_main()+0xf0) [0x7fceddf01830]
 8: (()+0x363777) [0x55e9a9c18777]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-02-12 10:01:15.966439 7fcee1302940 -1 osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const hobject_t&, const std::set<snapid_t>&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7fcee1302940 time 2017-02-12 10:01:15.964239
osd/SnapMapper.cc: 228: FAILED assert(r == -2)

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x55e9aa2c38db]
 2: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 3: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 6: (main()+0x2dce) [0x55e9a9bccc3e]
 7: (__libc_start_main()+0xf0) [0x7fceddf01830]
 8: (()+0x363777) [0x55e9a9c18777]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -500> 2017-02-12 10:01:13.459352 7fcee1302940  3 journal journal_replay: applying op seq 26517079
  -499> 2017-02-12 10:01:13.459543 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517079
  -498> 2017-02-12 10:01:13.459556 7fcee1302940  2 journal read_entry 4927377408 : seq 26517080 733 bytes
  -497> 2017-02-12 10:01:13.459561 7fcee1302940  3 journal journal_replay: applying op seq 26517080
  -496> 2017-02-12 10:01:13.459765 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517080
  -495> 2017-02-12 10:01:13.459778 7fcee1302940  2 journal read_entry 4927381504 : seq 26517081 733 bytes
  -494> 2017-02-12 10:01:13.459782 7fcee1302940  3 journal journal_replay: applying op seq 26517081
  -493> 2017-02-12 10:01:13.459958 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517081
  -492> 2017-02-12 10:01:13.459970 7fcee1302940  2 journal read_entry 4927385600 : seq 26517082 731 bytes
  -491> 2017-02-12 10:01:13.459974 7fcee1302940  3 journal journal_replay: applying op seq 26517082
  -490> 2017-02-12 10:01:13.460163 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517082
  -489> 2017-02-12 10:01:13.460177 7fcee1302940  2 journal read_entry 4927389696 : seq 26517083 733 bytes
  -488> 2017-02-12 10:01:13.460181 7fcee1302940  3 journal journal_replay: applying op seq 26517083
  -487> 2017-02-12 10:01:13.460375 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517083
  -486> 2017-02-12 10:01:13.475462 7fcee1302940  2 journal read_entry 4927393792 : seq 26517084 4195339 bytes
  -485> 2017-02-12 10:01:13.475474 7fcee1302940  3 journal journal_replay: applying op seq 26517084
  -484> 2017-02-12 10:01:13.477162 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517084
  -483> 2017-02-12 10:01:13.477418 7fcee1302940  2 journal read_entry 4931596288 : seq 26517085 731 bytes
  -482> 2017-02-12 10:01:13.477424 7fcee1302940  3 journal journal_replay: applying op seq 26517085
  -481> 2017-02-12 10:01:13.477616 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517085
  -480> 2017-02-12 10:01:13.477629 7fcee1302940  2 journal read_entry 4931600384 : seq 26517086 731 bytes
  -479> 2017-02-12 10:01:13.477631 7fcee1302940  3 journal journal_replay: applying op seq 26517086
  -478> 2017-02-12 10:01:13.477805 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517086
  -477> 2017-02-12 10:01:13.493013 7fcee1302940  2 journal read_entry 4931604480 : seq 26517087 4195339 bytes
  -476> 2017-02-12 10:01:13.493027 7fcee1302940  3 journal journal_replay: applying op seq 26517087
  -475> 2017-02-12 10:01:13.494840 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517087
  -474> 2017-02-12 10:01:13.495114 7fcee1302940  2 journal read_entry 4935806976 : seq 26517088 731 bytes
  -473> 2017-02-12 10:01:13.495120 7fcee1302940  3 journal journal_replay: applying op seq 26517088
  -472> 2017-02-12 10:01:13.495345 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517088
  -471> 2017-02-12 10:01:13.495370 7fcee1302940  2 journal read_entry 4935811072 : seq 26517089 733 bytes
  -470> 2017-02-12 10:01:13.495373 7fcee1302940  3 journal journal_replay: applying op seq 26517089
  -469> 2017-02-12 10:01:13.495594 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517089
  -468> 2017-02-12 10:01:13.495618 7fcee1302940  2 journal read_entry 4935815168 : seq 26517090 731 bytes
  -467> 2017-02-12 10:01:13.495621 7fcee1302940  3 journal journal_replay: applying op seq 26517090
  -466> 2017-02-12 10:01:13.495840 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517090
  -465> 2017-02-12 10:01:13.495866 7fcee1302940  2 journal read_entry 4935819264 : seq 26517091 731 bytes
  -464> 2017-02-12 10:01:13.495877 7fcee1302940  3 journal journal_replay: applying op seq 26517091
  -463> 2017-02-12 10:01:13.496078 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517091
  -462> 2017-02-12 10:01:13.496092 7fcee1302940  2 journal read_entry 4935823360 : seq 26517092 731 bytes
  -461> 2017-02-12 10:01:13.496095 7fcee1302940  3 journal journal_replay: applying op seq 26517092
  -460> 2017-02-12 10:01:13.496279 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517092
  -459> 2017-02-12 10:01:13.496302 7fcee1302940  2 journal read_entry 4935827456 : seq 26517093 731 bytes
  -458> 2017-02-12 10:01:13.496305 7fcee1302940  3 journal journal_replay: applying op seq 26517093
  -457> 2017-02-12 10:01:13.496498 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517093
  -456> 2017-02-12 10:01:13.496521 7fcee1302940  2 journal read_entry 4935831552 : seq 26517094 731 bytes
  -455> 2017-02-12 10:01:13.496524 7fcee1302940  3 journal journal_replay: applying op seq 26517094
  -454> 2017-02-12 10:01:13.496695 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517094
  -453> 2017-02-12 10:01:13.511914 7fcee1302940  2 journal read_entry 4935835648 : seq 26517095 4196903 bytes
  -452> 2017-02-12 10:01:13.511928 7fcee1302940  3 journal journal_replay: applying op seq 26517095
  -451> 2017-02-12 10:01:13.513690 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517095
  -450> 2017-02-12 10:01:13.513960 7fcee1302940  2 journal read_entry 4940038144 : seq 26517096 729 bytes
  -449> 2017-02-12 10:01:13.513967 7fcee1302940  3 journal journal_replay: applying op seq 26517096
  -448> 2017-02-12 10:01:13.514152 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517096
  -447> 2017-02-12 10:01:13.529447 7fcee1302940  2 journal read_entry 4940042240 : seq 26517097 4195341 bytes
  -446> 2017-02-12 10:01:13.529459 7fcee1302940  3 journal journal_replay: applying op seq 26517097
  -445> 2017-02-12 10:01:13.531281 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517097
  -444> 2017-02-12 10:01:13.531608 7fcee1302940  2 journal read_entry 4944244736 : seq 26517098 731 bytes
  -443> 2017-02-12 10:01:13.531615 7fcee1302940  3 journal journal_replay: applying op seq 26517098
  -442> 2017-02-12 10:01:13.531818 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517098
  -441> 2017-02-12 10:01:13.531832 7fcee1302940  2 journal read_entry 4944248832 : seq 26517099 731 bytes
  -440> 2017-02-12 10:01:13.531835 7fcee1302940  3 journal journal_replay: applying op seq 26517099
  -439> 2017-02-12 10:01:13.532024 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517099
  -438> 2017-02-12 10:01:13.532037 7fcee1302940  2 journal read_entry 4944252928 : seq 26517100 729 bytes
  -437> 2017-02-12 10:01:13.532040 7fcee1302940  3 journal journal_replay: applying op seq 26517100
  -436> 2017-02-12 10:01:13.532231 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517100
  -435> 2017-02-12 10:01:13.532244 7fcee1302940  2 journal read_entry 4944257024 : seq 26517101 729 bytes
  -434> 2017-02-12 10:01:13.532248 7fcee1302940  3 journal journal_replay: applying op seq 26517101
  -433> 2017-02-12 10:01:13.532433 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517101
  -432> 2017-02-12 10:01:13.532445 7fcee1302940  2 journal read_entry 4944261120 : seq 26517102 731 bytes
  -431> 2017-02-12 10:01:13.532450 7fcee1302940  3 journal journal_replay: applying op seq 26517102
  -430> 2017-02-12 10:01:13.532624 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517102
  -429> 2017-02-12 10:01:13.532637 7fcee1302940  2 journal read_entry 4944265216 : seq 26517103 731 bytes
  -428> 2017-02-12 10:01:13.532641 7fcee1302940  3 journal journal_replay: applying op seq 26517103
  -427> 2017-02-12 10:01:13.532844 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517103
  -426> 2017-02-12 10:01:13.532858 7fcee1302940  2 journal read_entry 4944269312 : seq 26517104 733 bytes
  -425> 2017-02-12 10:01:13.532863 7fcee1302940  3 journal journal_replay: applying op seq 26517104
  -424> 2017-02-12 10:01:13.533089 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517104
  -423> 2017-02-12 10:01:13.533102 7fcee1302940  2 journal read_entry 4944273408 : seq 26517105 733 bytes
  -422> 2017-02-12 10:01:13.533106 7fcee1302940  3 journal journal_replay: applying op seq 26517105
  -421> 2017-02-12 10:01:13.533322 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517105
  -420> 2017-02-12 10:01:13.533335 7fcee1302940  2 journal read_entry 4944277504 : seq 26517106 733 bytes
  -419> 2017-02-12 10:01:13.533339 7fcee1302940  3 journal journal_replay: applying op seq 26517106
  -418> 2017-02-12 10:01:13.533512 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517106
  -417> 2017-02-12 10:01:13.541434 7fcee1302940  2 journal read_entry 4944281600 : seq 26517107 2220991 bytes
  -416> 2017-02-12 10:01:13.541449 7fcee1302940  3 journal journal_replay: applying op seq 26517107
  -415> 2017-02-12 10:01:13.542575 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517107
  -414> 2017-02-12 10:01:13.542708 7fcee1302940  2 journal read_entry 4946509824 : seq 26517108 733 bytes
  -413> 2017-02-12 10:01:13.542714 7fcee1302940  3 journal journal_replay: applying op seq 26517108
  -412> 2017-02-12 10:01:13.542914 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517108
  -411> 2017-02-12 10:01:13.558029 7fcee1302940  2 journal read_entry 4946513920 : seq 26517109 4195341 bytes
  -410> 2017-02-12 10:01:13.558042 7fcee1302940  3 journal journal_replay: applying op seq 26517109
  -409> 2017-02-12 10:01:13.559813 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517109
  -408> 2017-02-12 10:01:13.560080 7fcee1302940  2 journal read_entry 4950716416 : seq 26517110 733 bytes
  -407> 2017-02-12 10:01:13.560087 7fcee1302940  3 journal journal_replay: applying op seq 26517110
  -406> 2017-02-12 10:01:13.560278 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517110
  -405> 2017-02-12 10:01:13.575456 7fcee1302940  2 journal read_entry 4950720512 : seq 26517111 4195337 bytes
  -404> 2017-02-12 10:01:13.575469 7fcee1302940  3 journal journal_replay: applying op seq 26517111
  -403> 2017-02-12 10:01:13.577257 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517111
  -402> 2017-02-12 10:01:13.577521 7fcee1302940  2 journal read_entry 4954923008 : seq 26517112 731 bytes
  -401> 2017-02-12 10:01:13.577527 7fcee1302940  3 journal journal_replay: applying op seq 26517112
  -400> 2017-02-12 10:01:13.577738 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517112
  -399> 2017-02-12 10:01:13.577752 7fcee1302940  2 journal read_entry 4954927104 : seq 26517113 731 bytes
  -398> 2017-02-12 10:01:13.577756 7fcee1302940  3 journal journal_replay: applying op seq 26517113
  -397> 2017-02-12 10:01:13.577956 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517113
  -396> 2017-02-12 10:01:13.577970 7fcee1302940  2 journal read_entry 4954931200 : seq 26517114 733 bytes
  -395> 2017-02-12 10:01:13.577974 7fcee1302940  3 journal journal_replay: applying op seq 26517114
  -394> 2017-02-12 10:01:13.578236 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517114
  -393> 2017-02-12 10:01:13.593363 7fcee1302940  2 journal read_entry 4954935296 : seq 26517115 4195341 bytes
  -392> 2017-02-12 10:01:13.593375 7fcee1302940  3 journal journal_replay: applying op seq 26517115
  -391> 2017-02-12 10:01:13.595165 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517115
  -390> 2017-02-12 10:01:13.595435 7fcee1302940  2 journal read_entry 4959137792 : seq 26517116 733 bytes
  -389> 2017-02-12 10:01:13.595442 7fcee1302940  3 journal journal_replay: applying op seq 26517116
  -388> 2017-02-12 10:01:13.595653 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517116
  -387> 2017-02-12 10:01:13.595667 7fcee1302940  2 journal read_entry 4959141888 : seq 26517117 733 bytes
  -386> 2017-02-12 10:01:13.595680 7fcee1302940  3 journal journal_replay: applying op seq 26517117
  -385> 2017-02-12 10:01:13.595885 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517117
  -384> 2017-02-12 10:01:13.595900 7fcee1302940  2 journal read_entry 4959145984 : seq 26517118 733 bytes
  -383> 2017-02-12 10:01:13.595905 7fcee1302940  3 journal journal_replay: applying op seq 26517118
  -382> 2017-02-12 10:01:13.596088 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517118
  -381> 2017-02-12 10:01:13.596101 7fcee1302940  2 journal read_entry 4959150080 : seq 26517119 731 bytes
  -380> 2017-02-12 10:01:13.596105 7fcee1302940  3 journal journal_replay: applying op seq 26517119
  -379> 2017-02-12 10:01:13.596290 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517119
  -378> 2017-02-12 10:01:13.596303 7fcee1302940  2 journal read_entry 4959154176 : seq 26517120 731 bytes
  -377> 2017-02-12 10:01:13.596308 7fcee1302940  3 journal journal_replay: applying op seq 26517120
  -376> 2017-02-12 10:01:13.596506 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517120
  -375> 2017-02-12 10:01:13.596519 7fcee1302940  2 journal read_entry 4959158272 : seq 26517121 731 bytes
  -374> 2017-02-12 10:01:13.596523 7fcee1302940  3 journal journal_replay: applying op seq 26517121
  -373> 2017-02-12 10:01:13.596703 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517121
  -372> 2017-02-12 10:01:13.596715 7fcee1302940  2 journal read_entry 4959162368 : seq 26517122 731 bytes
  -371> 2017-02-12 10:01:13.596719 7fcee1302940  3 journal journal_replay: applying op seq 26517122
  -370> 2017-02-12 10:01:13.596910 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517122
  -369> 2017-02-12 10:01:13.596924 7fcee1302940  2 journal read_entry 4959166464 : seq 26517123 731 bytes
  -368> 2017-02-12 10:01:13.596928 7fcee1302940  3 journal journal_replay: applying op seq 26517123
  -367> 2017-02-12 10:01:13.597123 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517123
  -366> 2017-02-12 10:01:13.612268 7fcee1302940  2 journal read_entry 4959170560 : seq 26517124 4195339 bytes
  -365> 2017-02-12 10:01:13.612281 7fcee1302940  3 journal journal_replay: applying op seq 26517124
  -364> 2017-02-12 10:01:13.614036 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517124
  -363> 2017-02-12 10:01:13.614301 7fcee1302940  2 journal read_entry 4963373056 : seq 26517125 733 bytes
  -362> 2017-02-12 10:01:13.614308 7fcee1302940  3 journal journal_replay: applying op seq 26517125
  -361> 2017-02-12 10:01:13.614580 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517125
  -360> 2017-02-12 10:01:13.614600 7fcee1302940  2 journal read_entry 4963377152 : seq 26517126 733 bytes
  -359> 2017-02-12 10:01:13.614608 7fcee1302940  3 journal journal_replay: applying op seq 26517126
  -358> 2017-02-12 10:01:13.614880 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517126
  -357> 2017-02-12 10:01:13.614924 7fcee1302940  2 journal read_entry 4963381248 : seq 26517127 731 bytes
  -356> 2017-02-12 10:01:13.614930 7fcee1302940  3 journal journal_replay: applying op seq 26517127
  -355> 2017-02-12 10:01:13.615136 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517127
  -354> 2017-02-12 10:01:13.615150 7fcee1302940  2 journal read_entry 4963385344 : seq 26517128 733 bytes
  -353> 2017-02-12 10:01:13.615157 7fcee1302940  3 journal journal_replay: applying op seq 26517128
  -352> 2017-02-12 10:01:13.615348 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517128
  -351> 2017-02-12 10:01:13.615361 7fcee1302940  2 journal read_entry 4963389440 : seq 26517129 729 bytes
  -350> 2017-02-12 10:01:13.615365 7fcee1302940  3 journal journal_replay: applying op seq 26517129
  -349> 2017-02-12 10:01:13.615585 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517129
  -348> 2017-02-12 10:01:13.615599 7fcee1302940  2 journal read_entry 4963393536 : seq 26517130 733 bytes
  -347> 2017-02-12 10:01:13.615603 7fcee1302940  3 journal journal_replay: applying op seq 26517130
  -346> 2017-02-12 10:01:13.615803 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517130
  -345> 2017-02-12 10:01:13.615817 7fcee1302940  2 journal read_entry 4963397632 : seq 26517131 733 bytes
  -344> 2017-02-12 10:01:13.615821 7fcee1302940  3 journal journal_replay: applying op seq 26517131
  -343> 2017-02-12 10:01:13.616005 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517131
  -342> 2017-02-12 10:01:13.616017 7fcee1302940  2 journal read_entry 4963401728 : seq 26517132 733 bytes
  -341> 2017-02-12 10:01:13.616021 7fcee1302940  3 journal journal_replay: applying op seq 26517132
  -340> 2017-02-12 10:01:13.616222 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517132
  -339> 2017-02-12 10:01:13.631408 7fcee1302940  2 journal read_entry 4963405824 : seq 26517133 4195339 bytes
  -338> 2017-02-12 10:01:13.631422 7fcee1302940  3 journal journal_replay: applying op seq 26517133
  -337> 2017-02-12 10:01:13.633199 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517133
  -336> 2017-02-12 10:01:13.648588 7fcee1302940  2 journal read_entry 4967608320 : seq 26517134 4195339 bytes
  -335> 2017-02-12 10:01:13.648602 7fcee1302940  3 journal journal_replay: applying op seq 26517134
  -334> 2017-02-12 10:01:13.650384 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517134
  -333> 2017-02-12 10:01:13.650649 7fcee1302940  2 journal read_entry 4971810816 : seq 26517135 733 bytes
  -332> 2017-02-12 10:01:13.650657 7fcee1302940  3 journal journal_replay: applying op seq 26517135
  -331> 2017-02-12 10:01:13.650864 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517135
  -330> 2017-02-12 10:01:13.665987 7fcee1302940  2 journal read_entry 4971814912 : seq 26517136 4195339 bytes
  -329> 2017-02-12 10:01:13.666001 7fcee1302940  3 journal journal_replay: applying op seq 26517136
  -328> 2017-02-12 10:01:13.667765 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517136
  -327> 2017-02-12 10:01:13.668037 7fcee1302940  2 journal read_entry 4976017408 : seq 26517137 731 bytes
  -326> 2017-02-12 10:01:13.668044 7fcee1302940  3 journal journal_replay: applying op seq 26517137
  -325> 2017-02-12 10:01:13.668241 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517137
  -324> 2017-02-12 10:01:13.668259 7fcee1302940  2 journal read_entry 4976021504 : seq 26517138 731 bytes
  -323> 2017-02-12 10:01:13.668267 7fcee1302940  3 journal journal_replay: applying op seq 26517138
  -322> 2017-02-12 10:01:13.668483 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517138
  -321> 2017-02-12 10:01:13.668497 7fcee1302940  2 journal read_entry 4976025600 : seq 26517139 731 bytes
  -320> 2017-02-12 10:01:13.668501 7fcee1302940  3 journal journal_replay: applying op seq 26517139
  -319> 2017-02-12 10:01:13.668689 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517139
  -318> 2017-02-12 10:01:13.668703 7fcee1302940  2 journal read_entry 4976029696 : seq 26517140 947 bytes
  -317> 2017-02-12 10:01:13.668707 7fcee1302940  3 journal journal_replay: applying op seq 26517140
  -316> 2017-02-12 10:01:13.668888 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517140
  -315> 2017-02-12 10:01:13.668901 7fcee1302940  2 journal read_entry 4976033792 : seq 26517141 731 bytes
  -314> 2017-02-12 10:01:13.668905 7fcee1302940  3 journal journal_replay: applying op seq 26517141
  -313> 2017-02-12 10:01:13.669109 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517141
  -312> 2017-02-12 10:01:13.683806 7fcee1302940  2 journal read_entry 4976037888 : seq 26517142 4088843 bytes
  -311> 2017-02-12 10:01:13.683820 7fcee1302940  3 journal journal_replay: applying op seq 26517142
  -310> 2017-02-12 10:01:13.685550 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517142
  -309> 2017-02-12 10:01:13.685814 7fcee1302940  2 journal read_entry 4980133888 : seq 26517143 731 bytes
  -308> 2017-02-12 10:01:13.685821 7fcee1302940  3 journal journal_replay: applying op seq 26517143
  -307> 2017-02-12 10:01:13.686024 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517143
  -306> 2017-02-12 10:01:13.686037 7fcee1302940  2 journal read_entry 4980137984 : seq 26517144 733 bytes
  -305> 2017-02-12 10:01:13.686042 7fcee1302940  3 journal journal_replay: applying op seq 26517144
  -304> 2017-02-12 10:01:13.686244 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517144
  -303> 2017-02-12 10:01:13.686257 7fcee1302940  2 journal read_entry 4980142080 : seq 26517145 733 bytes
  -302> 2017-02-12 10:01:13.686261 7fcee1302940  3 journal journal_replay: applying op seq 26517145
  -301> 2017-02-12 10:01:13.686441 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517145
  -300> 2017-02-12 10:01:13.686454 7fcee1302940  2 journal read_entry 4980146176 : seq 26517146 731 bytes
  -299> 2017-02-12 10:01:13.686458 7fcee1302940  3 journal journal_replay: applying op seq 26517146
  -298> 2017-02-12 10:01:13.686657 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517146
  -297> 2017-02-12 10:01:13.693483 7fcee1302940  2 journal read_entry 4980150272 : seq 26517147 1876851 bytes
  -296> 2017-02-12 10:01:13.693493 7fcee1302940  3 journal journal_replay: applying op seq 26517147
  -295> 2017-02-12 10:01:13.694409 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517147
  -294> 2017-02-12 10:01:13.694519 7fcee1302940  2 journal read_entry 4982034432 : seq 26517148 729 bytes
  -293> 2017-02-12 10:01:13.694525 7fcee1302940  3 journal journal_replay: applying op seq 26517148
  -292> 2017-02-12 10:01:13.694751 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517148
  -291> 2017-02-12 10:01:13.708414 7fcee1302940  2 journal read_entry 4982038528 : seq 26517149 3814411 bytes
  -290> 2017-02-12 10:01:13.708426 7fcee1302940  3 journal journal_replay: applying op seq 26517149
  -289> 2017-02-12 10:01:13.710047 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517149
  -288> 2017-02-12 10:01:13.710286 7fcee1302940  2 journal read_entry 4985860096 : seq 26517150 733 bytes
  -287> 2017-02-12 10:01:13.710293 7fcee1302940  3 journal journal_replay: applying op seq 26517150
  -286> 2017-02-12 10:01:13.710499 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517150
  -285> 2017-02-12 10:01:13.710513 7fcee1302940  2 journal read_entry 4985864192 : seq 26517151 733 bytes
  -284> 2017-02-12 10:01:13.710521 7fcee1302940  3 journal journal_replay: applying op seq 26517151
  -283> 2017-02-12 10:01:13.710708 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517151
  -282> 2017-02-12 10:01:13.710721 7fcee1302940  2 journal read_entry 4985868288 : seq 26517152 733 bytes
  -281> 2017-02-12 10:01:13.710726 7fcee1302940  3 journal journal_replay: applying op seq 26517152
  -280> 2017-02-12 10:01:13.710929 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517152
  -279> 2017-02-12 10:01:13.710943 7fcee1302940  2 journal read_entry 4985872384 : seq 26517153 733 bytes
  -278> 2017-02-12 10:01:13.710947 7fcee1302940  3 journal journal_replay: applying op seq 26517153
  -277> 2017-02-12 10:01:13.711136 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517153
  -276> 2017-02-12 10:01:13.711149 7fcee1302940  2 journal read_entry 4985876480 : seq 26517154 731 bytes
  -275> 2017-02-12 10:01:13.711153 7fcee1302940  3 journal journal_replay: applying op seq 26517154
  -274> 2017-02-12 10:01:13.711333 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517154
  -273> 2017-02-12 10:01:13.711345 7fcee1302940  2 journal read_entry 4985880576 : seq 26517155 733 bytes
  -272> 2017-02-12 10:01:13.711350 7fcee1302940  3 journal journal_replay: applying op seq 26517155
  -271> 2017-02-12 10:01:13.711542 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517155
  -270> 2017-02-12 10:01:13.711556 7fcee1302940  2 journal read_entry 4985884672 : seq 26517156 731 bytes
  -269> 2017-02-12 10:01:13.711560 7fcee1302940  3 journal journal_replay: applying op seq 26517156
  -268> 2017-02-12 10:01:13.711797 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517156
  -267> 2017-02-12 10:01:13.711810 7fcee1302940  2 journal read_entry 4985888768 : seq 26517157 731 bytes
  -266> 2017-02-12 10:01:13.711815 7fcee1302940  3 journal journal_replay: applying op seq 26517157
  -265> 2017-02-12 10:01:13.712002 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517157
  -264> 2017-02-12 10:01:13.712015 7fcee1302940  2 journal read_entry 4985892864 : seq 26517158 733 bytes
  -263> 2017-02-12 10:01:13.712020 7fcee1302940  3 journal journal_replay: applying op seq 26517158
  -262> 2017-02-12 10:01:13.712208 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517158
  -261> 2017-02-12 10:01:13.712221 7fcee1302940  2 journal read_entry 4985896960 : seq 26517159 731 bytes
  -260> 2017-02-12 10:01:13.712225 7fcee1302940  3 journal journal_replay: applying op seq 26517159
  -259> 2017-02-12 10:01:13.712403 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517159
  -258> 2017-02-12 10:01:13.712415 7fcee1302940  2 journal read_entry 4985901056 : seq 26517160 733 bytes
  -257> 2017-02-12 10:01:13.712420 7fcee1302940  3 journal journal_replay: applying op seq 26517160
  -256> 2017-02-12 10:01:13.712612 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517160
  -255> 2017-02-12 10:01:13.712625 7fcee1302940  2 journal read_entry 4985905152 : seq 26517161 731 bytes
  -254> 2017-02-12 10:01:13.712629 7fcee1302940  3 journal journal_replay: applying op seq 26517161
  -253> 2017-02-12 10:01:13.712832 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517161
  -252> 2017-02-12 10:01:13.712845 7fcee1302940  2 journal read_entry 4985909248 : seq 26517162 733 bytes
  -251> 2017-02-12 10:01:13.712849 7fcee1302940  3 journal journal_replay: applying op seq 26517162
  -250> 2017-02-12 10:01:13.713024 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517162
  -249> 2017-02-12 10:01:13.713038 7fcee1302940  2 journal read_entry 4985913344 : seq 26517163 733 bytes
  -248> 2017-02-12 10:01:13.713044 7fcee1302940  3 journal journal_replay: applying op seq 26517163
  -247> 2017-02-12 10:01:13.713230 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517163
  -246> 2017-02-12 10:01:13.713243 7fcee1302940  2 journal read_entry 4985917440 : seq 26517164 731 bytes
  -245> 2017-02-12 10:01:13.713247 7fcee1302940  3 journal journal_replay: applying op seq 26517164
  -244> 2017-02-12 10:01:13.713467 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517164
  -243> 2017-02-12 10:01:13.729052 7fcee1302940  2 journal read_entry 4985921536 : seq 26517165 4195339 bytes
  -242> 2017-02-12 10:01:13.729064 7fcee1302940  3 journal journal_replay: applying op seq 26517165
  -241> 2017-02-12 10:01:13.730825 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517165
  -240> 2017-02-12 10:01:13.731085 7fcee1302940  2 journal read_entry 4990124032 : seq 26517166 731 bytes
  -239> 2017-02-12 10:01:13.731091 7fcee1302940  3 journal journal_replay: applying op seq 26517166
  -238> 2017-02-12 10:01:13.731298 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517166
  -237> 2017-02-12 10:01:13.731312 7fcee1302940  2 journal read_entry 4990128128 : seq 26517167 731 bytes
  -236> 2017-02-12 10:01:13.731316 7fcee1302940  3 journal journal_replay: applying op seq 26517167
  -235> 2017-02-12 10:01:13.731545 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517167
  -234> 2017-02-12 10:01:13.731576 7fcee1302940  2 journal read_entry 4990132224 : seq 26517168 731 bytes
  -233> 2017-02-12 10:01:13.731584 7fcee1302940  3 journal journal_replay: applying op seq 26517168
  -232> 2017-02-12 10:01:13.731850 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517168
  -231> 2017-02-12 10:01:13.746518 7fcee1302940  2 journal read_entry 4990136320 : seq 26517169 4060171 bytes
  -230> 2017-02-12 10:01:13.746530 7fcee1302940  3 journal journal_replay: applying op seq 26517169
  -229> 2017-02-12 10:01:13.748307 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517169
  -228> 2017-02-12 10:01:13.748638 7fcee1302940  2 journal read_entry 4994203648 : seq 26517170 731 bytes
  -227> 2017-02-12 10:01:13.748645 7fcee1302940  3 journal journal_replay: applying op seq 26517170
  -226> 2017-02-12 10:01:13.748863 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517170
  -225> 2017-02-12 10:01:13.748888 7fcee1302940  2 journal read_entry 4994207744 : seq 26517171 731 bytes
  -224> 2017-02-12 10:01:13.748891 7fcee1302940  3 journal journal_replay: applying op seq 26517171
  -223> 2017-02-12 10:01:13.749100 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517171
  -222> 2017-02-12 10:01:13.749124 7fcee1302940  2 journal read_entry 4994211840 : seq 26517172 733 bytes
  -221> 2017-02-12 10:01:13.749128 7fcee1302940  3 journal journal_replay: applying op seq 26517172
  -220> 2017-02-12 10:01:13.749333 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517172
  -219> 2017-02-12 10:01:13.749357 7fcee1302940  2 journal read_entry 4994215936 : seq 26517173 731 bytes
  -218> 2017-02-12 10:01:13.749360 7fcee1302940  3 journal journal_replay: applying op seq 26517173
  -217> 2017-02-12 10:01:13.749554 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517173
  -216> 2017-02-12 10:01:13.764787 7fcee1302940  2 journal read_entry 4994220032 : seq 26517174 4195341 bytes
  -215> 2017-02-12 10:01:13.764800 7fcee1302940  3 journal journal_replay: applying op seq 26517174
  -214> 2017-02-12 10:01:13.766667 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517174
  -213> 2017-02-12 10:01:13.766926 7fcee1302940  2 journal read_entry 4998422528 : seq 26517175 733 bytes
  -212> 2017-02-12 10:01:13.766933 7fcee1302940  3 journal journal_replay: applying op seq 26517175
  -211> 2017-02-12 10:01:13.767135 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517175
  -210> 2017-02-12 10:01:13.767149 7fcee1302940  2 journal read_entry 4998426624 : seq 26517176 731 bytes
  -209> 2017-02-12 10:01:13.767153 7fcee1302940  3 journal journal_replay: applying op seq 26517176
  -208> 2017-02-12 10:01:13.767355 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517176
  -207> 2017-02-12 10:01:13.782511 7fcee1302940  2 journal read_entry 4998430720 : seq 26517177 4195339 bytes
  -206> 2017-02-12 10:01:13.782525 7fcee1302940  3 journal journal_replay: applying op seq 26517177
  -205> 2017-02-12 10:01:13.784307 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517177
  -204> 2017-02-12 10:01:13.784578 7fcee1302940  2 journal read_entry 5002633216 : seq 26517178 731 bytes
  -203> 2017-02-12 10:01:13.784585 7fcee1302940  3 journal journal_replay: applying op seq 26517178
  -202> 2017-02-12 10:01:13.784783 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517178
  -201> 2017-02-12 10:01:13.784797 7fcee1302940  2 journal read_entry 5002637312 : seq 26517179 731 bytes
  -200> 2017-02-12 10:01:13.784802 7fcee1302940  3 journal journal_replay: applying op seq 26517179
  -199> 2017-02-12 10:01:13.785000 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517179
  -198> 2017-02-12 10:01:13.785014 7fcee1302940  2 journal read_entry 5002641408 : seq 26517180 729 bytes
  -197> 2017-02-12 10:01:13.785018 7fcee1302940  3 journal journal_replay: applying op seq 26517180
  -196> 2017-02-12 10:01:13.785214 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517180
  -195> 2017-02-12 10:01:13.785227 7fcee1302940  2 journal read_entry 5002645504 : seq 26517181 733 bytes
  -194> 2017-02-12 10:01:13.785232 7fcee1302940  3 journal journal_replay: applying op seq 26517181
  -193> 2017-02-12 10:01:13.785426 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517181
  -192> 2017-02-12 10:01:13.785439 7fcee1302940  2 journal read_entry 5002649600 : seq 26517182 733 bytes
  -191> 2017-02-12 10:01:13.785443 7fcee1302940  3 journal journal_replay: applying op seq 26517182
  -190> 2017-02-12 10:01:13.785618 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517182
  -189> 2017-02-12 10:01:13.800747 7fcee1302940  2 journal read_entry 5002653696 : seq 26517183 4195339 bytes
  -188> 2017-02-12 10:01:13.800761 7fcee1302940  3 journal journal_replay: applying op seq 26517183
  -187> 2017-02-12 10:01:13.802526 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517183
  -186> 2017-02-12 10:01:13.802793 7fcee1302940  2 journal read_entry 5006856192 : seq 26517184 731 bytes
  -185> 2017-02-12 10:01:13.802800 7fcee1302940  3 journal journal_replay: applying op seq 26517184
  -184> 2017-02-12 10:01:13.803011 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517184
  -183> 2017-02-12 10:01:13.803025 7fcee1302940  2 journal read_entry 5006860288 : seq 26517185 731 bytes
  -182> 2017-02-12 10:01:13.803029 7fcee1302940  3 journal journal_replay: applying op seq 26517185
  -181> 2017-02-12 10:01:13.803210 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517185
  -180> 2017-02-12 10:01:13.803223 7fcee1302940  2 journal read_entry 5006864384 : seq 26517186 731 bytes
  -179> 2017-02-12 10:01:13.803228 7fcee1302940  3 journal journal_replay: applying op seq 26517186
  -178> 2017-02-12 10:01:13.803431 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517186
  -177> 2017-02-12 10:01:13.818637 7fcee1302940  2 journal read_entry 5006868480 : seq 26517187 4195339 bytes
  -176> 2017-02-12 10:01:13.818650 7fcee1302940  3 journal journal_replay: applying op seq 26517187
  -175> 2017-02-12 10:01:13.820425 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517187
  -174> 2017-02-12 10:01:13.820724 7fcee1302940  2 journal read_entry 5011070976 : seq 26517188 729 bytes
  -173> 2017-02-12 10:01:13.820733 7fcee1302940  3 journal journal_replay: applying op seq 26517188
  -172> 2017-02-12 10:01:13.820979 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517188
  -171> 2017-02-12 10:01:13.821009 7fcee1302940  2 journal read_entry 5011075072 : seq 26517189 733 bytes
  -170> 2017-02-12 10:01:13.821014 7fcee1302940  3 journal journal_replay: applying op seq 26517189
  -169> 2017-02-12 10:01:13.821308 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517189
  -168> 2017-02-12 10:01:13.821334 7fcee1302940  2 journal read_entry 5011079168 : seq 26517190 733 bytes
  -167> 2017-02-12 10:01:13.821338 7fcee1302940  3 journal journal_replay: applying op seq 26517190
  -166> 2017-02-12 10:01:13.821536 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517190
  -165> 2017-02-12 10:01:13.821560 7fcee1302940  2 journal read_entry 5011083264 : seq 26517191 731 bytes
  -164> 2017-02-12 10:01:13.821564 7fcee1302940  3 journal journal_replay: applying op seq 26517191
  -163> 2017-02-12 10:01:13.821765 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517191
  -162> 2017-02-12 10:01:13.837355 7fcee1302940  2 journal read_entry 5011087360 : seq 26517192 4194315 bytes
  -161> 2017-02-12 10:01:13.837368 7fcee1302940  3 journal journal_replay: applying op seq 26517192
  -160> 2017-02-12 10:01:13.839255 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517192
  -159> 2017-02-12 10:01:13.854692 7fcee1302940  2 journal read_entry 5015289856 : seq 26517193 4195339 bytes
  -158> 2017-02-12 10:01:13.854705 7fcee1302940  3 journal journal_replay: applying op seq 26517193
  -157> 2017-02-12 10:01:13.856482 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517193
  -156> 2017-02-12 10:01:13.856773 7fcee1302940  2 journal read_entry 5019492352 : seq 26517194 731 bytes
  -155> 2017-02-12 10:01:13.856779 7fcee1302940  3 journal journal_replay: applying op seq 26517194
  -154> 2017-02-12 10:01:13.857058 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517194
  -153> 2017-02-12 10:01:13.857071 7fcee1302940  2 journal read_entry 5019496448 : seq 26517195 731 bytes
  -152> 2017-02-12 10:01:13.857074 7fcee1302940  3 journal journal_replay: applying op seq 26517195
  -151> 2017-02-12 10:01:13.857281 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517195
  -150> 2017-02-12 10:01:13.857294 7fcee1302940  2 journal read_entry 5019500544 : seq 26517196 731 bytes
  -149> 2017-02-12 10:01:13.857296 7fcee1302940  3 journal journal_replay: applying op seq 26517196
  -148> 2017-02-12 10:01:13.857478 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517196
  -147> 2017-02-12 10:01:13.857490 7fcee1302940  2 journal read_entry 5019504640 : seq 26517197 731 bytes
  -146> 2017-02-12 10:01:13.857493 7fcee1302940  3 journal journal_replay: applying op seq 26517197
  -145> 2017-02-12 10:01:13.857711 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517197
  -144> 2017-02-12 10:01:13.857723 7fcee1302940  2 journal read_entry 5019508736 : seq 26517198 733 bytes
  -143> 2017-02-12 10:01:13.857726 7fcee1302940  3 journal journal_replay: applying op seq 26517198
  -142> 2017-02-12 10:01:13.857961 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517198
  -141> 2017-02-12 10:01:13.873153 7fcee1302940  2 journal read_entry 5019512832 : seq 26517199 4195341 bytes
  -140> 2017-02-12 10:01:13.873166 7fcee1302940  3 journal journal_replay: applying op seq 26517199
  -139> 2017-02-12 10:01:13.874968 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517199
  -138> 2017-02-12 10:01:13.875230 7fcee1302940  2 journal read_entry 5023715328 : seq 26517200 733 bytes
  -137> 2017-02-12 10:01:13.875237 7fcee1302940  3 journal journal_replay: applying op seq 26517200
  -136> 2017-02-12 10:01:13.875434 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517200
  -135> 2017-02-12 10:01:13.875447 7fcee1302940  2 journal read_entry 5023719424 : seq 26517201 731 bytes
  -134> 2017-02-12 10:01:13.875450 7fcee1302940  3 journal journal_replay: applying op seq 26517201
  -133> 2017-02-12 10:01:13.875691 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517201
  -132> 2017-02-12 10:01:13.875704 7fcee1302940  2 journal read_entry 5023723520 : seq 26517202 731 bytes
  -131> 2017-02-12 10:01:13.875707 7fcee1302940  3 journal journal_replay: applying op seq 26517202
  -130> 2017-02-12 10:01:13.875888 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517202
  -129> 2017-02-12 10:01:13.875901 7fcee1302940  2 journal read_entry 5023727616 : seq 26517203 733 bytes
  -128> 2017-02-12 10:01:13.875903 7fcee1302940  3 journal journal_replay: applying op seq 26517203
  -127> 2017-02-12 10:01:13.876092 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517203
  -126> 2017-02-12 10:01:13.876105 7fcee1302940  2 journal read_entry 5023731712 : seq 26517204 733 bytes
  -125> 2017-02-12 10:01:13.876107 7fcee1302940  3 journal journal_replay: applying op seq 26517204
  -124> 2017-02-12 10:01:13.876294 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517204
  -123> 2017-02-12 10:01:13.876306 7fcee1302940  2 journal read_entry 5023735808 : seq 26517205 733 bytes
  -122> 2017-02-12 10:01:13.876309 7fcee1302940  3 journal journal_replay: applying op seq 26517205
  -121> 2017-02-12 10:01:13.876504 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517205
  -120> 2017-02-12 10:01:13.876517 7fcee1302940  2 journal read_entry 5023739904 : seq 26517206 911 bytes
  -119> 2017-02-12 10:01:13.876521 7fcee1302940  3 journal journal_replay: applying op seq 26517206
  -118> 2017-02-12 10:01:13.876705 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517206
  -117> 2017-02-12 10:01:13.876717 7fcee1302940  2 journal read_entry 5023744000 : seq 26517207 733 bytes
  -116> 2017-02-12 10:01:13.876721 7fcee1302940  3 journal journal_replay: applying op seq 26517207
  -115> 2017-02-12 10:01:13.876894 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517207
  -114> 2017-02-12 10:01:13.876906 7fcee1302940  2 journal read_entry 5023748096 : seq 26517208 731 bytes
  -113> 2017-02-12 10:01:13.876910 7fcee1302940  3 journal journal_replay: applying op seq 26517208
  -112> 2017-02-12 10:01:13.877084 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517208
  -111> 2017-02-12 10:01:13.877096 7fcee1302940  2 journal read_entry 5023752192 : seq 26517209 733 bytes
  -110> 2017-02-12 10:01:13.877100 7fcee1302940  3 journal journal_replay: applying op seq 26517209
  -109> 2017-02-12 10:01:13.877290 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517209
  -108> 2017-02-12 10:01:13.877302 7fcee1302940  2 journal read_entry 5023756288 : seq 26517210 733 bytes
  -107> 2017-02-12 10:01:13.877306 7fcee1302940  3 journal journal_replay: applying op seq 26517210
  -106> 2017-02-12 10:01:13.877487 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517210
  -105> 2017-02-12 10:01:13.877499 7fcee1302940  2 journal read_entry 5023760384 : seq 26517211 731 bytes
  -104> 2017-02-12 10:01:13.877503 7fcee1302940  3 journal journal_replay: applying op seq 26517211
  -103> 2017-02-12 10:01:13.877699 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517211
  -102> 2017-02-12 10:01:13.877717 7fcee1302940  2 journal read_entry 5023764480 : seq 26517212 731 bytes
  -101> 2017-02-12 10:01:13.877724 7fcee1302940  3 journal journal_replay: applying op seq 26517212
  -100> 2017-02-12 10:01:13.877997 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517212
   -99> 2017-02-12 10:01:13.878015 7fcee1302940  2 journal read_entry 5023768576 : seq 26517213 733 bytes
   -98> 2017-02-12 10:01:13.878020 7fcee1302940  3 journal journal_replay: applying op seq 26517213
   -97> 2017-02-12 10:01:13.878267 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517213
   -96> 2017-02-12 10:01:13.878281 7fcee1302940  2 journal read_entry 5023772672 : seq 26517214 731 bytes
   -95> 2017-02-12 10:01:13.878285 7fcee1302940  3 journal journal_replay: applying op seq 26517214
   -94> 2017-02-12 10:01:13.878535 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517214
   -93> 2017-02-12 10:01:13.878548 7fcee1302940  2 journal read_entry 5023776768 : seq 26517215 731 bytes
   -92> 2017-02-12 10:01:13.878552 7fcee1302940  3 journal journal_replay: applying op seq 26517215
   -91> 2017-02-12 10:01:13.878729 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517215
   -90> 2017-02-12 10:01:13.878741 7fcee1302940  2 journal read_entry 5023780864 : seq 26517216 731 bytes
   -89> 2017-02-12 10:01:13.878745 7fcee1302940  3 journal journal_replay: applying op seq 26517216
   -88> 2017-02-12 10:01:13.878931 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517216
   -87> 2017-02-12 10:01:13.890247 7fcee1302940  2 journal read_entry 5023784960 : seq 26517217 3154955 bytes
   -86> 2017-02-12 10:01:13.890258 7fcee1302940  3 journal journal_replay: applying op seq 26517217
   -85> 2017-02-12 10:01:13.891685 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517217
   -84> 2017-02-12 10:01:13.891880 7fcee1302940  2 journal read_entry 5026947072 : seq 26517218 733 bytes
   -83> 2017-02-12 10:01:13.891887 7fcee1302940  3 journal journal_replay: applying op seq 26517218
   -82> 2017-02-12 10:01:13.892087 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517218
   -81> 2017-02-12 10:01:13.892129 7fcee1302940  2 journal read_entry 5026951168 : seq 26517219 731 bytes
   -80> 2017-02-12 10:01:13.892135 7fcee1302940  3 journal journal_replay: applying op seq 26517219
   -79> 2017-02-12 10:01:13.892341 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517219
   -78> 2017-02-12 10:01:13.901314 7fcee1302940  2 journal read_entry 5026955264 : seq 26517220 2507711 bytes
   -77> 2017-02-12 10:01:13.901325 7fcee1302940  3 journal journal_replay: applying op seq 26517220
   -76> 2017-02-12 10:01:13.902459 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517220
   -75> 2017-02-12 10:01:13.902605 7fcee1302940  2 journal read_entry 5029470208 : seq 26517221 731 bytes
   -74> 2017-02-12 10:01:13.902611 7fcee1302940  3 journal journal_replay: applying op seq 26517221
   -73> 2017-02-12 10:01:13.902818 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517221
   -72> 2017-02-12 10:01:13.902832 7fcee1302940  2 journal read_entry 5029474304 : seq 26517222 733 bytes
   -71> 2017-02-12 10:01:13.902837 7fcee1302940  3 journal journal_replay: applying op seq 26517222
   -70> 2017-02-12 10:01:13.903055 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517222
   -69> 2017-02-12 10:01:13.903074 7fcee1302940  2 journal read_entry 5029478400 : seq 26517223 731 bytes
   -68> 2017-02-12 10:01:13.903081 7fcee1302940  3 journal journal_replay: applying op seq 26517223
   -67> 2017-02-12 10:01:13.903289 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517223
   -66> 2017-02-12 10:01:13.903302 7fcee1302940  2 journal read_entry 5029482496 : seq 26517224 733 bytes
   -65> 2017-02-12 10:01:13.903309 7fcee1302940  3 journal journal_replay: applying op seq 26517224
   -64> 2017-02-12 10:01:13.903537 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517224
   -63> 2017-02-12 10:01:13.903550 7fcee1302940  2 journal read_entry 5029486592 : seq 26517225 729 bytes
   -62> 2017-02-12 10:01:13.903554 7fcee1302940  3 journal journal_replay: applying op seq 26517225
   -61> 2017-02-12 10:01:13.903785 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517225
   -60> 2017-02-12 10:01:13.903799 7fcee1302940  2 journal read_entry 5029490688 : seq 26517226 733 bytes
   -59> 2017-02-12 10:01:13.903803 7fcee1302940  3 journal journal_replay: applying op seq 26517226
   -58> 2017-02-12 10:01:13.904060 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517226
   -57> 2017-02-12 10:01:13.919172 7fcee1302940  2 journal read_entry 5029494784 : seq 26517227 4195339 bytes
   -56> 2017-02-12 10:01:13.919184 7fcee1302940  3 journal journal_replay: applying op seq 26517227
   -55> 2017-02-12 10:01:13.920945 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517227
   -54> 2017-02-12 10:01:13.921206 7fcee1302940  2 journal read_entry 5033697280 : seq 26517228 729 bytes
   -53> 2017-02-12 10:01:13.921213 7fcee1302940  3 journal journal_replay: applying op seq 26517228
   -52> 2017-02-12 10:01:13.921432 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517228
   -51> 2017-02-12 10:01:13.921447 7fcee1302940  2 journal read_entry 5033701376 : seq 26517229 729 bytes
   -50> 2017-02-12 10:01:13.921455 7fcee1302940  3 journal journal_replay: applying op seq 26517229
   -49> 2017-02-12 10:01:13.921646 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517229
   -48> 2017-02-12 10:01:13.921659 7fcee1302940  2 journal read_entry 5033705472 : seq 26517230 733 bytes
   -47> 2017-02-12 10:01:13.921664 7fcee1302940  3 journal journal_replay: applying op seq 26517230
   -46> 2017-02-12 10:01:13.921863 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517230
   -45> 2017-02-12 10:01:13.921876 7fcee1302940  2 journal read_entry 5033709568 : seq 26517231 731 bytes
   -44> 2017-02-12 10:01:13.921880 7fcee1302940  3 journal journal_replay: applying op seq 26517231
   -43> 2017-02-12 10:01:13.922065 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517231
   -42> 2017-02-12 10:01:13.922078 7fcee1302940  2 journal read_entry 5033713664 : seq 26517232 731 bytes
   -41> 2017-02-12 10:01:13.922083 7fcee1302940  3 journal journal_replay: applying op seq 26517232
   -40> 2017-02-12 10:01:13.922285 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517232
   -39> 2017-02-12 10:01:13.937443 7fcee1302940  2 journal read_entry 5033717760 : seq 26517233 4195337 bytes
   -38> 2017-02-12 10:01:13.937456 7fcee1302940  3 journal journal_replay: applying op seq 26517233
   -37> 2017-02-12 10:01:13.939225 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517233
   -36> 2017-02-12 10:01:13.939504 7fcee1302940  2 journal read_entry 5037920256 : seq 26517234 729 bytes
   -35> 2017-02-12 10:01:13.939511 7fcee1302940  3 journal journal_replay: applying op seq 26517234
   -34> 2017-02-12 10:01:13.939739 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517234
   -33> 2017-02-12 10:01:13.939756 7fcee1302940  2 journal read_entry 5037924352 : seq 26517235 733 bytes
   -32> 2017-02-12 10:01:13.939761 7fcee1302940  3 journal journal_replay: applying op seq 26517235
   -31> 2017-02-12 10:01:13.939963 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517235
   -30> 2017-02-12 10:01:13.955088 7fcee1302940  2 journal read_entry 5037928448 : seq 26517236 4195339 bytes
   -29> 2017-02-12 10:01:13.955101 7fcee1302940  3 journal journal_replay: applying op seq 26517236
   -28> 2017-02-12 10:01:13.956919 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517236
   -27> 2017-02-12 10:01:13.957210 7fcee1302940  2 journal read_entry 5042130944 : seq 26517237 733 bytes
   -26> 2017-02-12 10:01:13.957218 7fcee1302940  3 journal journal_replay: applying op seq 26517237
   -25> 2017-02-12 10:01:13.957432 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517237
   -24> 2017-02-12 10:01:13.972537 7fcee1302940  2 journal read_entry 5042135040 : seq 26517238 4195339 bytes
   -23> 2017-02-12 10:01:13.972549 7fcee1302940  3 journal journal_replay: applying op seq 26517238
   -22> 2017-02-12 10:01:13.974242 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517238
   -21> 2017-02-12 10:01:13.974506 7fcee1302940  2 journal read_entry 5046337536 : seq 26517239 733 bytes
   -20> 2017-02-12 10:01:13.974513 7fcee1302940  3 journal journal_replay: applying op seq 26517239
   -19> 2017-02-12 10:01:13.974705 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517239
   -18> 2017-02-12 10:01:13.974718 7fcee1302940  2 journal read_entry 5046341632 : seq 26517240 731 bytes
   -17> 2017-02-12 10:01:13.974722 7fcee1302940  3 journal journal_replay: applying op seq 26517240
   -16> 2017-02-12 10:01:13.974914 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517240
   -15> 2017-02-12 10:01:13.974975 7fcee1302940  2 journal read_entry 5046345728 : seq 26517241 33577 bytes
   -14> 2017-02-12 10:01:13.974982 7fcee1302940  3 journal journal_replay: applying op seq 26517241
   -13> 2017-02-12 10:01:13.975216 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517241
   -12> 2017-02-12 10:01:13.975229 7fcee1302940  2 journal read_entry 5046382592 : seq 26517242 731 bytes
   -11> 2017-02-12 10:01:13.975234 7fcee1302940  3 journal journal_replay: applying op seq 26517242
   -10> 2017-02-12 10:01:13.975417 7fcee1302940  3 journal journal_replay: r = 0, op_seq now 26517242
    -9> 2017-02-12 10:01:13.975442 7fcee1302940  2 journal No further valid entries found, journal is most likely valid
    -8> 2017-02-12 10:01:13.975446 7fcee1302940  3 journal journal_replay: end of journal, done.
    -7> 2017-02-12 10:01:13.975447 7fcee1302940  3 journal journal_replay: total = 297
    -6> 2017-02-12 10:01:13.996938 7fcee1302940  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 12: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
    -5> 2017-02-12 10:01:13.997235 7fcee1302940 -1 filestore(/var/lib/ceph/osd/ceph-3/) ignoging invalid collection '29.45_head.moved'
    -4> 2017-02-12 10:01:13.997245 7fcee1302940 -1 filestore(/var/lib/ceph/osd/ceph-3/) ignoging invalid collection '29.45_TEMP.moved'
    -3> 2017-02-12 10:01:13.997585 7fcee1302940  1 filestore(/var/lib/ceph/osd/ceph-3/) upgrade
    -2> 2017-02-12 10:01:13.998066 7fcee1302940 -1 filestore(/var/lib/ceph/osd/ceph-3/) ignoging invalid collection '29.45_head.moved'
    -1> 2017-02-12 10:01:13.998075 7fcee1302940 -1 filestore(/var/lib/ceph/osd/ceph-3/) ignoging invalid collection '29.45_TEMP.moved'
     0> 2017-02-12 10:01:15.966439 7fcee1302940 -1 osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const hobject_t&, const std::set<snapid_t>&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7fcee1302940 time 2017-02-12 10:01:15.964239
osd/SnapMapper.cc: 228: FAILED assert(r == -2)

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x55e9aa2c38db]
 2: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 3: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 6: (main()+0x2dce) [0x55e9a9bccc3e]
 7: (__libc_start_main()+0xf0) [0x7fceddf01830]
 8: (()+0x363777) [0x55e9a9c18777]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file 
--- end dump of recent events ---
*** Caught signal (Aborted) **
 in thread 7fcee1302940 thread_name:ceph-objectstor
 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x926492) [0x55e9aa1db492]
 2: (()+0x11390) [0x7fcedfb78390]
 3: (gsignal()+0x38) [0x7fceddf16428]
 4: (abort()+0x16a) [0x7fceddf1802a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x265) [0x55e9aa2c3ab5]
 6: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 7: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 10: (main()+0x2dce) [0x55e9a9bccc3e]
 11: (__libc_start_main()+0xf0) [0x7fceddf01830]
 12: (()+0x363777) [0x55e9a9c18777]
2017-02-12 10:01:15.971968 7fcee1302940 -1 *** Caught signal (Aborted) **
 in thread 7fcee1302940 thread_name:ceph-objectstor

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x926492) [0x55e9aa1db492]
 2: (()+0x11390) [0x7fcedfb78390]
 3: (gsignal()+0x38) [0x7fceddf16428]
 4: (abort()+0x16a) [0x7fceddf1802a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x265) [0x55e9aa2c3ab5]
 6: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 7: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 10: (main()+0x2dce) [0x55e9a9bccc3e]
 11: (__libc_start_main()+0xf0) [0x7fceddf01830]
 12: (()+0x363777) [0x55e9a9c18777]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2017-02-12 10:01:15.971968 7fcee1302940 -1 *** Caught signal (Aborted) **
 in thread 7fcee1302940 thread_name:ceph-objectstor

 ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
 1: (()+0x926492) [0x55e9aa1db492]
 2: (()+0x11390) [0x7fcedfb78390]
 3: (gsignal()+0x38) [0x7fceddf16428]
 4: (abort()+0x16a) [0x7fceddf1802a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x265) [0x55e9aa2c3ab5]
 6: (SnapMapper::add_oid(hobject_t const&, std::set<snapid_t, std::less<snapid_t>, std::allocator<snapid_t> > const&, MapCacher::Transaction<std::string, ceph::buffer::list>*)+0x530) [0x55e9a9d925e0]
 7: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, SnapMapper&)+0x62a) [0x55e9a9c273aa]
 8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0xacd) [0x55e9a9c27fed]
 9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, std::string, ObjectStore::Sequencer&)+0x13e3) [0x55e9a9c299c3]
 10: (main()+0x2dce) [0x55e9a9bccc3e]
 11: (__libc_start_main()+0xf0) [0x7fceddf01830]
 12: (()+0x363777) [0x55e9a9c18777]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file 
--- end dump of recent events ---
Aborted
Actions
Copy link
 #8

Updated by Yuri Weinstein about 7 years ago

  • Status changed from New to Pending Backport
Actions
Copy link
 #9

Updated by Loïc Dachary about 7 years ago

  • Copied to Backport #18951: jewel: segfault in ceph-osd --flush-journal added
Actions
Copy link
 #10

Updated by Loïc Dachary about 7 years ago

  • Copied to Backport #18952: kraken: segfault in ceph-osd --flush-journal added
Actions
Copy link
 #11

Updated by Nathan Cutler about 7 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF