Project

General

Profile

Bug #4654

FileJournal replay crash -- specific to btrfs/parallel journaling

Added by Samuel Just almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-04-03 22:36:14.242946 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) queue_flusher ep 0 fd 36 0~4194304 qlen 17
2013-04-03 22:36:14.242967 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) write 0.c_head/2753cd9c/benchmark_data_mira084_26241_object409/head//0 0~4194304 = 4194304
2013-04-03 22:36:14.243083 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.c_head/2753cd9c/benchmark_data_mira084_26241_object409/head//0
2013-04-03 22:36:14.243162 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.c_head/2753cd9c/benchmark_data_mira084_26241_object409/head//0 = -2
2013-04-03 22:36:14.243244 7f43c8e3a780 20 filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 37 getting '_'
2013-04-03 22:36:14.243272 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.c_head/2753cd9c/benchmark_data_mira084_26241_object409/head//0
2013-04-03 22:36:14.243327 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.c_head/2753cd9c/benchmark_data_mira084_26241_object409/head//0 = -2
2013-04-03 22:36:14.243337 7f43c8e3a780 10 journal op_apply_finish 881 open_ops 1 -> 0, max_applied_seq 880 -> 881
2013-04-03 22:36:14.243342 7f43c8e3a780 3 journal journal_replay: r = 0, op_seq now 881
2013-04-03 22:36:14.274774 7f43c8e3a780 2 journal read_entry 43819008 : seq 882 4195884 bytes
2013-04-03 22:36:14.274806 7f43c8e3a780 3 journal journal_replay: applying op seq 882
2013-04-03 22:36:14.274815 7f43c8e3a780 10 journal op_apply_start 882 open_ops 0 -> 1
2013-04-03 22:36:14.274821 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) do_transaction on 0x17da3c0
2013-04-03 22:36:14.274835 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) _omap_setkeys meta/103b255f/pglog_0.a/0//-1
2013-04-03 22:36:14.275026 7f43c8e3a780 10 filestore hoid: 103b255f/pglog_0.a/0//-1 not skipping op, *spos 882.0.0
2013-04-03 22:36:14.275042 7f43c8e3a780 10 filestore > header.spos 0.0.0
2013-04-03 22:36:14.275161 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) _omap_setkeys meta/16ef7597/infos/head//-1
2013-04-03 22:36:14.275303 7f43c8e3a780 10 filestore hoid: 16ef7597/infos/head//-1 not skipping op, *spos 882.0.1
2013-04-03 22:36:14.275316 7f43c8e3a780 10 filestore > header.spos 0.0.0
2013-04-03 22:36:14.275355 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) _do_transaction on 0x17d9df0
2013-04-03 22:36:14.275368 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) write 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0 0~4194304
2013-04-03 22:36:14.278544 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) queue_flusher ep 0 fd 37 0~4194304 qlen 18
2013-04-03 22:36:14.278563 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) write 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0 0~4194304 = 4194304
2013-04-03 22:36:14.278674 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0
2013-04-03 22:36:14.278752 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0 = -2
2013-04-03 22:36:14.278834 7f43c8e3a780 20 filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 38 getting '
'
2013-04-03 22:36:14.278862 7f43c8e3a780 15 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0
2013-04-03 22:36:14.278916 7f43c8e3a780 10 filestore(/var/lib/ceph/osd/ceph-4) setattrs 0.a_head/ec2b33da/benchmark_data_mira084_26241_object413/head//0 = -2
2013-04-03 22:36:14.278927 7f43c8e3a780 10 journal op_apply_finish 882 open_ops 1 -> 0, max_applied_seq 881 -> 882
2013-04-03 22:36:14.278931 7f43c8e3a780 3 journal journal_replay: r = 0, op_seq now 882
2013-04-03 22:36:14.455407 7f43c8e3a780 2 journal read_entry 48021504 : seq 885 4195884 bytes
2013-04-03 22:36:14.456591 7f43c8e3a780 -1 os/JournalingObjectStore.cc: In function 'int JournalingObjectStore::journal_replay(uint64_t)' thread 7f43c8e3a780 time 2013-04-03 22:36:14.455451
os/JournalingObjectStore.cc: 72: FAILED assert(op_seq == seq-1)

ceph version 0.60-406-g4abf0f6 (4abf0f61265eda0f59ccd58279d6c27496568ee7)
1: (JournalingObjectStore::journal_replay(unsigned long)+0x1688) [0x75f508]
2: (FileStore::mount()+0x38f4) [0x742764]
3: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x60a70a]
4: (OSD::convertfs(std::string const&, std::string const&)+0x47) [0x60b177]
5: (main()+0x20c3) [0x57e3a3]
6: (__libc_start_main()+0xed) [0x7f43c6b4176d]
7: ceph-osd() [0x580bcd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-432> 2013-04-03 22:36:12.974501 7f43c8e3a780 5 asok(0x177f1c0) register_command perfcounters_dump hook 0x17805c0
-431> 2013-04-03 22:36:12.974560 7f43c8e3a780 5 asok(0x177f1c0) register_command 1 hook 0x17805c0
-430> 2013-04-03 22:36:12.974575 7f43c8e3a780 5 asok(0x177f1c0) register_command perf dump hook 0x17805c0

History

#1 Updated by Samuel Just almost 11 years ago

ubuntu@mira084:~/ceph$ less ceph-osd.4.log

full logs with journal and filestore logging.

#2 Updated by Samuel Just almost 11 years ago

  • Status changed from 12 to Resolved

Also available in: Atom PDF