Actions
Bug #8439
closedceph-osd crashing often
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We have most of our OSDs crashing at least twice a day since a couple of days.
Here's the logs of such a crash.
Versions of the binaries:
[ebuild R ] sys-fs/xfsprogs-3.1.10 USE="nls readline -libedit -static -static-libs" [ebuild NS ] sys-kernel/gentoo-sources-3.12.13 [3.11.10] USE="-build -deblob -experimental -symlink" [ebuild R ] sys-cluster/ceph-0.78 USE="xfs -btrfs -debug -fuse -gtk -libatomic -radosgw -static-libs -tcmalloc"
I've also attached the binary as suggested in the dump.
Thanks for your help.
-116> 2014-05-26 09:50:14.837087 7fc8e8ff9700 5 -- op tracker -- , seq: 1222096, time: 2014-05-26 09:50:14.837087, event: done, request: osd_sub_op(client.143277.0:177751 3.56 90520cd6/rbd_data.343571f3e82af.0000000000000116/head//3 [] v 7899'10109 snapset=0=[]:[] snapc=0=[]) v9
-115> 2014-05-26 09:50:14.837101 7fc8d6ff5700 5 -- op tracker -- , seq: 1222255, time: 2014-05-26 09:50:14.837089, event: commit_queued_for_journal_write, request: osd_sub_op(client.142962.0:319787 3.1a fc9b109a/rbd_data.285be4fef98b3.0000000000000159/head//3 [] v 7899'133030 snapset=0=[]:[] snapc=0=[]) v9
-114> 2014-05-26 09:50:14.837174 7fc8d6ff5700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.837151, event: reached_pg, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-113> 2014-05-26 09:50:14.837195 7fc8d6ff5700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.837195, event: started, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-112> 2014-05-26 09:50:14.837380 7fc8d6ff5700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.837380, event: started, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-111> 2014-05-26 09:50:14.837396 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 21381983 > 104857600
-110> 2014-05-26 09:50:14.839621 7fc8eaffd700 5 -- op tracker -- , seq: 1222147, time: 2014-05-26 09:50:14.839621, event: journaled_completion_queued, request: osd_sub_op(client.139840.0:287955 3.57 d0ed54d7/rbd_data.221dd64207aa8.0000000000000041/head//3 [] v 7899'9601 snapset=0=[]:[] snapc=0=[]) v9
-109> 2014-05-26 09:50:14.839677 7fc8e3fff700 5 -- op tracker -- , seq: 1222147, time: 2014-05-26 09:50:14.839676, event: commit_sent, request: osd_sub_op(client.139840.0:287955 3.57 d0ed54d7/rbd_data.221dd64207aa8.0000000000000041/head//3 [] v 7899'9601 snapset=0=[]:[] snapc=0=[]) v9
-108> 2014-05-26 09:50:14.839687 7fc8eb7fe700 5 -- op tracker -- , seq: 1222244, time: 2014-05-26 09:50:14.839687, event: write_thread_in_journal_buffer, request: osd_sub_op(client.142962.0:319785 3.72 f8867c72/rbd_data.285be4fef98b3.0000000000000160/head//3 [] v 7899'10189 snapset=0=[]:[] snapc=0=[]) v9
-107> 2014-05-26 09:50:14.839695 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.6 172.16.128.39:6809/27164 -- osd_sub_op_reply(client.139840.0:287955 3.57 d0ed54d7/rbd_data.221dd64207aa8.0000000000000041/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc71002cca0
-106> 2014-05-26 09:50:14.839717 7fc8eb7fe700 5 -- op tracker -- , seq: 1222164, time: 2014-05-26 09:50:14.839717, event: write_thread_in_journal_buffer, request: osd_sub_op(client.160234.0:191783 3.26 35e29b26/rbd_data.2712433b32b6b.0000000000000125/head//3 [] v 7899'88753 snapset=0=[]:[] snapc=0=[]) v9
-105> 2014-05-26 09:50:14.839746 7fc8eb7fe700 5 -- op tracker -- , seq: 1222205, time: 2014-05-26 09:50:14.839746, event: write_thread_in_journal_buffer, request: osd_sub_op(client.143149.0:140973 3.6e 8ab7446e/rbd_data.3992837b908ce.00000000000000f0/head//3 [] v 7899'92146 snapset=0=[]:[] snapc=0=[]) v9
-104> 2014-05-26 09:50:14.839774 7fc8eb7fe700 5 -- op tracker -- , seq: 1222204, time: 2014-05-26 09:50:14.839774, event: write_thread_in_journal_buffer, request: osd_sub_op(client.143149.0:140970 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15366 snapset=0=[]:[] snapc=0=[]) v9
-103> 2014-05-26 09:50:14.839794 7fc8eb7fe700 5 -- op tracker -- , seq: 1222255, time: 2014-05-26 09:50:14.839794, event: write_thread_in_journal_buffer, request: osd_sub_op(client.142962.0:319787 3.1a fc9b109a/rbd_data.285be4fef98b3.0000000000000159/head//3 [] v 7899'133030 snapset=0=[]:[] snapc=0=[]) v9
-102> 2014-05-26 09:50:14.841459 7fc8eaffd700 5 -- op tracker -- , seq: 1222244, time: 2014-05-26 09:50:14.841459, event: journaled_completion_queued, request: osd_sub_op(client.142962.0:319785 3.72 f8867c72/rbd_data.285be4fef98b3.0000000000000160/head//3 [] v 7899'10189 snapset=0=[]:[] snapc=0=[]) v9
-101> 2014-05-26 09:50:14.841484 7fc8eaffd700 5 -- op tracker -- , seq: 1222164, time: 2014-05-26 09:50:14.841484, event: journaled_completion_queued, request: osd_sub_op(client.160234.0:191783 3.26 35e29b26/rbd_data.2712433b32b6b.0000000000000125/head//3 [] v 7899'88753 snapset=0=[]:[] snapc=0=[]) v9
-100> 2014-05-26 09:50:14.841536 7fc8e3fff700 5 -- op tracker -- , seq: 1222244, time: 2014-05-26 09:50:14.841507, event: commit_sent, request: osd_sub_op(client.142962.0:319785 3.72 f8867c72/rbd_data.285be4fef98b3.0000000000000160/head//3 [] v 7899'10189 snapset=0=[]:[] snapc=0=[]) v9
-99> 2014-05-26 09:50:14.841557 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.9 172.16.128.33:6810/1031764 -- osd_sub_op_reply(client.142962.0:319785 3.72 f8867c72/rbd_data.285be4fef98b3.0000000000000160/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710055770
-98> 2014-05-26 09:50:14.841556 7fc8eaffd700 5 -- op tracker -- , seq: 1222205, time: 2014-05-26 09:50:14.841544, event: journaled_completion_queued, request: osd_sub_op(client.143149.0:140973 3.6e 8ab7446e/rbd_data.3992837b908ce.00000000000000f0/head//3 [] v 7899'92146 snapset=0=[]:[] snapc=0=[]) v9
-97> 2014-05-26 09:50:14.841577 7fc8e3fff700 5 -- op tracker -- , seq: 1222164, time: 2014-05-26 09:50:14.841577, event: commit_sent, request: osd_sub_op(client.160234.0:191783 3.26 35e29b26/rbd_data.2712433b32b6b.0000000000000125/head//3 [] v 7899'88753 snapset=0=[]:[] snapc=0=[]) v9
-96> 2014-05-26 09:50:14.841590 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.15 172.16.128.35:6811/10008850 -- osd_sub_op_reply(client.160234.0:191783 3.26 35e29b26/rbd_data.2712433b32b6b.0000000000000125/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710019d40
-95> 2014-05-26 09:50:14.841590 7fc8eaffd700 5 -- op tracker -- , seq: 1222204, time: 2014-05-26 09:50:14.841578, event: journaled_completion_queued, request: osd_sub_op(client.143149.0:140970 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15366 snapset=0=[]:[] snapc=0=[]) v9
-94> 2014-05-26 09:50:14.841607 7fc8e3fff700 5 -- op tracker -- , seq: 1222205, time: 2014-05-26 09:50:14.841607, event: commit_sent, request: osd_sub_op(client.143149.0:140973 3.6e 8ab7446e/rbd_data.3992837b908ce.00000000000000f0/head//3 [] v 7899'92146 snapset=0=[]:[] snapc=0=[]) v9
-93> 2014-05-26 09:50:14.841621 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.20 172.16.128.41:6801/18217 -- osd_sub_op_reply(client.143149.0:140973 3.6e 8ab7446e/rbd_data.3992837b908ce.00000000000000f0/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710028ef0
-92> 2014-05-26 09:50:14.841620 7fc8eaffd700 5 -- op tracker -- , seq: 1222255, time: 2014-05-26 09:50:14.841612, event: journaled_completion_queued, request: osd_sub_op(client.142962.0:319787 3.1a fc9b109a/rbd_data.285be4fef98b3.0000000000000159/head//3 [] v 7899'133030 snapset=0=[]:[] snapc=0=[]) v9
-91> 2014-05-26 09:50:14.841639 7fc8e3fff700 5 -- op tracker -- , seq: 1222204, time: 2014-05-26 09:50:14.841639, event: commit_sent, request: osd_sub_op(client.143149.0:140970 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15366 snapset=0=[]:[] snapc=0=[]) v9
-90> 2014-05-26 09:50:14.841656 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.21 172.16.128.41:6806/18691 -- osd_sub_op_reply(client.143149.0:140970 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc7100372e0
-89> 2014-05-26 09:50:14.841683 7fc8e3fff700 5 -- op tracker -- , seq: 1222255, time: 2014-05-26 09:50:14.841683, event: commit_sent, request: osd_sub_op(client.142962.0:319787 3.1a fc9b109a/rbd_data.285be4fef98b3.0000000000000159/head//3 [] v 7899'133030 snapset=0=[]:[] snapc=0=[]) v9
-88> 2014-05-26 09:50:14.841703 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.1 172.16.128.39:6814/27216 -- osd_sub_op_reply(client.142962.0:319787 3.1a fc9b109a/rbd_data.285be4fef98b3.0000000000000159/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710005df0
-87> 2014-05-26 09:50:14.848624 7fc8e8ff9700 1 -- 172.16.128.36:6801/22825 --> 172.16.128.41:6801/18217 -- MOSDPGPushReply(3.67 7899 [PushReplyOp(a9997867/rbd_data.27d36b8b4567.0000000000018081/head//3),PushReplyOp(5f30a867/rbd_data.21efb4624c7dd.000000000000093f/head//3)]) v2 -- ?+0 0x7fc66d2f6590 con 0x7fc874000b30
-86> 2014-05-26 09:50:14.848925 7fc8d6ff5700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.848924, event: commit_queued_for_journal_write, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-85> 2014-05-26 09:50:14.849014 7fc8d6ff5700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.849014, event: reached_pg, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-84> 2014-05-26 09:50:14.849034 7fc8d6ff5700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.849034, event: started, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-83> 2014-05-26 09:50:14.849082 7fc8eb7fe700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.849081, event: write_thread_in_journal_buffer, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-82> 2014-05-26 09:50:14.849213 7fc8d6ff5700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.849213, event: started, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-81> 2014-05-26 09:50:14.849233 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 8794929 > 104857600
-80> 2014-05-26 09:50:14.849289 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-79> 2014-05-26 09:50:14.849645 7fc8e8ff9700 5 -- op tracker -- , seq: 1222108, time: 2014-05-26 09:50:14.849645, event: sub_op_applied, request: osd_sub_op(client.142962.0:319782 3.a b7c3488a/rbd_data.285be4fef98b3.0000000000000129/head//3 [] v 7899'159068 snapset=0=[]:[] snapc=0=[]) v9
-78> 2014-05-26 09:50:14.849678 7fc8d6ff5700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.849663, event: commit_queued_for_journal_write, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-77> 2014-05-26 09:50:14.849715 7fc8e8ff9700 5 -- op tracker -- , seq: 1222108, time: 2014-05-26 09:50:14.849701, event: done, request: osd_sub_op(client.142962.0:319782 3.a b7c3488a/rbd_data.285be4fef98b3.0000000000000129/head//3 [] v 7899'159068 snapset=0=[]:[] snapc=0=[]) v9
-76> 2014-05-26 09:50:14.849766 7fc8d6ff5700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.849736, event: reached_pg, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-75> 2014-05-26 09:50:14.849790 7fc8d6ff5700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.849790, event: started, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-74> 2014-05-26 09:50:14.849921 7fc8d6ff5700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.849921, event: started, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-73> 2014-05-26 09:50:14.849940 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 8782473 > 104857600
-72> 2014-05-26 09:50:14.851851 7fc8eaffd700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.851850, event: journaled_completion_queued, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-71> 2014-05-26 09:50:14.851907 7fc8e3fff700 5 -- op tracker -- , seq: 1222218, time: 2014-05-26 09:50:14.851907, event: commit_sent, request: osd_sub_op(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18490 snapset=0=[]:[] snapc=0=[]) v9
-70> 2014-05-26 09:50:14.851929 7fc8eb7fe700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.851929, event: write_thread_in_journal_buffer, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-69> 2014-05-26 09:50:14.851945 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.2 172.16.128.39:6801/19743 -- osd_sub_op_reply(client.157357.0:243216 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710069900
-68> 2014-05-26 09:50:14.852456 7fc8e8ff9700 1 -- 172.16.128.36:6801/22825 --> 172.16.128.41:6801/18217 -- MOSDPGPushReply(3.67 7899 [PushReplyOp(c47ba867/rbd_data.204046b8b4567.0000000000002d20/head//3)]) v2 -- ?+0 0x7fc8de9cab20 con 0x7fc874000b30
-67> 2014-05-26 09:50:14.852519 7fc8d6ff5700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.852518, event: commit_queued_for_journal_write, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-66> 2014-05-26 09:50:14.852589 7fc8d6ff5700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.852588, event: reached_pg, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-65> 2014-05-26 09:50:14.852615 7fc8d6ff5700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.852614, event: started, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-64> 2014-05-26 09:50:14.852796 7fc8d6ff5700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.852795, event: started, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-63> 2014-05-26 09:50:14.852818 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 4603959 > 104857600
-62> 2014-05-26 09:50:14.853395 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-61> 2014-05-26 09:50:14.853662 7fc8d6ff5700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.853662, event: commit_queued_for_journal_write, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-60> 2014-05-26 09:50:14.853684 7fc8e8ff9700 5 -- op tracker -- , seq: 1222105, time: 2014-05-26 09:50:14.853671, event: sub_op_applied, request: osd_sub_op(client.230360.0:680229 3.51 f4760ad1/rbd_data.38e543be927fb.0000000000000110/head//3 [] v 7899'12340 snapset=0=[]:[] snapc=0=[]) v9
-59> 2014-05-26 09:50:14.853709 7fc8d6ff5700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.853709, event: reached_pg, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-58> 2014-05-26 09:50:14.853726 7fc8d6ff5700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.853726, event: started, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-57> 2014-05-26 09:50:14.853739 7fc8e8ff9700 5 -- op tracker -- , seq: 1222105, time: 2014-05-26 09:50:14.853730, event: done, request: osd_sub_op(client.230360.0:680229 3.51 f4760ad1/rbd_data.38e543be927fb.0000000000000110/head//3 [] v 7899'12340 snapset=0=[]:[] snapc=0=[]) v9
-56> 2014-05-26 09:50:14.853857 7fc8d6ff5700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.853857, event: started, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-55> 2014-05-26 09:50:14.853873 7fc8e8ff9700 5 -- op tracker -- , seq: 1222103, time: 2014-05-26 09:50:14.853861, event: sub_op_applied, request: osd_sub_op(client.143277.0:177747 3.67 c110a067/rbd_data.343571f3e82af.0000000000000001/head//3 [] v 7899'117915 snapset=0=[]:[] snapc=0=[]) v9
-54> 2014-05-26 09:50:14.853893 7fc8d6ff5700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.853893, event: commit_queued_for_journal_write, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-53> 2014-05-26 09:50:14.853908 7fc8e8ff9700 5 -- op tracker -- , seq: 1222103, time: 2014-05-26 09:50:14.853908, event: done, request: osd_sub_op(client.143277.0:177747 3.67 c110a067/rbd_data.343571f3e82af.0000000000000001/head//3 [] v 7899'117915 snapset=0=[]:[] snapc=0=[]) v9
-52> 2014-05-26 09:50:14.853943 7fc8d6ff5700 5 -- op tracker -- , seq: 1222230, time: 2014-05-26 09:50:14.853943, event: reached_pg, request: osd_sub_op(client.143241.0:143379 3.67 43caff67/rbd_data.32d432913a7a3.00000000000000e0/head//3 [] v 7899'117918 snapset=0=[]:[] snapc=0=[]) v9
-51> 2014-05-26 09:50:14.853958 7fc8d6ff5700 5 -- op tracker -- , seq: 1222230, time: 2014-05-26 09:50:14.853958, event: started, request: osd_sub_op(client.143241.0:143379 3.67 43caff67/rbd_data.32d432913a7a3.00000000000000e0/head//3 [] v 7899'117918 snapset=0=[]:[] snapc=0=[]) v9
-50> 2014-05-26 09:50:14.854031 7fc8eaffd700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.854031, event: journaled_completion_queued, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-49> 2014-05-26 09:50:14.854048 7fc8e3fff700 5 -- op tracker -- , seq: 1222224, time: 2014-05-26 09:50:14.854048, event: commit_sent, request: osd_sub_op(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] v 7899'198539 snapset=0=[]:[] snapc=0=[]) v9
-48> 2014-05-26 09:50:14.854070 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.5 172.16.128.31:6801/637 -- osd_sub_op_reply(client.142998.0:106879 3.36 4782a936/rbd_data.289153346db8b.00000000000000d2/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc7100134c0
-47> 2014-05-26 09:50:14.854086 7fc8eb7fe700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.854085, event: write_thread_in_journal_buffer, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-46> 2014-05-26 09:50:14.854142 7fc8eb7fe700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.854142, event: write_thread_in_journal_buffer, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-45> 2014-05-26 09:50:14.854176 7fc8d6ff5700 5 -- op tracker -- , seq: 1222230, time: 2014-05-26 09:50:14.854144, event: started, request: osd_sub_op(client.143241.0:143379 3.67 43caff67/rbd_data.32d432913a7a3.00000000000000e0/head//3 [] v 7899'117918 snapset=0=[]:[] snapc=0=[]) v9
-44> 2014-05-26 09:50:14.854202 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 4598161 > 104857600
-43> 2014-05-26 09:50:14.854199 7fc8eb7fe700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.854184, event: write_thread_in_journal_buffer, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-42> 2014-05-26 09:50:14.854431 7fc8e97fa700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-41> 2014-05-26 09:50:14.854696 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-40> 2014-05-26 09:50:14.854734 7fc8e8ff9700 5 -- op tracker -- , seq: 1222130, time: 2014-05-26 09:50:14.854734, event: sub_op_applied, request: osd_sub_op(client.229847.0:770798 3.5f 73c279df/rbd_data.38b33698779de.0000000000000157/head//3 [] v 7899'19413 snapset=0=[]:[] snapc=0=[]) v9
-39> 2014-05-26 09:50:14.854776 7fc8e8ff9700 5 -- op tracker -- , seq: 1222130, time: 2014-05-26 09:50:14.854776, event: done, request: osd_sub_op(client.229847.0:770798 3.5f 73c279df/rbd_data.38b33698779de.0000000000000157/head//3 [] v 7899'19413 snapset=0=[]:[] snapc=0=[]) v9
-38> 2014-05-26 09:50:14.854787 7fc8d6ff5700 5 -- op tracker -- , seq: 1222230, time: 2014-05-26 09:50:14.854787, event: commit_queued_for_journal_write, request: osd_sub_op(client.143241.0:143379 3.67 43caff67/rbd_data.32d432913a7a3.00000000000000e0/head//3 [] v 7899'117918 snapset=0=[]:[] snapc=0=[]) v9
-37> 2014-05-26 09:50:14.854846 7fc8d6ff5700 5 -- op tracker -- , seq: 1222206, time: 2014-05-26 09:50:14.854846, event: reached_pg, request: osd_sub_op(client.143149.0:140971 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15367 snapset=0=[]:[] snapc=0=[]) v9
-36> 2014-05-26 09:50:14.854862 7fc8d6ff5700 5 -- op tracker -- , seq: 1222206, time: 2014-05-26 09:50:14.854862, event: started, request: osd_sub_op(client.143149.0:140971 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15367 snapset=0=[]:[] snapc=0=[]) v9
-35> 2014-05-26 09:50:14.854986 7fc8d6ff5700 5 -- op tracker -- , seq: 1222206, time: 2014-05-26 09:50:14.854986, event: started, request: osd_sub_op(client.143149.0:140971 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15367 snapset=0=[]:[] snapc=0=[]) v9
-34> 2014-05-26 09:50:14.855015 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 4597961 > 104857600
-33> 2014-05-26 09:50:14.855134 7fc8e8ff9700 5 -- op tracker -- , seq: 1222199, time: 2014-05-26 09:50:14.855134, event: sub_op_applied, request: osd_sub_op(client.239894.0:80430 3.37 e758fc37/rbd_data.3a8c857389a05.000000000000021c/head//3 [] v 7899'21698 snapset=0=[]:[] snapc=0=[]) v9
-32> 2014-05-26 09:50:14.855150 7fc8d6ff5700 5 -- op tracker -- , seq: 1222206, time: 2014-05-26 09:50:14.855150, event: commit_queued_for_journal_write, request: osd_sub_op(client.143149.0:140971 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15367 snapset=0=[]:[] snapc=0=[]) v9
-31> 2014-05-26 09:50:14.855181 7fc8e8ff9700 5 -- op tracker -- , seq: 1222199, time: 2014-05-26 09:50:14.855175, event: done, request: osd_sub_op(client.239894.0:80430 3.37 e758fc37/rbd_data.3a8c857389a05.000000000000021c/head//3 [] v 7899'21698 snapset=0=[]:[] snapc=0=[]) v9
-30> 2014-05-26 09:50:14.855224 7fc8d6ff5700 5 -- op tracker -- , seq: 1222219, time: 2014-05-26 09:50:14.855201, event: reached_pg, request: osd_sub_op(client.157357.0:243217 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18491 snapset=0=[]:[] snapc=0=[]) v9
-29> 2014-05-26 09:50:14.855247 7fc8d6ff5700 5 -- op tracker -- , seq: 1222219, time: 2014-05-26 09:50:14.855247, event: started, request: osd_sub_op(client.157357.0:243217 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18491 snapset=0=[]:[] snapc=0=[]) v9
-28> 2014-05-26 09:50:14.855307 7fc8e97fa700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-27> 2014-05-26 09:50:14.855415 7fc8d6ff5700 5 -- op tracker -- , seq: 1222219, time: 2014-05-26 09:50:14.855415, event: started, request: osd_sub_op(client.157357.0:243217 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18491 snapset=0=[]:[] snapc=0=[]) v9
-26> 2014-05-26 09:50:14.855433 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 4599976 > 104857600
-25> 2014-05-26 09:50:14.855690 7fc8e8ff9700 5 -- op tracker -- , seq: 1222111, time: 2014-05-26 09:50:14.855690, event: sub_op_applied, request: osd_sub_op(client.206906.0:148849 3.56 a677b2d6/rbd_data.32b4b6a5e026e.0000000000000120/head//3 [] v 7899'10110 snapset=0=[]:[] snapc=0=[]) v9
-24> 2014-05-26 09:50:14.855721 7fc8d6ff5700 5 -- op tracker -- , seq: 1222219, time: 2014-05-26 09:50:14.855695, event: commit_queued_for_journal_write, request: osd_sub_op(client.157357.0:243217 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18491 snapset=0=[]:[] snapc=0=[]) v9
-23> 2014-05-26 09:50:14.855780 7fc8d6ff5700 5 -- op tracker -- , seq: 1222225, time: 2014-05-26 09:50:14.855780, event: reached_pg, request: osd_sub_op(client.5199.0:70552 3.62 91319462/rbd_data.2026f36be830c.00000000000000a0/head//3 [] v 7899'140681 snapset=0=[]:[] snapc=0=[]) v9
-22> 2014-05-26 09:50:14.855798 7fc8d6ff5700 5 -- op tracker -- , seq: 1222225, time: 2014-05-26 09:50:14.855798, event: started, request: osd_sub_op(client.5199.0:70552 3.62 91319462/rbd_data.2026f36be830c.00000000000000a0/head//3 [] v 7899'140681 snapset=0=[]:[] snapc=0=[]) v9
-21> 2014-05-26 09:50:14.855821 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
-20> 2014-05-26 09:50:14.855868 7fc8eaffd700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.855868, event: journaled_completion_queued, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-19> 2014-05-26 09:50:14.855889 7fc8e3fff700 5 -- op tracker -- , seq: 1222153, time: 2014-05-26 09:50:14.855889, event: commit_sent, request: osd_sub_op(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] v 7899'179827 snapset=0=[]:[] snapc=0=[]) v9
-18> 2014-05-26 09:50:14.855910 7fc8eaffd700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.855894, event: journaled_completion_queued, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-17> 2014-05-26 09:50:14.855931 7fc8eaffd700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.855930, event: journaled_completion_queued, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-16> 2014-05-26 09:50:14.855927 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.6 172.16.128.39:6809/27164 -- osd_sub_op_reply(client.153793.0:488591 3.9 fcf66089/rbd_data.2992641f122bd.0000000000000427/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc71001a470
-15> 2014-05-26 09:50:14.855978 7fc8e3fff700 5 -- op tracker -- , seq: 1222256, time: 2014-05-26 09:50:14.855978, event: commit_sent, request: osd_sub_op(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] v 7899'108013 snapset=0=[]:[] snapc=0=[]) v9
-14> 2014-05-26 09:50:14.855997 7fc8d6ff5700 5 -- op tracker -- , seq: 1222225, time: 2014-05-26 09:50:14.855987, event: started, request: osd_sub_op(client.5199.0:70552 3.62 91319462/rbd_data.2026f36be830c.00000000000000a0/head//3 [] v 7899'140681 snapset=0=[]:[] snapc=0=[]) v9
-13> 2014-05-26 09:50:14.856016 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.9 172.16.128.33:6810/1031764 -- osd_sub_op_reply(client.142944.0:209800 3.16 5f849196/rbd_data.270827459abfd.0000000000000042/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710011960
-12> 2014-05-26 09:50:14.856032 7fc8eb7fe700 5 -- op tracker -- , seq: 1222230, time: 2014-05-26 09:50:14.856012, event: write_thread_in_journal_buffer, request: osd_sub_op(client.143241.0:143379 3.67 43caff67/rbd_data.32d432913a7a3.00000000000000e0/head//3 [] v 7899'117918 snapset=0=[]:[] snapc=0=[]) v9
-11> 2014-05-26 09:50:14.856070 7fc8e3fff700 5 -- op tracker -- , seq: 1222165, time: 2014-05-26 09:50:14.856035, event: commit_sent, request: osd_sub_op(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] v 7899'88754 snapset=0=[]:[] snapc=0=[]) v9
-10> 2014-05-26 09:50:14.856079 7fc8d6ff5700 2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 4601517 > 104857600
-9> 2014-05-26 09:50:14.856102 7fc8e3fff700 1 -- 172.16.128.36:6801/22825 --> osd.15 172.16.128.35:6811/10008850 -- osd_sub_op_reply(client.207743.0:153575 3.26 17e00226/rbd_data.32f4d3083f808.0000000000000125/head//3 [] ondisk, result = 0) v2 -- ?+0 0x7fc710067b80
-8> 2014-05-26 09:50:14.856104 7fc8eb7fe700 5 -- op tracker -- , seq: 1222206, time: 2014-05-26 09:50:14.856086, event: write_thread_in_journal_buffer, request: osd_sub_op(client.143149.0:140971 3.6c a595f96c/rbd_data.3992837b908ce.0000000000000001/head//3 [] v 7899'15367 snapset=0=[]:[] snapc=0=[]) v9
-7> 2014-05-26 09:50:14.856154 7fc8eb7fe700 5 -- op tracker -- , seq: 1222219, time: 2014-05-26 09:50:14.856154, event: write_thread_in_journal_buffer, request: osd_sub_op(client.157357.0:243217 3.3c 637643c/rbd_data.2bbf42620642.0000000000000200/head//3 [] v 7899'18491 snapset=0=[]:[] snapc=0=[]) v9
-6> 2014-05-26 09:50:14.856249 7fc8e8ff9700 5 -- op tracker -- , seq: 1222111, time: 2014-05-26 09:50:14.856249, event: done, request: osd_sub_op(client.206906.0:148849 3.56 a677b2d6/rbd_data.32b4b6a5e026e.0000000000000120/head//3 [] v 7899'10110 snapset=0=[]:[] snapc=0=[]) v9
-5> 2014-05-26 09:50:14.856272 7fc8d6ff5700 5 -- op tracker -- , seq: 1222225, time: 2014-05-26 09:50:14.856272, event: commit_queued_for_journal_write, request: osd_sub_op(client.5199.0:70552 3.62 91319462/rbd_data.2026f36be830c.00000000000000a0/head//3 [] v 7899'140681 snapset=0=[]:[] snapc=0=[]) v9
-4> 2014-05-26 09:50:14.856315 7fc8d6ff5700 5 -- op tracker -- , seq: 1222172, time: 2014-05-26 09:50:14.856315, event: reached_pg, request: osd_sub_op(client.143259.0:154379 3.8 34f9b08/rbd_data.3310f751e7e75.000000000000001b/head//3 [] v 7899'10567 snapset=0=[]:[] snapc=0=[]) v9
-3> 2014-05-26 09:50:14.856330 7fc8d6ff5700 5 -- op tracker -- , seq: 1222172, time: 2014-05-26 09:50:14.856330, event: started, request: osd_sub_op(client.143259.0:154379 3.8 34f9b08/rbd_data.3310f751e7e75.000000000000001b/head//3 [] v 7899'10567 snapset=0=[]:[] snapc=0=[]) v9
-2> 2014-05-26 09:50:14.856342 7fc8e8ff9700 5 -- op tracker -- , seq: 1222106, time: 2014-05-26 09:50:14.856333, event: sub_op_applied, request: osd_sub_op(client.207743.0:153572 3.43 8728b643/rbd_data.32f4d3083f808.0000000000000124/head//3 [] v 7899'140172 snapset=0=[]:[] snapc=0=[]) v9
-1> 2014-05-26 09:50:14.856399 7fc8e8ff9700 5 -- op tracker -- , seq: 1222106, time: 2014-05-26 09:50:14.856399, event: done, request: osd_sub_op(client.207743.0:153572 3.43 8728b643/rbd_data.32f4d3083f808.0000000000000124/head//3 [] v 7899'140172 snapset=0=[]:[] snapc=0=[]) v9
0> 2014-05-26 09:50:14.856385 7fc8d77f6700 -1 *** Caught signal (Aborted) **
in thread 7fc8d77f6700
ceph version 0.78 (f6c746c314d7b87b8419b6e584c94bfe4511dbd4)
1: /usr/bin/ceph-osd() [0xa884dd]
2: (()+0x10b10) [0x7fc900ea7b10]
3: (gsignal()+0x35) [0x7fc8ff329535]
4: (abort()+0x148) [0x7fc8ff32a9b8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc8ffc1596d]
6: (()+0x63a66) [0x7fc8ffc13a66]
7: (()+0x63a93) [0x7fc8ffc13a93]
8: (()+0x63cbe) [0x7fc8ffc13cbe]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x42a) [0xb5d13a]
10: (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0x12f) [0x8e8aef]
11: /usr/bin/ceph-osd() [0x918de9]
12: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list2<boost::statechart::custom_reaction<PG::FlushedEvt>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &boost::statechart::detail::no_context<boost::statechart::event_base>::no_function> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0xbf) [0x943b5f]
13: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list4<boost::statechart::custom_reaction<PG::AdvMap>, boost::statechart::custom_reaction<PG::NullEvt>, boost::statechart::custom_reaction<PG::FlushedEvt>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &boost::statechart::detail::no_context<boost::statechart::event_base>::no_function> >, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x57) [0x943ce7]
14: (boost::statechart::detail::reaction_result boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<PG::QueryState>, boost::statechart::custom_reaction<PG::AdvMap>, boost::statechart::custom_reaction<PG::NullEvt>, boost::statechart::custom_reaction<PG::FlushedEvt>, boost::statechart::transition<boost::statechart::event_base, PG::RecoveryState::Crashed, boost::statechart::detail::no_context<boost::statechart::event_base>, &boost::statechart::detail::no_context<boost::statechart::event_base>::no_function>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0> >(boost::statechart::simple_state<PG::RecoveryState::Started, PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Start, (boost::statechart::history_mode)0>&, boost::statechart::event_base const&, void const*)+0x57) [0x943d97]
15: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base const&)+0x5b) [0x926c1b]
16: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x19) [0x926ca9]
17: (PG::RecoveryState::handle_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x31) [0x926d41]
18: (PG::handle_peering_event(std::tr1::shared_ptr<PG::CephPeeringEvt>, PG::RecoveryCtx*)+0x338) [0x8dedd8]
19: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x400) [0x761110]
20: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x14) [0x7be7c4]
21: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68c) [0xb4ee1c]
22: (ThreadPool::WorkThread::entry()+0x10) [0xb50070]
23: (()+0x8f3a) [0x7fc900e9ff3a]
24: (clone()+0x6d) [0x7fc8ff3dedad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2014-05-26 09:50:14.856801 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.857217 7fc8e97fa700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.859191 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.860063 7fc8e97fa700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.860219 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.861188 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-26 09:50:14.861881 7fc8e9ffb700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) set_extsize: FSSETXATTR: (22) Invalid argument
Files
Actions