Actions
Bug #9727
closed0.86 EC+ KV OSDs crashing
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hi, testing our Tiering setup with EC+KV backend a bit further on the latest dev release, our OSDS started to crash after some hours:
ceph osd tree:
[root@ceph003 ~]# ceph osd tree # id weight type name up/down reweight -12 23.44 root default-cache -9 7.812 host ceph001-cache 1 3.906 osd.1 up 1 3 3.906 osd.3 up 1 -10 7.812 host ceph002-cache 17 3.906 osd.17 up 1 19 3.906 osd.19 up 1 -11 7.812 host ceph003-cache 33 3.906 osd.33 up 1 35 3.906 osd.35 up 1 -8 140.6 root default-ec -5 46.87 host ceph001-ec 4 3.906 osd.4 down 0 5 3.906 osd.5 down 0 6 3.906 osd.6 up 1 7 3.906 osd.7 down 0 8 3.906 osd.8 down 0 9 3.906 osd.9 down 0 10 3.906 osd.10 up 1 11 3.906 osd.11 up 1 12 3.906 osd.12 down 0 13 3.906 osd.13 down 0 14 3.906 osd.14 down 0 15 3.906 osd.15 down 0 -6 46.87 host ceph002-ec 20 3.906 osd.20 down 0 21 3.906 osd.21 up 1 22 3.906 osd.22 up 1 23 3.906 osd.23 down 0 24 3.906 osd.24 down 0 25 3.906 osd.25 down 0 26 3.906 osd.26 down 0 27 3.906 osd.27 down 0 28 3.906 osd.28 up 1 29 3.906 osd.29 down 0 30 3.906 osd.30 down 0 31 3.906 osd.31 down 0 -7 46.87 host ceph003-ec 36 3.906 osd.36 up 1 37 3.906 osd.37 up 1 38 3.906 osd.38 down 0 39 3.906 osd.39 down 0 40 3.906 osd.40 down 0 41 3.906 osd.41 up 1 42 3.906 osd.42 down 0 43 3.906 osd.43 down 0 44 3.906 osd.44 down 0 45 3.906 osd.45 up 1 46 3.906 osd.46 down 0 47 3.906 osd.47 down 0 -4 23.44 root default-ssd -1 7.812 host ceph001-ssd 0 3.906 osd.0 up 1 2 3.906 osd.2 up 1 -2 7.812 host ceph002-ssd 16 3.906 osd.16 up 1 18 3.906 osd.18 up 1 -3 7.812 host ceph003-ssd 32 3.906 osd.32 up 1 34 3.906 osd.34 up 1
The stack from the log file of a crashed OSD:
-37> 2014-10-10 16:13:28.624571 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 <== osd.6 10.143.8.180:0/44498 28286 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:28.624015) v2 ==== 47+0+0 (401773946 0 0) 0xa3ee900 con 0x58f09a0
-36> 2014-10-10 16:13:28.624595 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 --> 10.143.8.180:0/44498 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:28.624015) v2 -- ?+0 0x6e1a940 con 0x58f09a0
-35> 2014-10-10 16:13:28.660956 7f10d44c4700 1 -- 10.143.8.182:6831/1477 <== osd.21 10.141.8.181:0/39311 1186 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:28.660226) v2 ==== 47+0+0 (2490584147 0 0) 0x73a2940 con 0x67644c0
-34> 2014-10-10 16:13:28.660978 7f10d44c4700 1 -- 10.143.8.182:6831/1477 --> 10.141.8.181:0/39311 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:28.660226) v2 -- ?+0 0xc223fc0 con 0x67644c0
-33> 2014-10-10 16:13:28.660991 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 <== osd.21 10.141.8.181:0/39311 1186 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:28.660226) v2 ==== 47+0+0 (2490584147 0 0) 0x1bcaf080 con 0x1666a100
-32> 2014-10-10 16:13:28.661032 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 --> 10.141.8.181:0/39311 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:28.660226) v2 -- ?+0 0xa3ee900 con 0x1666a100
-31> 2014-10-10 16:13:28.750100 7f10b7bc9700 1 -- 10.143.8.182:6830/1477 <== osd.21 10.143.8.181:6810/39311 21434 ==== MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=)) v1 ==== 456+0+0 (259409574 0 0) 0x5b34ec0 con 0x5fb2ec0
-30> 2014-10-10 16:13:28.750119 7f10b7bc9700 5 -- op tracker -- seq: 1058580, time: 2014-10-10 16:13:28.750047, event: header_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-29> 2014-10-10 16:13:28.750131 7f10b7bc9700 5 -- op tracker -- seq: 1058580, time: 2014-10-10 16:13:28.750050, event: throttled, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-28> 2014-10-10 16:13:28.750141 7f10b7bc9700 5 -- op tracker -- seq: 1058580, time: 2014-10-10 16:13:28.750096, event: all_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-27> 2014-10-10 16:13:28.750151 7f10b7bc9700 5 -- op tracker -- seq: 1058580, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-26> 2014-10-10 16:13:28.750186 7f10cf4ba700 5 -- op tracker -- seq: 1058580, time: 2014-10-10 16:13:28.750185, event: reached_pg, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-25> 2014-10-10 16:13:28.756626 7f10cf4ba700 1 -- 10.143.8.182:6830/1477 --> 10.143.8.181:6810/39311 -- MOSDECSubOpReadReply(2.1es0 169 ECSubReadReply(tid=32298, attrs_read=0)) v1 -- ?+0 0x21dc2300 con 0x5fb2ec0
-24> 2014-10-10 16:13:28.756800 7f10cf4ba700 5 -- op tracker -- seq: 1058580, time: 2014-10-10 16:13:28.756799, event: done, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32298, to_read={5fc9c59e/1000008bcd3.00000000/head//2=0,839072,152ac59e/1000008ce45.00000000/head//2=0,839072,2b0bc59e/100000ca195.00000000/head//2=0,839072,425bc59e/100000dc199.00000000/head//2=0,839072,288bc59e/1000001df08.00000000/head//2=0,839072}, attrs_to_read=))
-23> 2014-10-10 16:13:28.855939 7f10b7bc9700 1 -- 10.143.8.182:6830/1477 <== osd.21 10.143.8.181:6810/39311 21435 ==== MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=)) v1 ==== 456+0+0 (3334140587 0 0) 0x5b306c0 con 0x5fb2ec0
-22> 2014-10-10 16:13:28.855968 7f10b7bc9700 5 -- op tracker -- seq: 1058581, time: 2014-10-10 16:13:28.855886, event: header_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-21> 2014-10-10 16:13:28.855981 7f10b7bc9700 5 -- op tracker -- seq: 1058581, time: 2014-10-10 16:13:28.855889, event: throttled, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-20> 2014-10-10 16:13:28.855991 7f10b7bc9700 5 -- op tracker -- seq: 1058581, time: 2014-10-10 16:13:28.855934, event: all_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-19> 2014-10-10 16:13:28.856001 7f10b7bc9700 5 -- op tracker -- seq: 1058581, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-18> 2014-10-10 16:13:28.856046 7f10cccb5700 5 -- op tracker -- seq: 1058581, time: 2014-10-10 16:13:28.856046, event: reached_pg, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-17> 2014-10-10 16:13:28.865364 7f10cccb5700 1 -- 10.143.8.182:6830/1477 --> 10.143.8.181:6810/39311 -- MOSDECSubOpReadReply(2.1es0 169 ECSubReadReply(tid=32299, attrs_read=0)) v1 -- ?+0 0x21dc2800 con 0x5fb2ec0
-16> 2014-10-10 16:13:28.865446 7f10cccb5700 5 -- op tracker -- seq: 1058581, time: 2014-10-10 16:13:28.865446, event: done, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32299, to_read={cf2cc59e/100000b5ec2.00000000/head//2=0,839072,c5acc59e/100000a6894.00000000/head//2=0,839072,c53dc59e/100000c62fd.00000000/head//2=0,839072,d11ec59e/1000005a942.00000000/head//2=0,839072,c88ec59e/100000e4348.00000000/head//2=0,839072}, attrs_to_read=))
-15> 2014-10-10 16:13:28.883745 7f10de654700 5 osd.47 169 tick
-14> 2014-10-10 16:13:29.387688 7f10d44c4700 1 -- 10.143.8.182:6831/1477 <== osd.8 10.141.8.180:0/50167 28373 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:29.383149) v2 ==== 47+0+0 (972166668 0 0) 0x19ecf260 con 0x644a7e0
-13> 2014-10-10 16:13:29.387722 7f10d44c4700 1 -- 10.143.8.182:6831/1477 --> 10.141.8.180:0/50167 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:29.383149) v2 -- ?+0 0x73a2940 con 0x644a7e0
-12> 2014-10-10 16:13:29.387727 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 <== osd.8 10.141.8.180:0/50167 28373 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:29.383149) v2 ==== 47+0+0 (972166668 0 0) 0x1c7005a0 con 0x58f3de0
-11> 2014-10-10 16:13:29.387782 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 --> 10.141.8.180:0/50167 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:29.383149) v2 -- ?+0 0x1bcaf080 con 0x58f3de0
-10> 2014-10-10 16:13:29.439085 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 <== osd.37 10.141.8.182:0/39363 28314 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:29.438577) v2 ==== 47+0+0 (3411033659 0 0) 0x9ac1680 con 0x6763860
-9> 2014-10-10 16:13:29.439117 7f10d5cc7700 1 -- 10.141.8.182:6846/1477 --> 10.141.8.182:0/39363 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:29.438577) v2 -- ?+0 0x1c7005a0 con 0x6763860
-8> 2014-10-10 16:13:29.439134 7f10d44c4700 1 -- 10.143.8.182:6831/1477 <== osd.37 10.141.8.182:0/39363 28314 ==== osd_ping(ping e169 stamp 2014-10-10 16:13:29.438577) v2 ==== 47+0+0 (3411033659 0 0) 0x1b0eecc0 con 0x6764ba0
-7> 2014-10-10 16:13:29.439153 7f10d44c4700 1 -- 10.143.8.182:6831/1477 --> 10.141.8.182:0/39363 -- osd_ping(ping_reply e169 stamp 2014-10-10 16:13:29.438577) v2 -- ?+0 0x19ecf260 con 0x6764ba0
-6> 2014-10-10 16:13:29.522560 7f10b7bc9700 1 -- 10.143.8.182:6830/1477 <== osd.21 10.143.8.181:6810/39311 21436 ==== MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=)) v1 ==== 456+0+0 (1404608695 0 0) 0x5b37980 con 0x5fb2ec0
-5> 2014-10-10 16:13:29.522593 7f10b7bc9700 5 -- op tracker -- seq: 1058582, time: 2014-10-10 16:13:29.522506, event: header_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=))
-4> 2014-10-10 16:13:29.522608 7f10b7bc9700 5 -- op tracker -- seq: 1058582, time: 2014-10-10 16:13:29.522508, event: throttled, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=))
-3> 2014-10-10 16:13:29.522619 7f10b7bc9700 5 -- op tracker -- seq: 1058582, time: 2014-10-10 16:13:29.522554, event: all_read, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=))
-2> 2014-10-10 16:13:29.522629 7f10b7bc9700 5 -- op tracker -- seq: 1058582, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=))
-1> 2014-10-10 16:13:29.522663 7f10cf4ba700 5 -- op tracker -- seq: 1058582, time: 2014-10-10 16:13:29.522663, event: reached_pg, op: MOSDECSubOpRead(2.1es6 169 ECSubRead(tid=32300, to_read={8d0fc59e/100000c66b4.00000000/head//2=0,839072,1fafc59e/10000087781.00000000/head//2=0,839072,bcd1d59e/1000005e23e.00000000/head//2=0,839072,c7d3d59e/10000004853.00000000/head//2=0,839072,20e3d59e/10000000447.00000791/head//2=0,839072}, attrs_to_read=))
0> 2014-10-10 16:13:29.526506 7f10cf4ba700 -1 osd/ECBackend.cc: In function 'void ECBackend::handle_sub_read(pg_shard_t, ECSubRead&, ECSubReadReply*)' thread 7f10cf4ba700 time 2014-10-10 16:13:29.524269
osd/ECBackend.cc: 876: FAILED assert(0)
ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb79235]
2: (ECBackend::handle_sub_read(pg_shard_t, ECSubRead&, ECSubReadReply*)+0x77e) [0x9ecdfe]
3: (ECBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x4b5) [0x9f59b5]
4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x26b) [0x80455b]
5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x6719f5]
6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x359) [0x671f89]
7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x828) [0xb68e88]
8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb6afb0]
9: (()+0x7df3) [0x7f10e6e1fdf3]
10: (clone()+0x6d) [0x7f10e58e63dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.47.log
--- end dump of recent events ---
2014-10-10 16:13:29.644863 7f10cf4ba700 -1 *** Caught signal (Aborted) **
in thread 7f10cf4ba700
ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82)
1: /usr/bin/ceph-osd() [0xa86f92]
2: (()+0xf130) [0x7f10e6e27130]
3: (gsignal()+0x39) [0x7f10e5825989]
4: (abort()+0x148) [0x7f10e5827098]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f10e61389d5]
6: (()+0x5e946) [0x7f10e6136946]
7: (()+0x5e973) [0x7f10e6136973]
8: (()+0x5eb9f) [0x7f10e6136b9f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xb7942a]
10: (ECBackend::handle_sub_read(pg_shard_t, ECSubRead&, ECSubReadReply*)+0x77e) [0x9ecdfe]
11: (ECBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x4b5) [0x9f59b5]
12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x26b) [0x80455b]
13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x6719f5]
14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x359) [0x671f89]
15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x828) [0xb68e88]
16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb6afb0]
17: (()+0x7df3) [0x7f10e6e1fdf3]
18: (clone()+0x6d) [0x7f10e58e63dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2014-10-10 16:13:29.644863 7f10cf4ba700 -1 *** Caught signal (Aborted) **
in thread 7f10cf4ba700
ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82)
1: /usr/bin/ceph-osd() [0xa86f92]
2: (()+0xf130) [0x7f10e6e27130]
3: (gsignal()+0x39) [0x7f10e5825989]
4: (abort()+0x148) [0x7f10e5827098]
5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f10e61389d5]
6: (()+0x5e946) [0x7f10e6136946]
7: (()+0x5e973) [0x7f10e6136973]
8: (()+0x5eb9f) [0x7f10e6136b9f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xb7942a]
10: (ECBackend::handle_sub_read(pg_shard_t, ECSubRead&, ECSubReadReply*)+0x77e) [0x9ecdfe]
11: (ECBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x4b5) [0x9f59b5]
12: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x26b) [0x80455b]
13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405) [0x6719f5]
14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x359) [0x671f89]
15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x828) [0xb68e88]
16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb6afb0]
17: (()+0x7df3) [0x7f10e6e1fdf3]
18: (clone()+0x6d) [0x7f10e58e63dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Thanks!
Actions