Actions
Bug #3699
closedosds crashed in ReplicatedPG::sub_op_modify on a mixed node cluster
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
cluster: burnupi06 [running osd.1 on v0.55.1] , burnupi07[running osd.3, osd.4, mon.b on argonaut], burnupi08[running osd.5,osd.6,mon.c,mds.a on argonaut]
steps to reproduce:
1. All 3 nodes were on running on argonaut and had IO pumped in to the cluster from client.
2. when IO is in progress, upgraded osd.1 on burnupi06 to v0.55.1 and restarted osd.1
3. Moved osd.1 out of the cluster with the command "ceph osd out 1", which is when all other osds running on argonaut crashed
2012-12-29 14:05:37.654117 7fb605632700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::sub_op_modify(OpRequestRef)' thread 7fb605632700 time 2012-12-29 14:05:37.571094 osd/ReplicatedPG.cc: 4192: FAILED assert(is_active()) ceph version 0.48.2argonaut-61-g9483a03 (commit:9483a032f750572586f146c696ec6501d3df0383) 1: (ReplicatedPG::sub_op_modify(std::tr1::shared_ptr<OpRequest>)+0xbbd) [0x54369d] 2: (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0xff) [0x55711f] 3: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x9f) [0x60b89f] 4: (OSD::dequeue_op(PG*)+0x238) [0x5cab78] 5: (ThreadPool::worker()+0x4c4) [0x7aa554] 6: (ThreadPool::WorkThread::entry()+0xd) [0x5e383d] 7: (()+0x7e9a) [0x7fb6164ace9a] 8: (clone()+0x6d) [0x7fb614a5ecbd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ubuntu@burnupi07:/var/log/ceph$ sudo cat /etc/ceph/ceph.conf [global] auth client required = none auth cluster required = none auth service required = none [osd] osd journal size = 1000 filestore xattr use omap = true [osd.1] host = burnupi06 [osd.3] host = burnupi07 [osd.4] host = burnupi07 [osd.5] host = burnupi08 [osd.6] host = burnupi08 [mon.b] host = burnupi07 mon addr = 10.214.134.38:6789 [mon.c] host = burnupi08 mon addr = 10.214.134.36:6789 [mds.a] host = burnupi08 ubuntu@burnupi06:/var/log/ceph$ sudo cat /etc/ceph/ceph.conf [global] auth client required = none auth cluster required = none auth service required = none [osd] osd journal size = 1000 filestore xattr use omap = true [osd.1] osd min pg log entries = 10 host = burnupi06 [osd.3] host = burnupi07 [osd.4] host = burnupi07 [osd.5] host = burnupi08 [osd.6] host = burnupi08 [mon.b] host = burnupi07 mon addr = 10.214.134.38:6789 [mon.c] host = burnupi08 mon addr = 10.214.134.36:6789 [mds.a] host = burnupi08
leaving the cluster as it is for reference.
Actions