Project

General

Profile

Actions
Copy link

Bug #3699

closed

osds crashed in ReplicatedPG::sub_op_modify on a mixed node cluster

Added by Tamilarasi muthamizhan over 11 years ago. Updated over 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Samuel Just
Category:
OSD
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

cluster: burnupi06 [running osd.1 on v0.55.1] , burnupi07[running osd.3, osd.4, mon.b on argonaut], burnupi08[running osd.5,osd.6,mon.c,mds.a on argonaut]

steps to reproduce:
1. All 3 nodes were on running on argonaut and had IO pumped in to the cluster from client.
2. when IO is in progress, upgraded osd.1 on burnupi06 to v0.55.1 and restarted osd.1
3. Moved osd.1 out of the cluster with the command "ceph osd out 1", which is when all other osds running on argonaut crashed

2012-12-29 14:05:37.654117 7fb605632700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::sub_op_modify(OpRequestRef)' thread 7fb605632700 time 2012-12-29 14:05:37.571094
osd/ReplicatedPG.cc: 4192: FAILED assert(is_active())

 ceph version 0.48.2argonaut-61-g9483a03 (commit:9483a032f750572586f146c696ec6501d3df0383)
 1: (ReplicatedPG::sub_op_modify(std::tr1::shared_ptr<OpRequest>)+0xbbd) [0x54369d]
 2: (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0xff) [0x55711f]
 3: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x9f) [0x60b89f]
 4: (OSD::dequeue_op(PG*)+0x238) [0x5cab78]
 5: (ThreadPool::worker()+0x4c4) [0x7aa554]
 6: (ThreadPool::WorkThread::entry()+0xd) [0x5e383d]
 7: (()+0x7e9a) [0x7fb6164ace9a]
 8: (clone()+0x6d) [0x7fb614a5ecbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@burnupi07:/var/log/ceph$ sudo cat /etc/ceph/ceph.conf 
[global]
    auth client required = none
    auth cluster required = none
    auth service required = none

[osd]
    osd journal size = 1000
    filestore xattr use omap = true

[osd.1]
    host = burnupi06

[osd.3]
    host = burnupi07

[osd.4]
    host = burnupi07

[osd.5]
    host = burnupi08

[osd.6]
    host = burnupi08

[mon.b]
    host = burnupi07
    mon addr = 10.214.134.38:6789

[mon.c]
    host = burnupi08
    mon addr = 10.214.134.36:6789

[mds.a]
    host = burnupi08

ubuntu@burnupi06:/var/log/ceph$ sudo cat /etc/ceph/ceph.conf
[global]
    auth client required = none
    auth cluster required = none
    auth service required = none

[osd]
    osd journal size = 1000
    filestore xattr use omap = true

[osd.1]
    osd min pg log entries = 10
    host = burnupi06

[osd.3]
    host = burnupi07

[osd.4]
    host = burnupi07

[osd.5]
    host = burnupi08

[osd.6]
    host = burnupi08

[mon.b]
    host = burnupi07
    mon addr = 10.214.134.38:6789

[mon.c]
    host = burnupi08
    mon addr = 10.214.134.36:6789

[mds.a]
    host = burnupi08

leaving the cluster as it is for reference.

Actions
Copy link

Also available in: Atom PDF