Project

General

Profile

Actions

Bug #10681

closed

osd crashed (in many jobs) in upgrade:firefly-x-giant-distro-basic-multi run

Added by Yuri Weinstein about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Urgent
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/
Jobs: 722253 (many)
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/

/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103297: ceph version 0.87-81-ga463b92 (a463b92e475cd1f4cdb963e402033ebc9d37dbdc)
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103371- 1: ceph-osd() [0xa97bca]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103397- 2: (()+0x10340) [0x7f5f356b5340]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103431- 3: (gsignal()+0x39) [0x7f5f33b54f89]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103469- 4: (abort()+0x148) [0x7f5f33b58398]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103506- 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5f344606b5]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103576- 6: (()+0x5e836) [0x7f5f3445e836]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103610- 7: (()+0x5e863) [0x7f5f3445e863]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103644- 8: (()+0x5eaa2) [0x7f5f3445eaa2]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103678- 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xb82a58]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103770- 10: (std::tr1::_Sp_counted_base_impl<ObjectContext*, SharedPtrRegistry<hobject_t, ObjectContext>::OnRemoval, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x207) [0x8ad6f7]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727103940- 11: (std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x39) [0x6c2e39]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104032- 12: (ReplicatedPG::OpContext::~OpContext()+0xf9) [0x8a4609]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104093- 13: (intrusive_ptr_release(ReplicatedPG::RepGather*)+0x2b) [0x82296b]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104164- 14: (C_OSD_RepopCommit::~C_OSD_RepopCommit()+0x25) [0x8a5285]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104227- 15: (ReplicatedBackend::sub_op_modify_reply(std::tr1::shared_ptr<OpRequest>)+0x23c) [0x9f4e1c]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104323- 16: (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x1e6) [0x9f5426]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104414- 17: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x23b) [0x81f44b]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104520- 18: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x686b35]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104642- 19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x37b) [0x6870cb]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104734- 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x821) [0xb72971]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104816- 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb74a80]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104885- 22: (()+0x8182) [0x7f5f356ad182]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104919- 23: (clone()+0x6d) [0x7f5f33c1938d]
/a/teuthology-2015-01-25_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/722253/remote/burnupi15/log/ceph-osd.1.log.gz:1727104956- NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open1 closed)

Has duplicate Ceph - Bug #10659: "Caught signal (Aborted)" in upgrade:dumpling-firefly-x-giant-distro-basic-multi runDuplicate01/28/2015

Actions
Actions #1

Updated by Yuri Weinstein about 9 years ago

  • Subject changed from osd crashed in upgrade:firefly-x-giant-distro-basic-multi run to osd crashed (in many jobs)in upgrade:firefly-x-giant-distro-basic-multi run
Actions #2

Updated by Yuri Weinstein about 9 years ago

  • Subject changed from osd crashed (in many jobs)in upgrade:firefly-x-giant-distro-basic-multi run to osd crashed (in many jobs) in upgrade:firefly-x-giant-distro-basic-multi run
Actions #3

Updated by Samuel Just about 9 years ago

  • Priority changed from High to Urgent
Actions #4

Updated by Samuel Just about 9 years ago

8d29a4a231666830914903b95599d80da7b97def was recently backported to firefly. It includes a test for the fixed problem, but the problem is not fixed in giant. When the test runs against giant, boom. Quick fix would be to backport to giant as well.

Actions #5

Updated by Samuel Just about 9 years ago

  • Status changed from New to Resolved

backported to giant

Actions #6

Updated by Yuri Weinstein about 9 years ago

  • Status changed from Resolved to 7
  • Assignee set to Yuri Weinstein
Actions #7

Updated by Yuri Weinstein about 9 years ago

  • Status changed from 7 to Resolved
Actions

Also available in: Atom PDF