Bug #13811: "osd/ReplicatedPG.cc: 10525: FAILED assert(obc)" in upgrade:firefly-hammer-x-jewel-distro-basic-openstack - Ceph - Ceph

Actions

Copy link

Bug #13811

closed

"osd/ReplicatedPG.cc: 10525: FAILED assert(obc)" in upgrade:firefly-hammer-x-jewel-distro-basic-openstack

Added by Yuri Weinstein over 8 years ago. Updated over 7 years ago.

Status:

Can't reproduce

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

upgrade/firefly-hammer-x, upgrade/hammer-x, upgrade/infernalis-x

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2015-11-12_21:36:48-upgrade:firefly-hammer-x-jewel-distro-basic-openstack/
Jobs: ['14759', '14761']
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2015-11-12_21:36:48-upgrade:firefly-hammer-x-jewel-distro-basic-openstack/14759/teuthology.log

2015-11-12T23:50:46.067 INFO:tasks.ceph.osd.0.target089133.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7f604f6b2700 time 2015-11-12 23:50:45.964712
2015-11-12T23:50:46.068 INFO:tasks.ceph.osd.0.target089133.stderr:osd/ReplicatedPG.cc: 10525: FAILED assert(obc)
2015-11-12T23:50:46.070 INFO:tasks.ceph.osd.0.target089133.stderr: ceph version 0.94.5-86-ga79acd4 (a79acd41187e6b049432bdc314f192e3fbb560a3)
2015-11-12T23:50:46.070 INFO:tasks.ceph.osd.0.target089133.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc86fb]
2015-11-12T23:50:46.070 INFO:tasks.ceph.osd.0.target089133.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x73f) [0x884c4f]
2015-11-12T23:50:46.071 INFO:tasks.ceph.osd.0.target089133.stderr: 3: (ReplicatedPG::hit_set_persist()+0xdc4) [0x887384]
2015-11-12T23:50:46.076 INFO:tasks.ceph.osd.0.target089133.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe38) [0x8a84e8]
2015-11-12T23:50:46.076 INFO:tasks.ceph.osd.0.target089133.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x84558a]
2015-11-12T23:50:46.076 INFO:tasks.ceph.osd.0.target089133.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x6a1be5]
2015-11-12T23:50:46.076 INFO:tasks.ceph.osd.0.target089133.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x331) [0x6a2131]
2015-11-12T23:50:46.076 INFO:tasks.ceph.osd.0.target089133.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0xbb824f]
2015-11-12T23:50:46.077 INFO:tasks.ceph.osd.0.target089133.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xbba370]
2015-11-12T23:50:46.077 INFO:tasks.ceph.osd.0.target089133.stderr: 10: (()+0x8182) [0x7f606af63182]
2015-11-12T23:50:46.077 INFO:tasks.ceph.osd.0.target089133.stderr: 11: (clone()+0x6d) [0x7f60694ce47d]
2015-11-12T23:50:46.077 INFO:tasks.ceph.osd.0.target089133.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Samuel Just over 8 years ago

maybe http://tracker.ceph.com/issues/13185 investigating

Actions

Copy link

Updated by Yuri Weinstein over 8 years ago

Related to Bug #14399: "ReplicatedPG.cc: 10483: FAILED assert(obc)" in rados-hammer-distro-basic-mira added

Actions

Copy link

Updated by Yuri Weinstein about 8 years ago

Release set to infernalis
ceph-qa-suite upgrade/hammer-x added

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-04-07_17:02:02-upgrade:hammer-x-infernalis-distro-basic-openstack/
Job: 30582
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-04-07_17:02:02-upgrade:hammer-x-infernalis-distro-basic-openstack/30582/teuthology.log

2016-04-07T18:40:29.987 INFO:tasks.ceph.osd.5.target087190.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7f7306274700 time 2016-04-07 18:40:29.940474
2016-04-07T18:40:29.987 INFO:tasks.ceph.osd.5.target087190.stderr:osd/ReplicatedPG.cc: 10514: FAILED assert(obc)
2016-04-07T18:40:29.987 INFO:tasks.ceph.osd.5.target087190.stderr: ceph version 0.94.6-254-ge219e85 (e219e85be00088eecde7b1f29d7699493a79bc4d)
2016-04-07T18:40:29.988 INFO:tasks.ceph.osd.5.target087190.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbb1c6b]
2016-04-07T18:40:29.988 INFO:tasks.ceph.osd.5.target087190.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x73f) [0x86c22f]
2016-04-07T18:40:29.988 INFO:tasks.ceph.osd.5.target087190.stderr: 3: (ReplicatedPG::hit_set_persist()+0xee7) [0x86d237]
2016-04-07T18:40:29.989 INFO:tasks.ceph.osd.5.target087190.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe2e) [0x88e8ce]
2016-04-07T18:40:29.989 INFO:tasks.ceph.osd.5.target087190.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x82b48a]
2016-04-07T18:40:29.989 INFO:tasks.ceph.osd.5.target087190.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x697e55]
2016-04-07T18:40:29.989 INFO:tasks.ceph.osd.5.target087190.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x338) [0x6983a8]
2016-04-07T18:40:29.990 INFO:tasks.ceph.osd.5.target087190.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0xba17bf]
2016-04-07T18:40:29.990 INFO:tasks.ceph.osd.5.target087190.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xba38e0]
2016-04-07T18:40:29.990 INFO:tasks.ceph.osd.5.target087190.stderr: 10: (()+0x8182) [0x7f73209e5182]
2016-04-07T18:40:29.990 INFO:tasks.ceph.osd.5.target087190.stderr: 11: (clone()+0x6d) [0x7f731ef5047d]
2016-04-07T18:40:29.991 INFO:tasks.ceph.osd.5.target087190.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions

Copy link

Updated by Yuri Weinstein about 8 years ago

ceph-qa-suite upgrade/infernalis-x added

http://qa-proxy.ceph.com/teuthology/teuthology-2016-04-17_17:10:08-upgrade:hammer-x-infernalis-distro-basic-vps/137490/teuthology.log

2016-04-17T18:45:59.786 INFO:tasks.ceph.osd.5.vpm051.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7fbdd81e0700 time 2016-04-18 01:45:59.780936
2016-04-17T18:45:59.786 INFO:tasks.ceph.osd.5.vpm051.stderr:osd/ReplicatedPG.cc: 10514: FAILED assert(obc)
2016-04-17T18:45:59.788 INFO:tasks.ceph.osd.5.vpm051.stderr: ceph version 0.94.6-254-ge219e85 (e219e85be00088eecde7b1f29d7699493a79bc4d)
2016-04-17T18:45:59.788 INFO:tasks.ceph.osd.5.vpm051.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbb1c6b]
2016-04-17T18:45:59.789 INFO:tasks.ceph.osd.5.vpm051.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x73f) [0x86c22f]
2016-04-17T18:45:59.789 INFO:tasks.ceph.osd.5.vpm051.stderr: 3: (ReplicatedPG::hit_set_persist()+0xee7) [0x86d237]
2016-04-17T18:45:59.789 INFO:tasks.ceph.osd.5.vpm051.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe2e) [0x88e8ce]
2016-04-17T18:45:59.789 INFO:tasks.ceph.osd.5.vpm051.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x82b48a]
2016-04-17T18:45:59.789 INFO:tasks.ceph.osd.5.vpm051.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x697e55]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x338) [0x6983a8]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0xba17bf]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xba38e0]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: 10: (()+0x8182) [0x7fbdf2881182]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: 11: (clone()+0x6d) [0x7fbdf0dec47d]
2016-04-17T18:45:59.790 INFO:tasks.ceph.osd.5.vpm051.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions

Copy link

Updated by Yuri Weinstein almost 8 years ago

Run: http://pulpito.ceph.com/teuthology-2016-05-26_18:05:02-upgrade:hammer-x-infernalis-distro-basic-vps/
Job: 217744
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-05-26_18:05:02-upgrade:hammer-x-infernalis-distro-basic-vps/217744/teuthology.log

2016-05-26T19:46:59.044 INFO:tasks.ceph.osd.4.vpm165.stderr:osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7fbab8c21700 time 2016-05-27 02:46:59.041141
2016-05-26T19:46:59.044 INFO:tasks.ceph.osd.4.vpm165.stderr:osd/ReplicatedPG.cc: 10514: FAILED assert(obc)
2016-05-26T19:46:59.045 INFO:tasks.ceph.osd.4.vpm165.stderr: ceph version 0.94.7-8-ga336291 (a33629106ec64acdaba45bf9db18d2402a926b8e)
2016-05-26T19:46:59.046 INFO:tasks.ceph.osd.4.vpm165.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbb1e0b]
2016-05-26T19:46:59.046 INFO:tasks.ceph.osd.4.vpm165.stderr: 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x73f) [0x86c3cf]
2016-05-26T19:46:59.046 INFO:tasks.ceph.osd.4.vpm165.stderr: 3: (ReplicatedPG::hit_set_persist()+0xee7) [0x86d3d7]
2016-05-26T19:46:59.047 INFO:tasks.ceph.osd.4.vpm165.stderr: 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe2e) [0x88ea6e]
2016-05-26T19:46:59.047 INFO:tasks.ceph.osd.4.vpm165.stderr: 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x66a) [0x82b62a]
2016-05-26T19:46:59.047 INFO:tasks.ceph.osd.4.vpm165.stderr: 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3d5) [0x697e55]
2016-05-26T19:46:59.048 INFO:tasks.ceph.osd.4.vpm165.stderr: 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x338) [0x6983a8]
2016-05-26T19:46:59.048 INFO:tasks.ceph.osd.4.vpm165.stderr: 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0xba195f]
2016-05-26T19:46:59.048 INFO:tasks.ceph.osd.4.vpm165.stderr: 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xba3a80]
2016-05-26T19:46:59.048 INFO:tasks.ceph.osd.4.vpm165.stderr: 10: (()+0x8184) [0x7fbad5ca8184]
2016-05-26T19:46:59.049 INFO:tasks.ceph.osd.4.vpm165.stderr: 11: (clone()+0x6d) [0x7fbad421337d]
2016-05-26T19:46:59.049 INFO:tasks.ceph.osd.4.vpm165.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions

Copy link