Project

General

Profile

Bug #13185

osd/ReplicatedPG.cc: 11062: FAILED assert(obc) on hammer -> infernalis upgrade

Added by Sage Weil over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

    -4> 2015-09-19 23:19:01.650164 7f6d6f930700 15 filestore(/var/lib/ceph/osd/ceph-3) getattr 429.6_head/429/00000006:.ceph-internal/hit_set_429.6_archive_2015-09-19 23:18:49.366823Z_2015-09-19 23:18:52.551666Z/head '_'
    -3> 2015-09-19 23:19:01.650242 7f6d6f930700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/429.6_head/hit\uset\u429.6\uarchive\u2015-09-19 23:18:49.366823Z\u2015-09-19 23:18:52.551666Z__head_00000006_.ceph-internal_1ad with flags=2: (2) No such file or directory
    -2> 2015-09-19 23:19:01.650254 7f6d6f930700 10 filestore(/var/lib/ceph/osd/ceph-3) getattr 429.6_head/429/00000006:.ceph-internal/hit_set_429.6_archive_2015-09-19 23:18:49.366823Z_2015-09-19 23:18:52.551666Z/head '_' = -2
    -1> 2015-09-19 23:19:01.650259 7f6d6f930700 10 osd.3 pg_epoch: 2554 pg[429.6( v 2554'19 (0'0,2554'19] local-les=2554 n=4 ec=2464 les/c 2554/2554 2553/2553/2480) [3,5] r=0 lpr=2553 crt=2554'17 lcod 2554'18 mlcod 2554'18 active+clean NIBBLEWISE] get_object_context: no obc for soid 429/00000006:.ceph-internal/hit_set_429.6_archive_2015-09-19 23:18:49.366823Z_2015-09-19 23:18:52.551666Z/head and !can_create
     0> 2015-09-19 23:19:01.668845 7f6d6f930700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 7f6d6f930700 time 2015-09-19 23:19:01.650276
osd/ReplicatedPG.cc: 11062: FAILED assert(obc)

 ceph version 9.0.3-1735-g210156f (210156f3d22c491e8e7f6e5c797464c45978f8db)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f6d9099df6b]
 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x54e) [0x7f6d905f80fe]
 3: (ReplicatedPG::hit_set_persist()+0xcc1) [0x7f6d905fb611]
 4: (ReplicatedPG::do_op(std::shared_ptr<OpRequest>&)+0xf8f) [0x7f6d90603d1f]
 5: (ReplicatedPG::do_request(std::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x6dd) [0x7f6d9059da4d]
 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3bd) [0x7f6d903fff2d]
 7: (PGQueueable::RunVis::operator()(std::shared_ptr<OpRequest>&)+0x5d) [0x7f6d9040014d]
 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8c4) [0x7f6d90424ed4]
 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x85f) [0x7f6d9098e98f]
 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f6d90990890]
 11: (()+0x8182) [0x7f6d8ed55182]
 12: (clone()+0x6d) [0x7f6d8d09c47d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

i thought this was mismatched timezones but it's not ... the vms all have the same tz.

/a/sage-2015-09-19_14:20:08-upgrade:hammer-x-master---basic-vps/1063065


Related issues

Related to Ceph - Bug #13158: hammer -> infernalis upgrade tests fail with hit_set_trim assert (timezones!) Rejected 09/18/2015
Related to Ceph - Bug #14399: "ReplicatedPG.cc: 10483: FAILED assert(obc)" in rados-hammer-distro-basic-mira Duplicate 01/18/2016

History

#1 Updated by Sage Weil over 8 years ago

  • Assignee set to Samuel Just

#2 Updated by Samuel Just over 8 years ago

  • Status changed from New to 7

#3 Updated by Samuel Just over 8 years ago

  • Status changed from 7 to Resolved

#4 Updated by Yuri Weinstein about 8 years ago

  • Related to Bug #14399: "ReplicatedPG.cc: 10483: FAILED assert(obc)" in rados-hammer-distro-basic-mira added

Also available in: Atom PDF