Project

General

Profile

Actions

Bug #10654

closed

osd: balance_reads hits osd/ReplicatedPG.cc: 398: FAILED assert(needs_recovery)

Added by Sage Weil about 9 years ago. Updated over 8 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

   -14> 2015-01-26 15:14:45.264963 7f7d7e770700 10 osd.4 pg_epoch: 21 pg[2.4( v 10'8 lc 10'3 (0'0,10'8] local-les=21 n=2 ec=9 les/c 16/16 20/20/15) [0,4] r=1 lpr=21 pi=15-19/1 luod=0'0 crt=10'8 lcod 0'0 active m=2] handle_message: 0x3aada00
   -13> 2015-01-26 15:14:45.264976 7f7d7e770700 10 osd.4 pg_epoch: 21 pg[2.4( v 10'8 lc 10'3 (0'0,10'8] local-les=21 n=2 ec=9 les/c 16/16 20/20/15) [0,4] r=1 lpr=21 pi=15-19/1 luod=0'0 crt=10'8 lcod 0'0 active m=2] do_op osd_op(client.4114.0:88 plana8818273-9 [read 0~3288050,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 2.750c3fa4 ack+read+balance_reads+known_if_redirect
ed e21) v4 may_read -> read-ordered flags ack+read+balance_reads+known_if_redirected
   -12> 2015-01-26 15:14:45.265671 7f7d88f85700  1 -- 10.214.132.3:6805/23794 <== osd.0 10.214.133.28:6801/17188 69 ==== pg_info(1 pgs e21:2.4) v4 ==== 743+0+0 (3199552171 0 0) 0x40a6200 con 0x3b5d580
   -11> 2015-01-26 15:14:45.265701 7f7d88f85700 10 osd.4 21 do_waiters -- start
   -10> 2015-01-26 15:14:45.265705 7f7d88f85700 10 osd.4 21 do_waiters -- finish
    -9> 2015-01-26 15:14:45.265707 7f7d88f85700 20 osd.4 21 _dispatch 0x40a6200 pg_info(1 pgs e21:2.4) v4
    -8> 2015-01-26 15:14:45.265712 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265558, event: header_read, op: pg_info(1 pgs e21:2.4)
    -7> 2015-01-26 15:14:45.265719 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265559, event: throttled, op: pg_info(1 pgs e21:2.4)
    -6> 2015-01-26 15:14:45.265723 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265611, event: all_read, op: pg_info(1 pgs e21:2.4)
    -5> 2015-01-26 15:14:45.265727 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265699, event: dispatched, op: pg_info(1 pgs e21:2.4)
    -4> 2015-01-26 15:14:45.265732 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265731, event: waiting_for_osdmap, op: pg_info(1 pgs e21:2.4)
    -3> 2015-01-26 15:14:45.265736 7f7d88f85700  7 osd.4 21 handle_pg_info pg_info(1 pgs e21:2.4) v4 from osd.0
    -2> 2015-01-26 15:14:45.265739 7f7d88f85700 15 osd.4 21 require_same_or_newer_map 21 (i am 21) 0x40a6200
    -1> 2015-01-26 15:14:45.265743 7f7d88f85700  5 -- op tracker -- seq: 195, time: 2015-01-26 15:14:45.265743, event: started, op: pg_info(1 pgs e21:2.4)
     0> 2015-01-26 15:14:45.268254 7f7d7e770700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::wait_for_unreadable_object(const hobject_t&, OpRequestRef)' thread 7f7d7e770700 time 2015-01-26 15:14:45.265010
osd/ReplicatedPG.cc: 398: FAILED assert(needs_recovery)

 ceph version 0.91-677-g0bd69ba (0bd69bae7e05a2a5bb7c5fd4331b4fc2fb0f0805)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xadfe0f]
 2: (ReplicatedPG::wait_for_unreadable_object(hobject_t const&, std::tr1::shared_ptr<OpRequest>)+0x50f) [0x81f6bf]
 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x44e) [0x85175e]
 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x63f) [0x7ebd2f]
 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x17f) [0x666cff]
 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x6c1) [0x6677c1]
 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x652) [0xad0ab2]
 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xad21e0]
 9: (()+0x7e9a) [0x7f7d99c8fe9a]
 10: (clone()+0x6d) [0x7f7d984393fd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions #1

Updated by Sage Weil over 8 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF