Bug #21629: interval_map.h: 161: FAILED assert(len > 0) - RADOS - Ceph

Actions

Copy link

Bug #21629

closed

interval_map.h: 161: FAILED assert(len > 0)

Added by Jason Dillaman over 6 years ago. Updated over 4 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Jason Dillaman

Category:

Target version:

% Done:

Source:

Tags:

Backport:

luminous

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

 2: (CallClientContexts::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0x1207) [0x55f8e7013d17]
 3: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x81) [0x55f8e6fe4f91]
 4: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x11f6) [0x55f8e6feefe6]
 5: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1bf) [0x55f8e7000d1f]
 6: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x55f8e6ee2e00]
 7: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x503) [0x55f8e6e48a83]
 8: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a7) [0x55f8e6cd67e7]
 9: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x5a) [0x55f8e6f6245a]
 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x11ea) [0x55f8e6cf978a]
 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x8ef) [0x55f8e72cfb4f]
 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55f8e72d2e50]

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Jason Dillaman over 6 years ago

Priority changed from Normal to Urgent

Actions

Copy link

Updated by Jason Dillaman over 6 years ago

The compare-extent op was beyond the truncated extent of the object. The EC async read code does not handle zero-length read results from the object store.

2017-10-02 13:46:17.221566 7f40ff5dd700 20 osd.1 pg_epoch: 129 pg[1.3s0( v 129'5725 (106'4136,129'5725] local-lis/les=16/17 n=816 ec=16/16 lis/c 16/16 les/c/f 17/17/0 16/16/16) [1,0,2] r=0 lpr=16 crt=129'5725 lcod 129'5724 mlcod 129'5724 active+clean] do_op: op osd_op(client.14116.0:449747 1.3s0 1:dd083836:::rbd_data.3.3724649bb77c.0000000000000005:head [set-alloc-hint object_size 2097152 write_size 2097152,write 1540096~16384] snapc 0=[] ondisk+write+known_if_redirected e129) v8

2017-10-02 16:24:05.176279 7ff7e3bdb700 20 osd.2 pg_epoch: 335 pg[1.1s0( v 335'10475 (311'8942,335'10475] local-lis/les=202/203 n=1478 ec=16/16 lis/c 202/202 les/c/f 203/203/0 202/202/202) [2,1,0] r=0 lpr=202 crt=335'10475 lcod 335'10474 mlcod 335'10474 active+clean] do_op obc obc(1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head(dne) rwstate(write n=1 w=0))

2017-10-02 13:46:17.355555 7f40ff5dd700 20 osd.1 pg_epoch: 129 pg[1.3s0( v 129'5728 (106'4136,129'5728] local-lis/les=16/17 n=818 ec=16/16 lis/c 16/16 les/c/f 17/17/0 16/16/16) [1,0,2] r=0 lpr=16 crt=129'5728 lcod 129'5727 mlcod 129'5727 active+clean] do_op: op osd_op(client.14116.0:449764 1.3s0 1:dd083836:::rbd_data.3.3724649bb77c.0000000000000005:head [read 1982464~423] snapc 0=[] ondisk+read+known_if_redirected e129) v8

2017-10-02 16:24:05.232113 7ff7e3bdb700 10 osd.2 pg_epoch: 335 pg[1.1s0( v 335'10478 (311'8942,335'10478] local-lis/les=202/203 n=1480 ec=16/16 lis/c 202/202 les/c/f 203/203/0 202/202/202) [2,1,0] r=0 lpr=202 crt=335'10477 lcod 335'10477 mlcod 335'10477 active+clean] get_object_context: 0x564c4f849480 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head rwstate(none n=0 w=0) oi: 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head(335'10476 client.74117.0:450198 dirty|extents s 1556480 uv 10476 alloc_hint [2097152 2097152 0]) exists: 1 ssc: 0x564c4efbcf20 snapset: 0=[]:{}

2017-10-02 13:46:17.550584 7f40ff5dd700 20 osd.1 pg_epoch: 129 pg[1.3s0( v 129'5734 (106'4136,129'5734] local-lis/les=16/17 n=812 ec=16/16 lis/c 16/16 les/c/f 17/17/0 16/16/16) [1,0,2] r=0 lpr=16 crt=129'5734 lcod 129'5733 mlcod 129'5733 active+clean] do_op: op osd_op(client.14116.0:449803 1.3s0 1:dd083836:::rbd_data.3.3724649bb77c.0000000000000005:head [truncate 950272] snapc 0=[] ondisk+write+known_if_redirected e129) v8

2017-10-02 16:24:05.292878 7ff7e3bdb700 10 osd.2 pg_epoch: 335 pg[1.1s0( v 335'10483 (311'8942,335'10483] local-lis/les=202/203 n=1475 ec=16/16 lis/c 202/202 les/c/f 203/203/0 202/202/202) [2,1,0] r=0 lpr=202 crt=335'10483 lcod 335'10482 mlcod 335'10482 active+clean] get_object_context: 0x564c4f849480 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head rwstate(none n=0 w=0) oi: 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head(335'10476 client.74117.0:450198 dirty|extents s 1556480 uv 10476 alloc_hint [2097152 2097152 0]) exists: 1 ssc: 0x564c4efbcf20 snapset: 0=[]:{}

2017-10-02 13:46:17.668572 7f40ff5dd700 20 osd.1 pg_epoch: 129 pg[1.3s0( v 129'5735 (106'4136,129'5735] local-lis/les=16/17 n=812 ec=16/16 lis/c 16/16 les/c/f 17/17/0 16/16/16) [1,0,2] r=0 lpr=16 crt=129'5735 lcod 129'5734 mlcod 129'5734 active+clean] do_op: op osd_op(client.14116.0:449825 1.3s0 1:dd083836:::rbd_data.3.3724649bb77c.0000000000000005:head [read 245760~3361] snapc 0=[] ondisk+read+known_if_redirected e129) v8

2017-10-02 16:24:05.316514 7ff7e3bdb700 10 osd.2 pg_epoch: 335 pg[1.1s0( v 335'10485 (311'8942,335'10485] local-lis/les=202/203 n=1475 ec=16/16 lis/c 202/202 les/c/f 203/203/0 202/202/202) [2,1,0] r=0 lpr=202 crt=335'10485 lcod 335'10484 mlcod 335'10484 active+clean] get_object_context: 0x564c4f849480 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head rwstate(none n=0 w=0) oi: 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head(335'10484 client.74117.0:450252 dirty|extents s 950272 uv 10484 alloc_hint [2097152 2097152 0]) exists: 1 ssc: 0x564c4efbcf20 snapset: 0=[]:{}

2017-10-02 13:46:18.896901 7f40ff5dd700 20 osd.1 pg_epoch: 129 pg[1.3s0( v 129'5743 (108'4236,129'5743] local-lis/les=16/17 n=814 ec=16/16 lis/c 16/16 les/c/f 17/17/0 16/16/16) [1,0,2] r=0 lpr=16 crt=129'5743 lcod 129'5742 mlcod 129'5742 active+clean] do_op: op osd_op(client.14116.0:449976 1.3s0 1:dd083836:::rbd_data.3.3724649bb77c.0000000000000005:head [cmpext,write 1843047~144] snapc 0=[] ondisk+write+known_if_redirected e129) v8

2017-10-02 16:24:05.627814 7ff7e3bdb700 10 osd.2 pg_epoch: 335 pg[1.1s0( v 335'10489 (311'8942,335'10489] local-lis/les=202/203 n=1476 ec=16/16 lis/c 202/202 les/c/f 203/203/0 202/202/202) [2,1,0] r=0 lpr=202 crt=335'10489 lcod 335'10488 mlcod 335'10488 active+clean] find_object_context 1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head @head oi=1:8ded059a:::rbd_data.13.12185649bb77c.0000000000000005:head(335'10484 client.74117.0:450252 dirty|extents s 950272 uv 10484 alloc_hint [2097152 2097152 0])

Actions

Copy link