Project

General

Profile

Actions
Copy link

Bug #21766

closed

os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + compression)

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  -243> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore.extentmap(0x55c296c67030) dirty_range mark inline shard dirty
  -242> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x36000~2000: 0x6000~2000 Blob(0x55c29bbb56b0 blob([!~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x0) SharedBlob(0x55c296443790 loaded (sbid 0x57ad ref_map(0x10998000~4000=2))))
  -241> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x10998000~4000]
  -240> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [] from SharedBlob(0x55c296443790 loaded (sbid 0x57ad ref_map(0x10998000~4000=1)))
  -239> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x38000~8000: 0x0~8000 Blob(0x55c298b7cc30 blob([!~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x0) SharedBlob(0x55c296440700 loaded (sbid 0x57ae ref_map(0x1099c000~4000=1))))
  -238> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x1099c000~4000]
  -237> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x1099c000~4000] from SharedBlob(0x55c296440700 loaded (sbid 0x57ae ref_map()))
  -236> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x1099c000~4000
  -235> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x40000~4000: 0x0~4000 Blob(0x55c2976608b0 blob([!~4000] csum+shared crc32c/0x1000) use_tracker(0x4000 0x0) SharedBlob(0x55c293ad95d0 loaded (sbid 0x57af ref_map(0x109a0000~4000=1))))
  -234> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109a0000~4000]
  -233> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109a0000~4000] from SharedBlob(0x55c293ad95d0 loaded (sbid 0x57af ref_map()))
  -232> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109a0000~4000
  -231> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x44000~4000: 0x4000~4000 Blob(0x55c296fa5f80 blob([!~8000] csum+has_unused+shared crc32c/0x1000 unused=0xff) use_tracker(0x2*0x4000 0x[0,0]) SharedBlob(0x55c29bbb52c0 loaded (sbid 0x57b0 ref_map(0x109a4000~4000=1))))
  -230> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109a4000~4000]
  -229> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109a4000~4000] from SharedBlob(0x55c29bbb52c0 loaded (sbid 0x57b0 ref_map()))
  -228> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109a4000~4000
  -227> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x48000~8000: 0x0~8000 Blob(0x55c29911a7e0 blob([!~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x0) SharedBlob(0x55c296dbadf0 loaded (sbid 0x57b1 ref_map(0x109c0000~4000=1))))
  -226> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109c0000~4000]
  -225> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109c0000~4000] from SharedBlob(0x55c296dbadf0 loaded (sbid 0x57b1 ref_map()))
  -224> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109c0000~4000
  -223> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x50000~8000: 0x0~8000 Blob(0x55c296c59dc0 blob([!~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x0) SharedBlob(0x55c296db9570 loaded (sbid 0x57b2 ref_map(0x109c4000~4000=1))))
  -222> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109c4000~4000]
  -221> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109c4000~4000] from SharedBlob(0x55c296db9570 loaded (sbid 0x57b2 ref_map()))
  -220> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109c4000~4000
  -219> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x58000~8000: 0x0~8000 Blob(0x55c29911d4f0 blob([!~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x0) SharedBlob(0x55c296db8f50 loaded (sbid 0x57b3 ref_map(0x109c8000~4000=1))))
  -218> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109c8000~4000]
  -217> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109c8000~4000] from SharedBlob(0x55c296db8f50 loaded (sbid 0x57b3 ref_map()))
  -216> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109c8000~4000
  -215> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish lex_old 0x60000~4000: 0x0~4000 Blob(0x55c29bbb4d80 blob([!~4000] csum+shared crc32c/0x1000) use_tracker(0x4000 0x0) SharedBlob(0x55c29694ef50 loaded (sbid 0x57b4 ref_map(0x109e0000~4000=1))))
  -214> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  blob release [0x109e0000~4000]
  -213> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  shared_blob release [0x109e0000~4000] from SharedBlob(0x55c29694ef50 loaded (sbid 0x57b4 ref_map()))
  -212> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _wctx_finish  release 0x109e0000~4000
  -211> 2017-10-11 16:06:56.231 7f5ec8352700 10 bluestore(/var/lib/ceph/osd/ceph-2) _do_remove gen and maybe_unshared_blobs 0x55c293ad95d0,0x55c296440700,0x55c296443790,0x55c29694ef50,0x55c296db8f50,0x55c296db9570,0x55c296dbadf0,0x55c29bbb52c0
  -210> 2017-10-11 16:06:56.231 7f5ec8352700 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_remove checking for unshareable blobs on 0x55c297f22b40 2#3:9e2e6e14:::10000000cb6.00000000:head#
   -41> 2017-10-11 16:06:56.236 7f5ec8352700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.0-1856-gdeabb6c/rpm/el7/BUILD/ceph-13.0.0-1856-gdeabb6c/src/os/bluestore/bluestore_types.h: In function 'int bluestore_blob_t::map(uint64_t, uint64_t, std::function<int(long unsigned int, long unsigned int)>) const' thread 7f5ec8352700 time 2017-10-11 16:06:56.232622
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.0-1856-gdeabb6c/rpm/el7/BUILD/ceph-13.0.0-1856-gdeabb6c/src/os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end())

 ceph version 13.0.0-1856-gdeabb6c (deabb6c2a8e9f7004db10b7a03d26169f0d37f2b) mimic (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55c289784120]
 2: (bluestore_blob_t::map(unsigned long, unsigned long, std::function<int (unsigned long, unsigned long)>) const+0xfe) [0x55c289665fce]
 3: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0x1161) [0x55c289644b31]
 4: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x94) [0x55c289645404]
 5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x15af) [0x55c28965804f]
 6: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a0) [0x55c289658f70]
 7: (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x65) [0x55c2893d61b5]
 8: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, Context*)+0x610) [0x55c2894ee680]
 9: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x327) [0x55c2894fe8a7]
 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x55c28940bdb0]
 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5be) [0x55c28937831e]
 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f6) [0x55c289212966]
 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x5e) [0x55c2894772de]
 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1186) [0x55c289237736]
 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839) [0x55c289789c39]
 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c28978bbd0]
 17: (()+0x7e25) [0x7f5ee2628e25]

/a/pdonnell-2017-10-10_19:28:52-kcephfs-wip-pdonnell-testing-20171010-testing-basic-smithi/1722519
Actions
Copy link
 #1

Updated by Sage Weil over 6 years ago

problem seems to be that the unsharing code isn't handling compressed extents properly.

https://github.com/ceph/ceph/pull/18255

Actions
Copy link
 #2

Updated by Sage Weil over 6 years ago

  • Status changed from In Progress to 7
Actions
Copy link
 #3

Updated by Sage Weil over 6 years ago

  • Subject changed from bluestore: to os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + snaps + cephfs)
Actions
Copy link
 #4

Updated by Sage Weil over 6 years ago

  • Subject changed from os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + snaps + cephfs) to os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + compression + cephfs)
Actions
Copy link
 #5

Updated by Sage Weil over 6 years ago

  • Subject changed from os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + compression + cephfs) to os/bluestore/bluestore_types.h: 740: FAILED assert(p != extents.end()) (ec + compression)
Actions
Copy link
 #6

Updated by Sage Weil over 6 years ago

  • Status changed from 7 to Pending Backport
Actions
Copy link
 #7

Updated by Sage Weil over 6 years ago

  • Backport set to luminous
Actions
Copy link
 #8

Updated by Sage Weil over 6 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF