Project

General

Profile

Actions
Copy link

Bug #36526

closed

segv in BlueStore::OldExtent::create

Added by Sage Weil over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  -356> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.extentmap(0x55f8a3f7a570) fault_range open shard 0x238000 (576 bytes)
  -350> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x3787 opened SharedBlob(0x55f8a0954380 sbid 0x3787)
  -349> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0954700) get_ref 0x0~8000 Blob(0x55f8a0954700 blob([0x684000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0954380 sbid 0x3787))
  -348> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0954700) get_ref init 0x8000, 4000
  -346> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x3788 opened SharedBlob(0x55f8a0954150 sbid 0x3788)
  -343> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a09542a0) get_ref 0x0~8000 Blob(0x55f8a09542a0 blob([0x68c000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0954150 sbid 0x3788))
  -342> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a09542a0) get_ref init 0x8000, 4000
  -340> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x3789 opened SharedBlob(0x55f89fa361c0 sbid 0x3789)
  -338> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0cf5f10) get_ref 0x0~8000 Blob(0x55f8a0cf5f10 blob([0x694000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f89fa361c0 sbid 0x3789))
  -337> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0cf5f10) get_ref init 0x8000, 4000
  -336> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x378a opened SharedBlob(0x55f89fcc6150 sbid 0x378a)
  -334> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f89fa360e0) get_ref 0x0~8000 Blob(0x55f89fa360e0 blob([0x69c000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f89fcc6150 sbid 0x378a))
  -333> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f89fa360e0) get_ref init 0x8000, 4000
  -330> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x378b opened SharedBlob(0x55f89fcc6000 sbid 0x378b)
  -328> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f89fcc60e0) get_ref 0x0~8000 Blob(0x55f89fcc60e0 blob([0x6a4000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f89fcc6000 sbid 0x378b))
  -327> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f89fcc60e0) get_ref init 0x8000, 4000
  -325> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x378c opened SharedBlob(0x55f8a0d81d50 sbid 0x378c)
  -323> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0796770) get_ref 0x0~8000 Blob(0x55f8a0796770 blob([0x6ac000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0d81d50 sbid 0x378c))
  -322> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0796770) get_ref init 0x8000, 4000
  -319> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x378d opened SharedBlob(0x55f8a0d81960 sbid 0x378d)
  -318> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0d81ce0) get_ref 0x0~8000 Blob(0x55f8a0d81ce0 blob([0x6b4000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0d81960 sbid 0x378d))
  -316> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0d81ce0) get_ref init 0x8000, 4000
  -314> 2018-10-18 03:29:43.559 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x378e opened SharedBlob(0x55f8a0d817a0 sbid 0x378e)
  -313> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0d818f0) get_ref 0x0~37dd Blob(0x55f8a0d818f0 blob([0x6bc000~4000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0d817a0 sbid 0x378e))
  -312> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0d818f0) get_ref init 0x4000, 4000
  -310> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.extentmap(0x55f8a3f7a570) fault_range open shard 0x2f8000 (400 bytes)
  -307> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6a80) put_ref 0x0~8000 Blob(0x55f8a08c6a80 blob([0x219c000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f89fe31c00 loaded (sbid 0x33f9 ref_map(0x219c000~4000=3))))
  -304> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3dc0) put_ref 0x0~8000 Blob(0x55f8a0be3dc0 blob([0x21a0000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0799110 loaded (sbid 0x33fa ref_map(0x21a0000~4000=3))))
  -302> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0954000) put_ref 0x0~8000 Blob(0x55f8a0954000 blob([0x21a4000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a084bea0 loaded (sbid 0x33fb ref_map(0x21a4000~4000=3))))
  -300> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6af0) put_ref 0x0~8000 Blob(0x55f8a08c6af0 blob([0x21a8000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a084bc00 loaded (sbid 0x33fc ref_map(0x21a8000~4000=3))))
  -299> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3d50) put_ref 0x0~8000 Blob(0x55f8a0be3d50 blob([0x21ac000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a084b960 loaded (sbid 0x33fd ref_map(0x21ac000~4000=3))))
  -296> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6bd0) put_ref 0x0~8000 Blob(0x55f8a08c6bd0 blob([0x21b0000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0a93490 loaded (sbid 0x33fe ref_map(0x21b0000~4000=3))))
  -293> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0954070) put_ref 0x0~8000 Blob(0x55f8a0954070 blob([0x21b4000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0a93f10 loaded (sbid 0x33ff ref_map(0x21b4000~4000=3))))
  -292> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3ce0) put_ref 0x0~8000 Blob(0x55f8a0be3ce0 blob([0x21b8000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a11fbd50 loaded (sbid 0x3400 ref_map(0x21b8000~4000=3))))
  -290> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6c40) put_ref 0x0~8000 Blob(0x55f8a08c6c40 blob([0x21bc000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a11fbf80 loaded (sbid 0x3401 ref_map(0x21bc000~4000=3))))
  -288> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a09540e0) put_ref 0x0~8000 Blob(0x55f8a09540e0 blob([0x21c0000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0d80380 loaded (sbid 0x3402 ref_map(0x21c0000~4000=3))))
  -286> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3c70) put_ref 0x0~8000 Blob(0x55f8a0be3c70 blob([0x21c4000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0d800e0 loaded (sbid 0x3403 ref_map(0x21c4000~4000=3))))
  -284> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6cb0) put_ref 0x0~8000 Blob(0x55f8a08c6cb0 blob([0x21c8000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0d815e0 loaded (sbid 0x3404 ref_map(0x21c8000~4000=3))))
  -281> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a09541c0) put_ref 0x0~8000 Blob(0x55f8a09541c0 blob([0x21cc000~4000] clen 0x8000 -> 0x610 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0d81340 loaded (sbid 0x3405 ref_map(0x21cc000~4000=3))))
  -280> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3c00) put_ref 0x0~8000 Blob(0x55f8a0be3c00 blob([0x21dc000~4000] clen 0x8000 -> 0x3183 compressed+shared) use_tracker(0x8000 0x8000) SharedBlob(0x55f8a0d810a0 loaded (sbid 0x3406 ref_map(0x21dc000~4000=3))))
  -278> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a08c6d20) put_ref 0x0~8000 Blob(0x55f8a08c6d20 blob([0x21e0000~8000] csum+shared crc32c/0x1000) use_tracker(0x2*0x4000 0x[4000,4000]) SharedBlob(0x55f8a0d80cb0 loaded (sbid 0x3407 ref_map(0x21e0000~8000=3))))
  -276> 2018-10-18 03:29:43.559 7f788e21c700 20 bluestore.blob(0x55f8a0be3b90) put_ref 0x0~8000 Blob(0x55f8a0be3b90 blob([0x21e8000~8000] csum+shared crc32c/0x1000) use_tracker(0x2*0x4000 0x[4000,4000]) SharedBlob(0x55f8a0d81f80 loaded (sbid 0x3408 ref_map(0x21e8000~8000=3))))
    -3> 2018-10-18 03:29:43.568 7f788e21c700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f788e21c700 thread_name:tp_osd_tp

 ceph version 14.0.0-4266-g4bb9c6a (4bb9c6abedbceb69fa830521ffbfd8d1ef707be2) nautilus (dev)
 1: (()+0xf6d0) [0x7f78b2de96d0]
 2: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x9b) [0x55f8939461db]
 3: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::list_member_hook<void, void, void>, &BlueStore::OldExtent::old_extent_item>, void, void
, void>*)+0x43b) [0x55f89394670b]
 4: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::allocator<BlueStore::SharedBlob*> >*)+0x104) [0x55f893991384]
 5: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xbf) [0x55f893993cef]
 6: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x5f) [0x55f89399586f]
 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x108d) [0x55f8939a424d]
 8: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x54c) [0x55f8939a901c]
 9: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x82) [0x55f893580132]
 10: (PrimaryLogPG::remove_missing_object(hobject_t const&, eversion_t, Context*)+0x685) [0x55f893675025]
 11: (PGBackend::handle_recovery_delete(boost::intrusive_ptr<OpRequest>)+0x169) [0x55f893713409]
 12: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0xbb) [0x55f893714f9b]
 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x6c9) [0x55f8936c5079]
 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55f89350e889]
...

/a/sage-2018-10-17_20:20:33-rados-nautilus-distro-basic-smithi/3154345

Related issues 5 (0 open — 5 closed)

Related to bluestore - Bug #36567: Segmentation fault in BlueStore::Blob::discard_unallocatedDuplicate10/23/2018

Actions
Related to bluestore - Bug #37090: BlueStore.cc: 3099: FAILED assert(0 == "uh oh, missing shared_blob")Can't reproduce11/13/2018

Actions
Has duplicate bluestore - Bug #36303: luminous: 12.2.8 - FAILED assert(0 == "put on missing extent (nothing before)")Duplicate10/03/2018

Actions
Copied to bluestore - Backport #36591: luminous: segv in BlueStore::OldExtent::createResolvedNathan CutlerActions
Copied to bluestore - Backport #36592: mimic: segv in BlueStore::OldExtent::createResolvedNathan CutlerActions
Actions
Copy link
 #1

Updated by Sage Weil over 5 years ago 

  -824> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d81d50) put 0x55f8a0d81d50 removing self from set 0x55f89ea453a8
  -823> 2018-10-18 03:29:43.558 7f788e21c700 20 bluestore.blob(0x55f8a0be3b90) get_ref init 0x8000, 4000
  -822> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d81d68 in 0x55f89ea1efc0) _clear
  -821> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d81ce0) put 0x55f8a0d81ce0 removing self from set 0x55f89ea453a8
  -820> 2018-10-18 03:29:43.558 7f788aa15700 20 bluestore(/var/lib/ceph/osd/ceph-4) _do_alloc_write  compressed 0x8000 -> 0x610 => 0x4000 with 1
  -819> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d81cf8 in 0x55f89ea1efc0) _clear
  -818> 2018-10-18 03:29:43.558 7f788e21c700 10 bluestore(/var/lib/ceph/osd/ceph-4).collection(2.6_head 0x55f89ea452c0) open_shared_blob sbid 0x3409 had SharedBlob(0x55f8a0d81ce0 loaded (sbid 0x3409 ref_map(0x21f0000~8000=3)))
  -817> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d81a40) put 0x55f8a0d81a40 removing self from set 0x55f89ea453a8
  -816> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d81a58 in 0x55f89ea1efc0) _clear
  -815> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d81960) put 0x55f8a0d81960 removing self from set 0x55f89ea453a8
  -814> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d81978 in 0x55f89ea1efc0) _clear
  -813> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d818f0) put 0x55f8a0d818f0 removing self from set 0x55f89ea453a8
  -812> 2018-10-18 03:29:43.558 7f788e21c700 20 bluestore.blob(0x55f8a0d80d90) get_ref 0x0~8000 Blob(0x55f8a0d80d90 blob([0x21f0000~8000] csum+shared crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x55f8a0d81ce0 loaded (sbid 0x55f8a02beb40 ref_map(0x21f0000~8000=3))))
  -811> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d81908 in 0x55f89ea1efc0) _clear
  -810> 2018-10-18 03:29:43.558 7f788e21c700 20 bluestore.blob(0x55f8a0d80d90) get_ref init 0x8000, 4000
  -809> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d817a0) put 0x55f8a0d817a0 removing self from set 0x55f89ea453a8
  -808> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.BufferSpace(0x55f8a0d817b8 in 0x55f89ea1efc0) _clear
  -807> 2018-10-18 03:29:43.558 7f789d23a700 20 bluestore.sharedblob(0x55f8a0d81730) put 0x55f8a0d81730 removing self from set 0x55f89ea453a8

looks like a race between removing the sharedblob and lookup
Actions
Copy link
 #2

Updated by Sage Weil over 5 years ago

  • Status changed from 12 to Fix Under Review
  • Backport set to mimic,luminous
Actions
Copy link
 #4

Updated by Sage Weil over 5 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Priority changed from High to Urgent
Actions
Copy link
 #5

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36591: luminous: segv in BlueStore::OldExtent::create added
Actions
Copy link
 #6

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36592: mimic: segv in BlueStore::OldExtent::create added
Actions
Copy link
 #7

Updated by Sage Weil over 5 years ago

  • Related to Bug #36567: Segmentation fault in BlueStore::Blob::discard_unallocated added
Actions
Copy link
 #8

Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #9

Updated by Sage Weil over 5 years ago

  • Related to Bug #37090: BlueStore.cc: 3099: FAILED assert(0 == "uh oh, missing shared_blob") added
Actions
Copy link
 #10

Updated by Josh Durgin over 5 years ago

  • Has duplicate Bug #36303: luminous: 12.2.8 - FAILED assert(0 == "put on missing extent (nothing before)") added
Actions
Copy link

Also available in: Atom PDF