Bug #22540
bluestore crush when deleting pool
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I was testing 4K random write. when i delete my test pool, osd crush, and never get up.
2017-12-26 10:52:39.989114 7f00f03fe700 0 osd.1 168 crush map has features 2199057072128 was 2200130822657, adjusting msgr requires for mons 2017-12-26 10:52:39.989121 7f00f03fe700 0 osd.1 168 crush map has features 2199057072128, adjusting msgr requires for osds 2017-12-26 10:52:40.054375 7f00d7bff700 0 bluestore(/var/lib/ceph/osd/ceph-1) error (39) Directory not empty not handled on operation 21 (op 1, counting from 0) 2017-12-26 10:52:40.054398 7f00d7bff700 0 bluestore(/var/lib/ceph/osd/ceph-1) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "1.225_head", "oid": "#1:a4400000::::head#" }, { "op_num": 1, "op_name": "rmcoll", "collection": "1.225_head" } ] } 2017-12-26 10:52:40.057131 7f00d7bff700 -1 os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f00d7bff700 time 2017-12-26 10:52:40.054428 os/bluestore/BlueStore.cc: 4425: FAILED assert(0 == "unexpected error") ceph version 10.2.10.1 (a17b5433d4f35a14f6d3e4c9c42ca8e18298fd25) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f00ffd62a25] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x77a) [0x7f00ff9ece7a] 3: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, std::shared_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a5) [0x7f00ff9ef6a5] 4: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, std::shared_ptr<TrackedOp>)+0x2a6) [0x7f00ff99a956] 5: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x6b4) [0x7f00ff6bc164] 6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11a) [0x7f00ff718e6a] 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x7f00ffd53a6e] 8: (ThreadPool::WorkThread::entry()+0x10) [0x7f00ffd54950] 9: (()+0x72f5) [0x7f00fdab72f5] 10: (clone()+0x6d) [0x7f00fc08404d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
after restart osd
-1> 2017-12-26 11:08:32.872624 7f03ee9a7880 20 osd.24 0 get_map 168 - loading and decoding 0x7f03e94c6780 0> 2017-12-26 11:08:32.874402 7f03ee9a7880 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f03ee9a7880 time 2017-12-26 11:08:32.872869 osd/OSD.h: 894: FAILED assert(ret) ceph version 10.2.10.1 (a17b5433d4f35a14f6d3e4c9c42ca8e18298fd25) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f03ee439a25] 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f03ede16ecd] 3: (OSD::init()+0x1fe2) [0x7f03eddca8c2] 4: (main()+0x2c01) [0x7f03edd2f201] 5: (__libc_start_main()+0xf5) [0x7f03ea75ca05] 6: (()+0x35c699) [0x7f03edd78699] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
this is similar to http://tracker.ceph.com/issues/15636
History
#1 Updated by Sage Weil about 6 years ago
- Status changed from New to Won't Fix
This is teh jewel bluestore, which is experiemental and very different from the luminous version.!