Project

General

Profile

Bug #22540

bluestore crush when deleting pool

Added by hongpeng lu over 6 years ago. Updated about 6 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I was testing 4K random write. when i delete my test pool, osd crush, and never get up.

2017-12-26 10:52:39.989114 7f00f03fe700  0 osd.1 168 crush map has features 2199057072128 was 2200130822657, adjusting msgr requires for mons
2017-12-26 10:52:39.989121 7f00f03fe700  0 osd.1 168 crush map has features 2199057072128, adjusting msgr requires for osds
2017-12-26 10:52:40.054375 7f00d7bff700  0 bluestore(/var/lib/ceph/osd/ceph-1)  error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
2017-12-26 10:52:40.054398 7f00d7bff700  0 bluestore(/var/lib/ceph/osd/ceph-1)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "1.225_head",
            "oid": "#1:a4400000::::head#" 
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "1.225_head" 
        }
    ]
}

2017-12-26 10:52:40.057131 7f00d7bff700 -1 os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f00d7bff700 time 2017-12-26 10:52:40.054428
os/bluestore/BlueStore.cc: 4425: FAILED assert(0 == "unexpected error")

 ceph version 10.2.10.1 (a17b5433d4f35a14f6d3e4c9c42ca8e18298fd25)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f00ffd62a25]
 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x77a) [0x7f00ff9ece7a]
 3: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, std::shared_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a5) [0x7f00ff9ef6a5]
 4: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, std::shared_ptr<TrackedOp>)+0x2a6) [0x7f00ff99a956]
 5: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x6b4) [0x7f00ff6bc164]
 6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11a) [0x7f00ff718e6a]
 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x7f00ffd53a6e]
 8: (ThreadPool::WorkThread::entry()+0x10) [0x7f00ffd54950]
 9: (()+0x72f5) [0x7f00fdab72f5]
 10: (clone()+0x6d) [0x7f00fc08404d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

after restart osd

    -1> 2017-12-26 11:08:32.872624 7f03ee9a7880 20 osd.24 0 get_map 168 - loading and decoding 0x7f03e94c6780
     0> 2017-12-26 11:08:32.874402 7f03ee9a7880 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f03ee9a7880 time 2017-12-26 11:08:32.872869
osd/OSD.h: 894: FAILED assert(ret)

 ceph version 10.2.10.1 (a17b5433d4f35a14f6d3e4c9c42ca8e18298fd25)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f03ee439a25]
 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f03ede16ecd]
 3: (OSD::init()+0x1fe2) [0x7f03eddca8c2]
 4: (main()+0x2c01) [0x7f03edd2f201]
 5: (__libc_start_main()+0xf5) [0x7f03ea75ca05]
 6: (()+0x35c699) [0x7f03edd78699]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

this is similar to http://tracker.ceph.com/issues/15636

History

#1 Updated by Sage Weil about 6 years ago

  • Status changed from New to Won't Fix

This is teh jewel bluestore, which is experiemental and very different from the luminous version.!

Also available in: Atom PDF