Bug #21250: os/bluestore/BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty()) - RADOS - Ceph

Actions

Copy link

Bug #21250

closed

os/bluestore/BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

High

Assignee:

Sage Weil

Category:

Target version:

% Done:

Source:

Tags:

Backport:

luminous

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

     0> 2017-09-02 23:59:44.253608 7f1d05399700 -1 /build/ceph-13.0.0-338-gbf905f8/src/os/bluestore/BlueFS.cc: In function 'void BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)' thread 7f1d05399700 time 2017-09-02 23:59:44.248140
/build/ceph-13.0.0-338-gbf905f8/src/os/bluestore/BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())

 ceph version 13.0.0-338-gbf905f8 (bf905f878cf6a2ef02738b2e36188c0039afb458) mimic (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0xdd7c35a2e2]
 2: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0xd0e) [0xdd7c2e309e]
 3: (BlueFS::sync_metadata()+0x4ee) [0xdd7c2e470e]
 4: (BlueRocksDirectory::Fsync()+0xd) [0xdd7c2f1d8d]
 5: (rocksdb::FlushJob::WriteLevel0Table()+0x1048) [0xdd7c7b6668]
 6: (rocksdb::FlushJob::Run(rocksdb::FileMetaData*)+0xf3) [0xdd7c7b8653]
 7: (rocksdb::DBImpl::FlushMemTableToOutputFile(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*)+0x513) [0xdd7c78abb3]
 8: (rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*)+0x47e) [0xdd7c78b76e]
 9: (rocksdb::DBImpl::BackgroundCallFlush()+0x10b) [0xdd7c79060b]
 10: (rocksdb::DBImpl::BGWorkFlush(void*)+0x82) [0xdd7c790b12]
 11: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x24e) [0xdd7c80316e]
 12: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0xdd7c80334d]
 13: (()+0xb8c80) [0x7f1d2a007c80]
 14: (()+0x770a) [0x7f1d2a6f670a]
 15: (clone()+0x6d) [0x7f1d2976d82d]

/a/sage-2017-09-02_21:10:18-rados-wip-sage-testing-2017-09-02-1439-distro-basic-smithi/1591262

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Sage Weil over 6 years ago

looks like two concurrent threads trying to compact_log_async:

  -268> 2017-09-02 23:59:44.246520 7f1d05399700 10 bluefs _compact_log_async
  -267> 2017-09-02 23:59:44.246526 7f1d05399700 10 bluefs _compact_log_async old_log_jump_to 0x2800000 need 0x2c00000
...
  -195> 2017-09-02 23:59:44.247389 7f1d1b3c5700 10 bluefs _compact_log_async
  -194> 2017-09-02 23:59:44.247390 7f1d1b3c5700 10 bluefs _compact_log_async old_log_jump_to 0x2c00000 need 0x3000000

Actions

Copy link