Project

General

Profile

Actions
Copy link

Bug #46804

closed

nautilus: upgrade:nautilus-p2p/nautilus-p2p-stress-split: BlueFS::_flush_and_sync_log ceph_assert(r == 0)

Added by Neha Ojha almost 4 years ago. Updated over 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-07-30T21:32:34.927 INFO:tasks.ceph.osd.0.smithi058.stderr:2020-07-30 21:32:34.921 7fd444355700 -1 bluestore(/var/lib/ceph/osd/ceph-0) allocate_bluefs_freespace failed to allocate on 0x3ff80000 min_size 0x400000 > allocated total 0x0 bluefs_shared_alloc_size 0x10000 allocated 0x0 available 0x 1e5e5c000
2020-07-30T21:32:34.927 INFO:tasks.ceph.osd.0.smithi058.stderr:/build/ceph-14.2.10-254-g76851635289/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, uint64_t, uint64_t)' thread 7fd444355700 time 2020-07-30 21:32:34.925698
2020-07-30T21:32:34.927 INFO:tasks.ceph.osd.0.smithi058.stderr:/build/ceph-14.2.10-254-g76851635289/src/os/bluestore/BlueFS.cc: 2197: FAILED ceph_assert(r == 0)
2020-07-30T21:32:34.928 INFO:tasks.ceph.osd.0.smithi058.stderr:2020-07-30 21:32:34.921 7fd444355700 -1 bluefs _allocate failed to expand slow device to fit +0x400000
2020-07-30T21:32:34.939 INFO:teuthology.orchestra.run.smithi058.stderr:osd.1: osd_enable_op_tracker = 'true'
2020-07-30T21:32:34.940 INFO:tasks.ceph.osd.0.smithi058.stderr: ceph version 14.2.10-254-g76851635289 (7685163528909e74749288840a480c00c24d4152) nautilus (stable)
2020-07-30T21:32:34.940 INFO:tasks.ceph.osd.0.smithi058.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x562c8121bb14]
2020-07-30T21:32:34.941 INFO:tasks.ceph.osd.0.smithi058.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x562c8121bcef]
2020-07-30T21:32:34.941 INFO:tasks.ceph.osd.0.smithi058.stderr: 3: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0x1a95) [0x562c8182b3c5]
2020-07-30T21:32:34.941 INFO:tasks.ceph.osd.0.smithi058.stderr: 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x9c) [0x562c8182c6fc]
2020-07-30T21:32:34.941 INFO:tasks.ceph.osd.0.smithi058.stderr: 5: (BlueRocksWritableFile::Sync()+0x63) [0x562c8184cd23]
2020-07-30T21:32:34.941 INFO:tasks.ceph.osd.0.smithi058.stderr: 6: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x3e9) [0x562c81ebd759]
2020-07-30T21:32:34.942 INFO:tasks.ceph.osd.0.smithi058.stderr: 7: (rocksdb::WritableFileWriter::Sync(bool)+0x3a6) [0x562c81ec0bf6]
2020-07-30T21:32:34.942 INFO:tasks.ceph.osd.0.smithi058.stderr: 8: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x32c) [0x562c81d2b93c]
2020-07-30T21:32:34.942 INFO:tasks.ceph.osd.0.smithi058.stderr: 9: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x245e) [0x562c81d349be]
2020-07-30T21:32:34.942 INFO:tasks.ceph.osd.0.smithi058.stderr: 10: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x30) [0x562c81d35030]
2020-07-30T21:32:34.942 INFO:tasks.ceph.osd.0.smithi058.stderr: 11: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x81) [0x562c81ce27b1]
2020-07-30T21:32:34.943 INFO:tasks.ceph.osd.0.smithi058.stderr: 12: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x97) [0x562c81ce3197]
2020-07-30T21:32:34.943 INFO:tasks.ceph.osd.0.smithi058.stderr: 13: (BlueStore::_kv_sync_thread()+0x205a) [0x562c8178da6a]
2020-07-30T21:32:34.943 INFO:tasks.ceph.osd.0.smithi058.stderr: 14: (BlueStore::KVSyncThread::entry()+0xd) [0x562c817af9ed]
2020-07-30T21:32:34.943 INFO:tasks.ceph.osd.0.smithi058.stderr: 15: (()+0x76db) [0x7fd4530686db]
2020-07-30T21:32:34.943 INFO:tasks.ceph.osd.0.smithi058.stderr: 16: (clone()+0x3f) [0x7fd451e08a3f]

Note that this is an upgrade test, where half of the cluster has been upgraded to latest nautilus. Also, the test is using objectstore/default, which means different allocators for upgraded vs not upgraded osds.

upgrade:nautilus-p2p/nautilus-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 7-final-workload/{rbd-python rgw-swift snaps-many-objects} objectstore/default supported-all-distro/ubuntu_latest thrashosds-health}

/a/yuriw-2020-07-30_20:39:04-upgrade:nautilus-p2p-wip-yuri-testing-2020-07-30-1921-nautilus-distro-basic-smithi/5270997

Related issues 1 (0 open — 1 closed)

Related to bluestore - Bug #50656: bluefs _allocate unable to allocate, though enough freeResolvedIgor Fedotov

Actions
Actions
Copy link

Also available in: Atom PDF