Project

General

Profile

Actions
Copy link

Bug #53906

closed

BlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size)

Added by Vikhyat Umrao over 2 years ago. Updated about 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
44713
Crash signature (v1):
Crash signature (v2):

Description

  -1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size)

 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x56347eb33b8e]
 2: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf]
 3: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00]
 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009]
 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de]
 7: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8]
 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f]
 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22]
 10: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8]
 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701]
 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454]
 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1]
 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84]
 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a]
 16: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390]
 17: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1]
 18: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a]
 19: clone()

     0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) **
 in thread 7f2bd48ec700 thread_name:bstore_kv_sync

 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec]
 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf]
 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00]
 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009]
 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de]
 10: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8]
 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f]
 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22]
 13: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8]
 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701]
 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454]
 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1]
 17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84]
 18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a]
 19: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390]
 20: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1]
 21: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a]
 22: clone()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open — 1 closed)

Is duplicate of bluestore - Bug #53907: BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length)ResolvedAdam Kupczyk

Actions
Actions
Copy link
 #1

Updated by Vikhyat Umrao over 2 years ago

- Looks like the systemd restarted the OSD container pod after hitting this crash and after the restart, OSD is running fine!

Actions
Copy link
 #2

Updated by Vikhyat Umrao over 2 years ago

OSD.138 logs for thread id - 7f2bd48ec700:

 -7348> 2022-01-17T22:02:20.534+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1229. Immutable memtables: 0.
 -7247> 2022-01-17T22:02:30.137+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.871898651s of 10.000083923s, submitted: 36357
 -7237> 2022-01-17T22:02:32.410+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1236. Immutable memtables: 0.
 -7196> 2022-01-17T22:02:40.137+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.393852472s of 10.000052452s, submitted: 34450
 -7168> 2022-01-17T22:02:46.790+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1238. Immutable memtables: 0.
 -7145> 2022-01-17T22:02:50.137+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.345420837s of 10.000248909s, submitted: 33896
 -7106> 2022-01-17T22:03:00.138+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.035129309s of 10.000363350s, submitted: 35595
 -7099> 2022-01-17T22:03:01.211+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1240. Immutable memtables: 0.
 -7069> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1229. Total log size is 1073743393 while max_total_wal_size is 1073741824
 -7068> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1243. Immutable memtables: 0.
 -7067> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1243. Immutable memtables: 0.
 -7066> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1243. Immutable memtables: 0.
 -7065> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1243. Immutable memtables: 0.
 -7064> 2022-01-17T22:03:03.228+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1243. Immutable memtables: 0.
 -6924> 2022-01-17T22:03:10.139+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 1.422380924s of 10.001676559s, submitted: 36202
 -6839> 2022-01-17T22:03:15.121+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1281. Immutable memtables: 0.
 -6803> 2022-01-17T22:03:20.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.232607126s of 10.000235558s, submitted: 36495
 -6767> 2022-01-17T22:03:29.197+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1283. Immutable memtables: 0.
 -6752> 2022-01-17T22:03:30.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.836078882s of 10.000127792s, submitted: 37565
 -6713> 2022-01-17T22:03:40.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.808803082s of 10.000279427s, submitted: 36784
 -6699> 2022-01-17T22:03:43.174+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1285. Immutable memtables: 0.
 -6685> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1243. Total log size is 1073744078 while max_total_wal_size is 1073741824
 -6684> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1287. Immutable memtables: 0.
 -6683> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1287. Immutable memtables: 0.
 -6682> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1287. Immutable memtables: 0.
 -6681> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1287. Immutable memtables: 0.
 -6680> 2022-01-17T22:03:44.608+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1287. Immutable memtables: 0.
 -6532> 2022-01-17T22:03:50.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.153817177s of 10.000497818s, submitted: 35734
 -6373> 2022-01-17T22:03:58.888+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1342. Immutable memtables: 0.
 -6314> 2022-01-17T22:04:00.142+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.471657276s of 10.000211716s, submitted: 31636
 -6274> 2022-01-17T22:04:10.142+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.041631460s of 10.000046730s, submitted: 35494
 -6256> 2022-01-17T22:04:14.881+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1348. Immutable memtables: 0.
 -6222> 2022-01-17T22:04:20.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.624328852s of 10.000032425s, submitted: 33670
 -6184> 2022-01-17T22:04:30.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.294847488s of 10.000163078s, submitted: 35205
 -6182> 2022-01-17T22:04:30.338+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1350. Immutable memtables: 0.
 -6166> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1287. Total log size is 1073743362 while max_total_wal_size is 1073741824
 -6165> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1352. Immutable memtables: 0.
 -6164> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1352. Immutable memtables: 0.
 -6163> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1352. Immutable memtables: 0.
 -6162> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1352. Immutable memtables: 0.
 -6161> 2022-01-17T22:04:31.109+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1352. Immutable memtables: 0.
 -6074> 2022-01-17T22:04:40.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.215115070s of 10.000009537s, submitted: 31125
 -6037> 2022-01-17T22:04:50.141+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.136115074s of 10.000347137s, submitted: 21522
 -6006> 2022-01-17T22:04:56.481+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1358. Immutable memtables: 0.
 -5983> 2022-01-17T22:05:00.142+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.058140278s of 10.000130653s, submitted: 22103
 -5944> 2022-01-17T22:05:10.142+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.180792332s of 10.000183105s, submitted: 26826
 -5905> 2022-01-17T22:05:20.143+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.412753105s of 10.000147820s, submitted: 25214
 -5892> 2022-01-17T22:05:22.452+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1360. Immutable memtables: 0.
 -5840> 2022-01-17T22:05:30.143+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.213577747s of 10.000043869s, submitted: 20253
 -5798> 2022-01-17T22:05:40.143+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.740430355s of 10.000234604s, submitted: 18681
 -5759> 2022-01-17T22:05:50.144+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.200869083s of 10.000736237s, submitted: 21460
 -5742> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1352. Total log size is 1073746833 while max_total_wal_size is 1073741824
 -5741> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1363. Immutable memtables: 0.
 -5740> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1363. Immutable memtables: 0.
 -5739> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1363. Immutable memtables: 0.
 -5738> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1363. Immutable memtables: 0.
 -5737> 2022-01-17T22:05:53.779+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1363. Immutable memtables: 0.
 -5653> 2022-01-17T22:05:59.071+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1370. Immutable memtables: 0.
 -5636> 2022-01-17T22:06:00.146+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.126998901s of 10.001685143s, submitted: 21063
 -5594> 2022-01-17T22:06:10.146+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.532536030s of 10.000311852s, submitted: 20146
 -5552> 2022-01-17T22:06:20.146+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.273554325s of 10.000081062s, submitted: 16015
 -5506> 2022-01-17T22:06:30.146+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.303408146s of 10.000711441s, submitted: 15952
 -5466> 2022-01-17T22:06:40.149+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.279325008s of 10.002703667s, submitted: 15080
 -5447> 2022-01-17T22:06:44.362+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1372. Immutable memtables: 0.
 -5414> 2022-01-17T22:06:50.149+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.105763912s of 10.000024796s, submitted: 16855
 -5373> 2022-01-17T22:07:00.149+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.872070789s of 10.000138283s, submitted: 18359
 -5334> 2022-01-17T22:07:10.150+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.261459351s of 10.000664711s, submitted: 16004
 -5295> 2022-01-17T22:07:20.151+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.133035183s of 10.000041008s, submitted: 16842
 -5256> 2022-01-17T22:07:29.434+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1374. Immutable memtables: 0.
 -5244> 2022-01-17T22:07:30.151+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.163985252s of 10.000092506s, submitted: 16358
 -5203> 2022-01-17T22:07:40.151+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.522592545s of 10.000003815s, submitted: 14237
 -5163> 2022-01-17T22:07:50.151+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.391773224s of 10.000718117s, submitted: 14934
 -5122> 2022-01-17T22:08:00.152+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.567738056s of 10.000141144s, submitted: 13861
 -5112> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1363. Total log size is 1073753040 while max_total_wal_size is 1073741824
 -5111> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1376. Immutable memtables: 0.
 -5110> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1376. Immutable memtables: 0.
 -5109> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1376. Immutable memtables: 0.
 -5108> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1376. Immutable memtables: 0.
 -5107> 2022-01-17T22:08:01.872+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1376. Immutable memtables: 0.
 -4919> 2022-01-17T22:08:10.152+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.072527409s of 10.000085831s, submitted: 13343
 -4879> 2022-01-17T22:08:20.155+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.948874950s of 10.003495216s, submitted: 11787
 -4849> 2022-01-17T22:08:26.958+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1406. Immutable memtables: 0.
 -4813> 2022-01-17T22:08:30.155+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.690354824s of 10.000430107s, submitted: 13232
 -4774> 2022-01-17T22:08:40.156+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.019449234s of 10.001420021s, submitted: 11662
 -4733> 2022-01-17T22:08:50.158+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.823121071s of 10.001865387s, submitted: 12609
 -4692> 2022-01-17T22:09:00.159+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.787741661s of 10.000707626s, submitted: 12718
 -4652> 2022-01-17T22:09:10.159+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.782723427s of 10.000552177s, submitted: 12703
 -4610> 2022-01-17T22:09:20.161+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.186806679s of 10.001282692s, submitted: 10587
 -4570> 2022-01-17T22:09:30.162+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.969998360s of 10.000073433s, submitted: 11710
 -4556> 2022-01-17T22:09:32.486+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1409. Immutable memtables: 0.
 -4520> 2022-01-17T22:09:40.162+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.065218925s of 10.000068665s, submitted: 11141
 -4480> 2022-01-17T22:09:50.162+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.326082230s of 10.000044823s, submitted: 9633
 -4438> 2022-01-17T22:10:00.162+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.749199390s of 10.000402451s, submitted: 13089
 -4399> 2022-01-17T22:10:10.165+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.939408302s of 10.002837181s, submitted: 11599
 -4359> 2022-01-17T22:10:20.167+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.809356213s of 10.002017021s, submitted: 12853
 -4320> 2022-01-17T22:10:30.166+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.689060688s of 10.000018120s, submitted: 13281
 -4293> 2022-01-17T22:10:36.797+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1411. Immutable memtables: 0.
 -4269> 2022-01-17T22:10:40.169+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.554021835s of 10.002296448s, submitted: 14190
 -4226> 2022-01-17T22:10:50.173+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.807527542s of 10.004211426s, submitted: 12677
 -4194> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1376. Total log size is 1073755738 while max_total_wal_size is 1073741824
 -4193> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1413. Immutable memtables: 0.
 -4192> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1413. Immutable memtables: 0.
 -4191> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1413. Immutable memtables: 0.
 -4190> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1413. Immutable memtables: 0.
 -4189> 2022-01-17T22:10:57.760+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1413. Immutable memtables: 0.
 -4093> 2022-01-17T22:11:00.174+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.839452267s of 10.000724792s, submitted: 10763
 -3847> 2022-01-17T22:11:10.174+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.662105560s of 10.000135422s, submitted: 12848
 -3731> 2022-01-17T22:11:20.176+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.784808159s of 10.002503395s, submitted: 10863
 -3691> 2022-01-17T22:11:30.180+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.380425453s of 10.003009796s, submitted: 9487
 -3651> 2022-01-17T22:11:40.181+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.965118408s of 10.000519753s, submitted: 11953
 -3626> 2022-01-17T22:11:45.699+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1490. Immutable memtables: 0.
 -3593> 2022-01-17T22:11:50.182+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.822191238s of 10.000876427s, submitted: 12874
 -3553> 2022-01-17T22:12:00.182+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.168907166s of 10.000414848s, submitted: 10564
 -3513> 2022-01-17T22:12:10.182+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.079604149s of 10.000261307s, submitted: 11301
 -3474> 2022-01-17T22:12:20.183+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.842392445s of 10.000760078s, submitted: 12579
 -3434> 2022-01-17T22:12:30.182+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.133393288s of 10.000002861s, submitted: 10912
 -3392> 2022-01-17T22:12:40.182+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.905469418s of 10.000060081s, submitted: 11891
 -3352> 2022-01-17T22:12:50.184+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.832632542s of 10.001495361s, submitted: 11825
 -3337> 2022-01-17T22:12:53.021+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1492. Immutable memtables: 0.
 -3286> 2022-01-17T22:13:00.184+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.737954617s of 10.000348091s, submitted: 13159
 -3246> 2022-01-17T22:13:10.184+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.913400173s of 10.000087738s, submitted: 12216
 -3205> 2022-01-17T22:13:20.184+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.016359329s of 10.000087738s, submitted: 11451
 -3164> 2022-01-17T22:13:30.186+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.681961536s of 10.000099182s, submitted: 13345
 -3121> 2022-01-17T22:13:40.188+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.648798943s of 10.002541542s, submitted: 13670
 -3081> 2022-01-17T22:13:50.188+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.625751972s of 10.000112534s, submitted: 13870
 -3064> 2022-01-17T22:13:54.171+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1495. Immutable memtables: 0.
 -3044> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1413. Total log size is 1073747433 while max_total_wal_size is 1073741824
 -3043> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1497. Immutable memtables: 0.
 -3042> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1497. Immutable memtables: 0.
 -3041> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1497. Immutable memtables: 0.
 -3040> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1497. Immutable memtables: 0.
 -3039> 2022-01-17T22:13:56.590+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1497. Immutable memtables: 0.
 -2973> 2022-01-17T22:14:00.191+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.800518036s of 10.003147125s, submitted: 12444
 -2934> 2022-01-17T22:14:10.192+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.312356949s of 10.000099182s, submitted: 10023
 -2895> 2022-01-17T22:14:20.192+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.982282639s of 10.000566483s, submitted: 11902
 -2857> 2022-01-17T22:14:30.191+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.601618767s of 10.000000954s, submitted: 13930
 -2815> 2022-01-17T22:14:40.191+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.767084122s of 10.000088692s, submitted: 13370
 -2776> 2022-01-17T22:14:50.192+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.071086884s of 10.000385284s, submitted: 11217
 -2735> 2022-01-17T22:15:00.193+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.009041786s of 10.001303673s, submitted: 11630
 -2567> 2022-01-17T22:15:42.301+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 42.061779022s of 42.107433319s, submitted: 210
 -2565> 2022-01-17T22:15:42.610+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1503. Immutable memtables: 0.
 -2516> 2022-01-17T22:15:52.302+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.030773640s of 10.000231743s, submitted: 27813
 -2477> 2022-01-17T22:16:02.302+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.171396255s of 10.000287056s, submitted: 22045
 -2446> 2022-01-17T22:16:09.237+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1505. Immutable memtables: 0.
 -2426> 2022-01-17T22:16:12.302+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.264243126s of 10.000252724s, submitted: 26308
 -2386> 2022-01-17T22:16:22.303+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.664372921s of 10.000242233s, submitted: 29573
 -2359> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1497. Total log size is 1073747405 while max_total_wal_size is 1073741824
 -2358> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1507. Immutable memtables: 0.
 -2357> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1507. Immutable memtables: 0.
 -2356> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1507. Immutable memtables: 0.
 -2355> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1507. Immutable memtables: 0.
 -2354> 2022-01-17T22:16:28.244+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1507. Immutable memtables: 0.
 -2278> 2022-01-17T22:16:32.187+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1514. Immutable memtables: 0.
 -2271> 2022-01-17T22:16:32.302+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.312994480s of 10.000098228s, submitted: 28693
 -2210> 2022-01-17T22:16:42.302+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.185116291s of 10.000041008s, submitted: 26557
 -2169> 2022-01-17T22:16:52.304+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.564402580s of 10.002439499s, submitted: 29118
 -2155> 2022-01-17T22:16:55.317+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1517. Immutable memtables: 0.
 -2119> 2022-01-17T22:17:02.304+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.003684521s of 10.000174522s, submitted: 30827
 -2080> 2022-01-17T22:17:12.305+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.199123859s of 10.000152588s, submitted: 26934
 -2053> 2022-01-17T22:17:18.397+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1519. Immutable memtables: 0.
 -2029> 2022-01-17T22:17:22.305+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.184415817s of 10.000242233s, submitted: 26777
 -1989> 2022-01-17T22:17:32.306+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.851082325s of 10.000052452s, submitted: 28557
 -1981> 2022-01-17T22:17:33.772+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1507. Total log size is 1073744900 while max_total_wal_size is 1073741824
 -1980> 2022-01-17T22:17:33.773+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1521. Immutable memtables: 0.
 -1979> 2022-01-17T22:17:33.773+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1521. Immutable memtables: 0.
 -1978> 2022-01-17T22:17:33.773+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1521. Immutable memtables: 0.
 -1977> 2022-01-17T22:17:33.773+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1521. Immutable memtables: 0.
 -1976> 2022-01-17T22:17:33.773+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1521. Immutable memtables: 0.
 -1770> 2022-01-17T22:17:41.368+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1555. Immutable memtables: 0.
 -1749> 2022-01-17T22:17:42.306+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.117579460s of 10.000198364s, submitted: 29013
 -1688> 2022-01-17T22:17:52.307+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.704406738s of 10.000500679s, submitted: 28234
 -1649> 2022-01-17T22:18:02.307+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.409285069s of 10.000104904s, submitted: 30374
 -1645> 2022-01-17T22:18:03.229+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1562. Immutable memtables: 0.
 -1585> 2022-01-17T22:18:12.307+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.343633175s of 10.000135422s, submitted: 29888
 -1545> 2022-01-17T22:18:22.307+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.203266621s of 10.000041008s, submitted: 26859
 -1527> 2022-01-17T22:18:26.800+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1565. Immutable memtables: 0.
 -1494> 2022-01-17T22:18:32.306+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.906566143s of 10.000028610s, submitted: 21825
 -1457> 2022-01-17T22:18:42.069+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1521. Total log size is 1073760304 while max_total_wal_size is 1073741824
 -1456> 2022-01-17T22:18:42.069+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1567. Immutable memtables: 0.
 -1455> 2022-01-17T22:18:42.069+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1567. Immutable memtables: 0.
 -1454> 2022-01-17T22:18:42.070+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1567. Immutable memtables: 0.
 -1453> 2022-01-17T22:18:42.070+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1567. Immutable memtables: 0.
 -1452> 2022-01-17T22:18:42.070+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1567. Immutable memtables: 0.
 -1431> 2022-01-17T22:18:42.308+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.714018345s of 10.001763344s, submitted: 22451
 -1202> 2022-01-17T22:18:52.309+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.199633837s of 10.001363754s, submitted: 26410
 -1189> 2022-01-17T22:18:53.532+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1612. Immutable memtables: 0.
 -1077> 2022-01-17T22:19:02.309+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.848275900s of 10.000105858s, submitted: 25472
 -1037> 2022-01-17T22:19:12.309+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.313353539s of 10.000037193s, submitted: 25113
 -1009> 2022-01-17T22:19:18.610+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1623. Immutable memtables: 0.
  -985> 2022-01-17T22:19:22.310+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.796754360s of 10.000080109s, submitted: 26829
  -945> 2022-01-17T22:19:32.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.205605030s of 10.000051498s, submitted: 30637
  -910> 2022-01-17T22:19:40.810+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1625. Immutable memtables: 0.
  -880> 2022-01-17T22:19:42.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.918146133s of 10.000528336s, submitted: 27804
  -851> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1567. Total log size is 1073770331 while max_total_wal_size is 1073741824
  -850> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1628. Immutable memtables: 0.
  -849> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1628. Immutable memtables: 0.
  -848> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1628. Immutable memtables: 0.
  -847> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1628. Immutable memtables: 0.
  -846> 2022-01-17T22:19:49.250+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1628. Immutable memtables: 0.
  -783> 2022-01-17T22:19:52.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.964078903s of 10.000203133s, submitted: 27526
  -744> 2022-01-17T22:20:02.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.992430687s of 10.000035286s, submitted: 27137
  -730> 2022-01-17T22:20:05.356+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1634. Immutable memtables: 0.
  -693> 2022-01-17T22:20:12.312+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.595205784s of 10.000067711s, submitted: 24924
  -651> 2022-01-17T22:20:22.312+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.253684044s of 10.000055313s, submitted: 26042
  -616> 2022-01-17T22:20:30.501+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1636. Immutable memtables: 0.
  -599> 2022-01-17T22:20:32.312+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.450284481s of 10.000133514s, submitted: 28891
  -559> 2022-01-17T22:20:42.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.229814053s of 10.000176430s, submitted: 30007
  -518> 2022-01-17T22:20:52.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.750452280s of 10.000203133s, submitted: 26781
  -514> 2022-01-17T22:20:53.534+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1638. Immutable memtables: 0.
  -486> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1628. Total log size is 1073746874 while max_total_wal_size is 1073741824
  -485> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1640. Immutable memtables: 0.
  -484> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1640. Immutable memtables: 0.
  -483> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1640. Immutable memtables: 0.
  -482> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1640. Immutable memtables: 0.
  -481> 2022-01-17T22:20:57.851+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1640. Immutable memtables: 0.
  -398> 2022-01-17T22:21:02.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.039410591s of 10.000135422s, submitted: 25658
  -358> 2022-01-17T22:21:12.311+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.254876137s of 10.000082970s, submitted: 27074
  -331> 2022-01-17T22:21:18.611+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1647. Immutable memtables: 0.
  -289> 2022-01-17T22:21:22.312+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.012214661s of 10.000073433s, submitted: 27295
  -243> 2022-01-17T22:21:32.312+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.111629486s of 10.000324249s, submitted: 27185
  -205> 2022-01-17T22:21:42.313+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.228207588s of 10.000223160s, submitted: 26464
  -202> 2022-01-17T22:21:43.106+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1650. Immutable memtables: 0.
  -152> 2022-01-17T22:21:52.316+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.147776604s of 10.002803802s, submitted: 21915
  -112> 2022-01-17T22:22:02.316+0000 7f2bd48ec700  5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.276202679s of 10.000174522s, submitted: 25334
   -78> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1640. Total log size is 1073768775 while max_total_wal_size is 1073741824
   -77> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1652. Immutable memtables: 0.
   -76> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1652. Immutable memtables: 0.
   -75> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1652. Immutable memtables: 0.
   -74> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1652. Immutable memtables: 0.
   -73> 2022-01-17T22:22:10.462+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1652. Immutable memtables: 0.
   -64> 2022-01-17T22:22:10.480+0000 7f2bd48ec700  4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1654. Immutable memtables: 0.
    -1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000
     0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) **
 in thread 7f2bd48ec700 thread_name:bstore_kv_sync
Actions
Copy link
 #3

Updated by Neha Ojha over 2 years ago

  • Status changed from New to Duplicate
Actions
Copy link
 #4

Updated by Neha Ojha over 2 years ago

  • Is duplicate of Bug #53907: BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length) added
Actions
Copy link
 #5

Updated by Adam Kupczyk about 2 years ago

  • Pull request ID set to 44713
Actions
Copy link

Also available in: Atom PDF