Actions
Bug #53906
closedBlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size)
Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
-1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size) ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x56347eb33b8e] 2: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] 3: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00] 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45] 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009] 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de] 7: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8] 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f] 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22] 10: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8] 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701] 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454] 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1] 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84] 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a] 16: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390] 17: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1] 18: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a] 19: clone() 0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) ** in thread 7f2bd48ec700 thread_name:bstore_kv_sync ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec] 5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf] 6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00] 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45] 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009] 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de] 10: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8] 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f] 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22] 13: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8] 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701] 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454] 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1] 17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84] 18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a] 19: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390] 20: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1] 21: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a] 22: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Vikhyat Umrao over 2 years ago
- Looks like the systemd restarted the OSD container pod after hitting this crash and after the restart, OSD is running fine!
Updated by Vikhyat Umrao over 2 years ago
OSD.138 logs for thread id - 7f2bd48ec700:
-7348> 2022-01-17T22:02:20.534+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1229. Immutable memtables: 0. -7247> 2022-01-17T22:02:30.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.871898651s of 10.000083923s, submitted: 36357 -7237> 2022-01-17T22:02:32.410+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1236. Immutable memtables: 0. -7196> 2022-01-17T22:02:40.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.393852472s of 10.000052452s, submitted: 34450 -7168> 2022-01-17T22:02:46.790+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1238. Immutable memtables: 0. -7145> 2022-01-17T22:02:50.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.345420837s of 10.000248909s, submitted: 33896 -7106> 2022-01-17T22:03:00.138+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.035129309s of 10.000363350s, submitted: 35595 -7099> 2022-01-17T22:03:01.211+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1240. Immutable memtables: 0. -7069> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1229. Total log size is 1073743393 while max_total_wal_size is 1073741824 -7068> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1243. Immutable memtables: 0. -7067> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1243. Immutable memtables: 0. -7066> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1243. Immutable memtables: 0. -7065> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1243. Immutable memtables: 0. -7064> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1243. Immutable memtables: 0. -6924> 2022-01-17T22:03:10.139+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 1.422380924s of 10.001676559s, submitted: 36202 -6839> 2022-01-17T22:03:15.121+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1281. Immutable memtables: 0. -6803> 2022-01-17T22:03:20.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.232607126s of 10.000235558s, submitted: 36495 -6767> 2022-01-17T22:03:29.197+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1283. Immutable memtables: 0. -6752> 2022-01-17T22:03:30.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.836078882s of 10.000127792s, submitted: 37565 -6713> 2022-01-17T22:03:40.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.808803082s of 10.000279427s, submitted: 36784 -6699> 2022-01-17T22:03:43.174+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1285. Immutable memtables: 0. -6685> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1243. Total log size is 1073744078 while max_total_wal_size is 1073741824 -6684> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1287. Immutable memtables: 0. -6683> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1287. Immutable memtables: 0. -6682> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1287. Immutable memtables: 0. -6681> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1287. Immutable memtables: 0. -6680> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1287. Immutable memtables: 0. -6532> 2022-01-17T22:03:50.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.153817177s of 10.000497818s, submitted: 35734 -6373> 2022-01-17T22:03:58.888+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1342. Immutable memtables: 0. -6314> 2022-01-17T22:04:00.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.471657276s of 10.000211716s, submitted: 31636 -6274> 2022-01-17T22:04:10.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.041631460s of 10.000046730s, submitted: 35494 -6256> 2022-01-17T22:04:14.881+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1348. Immutable memtables: 0. -6222> 2022-01-17T22:04:20.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.624328852s of 10.000032425s, submitted: 33670 -6184> 2022-01-17T22:04:30.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.294847488s of 10.000163078s, submitted: 35205 -6182> 2022-01-17T22:04:30.338+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1350. Immutable memtables: 0. -6166> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1287. Total log size is 1073743362 while max_total_wal_size is 1073741824 -6165> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1352. Immutable memtables: 0. -6164> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1352. Immutable memtables: 0. -6163> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1352. Immutable memtables: 0. -6162> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1352. Immutable memtables: 0. -6161> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1352. Immutable memtables: 0. -6074> 2022-01-17T22:04:40.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.215115070s of 10.000009537s, submitted: 31125 -6037> 2022-01-17T22:04:50.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.136115074s of 10.000347137s, submitted: 21522 -6006> 2022-01-17T22:04:56.481+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1358. Immutable memtables: 0. -5983> 2022-01-17T22:05:00.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.058140278s of 10.000130653s, submitted: 22103 -5944> 2022-01-17T22:05:10.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.180792332s of 10.000183105s, submitted: 26826 -5905> 2022-01-17T22:05:20.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.412753105s of 10.000147820s, submitted: 25214 -5892> 2022-01-17T22:05:22.452+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1360. Immutable memtables: 0. -5840> 2022-01-17T22:05:30.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.213577747s of 10.000043869s, submitted: 20253 -5798> 2022-01-17T22:05:40.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.740430355s of 10.000234604s, submitted: 18681 -5759> 2022-01-17T22:05:50.144+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.200869083s of 10.000736237s, submitted: 21460 -5742> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1352. Total log size is 1073746833 while max_total_wal_size is 1073741824 -5741> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1363. Immutable memtables: 0. -5740> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1363. Immutable memtables: 0. -5739> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1363. Immutable memtables: 0. -5738> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1363. Immutable memtables: 0. -5737> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1363. Immutable memtables: 0. -5653> 2022-01-17T22:05:59.071+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1370. Immutable memtables: 0. -5636> 2022-01-17T22:06:00.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.126998901s of 10.001685143s, submitted: 21063 -5594> 2022-01-17T22:06:10.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.532536030s of 10.000311852s, submitted: 20146 -5552> 2022-01-17T22:06:20.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.273554325s of 10.000081062s, submitted: 16015 -5506> 2022-01-17T22:06:30.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.303408146s of 10.000711441s, submitted: 15952 -5466> 2022-01-17T22:06:40.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.279325008s of 10.002703667s, submitted: 15080 -5447> 2022-01-17T22:06:44.362+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1372. Immutable memtables: 0. -5414> 2022-01-17T22:06:50.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.105763912s of 10.000024796s, submitted: 16855 -5373> 2022-01-17T22:07:00.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.872070789s of 10.000138283s, submitted: 18359 -5334> 2022-01-17T22:07:10.150+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.261459351s of 10.000664711s, submitted: 16004 -5295> 2022-01-17T22:07:20.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.133035183s of 10.000041008s, submitted: 16842 -5256> 2022-01-17T22:07:29.434+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1374. Immutable memtables: 0. -5244> 2022-01-17T22:07:30.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.163985252s of 10.000092506s, submitted: 16358 -5203> 2022-01-17T22:07:40.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.522592545s of 10.000003815s, submitted: 14237 -5163> 2022-01-17T22:07:50.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.391773224s of 10.000718117s, submitted: 14934 -5122> 2022-01-17T22:08:00.152+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.567738056s of 10.000141144s, submitted: 13861 -5112> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1363. Total log size is 1073753040 while max_total_wal_size is 1073741824 -5111> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1376. Immutable memtables: 0. -5110> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1376. Immutable memtables: 0. -5109> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1376. Immutable memtables: 0. -5108> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1376. Immutable memtables: 0. -5107> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1376. Immutable memtables: 0. -4919> 2022-01-17T22:08:10.152+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.072527409s of 10.000085831s, submitted: 13343 -4879> 2022-01-17T22:08:20.155+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.948874950s of 10.003495216s, submitted: 11787 -4849> 2022-01-17T22:08:26.958+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1406. Immutable memtables: 0. -4813> 2022-01-17T22:08:30.155+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.690354824s of 10.000430107s, submitted: 13232 -4774> 2022-01-17T22:08:40.156+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.019449234s of 10.001420021s, submitted: 11662 -4733> 2022-01-17T22:08:50.158+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.823121071s of 10.001865387s, submitted: 12609 -4692> 2022-01-17T22:09:00.159+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.787741661s of 10.000707626s, submitted: 12718 -4652> 2022-01-17T22:09:10.159+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.782723427s of 10.000552177s, submitted: 12703 -4610> 2022-01-17T22:09:20.161+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.186806679s of 10.001282692s, submitted: 10587 -4570> 2022-01-17T22:09:30.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.969998360s of 10.000073433s, submitted: 11710 -4556> 2022-01-17T22:09:32.486+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1409. Immutable memtables: 0. -4520> 2022-01-17T22:09:40.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.065218925s of 10.000068665s, submitted: 11141 -4480> 2022-01-17T22:09:50.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.326082230s of 10.000044823s, submitted: 9633 -4438> 2022-01-17T22:10:00.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.749199390s of 10.000402451s, submitted: 13089 -4399> 2022-01-17T22:10:10.165+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.939408302s of 10.002837181s, submitted: 11599 -4359> 2022-01-17T22:10:20.167+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.809356213s of 10.002017021s, submitted: 12853 -4320> 2022-01-17T22:10:30.166+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.689060688s of 10.000018120s, submitted: 13281 -4293> 2022-01-17T22:10:36.797+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1411. Immutable memtables: 0. -4269> 2022-01-17T22:10:40.169+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.554021835s of 10.002296448s, submitted: 14190 -4226> 2022-01-17T22:10:50.173+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.807527542s of 10.004211426s, submitted: 12677 -4194> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1376. Total log size is 1073755738 while max_total_wal_size is 1073741824 -4193> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1413. Immutable memtables: 0. -4192> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1413. Immutable memtables: 0. -4191> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1413. Immutable memtables: 0. -4190> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1413. Immutable memtables: 0. -4189> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1413. Immutable memtables: 0. -4093> 2022-01-17T22:11:00.174+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.839452267s of 10.000724792s, submitted: 10763 -3847> 2022-01-17T22:11:10.174+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.662105560s of 10.000135422s, submitted: 12848 -3731> 2022-01-17T22:11:20.176+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.784808159s of 10.002503395s, submitted: 10863 -3691> 2022-01-17T22:11:30.180+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.380425453s of 10.003009796s, submitted: 9487 -3651> 2022-01-17T22:11:40.181+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.965118408s of 10.000519753s, submitted: 11953 -3626> 2022-01-17T22:11:45.699+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1490. Immutable memtables: 0. -3593> 2022-01-17T22:11:50.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.822191238s of 10.000876427s, submitted: 12874 -3553> 2022-01-17T22:12:00.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.168907166s of 10.000414848s, submitted: 10564 -3513> 2022-01-17T22:12:10.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.079604149s of 10.000261307s, submitted: 11301 -3474> 2022-01-17T22:12:20.183+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.842392445s of 10.000760078s, submitted: 12579 -3434> 2022-01-17T22:12:30.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.133393288s of 10.000002861s, submitted: 10912 -3392> 2022-01-17T22:12:40.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.905469418s of 10.000060081s, submitted: 11891 -3352> 2022-01-17T22:12:50.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.832632542s of 10.001495361s, submitted: 11825 -3337> 2022-01-17T22:12:53.021+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1492. Immutable memtables: 0. -3286> 2022-01-17T22:13:00.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.737954617s of 10.000348091s, submitted: 13159 -3246> 2022-01-17T22:13:10.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.913400173s of 10.000087738s, submitted: 12216 -3205> 2022-01-17T22:13:20.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.016359329s of 10.000087738s, submitted: 11451 -3164> 2022-01-17T22:13:30.186+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.681961536s of 10.000099182s, submitted: 13345 -3121> 2022-01-17T22:13:40.188+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.648798943s of 10.002541542s, submitted: 13670 -3081> 2022-01-17T22:13:50.188+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.625751972s of 10.000112534s, submitted: 13870 -3064> 2022-01-17T22:13:54.171+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1495. Immutable memtables: 0. -3044> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1413. Total log size is 1073747433 while max_total_wal_size is 1073741824 -3043> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1497. Immutable memtables: 0. -3042> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1497. Immutable memtables: 0. -3041> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1497. Immutable memtables: 0. -3040> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1497. Immutable memtables: 0. -3039> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1497. Immutable memtables: 0. -2973> 2022-01-17T22:14:00.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.800518036s of 10.003147125s, submitted: 12444 -2934> 2022-01-17T22:14:10.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.312356949s of 10.000099182s, submitted: 10023 -2895> 2022-01-17T22:14:20.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.982282639s of 10.000566483s, submitted: 11902 -2857> 2022-01-17T22:14:30.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.601618767s of 10.000000954s, submitted: 13930 -2815> 2022-01-17T22:14:40.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.767084122s of 10.000088692s, submitted: 13370 -2776> 2022-01-17T22:14:50.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.071086884s of 10.000385284s, submitted: 11217 -2735> 2022-01-17T22:15:00.193+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.009041786s of 10.001303673s, submitted: 11630 -2567> 2022-01-17T22:15:42.301+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 42.061779022s of 42.107433319s, submitted: 210 -2565> 2022-01-17T22:15:42.610+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1503. Immutable memtables: 0. -2516> 2022-01-17T22:15:52.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.030773640s of 10.000231743s, submitted: 27813 -2477> 2022-01-17T22:16:02.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.171396255s of 10.000287056s, submitted: 22045 -2446> 2022-01-17T22:16:09.237+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1505. Immutable memtables: 0. -2426> 2022-01-17T22:16:12.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.264243126s of 10.000252724s, submitted: 26308 -2386> 2022-01-17T22:16:22.303+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.664372921s of 10.000242233s, submitted: 29573 -2359> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1497. Total log size is 1073747405 while max_total_wal_size is 1073741824 -2358> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1507. Immutable memtables: 0. -2357> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1507. Immutable memtables: 0. -2356> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1507. Immutable memtables: 0. -2355> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1507. Immutable memtables: 0. -2354> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1507. Immutable memtables: 0. -2278> 2022-01-17T22:16:32.187+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1514. Immutable memtables: 0. -2271> 2022-01-17T22:16:32.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.312994480s of 10.000098228s, submitted: 28693 -2210> 2022-01-17T22:16:42.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.185116291s of 10.000041008s, submitted: 26557 -2169> 2022-01-17T22:16:52.304+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.564402580s of 10.002439499s, submitted: 29118 -2155> 2022-01-17T22:16:55.317+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1517. Immutable memtables: 0. -2119> 2022-01-17T22:17:02.304+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.003684521s of 10.000174522s, submitted: 30827 -2080> 2022-01-17T22:17:12.305+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.199123859s of 10.000152588s, submitted: 26934 -2053> 2022-01-17T22:17:18.397+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1519. Immutable memtables: 0. -2029> 2022-01-17T22:17:22.305+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.184415817s of 10.000242233s, submitted: 26777 -1989> 2022-01-17T22:17:32.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.851082325s of 10.000052452s, submitted: 28557 -1981> 2022-01-17T22:17:33.772+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1507. Total log size is 1073744900 while max_total_wal_size is 1073741824 -1980> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1521. Immutable memtables: 0. -1979> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1521. Immutable memtables: 0. -1978> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1521. Immutable memtables: 0. -1977> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1521. Immutable memtables: 0. -1976> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1521. Immutable memtables: 0. -1770> 2022-01-17T22:17:41.368+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1555. Immutable memtables: 0. -1749> 2022-01-17T22:17:42.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.117579460s of 10.000198364s, submitted: 29013 -1688> 2022-01-17T22:17:52.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.704406738s of 10.000500679s, submitted: 28234 -1649> 2022-01-17T22:18:02.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.409285069s of 10.000104904s, submitted: 30374 -1645> 2022-01-17T22:18:03.229+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1562. Immutable memtables: 0. -1585> 2022-01-17T22:18:12.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.343633175s of 10.000135422s, submitted: 29888 -1545> 2022-01-17T22:18:22.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.203266621s of 10.000041008s, submitted: 26859 -1527> 2022-01-17T22:18:26.800+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1565. Immutable memtables: 0. -1494> 2022-01-17T22:18:32.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.906566143s of 10.000028610s, submitted: 21825 -1457> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1521. Total log size is 1073760304 while max_total_wal_size is 1073741824 -1456> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1567. Immutable memtables: 0. -1455> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1567. Immutable memtables: 0. -1454> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1567. Immutable memtables: 0. -1453> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1567. Immutable memtables: 0. -1452> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1567. Immutable memtables: 0. -1431> 2022-01-17T22:18:42.308+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.714018345s of 10.001763344s, submitted: 22451 -1202> 2022-01-17T22:18:52.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.199633837s of 10.001363754s, submitted: 26410 -1189> 2022-01-17T22:18:53.532+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1612. Immutable memtables: 0. -1077> 2022-01-17T22:19:02.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.848275900s of 10.000105858s, submitted: 25472 -1037> 2022-01-17T22:19:12.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.313353539s of 10.000037193s, submitted: 25113 -1009> 2022-01-17T22:19:18.610+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1623. Immutable memtables: 0. -985> 2022-01-17T22:19:22.310+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.796754360s of 10.000080109s, submitted: 26829 -945> 2022-01-17T22:19:32.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.205605030s of 10.000051498s, submitted: 30637 -910> 2022-01-17T22:19:40.810+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1625. Immutable memtables: 0. -880> 2022-01-17T22:19:42.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.918146133s of 10.000528336s, submitted: 27804 -851> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1567. Total log size is 1073770331 while max_total_wal_size is 1073741824 -850> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1628. Immutable memtables: 0. -849> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1628. Immutable memtables: 0. -848> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1628. Immutable memtables: 0. -847> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1628. Immutable memtables: 0. -846> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1628. Immutable memtables: 0. -783> 2022-01-17T22:19:52.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.964078903s of 10.000203133s, submitted: 27526 -744> 2022-01-17T22:20:02.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.992430687s of 10.000035286s, submitted: 27137 -730> 2022-01-17T22:20:05.356+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1634. Immutable memtables: 0. -693> 2022-01-17T22:20:12.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.595205784s of 10.000067711s, submitted: 24924 -651> 2022-01-17T22:20:22.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.253684044s of 10.000055313s, submitted: 26042 -616> 2022-01-17T22:20:30.501+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1636. Immutable memtables: 0. -599> 2022-01-17T22:20:32.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.450284481s of 10.000133514s, submitted: 28891 -559> 2022-01-17T22:20:42.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.229814053s of 10.000176430s, submitted: 30007 -518> 2022-01-17T22:20:52.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.750452280s of 10.000203133s, submitted: 26781 -514> 2022-01-17T22:20:53.534+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1638. Immutable memtables: 0. -486> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1628. Total log size is 1073746874 while max_total_wal_size is 1073741824 -485> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1640. Immutable memtables: 0. -484> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1640. Immutable memtables: 0. -483> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1640. Immutable memtables: 0. -482> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1640. Immutable memtables: 0. -481> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1640. Immutable memtables: 0. -398> 2022-01-17T22:21:02.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.039410591s of 10.000135422s, submitted: 25658 -358> 2022-01-17T22:21:12.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.254876137s of 10.000082970s, submitted: 27074 -331> 2022-01-17T22:21:18.611+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1647. Immutable memtables: 0. -289> 2022-01-17T22:21:22.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.012214661s of 10.000073433s, submitted: 27295 -243> 2022-01-17T22:21:32.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.111629486s of 10.000324249s, submitted: 27185 -205> 2022-01-17T22:21:42.313+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.228207588s of 10.000223160s, submitted: 26464 -202> 2022-01-17T22:21:43.106+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1650. Immutable memtables: 0. -152> 2022-01-17T22:21:52.316+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.147776604s of 10.002803802s, submitted: 21915 -112> 2022-01-17T22:22:02.316+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.276202679s of 10.000174522s, submitted: 25334 -78> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1640. Total log size is 1073768775 while max_total_wal_size is 1073741824 -77> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1652. Immutable memtables: 0. -76> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1652. Immutable memtables: 0. -75> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1652. Immutable memtables: 0. -74> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1652. Immutable memtables: 0. -73> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1652. Immutable memtables: 0. -64> 2022-01-17T22:22:10.480+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1654. Immutable memtables: 0. -1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000 0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) ** in thread 7f2bd48ec700 thread_name:bstore_kv_sync
Updated by Neha Ojha over 2 years ago
- Is duplicate of Bug #53907: BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length) added
Actions