Actions
Bug #53906
closedBlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size)
Added by Vikhyat Umrao over 2 years ago. Updated about 2 years ago.
Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
-1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: 4158: FAILED ceph_assert(cur >= fnode.size)
ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x56347eb33b8e]
2: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf]
3: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00]
4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009]
6: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de]
7: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8]
8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f]
9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22]
10: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8]
11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701]
12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454]
13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1]
14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84]
15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a]
16: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390]
17: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1]
18: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a]
19: clone()
0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) **
in thread 7f2bd48ec700 thread_name:bstore_kv_sync
ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
1: /lib64/libpthread.so.0(+0x12c20) [0x7f2beb318c20]
2: gsignal()
3: abort()
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56347eb33bec]
5: /usr/bin/ceph-osd(+0x5d5daf) [0x56347eb33daf]
6: (RocksDBBlueFSVolumeSelector::add_usage(void*, bluefs_fnode_t const&)+0) [0x56347f1f7d00]
7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x56347f295b45]
8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x56347f296009]
9: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x56347f2b16de]
10: (BlueRocksWritableFile::Sync()+0x18) [0x56347f2c1fb8]
11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x56347f7e574f]
12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x56347f915f22]
13: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x56347f9178e8]
14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x56347f7fc701]
15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x56347f804454]
16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x56347f8045a1]
17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x56347f79ef84]
18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x56347f79f98a]
19: (BlueStore::_kv_sync_thread()+0x3530) [0x56347f1f6390]
20: (BlueStore::KVSyncThread::entry()+0x11) [0x56347f2250b1]
21: /lib64/libpthread.so.0(+0x817a) [0x7f2beb30e17a]
22: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Vikhyat Umrao over 2 years ago
- Looks like the systemd restarted the OSD container pod after hitting this crash and after the restart, OSD is running fine!
Updated by Vikhyat Umrao over 2 years ago
OSD.138 logs for thread id - 7f2bd48ec700:
-7348> 2022-01-17T22:02:20.534+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1229. Immutable memtables: 0.
-7247> 2022-01-17T22:02:30.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.871898651s of 10.000083923s, submitted: 36357
-7237> 2022-01-17T22:02:32.410+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1236. Immutable memtables: 0.
-7196> 2022-01-17T22:02:40.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.393852472s of 10.000052452s, submitted: 34450
-7168> 2022-01-17T22:02:46.790+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1238. Immutable memtables: 0.
-7145> 2022-01-17T22:02:50.137+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.345420837s of 10.000248909s, submitted: 33896
-7106> 2022-01-17T22:03:00.138+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.035129309s of 10.000363350s, submitted: 35595
-7099> 2022-01-17T22:03:01.211+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1240. Immutable memtables: 0.
-7069> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1229. Total log size is 1073743393 while max_total_wal_size is 1073741824
-7068> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1243. Immutable memtables: 0.
-7067> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1243. Immutable memtables: 0.
-7066> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1243. Immutable memtables: 0.
-7065> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1243. Immutable memtables: 0.
-7064> 2022-01-17T22:03:03.228+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1243. Immutable memtables: 0.
-6924> 2022-01-17T22:03:10.139+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 1.422380924s of 10.001676559s, submitted: 36202
-6839> 2022-01-17T22:03:15.121+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1281. Immutable memtables: 0.
-6803> 2022-01-17T22:03:20.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.232607126s of 10.000235558s, submitted: 36495
-6767> 2022-01-17T22:03:29.197+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1283. Immutable memtables: 0.
-6752> 2022-01-17T22:03:30.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.836078882s of 10.000127792s, submitted: 37565
-6713> 2022-01-17T22:03:40.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.808803082s of 10.000279427s, submitted: 36784
-6699> 2022-01-17T22:03:43.174+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1285. Immutable memtables: 0.
-6685> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1243. Total log size is 1073744078 while max_total_wal_size is 1073741824
-6684> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1287. Immutable memtables: 0.
-6683> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1287. Immutable memtables: 0.
-6682> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1287. Immutable memtables: 0.
-6681> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1287. Immutable memtables: 0.
-6680> 2022-01-17T22:03:44.608+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1287. Immutable memtables: 0.
-6532> 2022-01-17T22:03:50.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.153817177s of 10.000497818s, submitted: 35734
-6373> 2022-01-17T22:03:58.888+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1342. Immutable memtables: 0.
-6314> 2022-01-17T22:04:00.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.471657276s of 10.000211716s, submitted: 31636
-6274> 2022-01-17T22:04:10.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.041631460s of 10.000046730s, submitted: 35494
-6256> 2022-01-17T22:04:14.881+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1348. Immutable memtables: 0.
-6222> 2022-01-17T22:04:20.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.624328852s of 10.000032425s, submitted: 33670
-6184> 2022-01-17T22:04:30.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.294847488s of 10.000163078s, submitted: 35205
-6182> 2022-01-17T22:04:30.338+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1350. Immutable memtables: 0.
-6166> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1287. Total log size is 1073743362 while max_total_wal_size is 1073741824
-6165> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1352. Immutable memtables: 0.
-6164> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1352. Immutable memtables: 0.
-6163> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1352. Immutable memtables: 0.
-6162> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1352. Immutable memtables: 0.
-6161> 2022-01-17T22:04:31.109+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1352. Immutable memtables: 0.
-6074> 2022-01-17T22:04:40.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.215115070s of 10.000009537s, submitted: 31125
-6037> 2022-01-17T22:04:50.141+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.136115074s of 10.000347137s, submitted: 21522
-6006> 2022-01-17T22:04:56.481+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1358. Immutable memtables: 0.
-5983> 2022-01-17T22:05:00.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.058140278s of 10.000130653s, submitted: 22103
-5944> 2022-01-17T22:05:10.142+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.180792332s of 10.000183105s, submitted: 26826
-5905> 2022-01-17T22:05:20.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.412753105s of 10.000147820s, submitted: 25214
-5892> 2022-01-17T22:05:22.452+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1360. Immutable memtables: 0.
-5840> 2022-01-17T22:05:30.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.213577747s of 10.000043869s, submitted: 20253
-5798> 2022-01-17T22:05:40.143+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.740430355s of 10.000234604s, submitted: 18681
-5759> 2022-01-17T22:05:50.144+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.200869083s of 10.000736237s, submitted: 21460
-5742> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1352. Total log size is 1073746833 while max_total_wal_size is 1073741824
-5741> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1363. Immutable memtables: 0.
-5740> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1363. Immutable memtables: 0.
-5739> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1363. Immutable memtables: 0.
-5738> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1363. Immutable memtables: 0.
-5737> 2022-01-17T22:05:53.779+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1363. Immutable memtables: 0.
-5653> 2022-01-17T22:05:59.071+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1370. Immutable memtables: 0.
-5636> 2022-01-17T22:06:00.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.126998901s of 10.001685143s, submitted: 21063
-5594> 2022-01-17T22:06:10.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.532536030s of 10.000311852s, submitted: 20146
-5552> 2022-01-17T22:06:20.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.273554325s of 10.000081062s, submitted: 16015
-5506> 2022-01-17T22:06:30.146+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.303408146s of 10.000711441s, submitted: 15952
-5466> 2022-01-17T22:06:40.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.279325008s of 10.002703667s, submitted: 15080
-5447> 2022-01-17T22:06:44.362+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1372. Immutable memtables: 0.
-5414> 2022-01-17T22:06:50.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.105763912s of 10.000024796s, submitted: 16855
-5373> 2022-01-17T22:07:00.149+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.872070789s of 10.000138283s, submitted: 18359
-5334> 2022-01-17T22:07:10.150+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.261459351s of 10.000664711s, submitted: 16004
-5295> 2022-01-17T22:07:20.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.133035183s of 10.000041008s, submitted: 16842
-5256> 2022-01-17T22:07:29.434+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1374. Immutable memtables: 0.
-5244> 2022-01-17T22:07:30.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.163985252s of 10.000092506s, submitted: 16358
-5203> 2022-01-17T22:07:40.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.522592545s of 10.000003815s, submitted: 14237
-5163> 2022-01-17T22:07:50.151+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.391773224s of 10.000718117s, submitted: 14934
-5122> 2022-01-17T22:08:00.152+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.567738056s of 10.000141144s, submitted: 13861
-5112> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1363. Total log size is 1073753040 while max_total_wal_size is 1073741824
-5111> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1376. Immutable memtables: 0.
-5110> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1376. Immutable memtables: 0.
-5109> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1376. Immutable memtables: 0.
-5108> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1376. Immutable memtables: 0.
-5107> 2022-01-17T22:08:01.872+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1376. Immutable memtables: 0.
-4919> 2022-01-17T22:08:10.152+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.072527409s of 10.000085831s, submitted: 13343
-4879> 2022-01-17T22:08:20.155+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.948874950s of 10.003495216s, submitted: 11787
-4849> 2022-01-17T22:08:26.958+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1406. Immutable memtables: 0.
-4813> 2022-01-17T22:08:30.155+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.690354824s of 10.000430107s, submitted: 13232
-4774> 2022-01-17T22:08:40.156+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.019449234s of 10.001420021s, submitted: 11662
-4733> 2022-01-17T22:08:50.158+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.823121071s of 10.001865387s, submitted: 12609
-4692> 2022-01-17T22:09:00.159+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.787741661s of 10.000707626s, submitted: 12718
-4652> 2022-01-17T22:09:10.159+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.782723427s of 10.000552177s, submitted: 12703
-4610> 2022-01-17T22:09:20.161+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.186806679s of 10.001282692s, submitted: 10587
-4570> 2022-01-17T22:09:30.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.969998360s of 10.000073433s, submitted: 11710
-4556> 2022-01-17T22:09:32.486+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1409. Immutable memtables: 0.
-4520> 2022-01-17T22:09:40.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.065218925s of 10.000068665s, submitted: 11141
-4480> 2022-01-17T22:09:50.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.326082230s of 10.000044823s, submitted: 9633
-4438> 2022-01-17T22:10:00.162+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.749199390s of 10.000402451s, submitted: 13089
-4399> 2022-01-17T22:10:10.165+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.939408302s of 10.002837181s, submitted: 11599
-4359> 2022-01-17T22:10:20.167+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.809356213s of 10.002017021s, submitted: 12853
-4320> 2022-01-17T22:10:30.166+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.689060688s of 10.000018120s, submitted: 13281
-4293> 2022-01-17T22:10:36.797+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1411. Immutable memtables: 0.
-4269> 2022-01-17T22:10:40.169+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.554021835s of 10.002296448s, submitted: 14190
-4226> 2022-01-17T22:10:50.173+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.807527542s of 10.004211426s, submitted: 12677
-4194> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1376. Total log size is 1073755738 while max_total_wal_size is 1073741824
-4193> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1413. Immutable memtables: 0.
-4192> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1413. Immutable memtables: 0.
-4191> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1413. Immutable memtables: 0.
-4190> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1413. Immutable memtables: 0.
-4189> 2022-01-17T22:10:57.760+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1413. Immutable memtables: 0.
-4093> 2022-01-17T22:11:00.174+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.839452267s of 10.000724792s, submitted: 10763
-3847> 2022-01-17T22:11:10.174+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.662105560s of 10.000135422s, submitted: 12848
-3731> 2022-01-17T22:11:20.176+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.784808159s of 10.002503395s, submitted: 10863
-3691> 2022-01-17T22:11:30.180+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.380425453s of 10.003009796s, submitted: 9487
-3651> 2022-01-17T22:11:40.181+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.965118408s of 10.000519753s, submitted: 11953
-3626> 2022-01-17T22:11:45.699+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1490. Immutable memtables: 0.
-3593> 2022-01-17T22:11:50.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.822191238s of 10.000876427s, submitted: 12874
-3553> 2022-01-17T22:12:00.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.168907166s of 10.000414848s, submitted: 10564
-3513> 2022-01-17T22:12:10.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.079604149s of 10.000261307s, submitted: 11301
-3474> 2022-01-17T22:12:20.183+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.842392445s of 10.000760078s, submitted: 12579
-3434> 2022-01-17T22:12:30.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.133393288s of 10.000002861s, submitted: 10912
-3392> 2022-01-17T22:12:40.182+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.905469418s of 10.000060081s, submitted: 11891
-3352> 2022-01-17T22:12:50.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.832632542s of 10.001495361s, submitted: 11825
-3337> 2022-01-17T22:12:53.021+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1492. Immutable memtables: 0.
-3286> 2022-01-17T22:13:00.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.737954617s of 10.000348091s, submitted: 13159
-3246> 2022-01-17T22:13:10.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.913400173s of 10.000087738s, submitted: 12216
-3205> 2022-01-17T22:13:20.184+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.016359329s of 10.000087738s, submitted: 11451
-3164> 2022-01-17T22:13:30.186+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.681961536s of 10.000099182s, submitted: 13345
-3121> 2022-01-17T22:13:40.188+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.648798943s of 10.002541542s, submitted: 13670
-3081> 2022-01-17T22:13:50.188+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.625751972s of 10.000112534s, submitted: 13870
-3064> 2022-01-17T22:13:54.171+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1495. Immutable memtables: 0.
-3044> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1413. Total log size is 1073747433 while max_total_wal_size is 1073741824
-3043> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1497. Immutable memtables: 0.
-3042> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1497. Immutable memtables: 0.
-3041> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1497. Immutable memtables: 0.
-3040> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1497. Immutable memtables: 0.
-3039> 2022-01-17T22:13:56.590+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1497. Immutable memtables: 0.
-2973> 2022-01-17T22:14:00.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.800518036s of 10.003147125s, submitted: 12444
-2934> 2022-01-17T22:14:10.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.312356949s of 10.000099182s, submitted: 10023
-2895> 2022-01-17T22:14:20.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.982282639s of 10.000566483s, submitted: 11902
-2857> 2022-01-17T22:14:30.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.601618767s of 10.000000954s, submitted: 13930
-2815> 2022-01-17T22:14:40.191+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 7.767084122s of 10.000088692s, submitted: 13370
-2776> 2022-01-17T22:14:50.192+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.071086884s of 10.000385284s, submitted: 11217
-2735> 2022-01-17T22:15:00.193+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 8.009041786s of 10.001303673s, submitted: 11630
-2567> 2022-01-17T22:15:42.301+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 42.061779022s of 42.107433319s, submitted: 210
-2565> 2022-01-17T22:15:42.610+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1503. Immutable memtables: 0.
-2516> 2022-01-17T22:15:52.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.030773640s of 10.000231743s, submitted: 27813
-2477> 2022-01-17T22:16:02.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.171396255s of 10.000287056s, submitted: 22045
-2446> 2022-01-17T22:16:09.237+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1505. Immutable memtables: 0.
-2426> 2022-01-17T22:16:12.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.264243126s of 10.000252724s, submitted: 26308
-2386> 2022-01-17T22:16:22.303+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.664372921s of 10.000242233s, submitted: 29573
-2359> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1497. Total log size is 1073747405 while max_total_wal_size is 1073741824
-2358> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1507. Immutable memtables: 0.
-2357> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1507. Immutable memtables: 0.
-2356> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1507. Immutable memtables: 0.
-2355> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1507. Immutable memtables: 0.
-2354> 2022-01-17T22:16:28.244+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1507. Immutable memtables: 0.
-2278> 2022-01-17T22:16:32.187+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1514. Immutable memtables: 0.
-2271> 2022-01-17T22:16:32.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.312994480s of 10.000098228s, submitted: 28693
-2210> 2022-01-17T22:16:42.302+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.185116291s of 10.000041008s, submitted: 26557
-2169> 2022-01-17T22:16:52.304+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.564402580s of 10.002439499s, submitted: 29118
-2155> 2022-01-17T22:16:55.317+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1517. Immutable memtables: 0.
-2119> 2022-01-17T22:17:02.304+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.003684521s of 10.000174522s, submitted: 30827
-2080> 2022-01-17T22:17:12.305+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.199123859s of 10.000152588s, submitted: 26934
-2053> 2022-01-17T22:17:18.397+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1519. Immutable memtables: 0.
-2029> 2022-01-17T22:17:22.305+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.184415817s of 10.000242233s, submitted: 26777
-1989> 2022-01-17T22:17:32.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.851082325s of 10.000052452s, submitted: 28557
-1981> 2022-01-17T22:17:33.772+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1507. Total log size is 1073744900 while max_total_wal_size is 1073741824
-1980> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1521. Immutable memtables: 0.
-1979> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1521. Immutable memtables: 0.
-1978> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1521. Immutable memtables: 0.
-1977> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1521. Immutable memtables: 0.
-1976> 2022-01-17T22:17:33.773+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1521. Immutable memtables: 0.
-1770> 2022-01-17T22:17:41.368+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1555. Immutable memtables: 0.
-1749> 2022-01-17T22:17:42.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.117579460s of 10.000198364s, submitted: 29013
-1688> 2022-01-17T22:17:52.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.704406738s of 10.000500679s, submitted: 28234
-1649> 2022-01-17T22:18:02.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.409285069s of 10.000104904s, submitted: 30374
-1645> 2022-01-17T22:18:03.229+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1562. Immutable memtables: 0.
-1585> 2022-01-17T22:18:12.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.343633175s of 10.000135422s, submitted: 29888
-1545> 2022-01-17T22:18:22.307+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.203266621s of 10.000041008s, submitted: 26859
-1527> 2022-01-17T22:18:26.800+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1565. Immutable memtables: 0.
-1494> 2022-01-17T22:18:32.306+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.906566143s of 10.000028610s, submitted: 21825
-1457> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1521. Total log size is 1073760304 while max_total_wal_size is 1073741824
-1456> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1567. Immutable memtables: 0.
-1455> 2022-01-17T22:18:42.069+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1567. Immutable memtables: 0.
-1454> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1567. Immutable memtables: 0.
-1453> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1567. Immutable memtables: 0.
-1452> 2022-01-17T22:18:42.070+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1567. Immutable memtables: 0.
-1431> 2022-01-17T22:18:42.308+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.714018345s of 10.001763344s, submitted: 22451
-1202> 2022-01-17T22:18:52.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 2.199633837s of 10.001363754s, submitted: 26410
-1189> 2022-01-17T22:18:53.532+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1612. Immutable memtables: 0.
-1077> 2022-01-17T22:19:02.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.848275900s of 10.000105858s, submitted: 25472
-1037> 2022-01-17T22:19:12.309+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.313353539s of 10.000037193s, submitted: 25113
-1009> 2022-01-17T22:19:18.610+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1623. Immutable memtables: 0.
-985> 2022-01-17T22:19:22.310+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.796754360s of 10.000080109s, submitted: 26829
-945> 2022-01-17T22:19:32.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.205605030s of 10.000051498s, submitted: 30637
-910> 2022-01-17T22:19:40.810+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1625. Immutable memtables: 0.
-880> 2022-01-17T22:19:42.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.918146133s of 10.000528336s, submitted: 27804
-851> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1567. Total log size is 1073770331 while max_total_wal_size is 1073741824
-850> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1628. Immutable memtables: 0.
-849> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1628. Immutable memtables: 0.
-848> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1628. Immutable memtables: 0.
-847> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1628. Immutable memtables: 0.
-846> 2022-01-17T22:19:49.250+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1628. Immutable memtables: 0.
-783> 2022-01-17T22:19:52.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.964078903s of 10.000203133s, submitted: 27526
-744> 2022-01-17T22:20:02.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.992430687s of 10.000035286s, submitted: 27137
-730> 2022-01-17T22:20:05.356+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1634. Immutable memtables: 0.
-693> 2022-01-17T22:20:12.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.595205784s of 10.000067711s, submitted: 24924
-651> 2022-01-17T22:20:22.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.253684044s of 10.000055313s, submitted: 26042
-616> 2022-01-17T22:20:30.501+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1636. Immutable memtables: 0.
-599> 2022-01-17T22:20:32.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.450284481s of 10.000133514s, submitted: 28891
-559> 2022-01-17T22:20:42.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 4.229814053s of 10.000176430s, submitted: 30007
-518> 2022-01-17T22:20:52.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 3.750452280s of 10.000203133s, submitted: 26781
-514> 2022-01-17T22:20:53.534+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1638. Immutable memtables: 0.
-486> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1628. Total log size is 1073746874 while max_total_wal_size is 1073741824
-485> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1640. Immutable memtables: 0.
-484> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1640. Immutable memtables: 0.
-483> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1640. Immutable memtables: 0.
-482> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1640. Immutable memtables: 0.
-481> 2022-01-17T22:20:57.851+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1640. Immutable memtables: 0.
-398> 2022-01-17T22:21:02.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.039410591s of 10.000135422s, submitted: 25658
-358> 2022-01-17T22:21:12.311+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.254876137s of 10.000082970s, submitted: 27074
-331> 2022-01-17T22:21:18.611+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1647. Immutable memtables: 0.
-289> 2022-01-17T22:21:22.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.012214661s of 10.000073433s, submitted: 27295
-243> 2022-01-17T22:21:32.312+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.111629486s of 10.000324249s, submitted: 27185
-205> 2022-01-17T22:21:42.313+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.228207588s of 10.000223160s, submitted: 26464
-202> 2022-01-17T22:21:43.106+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1650. Immutable memtables: 0.
-152> 2022-01-17T22:21:52.316+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 6.147776604s of 10.002803802s, submitted: 21915
-112> 2022-01-17T22:22:02.316+0000 7f2bd48ec700 5 bluestore(/var/lib/ceph/osd/ceph-138) _kv_sync_thread utilization: idle 5.276202679s of 10.000174522s, submitted: 25334
-78> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1298] Flushing all column families with data in WAL number 1640. Total log size is 1073768775 while max_total_wal_size is 1073741824
-77> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [default] New memtable created with log file: #1652. Immutable memtables: 0.
-76> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-0] New memtable created with log file: #1652. Immutable memtables: 0.
-75> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-1] New memtable created with log file: #1652. Immutable memtables: 0.
-74> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [O-2] New memtable created with log file: #1652. Immutable memtables: 0.
-73> 2022-01-17T22:22:10.462+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [P] New memtable created with log file: #1652. Immutable memtables: 0.
-64> 2022-01-17T22:22:10.480+0000 7f2bd48ec700 4 rocksdb: [db/db_impl/db_impl_write.cc:1739] [L] New memtable created with log file: #1654. Immutable memtables: 0.
-1> 2022-01-17T22:22:11.334+0000 7f2bd48ec700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f2bd48ec700 time 2022-01-17T22:22:11.323713+0000
0> 2022-01-17T22:22:11.369+0000 7f2bd48ec700 -1 *** Caught signal (Aborted) **
in thread 7f2bd48ec700 thread_name:bstore_kv_sync
Updated by Neha Ojha over 2 years ago
- Is duplicate of Bug #53907: BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length) added
Actions