debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable), process ceph-osd, pid 1 debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 pidfile_write: ignore empty --pid-file debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:06.608+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:06.840+0000 7f86cebf1080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal debug 2021-11-15T12:49:06.840+0000 7f86cebf1080 -1 Falling back to public interface debug 2021-11-15T12:49:06.852+0000 7f86cebf1080 0 load: jerasure load: lrc load: isa debug 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:07.452+0000 7f86cebf1080 0 osd.2:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:07.752+0000 7f86cebf1080 0 osd.2:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:08.048+0000 7f86cebf1080 0 osd.2:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:08.344+0000 7f86cebf1080 0 osd.2:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:08.628+0000 7f86cebf1080 0 osd.2:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:08.920+0000 7f86cebf1080 0 osd.2:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:09.216+0000 7f86cebf1080 0 osd.2:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 osd.2:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _mount path /var/lib/ceph/osd/ceph-2 debug 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 bluestore(/var/lib/ceph/osd/ceph-2) _open_db_and_around read-only:0 repair:0 debug 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs mount debug 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 0 debug 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561fe20cb080 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561fe20cbb20 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea6ab0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe205e7b0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980549639149, "job": 1, "event": "recovery_started", "log_files": [33446]} debug 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug 2021-11-15T12:49:12.072+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980552076908, "job": 1, "event": "recovery_finished"} debug 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 debug 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old nid_max 5338942 debug 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old blobid_max 296960 debug 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta freelist_type bitmap debug 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 debug 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta min_alloc_size 0x1000 debug 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist init debug 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist _read_cfg debug 2021-11-15T12:49:12.120+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc opening allocation metadata debug 2021-11-15T12:49:12.952+0000 7f86cebf1080 1 HybridAllocator _spillover_range constructing fallback allocator debug 2021-11-15T12:49:13.480+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc loaded 24 GiB in 1287936 extents, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5fd19c000, fragmentation 0.205099 debug 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work debug 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete debug 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bluefs umount debug 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) close debug 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs mount debug 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 756088832 debug 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561ff4e9cae0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561ff4945ba0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea73e0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe214e608) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980553626461, "job": 1, "event": "recovery_started", "log_files": [33446]} debug 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug 2021-11-15T12:49:14.524+0000 7f86cebf1080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. debug 2021-11-15T12:49:14.556+0000 7f86cebf1080 1 bluefs _allocate unable to allocate 0x90000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5d008c000, fragmentation 0.211058, allocated 0x0 debug 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _allocate allocation failed, needed 0x80c40 debug 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x80c40 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f86cebf1080 time 2021-11-15T12:49:14.560240+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x561fd7f10c8c] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 14: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 17: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 23: (BlueStore::_mount()+0x204) [0x561fd856f514] 24: (OSD::init()+0x380) [0x561fd8044a10] 25: main() 26: __libc_start_main() 27: _start() *** Caught signal (Aborted) ** in thread 7f86cebf1080 thread_name:ceph-osd debug 2021-11-15T12:49:14.568+0000 7f86cebf1080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f86cebf1080 time 2021-11-15T12:49:14.560240+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x561fd7f10c8c] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 14: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 17: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 23: (BlueStore::_mount()+0x204) [0x561fd856f514] 24: (OSD::init()+0x380) [0x561fd8044a10] 25: main() 26: __libc_start_main() 27: _start() ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f86ccb97b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x561fd7f10d5d] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 15: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 17: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 19: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 20: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 24: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 25: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 26: (BlueStore::_mount()+0x204) [0x561fd856f514] 27: (OSD::init()+0x380) [0x561fd8044a10] 28: main() 29: __libc_start_main() 30: _start() debug 2021-11-15T12:49:14.576+0000 7f86cebf1080 -1 *** Caught signal (Aborted) ** in thread 7f86cebf1080 thread_name:ceph-osd ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f86ccb97b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x561fd7f10d5d] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 15: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 17: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 19: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 20: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 24: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 25: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 26: (BlueStore::_mount()+0x204) [0x561fd856f514] 27: (OSD::init()+0x380) [0x561fd8044a10] 28: main() 29: __libc_start_main() 30: _start() NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- debug -767> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command assert hook 0x561fe205a540 debug -766> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command abort hook 0x561fe205a540 debug -765> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command leak_some_memory hook 0x561fe205a540 debug -764> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perfcounters_dump hook 0x561fe205a540 debug -763> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 1 hook 0x561fe205a540 debug -762> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf dump hook 0x561fe205a540 debug -761> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perfcounters_schema hook 0x561fe205a540 debug -760> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf histogram dump hook 0x561fe205a540 debug -759> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 2 hook 0x561fe205a540 debug -758> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf schema hook 0x561fe205a540 debug -757> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf histogram schema hook 0x561fe205a540 debug -756> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf reset hook 0x561fe205a540 debug -755> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config show hook 0x561fe205a540 debug -754> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config help hook 0x561fe205a540 debug -753> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config set hook 0x561fe205a540 debug -752> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config unset hook 0x561fe205a540 debug -751> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config get hook 0x561fe205a540 debug -750> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config diff hook 0x561fe205a540 debug -749> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config diff get hook 0x561fe205a540 debug -748> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command injectargs hook 0x561fe205a540 debug -747> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log flush hook 0x561fe205a540 debug -746> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log dump hook 0x561fe205a540 debug -745> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log reopen hook 0x561fe205a540 debug -744> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command dump_mempools hook 0x561fe205c5e8 debug -743> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: get_monmap_and_config debug -742> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: build_initial_monmap debug -741> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: monmap: epoch 0 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-11-15T12:49:06.294343+0000 created 2021-11-15T12:49:06.294343+0000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.noname-c 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.noname-b 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.noname-a debug -740> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -739> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -738> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -737> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: none debug -736> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -735> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -734> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -733> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -732> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -731> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -730> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -729> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -728> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -727> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -726> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -725> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -724> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -723> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: init debug -722> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -721> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -720> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -719> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: none debug -718> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -717> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -716> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -715> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -714> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -713> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -712> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -711> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -710> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -709> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -708> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -707> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -706> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -705> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -704> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: _reopen_session rank -1 debug -703> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: _add_conns ranks=[1,0,2] debug -702> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-b con 0x561fe2108c00 addr [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] debug -701> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-c con 0x561fe2109000 addr [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] debug -700> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-a con 0x561fe2109400 addr [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] debug -699> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -698> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -697> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -696> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): _renew_subs debug -695> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): authenticate will time out at 2021-11-15T12:54:06.295395+0000 debug -694> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): get_auth_request con 0x561fe2108c00 auth_method 0 debug -693> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] debug -692> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): _init_auth method 2 debug -691> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): _init_auth creating new auth debug -690> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more payload 9 debug -689> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 debug -688> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes debug -687> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): get_auth_request con 0x561fe2109400 auth_method 0 debug -686> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] debug -685> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): _init_auth method 2 debug -684> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): _init_auth creating new auth debug -683> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_done global_id 177814234 payload 274 debug -682> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient: _finish_hunting 0 debug -681> 2021-11-15T12:49:06.292+0000 7f86c839f700 1 monclient: found mon.noname-b debug -680> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient: _send_mon_message to mon.noname-b at v2:10.83.19.24:3300/0 debug -679> 2021-11-15T12:49:06.292+0000 7f86c7b9e700 10 monclient: get_auth_request con 0x561fe2109000 auth_method 0 debug -678> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_monmap mon_map magic: 0 v1 debug -677> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: got monmap 6 from mon.noname-b (according to old e6) debug -676> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: dump: epoch 6 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-10-19T09:21:56.109623+0000 created 2020-09-21T15:53:16.921162+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.d 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.e debug -675> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _finish_auth 0 debug -674> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-11-15T12:48:36.297485+0000) debug -673> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _send_mon_message to mon.d at v2:10.83.19.24:3300/0 debug -672> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 monclient: authenticate success, global_id 177814234 debug -671> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_config config(8 keys) v1 debug -670> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: get_monmap_and_config success debug -669> 2021-11-15T12:49:06.292+0000 7f86cebf1080 4 set_mon_vals no callback set debug -668> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_monmap mon_map magic: 0 v1 debug -667> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals log_to_file = false debug -666> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_allow_pool_delete = true debug -665> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_allow_pool_size_one = true debug -664> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_cluster_log_file = debug -663> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: got monmap 6 from mon.d (according to old e6) debug -662> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals mon_pg_warn_min_per_osd = 0 debug -661> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals osd_pool_default_pg_autoscale_mode = on debug -660> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals osd_scrub_auto_repair = true debug -659> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals rbd_default_features = 3 debug -658> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: dump: epoch 6 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-10-19T09:21:56.109623+0000 created 2020-09-21T15:53:16.921162+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.d 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.e debug -657> 2021-11-15T12:49:06.296+0000 7f86c739d700 10 monclient: _finish_auth 0 debug -656> 2021-11-15T12:49:06.296+0000 7f86c739d700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-11-15T12:48:36.298298+0000) debug -655> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 monclient: shutdown debug -654> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 set uid:gid to 167:167 (ceph:ceph) debug -653> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable), process ceph-osd, pid 1 debug -652> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 pidfile_write: ignore empty --pid-file debug -651> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) init /var/run/ceph/ceph-osd.2.asok debug -650> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) bind_and_listen /var/run/ceph/ceph-osd.2.asok debug -649> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 0 hook 0x561fe205e1e8 debug -648> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command version hook 0x561fe205e1e8 debug -647> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command git_version hook 0x561fe205e1e8 debug -646> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command help hook 0x561fe205a390 debug -645> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command get_command_descriptions hook 0x561fe205a330 debug -644> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 object store type is bluestore debug -643> 2021-11-15T12:49:06.296+0000 7f86c739d700 5 asok(0x561fe210c000) entry start debug -642> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -641> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -640> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -639> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a4e0 debug -638> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a4e0 debug -637> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a4e0 debug -636> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a4e0 debug -635> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -634> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -633> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -632> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs debug_inject_read_zeros debug -631> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs files list debug -630> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs stats debug -629> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluestore bluefs device info debug -628> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) close debug -627> 2021-11-15T12:49:06.608+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) close debug -626> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -625> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -624> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -623> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: none debug -622> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -621> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -620> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -619> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -618> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -617> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -616> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -615> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -614> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -613> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -612> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -611> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -610> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -609> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -608> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -607> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -606> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: none debug -605> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -604> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -603> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -602> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -601> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -600> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -599> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -598> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -597> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -596> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -595> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -594> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -593> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -592> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -591> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -590> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -589> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: none debug -588> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -587> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -586> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -585> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -584> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -583> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -582> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -581> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -580> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -579> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -578> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -577> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -576> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -575> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -574> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -573> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -572> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: none debug -571> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -570> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -569> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -568> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -567> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -566> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -565> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -564> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -563> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -562> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -561> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -560> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -559> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -558> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -557> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -556> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -555> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: none debug -554> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -553> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -552> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -551> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -550> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -549> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -548> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -547> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -546> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -545> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -544> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -543> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -542> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -541> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -540> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -539> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -538> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: none debug -537> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -536> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -535> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -534> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -533> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -532> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -531> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -530> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -529> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -528> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -527> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -526> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -525> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -524> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -523> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -522> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -521> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: none debug -520> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -519> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -518> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -517> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -516> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -515> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -514> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -513> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -512> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -511> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -510> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -509> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -508> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -507> 2021-11-15T12:49:06.840+0000 7f86cebf1080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal debug -506> 2021-11-15T12:49:06.840+0000 7f86cebf1080 -1 Falling back to public interface debug -505> 2021-11-15T12:49:06.844+0000 7f86cebf1080 10 monclient: build_initial_monmap debug -504> 2021-11-15T12:49:06.844+0000 7f86cebf1080 10 monclient: monmap: epoch 0 fsid 00000000-0000-0000-0000-000000000000 last_changed 0.000000 created 0.000000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.noname-a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.noname-b 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.noname-c debug -503> 2021-11-15T12:49:06.852+0000 7f86cebf1080 0 load: jerasure load: lrc load: isa debug -502> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -501> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -500> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -499> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -498> 2021-11-15T12:49:07.160+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command objecter_requests hook 0x561fe205a990 debug -497> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -496> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -495> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -494> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -493> 2021-11-15T12:49:07.452+0000 7f86cebf1080 0 osd.2:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -492> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -491> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -490> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -489> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -488> 2021-11-15T12:49:07.752+0000 7f86cebf1080 0 osd.2:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -487> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -486> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -485> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -484> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -483> 2021-11-15T12:49:08.048+0000 7f86cebf1080 0 osd.2:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -482> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -481> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -480> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -479> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -478> 2021-11-15T12:49:08.344+0000 7f86cebf1080 0 osd.2:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -477> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -476> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -475> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -474> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -473> 2021-11-15T12:49:08.628+0000 7f86cebf1080 0 osd.2:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -472> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -471> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -470> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -469> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -468> 2021-11-15T12:49:08.920+0000 7f86cebf1080 0 osd.2:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -467> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -466> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -465> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -464> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -463> 2021-11-15T12:49:09.216+0000 7f86cebf1080 0 osd.2:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -462> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -461> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -460> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -459> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -458> 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 osd.2:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -457> 2021-11-15T12:49:09.512+0000 7f86cebf1080 2 osd.2 0 init /var/lib/ceph/osd/ceph-2 (looks like ssd) debug -456> 2021-11-15T12:49:09.512+0000 7f86cebf1080 2 osd.2 0 journal /var/lib/ceph/osd/ceph-2/journal debug -455> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _mount path /var/lib/ceph/osd/ceph-2 debug -454> 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 bluestore(/var/lib/ceph/osd/ceph-2) _open_db_and_around read-only:0 repair:0 debug -453> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -452> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -451> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -450> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a9a0 debug -449> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a9a0 debug -448> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a9a0 debug -447> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a9a0 debug -446> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -445> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -444> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -443> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs mount debug -442> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug -441> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 0 debug -440> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug -439> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -438> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -437> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -436> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -435> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -434> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -433> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -432> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -431> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -430> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -429> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -428> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -427> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -426> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -425> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -424> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -423> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -422> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -421> 2021-11-15T12:49:09.604+0000 7f86cebf1080 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -420> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug -419> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -418> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug -417> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug -416> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug -415> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug -414> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug -413> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug -412> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug -411> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug -410> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug -409> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug -408> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug -407> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561fe20cb080 debug -406> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug -405> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561fe20cbb20 debug -404> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug -403> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug -402> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug -401> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug -400> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -399> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug -398> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug -397> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug -396> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug -395> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug -394> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug -393> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug -392> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -391> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug -390> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug -389> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug -388> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug -387> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug -386> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug -385> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -384> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -383> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -382> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -381> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -380> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug -379> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug -378> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea6ab0 debug -377> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -376> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug -375> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -374> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug -373> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug -372> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -371> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug -370> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug -369> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug -368> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug -367> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -366> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -365> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -364> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -363> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug -362> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug -361> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -360> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug -359> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug -358> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug -357> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug -356> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug -355> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -354> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug -353> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -352> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug -351> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -350> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug -349> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug -348> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -347> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug -346> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug -345> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug -344> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -343> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug -342> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug -341> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -340> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug -339> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug -338> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -337> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -336> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug -335> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug -334> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -333> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug -332> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug -331> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug -330> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug -329> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug -328> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug -327> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug -326> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -325> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug -324> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug -323> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -322> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -321> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug -320> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug -319> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug -318> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug -317> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug -316> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe205e7b0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -315> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug -314> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug -313> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug -312> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug -311> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug -310> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -309> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug -308> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -307> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -306> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -305> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -304> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -303> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -302> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -301> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -300> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -299> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug -298> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug -297> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug -296> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -295> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -294> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug -293> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -292> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -291> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -290> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug -289> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug -288> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -287> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug -286> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -285> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -284> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -283> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -282> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -281> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -280> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -279> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -278> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -277> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -276> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug -275> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -274> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -273> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -272> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug -271> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -270> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -269> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -268> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -267> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -266> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -265> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -264> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -263> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -262> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -261> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug -260> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug -259> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -258> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -257> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -256> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug -255> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug -254> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug -253> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -252> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug -251> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug -250> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug -249> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug -248> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -247> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -246> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug -245> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980549639149, "job": 1, "event": "recovery_started", "log_files": [33446]} debug -244> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug -243> 2021-11-15T12:49:12.072+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980552076908, "job": 1, "event": "recovery_finished"} debug -242> 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 debug -241> 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old nid_max 5338942 debug -240> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old blobid_max 296960 debug -239> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta freelist_type bitmap debug -238> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 debug -237> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta min_alloc_size 0x1000 debug -236> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0 debug -235> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 load compressor snappy debug -234> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 add compressor snappy 0x561fe2ea7e90 debug -233> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0x561fe2ea7e90 debug -232> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 load: compressor snappy loaded and registered debug -231> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0x561fe2ea7e90 debug -230> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist init debug -229> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist _read_cfg debug -228> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator dump block hook 0x561fe2ea7da0 debug -227> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator score block hook 0x561fe2ea7da0 debug -226> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator fragmentation block hook 0x561fe2ea7da0 debug -225> 2021-11-15T12:49:12.120+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc opening allocation metadata debug -224> 2021-11-15T12:49:12.952+0000 7f86cebf1080 1 HybridAllocator _spillover_range constructing fallback allocator debug -223> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator dump block.fallback hook 0x561fe2ea7d10 debug -222> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator score block.fallback hook 0x561fe2ea7d10 debug -221> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator fragmentation block.fallback hook 0x561fe2ea7d10 debug -220> 2021-11-15T12:49:13.480+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc loaded 24 GiB in 1287936 extents, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5fd19c000, fragmentation 0.205099 debug -219> 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work debug -218> 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete debug -217> 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bluefs umount debug -216> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs debug_inject_read_zeros debug -215> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs files list debug -214> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs stats debug -213> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluestore bluefs device info debug -212> 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) close debug -211> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a9a0 debug -210> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a9a0 debug -209> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a9a0 debug -208> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a9a0 debug -207> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -206> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -205> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -204> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs mount debug -203> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug -202> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 756088832 debug -201> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug -200> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -199> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -198> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -197> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -196> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -195> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -194> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -193> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -192> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -191> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -190> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -189> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -188> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -187> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -186> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -185> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -184> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -183> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -182> 2021-11-15T12:49:13.620+0000 7f86cebf1080 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -181> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug -180> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -179> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug -178> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug -177> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug -176> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug -175> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug -174> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug -173> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug -172> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug -171> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug -170> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug -169> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug -168> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561ff4e9cae0 debug -167> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug -166> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561ff4945ba0 debug -165> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug -164> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug -163> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug -162> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug -161> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -160> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug -159> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug -158> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug -157> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug -156> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug -155> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug -154> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug -153> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -152> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug -151> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug -150> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug -149> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug -148> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug -147> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug -146> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -145> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -144> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -143> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -142> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -141> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug -140> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug -139> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea73e0 debug -138> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -137> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug -136> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -135> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug -134> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug -133> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -132> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug -131> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug -130> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug -129> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug -128> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -127> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -126> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -125> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -124> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug -123> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug -122> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -121> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug -120> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug -119> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug -118> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug -117> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug -116> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -115> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug -114> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -113> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug -112> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -111> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug -110> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug -109> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -108> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug -107> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug -106> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug -105> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -104> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug -103> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug -102> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -101> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug -100> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug -99> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -98> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -97> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug -96> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug -95> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -94> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug -93> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug -92> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug -91> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug -90> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug -89> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug -88> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug -87> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -86> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug -85> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -84> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -83> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug -82> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug -81> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug -80> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug -79> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug -78> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe214e608) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -77> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug -76> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug -75> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug -74> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug -73> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug -72> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -71> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug -70> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -69> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -68> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -67> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -66> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -65> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -64> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -63> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -62> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -61> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug -60> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug -59> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug -58> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -57> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -56> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug -55> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -54> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -53> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -52> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug -51> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug -50> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -49> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug -48> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -47> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -46> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -45> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -44> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -43> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -42> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -41> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -40> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -39> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -38> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug -37> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -36> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -35> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -34> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug -33> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -32> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -31> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -30> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -29> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -28> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -27> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -26> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -25> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -24> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -23> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug -22> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug -21> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -20> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -19> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -18> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug -17> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug -16> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug -15> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -14> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug -13> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug -12> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug -11> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug -10> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -9> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -8> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug -7> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980553626461, "job": 1, "event": "recovery_started", "log_files": [33446]} debug -6> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug -5> 2021-11-15T12:49:14.524+0000 7f86cebf1080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. debug -4> 2021-11-15T12:49:14.556+0000 7f86cebf1080 1 bluefs _allocate unable to allocate 0x90000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5d008c000, fragmentation 0.211058, allocated 0x0 debug -3> 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _allocate allocation failed, needed 0x80c40 debug -2> 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x80c40 debug -1> 2021-11-15T12:49:14.568+0000 7f86cebf1080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f86cebf1080 time 2021-11-15T12:49:14.560240+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x561fd7f10c8c] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 14: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 17: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 23: (BlueStore::_mount()+0x204) [0x561fd856f514] 24: (OSD::init()+0x380) [0x561fd8044a10] 25: main() 26: __libc_start_main() 27: _start() debug 0> 2021-11-15T12:49:14.576+0000 7f86cebf1080 -1 *** Caught signal (Aborted) ** in thread 7f86cebf1080 thread_name:ceph-osd ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f86ccb97b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x561fd7f10d5d] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 15: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 17: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 19: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 20: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 24: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 25: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 26: (BlueStore::_mount()+0x204) [0x561fd856f514] 27: (OSD::init()+0x380) [0x561fd8044a10] 28: main() 29: __libc_start_main() 30: _start() NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 140216844801792 / admin_socket 140216853194496 / msgr-worker-2 140216861587200 / msgr-worker-1 140216869979904 / msgr-worker-0 140216970973312 / ceph-osd max_recent 10000 max_new 10000 log_file --- end dump of recent events --- --- begin dump of recent events --- debug -767> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command assert hook 0x561fe205a540 debug -766> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command abort hook 0x561fe205a540 debug -765> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command leak_some_memory hook 0x561fe205a540 debug -764> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perfcounters_dump hook 0x561fe205a540 debug -763> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 1 hook 0x561fe205a540 debug -762> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf dump hook 0x561fe205a540 debug -761> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perfcounters_schema hook 0x561fe205a540 debug -760> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf histogram dump hook 0x561fe205a540 debug -759> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 2 hook 0x561fe205a540 debug -758> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf schema hook 0x561fe205a540 debug -757> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf histogram schema hook 0x561fe205a540 debug -756> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command perf reset hook 0x561fe205a540 debug -755> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config show hook 0x561fe205a540 debug -754> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config help hook 0x561fe205a540 debug -753> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config set hook 0x561fe205a540 debug -752> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config unset hook 0x561fe205a540 debug -751> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config get hook 0x561fe205a540 debug -750> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config diff hook 0x561fe205a540 debug -749> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command config diff get hook 0x561fe205a540 debug -748> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command injectargs hook 0x561fe205a540 debug -747> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log flush hook 0x561fe205a540 debug -746> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log dump hook 0x561fe205a540 debug -745> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command log reopen hook 0x561fe205a540 debug -744> 2021-11-15T12:49:06.280+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command dump_mempools hook 0x561fe205c5e8 debug -743> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: get_monmap_and_config debug -742> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: build_initial_monmap debug -741> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: monmap: epoch 0 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-11-15T12:49:06.294343+0000 created 2021-11-15T12:49:06.294343+0000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.noname-c 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.noname-b 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.noname-a debug -740> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -739> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -738> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -737> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: none debug -736> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -735> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -734> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -733> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -732> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -731> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -730> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -729> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -728> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -727> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -726> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -725> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -724> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -723> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: init debug -722> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -721> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -720> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: cephx debug -719> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding auth protocol: none debug -718> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -717> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -716> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -715> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -714> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -713> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -712> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -711> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -710> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -709> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -708> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: crc debug -707> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 AuthRegistry(0x7fff8190f260) adding con mode: secure debug -706> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -705> 2021-11-15T12:49:06.292+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -704> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: _reopen_session rank -1 debug -703> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: _add_conns ranks=[1,0,2] debug -702> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-b con 0x561fe2108c00 addr [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] debug -701> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-c con 0x561fe2109000 addr [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] debug -700> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): picked mon.noname-a con 0x561fe2109400 addr [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] debug -699> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -698> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -697> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): start opening mon connection debug -696> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): _renew_subs debug -695> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient(hunting): authenticate will time out at 2021-11-15T12:54:06.295395+0000 debug -694> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): get_auth_request con 0x561fe2108c00 auth_method 0 debug -693> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] debug -692> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): _init_auth method 2 debug -691> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): _init_auth creating new auth debug -690> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more payload 9 debug -689> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 debug -688> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes debug -687> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): get_auth_request con 0x561fe2109400 auth_method 0 debug -686> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] debug -685> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): _init_auth method 2 debug -684> 2021-11-15T12:49:06.292+0000 7f86c8ba0700 10 monclient(hunting): _init_auth creating new auth debug -683> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient(hunting): handle_auth_done global_id 177814234 payload 274 debug -682> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient: _finish_hunting 0 debug -681> 2021-11-15T12:49:06.292+0000 7f86c839f700 1 monclient: found mon.noname-b debug -680> 2021-11-15T12:49:06.292+0000 7f86c839f700 10 monclient: _send_mon_message to mon.noname-b at v2:10.83.19.24:3300/0 debug -679> 2021-11-15T12:49:06.292+0000 7f86c7b9e700 10 monclient: get_auth_request con 0x561fe2109000 auth_method 0 debug -678> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_monmap mon_map magic: 0 v1 debug -677> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: got monmap 6 from mon.noname-b (according to old e6) debug -676> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: dump: epoch 6 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-10-19T09:21:56.109623+0000 created 2020-09-21T15:53:16.921162+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.d 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.e debug -675> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _finish_auth 0 debug -674> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-11-15T12:48:36.297485+0000) debug -673> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: _send_mon_message to mon.d at v2:10.83.19.24:3300/0 debug -672> 2021-11-15T12:49:06.292+0000 7f86cebf1080 5 monclient: authenticate success, global_id 177814234 debug -671> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_config config(8 keys) v1 debug -670> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 monclient: get_monmap_and_config success debug -669> 2021-11-15T12:49:06.292+0000 7f86cebf1080 4 set_mon_vals no callback set debug -668> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: handle_monmap mon_map magic: 0 v1 debug -667> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals log_to_file = false debug -666> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_allow_pool_delete = true debug -665> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_allow_pool_size_one = true debug -664> 2021-11-15T12:49:06.292+0000 7f86cebf1080 10 set_mon_vals mon_cluster_log_file = debug -663> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: got monmap 6 from mon.d (according to old e6) debug -662> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals mon_pg_warn_min_per_osd = 0 debug -661> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals osd_pool_default_pg_autoscale_mode = on debug -660> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals osd_scrub_auto_repair = true debug -659> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 set_mon_vals rbd_default_features = 3 debug -658> 2021-11-15T12:49:06.292+0000 7f86c739d700 10 monclient: dump: epoch 6 fsid addad34a-62d2-446a-8553-55461cacb31a last_changed 2021-10-19T09:21:56.109623+0000 created 2020-09-21T15:53:16.921162+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.d 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.e debug -657> 2021-11-15T12:49:06.296+0000 7f86c739d700 10 monclient: _finish_auth 0 debug -656> 2021-11-15T12:49:06.296+0000 7f86c739d700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-11-15T12:48:36.298298+0000) debug -655> 2021-11-15T12:49:06.296+0000 7f86cebf1080 10 monclient: shutdown debug -654> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 set uid:gid to 167:167 (ceph:ceph) debug -653> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable), process ceph-osd, pid 1 debug -652> 2021-11-15T12:49:06.296+0000 7f86cebf1080 0 pidfile_write: ignore empty --pid-file debug -651> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) init /var/run/ceph/ceph-osd.2.asok debug -650> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) bind_and_listen /var/run/ceph/ceph-osd.2.asok debug -649> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command 0 hook 0x561fe205e1e8 debug -648> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command version hook 0x561fe205e1e8 debug -647> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command git_version hook 0x561fe205e1e8 debug -646> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command help hook 0x561fe205a390 debug -645> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command get_command_descriptions hook 0x561fe205a330 debug -644> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 object store type is bluestore debug -643> 2021-11-15T12:49:06.296+0000 7f86c739d700 5 asok(0x561fe210c000) entry start debug -642> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -641> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -640> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -639> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a4e0 debug -638> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a4e0 debug -637> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a4e0 debug -636> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a4e0 debug -635> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -634> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -633> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -632> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs debug_inject_read_zeros debug -631> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs files list debug -630> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs stats debug -629> 2021-11-15T12:49:06.296+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluestore bluefs device info debug -628> 2021-11-15T12:49:06.296+0000 7f86cebf1080 1 bdev(0x561fe2108c00 /var/lib/ceph/osd/ceph-2/block) close debug -627> 2021-11-15T12:49:06.608+0000 7f86cebf1080 1 bdev(0x561fe2108800 /var/lib/ceph/osd/ceph-2/block) close debug -626> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -625> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -624> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: cephx debug -623> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding auth protocol: none debug -622> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -621> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -620> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -619> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -618> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -617> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -616> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -615> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -614> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -613> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -612> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: crc debug -611> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0a40) adding con mode: secure debug -610> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -609> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -608> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -607> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: cephx debug -606> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding auth protocol: none debug -605> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -604> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -603> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -602> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -601> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -600> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -599> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -598> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -597> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -596> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -595> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: crc debug -594> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce0140) adding con mode: secure debug -593> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -592> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -591> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -590> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: cephx debug -589> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding auth protocol: none debug -588> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -587> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -586> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -585> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -584> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -583> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -582> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -581> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -580> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -579> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -578> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: crc debug -577> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1340) adding con mode: secure debug -576> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -575> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -574> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -573> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: cephx debug -572> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding auth protocol: none debug -571> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -570> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -569> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -568> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -567> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -566> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -565> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -564> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -563> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -562> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -561> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: crc debug -560> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce1c40) adding con mode: secure debug -559> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -558> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -557> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -556> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: cephx debug -555> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding auth protocol: none debug -554> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -553> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -552> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -551> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -550> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -549> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -548> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -547> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -546> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -545> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -544> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: crc debug -543> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2540) adding con mode: secure debug -542> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -541> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -540> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -539> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: cephx debug -538> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding auth protocol: none debug -537> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -536> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -535> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -534> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -533> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -532> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -531> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -530> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -529> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -528> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -527> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: crc debug -526> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce2e40) adding con mode: secure debug -525> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -524> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -523> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -522> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: cephx debug -521> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding auth protocol: none debug -520> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -519> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -518> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -517> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -516> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -515> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -514> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -513> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -512> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -511> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -510> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: crc debug -509> 2021-11-15T12:49:06.840+0000 7f86cebf1080 5 AuthRegistry(0x561fe2ce3740) adding con mode: secure debug -508> 2021-11-15T12:49:06.840+0000 7f86cebf1080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring debug -507> 2021-11-15T12:49:06.840+0000 7f86cebf1080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal debug -506> 2021-11-15T12:49:06.840+0000 7f86cebf1080 -1 Falling back to public interface debug -505> 2021-11-15T12:49:06.844+0000 7f86cebf1080 10 monclient: build_initial_monmap debug -504> 2021-11-15T12:49:06.844+0000 7f86cebf1080 10 monclient: monmap: epoch 0 fsid 00000000-0000-0000-0000-000000000000 last_changed 0.000000 created 0.000000 min_mon_release 0 (unknown) election_strategy: 1 0: [v2:10.83.19.23:3300/0,v1:10.83.19.23:6789/0] mon.noname-a 1: [v2:10.83.19.24:3300/0,v1:10.83.19.24:6789/0] mon.noname-b 2: [v2:10.83.19.25:3300/0,v1:10.83.19.25:6789/0] mon.noname-c debug -503> 2021-11-15T12:49:06.852+0000 7f86cebf1080 0 load: jerasure load: lrc load: isa debug -502> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -501> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -500> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -499> 2021-11-15T12:49:06.852+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -498> 2021-11-15T12:49:07.160+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command objecter_requests hook 0x561fe205a990 debug -497> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -496> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -495> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -494> 2021-11-15T12:49:07.160+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -493> 2021-11-15T12:49:07.452+0000 7f86cebf1080 0 osd.2:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -492> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -491> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -490> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -489> 2021-11-15T12:49:07.452+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -488> 2021-11-15T12:49:07.752+0000 7f86cebf1080 0 osd.2:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -487> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -486> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -485> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -484> 2021-11-15T12:49:07.752+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -483> 2021-11-15T12:49:08.048+0000 7f86cebf1080 0 osd.2:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -482> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -481> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -480> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -479> 2021-11-15T12:49:08.048+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -478> 2021-11-15T12:49:08.344+0000 7f86cebf1080 0 osd.2:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -477> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -476> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -475> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -474> 2021-11-15T12:49:08.344+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -473> 2021-11-15T12:49:08.628+0000 7f86cebf1080 0 osd.2:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -472> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -471> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -470> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -469> 2021-11-15T12:49:08.628+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -468> 2021-11-15T12:49:08.920+0000 7f86cebf1080 0 osd.2:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -467> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -466> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -465> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -464> 2021-11-15T12:49:08.920+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -463> 2021-11-15T12:49:09.216+0000 7f86cebf1080 0 osd.2:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -462> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -461> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -460> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -459> 2021-11-15T12:49:09.216+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) close debug -458> 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 osd.2:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) debug -457> 2021-11-15T12:49:09.512+0000 7f86cebf1080 2 osd.2 0 init /var/lib/ceph/osd/ceph-2 (looks like ssd) debug -456> 2021-11-15T12:49:09.512+0000 7f86cebf1080 2 osd.2 0 journal /var/lib/ceph/osd/ceph-2/journal debug -455> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _mount path /var/lib/ceph/osd/ceph-2 debug -454> 2021-11-15T12:49:09.512+0000 7f86cebf1080 0 bluestore(/var/lib/ceph/osd/ceph-2) _open_db_and_around read-only:0 repair:0 debug -453> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -452> 2021-11-15T12:49:09.512+0000 7f86cebf1080 1 bdev(0x561fe2dde400 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -451> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 debug -450> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a9a0 debug -449> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a9a0 debug -448> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a9a0 debug -447> 2021-11-15T12:49:09.516+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a9a0 debug -446> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -445> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -444> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -443> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs mount debug -442> 2021-11-15T12:49:09.516+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug -441> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 0 debug -440> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug -439> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -438> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -437> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -436> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -435> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -434> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -433> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -432> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -431> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -430> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -429> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -428> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -427> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -426> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -425> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -424> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -423> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -422> 2021-11-15T12:49:09.600+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -421> 2021-11-15T12:49:09.604+0000 7f86cebf1080 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -420> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug -419> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -418> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug -417> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug -416> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug -415> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug -414> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug -413> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug -412> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug -411> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug -410> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug -409> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug -408> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug -407> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561fe20cb080 debug -406> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug -405> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561fe20cbb20 debug -404> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug -403> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug -402> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug -401> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug -400> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -399> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug -398> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug -397> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug -396> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug -395> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug -394> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug -393> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug -392> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -391> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug -390> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug -389> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug -388> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug -387> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug -386> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug -385> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -384> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -383> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -382> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -381> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -380> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug -379> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug -378> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea6ab0 debug -377> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -376> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug -375> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -374> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug -373> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug -372> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -371> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug -370> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug -369> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug -368> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug -367> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -366> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -365> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -364> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -363> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug -362> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug -361> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -360> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug -359> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug -358> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug -357> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug -356> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug -355> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -354> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug -353> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -352> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug -351> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -350> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug -349> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug -348> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -347> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug -346> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug -345> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug -344> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -343> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug -342> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug -341> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -340> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug -339> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug -338> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -337> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -336> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug -335> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug -334> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -333> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug -332> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug -331> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug -330> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug -329> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug -328> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug -327> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug -326> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -325> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_readonly.cc:23] Opening the db in read only mode debug -324> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug -323> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -322> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -321> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug -320> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug -319> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug -318> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug -317> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug -316> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe205e7b0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -315> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug -314> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug -313> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug -312> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug -311> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug -310> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -309> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug -308> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -307> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -306> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -305> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -304> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -303> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -302> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -301> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -300> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -299> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug -298> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug -297> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug -296> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -295> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -294> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug -293> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -292> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -291> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -290> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug -289> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug -288> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -287> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug -286> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -285> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -284> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -283> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -282> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -281> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -280> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -279> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -278> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -277> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -276> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug -275> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -274> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -273> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -272> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug -271> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -270> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -269> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -268> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -267> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -266> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -265> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -264> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -263> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -262> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -261> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug -260> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug -259> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -258> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -257> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -256> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug -255> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug -254> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug -253> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -252> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug -251> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug -250> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug -249> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug -248> 2021-11-15T12:49:09.604+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -247> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -246> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug -245> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980549639149, "job": 1, "event": "recovery_started", "log_files": [33446]} debug -244> 2021-11-15T12:49:09.636+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug -243> 2021-11-15T12:49:12.072+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980552076908, "job": 1, "event": "recovery_finished"} debug -242> 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824 debug -241> 2021-11-15T12:49:12.072+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old nid_max 5338942 debug -240> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta old blobid_max 296960 debug -239> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta freelist_type bitmap debug -238> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 debug -237> 2021-11-15T12:49:12.096+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_super_meta min_alloc_size 0x1000 debug -236> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0 debug -235> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 load compressor snappy debug -234> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 add compressor snappy 0x561fe2ea7e90 debug -233> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0x561fe2ea7e90 debug -232> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 load: compressor snappy loaded and registered debug -231> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 get compressor snappy = 0x561fe2ea7e90 debug -230> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist init debug -229> 2021-11-15T12:49:12.112+0000 7f86cebf1080 1 freelist _read_cfg debug -228> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator dump block hook 0x561fe2ea7da0 debug -227> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator score block hook 0x561fe2ea7da0 debug -226> 2021-11-15T12:49:12.120+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator fragmentation block hook 0x561fe2ea7da0 debug -225> 2021-11-15T12:49:12.120+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc opening allocation metadata debug -224> 2021-11-15T12:49:12.952+0000 7f86cebf1080 1 HybridAllocator _spillover_range constructing fallback allocator debug -223> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator dump block.fallback hook 0x561fe2ea7d10 debug -222> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator score block.fallback hook 0x561fe2ea7d10 debug -221> 2021-11-15T12:49:12.952+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore allocator fragmentation block.fallback hook 0x561fe2ea7d10 debug -220> 2021-11-15T12:49:13.480+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _init_alloc loaded 24 GiB in 1287936 extents, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5fd19c000, fragmentation 0.205099 debug -219> 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work debug -218> 2021-11-15T12:49:13.480+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete debug -217> 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bluefs umount debug -216> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs debug_inject_read_zeros debug -215> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs files list debug -214> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluefs stats debug -213> 2021-11-15T12:49:13.484+0000 7f86cebf1080 5 asok(0x561fe210c000) unregister_commands bluestore bluefs device info debug -212> 2021-11-15T12:49:13.484+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) close debug -211> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluestore bluefs device info hook 0x561fe205a9a0 debug -210> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs stats hook 0x561fe205a9a0 debug -209> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs files list hook 0x561fe205a9a0 debug -208> 2021-11-15T12:49:13.560+0000 7f86cebf1080 5 asok(0x561fe210c000) register_command bluefs debug_inject_read_zeros hook 0x561fe205a9a0 debug -207> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block debug -206> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bdev(0x561fe2dde800 /var/lib/ceph/osd/ceph-2/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size 4096 (4 KiB) non-rotational discard supported debug -205> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 200 GiB debug -204> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs mount debug -203> 2021-11-15T12:49:13.560+0000 7f86cebf1080 1 bluefs _init_alloc shared, id 1, capacity 0x31ffc00000, block size 0x10000 debug -202> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluefs mount shared_bdev_used = 756088832 debug -201> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_db_environment set db_paths to db,204006961971 db.slow,204006961971 debug -200> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -199> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -198> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -197> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -196> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -195> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -194> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -193> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -192> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -191> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_total_wal_size = 1073741824 debug -190> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compaction_readahead_size = 2097152 debug -189> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_write_buffer_number = 4 debug -188> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option max_background_compactions = 2 debug -187> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option compression = kNoCompression debug -186> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option writable_file_max_buffer_size = 0 debug -185> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option min_write_buffer_number_to_merge = 1 debug -184> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option recycle_log_file_num = 4 debug -183> 2021-11-15T12:49:13.616+0000 7f86cebf1080 1 set rocksdb option write_buffer_size = 268435456 debug -182> 2021-11-15T12:49:13.620+0000 7f86cebf1080 5 rocksdb: verify_sharding column families from rocksdb: [default] debug -181> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: RocksDB version: 6.8.1 debug -180> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug -179> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compile date Sep 16 2021 debug -178> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: DB SUMMARY debug -177> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: CURRENT file: CURRENT debug -176> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: IDENTITY file: IDENTITY debug -175> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: MANIFEST file: MANIFEST-031387 size: 238322 Bytes debug -174> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db dir, Total Num: 11, files: 033348.sst 033428.sst 033429.sst 033430.sst 033431.sst 033432.sst 033433.sst 033434.sst 033443.sst debug -173> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: SST files in db.slow dir, Total Num: 0, files: debug -172> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Write Ahead Log file in db.wal: 033446.log size: 221571739 ; debug -171> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.error_if_exists: 0 debug -170> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_if_missing: 0 debug -169> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_checks: 1 debug -168> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.env: 0x561ff4e9cae0 debug -167> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.fs: Legacy File System debug -166> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.info_log: 0x561ff4945ba0 debug -165> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_file_opening_threads: 16 debug -164> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.statistics: (nil) debug -163> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_fsync: 0 debug -162> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_log_file_size: 0 debug -161> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug -160> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_file_time_to_roll: 0 debug -159> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.keep_log_file_num: 1000 debug -158> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.recycle_log_file_num: 0 debug -157> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_fallocate: 1 debug -156> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_reads: 0 debug -155> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_mmap_writes: 0 debug -154> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_reads: 0 debug -153> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug -152> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.create_missing_column_families: 0 debug -151> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_log_dir: debug -150> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_dir: db.wal debug -149> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_cache_numshardbits: 6 debug -148> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_subcompactions: 1 debug -147> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_flushes: -1 debug -146> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_ttl_seconds: 0 debug -145> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.WAL_size_limit_MB: 0 debug -144> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug -143> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug -142> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.is_fd_close_on_exec: 1 debug -141> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.advise_random_on_open: 1 debug -140> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.db_write_buffer_size: 0 debug -139> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_manager: 0x561fe2ea73e0 debug -138> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug -137> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 debug -136> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug -135> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.use_adaptive_mutex: 0 debug -134> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limiter: (nil) debug -133> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug -132> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_recovery_mode: 2 debug -131> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_thread_tracking: 0 debug -130> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_pipelined_write: 0 debug -129> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.unordered_write: 0 debug -128> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug -127> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug -126> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug -125> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug -124> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.row_cache: None debug -123> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_filter: None debug -122> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug -121> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.allow_ingest_behind: 0 debug -120> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.preserve_deletes: 0 debug -119> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.two_write_queues: 0 debug -118> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.manual_wal_flush: 0 debug -117> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.atomic_flush: 0 debug -116> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug -115> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.persist_stats_to_disk: 0 debug -114> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_dbid_to_manifest: 0 debug -113> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.log_readahead_size: 0 debug -112> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.sst_file_checksum_func: Unknown debug -111> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_jobs: 2 debug -110> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_background_compactions: 2 debug -109> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug -108> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.writable_file_max_buffer_size: 0 debug -107> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delayed_write_rate : 16777216 debug -106> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_total_wal_size: 1073741824 debug -105> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug -104> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_dump_period_sec: 600 debug -103> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_persist_period_sec: 600 debug -102> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug -101> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_open_files: -1 debug -100> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bytes_per_sync: 0 debug -99> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.wal_bytes_per_sync: 0 debug -98> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.strict_bytes_per_sync: 0 debug -97> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_readahead_size: 2097152 debug -96> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Compression algorithms supported: debug -95> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug -94> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZSTD supported: 0 debug -93> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kXpressCompression supported: 0 debug -92> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4HCCompression supported: 1 debug -91> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kLZ4Compression supported: 1 debug -90> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kBZip2Compression supported: 0 debug -89> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kZlibCompression supported: 1 debug -88> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: kSnappyCompression supported: 1 debug -87> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Fast CRC32 supported: Supported on x86 debug -86> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: db/MANIFEST-031387 debug -85> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug -84> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug -83> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor debug -82> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter: None debug -81> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_filter_factory: None debug -80> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_factory: SkipListFactory debug -79> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_factory: BlockBasedTable debug -78> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561fe214e608) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x561fe20a7210 block_cache_name: BinnedLRUCache block_cache_options: capacity : 1449551462 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug -77> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.write_buffer_size: 268435456 debug -76> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number: 4 debug -75> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression: NoCompression debug -74> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression: Disabled debug -73> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.prefix_extractor: nullptr debug -72> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug -71> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.num_levels: 7 debug -70> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug -69> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug -68> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug -67> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug -66> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug -65> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug -64> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug -63> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug -62> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug -61> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.window_bits: -14 debug -60> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.level: 32767 debug -59> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.strategy: 0 debug -58> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug -57> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug -56> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compression_opts.enabled: false debug -55> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug -54> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug -53> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug -52> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_base: 67108864 debug -51> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.target_file_size_multiplier: 1 debug -50> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug -49> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 debug -48> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug -47> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug -46> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug -45> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug -44> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug -43> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug -42> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug -41> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug -40> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug -39> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug -38> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.arena_block_size: 33554432 debug -37> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug -36> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug -35> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug -34> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.disable_auto_compactions: 0 debug -33> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug -32> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug -31> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug -30> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug -29> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug -28> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug -27> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug -26> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug -25> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug -24> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug -23> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.table_properties_collectors: debug -22> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_support: 0 debug -21> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.inplace_update_num_locks: 10000 debug -20> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug -19> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug -18> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.memtable_huge_page_size: 0 debug -17> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.bloom_locality: 0 debug -16> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.max_successive_merges: 0 debug -15> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.optimize_filters_for_hits: 0 debug -14> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.paranoid_file_checks: 0 debug -13> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.force_consistency_checks: 0 debug -12> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.report_bg_io_stats: 0 debug -11> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.ttl: 2592000 debug -10> 2021-11-15T12:49:13.620+0000 7f86cebf1080 4 rocksdb: Options.periodic_compaction_seconds: 0 debug -9> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:db/MANIFEST-031387 succeeded,manifest_file_number is 31387, next_file_number is 33449, last_sequence is 9496239791, log_number is 33446,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug -8> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 33446 debug -7> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1636980553626461, "job": 1, "event": "recovery_started", "log_files": [33446]} debug -6> 2021-11-15T12:49:13.624+0000 7f86cebf1080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #33446 mode 2 debug -5> 2021-11-15T12:49:14.524+0000 7f86cebf1080 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. debug -4> 2021-11-15T12:49:14.556+0000 7f86cebf1080 1 bluefs _allocate unable to allocate 0x90000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5d008c000, fragmentation 0.211058, allocated 0x0 debug -3> 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _allocate allocation failed, needed 0x80c40 debug -2> 2021-11-15T12:49:14.556+0000 7f86cebf1080 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x80c40 debug -1> 2021-11-15T12:49:14.568+0000 7f86cebf1080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f86cebf1080 time 2021-11-15T12:49:14.560240+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc") ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x561fd7f10c8c] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 14: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 17: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 23: (BlueStore::_mount()+0x204) [0x561fd856f514] 24: (OSD::init()+0x380) [0x561fd8044a10] 25: main() 26: __libc_start_main() 27: _start() debug 0> 2021-11-15T12:49:14.576+0000 7f86cebf1080 -1 *** Caught signal (Aborted) ** in thread 7f86cebf1080 thread_name:ceph-osd ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f86ccb97b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x561fd7f10d5d] 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x561fd8607901] 6: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x561fd8607be0] 7: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock&)+0x32) [0x561fd8618cf2] 8: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x561fd863132b] 9: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561fd8ac3acf] 10: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x561fd8bd581a] 11: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x561fd8bd6c70] 12: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x561fd8cf2416] 13: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x561fd8cf2d5c] 14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x561fd8cf347c] 15: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x561fd8cf350d] 16: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x561fd8cf6978] 17: (rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x561fd8ca13d5] 18: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x561fd8b06415] 19: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1c2e) [0x561fd8b08b4e] 20: (rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, unsigned long*)+0xae8) [0x561fd8b09ea8] 21: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x59d) [0x561fd8b03bcd] 22: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x15) [0x561fd8b04f65] 23: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x10c1) [0x561fd8a7cec1] 24: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x561fd85024d8] 25: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x561fd856c657] 26: (BlueStore::_mount()+0x204) [0x561fd856f514] 27: (OSD::init()+0x380) [0x561fd8044a10] 28: main() 29: __libc_start_main() 30: _start() NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 140216844801792 / admin_socket 140216853194496 / msgr-worker-2 140216861587200 / msgr-worker-1 140216869979904 / msgr-worker-0 140216970973312 / ceph-osd max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2021-11-15T12:49:14.580310Z_a0aab6db-79d6-432e-8c4b-006640165a61/log --- end dump of recent events --- reraise_fatal: default handler for signal 6 didn't terminate the process?