Project

General

Profile

Bug #51866 » ceph-mon0.txt

David Piper, 07/27/2021 11:23 AM

 
-- Logs begin at Sat 2021-07-24 03:41:52 UTC, end at Tue 2021-07-27 11:19:36 UTC. --
Jul 27 09:28:00 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:00.060+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 331350016 full_alloc: 331350016 kv_alloc: 352321536
Jul 27 09:28:05 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:05.499+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 331350016 full_alloc: 331350016 kv_alloc: 352321536
Jul 27 09:28:11 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:11.489+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 331350016 full_alloc: 331350016 kv_alloc: 352321536
Jul 27 09:28:16 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:16.490+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 331350016 full_alloc: 331350016 kv_alloc: 352321536
Jul 27 09:28:21 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:21.492+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 331350016 full_alloc: 331350016 kv_alloc: 352321536
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.716+0000 7f72c8a5a700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.716+0000 7f72c8a5a700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 1 mon.newbrunswick0@0(leader).osd e87647 we're forcing failure of osd.2
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 0 log_channel(cluster) log [INF] : osd.2 failed (root=default,host=newbrunswick2) (connection refused reported by osd.0)
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:23 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:23.717+0000 7f72c8a5a700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.704+0000 7f72ca25d700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.705+0000 7f72ca25d700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.706+0000 7f72ca25d700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.706+0000 7f72ca25d700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.707+0000 7f72ca25d700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.707+0000 7f72ca25d700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.713+0000 7f72cca62700 1 mon.newbrunswick0@0(leader).osd e87647 do_prune osdmap full prune enabled
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.714+0000 7f72cca62700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
Jul 27 09:28:24 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:24.714+0000 7f72cca62700 0 log_channel(cluster) log [WRN] : Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN)
Jul 27 09:28:28 newbrunswick0 systemd[1]: Stopping Ceph Monitor...
Jul 27 09:28:29 newbrunswick0 docker[1215]: teardown: managing teardown after SIGTERM
Jul 27 09:28:29 newbrunswick0 docker[1215]: teardown: Sending SIGTERM to PID 74
Jul 27 09:28:29 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:29.241+0000 7f72d0a6a700 -1 received signal: Terminated from (PID: 1714) UID: 0
Jul 27 09:28:29 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:29.241+0000 7f72d0a6a700 -1 mon.newbrunswick0@0(leader) e2 *** Got Signal Terminated ***
Jul 27 09:28:29 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:29.241+0000 7f72d0a6a700 1 mon.newbrunswick0@0(leader) e2 shutdown
Jul 27 09:28:29 newbrunswick0 docker[1215]: teardown: Waiting PID 74 to terminate .debug 2021-07-27T09:28:29.263+0000 7f72e0bf46c0 1 rocksdb: close waiting for compaction thread to stop
Jul 27 09:28:29 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:29.266+0000 7f72e0bf46c0 1 rocksdb: close compaction thread to stopped
Jul 27 09:28:29 newbrunswick0 docker[1215]: debug 2021-07-27T09:28:29.266+0000 7f72e0bf46c0 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work
Jul 27 09:28:29 newbrunswick0 docker[1215]: .debug 2021-07-27T09:28:29.411+0000 7f72e0bf46c0 4 rocksdb: [db/db_impl.cc:563] Shutdown complete
Jul 27 09:28:29 newbrunswick0 docker[1215]: .
Jul 27 09:28:29 newbrunswick0 docker[1215]: teardown: Process 74 is terminated
Jul 27 09:28:29 newbrunswick0 docker[6644]: ceph-mon-newbrunswick0
Jul 27 09:28:29 newbrunswick0 systemd[1]: ceph-mon@newbrunswick0.service: main process exited, code=exited, status=143/n/a
Jul 27 09:28:29 newbrunswick0 systemd[1]: Stopped Ceph Monitor.
Jul 27 09:28:29 newbrunswick0 systemd[1]: Unit ceph-mon@newbrunswick0.service entered failed state.
Jul 27 09:28:29 newbrunswick0 systemd[1]: ceph-mon@newbrunswick0.service failed.
-- Reboot --
Jul 27 09:29:07 newbrunswick0 systemd[1]: Starting Ceph Monitor...
Jul 27 09:29:07 newbrunswick0 docker[1034]: Error response from daemon: No such container: ceph-mon-newbrunswick0
Jul 27 09:29:07 newbrunswick0 docker[1106]: Error: No such container: ceph-mon-newbrunswick0
Jul 27 09:29:07 newbrunswick0 systemd[1]: Started Ceph Monitor.
Jul 27 09:29:09 newbrunswick0 docker[1244]: 2021-07-27 09:29:09 /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster...
Jul 27 09:29:17 newbrunswick0 docker[1244]: 2021-07-27 09:29:17 /opt/ceph-container/bin/entrypoint.sh: SUCCESS
Jul 27 09:29:17 newbrunswick0 docker[1244]: exec: PID 71: spawning /usr/bin/ceph-mon --cluster ceph --setuser ceph --setgroup ceph --default-log-to-stderr=true --err-to-stderr=true --default-log-to-file=false --foreground --default-mon-cluster-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false -i newbrunswick0 --mon-data /var/lib/ceph/mon/ceph-newbrunswick0 --public-addr 10.245.0.40
Jul 27 09:29:17 newbrunswick0 docker[1244]: exec: Waiting 71 to quit
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.268+0000 7fb5f1b806c0 0 set uid:gid to 167:167 (ceph:ceph)
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.268+0000 7fb5f1b806c0 0 ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable), process ceph-mon, pid 71
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.268+0000 7fb5f1b806c0 0 pidfile_write: ignore empty --pid-file
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.426+0000 7fb5f1b806c0 0 load: jerasure load: lrc load: isa
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.449+0000 7fb5f1b806c0 0 set rocksdb option compression = kNoCompression
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.449+0000 7fb5f1b806c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.449+0000 7fb5f1b806c0 0 set rocksdb option write_buffer_size = 33554432
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.451+0000 7fb5f1b806c0 0 set rocksdb option compression = kNoCompression
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.451+0000 7fb5f1b806c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.451+0000 7fb5f1b806c0 0 set rocksdb option write_buffer_size = 33554432
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 1 rocksdb: do_open column families: [default]
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: RocksDB version: 6.1.2
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Compile date May 26 2021
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: DB SUMMARY
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: CURRENT file: CURRENT
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: IDENTITY file: IDENTITY
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: MANIFEST file: MANIFEST-170523 size: 4253 Bytes
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-newbrunswick0/store.db dir, Total Num: 2, files: 170596.sst 170597.sst
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-newbrunswick0/store.db: 170594.log size: 2915183 ;
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.error_if_exists: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.create_if_missing: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.paranoid_checks: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.env: 0x55ae0d3df1c0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.info_log: 0x55ae0f4b2dc0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.max_file_opening_threads: 16
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.statistics: (nil)
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.use_fsync: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.max_log_file_size: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.max_manifest_file_size: 1073741824
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.log_file_time_to_roll: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.keep_log_file_num: 1000
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.recycle_log_file_num: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.allow_fallocate: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.allow_mmap_reads: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.allow_mmap_writes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.use_direct_reads: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.create_missing_column_families: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.db_log_dir:
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-newbrunswick0/store.db
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.table_cache_numshardbits: 6
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.max_subcompactions: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.max_background_flushes: -1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.WAL_ttl_seconds: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.WAL_size_limit_MB: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.manifest_preallocation_size: 4194304
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.495+0000 7fb5f1b806c0 4 rocksdb: Options.is_fd_close_on_exec: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.496+0000 7fb5f1b806c0 4 rocksdb: Options.advise_random_on_open: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.496+0000 7fb5f1b806c0 4 rocksdb: Options.db_write_buffer_size: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.496+0000 7fb5f1b806c0 4 rocksdb: Options.write_buffer_manager: 0x55ae0f4715c0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.496+0000 7fb5f1b806c0 4 rocksdb: Options.access_hint_on_compaction_start: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.use_adaptive_mutex: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.rate_limiter: (nil)
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.wal_recovery_mode: 2
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.enable_thread_tracking: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.enable_pipelined_write: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.write_thread_max_yield_usec: 100
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.row_cache: None
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.wal_filter: None
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.avoid_flush_during_recovery: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.allow_ingest_behind: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.preserve_deletes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.two_write_queues: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.manual_wal_flush: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.atomic_flush: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_background_jobs: 2
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_background_compactions: -1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.delayed_write_rate : 16777216
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_total_wal_size: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.stats_dump_period_sec: 600
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.stats_persist_period_sec: 600
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.stats_history_buffer_size: 1048576
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_open_files: -1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bytes_per_sync: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.wal_bytes_per_sync: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_readahead_size: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Compression algorithms supported:
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kZSTDNotFinalCompression supported: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kZSTD supported: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kXpressCompression supported: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kLZ4HCCompression supported: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kLZ4Compression supported: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kBZip2Compression supported: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kZlibCompression supported: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: kSnappyCompression supported: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Fast CRC32 supported: Supported on x86
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-newbrunswick0/store.db/MANIFEST-170523
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]:
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.merge_operator:
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_filter: None
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_filter_factory: None
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.memtable_factory: SkipListFactory
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.table_factory: BlockBasedTable
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ae0f3cd2c0)
Jul 27 09:29:17 newbrunswick0 docker[1244]: cache_index_and_filter_blocks: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: cache_index_and_filter_blocks_with_high_priority: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: pin_l0_filter_and_index_blocks_in_cache: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: pin_top_level_index_and_filter: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: index_type: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: data_block_index_type: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: data_block_hash_table_util_ratio: 0.750000
Jul 27 09:29:17 newbrunswick0 docker[1244]: hash_index_allow_collision: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: checksum: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: no_block_cache: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_cache: 0x55ae0f404610
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_cache_name: BinnedLRUCache
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_cache_options:
Jul 27 09:29:17 newbrunswick0 docker[1244]: capacity : 536870912
Jul 27 09:29:17 newbrunswick0 docker[1244]: num_shard_bits : 4
Jul 27 09:29:17 newbrunswick0 docker[1244]: strict_capacity_limit : 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: high_pri_pool_ratio: 0.000
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_cache_compressed: (nil)
Jul 27 09:29:17 newbrunswick0 docker[1244]: persistent_cache: (nil)
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_size: 4096
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_size_deviation: 10
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_restart_interval: 16
Jul 27 09:29:17 newbrunswick0 docker[1244]: index_block_restart_interval: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: metadata_block_size: 4096
Jul 27 09:29:17 newbrunswick0 docker[1244]: partition_filters: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: use_delta_encoding: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: filter_policy: rocksdb.BuiltinBloomFilter
Jul 27 09:29:17 newbrunswick0 docker[1244]: whole_key_filtering: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: verify_compression: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: read_amp_bytes_per_bit: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: format_version: 2
Jul 27 09:29:17 newbrunswick0 docker[1244]: enable_index_compression: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: block_align: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.write_buffer_size: 33554432
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_write_buffer_number: 2
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression: NoCompression
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression: Disabled
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.prefix_extractor: nullptr
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.num_levels: 7
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.window_bits: -14
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.level: 32767
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.strategy: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compression_opts.enabled: false
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.level0_stop_writes_trigger: 36
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.target_file_size_base: 67108864
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.target_file_size_multiplier: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_compaction_bytes: 1677721600
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.arena_block_size: 4194304
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.disable_auto_compactions: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.table_properties_collectors:
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.inplace_update_support: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.inplace_update_num_locks: 10000
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.memtable_whole_key_filtering: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.memtable_huge_page_size: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.bloom_locality: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.max_successive_merges: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.optimize_filters_for_hits: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.paranoid_file_checks: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.force_consistency_checks: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.report_bg_io_stats: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.497+0000 7fb5f1b806c0 4 rocksdb: Options.ttl: 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.531+0000 7fb5f1b806c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-newbrunswick0/store.db/MANIFEST-170523 succeeded,manifest_file_number is 170523, next_file_number is 170599, last_sequence is 76973525, log_number is 170594,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.531+0000 7fb5f1b806c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 170594
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.532+0000 7fb5f1b806c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627378157533506, "job": 1, "event": "recovery_started", "log_files": [170594]}
Jul 27 09:29:17 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:17.532+0000 7fb5f1b806c0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #170594 mode 2
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.158+0000 7fb5f1b806c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627378158159238, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 170599, "file_size": 2910244, "table_properties": {"data_size": 2904817, "index_size": 4156, "filter_size": 453, "raw_key_size": 3390, "raw_average_key_size": 22, "raw_value_size": 2899505, "raw_average_value_size": 18951, "num_data_blocks": 118, "num_entries": 153, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.158+0000 7fb5f1b806c0 4 rocksdb: [db/version_set.cc:3036] Creating manifest 170600
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.284+0000 7fb5f1b806c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1627378158285006, "job": 1, "event": "recovery_finished"}
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.535+0000 7fb5f1b806c0 4 rocksdb: DB pointer 0x55ae10032400
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.535+0000 7fb5d89e4700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.535+0000 7fb5d89e4700 4 rocksdb: [db/db_impl.cc:778]
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** DB Stats **
Jul 27 09:29:18 newbrunswick0 docker[1244]: Uptime(secs): 1.0 total, 1.0 interval
Jul 27 09:29:18 newbrunswick0 docker[1244]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Jul 27 09:29:18 newbrunswick0 docker[1244]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Jul 27 09:29:18 newbrunswick0 docker[1244]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Jul 27 09:29:18 newbrunswick0 docker[1244]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Jul 27 09:29:18 newbrunswick0 docker[1244]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Jul 27 09:29:18 newbrunswick0 docker[1244]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** Compaction Stats [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jul 27 09:29:18 newbrunswick0 docker[1244]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 27 09:29:18 newbrunswick0 docker[1244]: L0 1/0 2.78 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: L6 2/0 64.54 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Sum 3/0 67.31 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** Compaction Stats [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jul 27 09:29:18 newbrunswick0 docker[1244]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 27 09:29:18 newbrunswick0 docker[1244]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Uptime(secs): 1.0 total, 1.0 interval
Jul 27 09:29:18 newbrunswick0 docker[1244]: Flush(GB): cumulative 0.003, interval 0.003
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(GB): cumulative 0.000, interval 0.000
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(Total Files): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(L0 Files): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(Keys): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Cumulative compaction: 0.00 GB write, 2.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.2 seconds
Jul 27 09:29:18 newbrunswick0 docker[1244]: Interval compaction: 0.00 GB write, 2.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.2 seconds
Jul 27 09:29:18 newbrunswick0 docker[1244]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** File Read Latency Histogram By Level [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** Compaction Stats [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jul 27 09:29:18 newbrunswick0 docker[1244]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 27 09:29:18 newbrunswick0 docker[1244]: L0 1/0 2.78 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: L6 2/0 64.54 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Sum 3/0 67.31 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** Compaction Stats [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jul 27 09:29:18 newbrunswick0 docker[1244]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jul 27 09:29:18 newbrunswick0 docker[1244]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 11.1 0.25 0.00 1 0.250 0 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Uptime(secs): 1.0 total, 0.0 interval
Jul 27 09:29:18 newbrunswick0 docker[1244]: Flush(GB): cumulative 0.003, interval 0.000
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(GB): cumulative 0.000, interval 0.000
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(Total Files): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(L0 Files): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: AddFile(Keys): cumulative 0, interval 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: Cumulative compaction: 0.00 GB write, 2.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.2 seconds
Jul 27 09:29:18 newbrunswick0 docker[1244]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jul 27 09:29:18 newbrunswick0 docker[1244]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Jul 27 09:29:18 newbrunswick0 docker[1244]: ** File Read Latency Histogram By Level [default] **
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.583+0000 7fb5f1b806c0 0 starting mon.newbrunswick0 rank 0 at public addrs [v2:10.245.0.40:3300/0,v1:10.245.0.40:6789/0] at bind addrs [v2:10.245.0.40:3300/0,v1:10.245.0.40:6789/0] mon_data /var/lib/ceph/mon/ceph-newbrunswick0 fsid 7a4265b6-605a-4dbc-9eaa-ec5d9ff62c2a
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.586+0000 7fb5f1b806c0 1 mon.newbrunswick0@-1(???) e2 preinit fsid 7a4265b6-605a-4dbc-9eaa-ec5d9ff62c2a
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.635+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).mds e287 new map
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.635+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).mds e287 print_map
Jul 27 09:29:18 newbrunswick0 docker[1244]: e287
Jul 27 09:29:18 newbrunswick0 docker[1244]: enable_multiple, ever_enabled_multiple: 0,0
Jul 27 09:29:18 newbrunswick0 docker[1244]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:29:18 newbrunswick0 docker[1244]: legacy client fscid: 1
Jul 27 09:29:18 newbrunswick0 docker[1244]: Filesystem 'cephfs' (1)
Jul 27 09:29:18 newbrunswick0 docker[1244]: fs_name cephfs
Jul 27 09:29:18 newbrunswick0 docker[1244]: epoch 287
Jul 27 09:29:18 newbrunswick0 docker[1244]: flags 12
Jul 27 09:29:18 newbrunswick0 docker[1244]: created 2021-04-21T13:23:09.271232+0000
Jul 27 09:29:18 newbrunswick0 docker[1244]: modified 2021-07-27T08:47:41.653179+0000
Jul 27 09:29:18 newbrunswick0 docker[1244]: tableserver 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: root 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: session_timeout 60
Jul 27 09:29:18 newbrunswick0 docker[1244]: session_autoclose 300
Jul 27 09:29:18 newbrunswick0 docker[1244]: max_file_size 1099511627776
Jul 27 09:29:18 newbrunswick0 docker[1244]: min_compat_client 0 (unknown)
Jul 27 09:29:18 newbrunswick0 docker[1244]: last_failure 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: last_failure_osd_epoch 87639
Jul 27 09:29:18 newbrunswick0 docker[1244]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:29:18 newbrunswick0 docker[1244]: max_mds 1
Jul 27 09:29:18 newbrunswick0 docker[1244]: in 0
Jul 27 09:29:18 newbrunswick0 docker[1244]: up {0=3344260}
Jul 27 09:29:18 newbrunswick0 docker[1244]: failed
Jul 27 09:29:18 newbrunswick0 docker[1244]: damaged
Jul 27 09:29:18 newbrunswick0 docker[1244]: stopped
Jul 27 09:29:18 newbrunswick0 docker[1244]: data_pools [1]
Jul 27 09:29:18 newbrunswick0 docker[1244]: metadata_pool 2
Jul 27 09:29:18 newbrunswick0 docker[1244]: inline_data disabled
Jul 27 09:29:18 newbrunswick0 docker[1244]: balancer
Jul 27 09:29:18 newbrunswick0 docker[1244]: standby_count_wanted 1
Jul 27 09:29:18 newbrunswick0 docker[1244]: [mds.newbrunswick2{0:3344260} state up:active seq 10 addr [v2:10.245.0.42:6800/3840387237,v1:10.245.0.42:6801/3840387237]]
Jul 27 09:29:18 newbrunswick0 docker[1244]: Standby daemons:
Jul 27 09:29:18 newbrunswick0 docker[1244]: [mds.newbrunswick3{-1:3310161} state up:standby seq 3 addr [v2:10.245.0.43:6800/3165326145,v1:10.245.0.43:6801/3165326145]]
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.691+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).osd e87647 crush map has features 283675107524608, adjusting msgr requires
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.691+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).osd e87647 crush map has features 283675107524608, adjusting msgr requires
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.691+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).osd e87647 crush map has features 3026702624700514304, adjusting msgr requires
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.691+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(???).osd e87647 crush map has features 283675107524608, adjusting msgr requires
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.726+0000 7fb5f1b806c0 1 mon.newbrunswick0@-1(???).paxosservice(auth 2751..2789) refresh upgraded, format 0 -> 3
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.820+0000 7fb5f1b806c0 0 mon.newbrunswick0@-1(probing) e2 my rank is now 0 (was -1)
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.823+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : mon.newbrunswick0 calling monitor election
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.823+0000 7fb5db1e9700 1 paxos.0).electionLogic(84512) init, last seen epoch 84512
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.839+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.843+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.844+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.872+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.873+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:18 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:18.894+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.079+0000 7fb5db1e9700 1 mon.newbrunswick0@0(electing) e2 collect_metadata : no unique device id for : fallback method has no model nor serial'
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.126+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.244+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.246+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.272+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.295+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.359+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.375+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.396+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.401+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.416+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.436+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.451+0000 7fb5df1f1700 1 mon.newbrunswick0@0(electing) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.511+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : mon.newbrunswick0 is new leader, mons newbrunswick0,newbrunswick1,newbrunswick2 in quorum (ranks 0,1,2)
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.560+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.576+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.597+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.617+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.640+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.675+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.962+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.977+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:19.998+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.018+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.046+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.048+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.074+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.097+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.724+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.763+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.778+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.800+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.820+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:20 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:20.926+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.001+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.034+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.049+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.203+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.235+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.237+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.250+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.272+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.327+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.438+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.474+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.605+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.637+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.643+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.645+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.652+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.672+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.694+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.840+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.844+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.846+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.874+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.875+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:21 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:21.895+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.128+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.245+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.247+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.275+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.296+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.360+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.376+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.397+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.407+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.417+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.439+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.454+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.561+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.578+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.598+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.618+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.641+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.676+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.963+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:22.979+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.000+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.020+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.047+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.048+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.077+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.098+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.724+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.764+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.780+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.801+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.821+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:23 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:23.926+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.002+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.034+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.050+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.204+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.236+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.237+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.252+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.273+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.327+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.359+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : monmap e2: 3 mons at {newbrunswick0=[v2:10.245.0.40:3300/0,v1:10.245.0.40:6789/0],newbrunswick1=[v2:10.245.0.41:3300/0,v1:10.245.0.41:6789/0],newbrunswick2=[v2:10.245.0.42:3300/0,v1:10.245.0.42:6789/0]}
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.359+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : fsmap cephfs:1 {0=newbrunswick2=up:active} 1 up:standby
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.359+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87647: 4 total, 4 up, 4 in
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.439+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.474+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.545+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e434: newbrunswick0(active, since 41m), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.605+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.637+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.3 [v2:10.245.0.41:6800/1799,v1:10.245.0.41:6801/1799] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.3 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 we're forcing failure of osd.3
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : osd.3 failed (root=default,host=newbrunswick1) (connection refused reported by osd.0)
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.3 [v2:10.245.0.41:6800/1799,v1:10.245.0.41:6801/1799] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.3 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.3 [v2:10.245.0.41:6800/1799,v1:10.245.0.41:6801/1799] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.3 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.1 [v2:10.245.0.40:6800/2021,v1:10.245.0.40:6801/2021] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.1 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 we're forcing failure of osd.1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : osd.1 failed (root=default,host=newbrunswick0) (connection refused reported by osd.0)
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.1 [v2:10.245.0.40:6800/2021,v1:10.245.0.40:6801/2021] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.1 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.1 [v2:10.245.0.40:6800/2021,v1:10.245.0.40:6801/2021] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.1 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 prepare_failure osd.2 [v2:10.245.0.42:6802/1902,v1:10.245.0.42:6803/1902] from osd.0 is reporting failure:1
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : osd.2 reported immediately failed by osd.0
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87647 we're forcing failure of osd.2
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.641+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : osd.2 failed (root=default,host=newbrunswick2) (connection refused reported by osd.0)
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.643+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.645+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.653+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.673+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.695+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.840+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.844+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.845+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.874+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.875+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:24.896+0000 7fb5df1f1700 1 mon.newbrunswick0@0(leader) e2 handle_auth_request failed to assign global_id
Jul 27 09:29:25 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:25.370+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87647 do_prune osdmap full prune enabled
Jul 27 09:29:25 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:25.372+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check failed: 3 osds down (OSD_DOWN)
Jul 27 09:29:25 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:25.372+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check failed: 3 hosts (3 osds) down (OSD_HOST_DOWN)
Jul 27 09:29:25 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:25.694+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : overall HEALTH_OK
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.583+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87647 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.585+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87647 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.586+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87648 e87648: 4 total, 1 up, 4 in
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.765+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87648: 4 total, 1 up, 4 in
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.767+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd stat", "format": "json"} v 0) v1
Jul 27 09:29:26 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:26.767+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='client.? 10.245.0.42:0/3329866461' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch
Jul 27 09:29:28 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:28.818+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87648 _set_new_cache_sizes cache_size:1019917737 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:29:29 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:29.658+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87648 do_prune osdmap full prune enabled
Jul 27 09:29:31 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:31.011+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]} v 0) v1
Jul 27 09:29:31 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:31.011+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.1 [v2:10.245.0.40:6800/2004,v1:10.245.0.40:6801/2004]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]}]: dispatch
Jul 27 09:29:32 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:32.539+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : Standby manager daemon newbrunswick2 restarted
Jul 27 09:29:32 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:32.539+0000 7fb5db1e9700 0 log_channel(cluster) log [DBG] : Standby manager daemon newbrunswick2 started
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.470+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87649 e87649: 4 total, 1 up, 4 in
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.608+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : daemon mds.newbrunswick2 restarted
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.609+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).mds e287 fail_mds_gid 3344260 mds.newbrunswick2 role 0
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.609+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87649 do_prune osdmap full prune enabled
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.609+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87649 encode_pending skipping prime_pg_temp; mapping job did not start
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.810+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87649: 4 total, 1 up, 4 in
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.846+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["3"]} v 0) v1
Jul 27 09:29:33 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:33.846+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["3"]}]: dispatch
Jul 27 09:29:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:34.035+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).mds e287 taking over failed mds.0 with 3364108/newbrunswick2 [v2:10.245.0.42:6800/3891958835,v1:10.245.0.42:6801/3891958835]
Jul 27 09:29:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:34.035+0000 7fb5dd9ee700 0 log_channel(cluster) log [INF] : Standby daemon mds.newbrunswick2 assigned to filesystem cephfs as rank 0
Jul 27 09:29:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:34.035+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check failed: 1 filesystem is degraded (FS_DEGRADED)
Jul 27 09:29:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:34.730+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["2"]} v 0) v1
Jul 27 09:29:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:34.731+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["2"]}]: dispatch
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.520+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87650 e87650: 4 total, 1 up, 4 in
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.849+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : Active manager daemon newbrunswick0 restarted
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.849+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).mgr e435 prepare_beacon: waiting for osdmon writeable to blacklist old instance.
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.849+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : Standby manager daemon newbrunswick1 restarted
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.849+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : Standby manager daemon newbrunswick1 started
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.852+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e435: newbrunswick0(active, since 42m), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:36 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:36.854+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87650: 4 total, 1 up, 4 in
Jul 27 09:29:37 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:37.520+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87650 do_prune osdmap full prune enabled
Jul 27 09:29:38 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:38.401+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e288 new map
Jul 27 09:29:38 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:38.401+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e288 print_map
Jul 27 09:29:38 newbrunswick0 docker[1244]: e288
Jul 27 09:29:38 newbrunswick0 docker[1244]: enable_multiple, ever_enabled_multiple: 0,0
Jul 27 09:29:38 newbrunswick0 docker[1244]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:29:38 newbrunswick0 docker[1244]: legacy client fscid: 1
Jul 27 09:29:38 newbrunswick0 docker[1244]: Filesystem 'cephfs' (1)
Jul 27 09:29:38 newbrunswick0 docker[1244]: fs_name cephfs
Jul 27 09:29:38 newbrunswick0 docker[1244]: epoch 288
Jul 27 09:29:38 newbrunswick0 docker[1244]: flags 12
Jul 27 09:29:38 newbrunswick0 docker[1244]: created 2021-04-21T13:23:09.271232+0000
Jul 27 09:29:38 newbrunswick0 docker[1244]: modified 2021-07-27T09:29:34.037104+0000
Jul 27 09:29:38 newbrunswick0 docker[1244]: tableserver 0
Jul 27 09:29:38 newbrunswick0 docker[1244]: root 0
Jul 27 09:29:38 newbrunswick0 docker[1244]: session_timeout 60
Jul 27 09:29:38 newbrunswick0 docker[1244]: session_autoclose 300
Jul 27 09:29:38 newbrunswick0 docker[1244]: max_file_size 1099511627776
Jul 27 09:29:38 newbrunswick0 docker[1244]: min_compat_client 0 (unknown)
Jul 27 09:29:38 newbrunswick0 docker[1244]: last_failure 0
Jul 27 09:29:38 newbrunswick0 docker[1244]: last_failure_osd_epoch 87650
Jul 27 09:29:38 newbrunswick0 docker[1244]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:29:38 newbrunswick0 docker[1244]: max_mds 1
Jul 27 09:29:38 newbrunswick0 docker[1244]: in 0
Jul 27 09:29:38 newbrunswick0 docker[1244]: up {0=3364108}
Jul 27 09:29:38 newbrunswick0 docker[1244]: failed
Jul 27 09:29:38 newbrunswick0 docker[1244]: damaged
Jul 27 09:29:38 newbrunswick0 docker[1244]: stopped
Jul 27 09:29:38 newbrunswick0 docker[1244]: data_pools [1]
Jul 27 09:29:38 newbrunswick0 docker[1244]: metadata_pool 2
Jul 27 09:29:38 newbrunswick0 docker[1244]: inline_data disabled
Jul 27 09:29:38 newbrunswick0 docker[1244]: balancer
Jul 27 09:29:38 newbrunswick0 docker[1244]: standby_count_wanted 1
Jul 27 09:29:38 newbrunswick0 docker[1244]: [mds.newbrunswick2{0:3364108} state up:replay seq 2 addr [v2:10.245.0.42:6800/3891958835,v1:10.245.0.42:6801/3891958835]]
Jul 27 09:29:38 newbrunswick0 docker[1244]: Standby daemons:
Jul 27 09:29:38 newbrunswick0 docker[1244]: [mds.newbrunswick3{-1:3310161} state up:standby seq 3 addr [v2:10.245.0.43:6800/3165326145,v1:10.245.0.43:6801/3165326145]]
Jul 27 09:29:38 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:38.406+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mds.? [v2:10.245.0.42:6800/3891958835,v1:10.245.0.42:6801/3891958835] up:boot
Jul 27 09:29:38 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:38.408+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : fsmap cephfs:1/1 {0=newbrunswick2=up:replay} 1 up:standby
Jul 27 09:29:40 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:40.838+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87651 e87651: 4 total, 1 up, 4 in
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.130+0000 7fb5d99e6700 0 log_channel(audit) log [INF] : from='osd.1 [v2:10.245.0.40:6800/2004,v1:10.245.0.40:6801/2004]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]}]': finished
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.130+0000 7fb5d99e6700 0 log_channel(audit) log [INF] : from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["3"]}]': finished
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.132+0000 7fb5d99e6700 0 log_channel(audit) log [INF] : from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["2"]}]': finished
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.132+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87651: 4 total, 1 up, 4 in
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.133+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : Active manager daemon newbrunswick0 restarted
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.133+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87651 do_prune osdmap full prune enabled
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.133+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87651 encode_pending skipping prime_pg_temp; mapping job 0x55ae101dc1e0 did not complete, 9 left
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.242+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : Activating manager daemon newbrunswick0
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.244+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e436: newbrunswick0(active, since 42m), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.245+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 1, "weight":0.0977, "args": ["host=newbrunswick0", "root=default"]} v 0) v1
Jul 27 09:29:41 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:41.245+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.1 [v2:10.245.0.40:6800/2004,v1:10.245.0.40:6801/2004]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0977, "args": ["host=newbrunswick0", "root=default"]}]: dispatch
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.573+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87652 e87652: 4 total, 1 up, 4 in
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.693+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).osd e87652 create-or-move crush item name 'osd.1' initial_weight 0.0977 at location {host=newbrunswick0,root=default}
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.694+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87652: 4 total, 1 up, 4 in
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.928+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 3, "weight":0.0977, "args": ["host=newbrunswick1", "root=default"]} v 0) v1
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.928+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0977, "args": ["host=newbrunswick1", "root=default"]}]: dispatch
Jul 27 09:29:42 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:42.928+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader).osd e87652 create-or-move crush item name 'osd.3' initial_weight 0.0977 at location {host=newbrunswick1,root=default}
Jul 27 09:29:43 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:43.573+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87652 do_prune osdmap full prune enabled
Jul 27 09:29:43 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:43.573+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: 2 osds down (OSD_DOWN)
Jul 27 09:29:43 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:43.573+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: 2 hosts (2 osds) down (OSD_HOST_DOWN)
Jul 27 09:29:43 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:43.742+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 2, "weight":0.0977, "args": ["host=newbrunswick2", "root=default"]} v 0) v1
Jul 27 09:29:43 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:43.742+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0977, "args": ["host=newbrunswick2", "root=default"]}]: dispatch
Jul 27 09:29:44 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:44.958+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e437: newbrunswick0(active, starting, since 3s), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.836+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mon metadata", "id": "newbrunswick0"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.838+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mon metadata", "id": "newbrunswick0"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.839+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mon metadata", "id": "newbrunswick1"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.840+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mon metadata", "id": "newbrunswick1"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.841+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mon metadata", "id": "newbrunswick2"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.842+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mon metadata", "id": "newbrunswick2"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.843+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mds metadata", "who": "newbrunswick3"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.844+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mds metadata", "who": "newbrunswick3"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.845+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).mds e288 all = 0
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.846+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mds metadata", "who": "newbrunswick2"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.847+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mds metadata", "who": "newbrunswick2"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.848+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).mds e288 all = 0
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.849+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mgr metadata", "who": "newbrunswick0", "id": "newbrunswick0"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.850+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mgr metadata", "who": "newbrunswick0", "id": "newbrunswick0"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.851+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mgr metadata", "who": "newbrunswick1", "id": "newbrunswick1"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.852+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mgr metadata", "who": "newbrunswick1", "id": "newbrunswick1"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.853+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mgr metadata", "who": "newbrunswick2", "id": "newbrunswick2"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.853+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mgr metadata", "who": "newbrunswick2", "id": "newbrunswick2"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.853+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.854+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.855+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 1} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.855+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.855+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.855+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.856+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.856+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.857+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mds metadata"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.857+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mds metadata"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.858+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).mds e288 all = 1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.858+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.858+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.859+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mon metadata"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.859+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mon metadata"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.864+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key ls"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.864+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key ls"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.866+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/crash/crash/2021-07-27T08:44:11.383348Z_59f6affa-730f-4349-812b-1e57f48441c8"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.866+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/crash/crash/2021-07-27T08:44:11.383348Z_59f6affa-730f-4349-812b-1e57f48441c8"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.867+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v1"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.867+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v1"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.869+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.869+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/accessdb_v2"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.871+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/dashboard/crt"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.871+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.872+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/dashboard/jwt_secret"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.872+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/jwt_secret"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.874+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/dashboard/key"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.874+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.875+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.876+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.877+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/progress/completed"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.877+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/progress/completed"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.878+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/telemetry/report_id"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.879+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.880+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "config-key get", "key": "mgr/telemetry/salt"} v 0) v1
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.880+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch
Jul 27 09:29:45 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:45.884+0000 7fb5db1e9700 0 log_channel(cluster) log [INF] : Manager daemon newbrunswick0 is now available
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.565+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87653 e87653: 4 total, 2 up, 4 in
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.710+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : osd.1 [v2:10.245.0.40:6800/2004,v1:10.245.0.40:6801/2004] boot
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.710+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).osd e87653 create-or-move crush item name 'osd.2' initial_weight 0.0977 at location {host=newbrunswick2,root=default}
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.710+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87653: 4 total, 2 up, 4 in
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.933+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 1} v 0) v1
Jul 27 09:29:46 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:46.933+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.489+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/mirror_snapshot_schedule"} v 0) v1
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.489+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/mirror_snapshot_schedule"}]: dispatch
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.493+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/trash_purge_schedule"} v 0) v1
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.493+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/trash_purge_schedule"}]: dispatch
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.513+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1} v 0) v1
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.513+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.517+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"} v 0) v1
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.517+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch
Jul 27 09:29:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:47.565+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87653 do_prune osdmap full prune enabled
Jul 27 09:29:48 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:48.146+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e438: newbrunswick0(active, since 6s), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.135+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 39 pgs inactive, 84 pgs peering (PG_AVAILABILITY)
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.135+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 24158/129252 objects degraded (18.691%), 20 pgs degraded (PG_DEGRADED)
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.383+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87654 e87654: 4 total, 3 up, 4 in
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.418+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : osd.3 [v2:10.245.0.41:6800/1744,v1:10.245.0.41:6801/1744] boot
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.418+0000 7fb5d99e6700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.418+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87654: 4 total, 3 up, 4 in
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.553+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
Jul 27 09:29:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:49.553+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
Jul 27 09:29:50 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:50.383+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87654 do_prune osdmap full prune enabled
Jul 27 09:29:52 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:52.648+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mgrmap e439: newbrunswick0(active, since 11s), standbys: newbrunswick1, newbrunswick2
Jul 27 09:29:54 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:54.038+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: 1 osds down (OSD_DOWN)
Jul 27 09:29:54 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:54.038+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: 1 host (1 osds) down (OSD_HOST_DOWN)
Jul 27 09:29:54 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:54.258+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87655 e87655: 4 total, 3 up, 4 in
Jul 27 09:29:54 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:54.312+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87655: 4 total, 3 up, 4 in
Jul 27 09:29:55 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:55.259+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87655 do_prune osdmap full prune enabled
Jul 27 09:29:55 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:55.259+0000 7fb5dd9ee700 0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
Jul 27 09:29:55 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:55.259+0000 7fb5dd9ee700 0 log_channel(cluster) log [INF] : Health check cleared: OSD_HOST_DOWN (was: 1 host (1 osds) down)
Jul 27 09:29:58 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:58.136+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 17 pgs inactive, 101 pgs peering (PG_AVAILABILITY)
Jul 27 09:29:58 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:58.136+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1373/129252 objects degraded (1.062%), 3 pgs degraded (PG_DEGRADED)
Jul 27 09:29:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.458+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87656 e87656: 4 total, 4 up, 4 in
Jul 27 09:29:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.552+0000 7fb5d99e6700 0 log_channel(cluster) log [INF] : osd.2 [v2:10.245.0.42:6802/1897,v1:10.245.0.42:6803/1897] boot
Jul 27 09:29:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.552+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87656: 4 total, 4 up, 4 in
Jul 27 09:29:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.626+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
Jul 27 09:29:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.626+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN 1 filesystem is degraded; Reduced data availability: 39 pgs inactive, 84 pgs peering; Degraded data redundancy: 24158/129252 objects degraded (18.691%), 20 pgs degraded
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : [WRN] FS_DEGRADED: 1 filesystem is degraded
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : fs cephfs is degraded
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : [WRN] PG_AVAILABILITY: Reduced data availability: 39 pgs inactive, 84 pgs peering
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.2 is stuck inactive for 42m, current state undersized+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.4 is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.5 is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.10 is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.12 is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.15 is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.16 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.17 is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.18 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.19 is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.1a is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.1b is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.1c is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 1.1e is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.0 is stuck peering for 45m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.1 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.7 is stuck inactive for 18h, current state undersized+degraded+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.10 is stuck peering for 45m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.11 is stuck inactive for 42m, current state undersized+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.12 is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.13 is stuck inactive for 18h, current state undersized+degraded+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.14 is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.16 is stuck peering for 43m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.17 is stuck peering for 45m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.18 is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.1a is stuck peering for 45m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.1b is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:29:59.999+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.1c is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.1d is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.10 is stuck peering for 3m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.11 is stuck peering for 26m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.13 is stuck peering for 116s, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.15 is stuck peering for 8m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.17 is stuck peering for 3m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.18 is stuck peering for 2m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1a is stuck peering for 7m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1b is stuck peering for 4m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1d is stuck peering for 6m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 4.1 is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.0 is stuck inactive for 18h, current state undersized+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.3 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.5 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.6 is stuck peering for 45m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.8 is stuck inactive for 42m, current state peering, last acting [0,1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.9 is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.a is stuck peering for 45m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.c is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.d is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.e is stuck peering for 42m, current state peering, last acting [1]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.12 is stuck peering for 42m, current state peering, last acting [1,0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 17.15 is stuck inactive for 45m, current state undersized+peered, last acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : [WRN] PG_DEGRADED: Degraded data redundancy: 24158/129252 objects degraded (18.691%), 20 pgs degraded
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.7 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 2.13 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.3 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.6 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.12 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.14 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.16 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.19 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1c is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 3.1f is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 4.0 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 8.3 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 10.1 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 10.2 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 11.2 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 12.0 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 12.1 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 13.2 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.000+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : pg 15.2 is undersized+degraded+peered, acting [0]
Jul 27 09:30:00 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:00.458+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87656 do_prune osdmap full prune enabled
Jul 27 09:30:02 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:02.461+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87657 e87657: 4 total, 4 up, 4 in
Jul 27 09:30:02 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:02.681+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87657: 4 total, 4 up, 4 in
Jul 27 09:30:03 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:03.461+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87657 do_prune osdmap full prune enabled
Jul 27 09:30:07 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:07.206+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 49 pgs peering (PG_AVAILABILITY)
Jul 27 09:30:07 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:07.206+0000 7fb5dd9ee700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 30/129252 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED)
Jul 27 09:30:10 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:10.080+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87658 e87658: 4 total, 4 up, 4 in
Jul 27 09:30:10 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:10.225+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87658: 4 total, 4 up, 4 in
Jul 27 09:30:10 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:10.511+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader).mds e288 prepare_beacon: marking rank 0 damaged
Jul 27 09:30:10 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:10.511+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).osd e87658 do_prune osdmap full prune enabled
Jul 27 09:30:11 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:11.080+0000 7fb5dd9ee700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)
Jul 27 09:30:11 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:11.080+0000 7fb5dd9ee700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE)
Jul 27 09:30:14 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:14.111+0000 7fb5dd9ee700 0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 49 pgs peering)
Jul 27 09:30:14 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:14.111+0000 7fb5dd9ee700 0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/129252 objects degraded (0.023%), 2 pgs degraded)
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.652+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e289 new map
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.653+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e289 print_map
Jul 27 09:30:15 newbrunswick0 docker[1244]: e289
Jul 27 09:30:15 newbrunswick0 docker[1244]: enable_multiple, ever_enabled_multiple: 0,0
Jul 27 09:30:15 newbrunswick0 docker[1244]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:30:15 newbrunswick0 docker[1244]: legacy client fscid: 1
Jul 27 09:30:15 newbrunswick0 docker[1244]: Filesystem 'cephfs' (1)
Jul 27 09:30:15 newbrunswick0 docker[1244]: fs_name cephfs
Jul 27 09:30:15 newbrunswick0 docker[1244]: epoch 289
Jul 27 09:30:15 newbrunswick0 docker[1244]: flags 12
Jul 27 09:30:15 newbrunswick0 docker[1244]: created 2021-04-21T13:23:09.271232+0000
Jul 27 09:30:15 newbrunswick0 docker[1244]: modified 2021-07-27T09:30:11.081490+0000
Jul 27 09:30:15 newbrunswick0 docker[1244]: tableserver 0
Jul 27 09:30:15 newbrunswick0 docker[1244]: root 0
Jul 27 09:30:15 newbrunswick0 docker[1244]: session_timeout 60
Jul 27 09:30:15 newbrunswick0 docker[1244]: session_autoclose 300
Jul 27 09:30:15 newbrunswick0 docker[1244]: max_file_size 1099511627776
Jul 27 09:30:15 newbrunswick0 docker[1244]: min_compat_client 0 (unknown)
Jul 27 09:30:15 newbrunswick0 docker[1244]: last_failure 0
Jul 27 09:30:15 newbrunswick0 docker[1244]: last_failure_osd_epoch 87659
Jul 27 09:30:15 newbrunswick0 docker[1244]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:30:15 newbrunswick0 docker[1244]: max_mds 1
Jul 27 09:30:15 newbrunswick0 docker[1244]: in 0
Jul 27 09:30:15 newbrunswick0 docker[1244]: up {}
Jul 27 09:30:15 newbrunswick0 docker[1244]: failed
Jul 27 09:30:15 newbrunswick0 docker[1244]: damaged 0
Jul 27 09:30:15 newbrunswick0 docker[1244]: stopped
Jul 27 09:30:15 newbrunswick0 docker[1244]: data_pools [1]
Jul 27 09:30:15 newbrunswick0 docker[1244]: metadata_pool 2
Jul 27 09:30:15 newbrunswick0 docker[1244]: inline_data disabled
Jul 27 09:30:15 newbrunswick0 docker[1244]: balancer
Jul 27 09:30:15 newbrunswick0 docker[1244]: standby_count_wanted 1
Jul 27 09:30:15 newbrunswick0 docker[1244]: Standby daemons:
Jul 27 09:30:15 newbrunswick0 docker[1244]: [mds.newbrunswick3{-1:3310161} state up:standby seq 3 addr [v2:10.245.0.43:6800/3165326145,v1:10.245.0.43:6801/3165326145]]
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.654+0000 7fb5d99e6700 1 mon.newbrunswick0@0(leader).osd e87659 e87659: 4 total, 4 up, 4 in
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.785+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : osdmap e87659: 4 total, 4 up, 4 in
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.786+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mds.? [v2:10.245.0.42:6800/3891958835,v1:10.245.0.42:6801/3891958835] down:damaged
Jul 27 09:30:15 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:15.787+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : fsmap cephfs:0/1 1 up:standby, 1 damaged
Jul 27 09:30:19 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:19.042+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020052669 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.928+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e290 new map
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.928+0000 7fb5d99e6700 0 mon.newbrunswick0@0(leader).mds e290 print_map
Jul 27 09:30:22 newbrunswick0 docker[1244]: e290
Jul 27 09:30:22 newbrunswick0 docker[1244]: enable_multiple, ever_enabled_multiple: 0,0
Jul 27 09:30:22 newbrunswick0 docker[1244]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:30:22 newbrunswick0 docker[1244]: legacy client fscid: 1
Jul 27 09:30:22 newbrunswick0 docker[1244]: Filesystem 'cephfs' (1)
Jul 27 09:30:22 newbrunswick0 docker[1244]: fs_name cephfs
Jul 27 09:30:22 newbrunswick0 docker[1244]: epoch 289
Jul 27 09:30:22 newbrunswick0 docker[1244]: flags 12
Jul 27 09:30:22 newbrunswick0 docker[1244]: created 2021-04-21T13:23:09.271232+0000
Jul 27 09:30:22 newbrunswick0 docker[1244]: modified 2021-07-27T09:30:11.081490+0000
Jul 27 09:30:22 newbrunswick0 docker[1244]: tableserver 0
Jul 27 09:30:22 newbrunswick0 docker[1244]: root 0
Jul 27 09:30:22 newbrunswick0 docker[1244]: session_timeout 60
Jul 27 09:30:22 newbrunswick0 docker[1244]: session_autoclose 300
Jul 27 09:30:22 newbrunswick0 docker[1244]: max_file_size 1099511627776
Jul 27 09:30:22 newbrunswick0 docker[1244]: min_compat_client 0 (unknown)
Jul 27 09:30:22 newbrunswick0 docker[1244]: last_failure 0
Jul 27 09:30:22 newbrunswick0 docker[1244]: last_failure_osd_epoch 87659
Jul 27 09:30:22 newbrunswick0 docker[1244]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
Jul 27 09:30:22 newbrunswick0 docker[1244]: max_mds 1
Jul 27 09:30:22 newbrunswick0 docker[1244]: in 0
Jul 27 09:30:22 newbrunswick0 docker[1244]: up {}
Jul 27 09:30:22 newbrunswick0 docker[1244]: failed
Jul 27 09:30:22 newbrunswick0 docker[1244]: damaged 0
Jul 27 09:30:22 newbrunswick0 docker[1244]: stopped
Jul 27 09:30:22 newbrunswick0 docker[1244]: data_pools [1]
Jul 27 09:30:22 newbrunswick0 docker[1244]: metadata_pool 2
Jul 27 09:30:22 newbrunswick0 docker[1244]: inline_data disabled
Jul 27 09:30:22 newbrunswick0 docker[1244]: balancer
Jul 27 09:30:22 newbrunswick0 docker[1244]: standby_count_wanted 1
Jul 27 09:30:22 newbrunswick0 docker[1244]: Standby daemons:
Jul 27 09:30:22 newbrunswick0 docker[1244]: [mds.newbrunswick3{-1:3310161} state up:standby seq 3 addr [v2:10.245.0.43:6800/3165326145,v1:10.245.0.43:6801/3165326145]]
Jul 27 09:30:22 newbrunswick0 docker[1244]: [mds.newbrunswick2{-1:3364295} state up:standby seq 2 addr [v2:10.245.0.42:6800/1642389801,v1:10.245.0.42:6801/1642389801]]
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.940+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : mds.? [v2:10.245.0.42:6800/1642389801,v1:10.245.0.42:6801/1642389801] up:boot
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.942+0000 7fb5d99e6700 0 log_channel(cluster) log [DBG] : fsmap cephfs:0/1 2 up:standby, 1 damaged
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.995+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix": "mds metadata", "who": "newbrunswick2"} v 0) v1
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.995+0000 7fb5db1e9700 0 log_channel(audit) log [DBG] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix": "mds metadata", "who": "newbrunswick2"}]: dispatch
Jul 27 09:30:22 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:22.995+0000 7fb5db1e9700 1 mon.newbrunswick0@0(leader).mds e290 all = 0
Jul 27 09:30:24 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:24.043+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054700 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:29 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:29.044+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:34 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:34.046+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:39 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:39.289+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:44 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:44.291+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:47.492+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/mirror_snapshot_schedule"} v 0) v1
Jul 27 09:30:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:47.492+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/mirror_snapshot_schedule"}]: dispatch
Jul 27 09:30:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:47.497+0000 7fb5db1e9700 0 mon.newbrunswick0@0(leader) e2 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/trash_purge_schedule"} v 0) v1
Jul 27 09:30:47 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:47.497+0000 7fb5db1e9700 0 log_channel(audit) log [INF] : from='mgr.3364124 10.245.0.40:0/55' entity='mgr.newbrunswick0' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/newbrunswick0/trash_purge_schedule"}]: dispatch
Jul 27 09:30:49 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:49.292+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:54 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:54.293+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
Jul 27 09:30:59 newbrunswick0 docker[1244]: debug 2021-07-27T09:30:59.295+0000 7fb5dd9ee700 1 mon.newbrunswick0@0(leader).osd e87659 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 373293056 full_alloc: 373293056 kv_alloc: 272629760
(3-3/12)