2019-03-06 15:27:55.425 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1665) extend_lease now+5 (2019-03-06 15:28:00.429596) 2019-03-06 15:27:55.425 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1665 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8f8b00 2019-03-06 15:27:55.425 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1665 fc 1005 pn 0 opn 0) v4 -- 0x56443e8f8b00 con 0x56443e61e900 2019-03-06 15:27:55.425 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1665 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440cd7700 2019-03-06 15:27:55.425 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1665 fc 1005 pn 0 opn 0) v4 -- 0x564440cd7700 con 0x56443e61ed80 2019-03-06 15:27:55.429 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1665) finish_round 2019-03-06 15:27:55.429 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1665) finish_round waiting_for_acting 2019-03-06 15:27:55.429 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1019) _active 2019-03-06 15:27:55.429 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1019) _active creating new pending 2019-03-06 15:27:55.429 7f72c901c700 10 mon.a@0(leader).mgrstat 1019 2019-03-06 15:27:55.429 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:27:55.429 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1665) finish_round waiting_for_readable 2019-03-06 15:27:55.429 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1665) finish_round waiting_for_writeable 2019-03-06 15:27:55.429 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1665) finish_round done w/ waiters, state active 2019-03-06 15:27:55.461 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5776 ==== paxos(lease_ack lc 1665 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440cd7700 con 0x56443e61ed80 2019-03-06 15:27:55.461 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:27:55.461 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:55.461 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow all 2019-03-06 15:27:55.461 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow all 2019-03-06 15:27:55.461 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1665) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:27:55.461 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4856 ==== paxos(lease_ack lc 1665 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e8f8b00 con 0x56443e61e900 2019-03-06 15:27:55.461 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:55.461 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:55.461 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow all 2019-03-06 15:27:55.461 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:55.461 7f72cc823700 20 allow all 2019-03-06 15:27:55.461 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1665) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:27:56.601 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 315 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x564440bbfe40 con 0x5644402c0400 2019-03-06 15:27:56.601 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:27:56.601 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:56.601 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440bbfe40 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:27:56.601 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) is_readable = 1 - now=2019-03-06 15:27:56.603612 lease_expire=2019-03-06 15:28:00.429596 has v0 lc 1665 2019-03-06 15:27:56.601 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:27:56.601 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:27:56.601 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:27:56.601 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:27:56.601 7f72cc823700 20 match 2019-03-06 15:27:56.601 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:27:56.601 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:27:56.613 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4857 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1494 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56444207c840 con 0x56443e61e900 2019-03-06 15:27:56.613 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:56.613 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:56.613 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:56.613 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.613 7f72cc823700 20 allow all 2019-03-06 15:27:56.613 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:27:56.613 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:56.613 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.613 7f72cc823700 20 allow all 2019-03-06 15:27:56.613 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:27:56.613 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:27:56.613 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56444199b600 from v2:192.168.178.28:40578/0 2019-03-06 15:27:56.617 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa89b00 for mgr.? 2019-03-06 15:27:56.617 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:56.617 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56444199b600 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444049a600 2019-03-06 15:27:56.617 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) is_readable = 1 - now=2019-03-06 15:27:56.617938 lease_expire=2019-03-06 15:28:00.429596 has v0 lc 1665 2019-03-06 15:27:56.617 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:27:56.617 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1494) v3 -- 0x56443ec0a300 con 0x56443e61e900 2019-03-06 15:27:56.617 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:27:56.617 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:27:56.617 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:27:56.617 7f72cc823700 20 match 2019-03-06 15:27:56.617 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:27:56.617 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:27:56.617 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:27:56.725 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:27:56.725 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2212.59s 2019-03-06 15:27:56.725 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:27:56.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) is_readable = 1 - now=2019-03-06 15:27:56.730190 lease_expire=2019-03-06 15:28:00.429596 has v0 lc 1665 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:27:56.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) is_readable = 1 - now=2019-03-06 15:27:56.730320 lease_expire=2019-03-06 15:28:00.429596 has v0 lc 1665 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).log v516 log 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..516) maybe_trim trim_to 16 would only trim 15 < paxos_service_trim_min 250 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:27:56.729 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1019) maybe_trim trim_to 1014 would only trim 244 < paxos_service_trim_min 250 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:27:56.729 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:27:56.729 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:27:56.729 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f7840 con 0x5644402c0400 2019-03-06 15:27:56.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 316 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f849200 con 0x5644402c0400 2019-03-06 15:27:56.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:27:56.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:56.813 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1019) dispatch 0x56443f849200 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:27:56.813 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) is_readable = 1 - now=2019-03-06 15:27:56.814444 lease_expire=2019-03-06 15:28:00.429596 has v0 lc 1665 2019-03-06 15:27:56.813 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:27:56.813 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:27:56.813 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804232 }, { "osd": 1, "seq": 42949673413 }, { "osd": 2, "seq": 60129542592 }, { "osd": 3, "seq": 73014444476 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:27:56.813 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:27:56.813 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:27:56.813 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1019) setting proposal_timer 0x56443ecbb230 with delay of 0.05 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1019) propose_pending 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader).mgrstat 1020 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:27:56.865 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1665) queue_pending_finisher 0x56443d581160 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1665) trigger_propose active, proposing now 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1665) propose_pending 1666 6340 bytes 2019-03-06 15:27:56.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) begin for 1666 6340 bytes 2019-03-06 15:27:56.897 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) sending begin to mon.1 2019-03-06 15:27:56.897 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1665 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e9f22c0 2019-03-06 15:27:56.897 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1665 fc 0 pn 300 opn 0) v4 -- 0x56443e9f22c0 con 0x56443e61e900 2019-03-06 15:27:56.897 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) sending begin to mon.2 2019-03-06 15:27:56.897 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1665 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f362c0 2019-03-06 15:27:56.897 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1665 fc 0 pn 300 opn 0) v4 -- 0x564440f362c0 con 0x56443e61ed80 2019-03-06 15:27:56.933 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5777 ==== paxos(accept lc 1665 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440f362c0 con 0x56443e61ed80 2019-03-06 15:27:56.933 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:27:56.933 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:56.933 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow all 2019-03-06 15:27:56.933 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow all 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) handle_accept paxos(accept lc 1665 fc 0 pn 300 opn 0) v4 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) now 0,2 have accepted 2019-03-06 15:27:56.933 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4858 ==== paxos(accept lc 1665 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e9f22c0 con 0x56443e61e900 2019-03-06 15:27:56.933 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:56.933 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:56.933 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow all 2019-03-06 15:27:56.933 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:56.933 7f72cc823700 20 allow all 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) handle_accept paxos(accept lc 1665 fc 0 pn 300 opn 0) v4 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) now 0,1,2 have accepted 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) got majority, committing, done with update 2019-03-06 15:27:56.933 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1665) commit_start 1666 2019-03-06 15:27:57.273 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1665) commit_finish 1666 2019-03-06 15:27:57.273 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1666) sending commit to mon.1 2019-03-06 15:27:57.273 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1666 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f383c0 2019-03-06 15:27:57.273 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1666 fc 0 pn 300 opn 0) v4 -- 0x564440f383c0 con 0x56443e61e900 2019-03-06 15:27:57.273 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1666) sending commit to mon.2 2019-03-06 15:27:57.273 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1666 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f38680 2019-03-06 15:27:57.273 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1666 fc 0 pn 300 opn 0) v4 -- 0x564440f38680 con 0x56443e61ed80 2019-03-06 15:27:57.273 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) refresh 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos version 516 summary v 516 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:27:57.277 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:27:57.285 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:27:57.285 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:27:57.301 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:27:57.301 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:27:57.301 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) refresh 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).mgrstat 1020 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1020 service_map e2 0progress events 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:27:57.301 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:27:57.301 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232ae00 con 0x5644402c0400 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1666) commit_proposal 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) _active - not active 2019-03-06 15:27:57.305 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1666) extend_lease now+5 (2019-03-06 15:28:02.307114) 2019-03-06 15:27:57.305 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1666 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440ee9b80 2019-03-06 15:27:57.305 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1666 fc 1005 pn 0 opn 0) v4 -- 0x564440ee9b80 con 0x56443e61e900 2019-03-06 15:27:57.305 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1666 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440ba6840 2019-03-06 15:27:57.305 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1666 fc 1005 pn 0 opn 0) v4 -- 0x564440ba6840 con 0x56443e61ed80 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1666) finish_round 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1666) finish_round waiting_for_acting 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) _active 2019-03-06 15:27:57.305 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1020) _active creating new pending 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).mgrstat 1020 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1666) finish_round waiting_for_readable 2019-03-06 15:27:57.305 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1666) finish_round waiting_for_writeable 2019-03-06 15:27:57.305 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1666) finish_round done w/ waiters, state active 2019-03-06 15:27:57.321 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4859 ==== paxos(lease_ack lc 1666 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440ee9b80 con 0x56443e61e900 2019-03-06 15:27:57.321 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:57.321 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:57.321 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow all 2019-03-06 15:27:57.321 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow all 2019-03-06 15:27:57.321 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1666) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:27:57.321 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5778 ==== paxos(lease_ack lc 1666 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440ba6840 con 0x56443e61ed80 2019-03-06 15:27:57.321 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:27:57.321 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:57.321 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow all 2019-03-06 15:27:57.321 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:57.321 7f72cc823700 20 allow all 2019-03-06 15:27:57.321 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1666) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:27:58.617 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4860 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1495 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440f383c0 con 0x56443e61e900 2019-03-06 15:27:58.617 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:58.617 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:58.617 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:58.617 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:58.617 7f72cc823700 20 allow all 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:27:58.617 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:58.617 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:58.617 7f72cc823700 20 allow all 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x564440eeb180 from v2:192.168.178.28:40578/0 2019-03-06 15:27:58.617 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa886c0 for mgr.? 2019-03-06 15:27:58.617 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440eeb180 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499e00 2019-03-06 15:27:58.617 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1666) is_readable = 1 - now=2019-03-06 15:27:58.620685 lease_expire=2019-03-06 15:28:02.307114 has v0 lc 1666 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:27:58.617 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1495) v3 -- 0x56443ec0b9c0 con 0x56443e61e900 2019-03-06 15:27:58.617 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:27:58.617 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:27:58.617 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:27:58.617 7f72cc823700 20 match 2019-03-06 15:27:58.617 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:27:58.617 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:27:58.621 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 317 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e9f2580 con 0x5644402c0400 2019-03-06 15:27:58.621 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:27:58.621 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:58.621 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e9f2580 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:27:58.621 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1666) is_readable = 1 - now=2019-03-06 15:27:58.622890 lease_expire=2019-03-06 15:28:02.307114 has v0 lc 1666 2019-03-06 15:27:58.621 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:27:58.621 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:27:58.621 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:27:58.621 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:27:58.621 7f72cc823700 20 match 2019-03-06 15:27:58.621 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:27:58.621 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:27:58.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 318 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f849c00 con 0x5644402c0400 2019-03-06 15:27:58.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:27:58.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:27:58.813 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) dispatch 0x56443f849c00 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:27:58.813 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1666) is_readable = 1 - now=2019-03-06 15:27:58.814797 lease_expire=2019-03-06 15:28:02.307114 has v0 lc 1666 2019-03-06 15:27:58.813 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:27:58.813 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:27:58.813 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804232 }, { "osd": 1, "seq": 42949673413 }, { "osd": 2, "seq": 60129542592 }, { "osd": 3, "seq": 73014444476 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:27:58.813 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:27:58.813 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:27:58.813 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) setting proposal_timer 0x564440345e00 with delay of 0.05 2019-03-06 15:27:58.861 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1020) propose_pending 2019-03-06 15:27:58.861 7f72cf028700 10 mon.a@0(leader).mgrstat 1021 2019-03-06 15:27:58.865 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:27:58.865 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1666) queue_pending_finisher 0x56443d581810 2019-03-06 15:27:58.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1666) trigger_propose active, proposing now 2019-03-06 15:27:58.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1666) propose_pending 1667 6340 bytes 2019-03-06 15:27:58.865 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) begin for 1667 6340 bytes 2019-03-06 15:27:59.017 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:27:59.017 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:27:59.017 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443eb3e000 con 0x5644403b1b00 2019-03-06 15:27:59.333 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) sending begin to mon.1 2019-03-06 15:27:59.333 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1666 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f36b00 2019-03-06 15:27:59.333 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1666 fc 0 pn 300 opn 0) v4 -- 0x564440f36b00 con 0x56443e61e900 2019-03-06 15:27:59.333 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) sending begin to mon.2 2019-03-06 15:27:59.333 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1666 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441e27e40 2019-03-06 15:27:59.333 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1666 fc 0 pn 300 opn 0) v4 -- 0x564441e27e40 con 0x56443e61ed80 2019-03-06 15:27:59.437 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5779 ==== paxos(accept lc 1666 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441e27e40 con 0x56443e61ed80 2019-03-06 15:27:59.437 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:27:59.437 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:59.437 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow all 2019-03-06 15:27:59.437 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow all 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) handle_accept paxos(accept lc 1666 fc 0 pn 300 opn 0) v4 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) now 0,2 have accepted 2019-03-06 15:27:59.437 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4861 ==== paxos(accept lc 1666 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440f36b00 con 0x56443e61e900 2019-03-06 15:27:59.437 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:27:59.437 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:27:59.437 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow all 2019-03-06 15:27:59.437 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:27:59.437 7f72cc823700 20 allow all 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) handle_accept paxos(accept lc 1666 fc 0 pn 300 opn 0) v4 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) now 0,1,2 have accepted 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) got majority, committing, done with update 2019-03-06 15:27:59.437 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1666) commit_start 1667 2019-03-06 15:27:59.921 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1666) commit_finish 1667 2019-03-06 15:27:59.921 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1667) sending commit to mon.1 2019-03-06 15:27:59.921 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1667 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e99600 2019-03-06 15:27:59.921 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1667 fc 0 pn 300 opn 0) v4 -- 0x564440e99600 con 0x56443e61e900 2019-03-06 15:27:59.921 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1667) sending commit to mon.2 2019-03-06 15:27:59.921 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1667 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f489180 2019-03-06 15:27:59.921 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1667 fc 0 pn 300 opn 0) v4 -- 0x56443f489180 con 0x56443e61ed80 2019-03-06 15:27:59.921 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:27:59.921 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) refresh 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos version 516 summary v 516 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:27:59.925 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:27:59.933 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:27:59.933 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:27:59.945 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:27:59.949 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:27:59.949 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:27:59.949 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).mgrstat 1021 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1021 service_map e2 0progress events 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:27:59.949 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232aa00 con 0x5644402c0400 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1667) commit_proposal 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) _active - not active 2019-03-06 15:27:59.949 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1667) extend_lease now+5 (2019-03-06 15:28:04.953181) 2019-03-06 15:27:59.949 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1667 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8683c0 2019-03-06 15:27:59.949 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1667 fc 1005 pn 0 opn 0) v4 -- 0x56443e8683c0 con 0x56443e61e900 2019-03-06 15:27:59.949 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1667 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8a2ec0 2019-03-06 15:27:59.949 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1667 fc 1005 pn 0 opn 0) v4 -- 0x56443e8a2ec0 con 0x56443e61ed80 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1667) finish_round 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1667) finish_round waiting_for_acting 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) _active 2019-03-06 15:27:59.949 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1021) _active creating new pending 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).mgrstat 1021 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1667) finish_round waiting_for_readable 2019-03-06 15:27:59.949 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1667) finish_round waiting_for_writeable 2019-03-06 15:27:59.949 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1667) finish_round done w/ waiters, state active 2019-03-06 15:28:00.029 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5780 ==== paxos(lease_ack lc 1667 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8a2ec0 con 0x56443e61ed80 2019-03-06 15:28:00.029 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:00.029 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:00.029 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow all 2019-03-06 15:28:00.029 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow all 2019-03-06 15:28:00.029 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1667) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:00.029 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4862 ==== paxos(lease_ack lc 1667 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e8683c0 con 0x56443e61e900 2019-03-06 15:28:00.029 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:00.029 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:00.029 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow all 2019-03-06 15:28:00.029 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.029 7f72cc823700 20 allow all 2019-03-06 15:28:00.029 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1667) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:00.621 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 319 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x564440e99600 con 0x5644402c0400 2019-03-06 15:28:00.621 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:00.621 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:00.621 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440e99600 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:00.621 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1667) is_readable = 1 - now=2019-03-06 15:28:00.624057 lease_expire=2019-03-06 15:28:04.953181 has v0 lc 1667 2019-03-06 15:28:00.621 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:00.621 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:00.621 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:00.621 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:00.621 7f72cc823700 20 match 2019-03-06 15:28:00.621 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:00.621 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:00.633 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4863 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1496 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564441203340 con 0x56443e61e900 2019-03-06 15:28:00.633 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:00.633 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:00.633 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:00.633 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.633 7f72cc823700 20 allow all 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:00.633 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:00.633 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:00.633 7f72cc823700 20 allow all 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e9f3e40 from v2:192.168.178.28:40578/0 2019-03-06 15:28:00.633 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa89440 for mgr.? 2019-03-06 15:28:00.633 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e9f3e40 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440498e00 2019-03-06 15:28:00.633 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1667) is_readable = 1 - now=2019-03-06 15:28:00.637677 lease_expire=2019-03-06 15:28:04.953181 has v0 lc 1667 2019-03-06 15:28:00.633 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:00.633 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1496) v3 -- 0x56443ec096c0 con 0x56443e61e900 2019-03-06 15:28:00.633 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:00.633 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:00.637 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:00.637 7f72cc823700 20 match 2019-03-06 15:28:00.637 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:00.637 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:00.637 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:00.817 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 320 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f848580 con 0x5644402c0400 2019-03-06 15:28:00.817 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:00.817 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:00.817 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) dispatch 0x56443f848580 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:00.817 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1667) is_readable = 1 - now=2019-03-06 15:28:00.818687 lease_expire=2019-03-06 15:28:04.953181 has v0 lc 1667 2019-03-06 15:28:00.817 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:00.817 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:00.817 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804233 }, { "osd": 1, "seq": 42949673414 }, { "osd": 2, "seq": 60129542593 }, { "osd": 3, "seq": 73014444477 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:00.817 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:00.817 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:00.817 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) setting proposal_timer 0x5644419fef60 with delay of 0.10418 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1021) propose_pending 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader).mgrstat 1022 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:00.921 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1667) queue_pending_finisher 0x56443d581cc0 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1667) trigger_propose active, proposing now 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1667) propose_pending 1668 6340 bytes 2019-03-06 15:28:00.921 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) begin for 1668 6340 bytes 2019-03-06 15:28:01.369 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) sending begin to mon.1 2019-03-06 15:28:01.369 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1667 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443fec2580 2019-03-06 15:28:01.369 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1667 fc 0 pn 300 opn 0) v4 -- 0x56443fec2580 con 0x56443e61e900 2019-03-06 15:28:01.369 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) sending begin to mon.2 2019-03-06 15:28:01.369 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1667 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8a0dc0 2019-03-06 15:28:01.369 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1667 fc 0 pn 300 opn 0) v4 -- 0x56443e8a0dc0 con 0x56443e61ed80 2019-03-06 15:28:01.697 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5781 ==== paxos(accept lc 1667 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e8a0dc0 con 0x56443e61ed80 2019-03-06 15:28:01.697 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:01.697 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:01.697 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow all 2019-03-06 15:28:01.697 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow all 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) handle_accept paxos(accept lc 1667 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) now 0,2 have accepted 2019-03-06 15:28:01.697 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4864 ==== paxos(accept lc 1667 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443fec2580 con 0x56443e61e900 2019-03-06 15:28:01.697 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:01.697 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:01.697 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow all 2019-03-06 15:28:01.697 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:01.697 7f72cc823700 20 allow all 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) handle_accept paxos(accept lc 1667 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) now 0,1,2 have accepted 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) got majority, committing, done with update 2019-03-06 15:28:01.697 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1667) commit_start 1668 2019-03-06 15:28:01.729 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:01.729 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2217.59s 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:01.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos writing c 1005..1667) is_readable = 1 - now=2019-03-06 15:28:01.732227 lease_expire=2019-03-06 15:28:04.953181 has v0 lc 1667 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:01.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos writing c 1005..1667) is_readable = 1 - now=2019-03-06 15:28:01.732258 lease_expire=2019-03-06 15:28:04.953181 has v0 lc 1667 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).log v516 log 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..516) maybe_trim trim_to 16 would only trim 15 < paxos_service_trim_min 250 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:01.729 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:01.729 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:01.729 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:01.729 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f6f40 con 0x5644402c0400 2019-03-06 15:28:02.053 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1667) commit_finish 1668 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1668) sending commit to mon.1 2019-03-06 15:28:02.053 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1668 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f5b3b80 2019-03-06 15:28:02.053 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1668 fc 0 pn 300 opn 0) v4 -- 0x56443f5b3b80 con 0x56443e61e900 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1668) sending commit to mon.2 2019-03-06 15:28:02.053 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1668 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e869440 2019-03-06 15:28:02.053 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1668 fc 0 pn 300 opn 0) v4 -- 0x56443e869440 con 0x56443e61ed80 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) refresh 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos version 516 summary v 516 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:02.053 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:02.061 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:02.061 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:02.077 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:02.077 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:02.081 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:02.081 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).mgrstat 1022 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1022 service_map e2 0progress events 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:02.081 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232b200 con 0x5644402c0400 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1668) commit_proposal 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) _active - not active 2019-03-06 15:28:02.081 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1668) extend_lease now+5 (2019-03-06 15:28:07.083941) 2019-03-06 15:28:02.081 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1668 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e9f3600 2019-03-06 15:28:02.081 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1668 fc 1005 pn 0 opn 0) v4 -- 0x56443e9f3600 con 0x56443e61e900 2019-03-06 15:28:02.081 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1668 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440e2ec00 2019-03-06 15:28:02.081 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1668 fc 1005 pn 0 opn 0) v4 -- 0x564440e2ec00 con 0x56443e61ed80 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1668) finish_round 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1668) finish_round waiting_for_acting 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) _active 2019-03-06 15:28:02.081 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1022) _active creating new pending 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).mgrstat 1022 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1668) finish_round waiting_for_readable 2019-03-06 15:28:02.081 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1668) finish_round waiting_for_writeable 2019-03-06 15:28:02.081 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1668) finish_round done w/ waiters, state active 2019-03-06 15:28:02.433 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4865 ==== paxos(lease_ack lc 1668 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e9f3600 con 0x56443e61e900 2019-03-06 15:28:02.433 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:02.433 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:02.433 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow all 2019-03-06 15:28:02.433 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow all 2019-03-06 15:28:02.433 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1668) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:02.433 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5782 ==== paxos(lease_ack lc 1668 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440e2ec00 con 0x56443e61ed80 2019-03-06 15:28:02.433 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:02.433 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:02.433 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow all 2019-03-06 15:28:02.433 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.433 7f72cc823700 20 allow all 2019-03-06 15:28:02.433 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1668) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:02.549 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 321 ==== log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 ==== 953+0+0 (crc 0 0 0) 0x56443f150480 con 0x5644402c0400 2019-03-06 15:28:02.549 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:02.549 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..516) dispatch 0x56443f150480 log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:02.549 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1668) is_readable = 1 - now=2019-03-06 15:28:02.552789 lease_expire=2019-03-06 15:28:07.083941 has v0 lc 1668 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 preprocess_query log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 preprocess_log log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 from mgr.6993 2019-03-06 15:28:02.549 7f72cc823700 20 is_capable service=log command= write addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:02.549 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:02.549 7f72cc823700 20 match 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 prepare_update log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 prepare_log log(5 entries from seq 140 at 2019-03-06 15:27:52.808800) v1 from mgr.6993 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 logging 2019-03-06 15:27:52.808800 mgr.z (mgr.6993) 140 : cluster [DBG] pgmap v137: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 logging 2019-03-06 15:27:54.811855 mgr.z (mgr.6993) 141 : cluster [DBG] pgmap v138: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 logging 2019-03-06 15:27:56.813491 mgr.z (mgr.6993) 142 : cluster [DBG] pgmap v139: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 logging 2019-03-06 15:27:58.814423 mgr.z (mgr.6993) 143 : cluster [DBG] pgmap v140: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).log v516 logging 2019-03-06 15:28:00.817498 mgr.z (mgr.6993) 144 : cluster [DBG] pgmap v141: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:02.549 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..516) setting proposal_timer 0x564441340750 with delay of 0.501529 2019-03-06 15:28:02.637 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4866 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1497 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56443f5b3b80 con 0x56443e61e900 2019-03-06 15:28:02.637 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:02.637 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:02.637 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:02.637 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.637 7f72cc823700 20 allow all 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:02.637 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:02.637 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:02.637 7f72cc823700 20 allow all 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x564440ee8840 from v2:192.168.178.28:40578/0 2019-03-06 15:28:02.637 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8a400 for mgr.? 2019-03-06 15:28:02.637 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440ee8840 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499c00 2019-03-06 15:28:02.637 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1668) is_readable = 1 - now=2019-03-06 15:28:02.639429 lease_expire=2019-03-06 15:28:07.083941 has v0 lc 1668 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:02.637 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1497) v3 -- 0x56443ec09500 con 0x56443e61e900 2019-03-06 15:28:02.637 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:02.637 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:02.637 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:02.637 7f72cc823700 20 match 2019-03-06 15:28:02.637 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:02.637 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:02.641 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 322 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e94c940 con 0x5644402c0400 2019-03-06 15:28:02.641 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:02.641 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:02.641 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e94c940 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:02.641 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1668) is_readable = 1 - now=2019-03-06 15:28:02.642064 lease_expire=2019-03-06 15:28:07.083941 has v0 lc 1668 2019-03-06 15:28:02.641 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:02.641 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:02.641 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:02.641 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:02.641 7f72cc823700 20 match 2019-03-06 15:28:02.641 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:02.641 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:02.817 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 323 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f848d00 con 0x5644402c0400 2019-03-06 15:28:02.817 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:02.817 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:02.817 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) dispatch 0x56443f848d00 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:02.817 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1668) is_readable = 1 - now=2019-03-06 15:28:02.819346 lease_expire=2019-03-06 15:28:07.083941 has v0 lc 1668 2019-03-06 15:28:02.817 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:02.817 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:02.817 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804233 }, { "osd": 1, "seq": 42949673414 }, { "osd": 2, "seq": 60129542593 }, { "osd": 3, "seq": 73014444477 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:02.817 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:02.817 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:02.817 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) setting proposal_timer 0x56444217c720 with delay of 0.234612 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1022) propose_pending 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader).mgrstat 1023 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:03.053 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1668) queue_pending_finisher 0x56443d581fc0 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1668) trigger_propose active, proposing now 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1668) propose_pending 1669 6340 bytes 2019-03-06 15:28:03.053 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) begin for 1669 6340 bytes 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) sending begin to mon.1 2019-03-06 15:28:03.089 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1668 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444104d600 2019-03-06 15:28:03.089 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1668 fc 0 pn 300 opn 0) v4 -- 0x56444104d600 con 0x56443e61e900 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) sending begin to mon.2 2019-03-06 15:28:03.089 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1668 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441306940 2019-03-06 15:28:03.089 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1668 fc 0 pn 300 opn 0) v4 -- 0x564441306940 con 0x56443e61ed80 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..516) propose_pending 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).log v516 encode_full log v 516 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).log v516 encode_pending v517 2019-03-06 15:28:03.089 7f72cf028700 5 mon.a@0(leader).paxos(paxos updating c 1005..1668) queue_pending_finisher 0x56443d581cd0 2019-03-06 15:28:03.089 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) trigger_propose not active, will propose later 2019-03-06 15:28:03.129 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5783 ==== paxos(accept lc 1668 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441306940 con 0x56443e61ed80 2019-03-06 15:28:03.129 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:03.129 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.129 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow all 2019-03-06 15:28:03.129 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow all 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) handle_accept paxos(accept lc 1668 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) now 0,2 have accepted 2019-03-06 15:28:03.129 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4867 ==== paxos(accept lc 1668 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56444104d600 con 0x56443e61e900 2019-03-06 15:28:03.129 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:03.129 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.129 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow all 2019-03-06 15:28:03.129 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.129 7f72cc823700 20 allow all 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) handle_accept paxos(accept lc 1668 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) now 0,1,2 have accepted 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) got majority, committing, done with update 2019-03-06 15:28:03.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1668) commit_start 1669 2019-03-06 15:28:03.169 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1668) commit_finish 1669 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1669) sending commit to mon.1 2019-03-06 15:28:03.169 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1669 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f37e40 2019-03-06 15:28:03.169 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1669 fc 0 pn 300 opn 0) v4 -- 0x564440f37e40 con 0x56443e61e900 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1669) sending commit to mon.2 2019-03-06 15:28:03.169 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1669 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f38100 2019-03-06 15:28:03.169 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1669 fc 0 pn 300 opn 0) v4 -- 0x564440f38100 con 0x56443e61ed80 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) refresh 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).log v516 update_from_paxos version 516 summary v 516 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:03.169 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:03.173 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:03.173 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:03.177 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:03.177 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:03.177 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgrstat 1023 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1023 service_map e2 0progress events 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:03.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232bc00 con 0x5644402c0400 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..516) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1669) commit_proposal 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) _active - not active 2019-03-06 15:28:03.177 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1669) extend_lease now+5 (2019-03-06 15:28:08.181347) 2019-03-06 15:28:03.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1669 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440ee9e40 2019-03-06 15:28:03.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1669 fc 1005 pn 0 opn 0) v4 -- 0x564440ee9e40 con 0x56443e61e900 2019-03-06 15:28:03.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1669 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8f98c0 2019-03-06 15:28:03.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1669 fc 1005 pn 0 opn 0) v4 -- 0x56443e8f98c0 con 0x56443e61ed80 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1669) finish_round 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1669) finish_round waiting_for_acting 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) _active 2019-03-06 15:28:03.177 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1023) _active creating new pending 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).mgrstat 1023 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1669) finish_round waiting_for_readable 2019-03-06 15:28:03.177 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1669) finish_round waiting_for_writeable 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1669) finish_round done w/ waiters, state active 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1669) propose_pending 1670 46991 bytes 2019-03-06 15:28:03.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) begin for 1670 46991 bytes 2019-03-06 15:28:03.225 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4868 ==== paxos(lease_ack lc 1669 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440ee9e40 con 0x56443e61e900 2019-03-06 15:28:03.261 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) sending begin to mon.1 2019-03-06 15:28:03.261 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1669 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e9f5700 2019-03-06 15:28:03.261 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1669 fc 0 pn 300 opn 0) v4 -- 0x56443e9f5700 con 0x56443e61e900 2019-03-06 15:28:03.261 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) sending begin to mon.2 2019-03-06 15:28:03.261 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1669 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e99b80 2019-03-06 15:28:03.261 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1669 fc 0 pn 300 opn 0) v4 -- 0x564440e99b80 con 0x56443e61ed80 2019-03-06 15:28:03.261 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:03.261 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.261 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow all 2019-03-06 15:28:03.261 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow all 2019-03-06 15:28:03.261 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) handle_lease_ack from mon.1 -- stray (probably since revoked) 2019-03-06 15:28:03.261 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5784 ==== paxos(lease_ack lc 1669 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8f98c0 con 0x56443e61ed80 2019-03-06 15:28:03.261 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:03.261 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.261 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow all 2019-03-06 15:28:03.261 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.261 7f72cc823700 20 allow all 2019-03-06 15:28:03.261 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) handle_lease_ack from mon.2 -- stray (probably since revoked) 2019-03-06 15:28:03.309 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4869 ==== paxos(accept lc 1669 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e9f5700 con 0x56443e61e900 2019-03-06 15:28:03.309 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:03.309 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.309 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow all 2019-03-06 15:28:03.309 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow all 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) handle_accept paxos(accept lc 1669 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) now 0,1 have accepted 2019-03-06 15:28:03.309 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5785 ==== paxos(accept lc 1669 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440e99b80 con 0x56443e61ed80 2019-03-06 15:28:03.309 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:03.309 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.309 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow all 2019-03-06 15:28:03.309 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.309 7f72cc823700 20 allow all 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) handle_accept paxos(accept lc 1669 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) now 0,1,2 have accepted 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) got majority, committing, done with update 2019-03-06 15:28:03.309 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1669) commit_start 1670 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1669) commit_finish 1670 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1670) sending commit to mon.1 2019-03-06 15:28:03.357 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1670 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e868ec0 2019-03-06 15:28:03.357 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1670 fc 0 pn 300 opn 0) v4 -- 0x56443e868ec0 con 0x56443e61e900 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1670) sending commit to mon.2 2019-03-06 15:28:03.357 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1670 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440d46c00 2019-03-06 15:28:03.357 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1670 fc 0 pn 300 opn 0) v4 -- 0x564440d46c00 con 0x56443e61ed80 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 516 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos latest full 516 2019-03-06 15:28:03.357 7f72c901c700 7 mon.a@0(leader).log v517 update_from_paxos applying incremental log 517 2019-03-06 15:27:52.808800 mgr.z (mgr.6993) 140 : cluster [DBG] pgmap v137: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:03.357 7f72c901c700 7 mon.a@0(leader).log v517 update_from_paxos applying incremental log 517 2019-03-06 15:27:54.811855 mgr.z (mgr.6993) 141 : cluster [DBG] pgmap v138: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:03.357 7f72c901c700 7 mon.a@0(leader).log v517 update_from_paxos applying incremental log 517 2019-03-06 15:27:56.813491 mgr.z (mgr.6993) 142 : cluster [DBG] pgmap v139: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:03.357 7f72c901c700 7 mon.a@0(leader).log v517 update_from_paxos applying incremental log 517 2019-03-06 15:27:58.814423 mgr.z (mgr.6993) 143 : cluster [DBG] pgmap v140: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:03.357 7f72c901c700 7 mon.a@0(leader).log v517 update_from_paxos applying incremental log 517 2019-03-06 15:28:00.817498 mgr.z (mgr.6993) 144 : cluster [DBG] pgmap v141: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:03.357 7f72c901c700 15 mon.a@0(leader).log v517 update_from_paxos logging for 1 channels 2019-03-06 15:28:03.357 7f72c901c700 15 mon.a@0(leader).log v517 update_from_paxos channel 'cluster' logging 755 bytes 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 check_subs 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 check_sub client wants log-info ver 517 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 _create_sub_incremental level 1 ver 517 cur summary ver 517 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).log v517 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 _create_sub_incremental incremental message ready (0 entries) 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).log v517 check_sub sending message to mgr.6993 with 0 entries (version 517) 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:03.357 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:03.357 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:03.361 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:03.365 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:03.365 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:03.365 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).mgrstat 1023 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1023 service_map e2 0progress events 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232a200 con 0x5644402c0400 2019-03-06 15:28:03.365 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:03.365 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1670) commit_proposal 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) _active - not active 2019-03-06 15:28:03.365 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1670) extend_lease now+5 (2019-03-06 15:28:08.367388) 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1670 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440e98580 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1670 fc 1005 pn 0 opn 0) v4 -- 0x564440e98580 con 0x56443e61e900 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1670 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f37b80 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1670 fc 1005 pn 0 opn 0) v4 -- 0x564440f37b80 con 0x56443e61ed80 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1670) finish_round 2019-03-06 15:28:03.365 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1670) finish_round waiting_for_acting 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) _active 2019-03-06 15:28:03.365 7f72c901c700 7 mon.a@0(leader).paxosservice(logm 1..517) _active creating new pending 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).log v517 create_pending v 518 2019-03-06 15:28:03.365 7f72c901c700 7 mon.a@0(leader).log v517 _updated_log for mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:03.365 7f72c901c700 2 mon.a@0(leader) e1 send_reply 0x564442180580 0x56443fd8c000 log(last 144) v1 2019-03-06 15:28:03.365 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- log(last 144) v1 -- 0x56443fd8c000 con 0x5644402c0400 2019-03-06 15:28:03.365 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1670) finish_round waiting_for_readable 2019-03-06 15:28:03.365 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1670) finish_round waiting_for_writeable 2019-03-06 15:28:03.365 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1670) finish_round done w/ waiters, state active 2019-03-06 15:28:03.477 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5786 ==== paxos(lease_ack lc 1670 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440f37b80 con 0x56443e61ed80 2019-03-06 15:28:03.477 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:03.477 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.477 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow all 2019-03-06 15:28:03.477 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow all 2019-03-06 15:28:03.477 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1670) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:03.477 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4870 ==== paxos(lease_ack lc 1670 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440e98580 con 0x56443e61e900 2019-03-06 15:28:03.477 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:03.477 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:03.477 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow all 2019-03-06 15:28:03.477 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:03.477 7f72cc823700 20 allow all 2019-03-06 15:28:03.477 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1670) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:04.017 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:04.017 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:04.017 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443eb40080 con 0x5644403b1b00 2019-03-06 15:28:04.641 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 324 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e868ec0 con 0x5644402c0400 2019-03-06 15:28:04.641 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:04.641 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:04.641 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e868ec0 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:04.641 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1670) is_readable = 1 - now=2019-03-06 15:28:04.643392 lease_expire=2019-03-06 15:28:08.367388 has v0 lc 1670 2019-03-06 15:28:04.641 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:04.641 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:04.641 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:04.641 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:04.641 7f72cc823700 20 match 2019-03-06 15:28:04.641 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:04.641 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:04.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4871 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1498 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440f37e40 con 0x56443e61e900 2019-03-06 15:28:04.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:04.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:04.657 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:04.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:04.657 7f72cc823700 20 allow all 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:04.657 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:04.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:04.657 7f72cc823700 20 allow all 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56444104d8c0 from v2:192.168.178.28:40578/0 2019-03-06 15:28:04.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa89200 for mgr.? 2019-03-06 15:28:04.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56444104d8c0 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499000 2019-03-06 15:28:04.657 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1670) is_readable = 1 - now=2019-03-06 15:28:04.658653 lease_expire=2019-03-06 15:28:08.367388 has v0 lc 1670 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:04.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1498) v3 -- 0x56443ec0af40 con 0x56443e61e900 2019-03-06 15:28:04.657 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:04.657 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:04.657 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:04.657 7f72cc823700 20 match 2019-03-06 15:28:04.657 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:04.657 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:04.821 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 325 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f846f00 con 0x5644402c0400 2019-03-06 15:28:04.821 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:04.821 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:04.821 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) dispatch 0x56443f846f00 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:04.821 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1670) is_readable = 1 - now=2019-03-06 15:28:04.823594 lease_expire=2019-03-06 15:28:08.367388 has v0 lc 1670 2019-03-06 15:28:04.821 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:04.821 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:04.821 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804234 }, { "osd": 1, "seq": 42949673415 }, { "osd": 2, "seq": 60129542594 }, { "osd": 3, "seq": 73014444478 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:04.821 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:04.821 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:04.821 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) setting proposal_timer 0x56444133c9f0 with delay of 0.05 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1023) propose_pending 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader).mgrstat 1024 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:04.873 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1670) queue_pending_finisher 0x56443d581a40 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1670) trigger_propose active, proposing now 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1670) propose_pending 1671 6340 bytes 2019-03-06 15:28:04.873 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) begin for 1671 6340 bytes 2019-03-06 15:28:05.833 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) sending begin to mon.1 2019-03-06 15:28:05.833 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1670 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441204940 2019-03-06 15:28:05.833 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1670 fc 0 pn 300 opn 0) v4 -- 0x564441204940 con 0x56443e61e900 2019-03-06 15:28:05.833 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) sending begin to mon.2 2019-03-06 15:28:05.833 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1670 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f36000 2019-03-06 15:28:05.833 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1670 fc 0 pn 300 opn 0) v4 -- 0x564440f36000 con 0x56443e61ed80 2019-03-06 15:28:05.837 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4872 ==== forward(osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 caps allow profile osd tid 1499 con_features 4611087854031667199) v4 ==== 437+0+0 (crc 0 0 0) 0x56443e8a02c0 con 0x56443e61e900 2019-03-06 15:28:05.837 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:05.837 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:05.837 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:05.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:05.837 7f72cc823700 20 allow all 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from osd [v2:192.168.178.28:6810/31407,v1:192.168.178.28:6811/31407] via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:05.837 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:05.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:05.837 7f72cc823700 20 allow all 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile osd 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader) e1 entity name 'osd.1' type 4 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443eacc400 from v2:192.168.178.28:40578/0 2019-03-06 15:28:05.837 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa88900 for osd.1 2019-03-06 15:28:05.837 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile osd 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader).paxosservice(osdmap 1..35) dispatch 0x56443eacc400 osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 from osd.1 v2:192.168.178.28:6810/31407 con 0x56444049be00 2019-03-06 15:28:05.837 7f72cc823700 5 mon.a@0(leader).paxos(paxos updating c 1005..1670) is_readable = 1 - now=2019-03-06 15:28:05.838423 lease_expire=2019-03-06 15:28:08.367388 has v0 lc 1670 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader).osd e35 preprocess_query osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 from osd.1 v2:192.168.178.28:6810/31407 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader) e1 no_reply to osd.1 v2:192.168.178.28:6810/31407 via v2:192.168.178.28:40578/0 for request osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 2019-03-06 15:28:05.837 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1499) v3 -- 0x56443ec09880 con 0x56443e61e900 2019-03-06 15:28:05.837 7f72cc823700 20 is_capable service=osd command= exec addr v2:192.168.178.28:6810/31407 on cap allow profile osd 2019-03-06 15:28:05.837 7f72cc823700 20 allow so far , doing grant allow profile osd 2019-03-06 15:28:05.837 7f72cc823700 20 match 2019-03-06 15:28:05.837 7f72cc823700 7 mon.a@0(leader).osd e35 prepare_update osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 from osd.1 v2:192.168.178.28:6810/31407 2019-03-06 15:28:05.837 7f72cc823700 10 mon.a@0(leader).osd e35 prepare_beacon osd_beacon(pgs [1.0,3.0,2.6,4.2,1.7,4.7,1.3,4.4,3.4,2.4] lec 34 v35) v1 from osd.1 2019-03-06 15:28:06.449 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5787 ==== paxos(accept lc 1670 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440f36000 con 0x56443e61ed80 2019-03-06 15:28:06.449 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:06.449 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:06.449 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow all 2019-03-06 15:28:06.449 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow all 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) handle_accept paxos(accept lc 1670 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) now 0,2 have accepted 2019-03-06 15:28:06.449 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4873 ==== paxos(accept lc 1670 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441204940 con 0x56443e61e900 2019-03-06 15:28:06.449 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:06.449 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:06.449 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow all 2019-03-06 15:28:06.449 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.449 7f72cc823700 20 allow all 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) handle_accept paxos(accept lc 1670 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) now 0,1,2 have accepted 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) got majority, committing, done with update 2019-03-06 15:28:06.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1670) commit_start 1671 2019-03-06 15:28:06.549 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1670) commit_finish 1671 2019-03-06 15:28:06.549 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1671) sending commit to mon.1 2019-03-06 15:28:06.553 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1671 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440b5adc0 2019-03-06 15:28:06.553 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1671 fc 0 pn 300 opn 0) v4 -- 0x564440b5adc0 con 0x56443e61e900 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1671) sending commit to mon.2 2019-03-06 15:28:06.553 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1671 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f378c0 2019-03-06 15:28:06.553 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1671 fc 0 pn 300 opn 0) v4 -- 0x564440f378c0 con 0x56443e61ed80 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 517 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:06.553 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:06.557 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:06.557 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:06.573 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:06.573 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:06.573 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) refresh 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).mgrstat 1024 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1024 service_map e2 0progress events 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:06.573 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:06.573 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x56444232a000 con 0x5644402c0400 2019-03-06 15:28:06.573 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:06.577 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1671) commit_proposal 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) _active - not active 2019-03-06 15:28:06.577 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1671) extend_lease now+5 (2019-03-06 15:28:11.579034) 2019-03-06 15:28:06.577 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1671 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443f46cb00 2019-03-06 15:28:06.577 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1671 fc 1005 pn 0 opn 0) v4 -- 0x56443f46cb00 con 0x56443e61e900 2019-03-06 15:28:06.577 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1671 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443ea8f440 2019-03-06 15:28:06.577 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1671 fc 1005 pn 0 opn 0) v4 -- 0x56443ea8f440 con 0x56443e61ed80 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1671) finish_round 2019-03-06 15:28:06.577 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1671) finish_round waiting_for_acting 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) _active 2019-03-06 15:28:06.577 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1024) _active creating new pending 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).mgrstat 1024 2019-03-06 15:28:06.577 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:06.577 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1671) finish_round waiting_for_readable 2019-03-06 15:28:06.577 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1671) finish_round waiting_for_writeable 2019-03-06 15:28:06.577 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1671) finish_round done w/ waiters, state active 2019-03-06 15:28:06.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5788 ==== paxos(lease_ack lc 1671 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443ea8f440 con 0x56443e61ed80 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1671) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:06.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4874 ==== paxos(lease_ack lc 1671 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443f46cb00 con 0x56443e61e900 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1671) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:06.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4875 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1500 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440b5adc0 con 0x56443e61e900 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:06.657 7f72cc823700 20 allow all 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e8a0580 from v2:192.168.178.28:40578/0 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa88b40 for mgr.? 2019-03-06 15:28:06.657 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e8a0580 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440498000 2019-03-06 15:28:06.657 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) is_readable = 1 - now=2019-03-06 15:28:06.661294 lease_expire=2019-03-06 15:28:11.579034 has v0 lc 1671 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:06.657 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1500) v3 -- 0x56443ec0bb80 con 0x56443e61e900 2019-03-06 15:28:06.657 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:06.657 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:06.657 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:06.657 7f72cc823700 20 match 2019-03-06 15:28:06.657 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:06.657 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:06.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 326 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x5644413063c0 con 0x5644402c0400 2019-03-06 15:28:06.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:06.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:06.661 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x5644413063c0 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:06.661 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) is_readable = 1 - now=2019-03-06 15:28:06.662864 lease_expire=2019-03-06 15:28:11.579034 has v0 lc 1671 2019-03-06 15:28:06.661 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:06.661 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:06.661 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:06.661 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:06.661 7f72cc823700 20 match 2019-03-06 15:28:06.661 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:06.661 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:06.729 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:06.729 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2222.59s 2019-03-06 15:28:06.729 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:06.729 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:06.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) is_readable = 1 - now=2019-03-06 15:28:06.733725 lease_expire=2019-03-06 15:28:11.579034 has v0 lc 1671 2019-03-06 15:28:06.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:06.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:06.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:06.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:06.729 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:06.729 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:06.729 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) is_readable = 1 - now=2019-03-06 15:28:06.733831 lease_expire=2019-03-06 15:28:11.579034 has v0 lc 1671 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).log v517 log 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..517) maybe_trim trim_to 17 would only trim 16 < paxos_service_trim_min 250 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:06.733 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) maybe_trim trim_to 1019 would only trim 249 < paxos_service_trim_min 250 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:06.733 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:06.733 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:06.733 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:06.733 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:06.733 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f7cc0 con 0x5644402c0400 2019-03-06 15:28:06.821 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 327 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f846000 con 0x5644402c0400 2019-03-06 15:28:06.821 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:06.821 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:06.821 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) dispatch 0x56443f846000 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:06.821 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) is_readable = 1 - now=2019-03-06 15:28:06.825312 lease_expire=2019-03-06 15:28:11.579034 has v0 lc 1671 2019-03-06 15:28:06.821 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:06.821 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:06.821 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804234 }, { "osd": 1, "seq": 42949673415 }, { "osd": 2, "seq": 60129542594 }, { "osd": 3, "seq": 73014444478 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:06.825 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:06.825 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:06.825 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) setting proposal_timer 0x56443ff0db90 with delay of 0.726532 2019-03-06 15:28:07.549 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1024) propose_pending 2019-03-06 15:28:07.553 7f72cf028700 10 mon.a@0(leader).mgrstat 1025 2019-03-06 15:28:07.553 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:07.553 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1671) queue_pending_finisher 0x5644401e40d0 2019-03-06 15:28:07.553 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1671) trigger_propose active, proposing now 2019-03-06 15:28:07.553 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1671) propose_pending 1672 6340 bytes 2019-03-06 15:28:07.553 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) begin for 1672 6340 bytes 2019-03-06 15:28:07.609 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) sending begin to mon.1 2019-03-06 15:28:07.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1671 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444199a000 2019-03-06 15:28:07.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1671 fc 0 pn 300 opn 0) v4 -- 0x56444199a000 con 0x56443e61e900 2019-03-06 15:28:07.609 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) sending begin to mon.2 2019-03-06 15:28:07.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1671 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8a2c00 2019-03-06 15:28:07.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1671 fc 0 pn 300 opn 0) v4 -- 0x56443e8a2c00 con 0x56443e61ed80 2019-03-06 15:28:07.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5789 ==== paxos(accept lc 1671 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e8a2c00 con 0x56443e61ed80 2019-03-06 15:28:07.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:07.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:07.661 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow all 2019-03-06 15:28:07.661 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow all 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) handle_accept paxos(accept lc 1671 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) now 0,2 have accepted 2019-03-06 15:28:07.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4876 ==== paxos(accept lc 1671 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56444199a000 con 0x56443e61e900 2019-03-06 15:28:07.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:07.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:07.661 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow all 2019-03-06 15:28:07.661 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.661 7f72cc823700 20 allow all 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) handle_accept paxos(accept lc 1671 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) now 0,1,2 have accepted 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) got majority, committing, done with update 2019-03-06 15:28:07.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1671) commit_start 1672 2019-03-06 15:28:07.713 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1671) commit_finish 1672 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1672) sending commit to mon.1 2019-03-06 15:28:07.713 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1672 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f442dc0 2019-03-06 15:28:07.713 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1672 fc 0 pn 300 opn 0) v4 -- 0x56443f442dc0 con 0x56443e61e900 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1672) sending commit to mon.2 2019-03-06 15:28:07.713 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1672 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e9f2dc0 2019-03-06 15:28:07.713 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1672 fc 0 pn 300 opn 0) v4 -- 0x56443e9f2dc0 con 0x56443e61ed80 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 517 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:07.713 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:07.717 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:07.717 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:07.721 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:07.721 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:07.721 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgrstat 1025 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1025 service_map e2 0progress events 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:07.721 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442329a00 con 0x5644402c0400 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1672) commit_proposal 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) _active - not active 2019-03-06 15:28:07.721 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1672) extend_lease now+5 (2019-03-06 15:28:12.724576) 2019-03-06 15:28:07.721 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1672 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440d463c0 2019-03-06 15:28:07.721 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1672 fc 1005 pn 0 opn 0) v4 -- 0x564440d463c0 con 0x56443e61e900 2019-03-06 15:28:07.721 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1672 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440e982c0 2019-03-06 15:28:07.721 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1672 fc 1005 pn 0 opn 0) v4 -- 0x564440e982c0 con 0x56443e61ed80 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1672) finish_round 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1672) finish_round waiting_for_acting 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) _active 2019-03-06 15:28:07.721 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1025) _active creating new pending 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).mgrstat 1025 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1672) finish_round waiting_for_readable 2019-03-06 15:28:07.721 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1672) finish_round waiting_for_writeable 2019-03-06 15:28:07.721 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1672) finish_round done w/ waiters, state active 2019-03-06 15:28:07.781 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5790 ==== paxos(lease_ack lc 1672 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440e982c0 con 0x56443e61ed80 2019-03-06 15:28:07.781 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:07.781 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:07.781 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow all 2019-03-06 15:28:07.781 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow all 2019-03-06 15:28:07.781 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1672) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:07.781 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4877 ==== paxos(lease_ack lc 1672 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440d463c0 con 0x56443e61e900 2019-03-06 15:28:07.781 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:07.781 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:07.781 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow all 2019-03-06 15:28:07.781 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:07.781 7f72cc823700 20 allow all 2019-03-06 15:28:07.781 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1672) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:08.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 328 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443f442dc0 con 0x5644402c0400 2019-03-06 15:28:08.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:08.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:08.661 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443f442dc0 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:08.661 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1672) is_readable = 1 - now=2019-03-06 15:28:08.664941 lease_expire=2019-03-06 15:28:12.724576 has v0 lc 1672 2019-03-06 15:28:08.661 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:08.661 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:08.661 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:08.661 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:08.661 7f72cc823700 20 match 2019-03-06 15:28:08.661 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:08.661 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:08.677 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4878 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1501 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564441927b80 con 0x56443e61e900 2019-03-06 15:28:08.677 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:08.677 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:08.677 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:08.677 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:08.677 7f72cc823700 20 allow all 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:08.677 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:08.677 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:08.677 7f72cc823700 20 allow all 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x564440ee9340 from v2:192.168.178.28:40578/0 2019-03-06 15:28:08.677 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8bcc0 for mgr.? 2019-03-06 15:28:08.677 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440ee9340 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444049ac00 2019-03-06 15:28:08.677 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1672) is_readable = 1 - now=2019-03-06 15:28:08.679905 lease_expire=2019-03-06 15:28:12.724576 has v0 lc 1672 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:08.677 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1501) v3 -- 0x56443ec0a140 con 0x56443e61e900 2019-03-06 15:28:08.677 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:08.677 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:08.677 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:08.677 7f72cc823700 20 match 2019-03-06 15:28:08.677 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:08.677 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:08.825 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 329 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x56443f847b80 con 0x5644402c0400 2019-03-06 15:28:08.825 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:08.825 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:08.825 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) dispatch 0x56443f847b80 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:08.825 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1672) is_readable = 1 - now=2019-03-06 15:28:08.827210 lease_expire=2019-03-06 15:28:12.724576 has v0 lc 1672 2019-03-06 15:28:08.825 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:08.825 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:08.825 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804234 }, { "osd": 1, "seq": 42949673415 }, { "osd": 2, "seq": 60129542594 }, { "osd": 3, "seq": 73014444478 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:08.825 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:08.825 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:08.825 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) setting proposal_timer 0x5644412838c0 with delay of 0.05 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1025) propose_pending 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader).mgrstat 1026 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:08.877 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1672) queue_pending_finisher 0x5644401e4ce0 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1672) trigger_propose active, proposing now 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1672) propose_pending 1673 6340 bytes 2019-03-06 15:28:08.877 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) begin for 1673 6340 bytes 2019-03-06 15:28:08.945 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) sending begin to mon.1 2019-03-06 15:28:08.945 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1672 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444184adc0 2019-03-06 15:28:08.945 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1672 fc 0 pn 300 opn 0) v4 -- 0x56444184adc0 con 0x56443e61e900 2019-03-06 15:28:08.945 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) sending begin to mon.2 2019-03-06 15:28:08.945 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1672 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441f903c0 2019-03-06 15:28:08.945 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1672 fc 0 pn 300 opn 0) v4 -- 0x564441f903c0 con 0x56443e61ed80 2019-03-06 15:28:09.005 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5791 ==== paxos(accept lc 1672 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441f903c0 con 0x56443e61ed80 2019-03-06 15:28:09.005 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:09.005 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:09.005 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow all 2019-03-06 15:28:09.005 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow all 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) handle_accept paxos(accept lc 1672 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) now 0,2 have accepted 2019-03-06 15:28:09.005 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4879 ==== paxos(accept lc 1672 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56444184adc0 con 0x56443e61e900 2019-03-06 15:28:09.005 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:09.005 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:09.005 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow all 2019-03-06 15:28:09.005 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:09.005 7f72cc823700 20 allow all 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) handle_accept paxos(accept lc 1672 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) now 0,1,2 have accepted 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) got majority, committing, done with update 2019-03-06 15:28:09.005 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1672) commit_start 1673 2019-03-06 15:28:09.017 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:09.017 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:09.017 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443eb3f380 con 0x5644403b1b00 2019-03-06 15:28:10.093 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1672) commit_finish 1673 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1673) sending commit to mon.1 2019-03-06 15:28:10.093 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1673 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f4c5440 2019-03-06 15:28:10.093 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1673 fc 0 pn 300 opn 0) v4 -- 0x56443f4c5440 con 0x56443e61e900 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1673) sending commit to mon.2 2019-03-06 15:28:10.093 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1673 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f3d0100 2019-03-06 15:28:10.093 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1673 fc 0 pn 300 opn 0) v4 -- 0x56443f3d0100 con 0x56443e61ed80 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 517 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:10.093 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:10.101 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:10.101 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:10.113 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:10.113 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:10.113 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:10.113 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:10.113 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) refresh 2019-03-06 15:28:10.113 7f72c901c700 10 mon.a@0(leader).mgrstat 1026 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1026 service_map e2 0progress events 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:10.117 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x5644406f6200 con 0x5644402c0400 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1673) commit_proposal 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) _active - not active 2019-03-06 15:28:10.117 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1673) extend_lease now+5 (2019-03-06 15:28:15.119376) 2019-03-06 15:28:10.117 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1673 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f37340 2019-03-06 15:28:10.117 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1673 fc 1005 pn 0 opn 0) v4 -- 0x564440f37340 con 0x56443e61e900 2019-03-06 15:28:10.117 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1673 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f37600 2019-03-06 15:28:10.117 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1673 fc 1005 pn 0 opn 0) v4 -- 0x564440f37600 con 0x56443e61ed80 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1673) finish_round 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1673) finish_round waiting_for_acting 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) _active 2019-03-06 15:28:10.117 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1026) _active creating new pending 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).mgrstat 1026 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1673) finish_round waiting_for_readable 2019-03-06 15:28:10.117 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1673) finish_round waiting_for_writeable 2019-03-06 15:28:10.117 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1673) finish_round done w/ waiters, state active 2019-03-06 15:28:10.417 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4880 ==== paxos(lease_ack lc 1673 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440f37340 con 0x56443e61e900 2019-03-06 15:28:10.417 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:10.417 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:10.417 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:10.417 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.417 7f72cc823700 20 allow all 2019-03-06 15:28:10.417 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:10.417 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.417 7f72cc823700 20 allow all 2019-03-06 15:28:10.417 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1673) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:10.421 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5792 ==== paxos(lease_ack lc 1673 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440f37600 con 0x56443e61ed80 2019-03-06 15:28:10.421 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:10.421 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:10.421 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:10.421 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.421 7f72cc823700 20 allow all 2019-03-06 15:28:10.421 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:10.421 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.421 7f72cc823700 20 allow all 2019-03-06 15:28:10.421 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1673) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:10.677 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4881 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1502 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56443f4c5440 con 0x56443e61e900 2019-03-06 15:28:10.677 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:10.677 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:10.677 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:10.677 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.677 7f72cc823700 20 allow all 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:10.677 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:10.677 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:10.677 7f72cc823700 20 allow all 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x5644412038c0 from v2:192.168.178.28:40578/0 2019-03-06 15:28:10.677 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa89d40 for mgr.? 2019-03-06 15:28:10.677 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x5644412038c0 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444049b600 2019-03-06 15:28:10.677 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1673) is_readable = 1 - now=2019-03-06 15:28:10.679657 lease_expire=2019-03-06 15:28:15.119376 has v0 lc 1673 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:10.677 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1502) v3 -- 0x56443ec0bd40 con 0x56443e61e900 2019-03-06 15:28:10.677 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:10.677 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:10.677 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:10.677 7f72cc823700 20 match 2019-03-06 15:28:10.677 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:10.677 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:10.681 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 330 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x5644410f58c0 con 0x5644402c0400 2019-03-06 15:28:10.681 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:10.681 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:10.681 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x5644410f58c0 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:10.685 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1673) is_readable = 1 - now=2019-03-06 15:28:10.685872 lease_expire=2019-03-06 15:28:15.119376 has v0 lc 1673 2019-03-06 15:28:10.685 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:10.685 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:10.685 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:10.685 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:10.685 7f72cc823700 20 match 2019-03-06 15:28:10.685 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:10.685 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:10.829 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 331 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441072800 con 0x5644402c0400 2019-03-06 15:28:10.829 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:10.829 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:10.829 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) dispatch 0x564441072800 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:10.829 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1673) is_readable = 1 - now=2019-03-06 15:28:10.830583 lease_expire=2019-03-06 15:28:15.119376 has v0 lc 1673 2019-03-06 15:28:10.829 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:10.829 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:10.829 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804235 }, { "osd": 1, "seq": 42949673416 }, { "osd": 2, "seq": 60129542595 }, { "osd": 3, "seq": 73014444479 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:10.829 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:10.829 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:10.829 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) setting proposal_timer 0x56443ff50870 with delay of 0.261353 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1026) propose_pending 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader).mgrstat 1027 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:11.093 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1673) queue_pending_finisher 0x5644401e4d00 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1673) trigger_propose active, proposing now 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1673) propose_pending 1674 6340 bytes 2019-03-06 15:28:11.093 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) begin for 1674 6340 bytes 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) sending begin to mon.1 2019-03-06 15:28:12.033 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1673 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8fb440 2019-03-06 15:28:12.033 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1673 fc 0 pn 300 opn 0) v4 -- 0x56443e8fb440 con 0x56443e61e900 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) sending begin to mon.2 2019-03-06 15:28:12.033 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1673 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440b5c3c0 2019-03-06 15:28:12.033 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1673 fc 0 pn 300 opn 0) v4 -- 0x564440b5c3c0 con 0x56443e61ed80 2019-03-06 15:28:12.033 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:12.033 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2227.89s 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:12.033 7f72cf028700 5 mon.a@0(leader).paxos(paxos updating c 1005..1673) is_readable = 1 - now=2019-03-06 15:28:12.035425 lease_expire=2019-03-06 15:28:15.119376 has v0 lc 1673 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:12.033 7f72cf028700 5 mon.a@0(leader).paxos(paxos updating c 1005..1673) is_readable = 1 - now=2019-03-06 15:28:12.035521 lease_expire=2019-03-06 15:28:15.119376 has v0 lc 1673 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).log v517 log 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..517) maybe_trim trim_to 17 would only trim 16 < paxos_service_trim_min 250 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:12.033 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:12.033 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:12.033 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:12.033 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f61c0 con 0x5644402c0400 2019-03-06 15:28:12.365 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5793 ==== paxos(accept lc 1673 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440b5c3c0 con 0x56443e61ed80 2019-03-06 15:28:12.365 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:12.365 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:12.365 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow all 2019-03-06 15:28:12.365 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow all 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) handle_accept paxos(accept lc 1673 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) now 0,2 have accepted 2019-03-06 15:28:12.365 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4882 ==== paxos(accept lc 1673 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e8fb440 con 0x56443e61e900 2019-03-06 15:28:12.365 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:12.365 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:12.365 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow all 2019-03-06 15:28:12.365 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.365 7f72cc823700 20 allow all 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) handle_accept paxos(accept lc 1673 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) now 0,1,2 have accepted 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) got majority, committing, done with update 2019-03-06 15:28:12.365 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1673) commit_start 1674 2019-03-06 15:28:12.397 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1673) commit_finish 1674 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1674) sending commit to mon.1 2019-03-06 15:28:12.397 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1674 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8fa680 2019-03-06 15:28:12.397 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1674 fc 0 pn 300 opn 0) v4 -- 0x56443e8fa680 con 0x56443e61e900 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1674) sending commit to mon.2 2019-03-06 15:28:12.397 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1674 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440eea3c0 2019-03-06 15:28:12.397 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1674 fc 0 pn 300 opn 0) v4 -- 0x564440eea3c0 con 0x56443e61ed80 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 517 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:12.397 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:12.401 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:12.405 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:12.405 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:12.417 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:12.417 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:12.421 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:12.421 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).mgrstat 1027 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1027 service_map e2 0progress events 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:12.421 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78000 con 0x5644402c0400 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1674) commit_proposal 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) _active - not active 2019-03-06 15:28:12.421 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1674) extend_lease now+5 (2019-03-06 15:28:17.424930) 2019-03-06 15:28:12.421 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1674 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440eea100 2019-03-06 15:28:12.421 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1674 fc 1005 pn 0 opn 0) v4 -- 0x564440eea100 con 0x56443e61e900 2019-03-06 15:28:12.421 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1674 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e9f2b00 2019-03-06 15:28:12.421 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1674 fc 1005 pn 0 opn 0) v4 -- 0x56443e9f2b00 con 0x56443e61ed80 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1674) finish_round 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1674) finish_round waiting_for_acting 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) _active 2019-03-06 15:28:12.421 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1027) _active creating new pending 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).mgrstat 1027 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1674) finish_round waiting_for_readable 2019-03-06 15:28:12.421 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1674) finish_round waiting_for_writeable 2019-03-06 15:28:12.421 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1674) finish_round done w/ waiters, state active 2019-03-06 15:28:12.449 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5794 ==== paxos(lease_ack lc 1674 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e9f2b00 con 0x56443e61ed80 2019-03-06 15:28:12.449 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:12.449 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:12.449 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow all 2019-03-06 15:28:12.449 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow all 2019-03-06 15:28:12.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1674) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:12.449 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4883 ==== paxos(lease_ack lc 1674 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440eea100 con 0x56443e61e900 2019-03-06 15:28:12.449 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:12.449 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:12.449 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow all 2019-03-06 15:28:12.449 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.449 7f72cc823700 20 allow all 2019-03-06 15:28:12.449 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1674) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:12.549 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 332 ==== log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 ==== 953+0+0 (crc 0 0 0) 0x5644411f61c0 con 0x5644402c0400 2019-03-06 15:28:12.549 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:12.549 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..517) dispatch 0x5644411f61c0 log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:12.549 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1674) is_readable = 1 - now=2019-03-06 15:28:12.553502 lease_expire=2019-03-06 15:28:17.424930 has v0 lc 1674 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).log v517 preprocess_query log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).log v517 preprocess_log log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 from mgr.6993 2019-03-06 15:28:12.549 7f72cc823700 20 is_capable service=log command= write addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:12.549 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:12.549 7f72cc823700 20 match 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).log v517 prepare_update log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).log v517 prepare_log log(5 entries from seq 145 at 2019-03-06 15:28:02.818744) v1 from mgr.6993 2019-03-06 15:28:12.549 7f72cc823700 10 mon.a@0(leader).log v517 logging 2019-03-06 15:28:02.818744 mgr.z (mgr.6993) 145 : cluster [DBG] pgmap v142: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:12.553 7f72cc823700 10 mon.a@0(leader).log v517 logging 2019-03-06 15:28:04.822504 mgr.z (mgr.6993) 146 : cluster [DBG] pgmap v143: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:12.553 7f72cc823700 10 mon.a@0(leader).log v517 logging 2019-03-06 15:28:06.824310 mgr.z (mgr.6993) 147 : cluster [DBG] pgmap v144: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:12.553 7f72cc823700 10 mon.a@0(leader).log v517 logging 2019-03-06 15:28:08.826230 mgr.z (mgr.6993) 148 : cluster [DBG] pgmap v145: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:12.553 7f72cc823700 10 mon.a@0(leader).log v517 logging 2019-03-06 15:28:10.829565 mgr.z (mgr.6993) 149 : cluster [DBG] pgmap v146: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:12.553 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..517) setting proposal_timer 0x56443ff0c750 with delay of 0.845478 2019-03-06 15:28:12.685 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 333 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e8fa680 con 0x5644402c0400 2019-03-06 15:28:12.685 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:12.685 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:12.685 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e8fa680 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:12.685 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1674) is_readable = 1 - now=2019-03-06 15:28:12.688281 lease_expire=2019-03-06 15:28:17.424930 has v0 lc 1674 2019-03-06 15:28:12.685 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:12.685 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:12.685 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:12.685 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:12.685 7f72cc823700 20 match 2019-03-06 15:28:12.685 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:12.685 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:12.705 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4884 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1503 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56444207de40 con 0x56443e61e900 2019-03-06 15:28:12.705 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:12.705 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:12.705 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.705 7f72cc823700 20 allow all 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:12.705 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:12.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:12.705 7f72cc823700 20 allow all 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e9f5440 from v2:192.168.178.28:40578/0 2019-03-06 15:28:12.705 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8aac0 for mgr.? 2019-03-06 15:28:12.705 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e9f5440 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440498c00 2019-03-06 15:28:12.705 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1674) is_readable = 1 - now=2019-03-06 15:28:12.706659 lease_expire=2019-03-06 15:28:17.424930 has v0 lc 1674 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:12.705 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1503) v3 -- 0x56443ec0ad80 con 0x56443e61e900 2019-03-06 15:28:12.705 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:12.705 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:12.705 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:12.705 7f72cc823700 20 match 2019-03-06 15:28:12.705 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:12.705 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:12.829 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 334 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441071180 con 0x5644402c0400 2019-03-06 15:28:12.829 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:12.829 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:12.829 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) dispatch 0x564441071180 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:12.829 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1674) is_readable = 1 - now=2019-03-06 15:28:12.832317 lease_expire=2019-03-06 15:28:17.424930 has v0 lc 1674 2019-03-06 15:28:12.829 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:12.829 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:12.829 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804235 }, { "osd": 1, "seq": 42949673416 }, { "osd": 2, "seq": 60129542595 }, { "osd": 3, "seq": 73014444479 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:12.833 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:12.833 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:12.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) setting proposal_timer 0x5644420b7fb0 with delay of 0.56504 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1027) propose_pending 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader).mgrstat 1028 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:13.397 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1674) queue_pending_finisher 0x5644401e43a0 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1674) trigger_propose active, proposing now 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1674) propose_pending 1675 6340 bytes 2019-03-06 15:28:13.397 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) begin for 1675 6340 bytes 2019-03-06 15:28:14.021 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:14.021 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:14.021 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443eb3fd40 con 0x5644403b1b00 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) sending begin to mon.1 2019-03-06 15:28:14.521 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1674 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644410f7440 2019-03-06 15:28:14.521 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1674 fc 0 pn 300 opn 0) v4 -- 0x5644410f7440 con 0x56443e61e900 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) sending begin to mon.2 2019-03-06 15:28:14.521 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1674 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e13440 2019-03-06 15:28:14.521 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1674 fc 0 pn 300 opn 0) v4 -- 0x564440e13440 con 0x56443e61ed80 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..517) propose_pending 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).log v517 encode_full log v 517 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).log v517 encode_pending v518 2019-03-06 15:28:14.521 7f72cf028700 5 mon.a@0(leader).paxos(paxos updating c 1005..1674) queue_pending_finisher 0x5644401e4790 2019-03-06 15:28:14.521 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) trigger_propose not active, will propose later 2019-03-06 15:28:14.569 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5795 ==== paxos(accept lc 1674 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440e13440 con 0x56443e61ed80 2019-03-06 15:28:14.569 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:14.569 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.569 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow all 2019-03-06 15:28:14.569 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow all 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) handle_accept paxos(accept lc 1674 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) now 0,2 have accepted 2019-03-06 15:28:14.569 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4885 ==== paxos(accept lc 1674 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x5644410f7440 con 0x56443e61e900 2019-03-06 15:28:14.569 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:14.569 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.569 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow all 2019-03-06 15:28:14.569 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.569 7f72cc823700 20 allow all 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) handle_accept paxos(accept lc 1674 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) now 0,1,2 have accepted 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) got majority, committing, done with update 2019-03-06 15:28:14.569 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1674) commit_start 1675 2019-03-06 15:28:14.601 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1674) commit_finish 1675 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1675) sending commit to mon.1 2019-03-06 15:28:14.601 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1675 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e9b700 2019-03-06 15:28:14.601 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1675 fc 0 pn 300 opn 0) v4 -- 0x564440e9b700 con 0x56443e61e900 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1675) sending commit to mon.2 2019-03-06 15:28:14.601 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1675 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440b6bc80 2019-03-06 15:28:14.601 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1675 fc 0 pn 300 opn 0) v4 -- 0x564440b6bc80 con 0x56443e61ed80 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) refresh 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).log v517 update_from_paxos version 517 summary v 517 2019-03-06 15:28:14.601 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:14.605 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:14.605 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:14.605 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:14.609 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:14.609 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:14.621 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:14.625 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:14.625 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:14.625 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) refresh 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).mgrstat 1028 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1028 service_map e2 0progress events 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:14.625 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78200 con 0x5644402c0400 2019-03-06 15:28:14.625 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:14.625 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:14.625 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..517) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1675) commit_proposal 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) _active - not active 2019-03-06 15:28:14.629 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1675) extend_lease now+5 (2019-03-06 15:28:19.630402) 2019-03-06 15:28:14.629 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1675 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440d44840 2019-03-06 15:28:14.629 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1675 fc 1005 pn 0 opn 0) v4 -- 0x564440d44840 con 0x56443e61e900 2019-03-06 15:28:14.629 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1675 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8fa3c0 2019-03-06 15:28:14.629 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1675 fc 1005 pn 0 opn 0) v4 -- 0x56443e8fa3c0 con 0x56443e61ed80 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1675) finish_round 2019-03-06 15:28:14.629 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1675) finish_round waiting_for_acting 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) _active 2019-03-06 15:28:14.629 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1028) _active creating new pending 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).mgrstat 1028 2019-03-06 15:28:14.629 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:14.629 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1675) finish_round waiting_for_readable 2019-03-06 15:28:14.629 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1675) finish_round waiting_for_writeable 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1675) finish_round done w/ waiters, state active 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1675) propose_pending 1676 46996 bytes 2019-03-06 15:28:14.629 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) begin for 1676 46996 bytes 2019-03-06 15:28:14.649 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4886 ==== paxos(lease_ack lc 1675 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440d44840 con 0x56443e61e900 2019-03-06 15:28:14.669 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) sending begin to mon.1 2019-03-06 15:28:14.669 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1675 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e9f4940 2019-03-06 15:28:14.669 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1675 fc 0 pn 300 opn 0) v4 -- 0x56443e9f4940 con 0x56443e61e900 2019-03-06 15:28:14.669 7f72c901c700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) sending begin to mon.2 2019-03-06 15:28:14.669 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1675 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644410f58c0 2019-03-06 15:28:14.669 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1675 fc 0 pn 300 opn 0) v4 -- 0x5644410f58c0 con 0x56443e61ed80 2019-03-06 15:28:14.669 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:14.669 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.669 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow all 2019-03-06 15:28:14.669 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow all 2019-03-06 15:28:14.669 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) handle_lease_ack from mon.1 -- stray (probably since revoked) 2019-03-06 15:28:14.669 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5796 ==== paxos(lease_ack lc 1675 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8fa3c0 con 0x56443e61ed80 2019-03-06 15:28:14.669 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:14.669 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.669 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow all 2019-03-06 15:28:14.669 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.669 7f72cc823700 20 allow all 2019-03-06 15:28:14.669 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) handle_lease_ack from mon.2 -- stray (probably since revoked) 2019-03-06 15:28:14.701 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 335 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e9f4940 con 0x5644402c0400 2019-03-06 15:28:14.701 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:14.701 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:14.701 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e9f4940 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:14.701 7f72cc823700 5 mon.a@0(leader).paxos(paxos updating c 1005..1675) is_readable = 1 - now=2019-03-06 15:28:14.704440 lease_expire=2019-03-06 15:28:19.630402 has v0 lc 1675 2019-03-06 15:28:14.701 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:14.701 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:14.701 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:14.701 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:14.701 7f72cc823700 20 match 2019-03-06 15:28:14.701 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:14.701 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:14.705 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5797 ==== paxos(accept lc 1675 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x5644410f58c0 con 0x56443e61ed80 2019-03-06 15:28:14.705 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:14.705 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.705 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow all 2019-03-06 15:28:14.705 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow all 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) handle_accept paxos(accept lc 1675 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) now 0,2 have accepted 2019-03-06 15:28:14.705 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4887 ==== paxos(accept lc 1675 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440e9b700 con 0x56443e61e900 2019-03-06 15:28:14.705 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:14.705 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.705 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow all 2019-03-06 15:28:14.705 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.705 7f72cc823700 20 allow all 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) handle_accept paxos(accept lc 1675 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) now 0,1,2 have accepted 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) got majority, committing, done with update 2019-03-06 15:28:14.705 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1675) commit_start 1676 2019-03-06 15:28:14.737 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1675) commit_finish 1676 2019-03-06 15:28:14.737 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1676) sending commit to mon.1 2019-03-06 15:28:14.737 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1676 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f4c5440 2019-03-06 15:28:14.737 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1676 fc 0 pn 300 opn 0) v4 -- 0x56443f4c5440 con 0x56443e61e900 2019-03-06 15:28:14.737 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1676) sending commit to mon.2 2019-03-06 15:28:14.737 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1676 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644412038c0 2019-03-06 15:28:14.737 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1676 fc 0 pn 300 opn 0) v4 -- 0x5644412038c0 con 0x56443e61ed80 2019-03-06 15:28:14.737 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:14.737 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 517 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos latest full 517 2019-03-06 15:28:14.741 7f72c901c700 7 mon.a@0(leader).log v518 update_from_paxos applying incremental log 518 2019-03-06 15:28:02.818744 mgr.z (mgr.6993) 145 : cluster [DBG] pgmap v142: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:14.741 7f72c901c700 7 mon.a@0(leader).log v518 update_from_paxos applying incremental log 518 2019-03-06 15:28:04.822504 mgr.z (mgr.6993) 146 : cluster [DBG] pgmap v143: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:14.741 7f72c901c700 7 mon.a@0(leader).log v518 update_from_paxos applying incremental log 518 2019-03-06 15:28:06.824310 mgr.z (mgr.6993) 147 : cluster [DBG] pgmap v144: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:14.741 7f72c901c700 7 mon.a@0(leader).log v518 update_from_paxos applying incremental log 518 2019-03-06 15:28:08.826230 mgr.z (mgr.6993) 148 : cluster [DBG] pgmap v145: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:14.741 7f72c901c700 7 mon.a@0(leader).log v518 update_from_paxos applying incremental log 518 2019-03-06 15:28:10.829565 mgr.z (mgr.6993) 149 : cluster [DBG] pgmap v146: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:14.741 7f72c901c700 15 mon.a@0(leader).log v518 update_from_paxos logging for 1 channels 2019-03-06 15:28:14.741 7f72c901c700 15 mon.a@0(leader).log v518 update_from_paxos channel 'cluster' logging 755 bytes 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 check_subs 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 check_sub client wants log-info ver 518 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 _create_sub_incremental level 1 ver 518 cur summary ver 518 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:14.741 7f72c901c700 20 mon.a@0(leader).log v518 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 _create_sub_incremental incremental message ready (0 entries) 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).log v518 check_sub sending message to mgr.6993 with 0 entries (version 518) 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:14.741 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:14.749 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:14.749 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:14.761 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:14.761 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:14.761 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:14.761 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:14.761 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) refresh 2019-03-06 15:28:14.761 7f72c901c700 10 mon.a@0(leader).mgrstat 1028 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1028 service_map e2 0progress events 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78400 con 0x5644402c0400 2019-03-06 15:28:14.765 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:14.765 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1676) commit_proposal 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) _active - not active 2019-03-06 15:28:14.765 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1676) extend_lease now+5 (2019-03-06 15:28:19.767023) 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1676 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f37600 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1676 fc 1005 pn 0 opn 0) v4 -- 0x564440f37600 con 0x56443e61e900 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1676 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f37340 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1676 fc 1005 pn 0 opn 0) v4 -- 0x564440f37340 con 0x56443e61ed80 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1676) finish_round 2019-03-06 15:28:14.765 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1676) finish_round waiting_for_acting 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) _active 2019-03-06 15:28:14.765 7f72c901c700 7 mon.a@0(leader).paxosservice(logm 1..518) _active creating new pending 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).log v518 create_pending v 519 2019-03-06 15:28:14.765 7f72c901c700 7 mon.a@0(leader).log v518 _updated_log for mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:14.765 7f72c901c700 2 mon.a@0(leader) e1 send_reply 0x564440936760 0x56443fd8cb40 log(last 149) v1 2019-03-06 15:28:14.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- log(last 149) v1 -- 0x56443fd8cb40 con 0x5644402c0400 2019-03-06 15:28:14.765 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1676) finish_round waiting_for_readable 2019-03-06 15:28:14.765 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1676) finish_round waiting_for_writeable 2019-03-06 15:28:14.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1676) finish_round done w/ waiters, state active 2019-03-06 15:28:14.789 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5798 ==== paxos(lease_ack lc 1676 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440f37340 con 0x56443e61ed80 2019-03-06 15:28:14.789 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:14.789 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.789 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow all 2019-03-06 15:28:14.789 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow all 2019-03-06 15:28:14.789 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1676) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:14.789 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4888 ==== paxos(lease_ack lc 1676 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440f37600 con 0x56443e61e900 2019-03-06 15:28:14.789 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:14.789 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.789 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow all 2019-03-06 15:28:14.789 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.789 7f72cc823700 20 allow all 2019-03-06 15:28:14.789 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1676) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:14.793 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4889 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1504 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56443f4c5440 con 0x56443e61e900 2019-03-06 15:28:14.793 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:14.793 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:14.793 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.793 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.793 7f72cc823700 20 allow all 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:14.793 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:14.793 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:14.793 7f72cc823700 20 allow all 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x5644419262c0 from v2:192.168.178.28:40578/0 2019-03-06 15:28:14.793 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa88240 for mgr.? 2019-03-06 15:28:14.793 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x5644419262c0 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444049b200 2019-03-06 15:28:14.793 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1676) is_readable = 1 - now=2019-03-06 15:28:14.795740 lease_expire=2019-03-06 15:28:19.767023 has v0 lc 1676 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:14.793 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1504) v3 -- 0x56443ec0abc0 con 0x56443e61e900 2019-03-06 15:28:14.793 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:14.793 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:14.793 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:14.793 7f72cc823700 20 match 2019-03-06 15:28:14.793 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:14.793 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:14.833 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 336 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441072580 con 0x5644402c0400 2019-03-06 15:28:14.833 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:14.833 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:14.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) dispatch 0x564441072580 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:14.833 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1676) is_readable = 1 - now=2019-03-06 15:28:14.834296 lease_expire=2019-03-06 15:28:19.767023 has v0 lc 1676 2019-03-06 15:28:14.833 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:14.833 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:14.833 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804236 }, { "osd": 1, "seq": 42949673417 }, { "osd": 2, "seq": 60129542596 }, { "osd": 3, "seq": 73014444480 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:14.833 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:14.833 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:14.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) setting proposal_timer 0x5644421d60f0 with delay of 0.905218 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1028) propose_pending 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader).mgrstat 1029 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:15.737 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1676) queue_pending_finisher 0x5644401e44e0 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1676) trigger_propose active, proposing now 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1676) propose_pending 1677 6340 bytes 2019-03-06 15:28:15.737 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) begin for 1677 6340 bytes 2019-03-06 15:28:16.609 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) sending begin to mon.1 2019-03-06 15:28:16.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1676 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444199d700 2019-03-06 15:28:16.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1676 fc 0 pn 300 opn 0) v4 -- 0x56444199d700 con 0x56443e61e900 2019-03-06 15:28:16.609 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) sending begin to mon.2 2019-03-06 15:28:16.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1676 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f4139c0 2019-03-06 15:28:16.609 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1676 fc 0 pn 300 opn 0) v4 -- 0x56443f4139c0 con 0x56443e61ed80 2019-03-06 15:28:16.685 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4890 ==== paxos(accept lc 1676 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56444199d700 con 0x56443e61e900 2019-03-06 15:28:16.685 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:16.685 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:16.685 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow all 2019-03-06 15:28:16.685 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow all 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) handle_accept paxos(accept lc 1676 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) now 0,1 have accepted 2019-03-06 15:28:16.685 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5799 ==== paxos(accept lc 1676 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443f4139c0 con 0x56443e61ed80 2019-03-06 15:28:16.685 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:16.685 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:16.685 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow all 2019-03-06 15:28:16.685 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.685 7f72cc823700 20 allow all 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) handle_accept paxos(accept lc 1676 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) now 0,1,2 have accepted 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) got majority, committing, done with update 2019-03-06 15:28:16.685 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1676) commit_start 1677 2019-03-06 15:28:16.705 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 337 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x564441e26840 con 0x5644402c0400 2019-03-06 15:28:16.705 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:16.705 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:16.705 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564441e26840 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:16.705 7f72cc823700 5 mon.a@0(leader).paxos(paxos writing c 1005..1676) is_readable = 1 - now=2019-03-06 15:28:16.706840 lease_expire=2019-03-06 15:28:19.767023 has v0 lc 1676 2019-03-06 15:28:16.705 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:16.705 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:16.705 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:16.705 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:16.705 7f72cc823700 20 match 2019-03-06 15:28:16.705 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:16.705 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:16.765 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1676) commit_finish 1677 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1677) sending commit to mon.1 2019-03-06 15:28:16.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1677 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444184adc0 2019-03-06 15:28:16.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1677 fc 0 pn 300 opn 0) v4 -- 0x56444184adc0 con 0x56443e61e900 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1677) sending commit to mon.2 2019-03-06 15:28:16.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1677 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441f903c0 2019-03-06 15:28:16.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1677 fc 0 pn 300 opn 0) v4 -- 0x564441f903c0 con 0x56443e61ed80 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 518 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:16.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:16.773 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:16.773 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:16.785 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:16.785 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:16.785 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:16.789 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).mgrstat 1029 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1029 service_map e2 0progress events 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:16.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78600 con 0x5644402c0400 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1677) commit_proposal 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) _active - not active 2019-03-06 15:28:16.789 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1677) extend_lease now+5 (2019-03-06 15:28:21.791204) 2019-03-06 15:28:16.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1677 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440ee9340 2019-03-06 15:28:16.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1677 fc 1005 pn 0 opn 0) v4 -- 0x564440ee9340 con 0x56443e61e900 2019-03-06 15:28:16.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1677 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443f442dc0 2019-03-06 15:28:16.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1677 fc 1005 pn 0 opn 0) v4 -- 0x56443f442dc0 con 0x56443e61ed80 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1677) finish_round 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1677) finish_round waiting_for_acting 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) _active 2019-03-06 15:28:16.789 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 770..1029) _active creating new pending 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).mgrstat 1029 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1677) finish_round waiting_for_readable 2019-03-06 15:28:16.789 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1677) finish_round waiting_for_writeable 2019-03-06 15:28:16.789 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1677) finish_round done w/ waiters, state active 2019-03-06 15:28:16.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5800 ==== paxos(lease_ack lc 1677 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443f442dc0 con 0x56443e61ed80 2019-03-06 15:28:16.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:16.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:16.813 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow all 2019-03-06 15:28:16.813 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow all 2019-03-06 15:28:16.813 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1677) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:16.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4891 ==== paxos(lease_ack lc 1677 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440ee9340 con 0x56443e61e900 2019-03-06 15:28:16.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:16.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:16.813 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.813 7f72cc823700 20 allow all 2019-03-06 15:28:16.817 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow all 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1677) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:16.817 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4892 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1505 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56444184adc0 con 0x56443e61e900 2019-03-06 15:28:16.817 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:16.817 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:16.817 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow all 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:16.817 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:16.817 7f72cc823700 20 allow all 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e94b080 from v2:192.168.178.28:40578/0 2019-03-06 15:28:16.817 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8b180 for mgr.? 2019-03-06 15:28:16.817 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e94b080 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499800 2019-03-06 15:28:16.817 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1677) is_readable = 1 - now=2019-03-06 15:28:16.819425 lease_expire=2019-03-06 15:28:21.791204 has v0 lc 1677 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:16.817 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1505) v3 -- 0x56443ec09f80 con 0x56443e61e900 2019-03-06 15:28:16.817 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:16.817 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:16.817 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:16.817 7f72cc823700 20 match 2019-03-06 15:28:16.817 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:16.817 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:16.833 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 338 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441072d00 con 0x5644402c0400 2019-03-06 15:28:16.833 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:16.833 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:16.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) dispatch 0x564441072d00 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:16.833 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1677) is_readable = 1 - now=2019-03-06 15:28:16.835199 lease_expire=2019-03-06 15:28:21.791204 has v0 lc 1677 2019-03-06 15:28:16.833 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:16.833 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:16.833 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804236 }, { "osd": 1, "seq": 42949673417 }, { "osd": 2, "seq": 60129542596 }, { "osd": 3, "seq": 73014444480 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:16.833 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:16.833 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:16.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) setting proposal_timer 0x56444217d6b0 with delay of 0.930218 2019-03-06 15:28:17.033 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:17.033 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2232.9s 2019-03-06 15:28:17.033 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:17.033 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:17.033 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1677) is_readable = 1 - now=2019-03-06 15:28:17.037671 lease_expire=2019-03-06 15:28:21.791204 has v0 lc 1677 2019-03-06 15:28:17.033 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:17.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:17.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:17.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:17.033 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:17.033 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:17.033 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1677) is_readable = 1 - now=2019-03-06 15:28:17.037802 lease_expire=2019-03-06 15:28:21.791204 has v0 lc 1677 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).log v518 log 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..518) maybe_trim trim_to 18 would only trim 17 < paxos_service_trim_min 250 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:17.037 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) maybe_trim trimming to 1024, 254 states 2019-03-06 15:28:17.037 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim from 770 to 1024 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 770 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 771 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 772 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 773 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 774 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 775 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 776 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 777 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 778 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 779 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 780 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 781 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 782 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 783 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 784 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 785 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 786 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 787 2019-03-06 15:28:17.037 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 788 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 789 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 790 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 791 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 792 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 793 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 794 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 795 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 796 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 797 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 798 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 799 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 800 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 801 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 802 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 803 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 804 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 805 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 806 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 807 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 808 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 809 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 810 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 811 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 812 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 813 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 814 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 815 2019-03-06 15:28:17.041 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 816 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 817 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 818 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 819 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 820 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 821 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 822 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 823 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 824 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 825 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 826 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 827 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 828 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 829 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 830 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 831 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 832 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 833 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 834 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 835 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 836 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 837 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 838 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 839 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 840 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 841 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 842 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 843 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 844 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 845 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 846 2019-03-06 15:28:17.045 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 847 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 848 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 849 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 850 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 851 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 852 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 853 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 854 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 855 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 856 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 857 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 858 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 859 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 860 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 861 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 862 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 863 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 864 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 865 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 866 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 867 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 868 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 869 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 870 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 871 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 872 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 873 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 874 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 875 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 876 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 877 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 878 2019-03-06 15:28:17.049 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 879 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 880 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 881 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 882 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 883 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 884 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 885 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 886 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 887 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 888 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 889 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 890 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 891 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 892 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 893 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 894 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 895 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 896 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 897 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 898 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 899 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 900 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 901 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 902 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 903 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 904 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 905 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 906 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 907 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 908 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 909 2019-03-06 15:28:17.053 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 910 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 911 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 912 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 913 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 914 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 915 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 916 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 917 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 918 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 919 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 920 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 921 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 922 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 923 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 924 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 925 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 926 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 927 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 928 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 929 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 930 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 931 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 932 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 933 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 934 2019-03-06 15:28:17.057 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 935 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 936 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 937 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 938 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 939 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 940 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 941 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 942 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 943 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 944 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 945 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 946 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 947 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 948 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 949 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 950 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 951 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 952 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 953 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 954 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 955 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 956 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 957 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 958 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 959 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 960 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 961 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 962 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 963 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 964 2019-03-06 15:28:17.061 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 965 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 966 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 967 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 968 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 969 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 970 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 971 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 972 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 973 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 974 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 975 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 976 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 977 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 978 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 979 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 980 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 981 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 982 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 983 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 984 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 985 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 986 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 987 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 988 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 989 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 990 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 991 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 992 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 993 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 994 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 995 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 996 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 997 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 998 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 999 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1000 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1001 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1002 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1003 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1004 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1005 2019-03-06 15:28:17.065 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1006 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1007 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1008 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1009 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1010 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1011 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1012 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1013 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1014 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1015 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1016 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1017 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1018 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1019 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1020 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1021 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1022 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) trim 1023 2019-03-06 15:28:17.069 7f72cf028700 20 mon.a@0(leader).paxosservice(mgrstat 770..1029) compacting prefix mgrstat 2019-03-06 15:28:17.069 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1677) trigger_propose active, proposing now 2019-03-06 15:28:17.069 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1677) propose_pending 1678 8569 bytes 2019-03-06 15:28:17.069 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) begin for 1678 8569 bytes 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) sending begin to mon.1 2019-03-06 15:28:17.109 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1677 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440b82680 2019-03-06 15:28:17.109 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1677 fc 0 pn 300 opn 0) v4 -- 0x564440b82680 con 0x56443e61e900 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) sending begin to mon.2 2019-03-06 15:28:17.109 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1677 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e98000 2019-03-06 15:28:17.109 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1677 fc 0 pn 300 opn 0) v4 -- 0x564440e98000 con 0x56443e61ed80 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:17.109 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:17.109 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:17.109 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:17.109 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:17.109 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f4240 con 0x5644402c0400 2019-03-06 15:28:17.149 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4893 ==== paxos(accept lc 1677 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440b82680 con 0x56443e61e900 2019-03-06 15:28:17.149 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:17.149 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.149 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow all 2019-03-06 15:28:17.149 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow all 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) handle_accept paxos(accept lc 1677 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) now 0,1 have accepted 2019-03-06 15:28:17.149 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5801 ==== paxos(accept lc 1677 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440e98000 con 0x56443e61ed80 2019-03-06 15:28:17.149 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:17.149 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.149 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow all 2019-03-06 15:28:17.149 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.149 7f72cc823700 20 allow all 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) handle_accept paxos(accept lc 1677 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) now 0,1,2 have accepted 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) got majority, committing, done with update 2019-03-06 15:28:17.149 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1677) commit_start 1678 2019-03-06 15:28:17.201 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1677) commit_finish 1678 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1678) sending commit to mon.1 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1678 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440d463c0 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1678 fc 0 pn 300 opn 0) v4 -- 0x564440d463c0 con 0x56443e61e900 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c801a700 4 rocksdb: [/home/sebastian/Repos/ceph/src/rocksdb/db/db_impl_compaction_flush.cc:1060] [default] Manual compaction starting 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1678) sending commit to mon.2 2019-03-06 15:28:17.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1678 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e982c0 2019-03-06 15:28:17.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1678 fc 0 pn 300 opn 0) v4 -- 0x564440e982c0 con 0x56443e61ed80 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 518 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:17.201 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:17.205 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:17.209 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:17.209 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:17.221 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:17.221 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:17.225 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:17.225 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1029) refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).mgrstat 1029 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1029 service_map e2 0progress events 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:17.225 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78800 con 0x5644402c0400 2019-03-06 15:28:17.225 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:17.225 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1029) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1678) commit_proposal 2019-03-06 15:28:17.225 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1678) extend_lease now+5 (2019-03-06 15:28:22.227425) 2019-03-06 15:28:17.225 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1678 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56444199a000 2019-03-06 15:28:17.225 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1678 fc 1005 pn 0 opn 0) v4 -- 0x56444199a000 con 0x56443e61e900 2019-03-06 15:28:17.225 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1678 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8a2c00 2019-03-06 15:28:17.225 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1678 fc 1005 pn 0 opn 0) v4 -- 0x56443e8a2c00 con 0x56443e61ed80 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1678) finish_round 2019-03-06 15:28:17.225 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1678) finish_round waiting_for_acting 2019-03-06 15:28:17.225 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1678) finish_round waiting_for_readable 2019-03-06 15:28:17.225 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1678) finish_round waiting_for_writeable 2019-03-06 15:28:17.225 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1678) finish_round done w/ waiters, state active 2019-03-06 15:28:17.265 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4894 ==== paxos(lease_ack lc 1678 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56444199a000 con 0x56443e61e900 2019-03-06 15:28:17.265 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:17.265 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.265 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.265 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.265 7f72cc823700 20 allow all 2019-03-06 15:28:17.265 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.265 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.265 7f72cc823700 20 allow all 2019-03-06 15:28:17.265 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1678) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:17.329 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5802 ==== paxos(lease_ack lc 1678 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8a2c00 con 0x56443e61ed80 2019-03-06 15:28:17.329 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:17.329 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.329 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.329 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.329 7f72cc823700 20 allow all 2019-03-06 15:28:17.329 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.329 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.329 7f72cc823700 20 allow all 2019-03-06 15:28:17.329 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1678) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1029) propose_pending 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader).mgrstat 1030 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:17.765 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1678) queue_pending_finisher 0x5644401e4680 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1678) trigger_propose active, proposing now 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1678) propose_pending 1679 6340 bytes 2019-03-06 15:28:17.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) begin for 1679 6340 bytes 2019-03-06 15:28:17.817 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) sending begin to mon.1 2019-03-06 15:28:17.817 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1678 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f3b4ec0 2019-03-06 15:28:17.817 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1678 fc 0 pn 300 opn 0) v4 -- 0x56443f3b4ec0 con 0x56443e61e900 2019-03-06 15:28:17.817 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) sending begin to mon.2 2019-03-06 15:28:17.817 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1678 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444184a2c0 2019-03-06 15:28:17.817 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1678 fc 0 pn 300 opn 0) v4 -- 0x56444184a2c0 con 0x56443e61ed80 2019-03-06 15:28:17.865 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4895 ==== paxos(accept lc 1678 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443f3b4ec0 con 0x56443e61e900 2019-03-06 15:28:17.865 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:17.865 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.865 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow all 2019-03-06 15:28:17.865 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow all 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) handle_accept paxos(accept lc 1678 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) now 0,1 have accepted 2019-03-06 15:28:17.865 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5803 ==== paxos(accept lc 1678 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56444184a2c0 con 0x56443e61ed80 2019-03-06 15:28:17.865 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:17.865 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.865 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow all 2019-03-06 15:28:17.865 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.865 7f72cc823700 20 allow all 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) handle_accept paxos(accept lc 1678 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) now 0,1,2 have accepted 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) got majority, committing, done with update 2019-03-06 15:28:17.865 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1678) commit_start 1679 2019-03-06 15:28:17.913 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1678) commit_finish 1679 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1679) sending commit to mon.1 2019-03-06 15:28:17.913 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1679 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644413063c0 2019-03-06 15:28:17.913 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1679 fc 0 pn 300 opn 0) v4 -- 0x5644413063c0 con 0x56443e61e900 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1679) sending commit to mon.2 2019-03-06 15:28:17.913 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1679 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440b5adc0 2019-03-06 15:28:17.913 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1679 fc 0 pn 300 opn 0) v4 -- 0x564440b5adc0 con 0x56443e61ed80 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 518 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:17.913 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:17.921 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:17.921 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:17.933 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:17.933 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:17.933 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:17.937 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).mgrstat 1030 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1030 service_map e2 0progress events 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:17.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78a00 con 0x5644402c0400 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1679) commit_proposal 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) _active - not active 2019-03-06 15:28:17.937 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1679) extend_lease now+5 (2019-03-06 15:28:22.940146) 2019-03-06 15:28:17.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1679 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8a0580 2019-03-06 15:28:17.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1679 fc 1005 pn 0 opn 0) v4 -- 0x56443e8a0580 con 0x56443e61e900 2019-03-06 15:28:17.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1679 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443f46cb00 2019-03-06 15:28:17.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1679 fc 1005 pn 0 opn 0) v4 -- 0x56443f46cb00 con 0x56443e61ed80 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1679) finish_round 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1679) finish_round waiting_for_acting 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) _active 2019-03-06 15:28:17.937 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1030) _active creating new pending 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).mgrstat 1030 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1679) finish_round waiting_for_readable 2019-03-06 15:28:17.937 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1679) finish_round waiting_for_writeable 2019-03-06 15:28:17.937 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1679) finish_round done w/ waiters, state active 2019-03-06 15:28:17.981 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5804 ==== paxos(lease_ack lc 1679 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443f46cb00 con 0x56443e61ed80 2019-03-06 15:28:17.981 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:17.981 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.981 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow all 2019-03-06 15:28:17.981 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow all 2019-03-06 15:28:17.981 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1679) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:17.981 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4896 ==== paxos(lease_ack lc 1679 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e8a0580 con 0x56443e61e900 2019-03-06 15:28:17.981 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:17.981 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:17.981 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow all 2019-03-06 15:28:17.981 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:17.981 7f72cc823700 20 allow all 2019-03-06 15:28:17.981 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1679) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:18.721 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 339 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x5644413063c0 con 0x5644402c0400 2019-03-06 15:28:18.721 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:18.725 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x5644413063c0 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:18.725 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1679) is_readable = 1 - now=2019-03-06 15:28:18.725934 lease_expire=2019-03-06 15:28:22.940146 has v0 lc 1679 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:18.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:18.725 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 20 match 2019-03-06 15:28:18.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:18.725 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4897 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1506 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440d463c0 con 0x56443e61e900 2019-03-06 15:28:18.725 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:18.725 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:18.725 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:18.725 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:18.725 7f72cc823700 20 allow all 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:18.725 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:18.725 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:18.725 7f72cc823700 20 allow all 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e869700 from v2:192.168.178.28:40578/0 2019-03-06 15:28:18.725 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa89f80 for mgr.? 2019-03-06 15:28:18.725 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e869700 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440498600 2019-03-06 15:28:18.725 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1679) is_readable = 1 - now=2019-03-06 15:28:18.729101 lease_expire=2019-03-06 15:28:22.940146 has v0 lc 1679 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:18.725 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1506) v3 -- 0x56443ec0aa00 con 0x56443e61e900 2019-03-06 15:28:18.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:18.725 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:18.725 7f72cc823700 20 match 2019-03-06 15:28:18.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:18.725 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:18.833 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 340 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441073c00 con 0x5644402c0400 2019-03-06 15:28:18.833 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:18.833 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:18.833 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) dispatch 0x564441073c00 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:18.833 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1679) is_readable = 1 - now=2019-03-06 15:28:18.837173 lease_expire=2019-03-06 15:28:22.940146 has v0 lc 1679 2019-03-06 15:28:18.833 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:18.833 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:18.833 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804236 }, { "osd": 1, "seq": 42949673417 }, { "osd": 2, "seq": 60129542596 }, { "osd": 3, "seq": 73014444480 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:18.837 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:18.837 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:18.837 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) setting proposal_timer 0x56444133d230 with delay of 0.0748842 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1030) propose_pending 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader).mgrstat 1031 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:18.913 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1679) queue_pending_finisher 0x5644401e4520 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1679) trigger_propose active, proposing now 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1679) propose_pending 1680 6340 bytes 2019-03-06 15:28:18.913 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) begin for 1680 6340 bytes 2019-03-06 15:28:19.021 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:19.021 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:19.021 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443eb40700 con 0x5644403b1b00 2019-03-06 15:28:19.565 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) sending begin to mon.1 2019-03-06 15:28:19.565 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1679 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644405a1080 2019-03-06 15:28:19.565 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1679 fc 0 pn 300 opn 0) v4 -- 0x5644405a1080 con 0x56443e61e900 2019-03-06 15:28:19.565 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) sending begin to mon.2 2019-03-06 15:28:19.565 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1679 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644419299c0 2019-03-06 15:28:19.565 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1679 fc 0 pn 300 opn 0) v4 -- 0x5644419299c0 con 0x56443e61ed80 2019-03-06 15:28:19.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5805 ==== paxos(accept lc 1679 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x5644419299c0 con 0x56443e61ed80 2019-03-06 15:28:19.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:19.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:19.661 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow all 2019-03-06 15:28:19.661 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow all 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) handle_accept paxos(accept lc 1679 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) now 0,2 have accepted 2019-03-06 15:28:19.661 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4898 ==== paxos(accept lc 1679 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x5644405a1080 con 0x56443e61e900 2019-03-06 15:28:19.661 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:19.661 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:19.661 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow all 2019-03-06 15:28:19.661 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.661 7f72cc823700 20 allow all 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) handle_accept paxos(accept lc 1679 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) now 0,1,2 have accepted 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) got majority, committing, done with update 2019-03-06 15:28:19.661 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1679) commit_start 1680 2019-03-06 15:28:19.733 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1679) commit_finish 1680 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1680) sending commit to mon.1 2019-03-06 15:28:19.733 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1680 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443ea8f440 2019-03-06 15:28:19.733 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1680 fc 0 pn 300 opn 0) v4 -- 0x56443ea8f440 con 0x56443e61e900 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1680) sending commit to mon.2 2019-03-06 15:28:19.733 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1680 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441204940 2019-03-06 15:28:19.733 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1680 fc 0 pn 300 opn 0) v4 -- 0x564441204940 con 0x56443e61ed80 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 518 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:19.733 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:19.737 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:19.737 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:19.737 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:19.745 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:19.745 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:19.745 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).mgrstat 1031 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1031 service_map e2 0progress events 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:19.745 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78c00 con 0x5644402c0400 2019-03-06 15:28:19.745 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:19.745 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:19.745 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1680) commit_proposal 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) _active - not active 2019-03-06 15:28:19.749 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1680) extend_lease now+5 (2019-03-06 15:28:24.750049) 2019-03-06 15:28:19.749 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1680 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440f36000 2019-03-06 15:28:19.749 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1680 fc 1005 pn 0 opn 0) v4 -- 0x564440f36000 con 0x56443e61e900 2019-03-06 15:28:19.749 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1680 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8a02c0 2019-03-06 15:28:19.749 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1680 fc 1005 pn 0 opn 0) v4 -- 0x56443e8a02c0 con 0x56443e61ed80 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1680) finish_round 2019-03-06 15:28:19.749 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1680) finish_round waiting_for_acting 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) _active 2019-03-06 15:28:19.749 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1031) _active creating new pending 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).mgrstat 1031 2019-03-06 15:28:19.749 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:19.749 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1680) finish_round waiting_for_readable 2019-03-06 15:28:19.749 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1680) finish_round waiting_for_writeable 2019-03-06 15:28:19.749 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1680) finish_round done w/ waiters, state active 2019-03-06 15:28:19.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4899 ==== paxos(lease_ack lc 1680 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440f36000 con 0x56443e61e900 2019-03-06 15:28:19.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:19.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:19.813 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow all 2019-03-06 15:28:19.813 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow all 2019-03-06 15:28:19.813 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1680) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:19.813 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5806 ==== paxos(lease_ack lc 1680 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8a02c0 con 0x56443e61ed80 2019-03-06 15:28:19.813 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:19.813 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:19.813 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow all 2019-03-06 15:28:19.813 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:19.813 7f72cc823700 20 allow all 2019-03-06 15:28:19.813 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1680) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:20.725 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 341 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443ea8f440 con 0x5644402c0400 2019-03-06 15:28:20.725 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:20.725 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:20.725 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443ea8f440 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:20.725 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1680) is_readable = 1 - now=2019-03-06 15:28:20.727713 lease_expire=2019-03-06 15:28:24.750049 has v0 lc 1680 2019-03-06 15:28:20.725 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:20.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:20.725 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:20.725 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:20.725 7f72cc823700 20 match 2019-03-06 15:28:20.725 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:20.725 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:20.745 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4900 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1507 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56443f3e2000 con 0x56443e61e900 2019-03-06 15:28:20.745 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:20.745 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:20.745 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:20.745 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:20.745 7f72cc823700 20 allow all 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:20.745 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:20.745 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:20.745 7f72cc823700 20 allow all 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56444019c840 from v2:192.168.178.28:40578/0 2019-03-06 15:28:20.745 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8af40 for mgr.? 2019-03-06 15:28:20.745 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56444019c840 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499400 2019-03-06 15:28:20.745 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1680) is_readable = 1 - now=2019-03-06 15:28:20.746713 lease_expire=2019-03-06 15:28:24.750049 has v0 lc 1680 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:20.745 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1507) v3 -- 0x56443ec09c00 con 0x56443e61e900 2019-03-06 15:28:20.745 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:20.745 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:20.745 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:20.745 7f72cc823700 20 match 2019-03-06 15:28:20.745 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:20.745 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:20.837 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 342 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441070780 con 0x5644402c0400 2019-03-06 15:28:20.837 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:20.837 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:20.837 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) dispatch 0x564441070780 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:20.837 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1680) is_readable = 1 - now=2019-03-06 15:28:20.840257 lease_expire=2019-03-06 15:28:24.750049 has v0 lc 1680 2019-03-06 15:28:20.837 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:20.837 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:20.837 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804237 }, { "osd": 1, "seq": 42949673418 }, { "osd": 2, "seq": 60129542597 }, { "osd": 3, "seq": 73014444481 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:20.841 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:20.841 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:20.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) setting proposal_timer 0x56443ff49770 with delay of 0.05 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1031) propose_pending 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader).mgrstat 1032 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:20.889 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1680) queue_pending_finisher 0x5644401e4650 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1680) trigger_propose active, proposing now 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1680) propose_pending 1681 6340 bytes 2019-03-06 15:28:20.889 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) begin for 1681 6340 bytes 2019-03-06 15:28:21.077 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) sending begin to mon.1 2019-03-06 15:28:21.077 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1680 fc 0 pn 300 opn 0) v4 -- ?+0 0x5644410f6680 2019-03-06 15:28:21.077 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1680 fc 0 pn 300 opn 0) v4 -- 0x5644410f6680 con 0x56443e61e900 2019-03-06 15:28:21.077 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) sending begin to mon.2 2019-03-06 15:28:21.077 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1680 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441f8f8c0 2019-03-06 15:28:21.077 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1680 fc 0 pn 300 opn 0) v4 -- 0x564441f8f8c0 con 0x56443e61ed80 2019-03-06 15:28:21.129 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4901 ==== paxos(accept lc 1680 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x5644410f6680 con 0x56443e61e900 2019-03-06 15:28:21.129 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:21.129 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:21.129 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow all 2019-03-06 15:28:21.129 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow all 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) handle_accept paxos(accept lc 1680 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) now 0,1 have accepted 2019-03-06 15:28:21.129 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5807 ==== paxos(accept lc 1680 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441f8f8c0 con 0x56443e61ed80 2019-03-06 15:28:21.129 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:21.129 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:21.129 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow all 2019-03-06 15:28:21.129 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.129 7f72cc823700 20 allow all 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) handle_accept paxos(accept lc 1680 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) now 0,1,2 have accepted 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) got majority, committing, done with update 2019-03-06 15:28:21.129 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1680) commit_start 1681 2019-03-06 15:28:21.177 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1680) commit_finish 1681 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1681) sending commit to mon.1 2019-03-06 15:28:21.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1681 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f37e40 2019-03-06 15:28:21.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1681 fc 0 pn 300 opn 0) v4 -- 0x564440f37e40 con 0x56443e61e900 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1681) sending commit to mon.2 2019-03-06 15:28:21.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1681 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444104d8c0 2019-03-06 15:28:21.177 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1681 fc 0 pn 300 opn 0) v4 -- 0x56444104d8c0 con 0x56443e61ed80 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) refresh 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).log v518 update_from_paxos version 518 summary v 518 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:21.177 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:21.185 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:21.185 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:21.197 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:21.197 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:21.201 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:21.201 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).mgrstat 1032 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1032 service_map e2 0progress events 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:21.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e78e00 con 0x5644402c0400 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..518) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1681) commit_proposal 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) _active - not active 2019-03-06 15:28:21.201 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1681) extend_lease now+5 (2019-03-06 15:28:26.203833) 2019-03-06 15:28:21.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1681 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e868ec0 2019-03-06 15:28:21.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1681 fc 1005 pn 0 opn 0) v4 -- 0x56443e868ec0 con 0x56443e61e900 2019-03-06 15:28:21.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1681 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440e98580 2019-03-06 15:28:21.201 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1681 fc 1005 pn 0 opn 0) v4 -- 0x564440e98580 con 0x56443e61ed80 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1681) finish_round 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1681) finish_round waiting_for_acting 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) _active 2019-03-06 15:28:21.201 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1032) _active creating new pending 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).mgrstat 1032 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1681) finish_round waiting_for_readable 2019-03-06 15:28:21.201 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1681) finish_round waiting_for_writeable 2019-03-06 15:28:21.201 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1681) finish_round done w/ waiters, state active 2019-03-06 15:28:21.241 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5808 ==== paxos(lease_ack lc 1681 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x564440e98580 con 0x56443e61ed80 2019-03-06 15:28:21.241 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:21.241 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:21.241 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow all 2019-03-06 15:28:21.241 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow all 2019-03-06 15:28:21.241 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1681) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:21.241 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4902 ==== paxos(lease_ack lc 1681 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e868ec0 con 0x56443e61e900 2019-03-06 15:28:21.241 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:21.241 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:21.241 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow all 2019-03-06 15:28:21.241 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:21.241 7f72cc823700 20 allow all 2019-03-06 15:28:21.241 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1681) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:22.109 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:22.109 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2237.97s 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:22.109 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1681) is_readable = 1 - now=2019-03-06 15:28:22.112191 lease_expire=2019-03-06 15:28:26.203833 has v0 lc 1681 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:22.109 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1681) is_readable = 1 - now=2019-03-06 15:28:22.112274 lease_expire=2019-03-06 15:28:26.203833 has v0 lc 1681 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).log v518 log 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..518) maybe_trim trim_to 18 would only trim 17 < paxos_service_trim_min 250 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:22.109 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) maybe_trim trim_to 1027 would only trim 3 < paxos_service_trim_min 250 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:22.109 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:22.109 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:22.113 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f6ac0 con 0x5644402c0400 2019-03-06 15:28:22.553 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 343 ==== log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 ==== 953+0+0 (crc 0 0 0) 0x5644411f6ac0 con 0x5644402c0400 2019-03-06 15:28:22.553 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:22.553 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..518) dispatch 0x5644411f6ac0 log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:22.553 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1681) is_readable = 1 - now=2019-03-06 15:28:22.554076 lease_expire=2019-03-06 15:28:26.203833 has v0 lc 1681 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 preprocess_query log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 preprocess_log log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 from mgr.6993 2019-03-06 15:28:22.553 7f72cc823700 20 is_capable service=log command= write addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:22.553 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:22.553 7f72cc823700 20 match 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 prepare_update log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 from mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 prepare_log log(5 entries from seq 150 at 2019-03-06 15:28:12.831361) v1 from mgr.6993 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 logging 2019-03-06 15:28:12.831361 mgr.z (mgr.6993) 150 : cluster [DBG] pgmap v147: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 logging 2019-03-06 15:28:14.833662 mgr.z (mgr.6993) 151 : cluster [DBG] pgmap v148: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 logging 2019-03-06 15:28:16.834716 mgr.z (mgr.6993) 152 : cluster [DBG] pgmap v149: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 logging 2019-03-06 15:28:18.836259 mgr.z (mgr.6993) 153 : cluster [DBG] pgmap v150: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).log v518 logging 2019-03-06 15:28:20.839244 mgr.z (mgr.6993) 154 : cluster [DBG] pgmap v151: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.553 7f72cc823700 10 mon.a@0(leader).paxosservice(logm 1..518) setting proposal_timer 0x564440b8e150 with delay of 0.05 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..518) propose_pending 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).log v518 encode_full log v 518 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).log v518 encode_pending v519 2019-03-06 15:28:22.601 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1681) queue_pending_finisher 0x5644401e4b60 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1681) trigger_propose active, proposing now 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1681) propose_pending 1682 47001 bytes 2019-03-06 15:28:22.601 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) begin for 1682 47001 bytes 2019-03-06 15:28:22.661 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) sending begin to mon.1 2019-03-06 15:28:22.661 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1681 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441e28940 2019-03-06 15:28:22.661 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1681 fc 0 pn 300 opn 0) v4 -- 0x564441e28940 con 0x56443e61e900 2019-03-06 15:28:22.661 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) sending begin to mon.2 2019-03-06 15:28:22.661 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1681 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443fc679c0 2019-03-06 15:28:22.661 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1681 fc 0 pn 300 opn 0) v4 -- 0x56443fc679c0 con 0x56443e61ed80 2019-03-06 15:28:22.713 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4903 ==== paxos(accept lc 1681 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441e28940 con 0x56443e61e900 2019-03-06 15:28:22.713 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:22.713 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:22.713 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow all 2019-03-06 15:28:22.713 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow all 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) handle_accept paxos(accept lc 1681 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) now 0,1 have accepted 2019-03-06 15:28:22.713 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5809 ==== paxos(accept lc 1681 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443fc679c0 con 0x56443e61ed80 2019-03-06 15:28:22.713 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:22.713 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:22.713 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow all 2019-03-06 15:28:22.713 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.713 7f72cc823700 20 allow all 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) handle_accept paxos(accept lc 1681 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) now 0,1,2 have accepted 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) got majority, committing, done with update 2019-03-06 15:28:22.713 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1681) commit_start 1682 2019-03-06 15:28:22.741 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 344 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x564440f37e40 con 0x5644402c0400 2019-03-06 15:28:22.741 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:22.741 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:22.741 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440f37e40 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:22.741 7f72cc823700 5 mon.a@0(leader).paxos(paxos writing c 1005..1681) is_readable = 1 - now=2019-03-06 15:28:22.745675 lease_expire=2019-03-06 15:28:26.203833 has v0 lc 1681 2019-03-06 15:28:22.741 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:22.741 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:22.741 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:22.741 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:22.741 7f72cc823700 20 match 2019-03-06 15:28:22.741 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:22.745 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1681) commit_finish 1682 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1682) sending commit to mon.1 2019-03-06 15:28:22.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1682 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f37b80 2019-03-06 15:28:22.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1682 fc 0 pn 300 opn 0) v4 -- 0x564440f37b80 con 0x56443e61e900 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1682) sending commit to mon.2 2019-03-06 15:28:22.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1682 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e99b80 2019-03-06 15:28:22.765 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1682 fc 0 pn 300 opn 0) v4 -- 0x564440e99b80 con 0x56443e61ed80 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) refresh 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos version 519 summary v 518 2019-03-06 15:28:22.765 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos latest full 518 2019-03-06 15:28:22.765 7f72c901c700 7 mon.a@0(leader).log v519 update_from_paxos applying incremental log 519 2019-03-06 15:28:12.831361 mgr.z (mgr.6993) 150 : cluster [DBG] pgmap v147: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).log v519 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:22.765 7f72c901c700 7 mon.a@0(leader).log v519 update_from_paxos applying incremental log 519 2019-03-06 15:28:14.833662 mgr.z (mgr.6993) 151 : cluster [DBG] pgmap v148: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).log v519 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:22.765 7f72c901c700 7 mon.a@0(leader).log v519 update_from_paxos applying incremental log 519 2019-03-06 15:28:16.834716 mgr.z (mgr.6993) 152 : cluster [DBG] pgmap v149: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).log v519 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:22.765 7f72c901c700 7 mon.a@0(leader).log v519 update_from_paxos applying incremental log 519 2019-03-06 15:28:18.836259 mgr.z (mgr.6993) 153 : cluster [DBG] pgmap v150: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).log v519 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:22.765 7f72c901c700 7 mon.a@0(leader).log v519 update_from_paxos applying incremental log 519 2019-03-06 15:28:20.839244 mgr.z (mgr.6993) 154 : cluster [DBG] pgmap v151: 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail 2019-03-06 15:28:22.765 7f72c901c700 20 mon.a@0(leader).log v519 update_from_paxos logging for channel 'cluster' to file '/home/sebastian/Repos/ceph/build/out/cluster.mon.a.log' 2019-03-06 15:28:22.769 7f72c901c700 15 mon.a@0(leader).log v519 update_from_paxos logging for 1 channels 2019-03-06 15:28:22.769 7f72c901c700 15 mon.a@0(leader).log v519 update_from_paxos channel 'cluster' logging 755 bytes 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).log v519 check_subs 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).log v519 check_sub client wants log-info ver 519 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).log v519 _create_sub_incremental level 1 ver 519 cur summary ver 519 2019-03-06 15:28:22.769 7f72c901c700 20 mon.a@0(leader).log v519 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:22.769 7f72c901c700 20 mon.a@0(leader).log v519 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:22.769 7f72c901c700 20 mon.a@0(leader).log v519 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:22.769 7f72c901c700 20 mon.a@0(leader).log v519 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:22.769 7f72c901c700 20 mon.a@0(leader).log v519 _create_sub_incremental requested 1 entry [DBG] 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).log v519 _create_sub_incremental incremental message ready (0 entries) 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).log v519 check_sub sending message to mgr.6993 with 0 entries (version 519) 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:22.769 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:22.773 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:22.773 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:22.789 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:22.789 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:22.789 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).mgrstat 1032 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1032 service_map e2 0progress events 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:22.789 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e79000 con 0x5644402c0400 2019-03-06 15:28:22.789 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:22.789 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) post_refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:22.789 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) post_refresh 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1682) commit_proposal 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) _active - not active 2019-03-06 15:28:22.793 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1682) extend_lease now+5 (2019-03-06 15:28:27.794234) 2019-03-06 15:28:22.793 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1682 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e9f5700 2019-03-06 15:28:22.793 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1682 fc 1005 pn 0 opn 0) v4 -- 0x56443e9f5700 con 0x56443e61e900 2019-03-06 15:28:22.793 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1682 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e8f98c0 2019-03-06 15:28:22.793 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1682 fc 1005 pn 0 opn 0) v4 -- 0x56443e8f98c0 con 0x56443e61ed80 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1682) finish_round 2019-03-06 15:28:22.793 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1682) finish_round waiting_for_acting 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) _active 2019-03-06 15:28:22.793 7f72c901c700 7 mon.a@0(leader).paxosservice(logm 1..519) _active creating new pending 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).log v519 create_pending v 520 2019-03-06 15:28:22.793 7f72c901c700 7 mon.a@0(leader).log v519 _updated_log for mgr.6993 192.168.178.28:0/20119 2019-03-06 15:28:22.793 7f72c901c700 2 mon.a@0(leader) e1 send_reply 0x56443f587a20 0x56443fd8cf00 log(last 154) v1 2019-03-06 15:28:22.793 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- log(last 154) v1 -- 0x56443fd8cf00 con 0x5644402c0400 2019-03-06 15:28:22.793 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1682) finish_round waiting_for_readable 2019-03-06 15:28:22.793 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1682) finish_round waiting_for_writeable 2019-03-06 15:28:22.793 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1682) finish_round done w/ waiters, state active 2019-03-06 15:28:22.837 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5810 ==== paxos(lease_ack lc 1682 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e8f98c0 con 0x56443e61ed80 2019-03-06 15:28:22.837 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:22.837 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:22.837 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow all 2019-03-06 15:28:22.837 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow all 2019-03-06 15:28:22.837 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1682) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:22.837 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4904 ==== paxos(lease_ack lc 1682 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x56443e9f5700 con 0x56443e61e900 2019-03-06 15:28:22.837 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:22.837 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:22.837 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow all 2019-03-06 15:28:22.837 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.837 7f72cc823700 20 allow all 2019-03-06 15:28:22.837 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1682) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:22.841 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4905 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1508 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440f37b80 con 0x56443e61e900 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:22.841 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.841 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.841 7f72cc823700 20 allow all 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:22.841 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:22.841 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:22.841 7f72cc823700 20 allow all 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443f4c38c0 from v2:192.168.178.28:40578/0 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8a880 for mgr.? 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443f4c38c0 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440498800 2019-03-06 15:28:22.841 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1682) is_readable = 1 - now=2019-03-06 15:28:22.842619 lease_expire=2019-03-06 15:28:27.794234 has v0 lc 1682 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:22.841 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1508) v3 -- 0x56443ec0b800 con 0x56443e61e900 2019-03-06 15:28:22.841 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:22.841 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:22.841 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:22.841 7f72cc823700 20 match 2019-03-06 15:28:22.841 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:22.841 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 345 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441073200 con 0x5644402c0400 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) dispatch 0x564441073200 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:22.841 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1682) is_readable = 1 - now=2019-03-06 15:28:22.842925 lease_expire=2019-03-06 15:28:27.794234 has v0 lc 1682 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804237 }, { "osd": 1, "seq": 42949673418 }, { "osd": 2, "seq": 60129542597 }, { "osd": 3, "seq": 73014444481 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:22.841 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:22.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) setting proposal_timer 0x56443fe76030 with delay of 0.923313 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) propose_pending 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader).mgrstat 1033 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:23.765 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1682) queue_pending_finisher 0x5644401e4a10 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1682) trigger_propose active, proposing now 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1682) propose_pending 1683 6340 bytes 2019-03-06 15:28:23.765 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) begin for 1683 6340 bytes 2019-03-06 15:28:24.021 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:24.021 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:24.021 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443f08cd00 con 0x5644403b1b00 2019-03-06 15:28:24.741 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 346 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443f40e680 con 0x5644402c0400 2019-03-06 15:28:24.797 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) sending begin to mon.1 2019-03-06 15:28:24.797 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1682 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443ea8f700 2019-03-06 15:28:24.797 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1682 fc 0 pn 300 opn 0) v4 -- 0x56443ea8f700 con 0x56443e61e900 2019-03-06 15:28:24.797 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) sending begin to mon.2 2019-03-06 15:28:24.797 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1682 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f46d340 2019-03-06 15:28:24.797 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1682 fc 0 pn 300 opn 0) v4 -- 0x56443f46d340 con 0x56443e61ed80 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443f40e680 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:24.797 7f72cc823700 5 mon.a@0(leader).paxos(paxos updating c 1005..1682) is_readable = 1 - now=2019-03-06 15:28:24.799491 lease_expire=2019-03-06 15:28:27.794234 has v0 lc 1682 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:24.797 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:24.797 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 20 match 2019-03-06 15:28:24.797 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:24.797 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4906 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1509 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440e9a100 con 0x56443e61e900 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:24.797 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:24.797 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.797 7f72cc823700 20 allow all 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:24.797 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:24.797 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.797 7f72cc823700 20 allow all 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e869c80 from v2:192.168.178.28:40578/0 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa8b600 for mgr.? 2019-03-06 15:28:24.797 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e869c80 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444049b000 2019-03-06 15:28:24.797 7f72cc823700 5 mon.a@0(leader).paxos(paxos updating c 1005..1682) is_readable = 1 - now=2019-03-06 15:28:24.800401 lease_expire=2019-03-06 15:28:27.794234 has v0 lc 1682 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:24.797 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1509) v3 -- 0x56443ec0a680 con 0x56443e61e900 2019-03-06 15:28:24.797 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:24.797 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:24.797 7f72cc823700 20 match 2019-03-06 15:28:24.797 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:24.797 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:24.841 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 347 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441073480 con 0x5644402c0400 2019-03-06 15:28:24.841 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:24.841 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:24.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) dispatch 0x564441073480 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:24.841 7f72cc823700 5 mon.a@0(leader).paxos(paxos updating c 1005..1682) is_readable = 1 - now=2019-03-06 15:28:24.845617 lease_expire=2019-03-06 15:28:27.794234 has v0 lc 1682 2019-03-06 15:28:24.841 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:24.841 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1032) waiting for paxos -> writeable 2019-03-06 15:28:24.877 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5811 ==== paxos(accept lc 1682 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443f46d340 con 0x56443e61ed80 2019-03-06 15:28:24.877 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:24.877 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:24.877 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow all 2019-03-06 15:28:24.877 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow all 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) handle_accept paxos(accept lc 1682 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) now 0,2 have accepted 2019-03-06 15:28:24.877 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4907 ==== paxos(accept lc 1682 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443ea8f700 con 0x56443e61e900 2019-03-06 15:28:24.877 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:24.877 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:24.877 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow all 2019-03-06 15:28:24.877 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:24.877 7f72cc823700 20 allow all 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) handle_accept paxos(accept lc 1682 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) now 0,1,2 have accepted 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) got majority, committing, done with update 2019-03-06 15:28:24.877 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1682) commit_start 1683 2019-03-06 15:28:24.937 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1682) commit_finish 1683 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1683) sending commit to mon.1 2019-03-06 15:28:24.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1683 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440ee9e40 2019-03-06 15:28:24.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1683 fc 0 pn 300 opn 0) v4 -- 0x564440ee9e40 con 0x56443e61e900 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1683) sending commit to mon.2 2019-03-06 15:28:24.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1683 fc 0 pn 300 opn 0) v4 -- ?+0 0x56444104d600 2019-03-06 15:28:24.937 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1683 fc 0 pn 300 opn 0) v4 -- 0x56444104d600 con 0x56443e61ed80 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) refresh 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos version 519 summary v 519 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:24.937 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:24.941 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:24.941 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:24.945 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:24.945 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:24.945 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat 1033 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1033 service_map e2 0progress events 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:24.945 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e79200 con 0x5644402c0400 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1683) commit_proposal 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) _active - not active 2019-03-06 15:28:24.945 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1683) extend_lease now+5 (2019-03-06 15:28:29.948742) 2019-03-06 15:28:24.945 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1683 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564441306940 2019-03-06 15:28:24.945 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1683 fc 1005 pn 0 opn 0) v4 -- 0x564441306940 con 0x56443e61e900 2019-03-06 15:28:24.945 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1683 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e94c940 2019-03-06 15:28:24.945 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1683 fc 1005 pn 0 opn 0) v4 -- 0x56443e94c940 con 0x56443e61ed80 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1683) finish_round 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1683) finish_round waiting_for_acting 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) _active 2019-03-06 15:28:24.945 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1033) _active creating new pending 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat 1033 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) dispatch 0x564441073480 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:24.945 7f72c901c700 5 mon.a@0(leader).paxos(paxos active c 1005..1683) is_readable = 1 - now=2019-03-06 15:28:24.949010 lease_expire=2019-03-06 15:28:29.948742 has v0 lc 1683 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804238 }, { "osd": 1, "seq": 42949673419 }, { "osd": 2, "seq": 60129542598 }, { "osd": 3, "seq": 73014444482 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) setting proposal_timer 0x56444217c780 with delay of 0.989527 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1683) finish_round waiting_for_readable 2019-03-06 15:28:24.945 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1683) finish_round waiting_for_writeable 2019-03-06 15:28:24.945 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1683) finish_round done w/ waiters, state active 2019-03-06 15:28:25.009 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4908 ==== paxos(lease_ack lc 1683 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564441306940 con 0x56443e61e900 2019-03-06 15:28:25.009 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:25.009 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:25.009 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow all 2019-03-06 15:28:25.009 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow all 2019-03-06 15:28:25.009 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1683) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:25.009 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5812 ==== paxos(lease_ack lc 1683 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e94c940 con 0x56443e61ed80 2019-03-06 15:28:25.009 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:25.009 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:25.009 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow all 2019-03-06 15:28:25.009 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:25.009 7f72cc823700 20 allow all 2019-03-06 15:28:25.009 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1683) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1033) propose_pending 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader).mgrstat 1034 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:25.937 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1683) queue_pending_finisher 0x5644401e47a0 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1683) trigger_propose active, proposing now 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1683) propose_pending 1684 6340 bytes 2019-03-06 15:28:25.937 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) begin for 1684 6340 bytes 2019-03-06 15:28:26.233 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) sending begin to mon.1 2019-03-06 15:28:26.233 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1683 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440ee9600 2019-03-06 15:28:26.233 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1683 fc 0 pn 300 opn 0) v4 -- 0x564440ee9600 con 0x56443e61e900 2019-03-06 15:28:26.233 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) sending begin to mon.2 2019-03-06 15:28:26.233 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1683 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f484c00 2019-03-06 15:28:26.233 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1683 fc 0 pn 300 opn 0) v4 -- 0x56443f484c00 con 0x56443e61ed80 2019-03-06 15:28:26.485 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5813 ==== paxos(accept lc 1683 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443f484c00 con 0x56443e61ed80 2019-03-06 15:28:26.485 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:26.485 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:26.485 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow all 2019-03-06 15:28:26.485 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow all 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) handle_accept paxos(accept lc 1683 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) now 0,2 have accepted 2019-03-06 15:28:26.485 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4909 ==== paxos(accept lc 1683 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440ee9600 con 0x56443e61e900 2019-03-06 15:28:26.485 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:26.485 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:26.485 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow all 2019-03-06 15:28:26.485 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.485 7f72cc823700 20 allow all 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) handle_accept paxos(accept lc 1683 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) now 0,1,2 have accepted 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) got majority, committing, done with update 2019-03-06 15:28:26.485 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1683) commit_start 1684 2019-03-06 15:28:26.665 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1683) commit_finish 1684 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1684) sending commit to mon.1 2019-03-06 15:28:26.665 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1684 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443f5b3b80 2019-03-06 15:28:26.665 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1684 fc 0 pn 300 opn 0) v4 -- 0x56443f5b3b80 con 0x56443e61e900 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1684) sending commit to mon.2 2019-03-06 15:28:26.665 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1684 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440ee8840 2019-03-06 15:28:26.665 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1684 fc 0 pn 300 opn 0) v4 -- 0x564440ee8840 con 0x56443e61ed80 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) refresh 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos version 519 summary v 519 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:26.665 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:26.673 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:26.673 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:26.685 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:26.685 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:26.689 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:26.689 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).mgrstat 1034 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1034 service_map e2 0progress events 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:26.689 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e79400 con 0x5644402c0400 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1684) commit_proposal 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) _active - not active 2019-03-06 15:28:26.689 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1684) extend_lease now+5 (2019-03-06 15:28:31.691643) 2019-03-06 15:28:26.689 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1684 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564440e2ec00 2019-03-06 15:28:26.689 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1684 fc 1005 pn 0 opn 0) v4 -- 0x564440e2ec00 con 0x56443e61e900 2019-03-06 15:28:26.689 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1684 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e9f3600 2019-03-06 15:28:26.689 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1684 fc 1005 pn 0 opn 0) v4 -- 0x56443e9f3600 con 0x56443e61ed80 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1684) finish_round 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1684) finish_round waiting_for_acting 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) _active 2019-03-06 15:28:26.689 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1034) _active creating new pending 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).mgrstat 1034 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1684) finish_round waiting_for_readable 2019-03-06 15:28:26.689 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1684) finish_round waiting_for_writeable 2019-03-06 15:28:26.689 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1684) finish_round done w/ waiters, state active 2019-03-06 15:28:26.741 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4910 ==== paxos(lease_ack lc 1684 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564440e2ec00 con 0x56443e61e900 2019-03-06 15:28:26.741 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:26.741 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:26.741 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow all 2019-03-06 15:28:26.741 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow all 2019-03-06 15:28:26.741 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1684) handle_lease_ack from mon.1 -- still need 1 more 2019-03-06 15:28:26.741 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5814 ==== paxos(lease_ack lc 1684 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e9f3600 con 0x56443e61ed80 2019-03-06 15:28:26.741 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:26.741 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:26.741 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow all 2019-03-06 15:28:26.741 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.741 7f72cc823700 20 allow all 2019-03-06 15:28:26.741 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1684) handle_lease_ack from mon.2 -- got everyone 2019-03-06 15:28:26.761 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 348 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443f5b3b80 con 0x5644402c0400 2019-03-06 15:28:26.761 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:26.761 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:26.761 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443f5b3b80 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:26.761 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) is_readable = 1 - now=2019-03-06 15:28:26.764222 lease_expire=2019-03-06 15:28:31.691643 has v0 lc 1684 2019-03-06 15:28:26.761 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:26.761 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:26.761 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:26.761 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:26.761 7f72cc823700 20 match 2019-03-06 15:28:26.761 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:26.761 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:26.765 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4911 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1510 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x564440ee9e40 con 0x56443e61e900 2019-03-06 15:28:26.765 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:26.765 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:26.765 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.765 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.765 7f72cc823700 20 allow all 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:26.765 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:26.765 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:26.765 7f72cc823700 20 allow all 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x56443e8f9b80 from v2:192.168.178.28:40578/0 2019-03-06 15:28:26.765 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa88000 for mgr.? 2019-03-06 15:28:26.765 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e8f9b80 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x564440499a00 2019-03-06 15:28:26.765 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) is_readable = 1 - now=2019-03-06 15:28:26.767301 lease_expire=2019-03-06 15:28:31.691643 has v0 lc 1684 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:26.765 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1510) v3 -- 0x56443ec0b2c0 con 0x56443e61e900 2019-03-06 15:28:26.765 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:26.765 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:26.765 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:26.765 7f72cc823700 20 match 2019-03-06 15:28:26.765 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:26.765 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:26.845 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 349 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441073980 con 0x5644402c0400 2019-03-06 15:28:26.845 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:26.845 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:26.845 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) dispatch 0x564441073980 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:26.845 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) is_readable = 1 - now=2019-03-06 15:28:26.847392 lease_expire=2019-03-06 15:28:31.691643 has v0 lc 1684 2019-03-06 15:28:26.845 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:26.845 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:26.845 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804238 }, { "osd": 1, "seq": 42949673419 }, { "osd": 2, "seq": 60129542598 }, { "osd": 3, "seq": 73014444482 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:26.845 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:26.845 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:26.845 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) setting proposal_timer 0x56444133c450 with delay of 0.817433 2019-03-06 15:28:27.109 7f72cf028700 11 mon.a@0(leader) e1 tick 2019-03-06 15:28:27.113 7f72cf028700 1 mon.a@0(leader).mds e3 no beacon from mds.-1.0 (gid: 4299 addr: [v2:192.168.178.28:6834/386489708,v1:192.168.178.28:6835/386489708] state: up:standby) since 2242.97s 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).osd e35 e35: 4 total, 4 up, 4 in 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).osd e35 should_prune currently holding only 34 epochs (min osdmap epochs: 500); do not prune. 2019-03-06 15:28:27.113 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) is_readable = 1 - now=2019-03-06 15:28:27.114351 lease_expire=2019-03-06 15:28:31.691643 has v0 lc 1684 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps max_prune 100 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 1 nothing purged 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 2 nothing purged 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 3 nothing purged 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).osd e35 try_prune_purged_snaps 4 nothing purged 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).osd e35 try_prune_purged_snaps actually pruned 0 2019-03-06 15:28:27.113 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) is_readable = 1 - now=2019-03-06 15:28:27.114449 lease_expire=2019-03-06 15:28:31.691643 has v0 lc 1684 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).osd e35 min_last_epoch_clean 34 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).log v519 log 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).paxosservice(logm 1..519) maybe_trim trim_to 19 would only trim 18 < paxos_service_trim_min 250 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).auth v10 auth 2019-03-06 15:28:27.113 7f72cf028700 10 cephx keyserver: _check_rotating_secrets 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) maybe_trim trim_to 1029 would only trim 5 < paxos_service_trim_min 250 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).health tick 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).health check_member_health 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).health check_member_health avail 89% total 1.8 TiB, used 102 GiB, avail 1.6 TiB 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader).health check_leader_health 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).config tick 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).paxosservice(config 1..10) maybe_trim trim_to 5 would only trim 4 < paxos_service_trim_min 250 2019-03-06 15:28:27.113 7f72cf028700 20 mon.a@0(leader) e1 sync_trim_providers 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests 2019-03-06 15:28:27.113 7f72cf028700 10 mon.a@0(leader).mgr e95 send_digests sending digest to subscriber 0x5644402c0400 192.168.178.28:0/20119 2019-03-06 15:28:27.113 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- mgrdigest v1 -- 0x5644411f5680 con 0x5644402c0400 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1034) propose_pending 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader).mgrstat 1035 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:27.665 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1684) queue_pending_finisher 0x5644401e4080 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1684) trigger_propose active, proposing now 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1684) propose_pending 1685 6340 bytes 2019-03-06 15:28:27.665 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) begin for 1685 6340 bytes 2019-03-06 15:28:28.173 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) sending begin to mon.1 2019-03-06 15:28:28.173 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1684 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441306c00 2019-03-06 15:28:28.173 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1684 fc 0 pn 300 opn 0) v4 -- 0x564441306c00 con 0x56443e61e900 2019-03-06 15:28:28.173 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) sending begin to mon.2 2019-03-06 15:28:28.173 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1684 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440f36dc0 2019-03-06 15:28:28.173 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1684 fc 0 pn 300 opn 0) v4 -- 0x564440f36dc0 con 0x56443e61ed80 2019-03-06 15:28:28.237 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4912 ==== paxos(accept lc 1684 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441306c00 con 0x56443e61e900 2019-03-06 15:28:28.237 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:28.237 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:28.237 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow all 2019-03-06 15:28:28.237 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow all 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) handle_accept paxos(accept lc 1684 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) now 0,1 have accepted 2019-03-06 15:28:28.237 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5815 ==== paxos(accept lc 1684 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564440f36dc0 con 0x56443e61ed80 2019-03-06 15:28:28.237 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:28.237 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:28.237 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow all 2019-03-06 15:28:28.237 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.237 7f72cc823700 20 allow all 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) handle_accept paxos(accept lc 1684 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) now 0,1,2 have accepted 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) got majority, committing, done with update 2019-03-06 15:28:28.237 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1684) commit_start 1685 2019-03-06 15:28:28.321 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1684) commit_finish 1685 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1685) sending commit to mon.1 2019-03-06 15:28:28.321 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1685 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443fec2580 2019-03-06 15:28:28.321 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1685 fc 0 pn 300 opn 0) v4 -- 0x56443fec2580 con 0x56443e61e900 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1685) sending commit to mon.2 2019-03-06 15:28:28.321 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1685 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8a0dc0 2019-03-06 15:28:28.321 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1685 fc 0 pn 300 opn 0) v4 -- 0x56443e8a0dc0 con 0x56443e61ed80 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) refresh 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos version 519 summary v 519 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:28.321 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:28.329 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:28.329 7f72c901c700 20 mon.a@0(leader).config load_config config map: { "global": { "mon_pg_warn_min_per_osd": { "name": "mon_pg_warn_min_per_osd", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_min_size": { "name": "osd_pool_default_min_size", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_pool_default_size": { "name": "osd_pool_default_size", "value": "3", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "by_type": { "mds": { "debug_mds": { "name": "debug_mds", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mds_debug_auth_pins": { "name": "mds_debug_auth_pins", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_frag": { "name": "mds_debug_frag", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_scatterstat": { "name": "mds_debug_scatterstat", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_debug_subtrees": { "name": "mds_debug_subtrees", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "mds_verify_scatter": { "name": "mds_verify_scatter", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" } }, "mgr": { "debug_mgr": { "name": "debug_mgr", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mon": { "name": "debug_mon", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mgr/dashboard/x/server_port": { "name": "mgr/dashboard/server_port", "value": "41576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/y/server_port": { "name": "mgr/dashboard/server_port", "value": "43576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/dashboard/z/server_port": { "name": "mgr/dashboard/server_port", "value": "45576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/orchestrator_cli/orchestrator": { "name": "mgr/orchestrator_cli/orchestrator", "value": "test_orchestrator", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/x/server_port": { "name": "mgr/restful/server_port", "value": "42576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/y/server_port": { "name": "mgr/restful/server_port", "value": "44576", "level": "advanced", "can_update_at_runtime": false, "mask": "" }, "mgr/restful/z/server_port": { "name": "mgr/restful/server_port", "value": "46576", "level": "advanced", "can_update_at_runtime": false, "mask": "" } }, "mon": { "mon_allow_pool_delete": { "name": "mon_allow_pool_delete", "value": "true", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_crit": { "name": "mon_data_avail_crit", "value": "1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_data_avail_warn": { "name": "mon_data_avail_warn", "value": "2", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "mon_osd_reporter_subtree_level": { "name": "mon_osd_reporter_subtree_level", "value": "osd", "level": "advanced", "can_update_at_runtime": true, "mask": "" } }, "osd": { "debug_bdev": { "name": "debug_bdev", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluefs": { "name": "debug_bluefs", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_bluestore": { "name": "debug_bluestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_filestore": { "name": "debug_filestore", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_journal": { "name": "debug_journal", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_mgrc": { "name": "debug_mgrc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_monc": { "name": "debug_monc", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_ms": { "name": "debug_ms", "value": "1/1", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objclass": { "name": "debug_objclass", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_objecter": { "name": "debug_objecter", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_osd": { "name": "debug_osd", "value": "25/25", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_reserver": { "name": "debug_reserver", "value": "10/10", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "debug_rocksdb": { "name": "debug_rocksdb", "value": "20/20", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_copyfrom_max_chunk": { "name": "osd_copyfrom_max_chunk", "value": "524288", "level": "advanced", "can_update_at_runtime": true, "mask": "" }, "osd_debug_misdirected_ops": { "name": "osd_debug_misdirected_ops", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_debug_op_order": { "name": "osd_debug_op_order", "value": "true", "level": "dev", "can_update_at_runtime": true, "mask": "" }, "osd_scrub_load_threshold": { "name": "osd_scrub_load_threshold", "value": "2000.000000", "level": "advanced", "can_update_at_runtime": true, "mask": "" } } }, "by_id": {} } 2019-03-06 15:28:28.341 7f72c901c700 10 mon.a@0(leader).mgr e95 prime_mgr_client 2019-03-06 15:28:28.341 7f72c901c700 20 mgrc handle_mgr_map mgrmap(e 95) v1 2019-03-06 15:28:28.341 7f72c901c700 4 mgrc handle_mgr_map Got map version 95 2019-03-06 15:28:28.341 7f72c901c700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).mgrstat 1035 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).mgrstat update_from_paxos v1035 service_map e2 0progress events 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).mgrstat check_subs 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).mgrstat check_sub next 2 have 1 2019-03-06 15:28:28.345 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> 192.168.178.28:0/20119 -- service_map(e2 1 svc) v1 -- 0x564442e79600 con 0x5644402c0400 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).health update_from_paxos 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).health dump:{ "quorum_health": {}, "leader_health": {} } 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(health 1..1) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(config 1..10) post_refresh 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1685) commit_proposal 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) _active - not active 2019-03-06 15:28:28.345 7f72c901c700 7 mon.a@0(leader).paxos(paxos refresh c 1005..1685) extend_lease now+5 (2019-03-06 15:28:33.348139) 2019-03-06 15:28:28.345 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1685 fc 1005 pn 0 opn 0) v4 -- ?+0 0x564441203340 2019-03-06 15:28:28.345 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(lease lc 1685 fc 1005 pn 0 opn 0) v4 -- 0x564441203340 con 0x56443e61e900 2019-03-06 15:28:28.345 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1685 fc 1005 pn 0 opn 0) v4 -- ?+0 0x56443e9f3e40 2019-03-06 15:28:28.345 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(lease lc 1685 fc 1005 pn 0 opn 0) v4 -- 0x56443e9f3e40 con 0x56443e61ed80 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxos(paxos refresh c 1005..1685) finish_round 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1685) finish_round waiting_for_acting 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) _active 2019-03-06 15:28:28.345 7f72c901c700 7 mon.a@0(leader).paxosservice(mgrstat 1024..1035) _active creating new pending 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).mgrstat 1035 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).mgrstat update_logger 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1685) finish_round waiting_for_readable 2019-03-06 15:28:28.345 7f72c901c700 20 mon.a@0(leader).paxos(paxos active c 1005..1685) finish_round waiting_for_writeable 2019-03-06 15:28:28.345 7f72c901c700 10 mon.a@0(leader).paxos(paxos active c 1005..1685) finish_round done w/ waiters, state active 2019-03-06 15:28:28.409 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5816 ==== paxos(lease_ack lc 1685 fc 1005 pn 0 opn 0) v4 ==== 142+0+0 (crc 0 0 0) 0x56443e9f3e40 con 0x56443e61ed80 2019-03-06 15:28:28.409 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:28.409 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:28.409 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow all 2019-03-06 15:28:28.409 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow all 2019-03-06 15:28:28.409 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1685) handle_lease_ack from mon.2 -- still need 1 more 2019-03-06 15:28:28.409 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4913 ==== paxos(lease_ack lc 1685 fc 1005 pn 0 opn 0) v4 ==== 190+0+0 (crc 0 0 0) 0x564441203340 con 0x56443e61e900 2019-03-06 15:28:28.409 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:28.409 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:28.409 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow all 2019-03-06 15:28:28.409 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.409 7f72cc823700 20 allow all 2019-03-06 15:28:28.409 7f72cc823700 10 mon.a@0(leader).paxos(paxos active c 1005..1685) handle_lease_ack from mon.1 -- got everyone 2019-03-06 15:28:28.773 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4914 ==== forward(mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 caps allow profile mgr tid 1511 con_features 4611087854031667199) v4 ==== 15000+0+0 (crc 0 0 0) 0x56443fec2580 con 0x56443e61e900 2019-03-06 15:28:28.773 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:28.773 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:28.773 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.773 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.773 7f72cc823700 20 allow all 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader) e1 received forwarded message from mgr 192.168.178.28:0/20175 via mon.1 v2:192.168.178.28:40578/0 2019-03-06 15:28:28.773 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:28.773 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:28.773 7f72cc823700 20 allow all 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader) e1 caps are allow profile mgr 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader) e1 entity name 'mgr.x' type 16 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader) e1 mesg 0x564440e2d600 from v2:192.168.178.28:40578/0 2019-03-06 15:28:28.773 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443fa88480 for mgr.? 2019-03-06 15:28:28.773 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x564440e2d600 mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 from mgr.? 192.168.178.28:0/20175 con 0x56444023c000 2019-03-06 15:28:28.773 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1685) is_readable = 1 - now=2019-03-06 15:28:28.774671 lease_expire=2019-03-06 15:28:33.348139 has v0 lc 1685 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.? 192.168.178.28:0/20175 via v2:192.168.178.28:40578/0 for request mgrbeacon mgr.x(6d4a0a74-ba53-4305-a1f1-c35109bdea32,7006, , 0) v8 2019-03-06 15:28:28.773 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- route(no-reply tid 1511) v3 -- 0x56443ec0b640 con 0x56443e61e900 2019-03-06 15:28:28.773 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:28.773 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20175 on cap allow profile mgr 2019-03-06 15:28:28.773 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:28.773 7f72cc823700 20 match 2019-03-06 15:28:28.773 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 7006 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader).mgr e95 from existing standby 7006 2019-03-06 15:28:28.773 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:28.789 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 350 ==== mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 ==== 14864+0+0 (crc 0 0 0) 0x56443e8fb700 con 0x5644402c0400 2019-03-06 15:28:28.789 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:28.789 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:28.789 7f72cc823700 10 mon.a@0(leader).paxosservice(mgr 1..95) dispatch 0x56443e8fb700 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:28.789 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1685) is_readable = 1 - now=2019-03-06 15:28:28.790662 lease_expire=2019-03-06 15:28:33.348139 has v0 lc 1685 2019-03-06 15:28:28.789 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 mgrbeacon mgr.z(6d4a0a74-ba53-4305-a1f1-c35109bdea32,6993, [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119], 1) v8 2019-03-06 15:28:28.789 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:28.789 7f72cc823700 20 is_capable service=mgr command= exec addr 192.168.178.28:0/20119 on cap allow profile mgr 2019-03-06 15:28:28.789 7f72cc823700 20 allow so far , doing grant allow profile mgr 2019-03-06 15:28:28.789 7f72cc823700 20 match 2019-03-06 15:28:28.789 7f72cc823700 4 mon.a@0(leader).mgr e95 beacon from 6993 2019-03-06 15:28:28.789 7f72cc823700 10 mon.a@0(leader).mgr e95 no change 2019-03-06 15:28:28.845 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mgr.6993 192.168.178.28:0/20119 351 ==== monmgrreport(0 checks, 0 progress events) v2 ==== 36+0+5253 (crc 0 0 0) 0x564441070500 con 0x5644402c0400 2019-03-06 15:28:28.845 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443f8441c0 for mgr.6993 2019-03-06 15:28:28.845 7f72cc823700 20 mon.a@0(leader) e1 caps allow profile mgr 2019-03-06 15:28:28.845 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) dispatch 0x564441070500 monmgrreport(0 checks, 0 progress events) v2 from mgr.6993 192.168.178.28:0/20119 con 0x5644402c0400 2019-03-06 15:28:28.845 7f72cc823700 5 mon.a@0(leader).paxos(paxos active c 1005..1685) is_readable = 1 - now=2019-03-06 15:28:28.849208 lease_expire=2019-03-06 15:28:33.348139 has v0 lc 1685 2019-03-06 15:28:28.845 7f72cc823700 10 mon.a@0(leader) e1 no_reply to mgr.6993 192.168.178.28:0/20119 monmgrreport(0 checks, 0 progress events) v2 2019-03-06 15:28:28.845 7f72cc823700 10 mon.a@0(leader).mgrstat prepare_report 32 pgs: 32 active+clean; 3.8 KiB data, 278 MiB used, 36 GiB / 40 GiB avail, 0 health checks, 0progress events 2019-03-06 15:28:28.845 7f72cc823700 20 mon.a@0(leader).mgrstat pending_digest: { "num_pg": 32, "num_pg_active": 32, "num_pg_unknown": 0, "num_osd": 4, "pool_sum": { "stat_sum": { "num_bytes": 3884, "num_objects": 201, "num_object_clones": 0, "num_object_copies": 603, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 201, "num_whiteouts": 0, "num_read": 1861, "num_read_kb": 1686, "num_write": 1234, "num_write_kb": 19, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1395, "ondisk_log_size": 1395, "up": 96, "acting": 96, "num_store_stats": 96 }, "osd_sum": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] }, "osd_sum_by_class": { "hdd": { "up_from": 0, "seq": 0, "num_pgs": 96, "kb": 42205152, "kb_used": 4478752, "kb_used_data": 21888, "kb_used_omap": 0, "kb_used_meta": 4194304, "kb_avail": 37726400, "statfs": { "total": 43218075648, "available": 38631833600, "internally_reserved": 4294967296, "allocated": 22413312, "data_stored": 426328, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 4294967296 }, "hb_peers": [], "snap_trim_queue_len": 0, "num_snap_trimming": 0, "op_queue_age_hist": { "histogram": [], "upper_bound": 1 }, "perf_stat": { "commit_latency_ms": 0, "apply_latency_ms": 0, "commit_latency_ns": 0, "apply_latency_ns": 0 }, "alerts": [] } }, "pool_stats": [ { "poolid": 2, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 8, "num_object_clones": 0, "num_object_copies": 24, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 8, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 136, "ondisk_log_size": 136, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 1, "num_pg": 8, "stat_sum": { "num_bytes": 1246, "num_objects": 4, "num_object_clones": 0, "num_object_copies": 12, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 4, "num_whiteouts": 0, "num_read": 49, "num_read_kb": 49, "num_write": 4, "num_write_kb": 4, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 786432, "data_stored": 3738, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 4, "ondisk_log_size": 4, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 3, "num_pg": 8, "stat_sum": { "num_bytes": 2638, "num_objects": 14, "num_object_clones": 0, "num_object_copies": 42, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 14, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 22, "num_write_kb": 15, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 2752512, "data_stored": 7914, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 15, "ondisk_log_size": 15, "up": 24, "acting": 24, "num_store_stats": 24 }, { "poolid": 4, "num_pg": 8, "stat_sum": { "num_bytes": 0, "num_objects": 175, "num_object_clones": 0, "num_object_copies": 525, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 175, "num_whiteouts": 0, "num_read": 1812, "num_read_kb": 1637, "num_write": 1208, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0 }, "store_stats": { "total": 0, "available": 0, "internally_reserved": 0, "allocated": 0, "data_stored": 0, "data_compressed": 0, "data_compressed_allocated": 0, "data_compressed_original": 0, "omap_allocated": 0, "internal_metadata": 0 }, "log_size": 1240, "ondisk_log_size": 1240, "up": 24, "acting": 24, "num_store_stats": 24 } ], "osd_stats": [ { "osd": 0, "seq": 25769804238 }, { "osd": 1, "seq": 42949673419 }, { "osd": 2, "seq": 60129542598 }, { "osd": 3, "seq": 73014444482 } ], "num_pg_by_state": [ { "state": "active+clean", "num": 32 } ], "num_pg_by_osd": [ { "osd": 0, "num_primary_pg": 6, "num_acting_pg": 27, "num_up_pg": 27 }, { "osd": 1, "num_primary_pg": 10, "num_acting_pg": 25, "num_up_pg": 25 }, { "osd": 2, "num_primary_pg": 4, "num_acting_pg": 21, "num_up_pg": 21 }, { "osd": 3, "num_primary_pg": 12, "num_acting_pg": 23, "num_up_pg": 23 } ], "purged_snaps": [ { "pool": 1, "purged_snaps": {} }, { "pool": 2, "purged_snaps": {} }, { "pool": 3, "purged_snaps": {} }, { "pool": 4, "purged_snaps": {} } ] } 2019-03-06 15:28:28.849 7f72cc823700 20 mon.a@0(leader).mgrstat health checks: {} 2019-03-06 15:28:28.849 7f72cc823700 20 mon.a@0(leader).mgrstat progress events: {} 2019-03-06 15:28:28.849 7f72cc823700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) setting proposal_timer 0x5644401f2150 with delay of 0.471559 2019-03-06 15:28:29.025 7f72cb821700 20 mgrc operator() sending 53 counters (of possible 198), 0 new, 0 removed 2019-03-06 15:28:29.025 7f72cb821700 20 mgrc _send_report encoded 798 bytes 2019-03-06 15:28:29.025 7f72cb821700 1 -- 192.168.178.28:0/29961 --> [v2:192.168.178.28:6834/20119,v1:192.168.178.28:6835/20119] -- mgrreport(unknown.a +0-0 packed 798 daemon_metrics=1) v7 -- 0x56443f08c680 con 0x5644403b1b00 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader).paxosservice(mgrstat 1024..1035) propose_pending 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader).mgrstat 1036 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader) e1 log_health updated 0 previous 0 2019-03-06 15:28:29.321 7f72cf028700 5 mon.a@0(leader).paxos(paxos active c 1005..1685) queue_pending_finisher 0x5644401e4250 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1685) trigger_propose active, proposing now 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader).paxos(paxos active c 1005..1685) propose_pending 1686 6340 bytes 2019-03-06 15:28:29.321 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) begin for 1686 6340 bytes 2019-03-06 15:28:29.377 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) sending begin to mon.1 2019-03-06 15:28:29.377 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1685 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8fbc80 2019-03-06 15:28:29.377 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(begin lc 1685 fc 0 pn 300 opn 0) v4 -- 0x56443e8fbc80 con 0x56443e61e900 2019-03-06 15:28:29.377 7f72cf028700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) sending begin to mon.2 2019-03-06 15:28:29.377 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1685 fc 0 pn 300 opn 0) v4 -- ?+0 0x564441305b80 2019-03-06 15:28:29.377 7f72cf028700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(begin lc 1685 fc 0 pn 300 opn 0) v4 -- 0x564441305b80 con 0x56443e61ed80 2019-03-06 15:28:29.445 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.2 v2:192.168.178.28:40580/0 5817 ==== paxos(accept lc 1685 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x564441305b80 con 0x56443e61ed80 2019-03-06 15:28:29.445 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443e698480 for mon.2 2019-03-06 15:28:29.445 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:29.445 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow all 2019-03-06 15:28:29.445 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40580/0 on cap allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow all 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) handle_accept paxos(accept lc 1685 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) now 0,2 have accepted 2019-03-06 15:28:29.445 7f72cc823700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] <== mon.1 v2:192.168.178.28:40578/0 4915 ==== paxos(accept lc 1685 fc 0 pn 300 opn 0) v4 ==== 84+0+0 (crc 0 0 0) 0x56443e8fbc80 con 0x56443e61e900 2019-03-06 15:28:29.445 7f72cc823700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x56443d585840 for mon.1 2019-03-06 15:28:29.445 7f72cc823700 20 mon.a@0(leader) e1 caps allow * 2019-03-06 15:28:29.445 7f72cc823700 20 is_capable service=mon command= read addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow all 2019-03-06 15:28:29.445 7f72cc823700 20 is_capable service=mon command= exec addr v2:192.168.178.28:40578/0 on cap allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow so far , doing grant allow * 2019-03-06 15:28:29.445 7f72cc823700 20 allow all 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) handle_accept paxos(accept lc 1685 fc 0 pn 300 opn 0) v4 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) now 0,1,2 have accepted 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) got majority, committing, done with update 2019-03-06 15:28:29.445 7f72cc823700 10 mon.a@0(leader).paxos(paxos updating c 1005..1685) commit_start 1686 2019-03-06 15:28:29.509 7f72c901c700 20 mon.a@0(leader).paxos(paxos writing c 1005..1685) commit_finish 1686 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1686) sending commit to mon.1 2019-03-06 15:28:29.509 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1686 fc 0 pn 300 opn 0) v4 -- ?+0 0x564440e99600 2019-03-06 15:28:29.509 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40578/0,v1:192.168.178.28:40579/0] -- paxos(commit lc 1686 fc 0 pn 300 opn 0) v4 -- 0x564440e99600 con 0x56443e61e900 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxos(paxos writing c 1005..1686) sending commit to mon.2 2019-03-06 15:28:29.509 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] send_to--> mon [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1686 fc 0 pn 300 opn 0) v4 -- ?+0 0x56443e8683c0 2019-03-06 15:28:29.509 7f72c901c700 1 -- [v2:192.168.178.28:40576/0,v1:192.168.178.28:40577/0] --> [v2:192.168.178.28:40580/0,v1:192.168.178.28:40581/0] -- paxos(commit lc 1686 fc 0 pn 300 opn 0) v4 -- 0x56443e8683c0 con 0x56443e61ed80 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader) e1 refresh_from_paxos 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(mdsmap 1..3) refresh 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(osdmap 1..35) refresh 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(logm 1..519) refresh 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).log v519 update_from_paxos version 519 summary v 519 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(monmap 1..1) refresh 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(auth 1..10) refresh 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).auth v10 update_from_paxos 2019-03-06 15:28:29.509 7f72c901c700 10 mon.a@0(leader).paxosservice(mgr 1..95) refresh 2019-03-06 15:28:29.517 7f72c901c700 10 mon.a@0(leader).config load_config got 44 keys 2019-03-06 15:28:29.517 7f72c901c700 20 mon.a@0(leader).config load_config config map: