-251> 2022-12-06 18:52:56.238 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f98902ad0d0 -250> 2022-12-06 18:52:56.702 7f9861f7b640 0 log_channel(cluster) log [DBG] : pgmap v12: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 322 op/s; 13720/2501637 objects degraded (0.548%); 9.4 MiB/s, 7 objects/s recovering -249> 2022-12-06 18:52:56.702 7f9861f7b640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -248> 2022-12-06 18:52:56.827 7f9886ffd640 4 mgr ms_dispatch active log(1 entries from seq 9003 at 2022-12-06 18:52:56.685414) v1 -247> 2022-12-06 18:52:56.827 7f9886ffd640 4 mgr ms_dispatch log(1 entries from seq 9003 at 2022-12-06 18:52:56.685414) v1 -246> 2022-12-06 18:52:56.837 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -245> 2022-12-06 18:52:56.837 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -244> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/server_addr -243> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/server_addr -242> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/ssl -241> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/ssl -240> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]ssl not found -239> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/ssl_server_port -238> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/ssl_server_port -237> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]ssl_server_port not found -236> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/cartman09/crt -235> 2022-12-06 18:52:56.946 7f984bfff640 4 ceph_store_get cartman09/crt not found -234> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/crt -233> 2022-12-06 18:52:56.946 7f984bfff640 4 ceph_store_get crt not found -232> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/crt_file -231> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/crt_file -230> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]crt_file not found -229> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/cartman09/key -228> 2022-12-06 18:52:56.946 7f984bfff640 4 ceph_store_get cartman09/key not found -227> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/key -226> 2022-12-06 18:52:56.946 7f984bfff640 4 ceph_store_get key not found -225> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/key_file -224> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/key_file -223> 2022-12-06 18:52:56.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]key_file not found -222> 2022-12-06 18:52:56.991 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f9890213ae0 -221> 2022-12-06 18:52:57.015 7f989e59c640 10 monclient: handle_auth_request added challenge on 0x7f98902039b0 -220> 2022-12-06 18:52:57.610 7f987e7fc640 4 mgr send_beacon active -219> 2022-12-06 18:52:57.623 7f987e7fc640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -218> 2022-12-06 18:52:57.721 7f9862f7d640 4 mgr.server handle_report from 0x7f9890010150 osd.11 -217> 2022-12-06 18:52:57.723 7f9862f7d640 4 mgr.server handle_report from 0x7f989004a1d0 mon.cartman09 -216> 2022-12-06 18:52:57.726 7f9862f7d640 4 mgr.server handle_report from 0x7f98900386b0 osd.9 -215> 2022-12-06 18:52:57.731 7f9862f7d640 4 mgr.server handle_report from 0x7f98900650f0 osd.19 -214> 2022-12-06 18:52:57.736 7f9862f7d640 4 mgr.server handle_report from 0x7f989001ac00 osd.5 -213> 2022-12-06 18:52:57.738 7f9862f7d640 4 mgr.server handle_report from 0x7f989008cb70 osd.20 -212> 2022-12-06 18:52:57.739 7f9862f7d640 4 mgr.server handle_report from 0x7f98900a0490 osd.2 -211> 2022-12-06 18:52:57.739 7f9862f7d640 4 mgr.server handle_report from 0x7f98900c76d0 osd.3 -210> 2022-12-06 18:52:57.740 7f9862f7d640 4 mgr.server handle_report from 0x7f98900cf410 osd.18 -209> 2022-12-06 18:52:57.740 7f9862f7d640 4 mgr.server handle_report from 0x7f98900fe390 osd.14 -208> 2022-12-06 18:52:57.740 7f9862f7d640 4 mgr.server handle_report from 0x7f98900f27b0 osd.1 -207> 2022-12-06 18:52:57.760 7f9862f7d640 4 mgr.server handle_report from 0x7f9890202f30 mon.cartman06 -206> 2022-12-06 18:52:57.760 7f9862f7d640 4 mgr.server handle_report from 0x7f989002cee0 osd.16 -205> 2022-12-06 18:52:57.762 7f9862f7d640 4 mgr.server handle_report from 0x7f989020abd0 osd.4 -204> 2022-12-06 18:52:57.763 7f9862f7d640 4 mgr.server handle_report from 0x7f98902073a0 osd.12 -203> 2022-12-06 18:52:57.763 7f9862f7d640 4 mgr.server handle_report from 0x7f98900d7150 rgw.radosgw.cartman02 -202> 2022-12-06 18:52:57.764 7f9862f7d640 4 mgr.server handle_report from 0x7f98900cb570 osd.10 -201> 2022-12-06 18:52:57.764 7f9862f7d640 4 mgr.server handle_report from 0x7f9890084ec0 osd.17 -200> 2022-12-06 18:52:57.764 7f9862f7d640 4 mgr.server handle_report from 0x7f98900ee910 osd.0 -199> 2022-12-06 18:52:57.764 7f9862f7d640 4 mgr.server handle_report from 0x7f989005c690 osd.8 -198> 2022-12-06 18:52:57.764 7f9862f7d640 4 mgr.server handle_report from 0x7f989009c5f0 osd.6 -197> 2022-12-06 18:52:57.765 7f9862f7d640 4 mgr.server handle_report from 0x7f9890204430 mgr.cartman09 -196> 2022-12-06 18:52:57.765 7f9862f7d640 4 mgr.server handle_report from 0x7f9890023840 osd.15 -195> 2022-12-06 18:52:57.766 7f9862f7d640 4 mgr.server handle_report from 0x7f9890065b70 osd.13 -194> 2022-12-06 18:52:57.768 7f9862f7d640 4 mgr.server handle_report from 0x7f9890207e20 mon.patty -193> 2022-12-06 18:52:57.769 7f9862f7d640 4 mgr.server handle_report from 0x7f98901fcfb0 osd.7 -192> 2022-12-06 18:52:57.807 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f9890057680 -191> 2022-12-06 18:52:58.705 7f9861f7b640 0 log_channel(cluster) log [DBG] : pgmap v13: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 446 op/s; 13701/2501637 objects degraded (0.548%); 11 MiB/s, 9 objects/s recovering -190> 2022-12-06 18:52:58.705 7f9861f7b640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -189> 2022-12-06 18:52:58.845 7f9886ffd640 4 mgr ms_dispatch active osd_map(2237134..2237134 src has 2236384..2237134) v4 -188> 2022-12-06 18:52:58.845 7f9886ffd640 4 mgr ms_dispatch osd_map(2237134..2237134 src has 2236384..2237134) v4 -187> 2022-12-06 18:52:58.845 7f9886ffd640 10 monclient: _renew_subs -186> 2022-12-06 18:52:58.845 7f9886ffd640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -185> 2022-12-06 18:52:58.846 7f986377e640 4 mgr[progress] Processing OSDMap change 2237133..2237134 -184> 2022-12-06 18:52:58.856 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -183> 2022-12-06 18:52:58.856 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -182> 2022-12-06 18:52:59.389 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f9890204f10 -181> 2022-12-06 18:52:59.603 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f989014b760 -180> 2022-12-06 18:52:59.607 7f989e59c640 10 monclient: handle_auth_request added challenge on 0x7f9890239660 -179> 2022-12-06 18:52:59.623 7f987e7fc640 4 mgr send_beacon active -178> 2022-12-06 18:52:59.637 7f987e7fc640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -177> 2022-12-06 18:52:59.866 7f9886ffd640 4 mgr ms_dispatch active osd_map(2237135..2237135 src has 2236384..2237135) v4 -176> 2022-12-06 18:52:59.866 7f9886ffd640 4 mgr ms_dispatch osd_map(2237135..2237135 src has 2236384..2237135) v4 -175> 2022-12-06 18:52:59.866 7f9886ffd640 10 monclient: _renew_subs -174> 2022-12-06 18:52:59.866 7f9886ffd640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -173> 2022-12-06 18:52:59.867 7f986377e640 4 mgr[progress] Processing OSDMap change 2237134..2237135 -172> 2022-12-06 18:52:59.947 7f983b7fe640 10 monclient: tick -171> 2022-12-06 18:52:59.947 7f983b7fe640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-12-06 18:52:29.948606) -170> 2022-12-06 18:52:59.963 7f9824ff9640 10 monclient: tick -169> 2022-12-06 18:52:59.963 7f9824ff9640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-12-06 18:52:29.964489) -168> 2022-12-06 18:53:00.706 7f9861f7b640 0 log_channel(cluster) log [DBG] : pgmap v16: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 411 op/s; 13701/2501637 objects degraded (0.548%); 8.5 MiB/s, 7 objects/s recovering -167> 2022-12-06 18:53:00.706 7f9861f7b640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -166> 2022-12-06 18:53:00.761 7f9862f7d640 4 mgr.server _handle_command decoded 4 -165> 2022-12-06 18:53:00.761 7f9862f7d640 4 mgr.server _handle_command prefix=osd df -164> 2022-12-06 18:53:00.761 7f9862f7d640 0 log_channel(audit) log [DBG] : from='client.2745765143 v1:172.20.4.6:0/401948060' entity='client.admin' cmd=[{"prefix": "osd df", "output_method": "tree", "target": ["mgr", ""], "format": "json"}]: dispatch -163> 2022-12-06 18:53:00.762 7f9862f7d640 4 mgr.server reply reply success -162> 2022-12-06 18:53:00.881 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -161> 2022-12-06 18:53:00.881 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -160> 2022-12-06 18:53:00.970 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f9890204f10 -159> 2022-12-06 18:53:01.637 7f987e7fc640 4 mgr send_beacon active -158> 2022-12-06 18:53:01.650 7f987e7fc640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -157> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/server_addr -156> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/server_addr -155> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/ssl -154> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/ssl -153> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]ssl not found -152> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/ssl_server_port -151> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/ssl_server_port -150> 2022-12-06 18:53:01.946 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]ssl_server_port not found -149> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/cartman09/crt -148> 2022-12-06 18:53:01.947 7f984bfff640 4 ceph_store_get cartman09/crt not found -147> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/crt -146> 2022-12-06 18:53:01.947 7f984bfff640 4 ceph_store_get crt not found -145> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/crt_file -144> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/crt_file -143> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]crt_file not found -142> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/cartman09/key -141> 2022-12-06 18:53:01.947 7f984bfff640 4 ceph_store_get cartman09/key not found -140> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_store get_store key: mgr/dashboard/key -139> 2022-12-06 18:53:01.947 7f984bfff640 4 ceph_store_get key not found -138> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/cartman09/key_file -137> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_config get_config key: mgr/dashboard/key_file -136> 2022-12-06 18:53:01.947 7f984bfff640 4 mgr get_typed_config get_typed_config [cartman09/]key_file not found -135> 2022-12-06 18:53:01.966 7f9886ffd640 4 mgr ms_dispatch active log(1 entries from seq 9006 at 2022-12-06 18:53:01.686118) v1 -134> 2022-12-06 18:53:01.966 7f9886ffd640 4 mgr ms_dispatch log(1 entries from seq 9006 at 2022-12-06 18:53:01.686118) v1 -133> 2022-12-06 18:53:02.286 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f989014b760 -132> 2022-12-06 18:53:02.423 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f9890213ae0 -131> 2022-12-06 18:53:02.425 7f9862f7d640 4 mgr.server handle_open from mgr.patty 0x7f9890213ae0 -130> 2022-12-06 18:53:02.425 7f9862f7d640 2 mgr.server handle_open ignoring open from mgr.patty 172.20.4.10:0/390305; not ready for session (expect reconnect) -129> 2022-12-06 18:53:02.566 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f989023bd60 -128> 2022-12-06 18:53:02.707 7f9861f7b640 0 log_channel(cluster) log [DBG] : pgmap v17: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 977 KiB/s rd, 882 KiB/s wr, 186 op/s; 13701/2501637 objects degraded (0.548%); 2.5 MiB/s, 2 objects/s recovering -127> 2022-12-06 18:53:02.707 7f9861f7b640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -126> 2022-12-06 18:53:02.722 7f9862f7d640 4 mgr.server handle_report from 0x7f9890010150 osd.11 -125> 2022-12-06 18:53:02.723 7f9862f7d640 4 mgr.server handle_report from 0x7f989004a1d0 mon.cartman09 -124> 2022-12-06 18:53:02.727 7f9862f7d640 4 mgr.server handle_report from 0x7f98900386b0 osd.9 -123> 2022-12-06 18:53:02.732 7f9862f7d640 4 mgr.server handle_report from 0x7f98900650f0 osd.19 -122> 2022-12-06 18:53:02.738 7f9862f7d640 4 mgr.server handle_report from 0x7f989001ac00 osd.5 -121> 2022-12-06 18:53:02.739 7f9862f7d640 4 mgr.server handle_report from 0x7f989008cb70 osd.20 -120> 2022-12-06 18:53:02.740 7f9862f7d640 4 mgr.server handle_report from 0x7f98900a0490 osd.2 -119> 2022-12-06 18:53:02.740 7f9862f7d640 4 mgr.server handle_report from 0x7f98900c76d0 osd.3 -118> 2022-12-06 18:53:02.741 7f9862f7d640 4 mgr.server handle_report from 0x7f98900cf410 osd.18 -117> 2022-12-06 18:53:02.741 7f9862f7d640 4 mgr.server handle_report from 0x7f98900fe390 osd.14 -116> 2022-12-06 18:53:02.741 7f9862f7d640 4 mgr.server handle_report from 0x7f98900f27b0 osd.1 -115> 2022-12-06 18:53:02.760 7f9862f7d640 4 mgr.server handle_report from 0x7f9890202f30 mon.cartman06 -114> 2022-12-06 18:53:02.761 7f9862f7d640 4 mgr.server handle_report from 0x7f989002cee0 osd.16 -113> 2022-12-06 18:53:02.763 7f9862f7d640 4 mgr.server handle_report from 0x7f989020abd0 osd.4 -112> 2022-12-06 18:53:02.764 7f9862f7d640 4 mgr.server handle_report from 0x7f98900d7150 rgw.radosgw.cartman02 -111> 2022-12-06 18:53:02.764 7f9862f7d640 4 mgr.server handle_report from 0x7f98902073a0 osd.12 -110> 2022-12-06 18:53:02.764 7f9862f7d640 4 mgr.server handle_report from 0x7f98900cb570 osd.10 -109> 2022-12-06 18:53:02.764 7f9862f7d640 4 mgr.server handle_report from 0x7f9890084ec0 osd.17 -108> 2022-12-06 18:53:02.765 7f9862f7d640 4 mgr.server handle_report from 0x7f98900ee910 osd.0 -107> 2022-12-06 18:53:02.765 7f9862f7d640 4 mgr.server handle_report from 0x7f989005c690 osd.8 -106> 2022-12-06 18:53:02.765 7f9862f7d640 4 mgr.server handle_report from 0x7f989009c5f0 osd.6 -105> 2022-12-06 18:53:02.765 7f9862f7d640 4 mgr.server handle_report from 0x7f9890204430 mgr.cartman09 -104> 2022-12-06 18:53:02.766 7f9862f7d640 4 mgr.server handle_report from 0x7f9890023840 osd.15 -103> 2022-12-06 18:53:02.766 7f9862f7d640 4 mgr.server handle_report from 0x7f9890065b70 osd.13 -102> 2022-12-06 18:53:02.769 7f9862f7d640 4 mgr.server handle_report from 0x7f9890207e20 mon.patty -101> 2022-12-06 18:53:02.770 7f9862f7d640 4 mgr.server handle_report from 0x7f98901fcfb0 osd.7 -100> 2022-12-06 18:53:02.983 7f9825ffb640 4 mgrc handle_mgr_map Got map version 654 -99> 2022-12-06 18:53:02.983 7f9825ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -98> 2022-12-06 18:53:02.983 7f9848ff9640 4 mgrc handle_mgr_map Got map version 654 -97> 2022-12-06 18:53:02.983 7f9848ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -96> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr ms_dispatch active mgrmap(e 654) v1 -95> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr handle_mgr_map received map epoch 654 -94> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr handle_mgr_map active in map: 1 active is 2745812326 -93> 2022-12-06 18:53:02.995 7f9886ffd640 10 monclient: start_mon_command cmd=[{"prefix": "mgr metadata", "who": "patty", "id": "patty"}] -92> 2022-12-06 18:53:02.995 7f9886ffd640 10 monclient: _send_command 69 [{"prefix": "mgr metadata", "who": "patty", "id": "patty"}] -91> 2022-12-06 18:53:02.995 7f9886ffd640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -90> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr ms_dispatch mgrmap(e 654) v1 -89> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgrc handle_mgr_map Got map version 654 -88> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -87> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -86> 2022-12-06 18:53:02.995 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -85> 2022-12-06 18:53:02.996 7f9886ffd640 10 monclient: handle_mon_command_ack 69 [{"prefix": "mgr metadata", "who": "patty", "id": "patty"}] -84> 2022-12-06 18:53:02.996 7f9886ffd640 10 monclient: _finish_command 69 = 0 -83> 2022-12-06 18:53:02.996 7f98857fa640 4 mgr finish mon returned valid metadata JSON for mgr.patty -82> 2022-12-06 18:53:03.280 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f989014b760 -81> 2022-12-06 18:53:03.281 7f9862f7d640 4 mgr.server handle_open from mgr.cartman06 0x7f989014b760 -80> 2022-12-06 18:53:03.282 7f9862f7d640 2 mgr.server handle_open ignoring open from mgr.cartman06 172.20.4.101:0/934460; not ready for session (expect reconnect) -79> 2022-12-06 18:53:03.421 7f989dd9b640 10 monclient: handle_auth_request added challenge on 0x7f9890213ae0 -78> 2022-12-06 18:53:03.422 7f9862f7d640 4 mgr.server handle_open from mgr.patty 0x7f9890213ae0 -77> 2022-12-06 18:53:03.423 7f9862f7d640 4 mgr.server handle_report from 0x7f9890213ae0 mgr.patty -76> 2022-12-06 18:53:03.650 7f987e7fc640 4 mgr send_beacon active -75> 2022-12-06 18:53:03.663 7f987e7fc640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -74> 2022-12-06 18:53:03.994 7f9848ff9640 4 mgrc handle_mgr_map Got map version 655 -73> 2022-12-06 18:53:03.994 7f9848ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -72> 2022-12-06 18:53:03.995 7f9825ffb640 4 mgrc handle_mgr_map Got map version 655 -71> 2022-12-06 18:53:03.995 7f9825ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -70> 2022-12-06 18:53:04.013 7f9886ffd640 4 mgr ms_dispatch active mgrmap(e 655) v1 -69> 2022-12-06 18:53:04.013 7f9886ffd640 4 mgr handle_mgr_map received map epoch 655 -68> 2022-12-06 18:53:04.013 7f9886ffd640 4 mgr handle_mgr_map active in map: 1 active is 2745812326 -67> 2022-12-06 18:53:04.013 7f9886ffd640 10 monclient: start_mon_command cmd=[{"prefix": "mgr metadata", "who": "cartman06", "id": "cartman06"}] -66> 2022-12-06 18:53:04.013 7f9886ffd640 10 monclient: _send_command 70 [{"prefix": "mgr metadata", "who": "cartman06", "id": "cartman06"}] -65> 2022-12-06 18:53:04.013 7f9886ffd640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -64> 2022-12-06 18:53:04.014 7f9886ffd640 4 mgr ms_dispatch mgrmap(e 655) v1 -63> 2022-12-06 18:53:04.014 7f9886ffd640 4 mgrc handle_mgr_map Got map version 655 -62> 2022-12-06 18:53:04.014 7f9886ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.20.4.100:6814/274605,v1:172.20.4.100:6815/274605] -61> 2022-12-06 18:53:04.014 7f9886ffd640 10 monclient: handle_mon_command_ack 70 [{"prefix": "mgr metadata", "who": "cartman06", "id": "cartman06"}] -60> 2022-12-06 18:53:04.014 7f9886ffd640 10 monclient: _finish_command 70 = 0 -59> 2022-12-06 18:53:04.014 7f98857fa640 4 mgr finish mon returned valid metadata JSON for mgr.cartman06 -58> 2022-12-06 18:53:04.163 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f98902a62c0 -57> 2022-12-06 18:53:04.280 7f989e59c640 10 monclient: handle_auth_request added challenge on 0x7f9890294bc0 -56> 2022-12-06 18:53:04.281 7f9862f7d640 4 mgr.server handle_open from mgr.cartman06 0x7f9890294bc0 -55> 2022-12-06 18:53:04.282 7f9862f7d640 4 mgr.server handle_report from 0x7f9890294bc0 mgr.cartman06 -54> 2022-12-06 18:53:04.597 7f9885ffb640 10 monclient: tick -53> 2022-12-06 18:53:04.597 7f9885ffb640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-12-06 18:52:34.597953) -52> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client log_queue is 6 last_log 18 sent 12 num 6 unsent 6 sending 6 -51> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:52:54.702624 mgr.cartman09 (mgr.2745812326) 13 : cluster [DBG] pgmap v11: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 4.2 MiB/s rd, 1.8 MiB/s wr, 421 op/s; 13720/2501637 objects degraded (0.548%); 10 MiB/s, 8 objects/s recovering -50> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:52:56.703609 mgr.cartman09 (mgr.2745812326) 14 : cluster [DBG] pgmap v12: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 322 op/s; 13720/2501637 objects degraded (0.548%); 9.4 MiB/s, 7 objects/s recovering -49> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:52:58.706042 mgr.cartman09 (mgr.2745812326) 15 : cluster [DBG] pgmap v13: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 446 op/s; 13701/2501637 objects degraded (0.548%); 11 MiB/s, 9 objects/s recovering -48> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:53:00.706972 mgr.cartman09 (mgr.2745812326) 16 : cluster [DBG] pgmap v16: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 411 op/s; 13701/2501637 objects degraded (0.548%); 8.5 MiB/s, 7 objects/s recovering -47> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:53:00.762165 mgr.cartman09 (mgr.2745812326) 17 : audit [DBG] from='client.2745765143 v1:172.20.4.6:0/401948060' entity='client.admin' cmd=[{"prefix": "osd df", "output_method": "tree", "target": ["mgr", ""], "format": "json"}]: dispatch -46> 2022-12-06 18:53:04.597 7f9885ffb640 10 log_client will send 2022-12-06 18:53:02.707926 mgr.cartman09 (mgr.2745812326) 18 : cluster [DBG] pgmap v17: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 977 KiB/s rd, 882 KiB/s wr, 186 op/s; 13701/2501637 objects degraded (0.548%); 2.5 MiB/s, 2 objects/s recovering -45> 2022-12-06 18:53:04.597 7f9885ffb640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -44> 2022-12-06 18:53:04.709 7f9861f7b640 0 log_channel(cluster) log [DBG] : pgmap v18: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 545 active+clean, 5 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.6 MiB/s rd, 1.6 MiB/s wr, 412 op/s; 13637/2501637 objects degraded (0.545%); 23 MiB/s, 10 objects/s recovering -43> 2022-12-06 18:53:04.709 7f9861f7b640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -42> 2022-12-06 18:53:04.798 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f98902a62c0 -41> 2022-12-06 18:53:04.886 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f9890236560 -40> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client handle_log_ack log(last 18) v1 -39> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:52:54.702624 mgr.cartman09 (mgr.2745812326) 13 : cluster [DBG] pgmap v11: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 4.2 MiB/s rd, 1.8 MiB/s wr, 421 op/s; 13720/2501637 objects degraded (0.548%); 10 MiB/s, 8 objects/s recovering -38> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:52:56.703609 mgr.cartman09 (mgr.2745812326) 14 : cluster [DBG] pgmap v12: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 322 op/s; 13720/2501637 objects degraded (0.548%); 9.4 MiB/s, 7 objects/s recovering -37> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:52:58.706042 mgr.cartman09 (mgr.2745812326) 15 : cluster [DBG] pgmap v13: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 446 op/s; 13701/2501637 objects degraded (0.548%); 11 MiB/s, 9 objects/s recovering -36> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:53:00.706972 mgr.cartman09 (mgr.2745812326) 16 : cluster [DBG] pgmap v16: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 3.4 MiB/s rd, 2.1 MiB/s wr, 411 op/s; 13701/2501637 objects degraded (0.548%); 8.5 MiB/s, 7 objects/s recovering -35> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:53:00.762165 mgr.cartman09 (mgr.2745812326) 17 : audit [DBG] from='client.2745765143 v1:172.20.4.6:0/401948060' entity='client.admin' cmd=[{"prefix": "osd df", "output_method": "tree", "target": ["mgr", ""], "format": "json"}]: dispatch -34> 2022-12-06 18:53:04.998 7f9886ffd640 10 log_client logged 2022-12-06 18:53:02.707926 mgr.cartman09 (mgr.2745812326) 18 : cluster [DBG] pgmap v17: 552 pgs: 2 active+undersized+degraded+remapped+backfilling, 544 active+clean, 6 active+undersized+degraded+remapped+backfill_wait; 2.1 TiB data, 6.6 TiB used, 17 TiB / 23 TiB avail; 977 KiB/s rd, 882 KiB/s wr, 186 op/s; 13701/2501637 objects degraded (0.548%); 2.5 MiB/s, 2 objects/s recovering -33> 2022-12-06 18:53:05.005 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -32> 2022-12-06 18:53:05.005 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -31> 2022-12-06 18:53:05.217 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f98902a62c0 -30> 2022-12-06 18:53:05.281 7f9848ff9640 1 client.2745813838.objecter ms_handle_reset 0x7f984c0adfb0 session 0x7f984c049850 osd.4 -29> 2022-12-06 18:53:05.281 7f9848ff9640 10 monclient: _renew_subs -28> 2022-12-06 18:53:05.281 7f9848ff9640 10 monclient: _send_mon_message to mon.cartman09 at v1:172.20.4.100:6789/0 -27> 2022-12-06 18:53:05.405 7f9848ff9640 1 client.2745813838.objecter ms_handle_reset 0x7f983004ef10 session 0x7f984c049850 osd.4 -26> 2022-12-06 18:53:05.405 7f9848ff9640 10 monclient: _renew_subs -25> 2022-12-06 18:53:05.405 7f9848ff9640 10 monclient: _send_mon_message to mon.cartman09 at v1:172.20.4.100:6789/0 -24> 2022-12-06 18:53:05.405 7f9848ff9640 1 client.2745813838.objecter ms_handle_refused on osd.4 -23> 2022-12-06 18:53:05.606 7f9848ff9640 1 client.2745813838.objecter ms_handle_refused on osd.4 -22> 2022-12-06 18:53:05.663 7f987e7fc640 4 mgr send_beacon active -21> 2022-12-06 18:53:05.676 7f987e7fc640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -20> 2022-12-06 18:53:05.746 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f9890236560 -19> 2022-12-06 18:53:06.006 7f9886ffd640 4 mgr ms_dispatch active log(1 entries from seq 9018 at 2022-12-06 18:53:05.406348) v1 -18> 2022-12-06 18:53:06.006 7f9886ffd640 4 mgr ms_dispatch log(1 entries from seq 9018 at 2022-12-06 18:53:05.406348) v1 -17> 2022-12-06 18:53:06.007 7f9848ff9640 1 client.2745813838.objecter ms_handle_refused on osd.4 -16> 2022-12-06 18:53:06.014 7f9886ffd640 4 mgr ms_dispatch active osd_map(2237136..2237136 src has 2236635..2237136) v4 -15> 2022-12-06 18:53:06.014 7f9886ffd640 4 mgr ms_dispatch osd_map(2237136..2237136 src has 2236635..2237136) v4 -14> 2022-12-06 18:53:06.014 7f9886ffd640 10 monclient: _renew_subs -13> 2022-12-06 18:53:06.014 7f9886ffd640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -12> 2022-12-06 18:53:06.014 7f986377e640 4 mgr[progress] Processing OSDMap change 2237135..2237136 -11> 2022-12-06 18:53:06.018 7f9886ffd640 4 mgr ms_dispatch active mgrdigest v1 -10> 2022-12-06 18:53:06.018 7f9886ffd640 4 mgr ms_dispatch mgrdigest v1 -9> 2022-12-06 18:53:06.316 7f989fd9f640 10 monclient: handle_auth_request added challenge on 0x7f98902039b0 -8> 2022-12-06 18:53:06.577 7f9862f7d640 4 mgr.server _handle_command decoded 4 -7> 2022-12-06 18:53:06.577 7f9862f7d640 4 mgr.server _handle_command prefix=osd purge -6> 2022-12-06 18:53:06.577 7f9862f7d640 0 log_channel(audit) log [DBG] : from='client.2745765263 -' entity='client.admin' cmd=[{"prefix": "osd purge", "id": 4, "yes_i_really_mean_it": true, "target": ["mgr", ""]}]: dispatch -5> 2022-12-06 18:53:06.577 7f9862f7d640 10 monclient: start_mon_command cmd=[{"prefix": "osd purge-actual", "id": 4, "yes_i_really_mean_it": true}] -4> 2022-12-06 18:53:06.577 7f9862f7d640 10 monclient: _send_command 71 [{"prefix": "osd purge-actual", "id": 4, "yes_i_really_mean_it": true}] -3> 2022-12-06 18:53:06.577 7f9862f7d640 10 monclient: _send_mon_message to mon.patty at v1:172.20.4.10:6789/0 -2> 2022-12-06 18:53:06.594 7f9886ffd640 4 mgr ms_dispatch active osd_map(2237137..2237137 src has 2236635..2237137) v4 -1> 2022-12-06 18:53:06.594 7f9886ffd640 4 mgr ms_dispatch osd_map(2237137..2237137 src has 2236635..2237137) v4 0> 2022-12-06 18:53:06.596 7f9886ffd640 -1 *** Caught signal (Segmentation fault) ** in thread 7f9886ffd640 thread_name:ms_dispatch ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0x13700) [0x7f98a3b33700] 2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x13) [0x7f98a3995bd3] 3: (PGMap::apply_incremental(CephContext*, PGMap::Incremental const&)+0xf80) [0x557001741a40] 4: (ClusterState::notify_osdmap(OSDMap const&)+0x282) [0x5570017c8f42] 5: (Mgr::handle_osd_map()+0x57d) [0x5570018267cd] 6: (Mgr::ms_dispatch(Message*)+0x388) [0x557001826f18] 7: (MgrStandby::ms_dispatch(Message*)+0x1db) [0x557001837eab] 8: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x27) [0x5570018134d7] 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x4c8) [0x7f98a4bf1e88] 10: (DispatchQueue::entry()+0x6f7) [0x7f98a4bed7b7] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f98a4cc29cd] 12: (Thread::entry_wrapper()+0x50) [0x7f98a4a43c90] 13: (()+0x8e9e) [0x7f98a3b28e9e] 14: (clone()+0x3f) [0x7f98a36af4af] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 4/ 4 osd 0/ 5 optracker 0/ 5 objclass 4/ 4 filestore 4/ 4 journal 0/ 0 ms 4/ 4 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 4/ 4 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mgr.cartman09.log --- end dump of recent events ---