Project

General

Profile

Actions

Bug #40793

closed

mgr mon commands pile up

Added by Sage Weil almost 5 years ago. Updated almost 5 years ago.

Status:
Rejected
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

on lab cluster, a mon was down for a few days. on restart,

2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148399 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf11c00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169786 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d37800 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148400 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf11e00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169787 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d37a00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148401 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf12000 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169788 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d37c00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148402 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf12200 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169789 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d37e00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148403 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf12400 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169790 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d50000 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148404 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf12600 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169791 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d50200 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148405 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf12800 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169792 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d50400 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148406 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf12a00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169793 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d50600 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148407 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf12c00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169794 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d50800 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148408 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf12e00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169795 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d50a00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148409 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf13000 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169796 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d50c00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148410 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf13200 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169797 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d50e00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148411 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf13400 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169798 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d51000 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148412 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf13600 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169799 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d51200 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148413 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf13800 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169800 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d51400 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148414 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf13a00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169801 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d51600 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148415 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf13c00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169802 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d51800 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148416 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x1bf13e00 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169803 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x48d51a00 con 0x6720480
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058532 172.21.2.202:0/2325035 148417 ==== mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1 ==== 127+0+0 (crc 0 0 0) 0x1bf24000 con 0x6722400
2019-07-16 17:25:36.330 7f9afeca4700  5 mon.reesi005@4(synchronizing) e33 waitlisting message mon_command({"prefix": "osd pool set", "pool": "djf_tmp", "var": "pgp_num_actual", "val": "1465"} v 0) v1
2019-07-16 17:25:36.330 7f9afeca4700  1 -- [v2:172.21.2.205:3300/0,v1:172.21.2.205:6789/0] <== mgr.677058636 172.21.2.203:0/1095622 169804 ==== mon_command({"prefix": "osd pool set", "pool": "libvirt-pool", "var": "pgp_num_actual", "val": "334"} v 0) v1 ==== 131+0+0 (crc 0 0 0) 0x48d51c00 con 0x6720480

times a zillion


Related issues 1 (0 open1 closed)

Related to RADOS - Bug #40792: monc: send_command to specific down mon breaks other mon msgsResolvedSage Weil

Actions
Actions #1

Updated by Sage Weil almost 5 years ago

  • Status changed from 12 to Rejected

This was a side-effect of #40792. A targeted mon command was queued for down mon, which forced the MonClient to keep reconnecting to that (down) mon. Meanwhile, other mgr modules were sending untargetted mon commands, and zillions of them queued up. Once the mon finally came up, all the queued commands were sent.

Actions #2

Updated by Sage Weil almost 5 years ago

  • Related to Bug #40792: monc: send_command to specific down mon breaks other mon msgs added
Actions

Also available in: Atom PDF