See the zipped core.gz via gunzip core.gz at ceph-post-file: 7b0b4def-46e0-43a2-b819-0aafffd57b25 It was generated by the commands below. I started this instance, then I stopped the ceph-mgr service on the other nodes until crashy one woke up...which then crashed generating the 843M core. Shell details here: root@nocsupport4:/tmp# ulimit -c unlimited root@nocsupport4:/tmp# systemctl stop ceph-mgr ceph-mgr@nocsupport4.service ceph-mgr.target root@nocsupport4:/tmp# systemctl stop ceph-mgr@nocsupport4.service root@nocsupport4:/tmp# ceph config-key set config/mgr/mgr/devicehealth/enable_monitoring true set config/mgr/mgr/devicehealth/enable_monitoring root@nocsupport4:/tmp# /usr/bin/ceph-mgr -d --cluster ceph --id nocsupport4 --setuser ceph --setgroup ceph 2019-09-03 19:05:27.310 7f2288de3d00 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-09-03 19:05:27.314 7f2288de3d00 0 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-mgr, pid 2184 2019-09-03 19:05:27.314 7f2288de3d00 0 pidfile_write: ignore empty --pid-file 2019-09-03 19:05:27.542 7f2288de3d00 1 mgr[py] Loading python module 'ansible' 2019-09-03 19:05:28.990 7f2288de3d00 1 mgr[py] Loading python module 'balancer' 2019-09-03 19:05:29.226 7f2288de3d00 1 mgr[py] Loading python module 'crash' 2019-09-03 19:05:29.410 7f2288de3d00 1 mgr[py] Loading python module 'dashboard' 2019-09-03 19:05:32.186 7f2288de3d00 1 mgr[py] Loading python module 'deepsea' 2019-09-03 19:05:33.958 7f2288de3d00 1 mgr[py] Loading python module 'devicehealth' 2019-09-03 19:05:34.190 7f2288de3d00 1 mgr[py] Loading python module 'diskprediction_cloud' 2019-09-03 19:05:34.494 7f2288de3d00 1 mgr[py] Loading python module 'diskprediction_local' 2019-09-03 19:05:34.678 7f2288de3d00 1 mgr[py] Loading python module 'influx' 2019-09-03 19:05:34.874 7f2288de3d00 1 mgr[py] Loading python module 'insights' 2019-09-03 19:05:35.074 7f2288de3d00 1 mgr[py] Loading python module 'iostat' 2019-09-03 19:05:35.898 7f2288de3d00 1 mgr[py] Loading python module 'localpool' 2019-09-03 19:05:36.074 7f2288de3d00 1 mgr[py] Loading python module 'orchestrator_cli' 2019-09-03 19:05:36.294 7f2288de3d00 1 mgr[py] Loading python module 'pg_autoscaler' 2019-09-03 19:05:36.534 7f2288de3d00 1 mgr[py] Loading python module 'progress' 2019-09-03 19:05:36.726 7f2288de3d00 1 mgr[py] Loading python module 'prometheus' 2019-09-03 19:05:38.010 7f2288de3d00 1 mgr[py] Loading python module 'rbd_support' 2019-09-03 19:05:38.278 7f2288de3d00 1 mgr[py] Loading python module 'restful' 2019-09-03 19:05:40.314 7f2288de3d00 1 mgr[py] Loading python module 'rook' 2019-09-03 19:05:40.550 7f2288de3d00 1 mgr[py] Loading python module 'selftest' 2019-09-03 19:05:42.378 7f2288de3d00 1 mgr[py] Loading python module 'ssh' 2019-09-03 19:05:42.658 7f2288de3d00 1 mgr[py] Loading python module 'status' 2019-09-03 19:05:42.874 7f2288de3d00 1 mgr[py] Loading python module 'telegraf' 2019-09-03 19:05:43.070 7f2288de3d00 1 mgr[py] Loading python module 'telemetry' 2019-09-03 19:05:44.490 7f2288de3d00 1 mgr[py] Loading python module 'test_orchestrator' 2019-09-03 19:05:44.746 7f2288de3d00 1 mgr[py] Loading python module 'volumes' 2019-09-03 19:05:47.038 7f2288de3d00 1 mgr[py] Loading python module 'zabbix' 2019-09-03 19:05:47.230 7f2283ba2700 1 mgr load Constructed class from module: dashboard 2019-09-03 19:05:47.254 7f22833a1700 0 ms_deliver_dispatch: unhandled message 0x564f57009400 mon_map magic: 0 v1 from mon.2 v2:[fc00:1002:c7::b4]:3300/0 2019-09-03 19:05:47.262 7f22833a1700 0 client.0 ms_handle_reset on v2:[fc00:1002:c7::b2]:6802/1356 [03/Sep/2019:19:05:47] ENGINE Bus STARTING CherryPy Checker: The Application mounted at '' has an empty config. [03/Sep/2019:19:05:47] ENGINE Started monitor thread '_TimeoutMonitor'. [03/Sep/2019:19:05:47] ENGINE Serving on http://:::8080 [03/Sep/2019:19:05:47] ENGINE Bus STARTED ... -560> 2019-09-03 19:07:42.375 7f2273cd9700 4 mgr entry Entering thread for dashboard -559> 2019-09-03 19:07:42.379 7f2274cdb700 4 mgr entry Entering thread for devicehealth -558> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr[devicehealth] Starting -557> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/enable_monitoring -556> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/scrape_frequency -555> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_typed_config get_typed_config scrape_frequency not found -554> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/pool_name -553> 2019-09-03 19:07:42.675 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb43680 -552> 2019-09-03 19:07:42.675 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb43b00 -551> 2019-09-03 19:07:42.679 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f57c1c000 -550> 2019-09-03 19:07:42.683 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb80000 -549> 2019-09-03 19:07:42.683 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb45a80 -548> 2019-09-03 19:07:42.683 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb42480 -547> 2019-09-03 19:07:42.687 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb45600 -546> 2019-09-03 19:07:42.687 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb42000 -545> 2019-09-03 19:07:42.687 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb42900 -544> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb42d80 -543> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb44400 -542> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb44880 -541> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb45180 -540> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb80480 -539> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb43200 -538> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb43f80 -537> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdb200 -536> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb44d00 -535> 2019-09-03 19:07:42.695 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdb200 mds,nocsupport4 -534> 2019-09-03 19:07:42.699 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb81200 -533> 2019-09-03 19:07:42.699 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbda480 -532> 2019-09-03 19:07:42.703 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43680 osd,18 -531> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb80900 -530> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb81680 -529> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbda900 -528> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbdb680 -527> 2019-09-03 19:07:42.703 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdb200 mds,nocsupport4 -526> 2019-09-03 19:07:42.707 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdd600 -525> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb80d80 -524> 2019-09-03 19:07:42.707 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdd600 rgw,nocsupport4 -523> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbda000 -522> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdad80 -521> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdbb00 -520> 2019-09-03 19:07:42.711 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdd180 -519> 2019-09-03 19:07:42.711 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdc400 -518> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42000 osd,8 -517> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42900 osd,5 -516> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43680 osd,18 -515> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb44400 osd,16 -514> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45180 osd,9 -513> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb81200 osd,21 -512> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdda80 -511> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd02900 -510> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd03680 -509> 2019-09-03 19:07:42.739 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdbf80 -508> 2019-09-03 19:07:42.739 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdcd00 -507> 2019-09-03 19:07:42.743 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd04400 -506> 2019-09-03 19:07:42.747 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbdc880 -505> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd02480 -504> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd03200 -503> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd03f80 -502> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr ms_dispatch active mgrmap(e 1020) v1 -501> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr handle_mgr_map received map epoch 1020 -500> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr handle_mgr_map active in map: 1 active is 3774110 -499> 2019-09-03 19:07:42.763 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd02000 -498> 2019-09-03 19:07:42.767 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd03b00 -497> 2019-09-03 19:07:42.767 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd02d80 -496> 2019-09-03 19:07:42.783 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94000 -495> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server maybe_ready initial report from osd 18 -494> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server maybe_ready still waiting for 23 osds to report in before PGMap is ready -493> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdd600 rgw,nocsupport4 -492> 2019-09-03 19:07:43.207 7f22833a1700 4 mgr handle_mgr_map Map now says I am available -491> 2019-09-03 19:07:43.207 7f22833a1700 4 mgr ms_dispatch mgrmap(e 1020) v1 -490> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02000 osd,2 -489> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc handle_mgr_map Got map version 1020 -488> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -487> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02480 osd,20 -486> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -485> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02900 osd,17 -484> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -483> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -482> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03200 osd,10 -481> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03680 osd,19 -480> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03f80 mon,nocsupport2 -479> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd04400 osd,4 -478> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94000 mon,nocsupport4 -477> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42000 osd,8 -476> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active log(1 entries from seq 2383 at 2019-09-03 19:07:42.370148) v1 -475> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch log(1 entries from seq 2383 at 2019-09-03 19:07:42.370148) v1 -474> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -473> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -472> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42480 osd,14 -471> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42900 osd,5 -470> 2019-09-03 19:07:43.215 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42d80 osd,12 -469> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43200 osd,11 -468> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43b00 osd,6 -467> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43f80 osd,22 -466> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_report from 0x564f5bb44400 osd,16 -465> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb44d00 osd,1 -464> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45180 osd,9 -463> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45600 osd,3 -462> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45a80 rgw,nocsupport1 -461> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80480 mds,nocsupport1 -460> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80900 osd,7 -459> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80d80 osd,0 -458> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_report from 0x564f5bb81200 osd,21 -457> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bb81680 osd,15 -456> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbda000 osd,13 -455> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbda480 mds,nocsupport2 -454> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdad80 mon,nocsupport1 -453> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdb680 rgw,nocsupport3 -452> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdc400 mds,nocsupport3 -451> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdc880 mon,nocsupport3 -450> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdcd00 osd,23 -449> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdd180 rgw,nocsupport2 -448> 2019-09-03 19:07:43.239 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02000 osd,2 -447> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02480 osd,20 -446> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02900 osd,17 -445> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03680 osd,19 -444> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd04400 osd,4 -443> 2019-09-03 19:07:43.247 7f226eccf700 4 mgr.server maybe_ready initial report from osd 8 -442> 2019-09-03 19:07:43.247 7f226eccf700 4 mgr.server maybe_ready still waiting for 22 osds to report in before PGMap is ready -441> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 5 -440> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 21 osds to report in before PGMap is ready -439> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42d80 osd,12 -438> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 16 -437> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 20 osds to report in before PGMap is ready -436> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 9 -435> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 19 osds to report in before PGMap is ready -434> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80900 osd,7 -433> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 21 -432> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 18 osds to report in before PGMap is ready -431> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 2 -430> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 17 osds to report in before PGMap is ready -429> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 17 -428> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 16 osds to report in before PGMap is ready -427> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 19 -426> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 15 osds to report in before PGMap is ready -425> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 12 -424> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 14 osds to report in before PGMap is ready -423> 2019-09-03 19:07:43.255 7f2288bac700 10 monclient: get_auth_request con 0x564f5ad53600 auth_method 0 -422> 2019-09-03 19:07:43.267 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bf94480 -421> 2019-09-03 19:07:43.267 7f227fb9a700 4 mgr send_beacon active -420> 2019-09-03 19:07:43.267 7f227fb9a700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -419> 2019-09-03 19:07:43.287 7f226eccf700 4 mgr.server maybe_ready initial report from osd 4 -418> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server maybe_ready still waiting for 13 osds to report in before PGMap is ready -417> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94000 mon,nocsupport4 -416> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42480 osd,14 -415> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43200 osd,11 -414> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43b00 osd,6 -413> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43f80 osd,22 -412> 2019-09-03 19:07:43.295 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45600 osd,3 -411> 2019-09-03 19:07:43.295 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80480 mds,nocsupport1 -410> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server maybe_ready initial report from osd 7 -409> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server maybe_ready still waiting for 12 osds to report in before PGMap is ready -408> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80d80 osd,0 -407> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bb81680 osd,15 -406> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbda000 osd,13 -405> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbda480 mds,nocsupport2 -404> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdad80 mon,nocsupport1 -403> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdb680 rgw,nocsupport3 -402> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdc400 mds,nocsupport3 -401> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdc880 mon,nocsupport3 -400> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdcd00 osd,23 -399> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdd180 rgw,nocsupport2 -398> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server maybe_ready initial report from osd 20 -397> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server maybe_ready still waiting for 11 osds to report in before PGMap is ready -396> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03200 osd,10 -395> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03f80 mon,nocsupport2 -394> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94480 mgr,nocsupport4 -393> 2019-09-03 19:07:43.323 7f22833a1700 4 mgr ms_dispatch active mgrconfigure(period=5, threshold=5) v3 -392> 2019-09-03 19:07:43.323 7f22833a1700 4 mgr ms_dispatch mgrconfigure(period=5, threshold=5) v3 -391> 2019-09-03 19:07:43.323 7f22833a1700 4 mgrc handle_mgr_configure stats_period=5 -390> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 14 -389> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 10 osds to report in before PGMap is ready -388> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 11 -387> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 9 osds to report in before PGMap is ready -386> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 6 -385> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 8 osds to report in before PGMap is ready -384> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 22 -383> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 7 osds to report in before PGMap is ready -382> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server handle_report from 0x564f5bb44d00 osd,1 -381> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 3 -380> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 6 osds to report in before PGMap is ready -379> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45a80 rgw,nocsupport1 -378> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 0 -377> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 5 osds to report in before PGMap is ready -376> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready initial report from osd 15 -375> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready still waiting for 4 osds to report in before PGMap is ready -374> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready initial report from osd 13 -373> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready still waiting for 3 osds to report in before PGMap is ready -372> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 23 -371> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready still waiting for 2 osds to report in before PGMap is ready -370> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 10 -369> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready still waiting for 1 osds to report in before PGMap is ready -368> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94480 mgr,nocsupport4 -367> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 1 -366> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready all osds have reported, sending PG state to mon -365> 2019-09-03 19:07:43.347 7f226eccf700 0 log_channel(cluster) log [DBG] : pgmap v2: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail -364> 2019-09-03 19:07:43.347 7f226eccf700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -363> 2019-09-03 19:07:43.527 7f226f4d0700 1 mgr load Constructed class from module: orchestrator_cli -362> 2019-09-03 19:07:43.527 7f226f4d0700 4 mgr operator() Starting thread for orchestrator_cli -361> 2019-09-03 19:07:43.531 7f226f4d0700 1 mgr load Constructed class from module: progress -360> 2019-09-03 19:07:43.531 7f226f4d0700 4 mgr operator() Starting thread for progress -359> 2019-09-03 19:07:43.531 7f2271995700 4 mgr entry Entering thread for progress -358> 2019-09-03 19:07:43.535 7f2272196700 4 mgr entry Entering thread for orchestrator_cli -357> 2019-09-03 19:07:43.535 7f226f4d0700 1 mgr load Constructed class from module: status -356> 2019-09-03 19:07:43.535 7f226f4d0700 4 mgr operator() Starting thread for status -355> 2019-09-03 19:07:43.535 7f2271194700 4 mgr entry Entering thread for status -354> 2019-09-03 19:07:43.535 7f226f4d0700 1 mgr load Constructed class from module: telemetry -353> 2019-09-03 19:07:43.535 7f226f4d0700 4 mgr operator() Starting thread for telemetry -352> 2019-09-03 19:07:43.535 7f2271995700 4 mgr get_config get_config key: mgr/progress/max_completed_events -351> 2019-09-03 19:07:43.535 7f2271995700 4 mgr get_typed_config get_typed_config max_completed_events not found -350> 2019-09-03 19:07:43.539 7f2270993700 4 mgr entry Entering thread for telemetry -349> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_config get_config key: mgr/progress/persist_interval -348> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_typed_config get_typed_config persist_interval not found -347> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] Loading... -346> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_store get_store key: mgr/progress/completed -345> 2019-09-03 19:07:43.539 7f2271995700 4 ceph_store_get completed not found -344> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] No stored events to load -343> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] Loaded [] historic events -342> 2019-09-03 19:07:43.539 7f226f4d0700 1 mgr load Constructed class from module: volumes -341> 2019-09-03 19:07:43.539 7f226f4d0700 4 mgr operator() Starting thread for volumes -340> 2019-09-03 19:07:43.539 7f2270192700 4 mgr entry Entering thread for volumes -339> 2019-09-03 19:07:43.543 7f2271995700 4 mgr[progress] Loaded OSDMap, ready. -338> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config pool_name not found -337> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/retention_period -336> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config retention_period not found -335> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/mark_out_threshold -334> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config mark_out_threshold not found -333> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/warn_threshold -332> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_typed_config get_typed_config warn_threshold not found -331> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/self_heal -330> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_typed_config get_typed_config self_heal not found -329> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/sleep_interval -328> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr[dashboard] AC: Loading user roles DB version=1 -327> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_store get_store key: mgr/dashboard/accessdb_v1 -326> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr[dashboard] SSO: Loading SSO DB version=1 -325> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_store get_store key: mgr/dashboard/ssodb_v1 -324> 2019-09-03 19:07:43.567 7f2273cd9700 4 ceph_store_get ssodb_v1 not found -323> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/server_addr -322> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/server_addr -321> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_typed_config get_typed_config [nocsupport4/]server_addr not found -320> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/ssl -319> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/ssl -318> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] Optimize plan auto_2019-09-04_00:07:42 -317> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/mode -316> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] Mode upmap, max misplaced 0.050000 -315> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] do_upmap -314> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/upmap_max_iterations -313> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_typed_config get_typed_config upmap_max_iterations not found -312> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/upmap_max_deviation -311> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_typed_config get_typed_config upmap_max_deviation not found -310> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/server_port -309> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/server_port -308> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_typed_config get_typed_config [nocsupport4/]server_port not found -307> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr[dashboard] server: ssl=no host=:: port=8080 -306> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/url_prefix -305> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr get_typed_config get_typed_config sleep_interval not found -304> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr get_store get_store key: mgr/devicehealth/last_scrape -303> 2019-09-03 19:07:43.587 7f2274cdb700 4 ceph_store_get last_scrape not found -302> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr[devicehealth] Check health -301> 2019-09-03 19:07:43.595 7f2273cd9700 4 mgr get_typed_config get_typed_config url_prefix not found -300> 2019-09-03 19:07:43.611 7f2273cd9700 4 mgr[dashboard] Configured CherryPy, starting engine... -299> 2019-09-03 19:07:43.611 7f2273cd9700 4 mgr set_uri module dashboard set URI 'http://nocsupport4.1.quietfountain.com:8080/' -298> 2019-09-03 19:07:43.615 7f22734d8700 4 mgr[balancer] prepared 0/10 changes -297> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_store get_store key: mgr/telemetry/last_upload -296> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_store get_store key: mgr/telemetry/report_id -295> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/url -294> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config url not found -293> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/enabled -292> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/leaderboard -291> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config leaderboard not found -290> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/description -289> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config description not found -288> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/contact -287> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config contact not found -286> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/organization -285> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config organization not found -284> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/proxy -283> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config proxy not found -282> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/interval -281> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config interval not found -280> 2019-09-03 19:07:43.631 7f226dccd700 0 log_channel(cluster) log [DBG] : pgmap v3: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail -279> 2019-09-03 19:07:43.631 7f226dccd700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -278> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: get_monmap_and_config -277> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: build_initial_monmap -276> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: monmap: epoch 0 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-09-03 19:07:43.648217 created 2019-09-03 19:07:43.648217 min_mon_release 0 (unknown) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.noname-a 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.noname-b 2: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.noname-c 3: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.noname-d -275> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -274> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -273> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -272> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -271> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -270> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -269> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -268> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -267> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -266> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -265> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: init -264> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx -263> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx -262> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx -261> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -260> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -259> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -258> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -257> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -256> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc -255> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -254> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -253> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: _reopen_session rank -1 -252> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient(hunting): picked mon.noname-b con 0x564f5b19ad80 addr [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] -251> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient(hunting): picked mon.noname-c con 0x564f5b19b680 addr [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] -250> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): picked mon.noname-d con 0x564f5b19bf80 addr [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] -249> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection -248> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection -247> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection -246> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): _renew_subs -245> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): authenticate will time out at 2019-09-03 19:12:43.652328 -244> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): get_auth_request con 0x564f5b19ad80 auth_method 0 -243> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -242> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): _init_auth method 2 -241> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): get_auth_request con 0x564f5b19b680 auth_method 0 -240> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -239> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): _init_auth method 2 -238> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload 9 -237> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -236> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -235> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload 9 -234> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -233> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -232> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): get_auth_request con 0x564f5b19bf80 auth_method 0 -231> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -230> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): _init_auth method 2 -229> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_done global_id 3755231 payload 418 -228> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient: _finish_hunting 0 -227> 2019-09-03 19:07:43.647 7f2288bac700 1 monclient: found mon.noname-c -226> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient: _send_mon_message to mon.noname-c at v2:[fc00:1002:c7::b3]:3300/0 -225> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_monmap mon_map magic: 0 v1 -224> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: got monmap 6 from mon.noname-c (according to old e6) -223> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: dump: epoch 6 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-07-30 14:53:19.672269 created 2018-12-03 09:32:28.988007 min_mon_release 14 (nautilus) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2 2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4 3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3 -222> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _finish_auth 0 -221> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-03 19:07:13.657086) -220> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _send_mon_message to mon.nocsupport3 at v2:[fc00:1002:c7::b3]:3300/0 -219> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_config config(25 keys) v1 -218> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_monmap mon_map magic: 0 v1 -217> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: got monmap 6 from mon.nocsupport3 (according to old e6) -216> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: dump: epoch 6 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-07-30 14:53:19.672269 created 2018-12-03 09:32:28.988007 min_mon_release 14 (nautilus) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2 2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4 3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3 -215> 2019-09-03 19:07:43.651 7f226bcc9700 4 set_mon_vals no callback set -214> 2019-09-03 19:07:43.651 7f2274cdb700 5 monclient: authenticate success, global_id 3755231 -213> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/balancer/active = 1 (unrecognized option) -212> 2019-09-03 19:07:43.651 7f2274cdb700 10 monclient: get_monmap_and_config success -211> 2019-09-03 19:07:43.651 7f2274cdb700 10 monclient: shutdown -210> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/balancer/mode = upmap (unrecognized option) -209> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/password = $2b$12$X/T8Jev8/9YAPDBU.gFet.zN2WWr3tnQY71zo.qDMzN0t2vB/FRdm (unrecognized option) -208> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option) -207> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/username = admin (unrecognized option) -206> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/devicehealth/enable_monitoring = true (unrecognized option) -205> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/telemetry/enabled = true (unrecognized option) -204> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: build_initial_monmap -203> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: monmap: epoch 0 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-09-03 19:07:43.660749 created 2019-09-03 19:07:43.660749 min_mon_release 0 (unknown) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.noname-a 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.noname-b 2: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.noname-c 3: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.noname-d -202> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -201> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -200> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx -199> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -198> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -197> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -196> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -195> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -194> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc -193> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -192> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: starting msgr at -191> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: starting objecter -190> 2019-09-03 19:07:43.655 7f2274cdb700 5 asok(0x564f570ba000) register_command objecter_requests hook 0x564f5d373e20 EEXIST -189> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: setting wanted keys -188> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: calling monclient init -187> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: init -186> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx -185> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx -184> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx -183> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -182> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -181> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -180> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -179> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -178> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc -177> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -176> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring -175> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient: _reopen_session rank -1 -174> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-a con 0x564f5b19b680 addr [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] -173> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-c con 0x564f5b19ad80 addr [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] -172> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-b con 0x564f5b19c880 addr [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] -171> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection -170> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection -169> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection -168> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): _renew_subs -167> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): authenticate will time out at 2019-09-03 19:12:43.663758 -166> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): get_auth_request con 0x564f5b19ad80 auth_method 0 -165> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -164> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): _init_auth method 2 -163> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): get_auth_request con 0x564f5b19b680 auth_method 0 -162> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -161> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): _init_auth method 2 -160> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): get_auth_request con 0x564f5b19c880 auth_method 0 -159> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1] -158> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): _init_auth method 2 -157> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload 9 -156> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -155> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -154> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more payload 9 -153> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -152> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -151> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload 9 -150> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -149> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -148> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient(hunting): handle_auth_done global_id 3755053 payload 979 -147> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient: _finish_hunting 0 -146> 2019-09-03 19:07:43.663 7f22883ab700 1 monclient: found mon.noname-b -145> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient: _send_mon_message to mon.noname-b at v2:[fc00:1002:c7::b2]:3300/0 -144> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_monmap mon_map magic: 0 v1 -143> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: got monmap 6 from mon.noname-b (according to old e6) -142> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: dump: epoch 6 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-07-30 14:53:19.672269 created 2018-12-03 09:32:28.988007 min_mon_release 14 (nautilus) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2 2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4 3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3 -141> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _finish_auth 0 -140> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-03 19:07:13.674538) -139> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -138> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_config config(25 keys) v1 -137> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_monmap mon_map magic: 0 v1 -136> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: got monmap 6 from mon.nocsupport2 (according to old e6) -135> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: dump: epoch 6 fsid 4067126d-01cb-40af-824a-881c130140f8 last_changed 2019-07-30 14:53:19.672269 created 2018-12-03 09:32:28.988007 min_mon_release 14 (nautilus) 0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1 1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2 2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4 3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3 -134> 2019-09-03 19:07:43.667 7f226b4c8700 4 set_mon_vals no callback set -133> 2019-09-03 19:07:43.667 7f2274cdb700 5 monclient: authenticate success, global_id 3755053 -132> 2019-09-03 19:07:43.667 7f2274cdb700 10 monclient: _renew_subs -131> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/balancer/active = 1 (unrecognized option) -130> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/balancer/mode = upmap (unrecognized option) -129> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/password = $2b$12$X/T8Jev8/9YAPDBU.gFet.zN2WWr3tnQY71zo.qDMzN0t2vB/FRdm (unrecognized option) -128> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option) -127> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/username = admin (unrecognized option) -126> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/devicehealth/enable_monitoring = true (unrecognized option) -125> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/telemetry/enabled = true (unrecognized option) -124> 2019-09-03 19:07:43.667 7f2274cdb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -123> 2019-09-03 19:07:43.671 7f2274cdb700 10 monclient: _renew_subs -122> 2019-09-03 19:07:43.671 7f2274cdb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -121> 2019-09-03 19:07:43.671 7f2274cdb700 1 librados: init done -120> 2019-09-03 19:07:43.671 7f226cccb700 10 monclient: _finish_auth 0 -119> 2019-09-03 19:07:43.671 7f226cccb700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-09-03 19:07:13.677838) -118> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc handle_mgr_map Got map version 1020 -117> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -116> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -115> 2019-09-03 19:07:43.675 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0 -114> 2019-09-03 19:07:43.675 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94900 -113> 2019-09-03 19:07:43.675 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94900 client,nocsupport4 -112> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5 -111> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc handle_mgr_configure updated stats threshold: 5 -110> 2019-09-03 19:07:43.679 7f22883ab700 10 monclient: get_auth_request con 0x564f5b1ba480 auth_method 0 -109> 2019-09-03 19:07:43.679 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94900 client,nocsupport4 -108> 2019-09-03 19:07:43.679 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4 -107> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180 -106> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184 -105> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:44.679428 -104> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -103> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -102> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch active mgrdigest v1 -101> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch mgrdigest v1 -100> 2019-09-03 19:07:44.679 7f226acc7700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -99> 2019-09-03 19:07:44.679 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0 -98> 2019-09-03 19:07:44.679 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94d80 -97> 2019-09-03 19:07:44.679 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94d80 client,nocsupport4 -96> 2019-09-03 19:07:44.687 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5 -95> 2019-09-03 19:07:44.687 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94d80 client,nocsupport4 -94> 2019-09-03 19:07:44.687 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4 -93> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180 -92> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184 -91> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:45.683439 -90> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch active log(2 entries from seq 2385 at 2019-09-03 19:07:44.056172) v1 -89> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch log(2 entries from seq 2385 at 2019-09-03 19:07:44.056172) v1 -88> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -87> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -86> 2019-09-03 19:07:44.991 7f2273cd9700 4 mgr[dashboard] Initializing controller: Auth -> /api/auth -85> 2019-09-03 19:07:44.995 7f2273cd9700 4 mgr[dashboard] Initializing controller: CephFS -> /api/cephfs -84> 2019-09-03 19:07:44.995 7f2273cd9700 4 mgr[dashboard] Initializing controller: ClusterConfiguration -> /api/cluster_conf -83> 2019-09-03 19:07:44.999 7f2273cd9700 4 mgr[dashboard] Initializing controller: Docs -> /docs -82> 2019-09-03 19:07:44.999 7f2273cd9700 4 mgr[dashboard] Initializing controller: ErasureCodeProfile -> /api/erasure_code_profile -81> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Grafana -> /api/grafana -80> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Health -> /api/health -79> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Host -> /api/host -78> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: IscsiUi -> /ui-api/iscsi -77> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Iscsi -> /api/iscsi -76> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: IscsiTarget -> /api/iscsi/target -75> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Logging -> /ui-api/logging -74> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Logs -> /api/logs -73> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: MgrModules -> /api/mgr/module -72> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Monitor -> /api/monitor -71> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGanesha -> /api/nfs-ganesha -70> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaExports -> /api/nfs-ganesha/export -69> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaService -> /api/nfs-ganesha/daemon -68> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaUi -> /ui-api/nfs-ganesha -67> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: Osd -> /api/osd -66> 2019-09-03 19:07:45.015 7f2273cd9700 4 mgr[dashboard] Initializing controller: OsdFlagsController -> /api/osd/flags -65> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: MdsPerfCounter -> /api/perf_counters/mds -64> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: MonPerfCounter -> /api/perf_counters/mon -63> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: OsdPerfCounter -> /api/perf_counters/osd -62> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwPerfCounter -> /api/perf_counters/rgw -61> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror -60> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: MgrPerfCounter -> /api/perf_counters/mgr -59> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner -58> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: PerfCounters -> /api/perf_counters -57> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: Pool -> /api/pool -56> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: PrometheusReceiver -> /api/prometheus_receiver -55> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: Prometheus -> /api/prometheus -54> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: PrometheusNotifications -> /api/prometheus/notifications -53> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: Rbd -> /api/block/image -52> 2019-09-03 19:07:45.031 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdSnapshot -> /api/block/image/{pool_name}/{image_name}/snap -51> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdTrash -> /api/block/image/trash -50> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringSummary -> /api/block/mirroring/summary -49> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringPoolMode -> /api/block/mirroring/pool -48> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer -47> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: Rgw -> /api/rgw -46> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwDaemon -> /api/rgw/daemon -45> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwBucket -> /api/rgw/bucket -44> 2019-09-03 19:07:45.043 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwUser -> /api/rgw/user -43> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Role -> /api/role -42> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Scope -> /ui-api/scope -41> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Saml2 -> /auth/saml2 -40> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Settings -> /api/settings -39> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Summary -> /api/summary -38> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Task -> /api/task -37> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: User -> /api/user -36> 2019-09-03 19:07:45.055 7f2273cd9700 4 mgr[dashboard] Initializing controller: FeatureTogglesEndpoint -> /api/feature_toggles -35> 2019-09-03 19:07:45.139 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Bus STARTING -34> 2019-09-03 19:07:45.267 7f227fb9a700 4 mgr send_beacon active -33> 2019-09-03 19:07:45.271 7f227fb9a700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -32> 2019-09-03 19:07:45.291 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Started monitor thread '_TimeoutMonitor'. -31> 2019-09-03 19:07:45.391 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Serving on http://:::8080 -30> 2019-09-03 19:07:45.391 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Bus STARTED -29> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr[dashboard] Engine started. -28> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/GRAFANA_UPDATE_DASHBOARDS -27> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr get_typed_config get_typed_config GRAFANA_UPDATE_DASHBOARDS not found -26> 2019-09-03 19:07:45.635 7f226dccd700 0 log_channel(cluster) log [DBG] : pgmap v4: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail -25> 2019-09-03 19:07:45.635 7f226dccd700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0 -24> 2019-09-03 19:07:45.679 7f226acc7700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -23> 2019-09-03 19:07:45.679 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0 -22> 2019-09-03 19:07:45.679 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94900 -21> 2019-09-03 19:07:45.679 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94900 client,nocsupport4 -20> 2019-09-03 19:07:45.679 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5 -19> 2019-09-03 19:07:45.683 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94900 client,nocsupport4 -18> 2019-09-03 19:07:45.683 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4 -17> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180 -16> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184 -15> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:46.683821 -14> 2019-09-03 19:07:45.987 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -13> 2019-09-03 19:07:45.987 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -12> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch active mgrmap(e 1021) v1 -11> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr handle_mgr_map received map epoch 1021 -10> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr handle_mgr_map active in map: 1 active is 3774110 -9> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch mgrmap(e 1021) v1 -8> 2019-09-03 19:07:46.231 7f22833a1700 4 mgrc handle_mgr_map Got map version 1021 -7> 2019-09-03 19:07:46.231 7f22833a1700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -6> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc handle_mgr_map Got map version 1021 -5> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184] -4> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:46.683821 -3> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1 -2> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1 -1> 2019-09-03 19:07:46.263 7f2288bac700 10 monclient: get_auth_request con 0x564f5b1bb680 auth_method 0 0> 2019-09-03 19:07:46.275 7f2274cdb700 -1 *** Caught signal (Segmentation fault) ** in thread 7f2274cdb700 thread_name:devicehealth ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) 1: (()+0x14840) [0x7f2289a8d840] 2: (rados_write_op_omap_set()+0x138) [0x7f227df04368] 3: (()+0xc2a85) [0x7f227e0bba85] 4: (_PyMethodDef_RawFastCallDict()+0x1a2) [0x7f2289f6f4d2] 5: (()+0x25828b) [0x7f2289f6a28b] 6: (()+0x44586) [0x7f227e03d586] 7: (()+0x7f84e) [0x7f227e07884e] 8: (_PyObject_FastCallKeywords()+0xd7) [0x7f2289f70037] 9: (()+0x6dfc3) [0x7f2289d7ffc3] 10: (_PyEval_EvalFrameDefault()+0x68a2) [0x7f2289d86c42] 11: (()+0x7794b) [0x7f2289d8994b] 12: (()+0x6e015) [0x7f2289d80015] 13: (_PyEval_EvalFrameDefault()+0x242d) [0x7f2289d827cd] 14: (()+0x7794b) [0x7f2289d8994b] 15: (()+0x6e015) [0x7f2289d80015] 16: (_PyEval_EvalFrameDefault()+0x242d) [0x7f2289d827cd] 17: (()+0x7794b) [0x7f2289d8994b] 18: (_PyFunction_FastCallDict()+0x2c8) [0x7f2289f6fae8] 19: (_PyObject_Call_Prepend()+0xcd) [0x7f2289f713ad] 20: (_PyObject_FastCallDict()+0x8a) [0x7f2289f7021a] 21: (PyObject_CallMethod()+0xfe) [0x7f2289f70f7e] 22: (PyModuleRunner::serve()+0x66) [0x564f55645b86] 23: (PyModuleRunner::PyModuleRunnerThread::entry()+0x1d4) [0x564f556462e4] 24: (()+0x975a) [0x7f2289a8275a] 25: (clone()+0x43) [0x7f228964b9f3] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/lib/ceph/crash/2019-09-04_00:07:46.281830Z_7128b0c3-651f-4841-82bd-fc59d45e57aa/log --- end dump of recent events --- Segmentation fault (core dumped) root@nocsupport4:/tmp# ls core core root@nocsupport4:/tmp# ls -l core -rw------- 1 ceph ceph 873644032 Sep 3 19:07 core root@nocsupport4:/tmp#