Project

General

Profile

Bug #41618 » log.txt

Harry Coin, 09/04/2019 12:20 AM

 
See the zipped core.gz via gunzip core.gz at ceph-post-file: 7b0b4def-46e0-43a2-b819-0aafffd57b25

It was generated by the commands below. I started this instance, then I stopped the ceph-mgr service on the other nodes until crashy one woke up...which then crashed generating the 843M core.

Shell details here:


root@nocsupport4:/tmp# ulimit -c unlimited
root@nocsupport4:/tmp# systemctl stop ceph-mgr
ceph-mgr@nocsupport4.service ceph-mgr.target
root@nocsupport4:/tmp# systemctl stop ceph-mgr@nocsupport4.service
root@nocsupport4:/tmp# ceph config-key set config/mgr/mgr/devicehealth/enable_monitoring true
set config/mgr/mgr/devicehealth/enable_monitoring
root@nocsupport4:/tmp# /usr/bin/ceph-mgr -d --cluster ceph --id nocsupport4 --setuser ceph --setgroup ceph
2019-09-03 19:05:27.310 7f2288de3d00 0 set uid:gid to 64045:64045 (ceph:ceph)
2019-09-03 19:05:27.314 7f2288de3d00 0 ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-mgr, pid 2184
2019-09-03 19:05:27.314 7f2288de3d00 0 pidfile_write: ignore empty --pid-file
2019-09-03 19:05:27.542 7f2288de3d00 1 mgr[py] Loading python module 'ansible'
2019-09-03 19:05:28.990 7f2288de3d00 1 mgr[py] Loading python module 'balancer'
2019-09-03 19:05:29.226 7f2288de3d00 1 mgr[py] Loading python module 'crash'
2019-09-03 19:05:29.410 7f2288de3d00 1 mgr[py] Loading python module 'dashboard'
2019-09-03 19:05:32.186 7f2288de3d00 1 mgr[py] Loading python module 'deepsea'
2019-09-03 19:05:33.958 7f2288de3d00 1 mgr[py] Loading python module 'devicehealth'
2019-09-03 19:05:34.190 7f2288de3d00 1 mgr[py] Loading python module 'diskprediction_cloud'
2019-09-03 19:05:34.494 7f2288de3d00 1 mgr[py] Loading python module 'diskprediction_local'
2019-09-03 19:05:34.678 7f2288de3d00 1 mgr[py] Loading python module 'influx'
2019-09-03 19:05:34.874 7f2288de3d00 1 mgr[py] Loading python module 'insights'
2019-09-03 19:05:35.074 7f2288de3d00 1 mgr[py] Loading python module 'iostat'
2019-09-03 19:05:35.898 7f2288de3d00 1 mgr[py] Loading python module 'localpool'
2019-09-03 19:05:36.074 7f2288de3d00 1 mgr[py] Loading python module 'orchestrator_cli'
2019-09-03 19:05:36.294 7f2288de3d00 1 mgr[py] Loading python module 'pg_autoscaler'
2019-09-03 19:05:36.534 7f2288de3d00 1 mgr[py] Loading python module 'progress'
2019-09-03 19:05:36.726 7f2288de3d00 1 mgr[py] Loading python module 'prometheus'
2019-09-03 19:05:38.010 7f2288de3d00 1 mgr[py] Loading python module 'rbd_support'
2019-09-03 19:05:38.278 7f2288de3d00 1 mgr[py] Loading python module 'restful'
2019-09-03 19:05:40.314 7f2288de3d00 1 mgr[py] Loading python module 'rook'
2019-09-03 19:05:40.550 7f2288de3d00 1 mgr[py] Loading python module 'selftest'
2019-09-03 19:05:42.378 7f2288de3d00 1 mgr[py] Loading python module 'ssh'
2019-09-03 19:05:42.658 7f2288de3d00 1 mgr[py] Loading python module 'status'
2019-09-03 19:05:42.874 7f2288de3d00 1 mgr[py] Loading python module 'telegraf'
2019-09-03 19:05:43.070 7f2288de3d00 1 mgr[py] Loading python module 'telemetry'
2019-09-03 19:05:44.490 7f2288de3d00 1 mgr[py] Loading python module 'test_orchestrator'
2019-09-03 19:05:44.746 7f2288de3d00 1 mgr[py] Loading python module 'volumes'
2019-09-03 19:05:47.038 7f2288de3d00 1 mgr[py] Loading python module 'zabbix'
2019-09-03 19:05:47.230 7f2283ba2700 1 mgr load Constructed class from module: dashboard
2019-09-03 19:05:47.254 7f22833a1700 0 ms_deliver_dispatch: unhandled message 0x564f57009400 mon_map magic: 0 v1 from mon.2 v2:[fc00:1002:c7::b4]:3300/0
2019-09-03 19:05:47.262 7f22833a1700 0 client.0 ms_handle_reset on v2:[fc00:1002:c7::b2]:6802/1356
[03/Sep/2019:19:05:47] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.
[03/Sep/2019:19:05:47] ENGINE Started monitor thread '_TimeoutMonitor'.
[03/Sep/2019:19:05:47] ENGINE Serving on http://:::8080
[03/Sep/2019:19:05:47] ENGINE Bus STARTED
...
-560> 2019-09-03 19:07:42.375 7f2273cd9700 4 mgr entry Entering thread for dashboard
-559> 2019-09-03 19:07:42.379 7f2274cdb700 4 mgr entry Entering thread for devicehealth
-558> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr[devicehealth] Starting
-557> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/enable_monitoring
-556> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/scrape_frequency
-555> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_typed_config get_typed_config scrape_frequency not found
-554> 2019-09-03 19:07:42.387 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/pool_name
-553> 2019-09-03 19:07:42.675 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb43680
-552> 2019-09-03 19:07:42.675 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb43b00
-551> 2019-09-03 19:07:42.679 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f57c1c000
-550> 2019-09-03 19:07:42.683 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb80000
-549> 2019-09-03 19:07:42.683 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb45a80
-548> 2019-09-03 19:07:42.683 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb42480
-547> 2019-09-03 19:07:42.687 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb45600
-546> 2019-09-03 19:07:42.687 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb42000
-545> 2019-09-03 19:07:42.687 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb42900
-544> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb42d80
-543> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb44400
-542> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb44880
-541> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb45180
-540> 2019-09-03 19:07:42.691 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb80480
-539> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb43200
-538> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb43f80
-537> 2019-09-03 19:07:42.691 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdb200
-536> 2019-09-03 19:07:42.691 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb44d00
-535> 2019-09-03 19:07:42.695 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdb200 mds,nocsupport4
-534> 2019-09-03 19:07:42.699 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bb81200
-533> 2019-09-03 19:07:42.699 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbda480
-532> 2019-09-03 19:07:42.703 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43680 osd,18
-531> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb80900
-530> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bb81680
-529> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbda900
-528> 2019-09-03 19:07:42.703 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbdb680
-527> 2019-09-03 19:07:42.703 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdb200 mds,nocsupport4
-526> 2019-09-03 19:07:42.707 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdd600
-525> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bb80d80
-524> 2019-09-03 19:07:42.707 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdd600 rgw,nocsupport4
-523> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbda000
-522> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdad80
-521> 2019-09-03 19:07:42.707 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdbb00
-520> 2019-09-03 19:07:42.711 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdd180
-519> 2019-09-03 19:07:42.711 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdc400
-518> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42000 osd,8
-517> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42900 osd,5
-516> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43680 osd,18
-515> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb44400 osd,16
-514> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45180 osd,9
-513> 2019-09-03 19:07:42.719 7f226eccf700 4 mgr.server handle_open from 0x564f5bb81200 osd,21
-512> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bbdda80
-511> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd02900
-510> 2019-09-03 19:07:42.731 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd03680
-509> 2019-09-03 19:07:42.739 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdbf80
-508> 2019-09-03 19:07:42.739 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bbdcd00
-507> 2019-09-03 19:07:42.743 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bd04400
-506> 2019-09-03 19:07:42.747 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bbdc880
-505> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd02480
-504> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd03200
-503> 2019-09-03 19:07:42.751 7f2287baa700 10 monclient: handle_auth_request added challenge on 0x564f5bd03f80
-502> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr ms_dispatch active mgrmap(e 1020) v1
-501> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr handle_mgr_map received map epoch 1020
-500> 2019-09-03 19:07:42.759 7f22833a1700 4 mgr handle_mgr_map active in map: 1 active is 3774110
-499> 2019-09-03 19:07:42.763 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd02000
-498> 2019-09-03 19:07:42.767 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd03b00
-497> 2019-09-03 19:07:42.767 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bd02d80
-496> 2019-09-03 19:07:42.783 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94000
-495> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server maybe_ready initial report from osd 18
-494> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server maybe_ready still waiting for 23 osds to report in before PGMap is ready
-493> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdd600 rgw,nocsupport4
-492> 2019-09-03 19:07:43.207 7f22833a1700 4 mgr handle_mgr_map Map now says I am available
-491> 2019-09-03 19:07:43.207 7f22833a1700 4 mgr ms_dispatch mgrmap(e 1020) v1
-490> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02000 osd,2
-489> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc handle_mgr_map Got map version 1020
-488> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-487> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02480 osd,20
-486> 2019-09-03 19:07:43.207 7f22833a1700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-485> 2019-09-03 19:07:43.207 7f226eccf700 4 mgr.server handle_open from 0x564f5bd02900 osd,17
-484> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-483> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-482> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03200 osd,10
-481> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03680 osd,19
-480> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd03f80 mon,nocsupport2
-479> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bd04400 osd,4
-478> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94000 mon,nocsupport4
-477> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42000 osd,8
-476> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active log(1 entries from seq 2383 at 2019-09-03 19:07:42.370148) v1
-475> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch log(1 entries from seq 2383 at 2019-09-03 19:07:42.370148) v1
-474> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-473> 2019-09-03 19:07:43.211 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-472> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42480 osd,14
-471> 2019-09-03 19:07:43.211 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42900 osd,5
-470> 2019-09-03 19:07:43.215 7f226eccf700 4 mgr.server handle_open from 0x564f5bb42d80 osd,12
-469> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43200 osd,11
-468> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43b00 osd,6
-467> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_open from 0x564f5bb43f80 osd,22
-466> 2019-09-03 19:07:43.227 7f226eccf700 4 mgr.server handle_report from 0x564f5bb44400 osd,16
-465> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb44d00 osd,1
-464> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45180 osd,9
-463> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45600 osd,3
-462> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb45a80 rgw,nocsupport1
-461> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80480 mds,nocsupport1
-460> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80900 osd,7
-459> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_open from 0x564f5bb80d80 osd,0
-458> 2019-09-03 19:07:43.231 7f226eccf700 4 mgr.server handle_report from 0x564f5bb81200 osd,21
-457> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bb81680 osd,15
-456> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbda000 osd,13
-455> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbda480 mds,nocsupport2
-454> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdad80 mon,nocsupport1
-453> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdb680 rgw,nocsupport3
-452> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdc400 mds,nocsupport3
-451> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdc880 mon,nocsupport3
-450> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdcd00 osd,23
-449> 2019-09-03 19:07:43.235 7f226eccf700 4 mgr.server handle_open from 0x564f5bbdd180 rgw,nocsupport2
-448> 2019-09-03 19:07:43.239 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02000 osd,2
-447> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02480 osd,20
-446> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd02900 osd,17
-445> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03680 osd,19
-444> 2019-09-03 19:07:43.243 7f226eccf700 4 mgr.server handle_report from 0x564f5bd04400 osd,4
-443> 2019-09-03 19:07:43.247 7f226eccf700 4 mgr.server maybe_ready initial report from osd 8
-442> 2019-09-03 19:07:43.247 7f226eccf700 4 mgr.server maybe_ready still waiting for 22 osds to report in before PGMap is ready
-441> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 5
-440> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 21 osds to report in before PGMap is ready
-439> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42d80 osd,12
-438> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 16
-437> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 20 osds to report in before PGMap is ready
-436> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready initial report from osd 9
-435> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server maybe_ready still waiting for 19 osds to report in before PGMap is ready
-434> 2019-09-03 19:07:43.251 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80900 osd,7
-433> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 21
-432> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 18 osds to report in before PGMap is ready
-431> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 2
-430> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 17 osds to report in before PGMap is ready
-429> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 17
-428> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 16 osds to report in before PGMap is ready
-427> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 19
-426> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 15 osds to report in before PGMap is ready
-425> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready initial report from osd 12
-424> 2019-09-03 19:07:43.255 7f226eccf700 4 mgr.server maybe_ready still waiting for 14 osds to report in before PGMap is ready
-423> 2019-09-03 19:07:43.255 7f2288bac700 10 monclient: get_auth_request con 0x564f5ad53600 auth_method 0
-422> 2019-09-03 19:07:43.267 7f22883ab700 10 monclient: handle_auth_request added challenge on 0x564f5bf94480
-421> 2019-09-03 19:07:43.267 7f227fb9a700 4 mgr send_beacon active
-420> 2019-09-03 19:07:43.267 7f227fb9a700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-419> 2019-09-03 19:07:43.287 7f226eccf700 4 mgr.server maybe_ready initial report from osd 4
-418> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server maybe_ready still waiting for 13 osds to report in before PGMap is ready
-417> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94000 mon,nocsupport4
-416> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb42480 osd,14
-415> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43200 osd,11
-414> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43b00 osd,6
-413> 2019-09-03 19:07:43.291 7f226eccf700 4 mgr.server handle_report from 0x564f5bb43f80 osd,22
-412> 2019-09-03 19:07:43.295 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45600 osd,3
-411> 2019-09-03 19:07:43.295 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80480 mds,nocsupport1
-410> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server maybe_ready initial report from osd 7
-409> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server maybe_ready still waiting for 12 osds to report in before PGMap is ready
-408> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bb80d80 osd,0
-407> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bb81680 osd,15
-406> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbda000 osd,13
-405> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbda480 mds,nocsupport2
-404> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdad80 mon,nocsupport1
-403> 2019-09-03 19:07:43.311 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdb680 rgw,nocsupport3
-402> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdc400 mds,nocsupport3
-401> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdc880 mon,nocsupport3
-400> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdcd00 osd,23
-399> 2019-09-03 19:07:43.315 7f226eccf700 4 mgr.server handle_report from 0x564f5bbdd180 rgw,nocsupport2
-398> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server maybe_ready initial report from osd 20
-397> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server maybe_ready still waiting for 11 osds to report in before PGMap is ready
-396> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03200 osd,10
-395> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_report from 0x564f5bd03f80 mon,nocsupport2
-394> 2019-09-03 19:07:43.319 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94480 mgr,nocsupport4
-393> 2019-09-03 19:07:43.323 7f22833a1700 4 mgr ms_dispatch active mgrconfigure(period=5, threshold=5) v3
-392> 2019-09-03 19:07:43.323 7f22833a1700 4 mgr ms_dispatch mgrconfigure(period=5, threshold=5) v3
-391> 2019-09-03 19:07:43.323 7f22833a1700 4 mgrc handle_mgr_configure stats_period=5
-390> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 14
-389> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 10 osds to report in before PGMap is ready
-388> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 11
-387> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 9 osds to report in before PGMap is ready
-386> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 6
-385> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 8 osds to report in before PGMap is ready
-384> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 22
-383> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 7 osds to report in before PGMap is ready
-382> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server handle_report from 0x564f5bb44d00 osd,1
-381> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 3
-380> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 6 osds to report in before PGMap is ready
-379> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server handle_report from 0x564f5bb45a80 rgw,nocsupport1
-378> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready initial report from osd 0
-377> 2019-09-03 19:07:43.323 7f226eccf700 4 mgr.server maybe_ready still waiting for 5 osds to report in before PGMap is ready
-376> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready initial report from osd 15
-375> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready still waiting for 4 osds to report in before PGMap is ready
-374> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready initial report from osd 13
-373> 2019-09-03 19:07:43.327 7f226eccf700 4 mgr.server maybe_ready still waiting for 3 osds to report in before PGMap is ready
-372> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 23
-371> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready still waiting for 2 osds to report in before PGMap is ready
-370> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 10
-369> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready still waiting for 1 osds to report in before PGMap is ready
-368> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94480 mgr,nocsupport4
-367> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready initial report from osd 1
-366> 2019-09-03 19:07:43.331 7f226eccf700 4 mgr.server maybe_ready all osds have reported, sending PG state to mon
-365> 2019-09-03 19:07:43.347 7f226eccf700 0 log_channel(cluster) log [DBG] : pgmap v2: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail
-364> 2019-09-03 19:07:43.347 7f226eccf700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-363> 2019-09-03 19:07:43.527 7f226f4d0700 1 mgr load Constructed class from module: orchestrator_cli
-362> 2019-09-03 19:07:43.527 7f226f4d0700 4 mgr operator() Starting thread for orchestrator_cli
-361> 2019-09-03 19:07:43.531 7f226f4d0700 1 mgr load Constructed class from module: progress
-360> 2019-09-03 19:07:43.531 7f226f4d0700 4 mgr operator() Starting thread for progress
-359> 2019-09-03 19:07:43.531 7f2271995700 4 mgr entry Entering thread for progress
-358> 2019-09-03 19:07:43.535 7f2272196700 4 mgr entry Entering thread for orchestrator_cli
-357> 2019-09-03 19:07:43.535 7f226f4d0700 1 mgr load Constructed class from module: status
-356> 2019-09-03 19:07:43.535 7f226f4d0700 4 mgr operator() Starting thread for status
-355> 2019-09-03 19:07:43.535 7f2271194700 4 mgr entry Entering thread for status
-354> 2019-09-03 19:07:43.535 7f226f4d0700 1 mgr load Constructed class from module: telemetry
-353> 2019-09-03 19:07:43.535 7f226f4d0700 4 mgr operator() Starting thread for telemetry
-352> 2019-09-03 19:07:43.535 7f2271995700 4 mgr get_config get_config key: mgr/progress/max_completed_events
-351> 2019-09-03 19:07:43.535 7f2271995700 4 mgr get_typed_config get_typed_config max_completed_events not found
-350> 2019-09-03 19:07:43.539 7f2270993700 4 mgr entry Entering thread for telemetry
-349> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_config get_config key: mgr/progress/persist_interval
-348> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_typed_config get_typed_config persist_interval not found
-347> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] Loading...
-346> 2019-09-03 19:07:43.539 7f2271995700 4 mgr get_store get_store key: mgr/progress/completed
-345> 2019-09-03 19:07:43.539 7f2271995700 4 ceph_store_get completed not found
-344> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] No stored events to load
-343> 2019-09-03 19:07:43.539 7f2271995700 4 mgr[progress] Loaded [] historic events
-342> 2019-09-03 19:07:43.539 7f226f4d0700 1 mgr load Constructed class from module: volumes
-341> 2019-09-03 19:07:43.539 7f226f4d0700 4 mgr operator() Starting thread for volumes
-340> 2019-09-03 19:07:43.539 7f2270192700 4 mgr entry Entering thread for volumes
-339> 2019-09-03 19:07:43.543 7f2271995700 4 mgr[progress] Loaded OSDMap, ready.
-338> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config pool_name not found
-337> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/retention_period
-336> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config retention_period not found
-335> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/mark_out_threshold
-334> 2019-09-03 19:07:43.551 7f2274cdb700 4 mgr get_typed_config get_typed_config mark_out_threshold not found
-333> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/warn_threshold
-332> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_typed_config get_typed_config warn_threshold not found
-331> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/self_heal
-330> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_typed_config get_typed_config self_heal not found
-329> 2019-09-03 19:07:43.555 7f2274cdb700 4 mgr get_config get_config key: mgr/devicehealth/sleep_interval
-328> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr[dashboard] AC: Loading user roles DB version=1
-327> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_store get_store key: mgr/dashboard/accessdb_v1
-326> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr[dashboard] SSO: Loading SSO DB version=1
-325> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_store get_store key: mgr/dashboard/ssodb_v1
-324> 2019-09-03 19:07:43.567 7f2273cd9700 4 ceph_store_get ssodb_v1 not found
-323> 2019-09-03 19:07:43.567 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/server_addr
-322> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/server_addr
-321> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_typed_config get_typed_config [nocsupport4/]server_addr not found
-320> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/ssl
-319> 2019-09-03 19:07:43.571 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/ssl
-318> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] Optimize plan auto_2019-09-04_00:07:42
-317> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/mode
-316> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] Mode upmap, max misplaced 0.050000
-315> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr[balancer] do_upmap
-314> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/upmap_max_iterations
-313> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_typed_config get_typed_config upmap_max_iterations not found
-312> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_config get_config key: mgr/balancer/upmap_max_deviation
-311> 2019-09-03 19:07:43.575 7f22734d8700 4 mgr get_typed_config get_typed_config upmap_max_deviation not found
-310> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/nocsupport4/server_port
-309> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/server_port
-308> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_typed_config get_typed_config [nocsupport4/]server_port not found
-307> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr[dashboard] server: ssl=no host=:: port=8080
-306> 2019-09-03 19:07:43.575 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/url_prefix
-305> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr get_typed_config get_typed_config sleep_interval not found
-304> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr get_store get_store key: mgr/devicehealth/last_scrape
-303> 2019-09-03 19:07:43.587 7f2274cdb700 4 ceph_store_get last_scrape not found
-302> 2019-09-03 19:07:43.587 7f2274cdb700 4 mgr[devicehealth] Check health
-301> 2019-09-03 19:07:43.595 7f2273cd9700 4 mgr get_typed_config get_typed_config url_prefix not found
-300> 2019-09-03 19:07:43.611 7f2273cd9700 4 mgr[dashboard] Configured CherryPy, starting engine...
-299> 2019-09-03 19:07:43.611 7f2273cd9700 4 mgr set_uri module dashboard set URI 'http://nocsupport4.1.quietfountain.com:8080/'
-298> 2019-09-03 19:07:43.615 7f22734d8700 4 mgr[balancer] prepared 0/10 changes
-297> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_store get_store key: mgr/telemetry/last_upload
-296> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_store get_store key: mgr/telemetry/report_id
-295> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/url
-294> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config url not found
-293> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/enabled
-292> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/leaderboard
-291> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config leaderboard not found
-290> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/description
-289> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config description not found
-288> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/contact
-287> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config contact not found
-286> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/organization
-285> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config organization not found
-284> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/proxy
-283> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config proxy not found
-282> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_config get_config key: mgr/telemetry/interval
-281> 2019-09-03 19:07:43.623 7f2270993700 4 mgr get_typed_config get_typed_config interval not found
-280> 2019-09-03 19:07:43.631 7f226dccd700 0 log_channel(cluster) log [DBG] : pgmap v3: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail
-279> 2019-09-03 19:07:43.631 7f226dccd700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-278> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: get_monmap_and_config
-277> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: build_initial_monmap
-276> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: monmap:
epoch 0
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-09-03 19:07:43.648217
created 2019-09-03 19:07:43.648217
min_mon_release 0 (unknown)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.noname-a
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.noname-b
2: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.noname-c
3: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.noname-d
-275> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-274> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-273> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-272> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-271> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-270> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-269> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-268> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-267> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-266> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-265> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: init
-264> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx
-263> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx
-262> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding auth protocol: cephx
-261> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-260> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-259> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-258> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-257> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-256> 2019-09-03 19:07:43.643 7f2274cdb700 5 AuthRegistry(0x7f2274cd72c8) adding con mode: crc
-255> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-254> 2019-09-03 19:07:43.643 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-253> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient: _reopen_session rank -1
-252> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient(hunting): picked mon.noname-b con 0x564f5b19ad80 addr [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0]
-251> 2019-09-03 19:07:43.643 7f2274cdb700 10 monclient(hunting): picked mon.noname-c con 0x564f5b19b680 addr [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0]
-250> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): picked mon.noname-d con 0x564f5b19bf80 addr [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0]
-249> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection
-248> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection
-247> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): start opening mon connection
-246> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): _renew_subs
-245> 2019-09-03 19:07:43.647 7f2274cdb700 10 monclient(hunting): authenticate will time out at 2019-09-03 19:12:43.652328
-244> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): get_auth_request con 0x564f5b19ad80 auth_method 0
-243> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-242> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): _init_auth method 2
-241> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): get_auth_request con 0x564f5b19b680 auth_method 0
-240> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-239> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): _init_auth method 2
-238> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload 9
-237> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-236> 2019-09-03 19:07:43.647 7f2287baa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-235> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload 9
-234> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-233> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-232> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): get_auth_request con 0x564f5b19bf80 auth_method 0
-231> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-230> 2019-09-03 19:07:43.647 7f22883ab700 10 monclient(hunting): _init_auth method 2
-229> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient(hunting): handle_auth_done global_id 3755231 payload 418
-228> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient: _finish_hunting 0
-227> 2019-09-03 19:07:43.647 7f2288bac700 1 monclient: found mon.noname-c
-226> 2019-09-03 19:07:43.647 7f2288bac700 10 monclient: _send_mon_message to mon.noname-c at v2:[fc00:1002:c7::b3]:3300/0
-225> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_monmap mon_map magic: 0 v1
-224> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: got monmap 6 from mon.noname-c (according to old e6)
-223> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: dump:
epoch 6
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-07-30 14:53:19.672269
created 2018-12-03 09:32:28.988007
min_mon_release 14 (nautilus)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2
2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4
3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3
-222> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _finish_auth 0
-221> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-03 19:07:13.657086)
-220> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: _send_mon_message to mon.nocsupport3 at v2:[fc00:1002:c7::b3]:3300/0
-219> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_config config(25 keys) v1
-218> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: handle_monmap mon_map magic: 0 v1
-217> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: got monmap 6 from mon.nocsupport3 (according to old e6)
-216> 2019-09-03 19:07:43.651 7f226d4cc700 10 monclient: dump:
epoch 6
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-07-30 14:53:19.672269
created 2018-12-03 09:32:28.988007
min_mon_release 14 (nautilus)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2
2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4
3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3
-215> 2019-09-03 19:07:43.651 7f226bcc9700 4 set_mon_vals no callback set
-214> 2019-09-03 19:07:43.651 7f2274cdb700 5 monclient: authenticate success, global_id 3755231
-213> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/balancer/active = 1 (unrecognized option)
-212> 2019-09-03 19:07:43.651 7f2274cdb700 10 monclient: get_monmap_and_config success
-211> 2019-09-03 19:07:43.651 7f2274cdb700 10 monclient: shutdown
-210> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/balancer/mode = upmap (unrecognized option)
-209> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/password = $2b$12$X/T8Jev8/9YAPDBU.gFet.zN2WWr3tnQY71zo.qDMzN0t2vB/FRdm (unrecognized option)
-208> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option)
-207> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/dashboard/username = admin (unrecognized option)
-206> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/devicehealth/enable_monitoring = true (unrecognized option)
-205> 2019-09-03 19:07:43.651 7f226bcc9700 10 set_mon_vals mgr/telemetry/enabled = true (unrecognized option)
-204> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: build_initial_monmap
-203> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: monmap:
epoch 0
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-09-03 19:07:43.660749
created 2019-09-03 19:07:43.660749
min_mon_release 0 (unknown)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.noname-a
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.noname-b
2: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.noname-c
3: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.noname-d
-202> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-201> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-200> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding auth protocol: cephx
-199> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-198> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-197> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-196> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-195> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-194> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b0faa38) adding con mode: crc
-193> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-192> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: starting msgr at
-191> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: starting objecter
-190> 2019-09-03 19:07:43.655 7f2274cdb700 5 asok(0x564f570ba000) register_command objecter_requests hook 0x564f5d373e20 EEXIST
-189> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: setting wanted keys
-188> 2019-09-03 19:07:43.655 7f2274cdb700 1 librados: calling monclient init
-187> 2019-09-03 19:07:43.655 7f2274cdb700 10 monclient: init
-186> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx
-185> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx
-184> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding auth protocol: cephx
-183> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-182> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-181> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-180> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-179> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-178> 2019-09-03 19:07:43.655 7f2274cdb700 5 AuthRegistry(0x564f5b17f078) adding con mode: crc
-177> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-176> 2019-09-03 19:07:43.655 7f2274cdb700 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-nocsupport4/keyring
-175> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient: _reopen_session rank -1
-174> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-a con 0x564f5b19b680 addr [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0]
-173> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-c con 0x564f5b19ad80 addr [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0]
-172> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): picked mon.noname-b con 0x564f5b19c880 addr [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0]
-171> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection
-170> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection
-169> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): start opening mon connection
-168> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): _renew_subs
-167> 2019-09-03 19:07:43.659 7f2274cdb700 10 monclient(hunting): authenticate will time out at 2019-09-03 19:12:43.663758
-166> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): get_auth_request con 0x564f5b19ad80 auth_method 0
-165> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-164> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): _init_auth method 2
-163> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): get_auth_request con 0x564f5b19b680 auth_method 0
-162> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-161> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): _init_auth method 2
-160> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): get_auth_request con 0x564f5b19c880 auth_method 0
-159> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1]
-158> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): _init_auth method 2
-157> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload 9
-156> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-155> 2019-09-03 19:07:43.659 7f2287baa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-154> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more payload 9
-153> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-152> 2019-09-03 19:07:43.659 7f22883ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-151> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload 9
-150> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-149> 2019-09-03 19:07:43.659 7f2288bac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-148> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient(hunting): handle_auth_done global_id 3755053 payload 979
-147> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient: _finish_hunting 0
-146> 2019-09-03 19:07:43.663 7f22883ab700 1 monclient: found mon.noname-b
-145> 2019-09-03 19:07:43.663 7f22883ab700 10 monclient: _send_mon_message to mon.noname-b at v2:[fc00:1002:c7::b2]:3300/0
-144> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_monmap mon_map magic: 0 v1
-143> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: got monmap 6 from mon.noname-b (according to old e6)
-142> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: dump:
epoch 6
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-07-30 14:53:19.672269
created 2018-12-03 09:32:28.988007
min_mon_release 14 (nautilus)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2
2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4
3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3
-141> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _finish_auth 0
-140> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-09-03 19:07:13.674538)
-139> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-138> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_config config(25 keys) v1
-137> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: handle_monmap mon_map magic: 0 v1
-136> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: got monmap 6 from mon.nocsupport2 (according to old e6)
-135> 2019-09-03 19:07:43.667 7f226cccb700 10 monclient: dump:
epoch 6
fsid 4067126d-01cb-40af-824a-881c130140f8
last_changed 2019-07-30 14:53:19.672269
created 2018-12-03 09:32:28.988007
min_mon_release 14 (nautilus)
0: [v2:[fc00:1002:c7::b1]:3300/0,v1:[fc00:1002:c7::b1]:6789/0] mon.nocsupport1
1: [v2:[fc00:1002:c7::b2]:3300/0,v1:[fc00:1002:c7::b2]:6789/0] mon.nocsupport2
2: [v2:[fc00:1002:c7::b4]:3300/0,v1:[fc00:1002:c7::b4]:6789/0] mon.nocsupport4
3: [v2:[fc00:1002:c7::b3]:3300/0,v1:[fc00:1002:c7::b3]:6789/0] mon.nocsupport3
-134> 2019-09-03 19:07:43.667 7f226b4c8700 4 set_mon_vals no callback set
-133> 2019-09-03 19:07:43.667 7f2274cdb700 5 monclient: authenticate success, global_id 3755053
-132> 2019-09-03 19:07:43.667 7f2274cdb700 10 monclient: _renew_subs
-131> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/balancer/active = 1 (unrecognized option)
-130> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/balancer/mode = upmap (unrecognized option)
-129> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/password = $2b$12$X/T8Jev8/9YAPDBU.gFet.zN2WWr3tnQY71zo.qDMzN0t2vB/FRdm (unrecognized option)
-128> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option)
-127> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/dashboard/username = admin (unrecognized option)
-126> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/devicehealth/enable_monitoring = true (unrecognized option)
-125> 2019-09-03 19:07:43.667 7f226b4c8700 10 set_mon_vals mgr/telemetry/enabled = true (unrecognized option)
-124> 2019-09-03 19:07:43.667 7f2274cdb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-123> 2019-09-03 19:07:43.671 7f2274cdb700 10 monclient: _renew_subs
-122> 2019-09-03 19:07:43.671 7f2274cdb700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-121> 2019-09-03 19:07:43.671 7f2274cdb700 1 librados: init done
-120> 2019-09-03 19:07:43.671 7f226cccb700 10 monclient: _finish_auth 0
-119> 2019-09-03 19:07:43.671 7f226cccb700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-09-03 19:07:13.677838)
-118> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc handle_mgr_map Got map version 1020
-117> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-116> 2019-09-03 19:07:43.675 7f226cccb700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-115> 2019-09-03 19:07:43.675 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0
-114> 2019-09-03 19:07:43.675 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94900
-113> 2019-09-03 19:07:43.675 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94900 client,nocsupport4
-112> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5
-111> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc handle_mgr_configure updated stats threshold: 5
-110> 2019-09-03 19:07:43.679 7f22883ab700 10 monclient: get_auth_request con 0x564f5b1ba480 auth_method 0
-109> 2019-09-03 19:07:43.679 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94900 client,nocsupport4
-108> 2019-09-03 19:07:43.679 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4
-107> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180
-106> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184
-105> 2019-09-03 19:07:43.679 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:44.679428
-104> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-103> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-102> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch active mgrdigest v1
-101> 2019-09-03 19:07:44.279 7f22833a1700 4 mgr ms_dispatch mgrdigest v1
-100> 2019-09-03 19:07:44.679 7f226acc7700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-99> 2019-09-03 19:07:44.679 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0
-98> 2019-09-03 19:07:44.679 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94d80
-97> 2019-09-03 19:07:44.679 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94d80 client,nocsupport4
-96> 2019-09-03 19:07:44.687 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5
-95> 2019-09-03 19:07:44.687 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94d80 client,nocsupport4
-94> 2019-09-03 19:07:44.687 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4
-93> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180
-92> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184
-91> 2019-09-03 19:07:44.695 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:45.683439
-90> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch active log(2 entries from seq 2385 at 2019-09-03 19:07:44.056172) v1
-89> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch log(2 entries from seq 2385 at 2019-09-03 19:07:44.056172) v1
-88> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-87> 2019-09-03 19:07:44.747 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-86> 2019-09-03 19:07:44.991 7f2273cd9700 4 mgr[dashboard] Initializing controller: Auth -> /api/auth
-85> 2019-09-03 19:07:44.995 7f2273cd9700 4 mgr[dashboard] Initializing controller: CephFS -> /api/cephfs
-84> 2019-09-03 19:07:44.995 7f2273cd9700 4 mgr[dashboard] Initializing controller: ClusterConfiguration -> /api/cluster_conf
-83> 2019-09-03 19:07:44.999 7f2273cd9700 4 mgr[dashboard] Initializing controller: Docs -> /docs
-82> 2019-09-03 19:07:44.999 7f2273cd9700 4 mgr[dashboard] Initializing controller: ErasureCodeProfile -> /api/erasure_code_profile
-81> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Grafana -> /api/grafana
-80> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Health -> /api/health
-79> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Host -> /api/host
-78> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: IscsiUi -> /ui-api/iscsi
-77> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: Iscsi -> /api/iscsi
-76> 2019-09-03 19:07:45.003 7f2273cd9700 4 mgr[dashboard] Initializing controller: IscsiTarget -> /api/iscsi/target
-75> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Logging -> /ui-api/logging
-74> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Logs -> /api/logs
-73> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: MgrModules -> /api/mgr/module
-72> 2019-09-03 19:07:45.007 7f2273cd9700 4 mgr[dashboard] Initializing controller: Monitor -> /api/monitor
-71> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGanesha -> /api/nfs-ganesha
-70> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaExports -> /api/nfs-ganesha/export
-69> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaService -> /api/nfs-ganesha/daemon
-68> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: NFSGaneshaUi -> /ui-api/nfs-ganesha
-67> 2019-09-03 19:07:45.011 7f2273cd9700 4 mgr[dashboard] Initializing controller: Osd -> /api/osd
-66> 2019-09-03 19:07:45.015 7f2273cd9700 4 mgr[dashboard] Initializing controller: OsdFlagsController -> /api/osd/flags
-65> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: MdsPerfCounter -> /api/perf_counters/mds
-64> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: MonPerfCounter -> /api/perf_counters/mon
-63> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: OsdPerfCounter -> /api/perf_counters/osd
-62> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwPerfCounter -> /api/perf_counters/rgw
-61> 2019-09-03 19:07:45.019 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirrorPerfCounter -> /api/perf_counters/rbd-mirror
-60> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: MgrPerfCounter -> /api/perf_counters/mgr
-59> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: TcmuRunnerPerfCounter -> /api/perf_counters/tcmu-runner
-58> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: PerfCounters -> /api/perf_counters
-57> 2019-09-03 19:07:45.023 7f2273cd9700 4 mgr[dashboard] Initializing controller: Pool -> /api/pool
-56> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: PrometheusReceiver -> /api/prometheus_receiver
-55> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: Prometheus -> /api/prometheus
-54> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: PrometheusNotifications -> /api/prometheus/notifications
-53> 2019-09-03 19:07:45.027 7f2273cd9700 4 mgr[dashboard] Initializing controller: Rbd -> /api/block/image
-52> 2019-09-03 19:07:45.031 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdSnapshot -> /api/block/image/{pool_name}/{image_name}/snap
-51> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdTrash -> /api/block/image/trash
-50> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringSummary -> /api/block/mirroring/summary
-49> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringPoolMode -> /api/block/mirroring/pool
-48> 2019-09-03 19:07:45.035 7f2273cd9700 4 mgr[dashboard] Initializing controller: RbdMirroringPoolPeer -> /api/block/mirroring/pool/{pool_name}/peer
-47> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: Rgw -> /api/rgw
-46> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwDaemon -> /api/rgw/daemon
-45> 2019-09-03 19:07:45.039 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwBucket -> /api/rgw/bucket
-44> 2019-09-03 19:07:45.043 7f2273cd9700 4 mgr[dashboard] Initializing controller: RgwUser -> /api/rgw/user
-43> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Role -> /api/role
-42> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Scope -> /ui-api/scope
-41> 2019-09-03 19:07:45.047 7f2273cd9700 4 mgr[dashboard] Initializing controller: Saml2 -> /auth/saml2
-40> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Settings -> /api/settings
-39> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Summary -> /api/summary
-38> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: Task -> /api/task
-37> 2019-09-03 19:07:45.051 7f2273cd9700 4 mgr[dashboard] Initializing controller: User -> /api/user
-36> 2019-09-03 19:07:45.055 7f2273cd9700 4 mgr[dashboard] Initializing controller: FeatureTogglesEndpoint -> /api/feature_toggles
-35> 2019-09-03 19:07:45.139 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Bus STARTING
-34> 2019-09-03 19:07:45.267 7f227fb9a700 4 mgr send_beacon active
-33> 2019-09-03 19:07:45.271 7f227fb9a700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-32> 2019-09-03 19:07:45.291 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Started monitor thread '_TimeoutMonitor'.
-31> 2019-09-03 19:07:45.391 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Serving on http://:::8080
-30> 2019-09-03 19:07:45.391 7f2273cd9700 4 mgr[dashboard] [03/Sep/2019:19:07:45] ENGINE Bus STARTED
-29> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr[dashboard] Engine started.
-28> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr get_config get_config key: mgr/dashboard/GRAFANA_UPDATE_DASHBOARDS
-27> 2019-09-03 19:07:45.395 7f2273cd9700 4 mgr get_typed_config get_typed_config GRAFANA_UPDATE_DASHBOARDS not found
-26> 2019-09-03 19:07:45.635 7f226dccd700 0 log_channel(cluster) log [DBG] : pgmap v4: 1361 pgs: 2 active+clean+scrubbing+deep+repair, 1359 active+clean; 12 TiB data, 34 TiB used, 26 TiB / 61 TiB avail
-25> 2019-09-03 19:07:45.635 7f226dccd700 10 monclient: _send_mon_message to mon.nocsupport2 at v2:[fc00:1002:c7::b2]:3300/0
-24> 2019-09-03 19:07:45.679 7f226acc7700 4 mgrc reconnect Starting new session with [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-23> 2019-09-03 19:07:45.679 7f2287baa700 10 monclient: get_auth_request con 0x564f5b19d180 auth_method 0
-22> 2019-09-03 19:07:45.679 7f2288bac700 10 monclient: handle_auth_request added challenge on 0x564f5bf94900
-21> 2019-09-03 19:07:45.679 7f226eccf700 4 mgr.server handle_open from 0x564f5bf94900 client,nocsupport4
-20> 2019-09-03 19:07:45.679 7f226cccb700 4 mgrc handle_mgr_configure stats_period=5
-19> 2019-09-03 19:07:45.683 7f226eccf700 4 mgr.server handle_report from 0x564f5bf94900 client,nocsupport4
-18> 2019-09-03 19:07:45.683 7f226eccf700 4 mgr.server handle_report rejecting report from non-daemon client nocsupport4
-17> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc ms_handle_reset ms_handle_reset con 0x564f5b19d180
-16> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc reconnect Terminating session with v2:[fc00:1002:c7::b4]:6802/2184
-15> 2019-09-03 19:07:45.683 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:46.683821
-14> 2019-09-03 19:07:45.987 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-13> 2019-09-03 19:07:45.987 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-12> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch active mgrmap(e 1021) v1
-11> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr handle_mgr_map received map epoch 1021
-10> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr handle_mgr_map active in map: 1 active is 3774110
-9> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch mgrmap(e 1021) v1
-8> 2019-09-03 19:07:46.231 7f22833a1700 4 mgrc handle_mgr_map Got map version 1021
-7> 2019-09-03 19:07:46.231 7f22833a1700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-6> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc handle_mgr_map Got map version 1021
-5> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc handle_mgr_map Active mgr is now [v2:[fc00:1002:c7::b4]:6802/2184,v1:[fc00:1002:c7::b4]:6803/2184]
-4> 2019-09-03 19:07:46.231 7f226cccb700 4 mgrc reconnect waiting to retry connect until 2019-09-03 19:07:46.683821
-3> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch active service_map(e16 1 svc) v1
-2> 2019-09-03 19:07:46.231 7f22833a1700 4 mgr ms_dispatch service_map(e16 1 svc) v1
-1> 2019-09-03 19:07:46.263 7f2288bac700 10 monclient: get_auth_request con 0x564f5b1bb680 auth_method 0
0> 2019-09-03 19:07:46.275 7f2274cdb700 -1 *** Caught signal (Segmentation fault) **
in thread 7f2274cdb700 thread_name:devicehealth
ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
1: (()+0x14840) [0x7f2289a8d840]
2: (rados_write_op_omap_set()+0x138) [0x7f227df04368]
3: (()+0xc2a85) [0x7f227e0bba85]
4: (_PyMethodDef_RawFastCallDict()+0x1a2) [0x7f2289f6f4d2]
5: (()+0x25828b) [0x7f2289f6a28b]
6: (()+0x44586) [0x7f227e03d586]
7: (()+0x7f84e) [0x7f227e07884e]
8: (_PyObject_FastCallKeywords()+0xd7) [0x7f2289f70037]
9: (()+0x6dfc3) [0x7f2289d7ffc3]
10: (_PyEval_EvalFrameDefault()+0x68a2) [0x7f2289d86c42]
11: (()+0x7794b) [0x7f2289d8994b]
12: (()+0x6e015) [0x7f2289d80015]
13: (_PyEval_EvalFrameDefault()+0x242d) [0x7f2289d827cd]
14: (()+0x7794b) [0x7f2289d8994b]
15: (()+0x6e015) [0x7f2289d80015]
16: (_PyEval_EvalFrameDefault()+0x242d) [0x7f2289d827cd]
17: (()+0x7794b) [0x7f2289d8994b]
18: (_PyFunction_FastCallDict()+0x2c8) [0x7f2289f6fae8]
19: (_PyObject_Call_Prepend()+0xcd) [0x7f2289f713ad]
20: (_PyObject_FastCallDict()+0x8a) [0x7f2289f7021a]
21: (PyObject_CallMethod()+0xfe) [0x7f2289f70f7e]
22: (PyModuleRunner::serve()+0x66) [0x564f55645b86]
23: (PyModuleRunner::PyModuleRunnerThread::entry()+0x1d4) [0x564f556462e4]
24: (()+0x975a) [0x7f2289a8275a]
25: (clone()+0x43) [0x7f228964b9f3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/lib/ceph/crash/2019-09-04_00:07:46.281830Z_7128b0c3-651f-4841-82bd-fc59d45e57aa/log
--- end dump of recent events ---
Segmentation fault (core dumped)
root@nocsupport4:/tmp# ls core
core
root@nocsupport4:/tmp# ls -l core
-rw------- 1 ceph ceph 873644032 Sep 3 19:07 core
root@nocsupport4:/tmp#
(3-3/3)