extract.log - bluestore - Ceph

Bug #45335 » extract.log

Sebastian Wagner, 04/29/2020 03:59 PM

 -04-28T22:15:25.775 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.842415+0000 mgr.x (mgr.34535) 47 : cephadm [INF] Upgrade: Target is quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537 with id 9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9
 -04-28T22:15:25.775 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.843492+0000 mgr.x (mgr.34535) 48 : cephadm [INF] Upgrade: Checking mgr daemons...
 -04-28T22:15:25.775 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.848251+0000 mgr.x (mgr.34535) 49 : cephadm [INF] Upgrade: All mgr daemons are up to date.
 -04-28T22:15:25.775 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.848483+0000 mgr.x (mgr.34535) 50 : cephadm [INF] Upgrade: Checking mon daemons...
 -04-28T22:15:25.776 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.849302+0000 mgr.x (mgr.34535) 51 : cephadm [INF] Upgrade: Setting container_image for all mon...
 -04-28T22:15:25.776 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.868867+0000 mgr.x (mgr.34535) 52 : cephadm [INF] Upgrade: All mon daemons are up to date.
 -04-28T22:15:25.777 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.869043+0000 mgr.x (mgr.34535) 53 : cephadm [INF] Upgrade: Checking crash daemons...
 -04-28T22:15:25.777 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.869744+0000 mgr.x (mgr.34535) 54 : cephadm [INF] Upgrade: Setting container_image for all crash...
 -04-28T22:15:25.777 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.870444+0000 mgr.x (mgr.34535) 55 : cephadm [INF] Upgrade: All crash daemons are up to date.
 -04-28T22:15:25.778 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cephadm 2020-04-28T22:15:23.870641+0000 mgr.x (mgr.34535) 56 : cephadm [INF] Upgrade: Checking osd daemons...
 -04-28T22:15:25.778 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:25 smithi151 bash[10946]: cluster 2020-04-28T22:15:24.333492+0000 mon.a (mon.0) 109 : cluster [DBG] mgrmap e25: x(active, since 41s), standbys: y
 -04-28T22:15:26.521 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: cluster 2020-04-28T22:15:25.061515+0000 mgr.x (mgr.34535) 57 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:26.522 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.422931+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.522 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.423236+0000 mgr.x (mgr.34535) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.522 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: cephadm 2020-04-28T22:15:25.424056+0000 mgr.x (mgr.34535) 59 : cephadm [INF] Upgrade: It is safe to stop osd.0
 -04-28T22:15:26.523 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: cephadm 2020-04-28T22:15:25.424365+0000 mgr.x (mgr.34535) 60 : cephadm [INF] Upgrade: Redeploying osd.0
 -04-28T22:15:26.523 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.424676+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]: dispatch
 -04-28T22:15:26.523 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.428887+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]': finished
 -04-28T22:15:26.523 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.429664+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch
 -04-28T22:15:26.524 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.430373+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
 -04-28T22:15:26.524 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: cephadm 2020-04-28T22:15:25.431342+0000 mgr.x (mgr.34535) 61 : cephadm [INF] Deploying daemon osd.0 on smithi151
 -04-28T22:15:26.524 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[10946]: audit 2020-04-28T22:15:25.431836+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch
 -04-28T22:15:26.524 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: cluster 2020-04-28T22:15:25.061515+0000 mgr.x (mgr.34535) 57 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:26.525 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.422931+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.525 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.423236+0000 mgr.x (mgr.34535) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.525 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: cephadm 2020-04-28T22:15:25.424056+0000 mgr.x (mgr.34535) 59 : cephadm [INF] Upgrade: It is safe to stop osd.0
 -04-28T22:15:26.525 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: cephadm 2020-04-28T22:15:25.424365+0000 mgr.x (mgr.34535) 60 : cephadm [INF] Upgrade: Redeploying osd.0
 -04-28T22:15:26.526 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.424676+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]: dispatch
 -04-28T22:15:26.526 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.428887+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]': finished
 -04-28T22:15:26.526 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.429664+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch
 -04-28T22:15:26.527 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.430373+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
 -04-28T22:15:26.527 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: cephadm 2020-04-28T22:15:25.431342+0000 mgr.x (mgr.34535) 61 : cephadm [INF] Deploying daemon osd.0 on smithi151
 -04-28T22:15:26.527 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[11812]: audit 2020-04-28T22:15:25.431836+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch
 -04-28T22:15:26.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: cluster 2020-04-28T22:15:25.061515+0000 mgr.x (mgr.34535) 57 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:26.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.422931+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.423236+0000 mgr.x (mgr.34535) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"]}]: dispatch
 -04-28T22:15:26.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: cephadm 2020-04-28T22:15:25.424056+0000 mgr.x (mgr.34535) 59 : cephadm [INF] Upgrade: It is safe to stop osd.0
 -04-28T22:15:26.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: cephadm 2020-04-28T22:15:25.424365+0000 mgr.x (mgr.34535) 60 : cephadm [INF] Upgrade: Redeploying osd.0
 -04-28T22:15:26.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.424676+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]: dispatch
 -04-28T22:15:26.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.428887+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix": "config set", "name": "container_image", "value": "quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537", "who": "osd.0"}]': finished
 -04-28T22:15:26.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.429664+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch
 -04-28T22:15:26.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.430373+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
 -04-28T22:15:26.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: cephadm 2020-04-28T22:15:25.431342+0000 mgr.x (mgr.34535) 61 : cephadm [INF] Deploying daemon osd.0 on smithi151
 -04-28T22:15:26.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:26 smithi156 bash[4373]: audit 2020-04-28T22:15:25.431836+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch
 -04-28T22:15:26.991 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:26 smithi151 systemd[1]: Stopping Ceph osd.0 for 8e078f14-899c-11ea-a068-001a4aab830c...
 -04-28T22:15:26.992 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[28166]: debug 2020-04-28T22:15:26.838+0000 7fb6a45f8700 -1 received  signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
 -04-28T22:15:26.992 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[28166]: debug 2020-04-28T22:15:26.838+0000 7fb6a45f8700 -1 osd.0 51 *** Got signal Terminated ***
 -04-28T22:15:26.992 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:26 smithi151 bash[28166]: debug 2020-04-28T22:15:26.838+0000 7fb6a45f8700 -1 osd.0 51 *** Immediate shutdown (osd_fast_shutdown=true) ***
 -04-28T22:15:27.271 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:26 smithi151 podman[13417]: 2020-04-28 22:15:26.989657914 +0000 UTC m=+0.182639933 container died 816cfebb822abfc1a17c63b3c8dbf4d82a23b1f976117dbc325a433a615cc931 (image=docker.io/ceph/ceph:v15.2.0, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:27.272 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13417]: 2020-04-28 22:15:27.020897016 +0000 UTC m=+0.213879019 container stop 816cfebb822abfc1a17c63b3c8dbf4d82a23b1f976117dbc325a433a615cc931 (image=docker.io/ceph/ceph:v15.2.0, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:27.272 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13417]: 816cfebb822abfc1a17c63b3c8dbf4d82a23b1f976117dbc325a433a615cc931
 -04-28T22:15:27.647 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.875889+0000 mon.a (mon.0) 116 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.647 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.875926+0000 mon.a (mon.0) 117 : cluster [INF] osd.0 failed (root=default,host=smithi151) (connection refused reported by osd.2)
 -04-28T22:15:27.648 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876132+0000 mon.a (mon.0) 118 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.648 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876254+0000 mon.a (mon.0) 119 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.648 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876427+0000 mon.a (mon.0) 120 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.648 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876630+0000 mon.a (mon.0) 121 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.649 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876804+0000 mon.a (mon.0) 122 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.649 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.876953+0000 mon.a (mon.0) 123 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.649 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.877082+0000 mon.a (mon.0) 124 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.649 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.877205+0000 mon.a (mon.0) 125 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.650 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.877341+0000 mon.a (mon.0) 126 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.650 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:26.877492+0000 mon.a (mon.0) 127 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.650 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.074089+0000 mon.a (mon.0) 128 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.650 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.074241+0000 mon.a (mon.0) 129 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.651 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.075789+0000 mon.a (mon.0) 130 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.651 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.075870+0000 mon.a (mon.0) 131 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.651 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.076508+0000 mon.a (mon.0) 132 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.651 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.076604+0000 mon.a (mon.0) 133 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.652 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.076919+0000 mon.a (mon.0) 134 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.652 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077001+0000 mon.a (mon.0) 135 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.652 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077113+0000 mon.a (mon.0) 136 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.652 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077197+0000 mon.a (mon.0) 137 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.653 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077320+0000 mon.a (mon.0) 138 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.653 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077390+0000 mon.a (mon.0) 139 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.653 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077520+0000 mon.a (mon.0) 140 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.653 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.077634+0000 mon.a (mon.0) 141 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.654 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.875889+0000 mon.a (mon.0) 116 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.654 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.875926+0000 mon.a (mon.0) 117 : cluster [INF] osd.0 failed (root=default,host=smithi151) (connection refused reported by osd.2)
 -04-28T22:15:27.654 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876132+0000 mon.a (mon.0) 118 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.655 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876254+0000 mon.a (mon.0) 119 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.655 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876427+0000 mon.a (mon.0) 120 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.655 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876630+0000 mon.a (mon.0) 121 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.655 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876804+0000 mon.a (mon.0) 122 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.655 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.876953+0000 mon.a (mon.0) 123 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.656 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.877082+0000 mon.a (mon.0) 124 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.656 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.877205+0000 mon.a (mon.0) 125 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.656 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.877341+0000 mon.a (mon.0) 126 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.656 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:26.877492+0000 mon.a (mon.0) 127 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.657 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.074089+0000 mon.a (mon.0) 128 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.657 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.074241+0000 mon.a (mon.0) 129 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.657 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.075789+0000 mon.a (mon.0) 130 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.657 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.075870+0000 mon.a (mon.0) 131 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.658 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.076508+0000 mon.a (mon.0) 132 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.658 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.076604+0000 mon.a (mon.0) 133 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.658 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.076919+0000 mon.a (mon.0) 134 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.658 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077001+0000 mon.a (mon.0) 135 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.659 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077113+0000 mon.a (mon.0) 136 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.659 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077197+0000 mon.a (mon.0) 137 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.659 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077320+0000 mon.a (mon.0) 138 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.659 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077390+0000 mon.a (mon.0) 139 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.660 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077520+0000 mon.a (mon.0) 140 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.660 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:27 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.077634+0000 mon.a (mon.0) 141 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.875889+0000 mon.a (mon.0) 116 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.875926+0000 mon.a (mon.0) 117 : cluster [INF] osd.0 failed (root=default,host=smithi151) (connection refused reported by osd.2)
 -04-28T22:15:27.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876132+0000 mon.a (mon.0) 118 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876254+0000 mon.a (mon.0) 119 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876427+0000 mon.a (mon.0) 120 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876630+0000 mon.a (mon.0) 121 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876804+0000 mon.a (mon.0) 122 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.876953+0000 mon.a (mon.0) 123 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.877082+0000 mon.a (mon.0) 124 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.877205+0000 mon.a (mon.0) 125 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.690 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.877341+0000 mon.a (mon.0) 126 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.690 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:26.877492+0000 mon.a (mon.0) 127 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.690 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.074089+0000 mon.a (mon.0) 128 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.690 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.074241+0000 mon.a (mon.0) 129 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.691 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.075789+0000 mon.a (mon.0) 130 : cluster [DBG] osd.0 reported immediately failed by osd.3
 -04-28T22:15:27.691 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.075870+0000 mon.a (mon.0) 131 : cluster [DBG] osd.0 reported immediately failed by osd.2
 -04-28T22:15:27.691 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.076508+0000 mon.a (mon.0) 132 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.691 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.076604+0000 mon.a (mon.0) 133 : cluster [DBG] osd.0 reported immediately failed by osd.1
 -04-28T22:15:27.692 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.076919+0000 mon.a (mon.0) 134 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.692 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077001+0000 mon.a (mon.0) 135 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.692 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077113+0000 mon.a (mon.0) 136 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.692 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077197+0000 mon.a (mon.0) 137 : cluster [DBG] osd.0 reported immediately failed by osd.6
 -04-28T22:15:27.692 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077320+0000 mon.a (mon.0) 138 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:27.693 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077390+0000 mon.a (mon.0) 139 : cluster [DBG] osd.0 reported immediately failed by osd.4
 -04-28T22:15:27.693 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077520+0000 mon.a (mon.0) 140 : cluster [DBG] osd.0 reported immediately failed by osd.5
 -04-28T22:15:27.693 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:27 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.077634+0000 mon.a (mon.0) 141 : cluster [DBG] osd.0 reported immediately failed by osd.7
 -04-28T22:15:28.022 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13459]: 2020-04-28 22:15:27.64575936 +0000 UTC m=+0.606987472 container create c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.022 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13459]: 2020-04-28 22:15:27.820779587 +0000 UTC m=+0.782007706 container init c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.023 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13459]: 2020-04-28 22:15:27.862377714 +0000 UTC m=+0.823605831 container start c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.023 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:27 smithi151 podman[13459]: 2020-04-28 22:15:27.862442318 +0000 UTC m=+0.823670460 container attach c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.349 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 podman[13459]: 2020-04-28 22:15:28.087744802 +0000 UTC m=+1.048972928 container died c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.605 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.062007+0000 mgr.x (mgr.34535) 62 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:28.605 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.338144+0000 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
 -04-28T22:15:28.605 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[10946]: cluster 2020-04-28T22:15:27.347350+0000 mon.a (mon.0) 143 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in
 -04-28T22:15:28.605 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 podman[13459]: 2020-04-28 22:15:28.587885902 +0000 UTC m=+1.549114039 container remove c38c141a767f34ae26c7f4f2d575e2f0e779068022585eabc02d7f9832e2536a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-deactivate)
 -04-28T22:15:28.606 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 systemd[1]: Stopped Ceph osd.0 for 8e078f14-899c-11ea-a068-001a4aab830c.
 -04-28T22:15:28.606 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.062007+0000 mgr.x (mgr.34535) 62 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:28.606 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.338144+0000 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
 -04-28T22:15:28.607 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:28 smithi151 bash[11812]: cluster 2020-04-28T22:15:27.347350+0000 mon.a (mon.0) 143 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in
 -04-28T22:15:28.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:28 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.062007+0000 mgr.x (mgr.34535) 62 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:28.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:28 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.338144+0000 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
 -04-28T22:15:28.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:28 smithi156 bash[4373]: cluster 2020-04-28T22:15:27.347350+0000 mon.a (mon.0) 143 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in
 -04-28T22:15:29.021 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 systemd[1]: Starting Ceph osd.0 for 8e078f14-899c-11ea-a068-001a4aab830c...
 -04-28T22:15:29.022 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 podman[13562]: Error: no container with name or ID ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0 found: no such container
 -04-28T22:15:29.022 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:28 smithi151 systemd[1]: Started Ceph osd.0 for 8e078f14-899c-11ea-a068-001a4aab830c.
 -04-28T22:15:29.322 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.0207602 +0000 UTC m=+0.262426894 container create edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:29.322 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.120609901 +0000 UTC m=+0.362276575 container init edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:29.323 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.162161904 +0000 UTC m=+0.403828610 container start edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:29.323 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.162247399 +0000 UTC m=+0.403914112 container attach edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:29.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:29 smithi156 bash[4373]: cluster 2020-04-28T22:15:28.355727+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in
 -04-28T22:15:29.688 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:29 smithi156 bash[4373]: audit 2020-04-28T22:15:28.771642+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]: dispatch
 -04-28T22:15:29.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:29 smithi156 bash[4373]: audit 2020-04-28T22:15:28.776223+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]': finished
 -04-28T22:15:29.689 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:29 smithi156 bash[4373]: audit 2020-04-28T22:15:28.777097+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch
 -04-28T22:15:29.773 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[10946]: cluster 2020-04-28T22:15:28.355727+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in
 -04-28T22:15:29.773 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[10946]: audit 2020-04-28T22:15:28.771642+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]: dispatch
 -04-28T22:15:29.773 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[10946]: audit 2020-04-28T22:15:28.776223+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]': finished
 -04-28T22:15:29.774 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[10946]: audit 2020-04-28T22:15:28.777097+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch
 -04-28T22:15:29.774 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[11812]: cluster 2020-04-28T22:15:28.355727+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in
 -04-28T22:15:29.774 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[11812]: audit 2020-04-28T22:15:28.771642+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]: dispatch
 -04-28T22:15:29.775 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[11812]: audit 2020-04-28T22:15:28.776223+0000 mon.a (mon.0) 146 : audit [INF] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/host.smithi151","val":"{\"daemons\": {\"mon.a\": {\"hostname\": \"smithi151\", \"container_id\": \"9f105ca3fb16\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"a\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.345966\", \"created\": \"2020-04-28T22:07:26.951484\", \"started\": \"2020-04-28T22:14:38.744481\"}, \"mgr.y\": {\"hostname\": \"smithi151\", \"container_id\": \"1368bbb8452f\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"y\", \"daemon_type\": \"mgr\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346058\", \"created\": \"2020-04-28T22:07:33.398511\", \"started\": \"2020-04-28T22:15:19.047266\"}, \"mon.c\": {\"hostname\": \"smithi151\", \"container_id\": \"805c083817f4\", \"container_image_id\": \"9c90938ad11a31c5ba9b58ed052bf347591ae047e94bca695e7a022672efd3b9\", \"container_image_name\": \"quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537\", \"daemon_id\": \"c\", \"daemon_type\": \"mon\", \"version\": \"16.0.0-1168-g5729005\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346106\", \"created\": \"2020-04-28T22:08:53.300785\", \"started\": \"2020-04-28T22:14:52.418876\"}, \"osd.0\": {\"hostname\": \"smithi151\", \"daemon_id\": \"0\", \"daemon_type\": \"osd\", \"status\": 1, \"status_desc\": \"starting\"}, \"osd.1\": {\"hostname\": \"smithi151\", \"container_id\": \"bceb25f88a38\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"1\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346175\", \"created\": \"2020-04-28T22:09:49.827082\", \"started\": \"2020-04-28T22:09:51.486018\"}, \"osd.2\": {\"hostname\": \"smithi151\", \"container_id\": \"d0fe8a5110be\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"2\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346206\", \"created\": \"2020-04-28T22:10:06.690153\", \"started\": \"2020-04-28T22:10:08.418596\"}, \"osd.3\": {\"hostname\": \"smithi151\", \"container_id\": \"a1ebf099d815\", \"container_image_id\": \"204a01f9b0b6710dd0c0af7f37ce7139c47ff0f0105d778d7104c69282dfbbf1\", \"container_image_name\": \"docker.io/ceph/ceph:v15.2.0\", \"daemon_id\": \"3\", \"daemon_type\": \"osd\", \"version\": \"15.2.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346237\", \"created\": \"2020-04-28T22:10:23.301228\", \"started\": \"2020-04-28T22:10:24.871371\"}, \"node-exporter.a\": {\"hostname\": \"smithi151\", \"container_id\": \"973f2457bbad\", \"container_image_id\": \"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87\", \"container_image_name\": \"docker.io/prom/node-exporter:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"node-exporter\", \"version\": \"0.18.1\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346267\", \"created\": \"2020-04-28T22:11:45.281592\", \"started\": \"2020-04-28T22:11:49.308108\"}, \"alertmanager.a\": {\"hostname\": \"smithi151\", \"container_id\": \"1486112ad327\", \"container_image_id\": \"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0\", \"container_image_name\": \"docker.io/prom/alertmanager:latest\", \"daemon_id\": \"a\", \"daemon_type\": \"alertmanager\", \"version\": \"0.20.0\", \"status\": 1, \"status_desc\": \"running\", \"last_refresh\": \"2020-04-28T22:15:21.346299\", \"created\": \"2020-04-28T22:12:01.002659\", \"started\": \"2020-04-28T22:14:18.353609\"}}, \"devices\": [{\"rejected_reasons\": [\"locked\", \"LVM detected\", \"Insufficient space (<5GB) on vgs\"], \"available\": false, \"path\": \"/dev/nvme0n1\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"\", \"model\": \"INTEL SSDPEDMD400G4\", \"rev\": \"\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"512\", \"rotational\": \"0\", \"nr_requests\": \"1023\", \"scheduler_mode\": \"none\", \"partitions\": {}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 400088457216.0, \"human_readable_size\": \"372.61 GB\", \"path\": \"/dev/nvme0n1\", \"locked\": 1}, \"lvs\": [{\"name\": \"lv_1\", \"osd_id\": \"3\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"79e29313-9725-4bf4-92ba-472bea242cc3\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"SyzLct-FYSf-k3HO-wmCd-6TDm-cOJA-0P7N5k\"}, {\"name\": \"lv_2\", \"osd_id\": \"2\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"06746d34-1cbb-4b30-86f1-2ab82ec60c09\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"EzbYCq-y6AL-M5f0-5l0f-306e-j1Lg-nvA1eX\"}, {\"name\": \"lv_3\", \"osd_id\": \"1\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"de2c0974-60c7-45aa-83de-9df05a2c6d4a\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"JVsVDJ-cZJ2-btKf-OxM0-GbcG-mV7l-wco9Je\"}, {\"name\": \"lv_4\", \"osd_id\": \"0\", \"cluster_name\": \"ceph\", \"type\": \"block\", \"osd_fsid\": \"b2ffa6b2-4845-4e54-b49f-12b8de78edb6\", \"cluster_fsid\": \"8e078f14-899c-11ea-a068-001a4aab830c\", \"block_uuid\": \"x8Gd38-wRZy-8Kal-HnZ6-QZVs-QG2R-jsO0eP\"}, {\"name\": \"lv_5\", \"comment\": \"not used by ceph\"}], \"human_readable_type\": \"ssd\", \"device_id\": \"INTEL SSDPEDMD400G4_CVFT623300JJ400BGN\"}, {\"rejected_reasons\": [\"locked\"], \"available\": false, \"path\": \"/dev/sda\", \"sys_api\": {\"removable\": \"0\", \"ro\": \"0\", \"vendor\": \"ATA\", \"model\": \"ST1000NM0033-9ZM\", \"rev\": \"SN06\", \"sas_address\": \"\", \"sas_device_handle\": \"\", \"support_discard\": \"0\", \"rotational\": \"1\", \"nr_requests\": \"64\", \"scheduler_mode\": \"mq-deadline\", \"partitions\": {\"sda1\": {\"start\": \"2048\", \"sectors\": \"1953522688\", \"sectorsize\": 512, \"size\": 1000203616256.0, \"human_readable_size\": \"931.51 GB\", \"holders\": []}}, \"sectors\": 0, \"sectorsize\": \"512\", \"size\": 1000204886016.0, \"human_readable_size\": \"931.51 GB\", \"path\": \"/dev/sda\", \"locked\": 1}, \"lvs\": [], \"human_readable_type\": \"hdd\", \"device_id\": \"ST1000NM0033-9ZM173_Z1W5XK16\"}], \"daemon_config_deps\": {\"mon.c\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:50.025993\"}, \"mgr.y\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:06.735032\"}, \"osd.0\": {\"deps\": [], \"last_config\": \"2020-04-28T22:15:25.429458\"}, \"osd.1\": {\"deps\": [], \"last_config\": \"2020-04-28T22:09:48.348404\"}, \"osd.2\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:05.191401\"}, \"osd.3\": {\"deps\": [], \"last_config\": \"2020-04-28T22:10:21.787505\"}, \"node-exporter.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:11:44.825212\"}, \"alertmanager.a\": {\"deps\": [\"alertmanager.a\", \"mgr.x\", \"mgr.y\"], \"last_config\": \"2020-04-28T22:14:14.630536\"}, \"mon.a\": {\"deps\": [], \"last_config\": \"2020-04-28T22:14:36.546497\"}}, \"last_device_update\": \"2020-04-28T22:10:28.396419\", \"networks\": {\"172.21.0.0/20\": [\"172.21.15.151\"]}, \"last_host_check\": \"2020-04-28T22:07:52.154037\"}"}]': finished
 -04-28T22:15:29.775 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[11812]: audit 2020-04-28T22:15:28.777097+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.34535 172.21.15.156:0/2194410762' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch
 -04-28T22:15:29.776 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
 -04-28T22:15:29.776 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
 -04-28T22:15:29.776 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block
 -04-28T22:15:29.777 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
 -04-28T22:15:29.777 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
 -04-28T22:15:29.777 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
 -04-28T22:15:29.777 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 bash[13577]: --> ceph-volume lvm activate successful for osd ID: 0
 -04-28T22:15:29.778 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.444755363 +0000 UTC m=+0.686422056 container died edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:30.180 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:29 smithi151 podman[13580]: 2020-04-28 22:15:29.902456456 +0000 UTC m=+1.144123162 container remove edc631b77207be0034ca1916a33133fc983c923c2b05a13f6b48aa7e06025d40 (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0-activate)
 -04-28T22:15:30.180 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:30 smithi151 podman[13761]: 2020-04-28 22:15:30.137040328 +0000 UTC m=+0.215911961 container create 0ba1c66cfbf52fe97b55dfc3a43281b5629c624e0482caba90b4d1019b5c9f8a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:30.505 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:30 smithi151 bash[10946]: cluster 2020-04-28T22:15:29.062506+0000 mgr.x (mgr.34535) 63 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:30.506 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:30 smithi151 bash[11812]: cluster 2020-04-28T22:15:29.062506+0000 mgr.x (mgr.34535) 63 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:30.506 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:30 smithi151 podman[13761]: 2020-04-28 22:15:30.253693628 +0000 UTC m=+0.332565244 container init 0ba1c66cfbf52fe97b55dfc3a43281b5629c624e0482caba90b4d1019b5c9f8a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:30.506 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:30 smithi151 podman[13761]: 2020-04-28 22:15:30.295389095 +0000 UTC m=+0.374260713 container start 0ba1c66cfbf52fe97b55dfc3a43281b5629c624e0482caba90b4d1019b5c9f8a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:30.506 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:30 smithi151 podman[13761]: 2020-04-28 22:15:30.295458519 +0000 UTC m=+0.374330136 container attach 0ba1c66cfbf52fe97b55dfc3a43281b5629c624e0482caba90b4d1019b5c9f8a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:30.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:30 smithi156 bash[4373]: cluster 2020-04-28T22:15:29.062506+0000 mgr.x (mgr.34535) 63 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 0 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
 -04-28T22:15:31.077 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:30 smithi151 bash[13577]: debug 2020-04-28T22:15:30.801+0000 7f47628adec0 -1 Falling back to public interface
 -04-28T22:15:31.077 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[13577]: debug 2020-04-28T22:15:31.074+0000 7f47628adec0 -1 rocksdb: verify_sharding mismatch on sharding. requested = [(L,1,0-,),(O,3,0-13,),(m,3,0-,)] stored = []
 -04-28T22:15:31.078 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[13577]: debug 2020-04-28T22:15:31.074+0000 7f47628adec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening db:
 -04-28T22:15:31.687 INFO:ceph.mon.b.smithi156.stdout:Apr 28 22:15:31 smithi156 bash[4373]: cluster 2020-04-28T22:15:31.361256+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)
 -04-28T22:15:31.731 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[10946]: cluster 2020-04-28T22:15:31.361256+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)
 -04-28T22:15:31.731 INFO:ceph.mon.c.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[11812]: cluster 2020-04-28T22:15:31.361256+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)
 -04-28T22:15:31.732 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[13577]: debug 2020-04-28T22:15:31.589+0000 7f47628adec0 -1 osd.0 0 OSD:init: unable to mount object store
 -04-28T22:15:31.732 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:31 smithi151 bash[13577]: debug 2020-04-28T22:15:31.589+0000 7f47628adec0 -1  ** ERROR: osd init failed: (5) Input/output error
 -04-28T22:15:32.021 INFO:ceph.osd.0.smithi151.stdout:Apr 28 22:15:31 smithi151 podman[13761]: 2020-04-28 22:15:31.729840599 +0000 UTC m=+1.808712241 container died 0ba1c66cfbf52fe97b55dfc3a43281b5629c624e0482caba90b4d1019b5c9f8a (image=quay.io/ceph-ci/ceph:572900540492d8a60eef974379bcb0aa3e5c4537, name=ceph-8e078f14-899c-11ea-a068-001a4aab830c-osd.0)
 -04-28T22:15:32.614 INFO:ceph.mon.a.smithi151.stdout:Apr 28 22:15:32 smithi151 bash[10946]: cluster 2020-04-28T22:15:31.063005+0000 mgr.x (mgr.34535) 64 : cluster [DBG] pgmap v29: 1 pgs: 1 active+undersized+degraded; 0 B data, 4.0 MiB used, 707 GiB / 715 GiB avail; 1/3 objects degraded (33.333%)

(1-1/1)

Project

General

Profile

Ceph » bluestore

Bug #45335 » extract.log