Ceph » RADOS

/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984906

The MON log shows it came back up around 09:41:11.

2020-04-26T09:41:09.547+0000 7f3e7a667540  4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work
2020-04-26T09:41:09.551+0000 7f3e7a667540  4 rocksdb: [db/db_impl.cc:563] Shutdown complete
2020-04-26T09:41:09.551+0000 7f3e7a667540  0 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-d for mon.d
2020-04-26T09:41:11.139+0000 7f2561752540  0 ceph version 15.2.1-136-ga8c125c7d7 (a8c125c7d78f5cd973863993d258cd717ade4c99) octopus (stable), process ceph-mon, pid 12193

Around that time in the teuthology log we see.

2020-04-26T09:41:11.031 INFO:tasks.ceph.mon.d:Restarting daemon
2020-04-26T09:41:11.031 INFO:teuthology.orchestra.run.smithi101:> true
2020-04-26T09:41:11.036 INFO:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d
2020-04-26T09:41:11.082 INFO:tasks.ceph.mon.d:Started

So it is restarted around 09:41:11 but the warning is issued at 09:41:26. The monitor log shows that quorum wasn't achieved until 09:41:27.

2020-04-26T09:41:27.343+0000 7f254c186700  7 mon.d@2(peon).log v7 update_from_paxos applying incremental log 7 2020-04-26T09:41:26.428693+0000 mon.b (mon.0) 27 : cluster [INF] mon.b is new leader, mons b,a,d,c,f,e in quorum (ranks 0,1,2,3,4,5)

I don't think we can whitelist this message if we want to catch actual failures in the mons during the test.