Ceph » Orchestrator

Run: https://pulpito.ceph.com/teuthology-2020-12-04_19:57:53-upgrade:octopus-x-master-distro-basic-smithi/
Job:5681009, 5681010
Logs:/a/teuthology-2020-12-04_19:57:53-upgrade:octopus-x-master-distro-basic-smithi/5681010/teuthology.log

2020-12-04T20:53:16.470 INFO:teuthology.orchestra.run.smithi116:> sudo systemctl start ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116
2020-12-04T20:53:16.485 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:-- Logs begin at Sun 2020-11-29 12:19:31 UTC. --
2020-12-04T20:53:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:Job for ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116.service failed.
2020-12-04T20:53:16.505 INFO:teuthology.orchestra.run.smithi116.stderr:See "systemctl status ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116.service" and "journalctl -xe" for details.
2020-12-04T20:53:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-12-04T20:53:16.506 ERROR:tasks.mon_thrash.mon_thrasher:exception:
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_yuriw_ceph_wip-yuriw-octopus-x-parallel-master/qa/tasks/mon_thrash.py", line 232, in do_thrash
    self._do_thrash()
  File "/home/teuthworker/src/github.com_yuriw_ceph_wip-yuriw-octopus-x-parallel-master/qa/tasks/mon_thrash.py", line 312, in _do_thrash
    self.revive_mon(mon)
  File "/home/teuthworker/src/github.com_yuriw_ceph_wip-yuriw-octopus-x-parallel-master/qa/tasks/mon_thrash.py", line 215, in revive_mon
    self.manager.revive_mon(mon)
  File "/home/teuthworker/src/github.com_yuriw_ceph_wip-yuriw-octopus-x-parallel-master/qa/tasks/ceph_manager.py", line 2834, in revive_mon
    self.ctx.daemons.get_daemon('mon', mon, self.cluster).restart()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/cephadmunit.py", line 95, in restart
    self.remote.sh(self.start_cmd)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 259, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'sudo systemctl start ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116'
2020-12-04T20:53:16.710 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Dec 04 20:53:16 smithi135 bash[24913]: audit 2020-12-04T20:53:15.430282+0000 mon.smithi151 (mon.0) 144 : audit [INF] from='client.154209 172.21.15.151:0/3316206744' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi151-47528-76", "overlaypool": "test-rados-api-smithi151-47528-87"}]': finished
2020-12-04T20:53:16.711 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Dec 04 20:53:16 smithi135 bash[24913]: cluster 2020-12-04T20:53:15.430347+0000 mon.smithi151 (mon.0) 145 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in
2020-12-04T20:53:16.711 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Dec 04 20:53:16 smithi135 bash[24913]: audit 2020-12-04T20:53:15.431189+0000 mon.smithi151 (mon.0) 146 : audit [INF] from='client.154209 172.21.15.151:0/3316206744' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi151-47528-87", "mode": "writeback"}]: dispatch
2020-12-04T20:53:16.711 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Dec 04 20:53:16 smithi135 bash[24913]: cluster 2020-12-04T20:53:16.427548+0000 mon.smithi151 (mon.0) 147 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)
2020-12-04T20:53:16.751 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Dec 04 20:53:16 smithi151 bash[122979]: audit 2020-12-04T20:53:15.430282+0000 mon.smithi151 (mon.0) 144 : audit [INF] from='client.154209 172.21.15.151:0/3316206744' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi151-47528-76", "overlaypool": "test-rados-api-smithi151-47528-87"}]': finished
2020-12-04T20:53:16.751 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Dec 04 20:53:16 smithi151 bash[122979]: cluster 2020-12-04T20:53:15.430347+0000 mon.smithi151 (mon.0) 145 : cluster [DBG] osdmap e1154: 8 total, 8 up, 8 in
2020-12-04T20:53:16.752 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Dec 04 20:53:16 smithi151 bash[122979]: audit 2020-12-04T20:53:15.431189+0000 mon.smithi151 (mon.0) 146 : audit [INF] from='client.154209 172.21.15.151:0/3316206744' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi151-47528-87", "mode": "writeback"}]: dispatch
2020-12-04T20:53:16.752 INFO:journalctl@ceph.mon.smithi151.smithi151.stdout:Dec 04 20:53:16 smithi151 bash[122979]: cluster 2020-12-04T20:53:16.427548+0000 mon.smithi151 (mon.0) 147 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)
2020-12-04T20:53:16.918 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Dec 04 20:53:16 smithi116 systemd[1]: ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116.service: Start request repeated too quickly.
2020-12-04T20:53:16.918 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Dec 04 20:53:16 smithi116 systemd[1]: ceph-a0fd247a-366d-11eb-980d-001a4aab830c@mon.smithi116.service: Failed with result 'start-limit-hit'.
2020-12-04T20:53:16.919 INFO:journalctl@ceph.mon.smithi116.smithi116.stdout:Dec 04 20:53:16 smithi116 systemd[1]: Failed to start Ceph mon.smithi116 for a0fd247a-366d-11eb-980d-001a4aab830c.