Bug #46035
Updated by David Galloway almost 4 years ago
From http://qa-proxy.ceph.com/teuthology/yuriw-2020-06-15_23:54:55-rados-wip-yuri2-testing-2020-06-15-1645-octopus-distro-basic-smithi/5152401/teuthology.log
<pre>
2020-06-16T13:01:38.409 INFO:teuthology.orchestra.run.smithi007.stderr:INFO:cephadm:Pulling latest quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd container...
2020-06-16T13:01:38.409 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG:cephadm:Running command: /usr/bin/docker pull quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd
2020-06-16T13:01:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG:cephadm:/usr/bin/docker:stderr Error response from daemon: manifest for quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd not found: manifest unknown: manifest unknown
2020-06-16T13:01:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:INFO:cephadm:Non-zero exit code 1 from /usr/bin/docker pull quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd
2020-06-16T13:01:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:INFO:cephadm:/usr/bin/docker:stderr Error response from daemon: manifest for quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd not found: manifest unknown: manifest unknown
2020-06-16T13:01:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:Traceback (most recent call last):
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: File "/home/ubuntu/cephtest/cephadm", line 4761, in <module>
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: r = args.func()
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: File "/home/ubuntu/cephtest/cephadm", line 1162, in _default_image
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: return func()
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: File "/home/ubuntu/cephtest/cephadm", line 2366, in command_bootstrap
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: call_throws([container_path, 'pull', args.image])
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: File "/home/ubuntu/cephtest/cephadm", line 846, in call_throws
2020-06-16T13:01:42.724 INFO:teuthology.orchestra.run.smithi007.stderr: raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-06-16T13:01:42.725 INFO:teuthology.orchestra.run.smithi007.stderr:RuntimeError: Failed command: /usr/bin/docker pull quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd
2020-06-16T13:01:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG:cephadm:Releasing lock 140539053150448 on /run/cephadm/83387b56-afd1-11ea-a06d-001a4aab830c.lock
2020-06-16T13:01:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG:cephadm:Lock 140539053150448 released on /run/cephadm/83387b56-afd1-11ea-a06d-001a4aab830c.lock
2020-06-16T13:01:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-06-16T13:01:42.743 INFO:tasks.cephadm:Cleaning up testdir ceph.* files...
2020-06-16T13:01:42.744 INFO:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub
2020-06-16T13:01:42.749 INFO:teuthology.orchestra.run.smithi008:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub
2020-06-16T13:01:42.755 INFO:tasks.cephadm:Stopping all daemons...
2020-06-16T13:01:42.756 INFO:tasks.cephadm.mon.smithi007:Stopping mon.smithi007...
2020-06-16T13:01:42.756 INFO:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007
2020-06-16T13:01:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007.service: Unit ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007.service not loaded.
2020-06-16T13:01:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 5
2020-06-16T13:01:42.813 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi007"
Traceback (most recent call last):
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri2-testing-2020-06-15-1645-octopus/qa/tasks/cephadm.py", line 462, in ceph_bootstrap
ctx.daemons.get_daemon(type_, id_, cluster).stop()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/daemon/cephadmunit.py", line 149, in stop
self.remote.sh(self.stop_cmd)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 250, in sh
proc=self.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 206, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 475, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi007 with status 5: 'sudo systemctl stop ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007'
2020-06-16T13:01:42.814 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri2-testing-2020-06-15-1645-octopus/qa/tasks/cephadm.py", line 462, in ceph_bootstrap
ctx.daemons.get_daemon(type_, id_, cluster).stop()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/daemon/cephadmunit.py", line 149, in stop
self.remote.sh(self.stop_cmd)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 250, in sh
proc=self.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 206, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 475, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi007 with status 5: 'sudo systemctl stop ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007'
</pre>
<code>sudo systemctl stop ceph-83387b56-afd1-11ea-a06d-001a4aab830c@mon.smithi007</code> at first glance would lead me to believe a Ceph code change is to blame but the real error is <code>Error response from daemon: manifest for quay.io/ceph-ci/ceph:32f62798ee9a2145abdafd82203a5740d87c5dbd not found: manifest unknown: manifest unknown</code> which is kind of my responsibility.