Project

General

Profile

Bug #45420

cephadmunit.py: teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1'

Added by Brad Hubbard almost 4 years ago. Updated over 2 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
teuthology
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/bhubbard-2020-05-01_23:30:27-rados:thrash-old-clients-master-distro-basic-smithi/5014156

2020-05-02T23:16:25.929 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:25 smithi094 systemd[1]: Stopping Ceph osd.1 for d8648236-8cc8-11ea-a068-001a4aab830c...
2020-05-02T23:16:26.039 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 bash[17770]: debug 2020-05-02T23:16:26.051+0000 7fa396728700 -1 received  signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2020-05-02T23:16:26.039 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 bash[17770]: debug 2020-05-02T23:16:26.051+0000 7fa396728700 -1 osd.1 455 *** Got signal Terminated ***
2020-05-02T23:16:26.204 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 bash[17770]: debug 2020-05-02T23:16:26.051+0000 7fa396728700 -1 osd.1 455 *** Immediate shutdown (osd_fast_shutdown=true) ***
2020-05-02T23:16:26.204 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 podman[143190]: 2020-05-02 23:16:26.216427372 +0000 UTC m=+0.257080594 container died 900d37e94a3796ff360f9007f1d215afb583884e61cec3e809cbc59fde518bde (image=quay.io/ceph-ci/ceph:beaa4b04bc57ed43e98602e493e8a787a014b4e6, name=ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1)
2020-05-02T23:16:26.247 INFO:teuthology.orchestra.run.smithi094.stderr:Error: can only kill running containers. 900d37e94a3796ff360f9007f1d215afb583884e61cec3e809cbc59fde518bde is in state stopped: container state improper
2020-05-02T23:16:26.257 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 podman[143190]: 2020-05-02 23:16:26.257171524 +0000 UTC m=+0.297824636 container stop 900d37e94a3796ff360f9007f1d215afb583884e61cec3e809cbc59fde518bde (image=quay.io/ceph-ci/ceph:beaa4b04bc57ed43e98602e493e8a787a014b4e6, name=ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1)
2020-05-02T23:16:26.257 INFO:ceph.osd.1.smithi094.stdout:May 02 23:16:26 smithi094 podman[143190]: 900d37e94a3796ff360f9007f1d215afb583884e61cec3e809cbc59fde518bde
2020-05-02T23:16:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 125
2020-05-02T23:16:26.267 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 122, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 1095, in do_sighup
    self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True)
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 2797, in signal_osd
    self.cluster).signal(sig, silent=silent)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal
    self.remote.sh(self.kill_cmd(sig))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 247, in sh
    proc=self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 203, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1'

Related issues

Related to Orchestrator - Bug #46247: cephadm mon failure: Error: no container with name or ID ... no such container Can't reproduce

History

#1 Updated by Sebastian Wagner almost 4 years ago

  • Subject changed from teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1' to cephadmunit.py: teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 125: 'sudo docker kill -s 1 ceph-d8648236-8cc8-11ea-a068-001a4aab830c-osd.1'

#2 Updated by Deepika Upadhyay over 3 years ago

  • Related to Bug #46247: cephadm mon failure: Error: no container with name or ID ... no such container added

#3 Updated by Sebastian Wagner over 3 years ago

  • Category set to teuthology

#4 Updated by Sebastian Wagner about 3 years ago

  • Priority changed from High to Normal

#5 Updated by Sebastian Wagner over 2 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF