Actions
Bug #44526
closedsporatic cephadm bootstrap failures: 'timed out'
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:Non-zero exit code 1 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -e NODE_NAME=smithi123 -v /var/log/ceph/7d5d8d88-6202-11ea-9a3e-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpw7u46ywy:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpquh1zk_7:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -h 2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout 2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout General usage: 2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout ============== ... 2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout critical, interesting, useful, noninteresting, debug 2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout List shows a table of all available stats 2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout Run <count> times (default forever), 2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout once per <interval> seconds (default 1) 2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stderr timed out 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:Traceback (most recent call last): 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 3729, in <module> 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: r = args.func() 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 2053, in command_bootstrap 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: is_available('Dashboard', is_dashboard_available) 2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 575, in is_available 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: if func(): 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 2051, in is_dashboard_available 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: out = cli(['-h'], timeout=timeout) 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 1924, in cli 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: ).run(timeout=timeout) 2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 1681, in run 2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: self.run_cmd(), desc=self.entrypoint, timeout=timeout) 2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 518, in call_throws 2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2020-03-09T12:37:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -e NODE_NAME=smithi123 -v /var/log/ceph/7d5d8d88-6202-11ea-9a3e-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpw7u46ywy:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpquh1zk_7:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -h
http://pulpito.ceph.com/sage-2020-03-09_11:35:13-rados:thrash-old-clients-wip-sage-testing-2020-03-08-2121-distro-basic-smithi/
i'm able to locally reproduce ceph -h ending with 'timed out' if i set ms_inject_socket_failures = 2000.
Actions