Bug #44526
Updated by Sage Weil about 4 years ago
<pre>
2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:Non-zero exit code 1 from /bin/podman run --rm --net=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -e NODE_NAME=smithi123 -v /var/log/ceph/7d5d8d88-6202-11ea-9a3e-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpw7u46ywy:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpquh1zk_7:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -h
2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout
2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout General usage:
2020-03-09T12:37:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout ==============
...
2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout critical, interesting, useful, noninteresting, debug
2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout List shows a table of all available stats
2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout Run <count> times (default forever),
2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout once per <interval> seconds (default 1)
2020-03-09T12:37:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stdout
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:INFO:cephadm:/usr/bin/ceph:stderr timed out
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr:Traceback (most recent call last):
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 3729, in <module>
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: r = args.func()
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 2053, in command_bootstrap
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: is_available('Dashboard', is_dashboard_available)
2020-03-09T12:37:04.321 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 575, in is_available
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: if func():
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 2051, in is_dashboard_available
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: out = cli(['-h'], timeout=timeout)
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 1924, in cli
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: ).run(timeout=timeout)
2020-03-09T12:37:04.322 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 1681, in run
2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: self.run_cmd(), desc=self.entrypoint, timeout=timeout)
2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: File "/home/ubuntu/cephtest/cephadm", line 518, in call_throws
2020-03-09T12:37:04.323 INFO:teuthology.orchestra.run.smithi123.stderr: raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-03-09T12:37:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:RuntimeError: Failed command: /bin/podman run --rm --net=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -e NODE_NAME=smithi123 -v /var/log/ceph/7d5d8d88-6202-11ea-9a3e-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpw7u46ywy:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpquh1zk_7:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph quay.io/ceph-ci/ceph:8c7e5228ba7bdea07454d0dd9c211c1b6b6dcfc8 -h
</pre>
http://pulpito.ceph.com/sage-2020-03-09_11:35:13-rados:thrash-old-clients-wip-sage-testing-2020-03-08-2121-distro-basic-smithi/
i'm able to locally reproduce ceph -h ending with 'timed out' if i set ms_inject_socket_failures = 2000.