Ceph » RADOS

/a/yuriw-2022-03-04_00:56:58-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6718934

teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rgw.sh) on smithi201 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c8f79f870e0d6a996c92d420e6256d312bac1c7c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'
2022-03-04T06:35:13.412 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds
2022-03-04T06:35:13.420 INFO:tasks.thrashosds:joining thrashosds
2022-03-04T06:35:13.420 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet
2022-03-04T06:35:13.447 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet
2022-03-04T06:35:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 22
2022-03-04T06:35:13.449 INFO:teuthology.orchestra.run.smithi201.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
2022-03-04T06:35:13.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight
2022-03-04T06:35:13.568 INFO:teuthology.orchestra.run.smithi204.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2022-03-04T06:35:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 22
2022-03-04T06:35:13.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops
2022-03-04T06:35:13.679 INFO:teuthology.orchestra.run.smithi204.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2022-03-04T06:35:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 22
2022-03-04T06:35:13.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops
2022-03-04T06:35:13.790 INFO:teuthology.orchestra.run.smithi204.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2022-03-04T06:35:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 22
2022-03-04T06:36:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 124
2022-03-04T06:36:50.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring
2022-03-04T06:36:52.447 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet
2022-03-04T06:36:52.448 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet
2022-03-04T06:36:52.448 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up
2022-03-04T06:36:52.449 DEBUG:teuthology.orchestra.run.smithi201:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
2022-03-04T06:38:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 124
2022-03-04T06:38:52.499 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/run_tasks.py", line 176, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/thrashosds.py", line 215, in task
    cluster_manager.wait_for_all_osds_up()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 2759, in wait_for_all_osds_up
    while not self.are_all_osds_up():
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 2749, in are_all_osds_up
    x = self.get_osd_dump()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 2522, in get_osd_dump
    return self.get_osd_dump_json()['osds']
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 2514, in get_osd_dump_json
    out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd
    return self.run_cluster_cmd(**kwargs).stdout.getvalue()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c8f79f870e0d6a996c92d420e6256d312bac1c7c/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd
    return self.controller.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label