Ceph » RADOS

2020-09-14T08:52:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:2020-09-14T08:52:47.357+0000 7fe2917fa700  1 -- 172.21.15.58:0/2033301835 <== mon.2 v2:172.21.15.58:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=-22 MDS named 'a' does not exist, or is not up v10) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe2a0077950 con 0x7fe2a4067eb0
2020-09-14T08:52:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:2020-09-14T08:52:47.356+0000 7fead391c700  1 -- 172.21.15.58:0/3567309816 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7feacc044b60 con 0x7feacc09ce80
2020-09-14T08:52:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:2020-09-14T08:52:47.357+0000 7feabd7fa700  1 -- 172.21.15.58:0/3567309816 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=-22 MDS named 'b' does not exist, or is not up v10) v1 ==== 118+0+0 (secure 0 0 0) 0x7feac0077c30 con 0x7feacc09ce80
2020-09-14T08:52:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:Error EINVAL: MDS named 'a' does not exist, or is not up
2020-09-14T08:52:47.360 INFO:teuthology.orchestra.run.smithi058.stderr:Error EINVAL: MDS named 'b' does not exist, or is not up
2020-09-14T08:52:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 22
2020-09-14T08:52:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 22
2020-09-14T08:52:47.364 INFO:tasks.cephfs_test_runner:test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) ... ERROR
2020-09-14T08:52:47.364 INFO:tasks.cephfs_test_runner:
2020-09-14T08:52:47.364 INFO:tasks.cephfs_test_runner:======================================================================
2020-09-14T08:52:47.365 INFO:tasks.cephfs_test_runner:ERROR: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress)
2020-09-14T08:52:47.365 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-09-14T08:52:47.365 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-09-14T08:52:47.365 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/mgr/test_progress.py", line 79, in setUp
2020-09-14T08:52:47.366 INFO:tasks.cephfs_test_runner:    self.mds_cluster.mds_fail()
2020-09-14T08:52:47.366 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/filesystem.py", line 309, in mds_fail
2020-09-14T08:52:47.366 INFO:tasks.cephfs_test_runner:    self._one_or_all(mds_id, lambda id_: self.mon_manager.raw_cluster_cmd("mds", "fail", id_))
2020-09-14T08:52:47.367 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/filesystem.py", line 278, in _one_or_all
2020-09-14T08:52:47.367 INFO:tasks.cephfs_test_runner:    p.spawn(cb, mds_id)
2020-09-14T08:52:47.367 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__
2020-09-14T08:52:47.367 INFO:tasks.cephfs_test_runner:    for result in self:
2020-09-14T08:52:47.367 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__
2020-09-14T08:52:47.368 INFO:tasks.cephfs_test_runner:    resurrect_traceback(result)
2020-09-14T08:52:47.368 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback
2020-09-14T08:52:47.368 INFO:tasks.cephfs_test_runner:    raise exc.exc_info[1]
2020-09-14T08:52:47.368 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback
2020-09-14T08:52:47.368 INFO:tasks.cephfs_test_runner:    return func(*args, **kwargs)
2020-09-14T08:52:47.369 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/cephfs/filesystem.py", line 309, in <lambda>
2020-09-14T08:52:47.369 INFO:tasks.cephfs_test_runner:    self._one_or_all(mds_id, lambda id_: self.mon_manager.raw_cluster_cmd("mds", "fail", id_))
2020-09-14T08:52:47.369 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1354, in raw_cluster_cmd
2020-09-14T08:52:47.369 INFO:tasks.cephfs_test_runner:    'stdout': StringIO()}).stdout.getvalue()
2020-09-14T08:52:47.369 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1347, in run_cluster_cmd
2020-09-14T08:52:47.370 INFO:tasks.cephfs_test_runner:    return self.controller.run(**kwargs)
2020-09-14T08:52:47.371 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-09-14T08:52:47.371 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-09-14T08:52:47.371 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
2020-09-14T08:52:47.371 INFO:tasks.cephfs_test_runner:    r.wait()
2020-09-14T08:52:47.372 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
2020-09-14T08:52:47.372 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-09-14T08:52:47.372 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
2020-09-14T08:52:47.372 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-09-14T08:52:47.372 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi058 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mds fail b'

rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/progress}

/a/teuthology-2020-09-11_21:54:16-rados-master-distro-basic-smithi/5426411
/a/teuthology-2020-09-14_07:01:01-rados-master-distro-basic-smithi/5433980