Actions
Bug #50021
closedqa: snaptest-git-ceph failure during mon thrashing
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure, snapshots
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-03-25T09:56:11.131 INFO:tasks.workunit.client.0.smithi016.stderr:Checking out files: 2% (6/227) ^MChecking out files: 3% (7/227) ^MChecking out files: 4% (10/227) ^MChecking out files: 5% (12/227) ^MChecking out files: 6% (14/227) ^MChecking out files: 7% (16/227) ^MChecking out files: 7% (18/227) ^MChecking out files: 8% (19/227) ^MChecking out files: 8% (20/227) ^MChecking out files: 9% (21/227) ^MChecking out files: 10% (23/227) ^MChecking out files: 11% (25/227) ^MChecking out files: 12% (28/227) ^MChecking out files: 13% (30/227) ^MChecking out files: 14% (32/227) ^MChecking out files: 15% (35/227) ^MChecking out files: 15% (36/227) ^MChecking out files: 16% (37/227) ^MChecking out files: 16% (38/227) ^MChecking out files: 17% (39/227) ^MChecking out files: 18% (41/227) ^MChecking out files: 19% (44/227) ^MChecking out files: 19% (45/227) ^MChecking out files: 20% (46/227) ^MChecking out files: 21% (48/227) ^MChecking out files: 22% (50/227) ^MChecking out files: 23% (53/227) ^MChecking out files: 24% (55/227) ^MChecking out files: 25% (57/227) ^MChecking out files: 26% (60/227) ^MChecking out files: 27% (62/227) ^MChecking out files: 28% (64/227) ^MChecking out files: 29% (66/227) ^MChecking out files: 30% (69/227) ^MChecking out files: 31% (71/227) ^MChecking out files: 32% (73/227) ^MChecking out files: 33% (75/227) ^MChecking out files: 34% (78/227) ^MChecking out files: 35% (80/227) ^MChecking out files: 36% (82/227) ^MChecking out files: 37% (84/227) ^MChecking out files: 37% (86/227) ^MChecking out files: 38% (87/227) ^MChecking out files: 39% (89/227) ^MChecking out files: 39% (90/227) ^MChecking out files: 40% (91/227) ^MChecking out files: 40% (92/227) ^MChecking out files: 40% (93/227) ^MChecking out files: 41% (94/227) ^MChecking out files: 41% (95/227) ^MChecking out files: 42% (96/227) ^MChecking out files: 42% (97/227) ^MChecking out files: 43% (98/227) ^MChecking out files: 43% (99/227) ^MChecking out files: 44% (100/227) ^MChecking out files: 44% (101/227) ^MChecking out files: 44% (102/227) ^MChecking out files: 45% (103/227) ^MChecking out files: 45% (104/227) ^MChecking out files: 46% (105/227) ^MChecking out files: 46% (106/227) ^MChecking out files: 47% (107/227) ^MChecking out files: 47% (108/227) ^MChecking out files: 48% (109/227) ^MChecking out files: 48% (110/227) ^MChecking out files: 48% (111/227) ^MChecking out files: 49% (112/227) ^MChecking out files: 49% (113/227) ^MChecking out files: 50% (114/227) ^MChecking out files: 50% (115/227) ^MChecking out files: 51% (116/227) 2021-03-25T09:56:11.132 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2021-03-25T09:56:11.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-03-25T09:56:11.325 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 147, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 425, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0691d6bed3e3aaf89688b125297e25f6f6c3fae2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2021-03-25T09:56:11.783 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d943e434792545d5bcd96d9d39646764 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 147, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_0691d6bed3e3aaf89688b125297e25f6f6c3fae2/qa/tasks/workunit.py", line 425, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0691d6bed3e3aaf89688b125297e25f6f6c3fae2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
From: /ceph/teuthology-archive/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/5995979/teuthology.log
It's not clear to me from the log what actually broke.
Actions