Actions
Bug #48640
closedqa: snapshot mismatch during mds thrashing
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-12-17T00:01:48.575 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0)
2020-12-17T00:01:48.576 DEBUG:tasks.ceph.mds.b:waiting for process to exit
2020-12-17T00:01:48.576 INFO:teuthology.orchestra.run:waiting for 300
2020-12-17T00:01:48.594 INFO:tasks.ceph.mds.b:Stopped
...
2020-12-17T00:02:23.217 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5220
2020-12-17T00:02:23.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b
2020-12-17T00:02:23.325 INFO:tasks.workunit.client.0.smithi183.stdout:HEAD is now at a4c752df71 v0.6
2020-12-17T00:02:23.346 INFO:tasks.workunit.client.0.smithi183.stderr:+ mkdir .snap/v0.6
2020-12-17T00:02:23.366 INFO:tasks.workunit.client.0.smithi183.stderr:+ for v in $versions
2020-12-17T00:02:23.366 INFO:tasks.workunit.client.0.smithi183.stderr:+ ver=v0.7
2020-12-17T00:02:23.366 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo v0.7
2020-12-17T00:02:23.367 INFO:tasks.workunit.client.0.smithi183.stderr:+ git reset --hard v0.7
...
2020-12-17T00:02:58.017 INFO:tasks.workunit.client.0.smithi183.stdout:HEAD is now at 090436f56a v0.21
2020-12-17T00:02:58.063 INFO:tasks.workunit.client.0.smithi183.stderr:+ mkdir .snap/v0.21
2020-12-17T00:02:58.169 INFO:tasks.workunit.client.0.smithi183.stderr:+ for v in $versions
2020-12-17T00:02:58.169 INFO:tasks.workunit.client.0.smithi183.stderr:+ ver=v0.1
2020-12-17T00:02:58.170 INFO:tasks.workunit.client.0.smithi183.stderr:+ echo checking v0.1
2020-12-17T00:02:58.170 INFO:tasks.workunit.client.0.smithi183.stderr:+ cd .snap/v0.1
2020-12-17T00:02:58.170 INFO:tasks.workunit.client.0.smithi183.stdout:checking v0.1
2020-12-17T00:02:58.171 INFO:tasks.workunit.client.0.smithi183.stderr:+ git diff --exit-code
2020-12-17T00:03:04.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-12-17T00:03:04.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-12-17T00:03:14.037 INFO:tasks.workunit.client.0.smithi183.stdout:diff --git a/src/ebofs/Cnode.h b/src/ebofs/Cnode.h
2020-12-17T00:03:14.037 INFO:tasks.workunit.client.0.smithi183.stdout:index 810124532c..e69de29bb2 100644
2020-12-17T00:03:14.038 INFO:tasks.workunit.client.0.smithi183.stdout:--- a/src/ebofs/Cnode.h
2020-12-17T00:03:14.038 INFO:tasks.workunit.client.0.smithi183.stdout:+++ b/src/ebofs/Cnode.h
...
2020-12-17T00:03:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-12-17T00:03:14.062 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2020-12-17T00:03:14.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2020-12-17T00:03:15.067 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201214.211711/qa/tasks/workunit.py", line 147, in task
cleanup=cleanup)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201214.211711/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201214.211711/qa/tasks/workunit.py", line 425, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81d0278a02295c5a0d481303def7f69477d4ff55 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2020-12-17T00:03:15.157 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5e1183f827214bc9b23ceabe848260a9
From: /ceph/teuthology-archive/pdonnell-2020-12-16_23:01:45-fs-wip-pdonnell-testing-20201214.211711-distro-basic-smithi/5714203/teuthology.log
Actions