Bug #48833: snap_rm hang during osd thrashing - CephFS - Ceph

Actions

Copy link

Bug #48833

open

snap_rm hang during osd thrashing

Added by Patrick Donnelly over 3 years ago. Updated almost 2 years ago.

Status:

New

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

pacific,octopus,nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

kceph, qa-suite

Labels (FS):

qa-failure, snapshots

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2021-01-10T19:07:10.487 DEBUG:teuthology.orchestra.run.smithi061:workunit test fs/snaps/untar_snap_rm.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e9d58097ffdbebcb80f58de46bac71db7077a422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh
...
2021-01-10T19:23:09.097 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/ioctl.c' -> './k/linux-2.6.33/fs/ioctl.c'
2021-01-10T19:23:09.098 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/ioprio.c' -> './k/linux-2.6.33/fs/ioprio.c'
2021-01-10T19:23:09.098 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs' -> './k/linux-2.6.33/fs/isofs'
2021-01-10T19:23:09.098 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/Kconfig' -> './k/linux-2.6.33/fs/isofs/Kconfig'
2021-01-10T19:23:09.098 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/Makefile' -> './k/linux-2.6.33/fs/isofs/Makefile'
2021-01-10T19:23:09.099 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/compress.c' -> './k/linux-2.6.33/fs/isofs/compress.c'
2021-01-10T19:23:09.099 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/dir.c' -> './k/linux-2.6.33/fs/isofs/dir.c'
2021-01-10T19:23:09.099 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/export.c' -> './k/linux-2.6.33/fs/isofs/export.c'
2021-01-10T19:23:09.099 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/inode.c' -> './k/linux-2.6.33/fs/isofs/inode.c'
2021-01-10T19:23:09.099 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/isofs.h' -> './k/linux-2.6.33/fs/isofs/isofs.h'
2021-01-10T19:23:09.100 INFO:tasks.workunit.client.0.smithi061.stdout:'.snap/k/linux-2.6.33/fs/isofs/joliet.c' -> './k/linux-2.6.33/fs/isofs/joliet.c'
...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210110.050947/qa/tasks/workunit.py", line 147, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210110.050947/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210110.050947/qa/tasks/workunit.py", line 425, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi061 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e9d58097ffdbebcb80f58de46bac71db7077a422 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'
2021-01-10T22:07:11.721 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=27bbe4594b264097b338fcd6f4d71d0a

From: /ceph/teuthology-archive/pdonnell-2021-01-10_18:07:36-fs-wip-pdonnell-testing-20210110.050947-distro-basic-smithi/5773652/teuthology.log

with k-testing client but may affect other clients too.