Project

General

Profile

Actions

Bug #42496

closed

cbt task does not clean up

Added by Sage Weil over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-10-25T09:00:00.124 INFO:teuthology.task.internal:Tidying up after the test...
2019-10-25T09:00:00.125 INFO:teuthology.orchestra.run.smithi016:Running:
2019-10-25T09:00:00.125 INFO:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2019-10-25T09:00:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:394132    4 drwxr-xr-x   3 ubuntu   ubuntu       4096 Oct 25 08:59 /home/ubuntu/cephtest
2019-10-25T09:00:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:658927    4 drwxrwxr-x   3 ubuntu   ubuntu       4096 Oct 25 08:58 /home/ubuntu/cephtest/0.4.2.c3
2019-10-25T09:00:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:658928    4 drwxrwxr-x   3 ubuntu   ubuntu       4096 Oct 25 08:58 /home/ubuntu/cephtest/0.4.2.c3/log
2019-10-25T09:00:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:658931    4 drwxrwxr-x   2 ubuntu   ubuntu       4096 Oct 25 08:58 /home/ubuntu/cephtest/0.4.2.c3/log/mission
2019-10-25T09:00:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:658932 3168 -rw-rw-r--   1 ubuntu   ubuntu    3242057 Oct 25 08:58 /home/ubuntu/cephtest/0.4.2.c3/log/mission/M621DDCAF8.log
2019-10-25T09:00:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove ‘/home/ubuntu/cephtest’: Directory not empty
2019-10-25T09:00:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1
2019-10-25T09:00:00.220 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 50, in base
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 461, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi016 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

/a/sage-2019-10-25_05:04:47-rados-wip-sage2-testing-2019-10-24-1944-distro-basic-smithi/4443084
/a/sage-2019-10-25_05:04:47-rados-wip-sage2-testing-2019-10-24-1944-distro-basic-smithi/4443148

(The task needs to clean up everything it writes to the cephtest directory.)

This is causing regular failures in the rados suite runs.


Related issues 3 (0 open3 closed)

Has duplicate RADOS - Bug #41946: cbt perf test fails due to leftover in /home/ubuntu/cephtestDuplicate09/22/2019

Actions
Copied to Ceph - Backport #42544: mimic: cbt task does not clean upResolvedNathan CutlerActions
Copied to Ceph - Backport #42545: nautilus: cbt task does not clean upResolvedNathan CutlerActions
Actions #1

Updated by Sage Weil over 4 years ago

looks like the task tried to clean up, but a process was still running and created more file(s).

Actions #2

Updated by Sage Weil over 4 years ago

  • Status changed from 12 to Fix Under Review
  • Pull request ID set to 31171
Actions #3

Updated by Kefu Chai over 4 years ago

  • Backport set to mimic,nautilus
Actions #4

Updated by Kefu Chai over 4 years ago

  • Has duplicate Bug #41946: cbt perf test fails due to leftover in /home/ubuntu/cephtest added
Actions #5

Updated by Kefu Chai over 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #6

Updated by Nathan Cutler over 4 years ago

Actions #7

Updated by Nathan Cutler over 4 years ago

Actions #8

Updated by Nathan Cutler about 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions

Also available in: Atom PDF