Project

General

Profile

Bug #44293

nautilus: pybind/mgr/volumes: incomplete async unlink

Added by Patrick Donnelly 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/volumes
Labels (FS):
Pull request ID:
Crash signature:

Description

2020-02-25T17:13:24.402 INFO:tasks.cephfs_test_runner:======================================================================
2020-02-25T17:13:24.402 INFO:tasks.cephfs_test_runner:ERROR: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes)
2020-02-25T17:13:24.403 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-02-25T17:13:24.403 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-02-25T17:13:24.403 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/cephfs/test_volumes.py", line 1240, in test_async_subvolume_rm
2020-02-25T17:13:24.403 INFO:tasks.cephfs_test_runner:    self._wait_for_trash_empty(timeout=300)
2020-02-25T17:13:24.403 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/cephfs/test_volumes.py", line 158, in _wait_for_trash_empty
2020-02-25T17:13:24.404 INFO:tasks.cephfs_test_runner:    self.mount_a.wait_for_dir_empty(trashdir, timeout=timeout)
2020-02-25T17:13:24.404 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/cephfs/mount.py", line 234, in wait_for_dir_empty
2020-02-25T17:13:24.404 INFO:tasks.cephfs_test_runner:    i, dirname, self.client_id))
2020-02-25T17:13:24.404 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out after 300s waiting for ./volumes/_deleting to become empty from 0

From: /ceph/teuthology-archive/pdonnell-2020-02-25_16:29:58-fs-nautilus-distro-basic-smithi/4801927/teuthology.log


Related issues

Copied to fs - Backport #44315: nautilus: nautilus: pybind/mgr/volumes: incomplete async unlink Resolved

History

#1 Updated by Patrick Donnelly 7 months ago

See also: /ceph/teuthology-archive/pdonnell-2020-02-25_16:29:58-fs-nautilus-distro-basic-smithi/4801912/teuthology.log

#2 Updated by Venky Shankar 7 months ago

exception in purge threads:

2020-02-25 17:06:23.807 7ffb869c5700  1 mgr[volumes] traceback: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/async_job.py", line 42, in run
    self.async_job.execute_job(vol_job[0], vol_job[1], should_cancel=lambda: thread_id.should_cancel())
  File "/usr/share/ceph/mgr/volumes/fs/purge_queue.py", line 59, in execute_job
    purge_trash_entry_for_volume(self.vc, volname, job, should_cancel)
  File "/usr/share/ceph/mgr/volumes/fs/purge_queue.py", line 38, in purge_trash_entry_for_volume
    trashcan.purge(purge_dir, should_cancel)
  File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 88, in purge
    rmtree(trashpath)
  File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 72, in rmtree
    rmtree(d_full)
  File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 64, in rmtree
    log.debug("rmtree {0}".format(root_path))
  File "/usr/lib64/python2.7/logging/__init__.py", line 1137, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/usr/share/ceph/mgr/mgr_module.py", line 65, in emit
    self._module._ceph_log(ceph_level, self.format(record))
TypeError: must be string without null bytes, not str

happens during log.debug call -- maybe a py2 thing.

#3 Updated by Patrick Donnelly 7 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 33547

PR 33547 fixes the exception handling but not the underlying cause (the py2 exception). I'm marking this ticket as fixed by that. Venky, please create a new issue for the py2 exception.

#4 Updated by Patrick Donnelly 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#5 Updated by Patrick Donnelly 7 months ago

  • Copied to Backport #44315: nautilus: nautilus: pybind/mgr/volumes: incomplete async unlink added

#6 Updated by Nathan Cutler 7 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF