Project

General

Profile

Bug #49928

Bug #48679: client: items pinned in cache preventing unmount

client: items pinned in cache preventing unmount x2

Added by Patrick Donnelly 30 days ago. Updated 29 days ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-03-19T00:30:04.457 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse.
2021-03-19T00:30:04.489 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_42a957504f98012201fffd5a4e0424c0b4624e8b/qa/tasks/ceph_fuse.py", line 175, in task
    mount.umount_wait()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_42a957504f98012201fffd5a4e0424c0b4624e8b/qa/tasks/cephfs/fuse_mount.py", line 393, in umount_wait
    run.wait([self.fuse_daemon], timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 473, in wait
    check_time()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (150) after waiting for 900 seconds

From: /ceph/teuthology-archive/pdonnell-2021-03-18_13:46:31-fs-wip-pdonnell-testing-20210318.024145-distro-basic-smithi/5977282/teuthology.log

History

#1 Updated by Patrick Donnelly 30 days ago

  • Related to Bug #48679: client: items pinned in cache preventing unmount added

#2 Updated by Xiubo Li 29 days ago

  • Status changed from New to In Progress

#3 Updated by Xiubo Li 29 days ago

For the inode `0x10000001949`, since it has Fb cap and the flush cap snap was delayed, but never did it after that:

2021-03-18T22:53:20.785+0000 7fb07ffff700 10 client.4755 finish_cap_snap 0x10000001949.head(faked_ino=0 ref=7 ll_ref=1 cap_refs={4=0,1024=1,4096=0,8192=1} open={2=0} mode=100664 size=10/4194304 nlink=1 btime=2021-03-18T22:53:20.346701+0000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:20.348022+0000 caps=pAsxLsXsxFsxcrwb(0=pAsxLsXsxFsxcrwb) dirty_caps=Fw objectset[0x10000001949 ts 0/0 objects 1 dirty_or_tx 10] parents=0x10000001946.head["file2"] 0x7fb078696dc0) capsnap 0x7fb042f96768 used Fcb
2021-03-18T22:53:20.785+0000 7fb07ffff700 10 client.4755 finish_cap_snap 0x10000001949.head(faked_ino=0 ref=7 ll_ref=1 cap_refs={4=0,1024=1,4096=0,8192=1} open={2=0} mode=100664 size=10/4194304 nlink=1 btime=2021-03-18T22:53:20.346701+0000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:20.348022+0000 caps=pAsxLsXsxFsxcrwb(0=pAsxLsXsxFsxcrwb) dirty_caps=Fw objectset[0x10000001949 ts 0/0 objects 1 dirty_or_tx 10] parents=0x10000001946.head["file2"] 0x7fb078696dc0) cap_snap 0x7fb042f96768 used 9216 WRBUFFER, delaying

And didn't see any `handle_cap_flushsnap_ack()` for the inode.

And then the `unmount` was stuck and waiting:

2021-03-19T01:15:35.302+0000 7fb09d652200  1 client.4755 dump_cache
2021-03-19T01:15:35.302+0000 7fb09d652200  1 client.4755 dump_inode: DISCONNECTED inode 0x10000001949 #0x10000001949 ref 2 0x10000001949.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={4=0,1024=0,4096=0,8192=0} open={2=0} mode=100664 size=10/4194304 nlink=0 btime=0.000000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:22.173527+0000 caps=-(0=pFxcb) objectset[0x10000001949 ts 0/0 objects 0 dirty_or_tx 0] 0x7fb078696dc0)
2021-03-19T01:15:35.302+0000 7fb09d652200  2 client.4755 cache still has 0+1 items, waiting (for caps to release?)

This is the same issue with https://tracker.ceph.com/issues/48679.

#4 Updated by Xiubo Li 29 days ago

  • Status changed from In Progress to Duplicate
  • Parent task set to #48679

Also available in: Atom PDF