Bug #49928
Bug #48679: client: items pinned in cache preventing unmount
client: items pinned in cache preventing unmount x2
% Done:
0%
Source:
Q/A
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-03-19T00:30:04.457 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2021-03-19T00:30:04.489 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_42a957504f98012201fffd5a4e0424c0b4624e8b/qa/tasks/ceph_fuse.py", line 175, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_42a957504f98012201fffd5a4e0424c0b4624e8b/qa/tasks/cephfs/fuse_mount.py", line 393, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (150) after waiting for 900 seconds
From: /ceph/teuthology-archive/pdonnell-2021-03-18_13:46:31-fs-wip-pdonnell-testing-20210318.024145-distro-basic-smithi/5977282/teuthology.log
History
#1 Updated by Patrick Donnelly about 3 years ago
- Related to Bug #48679: client: items pinned in cache preventing unmount added
#2 Updated by Xiubo Li about 3 years ago
- Status changed from New to In Progress
#3 Updated by Xiubo Li about 3 years ago
For the inode `0x10000001949`, since it has Fb cap and the flush cap snap was delayed, but never did it after that:
2021-03-18T22:53:20.785+0000 7fb07ffff700 10 client.4755 finish_cap_snap 0x10000001949.head(faked_ino=0 ref=7 ll_ref=1 cap_refs={4=0,1024=1,4096=0,8192=1} open={2=0} mode=100664 size=10/4194304 nlink=1 btime=2021-03-18T22:53:20.346701+0000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:20.348022+0000 caps=pAsxLsXsxFsxcrwb(0=pAsxLsXsxFsxcrwb) dirty_caps=Fw objectset[0x10000001949 ts 0/0 objects 1 dirty_or_tx 10] parents=0x10000001946.head["file2"] 0x7fb078696dc0) capsnap 0x7fb042f96768 used Fcb 2021-03-18T22:53:20.785+0000 7fb07ffff700 10 client.4755 finish_cap_snap 0x10000001949.head(faked_ino=0 ref=7 ll_ref=1 cap_refs={4=0,1024=1,4096=0,8192=1} open={2=0} mode=100664 size=10/4194304 nlink=1 btime=2021-03-18T22:53:20.346701+0000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:20.348022+0000 caps=pAsxLsXsxFsxcrwb(0=pAsxLsXsxFsxcrwb) dirty_caps=Fw objectset[0x10000001949 ts 0/0 objects 1 dirty_or_tx 10] parents=0x10000001946.head["file2"] 0x7fb078696dc0) cap_snap 0x7fb042f96768 used 9216 WRBUFFER, delaying
And didn't see any `handle_cap_flushsnap_ack()` for the inode.
And then the `unmount` was stuck and waiting:
2021-03-19T01:15:35.302+0000 7fb09d652200 1 client.4755 dump_cache 2021-03-19T01:15:35.302+0000 7fb09d652200 1 client.4755 dump_inode: DISCONNECTED inode 0x10000001949 #0x10000001949 ref 2 0x10000001949.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={4=0,1024=0,4096=0,8192=0} open={2=0} mode=100664 size=10/4194304 nlink=0 btime=0.000000 mtime=2021-03-18T22:53:20.348022+0000 ctime=2021-03-18T22:53:22.173527+0000 caps=-(0=pFxcb) objectset[0x10000001949 ts 0/0 objects 0 dirty_or_tx 0] 0x7fb078696dc0) 2021-03-19T01:15:35.302+0000 7fb09d652200 2 client.4755 cache still has 0+1 items, waiting (for caps to release?)
This is the same issue with https://tracker.ceph.com/issues/48679.
#4 Updated by Xiubo Li about 3 years ago
- Status changed from In Progress to Duplicate
- Parent task set to #48679