Bug #39571
closedxfstest generic/452 exposes inode refcount leak
0%
Description
After running xfstest generic/452, I saw this message pop up in the console:
[ 44.795039] VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds. Have a nice day...
Basically we have some inodes that have elevated refcounts even after the superblock is gone. Eventually we'll trip over them and the box will crash.
Files
Updated by Jeff Layton about 5 years ago
This problem is reliably reproducible too.
Updated by Jeff Layton about 5 years ago
This reproduces on a stock v5.0.5 kernel too, so I'm pretty sure none of my patches broke it. I'm working on nailing down the specific sequence of things that happen to trigger this. It reliably reproduces under xfstests when I run this:
$ sudo ./check generic/452
...however, if I run the test itself directly it does not reproduce:
$ sudo ./tests/generic/452
...if I run it a second time, then the message does pop. That leads be to believe that this is happening when cleaning up the scratch directory from a previous run.
Updated by Jeff Layton about 5 years ago
- File 452.filtered 452.filtered added
Filtered strace of the test, showing all syscalls that touch /mnt/scratch or /mnt/scratch/ls_on_scratch.
Updated by Jeff Layton almost 5 years ago
With some printk debugging, leftover inode is the one for /mnt/scratch/ls_on_scratch. Still not able to reproduce this by hand though, so I wonder if there is some raciness involved.
Updated by Jeff Layton almost 5 years ago
Single shell-script reproducer:
#!/bin/bash mount /mnt/scratch rm -r /mnt/scratch/* umount /mnt/scratch mount /mnt/scratch umount /mnt/scratch mount /mnt/scratch ls /mnt/scratch umount /mnt/scratch mount /mnt/scratch cp /usr/bin/ls /mnt/scratch/ls_on_scratch /mnt/scratch/ls_on_scratch /mnt/scratch/ls_on_scratch mount -o remount,ro /mnt/scratch /mnt/scratch/ls_on_scratch /mnt/scratch/ls_on_scratch umount /mnt/scratch
...with this in fstab:
192.168.XXX.YYY:40527:/scratch /mnt/scratch ceph noauto,context="system_u:object_r:root_t:s0",acl 0 0
Probably, some of these steps are not needed so I'll try whittling this down next.
Updated by Jeff Layton almost 5 years ago
Slimmed-down reproducer:
#!/bin/bash mount /mnt/scratch cp /usr/bin/ls /mnt/scratch/ls_on_scratch mount -o remount,ro /mnt/scratch umount /mnt/scratch
...also, the extra mount options don't matter.
More interestingly, after the last umount, the ls_on_scratch file in cephfs seems to be the correct length, but it's completely zero-filled. I wonder if the remount,ro is occurring before we have a chance to flush the cache, and then that prevents writeback from succeeding?
EDIT: calling sync after doing the copy, but before the remount works around the issue.
Updated by Jeff Layton almost 5 years ago
Found the problem. ceph didn't have a remount_sb operation, so we just need that and to have that call sync_filesystem(). Patch posted: