Ceph » Linux kernel client

This problem is reliably reproducible too.

This reproduces on a stock v5.0.5 kernel too, so I'm pretty sure none of my patches broke it. I'm working on nailing down the specific sequence of things that happen to trigger this. It reliably reproduces under xfstests when I run this:

$ sudo ./check generic/452

...however, if I run the test itself directly it does not reproduce:

$ sudo ./tests/generic/452

...if I run it a second time, then the message does pop. That leads be to believe that this is happening when cleaning up the scratch directory from a previous run.

With some printk debugging, leftover inode is the one for /mnt/scratch/ls_on_scratch. Still not able to reproduce this by hand though, so I wonder if there is some raciness involved.

Single shell-script reproducer:

#!/bin/bash

mount /mnt/scratch
rm -r /mnt/scratch/*
umount /mnt/scratch
mount /mnt/scratch
umount /mnt/scratch
mount /mnt/scratch
ls /mnt/scratch
umount /mnt/scratch
mount /mnt/scratch
cp /usr/bin/ls /mnt/scratch/ls_on_scratch
/mnt/scratch/ls_on_scratch /mnt/scratch/ls_on_scratch
mount -o remount,ro /mnt/scratch
/mnt/scratch/ls_on_scratch /mnt/scratch/ls_on_scratch
umount /mnt/scratch

...with this in fstab:

192.168.XXX.YYY:40527:/scratch    /mnt/scratch    ceph    noauto,context="system_u:object_r:root_t:s0",acl    0 0

Probably, some of these steps are not needed so I'll try whittling this down next.

Slimmed-down reproducer:

#!/bin/bash

mount /mnt/scratch
cp /usr/bin/ls /mnt/scratch/ls_on_scratch
mount -o remount,ro /mnt/scratch
umount /mnt/scratch

...also, the extra mount options don't matter.

More interestingly, after the last umount, the ls_on_scratch file in cephfs seems to be the correct length, but it's completely zero-filled. I wonder if the remount,ro is occurring before we have a chance to flush the cache, and then that prevents writeback from succeeding?

EDIT: calling sync after doing the copy, but before the remount works around the issue.

Found the problem. ceph didn't have a remount_sb operation, so we just need that and to have that call sync_filesystem(). Patch posted:

https://marc.info/?l=ceph-devel&m=155723567025589&w=2