cephfs snapshot remains visible in nfs export after deletion and new snaps not shown
When a snapshot is taken of the subvolume, the .snap directory shows the snapshot when viewed from the NFS mount and cephfs mount.
However, when the snapshot is deleted the cephfs mount correctly shows an empty .snap directory - but the NFS mount still shows the snapshot entry and the directories and files listed. You can even copy a file from the snapshot to the main filesystem!
root@nfs-client-02 ~]# ls /mnt/cephfs/.snap/_snapit_1099511628282/ 172.18.199.3.workload.0.0 smalldfile smallfile specstorage testfile [root@nfs-client-02 ~]# ls /mnt/nfs/.snap/_snapit_1099511628282/ 172.18.199.3.workload.0.0 smalldfile smallfile specstorage testfile [root@nfs-client-02 ~]# # snapshot has been deleted [root@nfs-client-02 ~]# ls /mnt/cephfs/.snap [root@nfs-client-02 ~]# ls /mnt/nfs/.snap/_snapit_1099511628282/ 172.18.199.3.workload.0.0 smalldfile smallfile specstorage testfile <--- still shows a snapshot! [root@nfs-client-02 ~]# cp /mnt/nfs/.snap/_snapit_1099511628282/testfile ~/. <- copying a file from a snapshot that has been deleted worked! [root@nfs-client-02 ~]# cat testfile hello from the test file
df on the client shows the following
172.16.36.61:/volumes/_nogroup/nfs-client-02/c53381bb-c6e8-4ac2-89bf-c07b042ab56b 100G 5.0G 95G 5% /mnt/cephfs 172.18.200.10:/nfs-client-02 100G 5.0G 95G 5% /mnt/nfs 172.18.200.10:/nfs-client-02/.snap 100G 5.0G 95G 5% /mnt/nfs/.snap 172.18.200.10:/nfs-client-02/.snap/_snapit_1099511628282 100G 5.0G 95G 5% /mnt/nfs/.snap/_snapit_1099511628282
There appears to be a sync issue when there are multiple snapshots too.
When I create several snapshots on cephfs I see this
[root@nfs-client-02 ~]# ls /mnt/cephfs/.snap _snapit-again_1099511628282 _snapit-again-2_1099511628282
But when I view the nfs mount which is the same subvolume
[root@nfs-client-02 ~]# ls /mnt/nfs/.snap <-- _snapit-again-2 is missing! _snapit-again_1099511628282
I'm not sure where the issue lies - Ganesha or cephfs, so raising against cephfs
#2 Updated by Venky Shankar 3 months ago
- Status changed from New to Duplicate
You are probably running into https://tracker.ceph.com/issues/59041 - at least for the part for listing snapshots. The mtime for the .snap dire doesn't get updated on snapshot creation/deletion causing the NFS client to serve the snapshot lists from its cache. I see the affected versions are 18.*, however, those releases should have the fix, so, I'm a bit worried what's going on. Can you share the result from `ceph versions`?
As far as being able to copy data out from the deleted snapshot, I recall another tracker listing out the same issue and I don't think it was RCA'd. I'll dig that up and follow up on the details.