Ceph » CephFS

Summary¶

When cloning a Ceph FS volume containing fscrypt-enabled subtrees,
the clone misses fscrypt metadata. The fscrypted data becomes inaccessible.

Steps to reproduce¶

Ceph commands:

ceph fs subvolume create subvol a
ceph fs subvolume snapshot create a subvol snap
ceph fs subvolume snapshot clone a subvol snap subvolclone

On a host running kernel git://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux.git branch ceph-fscrypt commitish 46d15ce3eb30

# subvolume
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# mkdir encrypted
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# fscryptctl add_key encrypted
asdfasdfasdfasdfasdfasdfasdfasdf
35601b42a0fb06b9ed6145adf29021a0
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# fscryptctl set_policy 35601b42a0fb06b9ed6145adf29021a0 encrypted/
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# fscryptctl get_policy encrypted
Encryption policy for encrypted:
        Policy version: 2
        Master key identifier: 35601b42a0fb06b9ed6145adf29021a0
        Contents encryption mode: AES-256-XTS
        Filenames encryption mode: AES-256-CTS
        Flags: PAD_32

# snapshot
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/.snap/snap/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# ls encrypted/
bar
rapido1:/mnt/cephfs/volumes/_nogroup/subvol/.snap/snap/b2ffc43d-c28a-48a8-bb92-e759ae84cd07# fscryptctl get_policy encrypted/
Encryption policy for encrypted/:
        Policy version: 2
        Master key identifier: 35601b42a0fb06b9ed6145adf29021a0
        Contents encryption mode: AES-256-XTS
        Filenames encryption mode: AES-256-CTS
        Flags: PAD_32

# clone
rapido1:/mnt/cephfs/volumes/_nogroup/subvolclone/c0053865-5f33-413f-9116-49a9ef6ee641# ls encrypted/
GZNGZhZa7K9FXj+ShkjhU77lk3CkwjAw93itRdjB,oc
rapido1:/mnt/cephfs/volumes/_nogroup/subvolclone/c0053865-5f33-413f-9116-49a9ef6ee641# fscryptctl get_policy encrypted/
error: getting policy for encrypted/: file or directory not encrypted

# xattrs
rapido1:/mnt/cephfs/volumes/_nogroup# getfattr -n ceph.fscrypt.auth subvol/*/encrypted
# file: subvol/b2ffc43d-c28a-48a8-bb92-e759ae84cd07/encrypted
ceph.fscrypt.auth=0sAQAAACgAAAACAQQDAAAAADVgG0Kg+wa57WFFrfKQIaCzVphLRaRLAru8cny30lOh

rapido1:/mnt/cephfs/volumes/_nogroup# getfattr -n ceph.fscrypt.auth subvolclone/*/encrypted
subvolclone/c0053865-5f33-413f-9116-49a9ef6ee641/encrypted: ceph.fscrypt.auth: No such attribute

Possible fix¶

Copying the ceph.fscrypt.auth xattr in the mgr/volumes async clone seems to work:
https://github.com/irq0/ceph/commit/001f5e88c3486ea370abc82546f3a8d98725090d