qa: add network namespaces to kernel/ceph-fuse mounts for partition testing
In teuthology, we want to shutdown the kernel mount without any kind of cleanup like sending SIGKILL to ceph-fuse. We end up doing this by putting the kernel client on a separate node and use impi to hard reset the machine. This is not optimal because we require a separate node for each kernel client.
It'd be better if we had a way to shutdown the cephfs mount without any kind of cleanup. This would allow us to have kernel clients all on the same node and selectively "kill" them.
Obviously, this shouldn't necessarily cause the unmount. Applications may still be using the mount (open fd) or their cwd is on the mount. All operations should return ESHUTDOWN (or similar). `umount [-f]` should work normally.
From Jeff's idea and comments of the first version to fulfill the "halt" mount option, which will try to close all the monc/osdc/mdsc connections without doing any cleanup beforehand, but the socket close routine will send one FIN to the peer, so this couldn't be 100% simulate pulling cable or hard reset the node case.
Dig into the iptable/netfilter code, we can fulfill the iptable DROP rules in kceph directly if there is no any potential problems for this, but it will by pass the userspace iptable app.
This is for ceph-fuse: https://github.com/ceph/ceph/pull/33576
This will use a separating network namespace to isolate the fuse client from the os, then we can just shutdown
the veth inferace of the network namespace container, with this it will just DROP all the socket packets from the cluster without any response.
This is just for fuse client in userspace, next will try this in kclient.
For now both kernel and fuse are working the https://github.com/ceph/ceph/pull/33576.
# ./unshare_ns_mount.sh This will help to isolate the network namespace from OS for the mount client! usage: unshare_ns_mount.sh [OPTIONS [paramters]] [--brxip <ip_address/mask>] OPTIONS: --fuse <ceph-fuse options> The ceph-fuse command options $ unshare_ns_mount.sh --fuse -m 192.168.0.1:6789 /mnt/cephfs -o nonempty --kernel <mount options> The mount command options $ unshare_ns_mount.sh --kernel -t ceph 192.168.0.1:6789:/ /mnt/cephfs -o fs=a --suspend <mountpoint> Down the veth interface in the network namespace $ unshare_ns_mount.sh --suspend /mnt/cephfs --resume <mountpoint> Up the veth interface in the network namespace $ unshare_ns_mount.sh --resume /mnt/cephfs --umount <mountpoint> Umount and delete the network namespace $ unshare_ns_mount.sh --umount /mnt/cephfs --brxip <ip_address/mask> Specify ip/mask for ceph-brx and it only makes sense for --fuse/--kernel options (default: 192.168.255.254/16, netns ip: 192.168.0.1/16 ~ 192.168.255.253/16) $ unshare_ns_mount.sh --fuse -m 192.168.0.1:6789 /mnt/cephfs --brxip 172.19.255.254/12 $ unshare_ns_mount.sh --kernel 192.168.0.1:6789:/ /mnt/cephfs --brxip 172.19.255.254/12 -h, --help Print help
Defaultly it will use the 192.168.X.Y/16 private network IPs for the ceph-brx and netnses as above. And you can also specify your own new ip/mask for the ceph-brx, like:
$ unshare_ns_mount.sh --fuse /mnt/cephfs --brxip 172.19.100.100/12
Then the each netns will get a new ip from the ranges:
[172.16.0.1 ~ 172.19.100.99]/12 and [172.19.100.101 ~ 172.31.255.254]/12
#6 Updated by Patrick Donnelly 11 months ago
- Tracker changed from Bug to Feature
- Project changed from Linux kernel client to CephFS
- Subject changed from fs/ceph: add sysfs control file to hard shutdown mount to qa: add network namespaces to kernel/ceph-fuse mounts for partition testing
- Target version set to v16.0.0
- Source set to Development
- Pull request ID set to 33576
- Component(FS) ceph-fuse, kceph, qa-suite added
- Labels (FS) qa added