Ceph » CephFS

Looks unrelated to anything ceph-specific. My guess is that this is an selinux policy bug, since rpc.mountd should be able to write to the auth.unix.ip file. I'd suggest reporting this to the ubuntu folks.

Ahh, hmm -- just noticed the "add name" deinal too. Does the path "/proc/net/rpc/auth.unix.ip/channel" even exist? Maybe the sunrpc kernel module didn't get fully plugged in before mountd got started?

Anyway, the first AVC denial is here:

avc: denied { add_name } for pid=22038 comm="rpc.mountd" name="channel" scontext=system_u:system_r:nfsd_t:s0 tcontext=system_u:object_r:sysctl_rpc_t:s0 tclass=dir permissive=1'

The channel file there is a conduit to the kernel for handling upcall/downcall messasges. It's very odd that mountd would be trying to create this file as I don't think it ever passes the O_CREAT flag to the open call for this file. I'm checking with the selinux folks now to see whether you can get that warning without O_CREAT being involved...

Just some notes. It looks like the machine has already been torn down and rebuilt, but the new machine is using the same OS:

$ rpm -qa nfs-utils \*selinux\*
selinux-policy-targeted-3.13.1-60.el7_2.7.noarch
selinux-policy-3.13.1-60.el7_2.7.noarch
libselinux-2.2.2-6.el7.x86_64
ceph-selinux-10.2.2-98.gbb94997.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
nfs-utils-1.3.0-0.21.el7_2.x86_64

$ cat /lib/systemd/system/nfs-mountd.service 
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS

...the fact that it requires proc-fs-nfsd.mount would seem to mean that sunrpc.ko should also be plugged in before it can start.

I'm trying to get some clarification of what the application was doing when it got these AVC denials. In the meantime, I took a look at the code. These files are multiplexed over the network namespaces in the kernel. When the module is plugged in, the kernel will create these files for the init_net namespace.

Is it possible that mountd is being run within an different network namespace entirely that isn't yet set up properly? Seems like a stretch, but I don't see any way that these files wouldn't already exist otherwise.

As Scott Mayhew pointed out, the version of nfs-utils that ships in RHEL7.2 uses fopen to open the channel file, and that will add an implicit O_CREAT. So these AVC messages are consistent with the files not being present at the time that the daemon tried to call into the kernel. I'm still a little unclear on how that could occur however.

The other thing of note is the logs seem to indicate that these hosts are running pretty bleeding-edge kernels -- 4.7.0-rc3. It's possible that this is a regression that crept in there.

I notice a recent run that didn't have any of these failures, possibly the kernel got upgraded?
http://pulpito.ceph.com/teuthology-2016-09-17_17:35:03-knfs-master-testing-basic-mira/

John Spray wrote:

I notice a recent run that didn't have any of these failures, possibly the kernel got upgraded?
http://pulpito.ceph.com/teuthology-2016-09-17_17:35:03-knfs-master-testing-basic-mira/

More likely just luck and the files were already present when mountd went to open them. FWIW, I think someone pointed out the problem upstream recently. The nfs-utils package in RHEL7 uses fopen() to open some files under /proc, and that passes in an implicit O_CREAT flag. SELinux is just catching that occurrence, but it wouldn't be allowed by the kernel anyway, so this shouldn't be causing any issues.

Upstream nfs-utils got a set of patches a year or two ago that changed it to use open() instead, and I think that would eliminate the issue there. We may want to have the RH NFS team backport those, but it'll probably be hard to convince them w/o a consistent reproducer. I'll look at that when I get the time.

Ok, sorry for the delay on this. Finally got around to opening a RHBZ:

https://bugzilla.redhat.com/show_bug.cgi?id=1409903

Maybe we'll be able to make something happen for 7.5.

<dgalloway> jcsp, jlayton all reachable miras and smithi have been updated to nfs-utils-1.3.0-0.35.el7.x86_64

I've pushed a branch that pins the knfs tests to centos instead of ubuntu, and doing a run to check it's passing now:
http://pulpito.ceph.com/jspray-2017-01-24_22:02:27-knfs-master-testing-basic-smithi/

I'll plan to leave this open for the next week or two and we can see if any failures crop up between now and then. If there are none, I'll plan to go ahead and close this.