Project

General

Profile

Bug #16397

nfsd selinux denials causing knfs tests to fail

Added by John Spray about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Testing
Target version:
-
Start date:
06/21/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Component(FS):
qa-suite
Needs Doc:
No


Related issues

Duplicated by fs - Bug #17192: "SELinux denials" in knfs-master-testing-basic-smithi Duplicate 09/01/2016

History

#1 Updated by Jeff Layton about 1 year ago

Looks unrelated to anything ceph-specific. My guess is that this is an selinux policy bug, since rpc.mountd should be able to write to the auth.unix.ip file. I'd suggest reporting this to the ubuntu folks.

#2 Updated by Jeff Layton about 1 year ago

Ahh, hmm -- just noticed the "add name" deinal too. Does the path "/proc/net/rpc/auth.unix.ip/channel" even exist? Maybe the sunrpc kernel module didn't get fully plugged in before mountd got started?

#3 Updated by John Spray about 1 year ago

  • Assignee set to Jeff Layton

#4 Updated by Jeff Layton about 1 year ago

  • Priority changed from Urgent to Normal

Ok, looking at the log, I do see the SELinux denials. I am new teuthology though...

So you have ubuntu boxes that are running this code, and then you enable SELinux on them? Who actually maintains the policy that you're loading on the ubuntu boxes?

Regardless, I'm reducing the priority here since the AVC messages indicate that the box was in permissive mode, so this should have had no effect other than the warnings in the log.

#5 Updated by Jeff Layton about 1 year ago

Anyway, the first AVC denial is here:

avc: denied { add_name } for pid=22038 comm="rpc.mountd" name="channel" scontext=system_u:system_r:nfsd_t:s0 tcontext=system_u:object_r:sysctl_rpc_t:s0 tclass=dir permissive=1'

The channel file there is a conduit to the kernel for handling upcall/downcall messasges. It's very odd that mountd would be trying to create this file as I don't think it ever passes the O_CREAT flag to the open call for this file. I'm checking with the selinux folks now to see whether you can get that warning without O_CREAT being involved...

#6 Updated by Jeff Layton about 1 year ago

  • Status changed from New to Can't reproduce

Ok, talked with Bruce (knfsd maintainer) and the SELinux folks and the consensus is that we have no clue as to why this would have occurred. I think the most likely scenario is a race between mountd starting up and the kernel module being plugged in. IOW, mountd got there before the procfiles were created.

Since we haven't seen any other occurrences of this, I suggest we just close this out and plan to reopen it if it occurs again.

#8 Updated by Jeff Layton about 1 year ago

Just some notes. It looks like the machine has already been torn down and rebuilt, but the new machine is using the same OS:


$ rpm -qa nfs-utils \*selinux\*
selinux-policy-targeted-3.13.1-60.el7_2.7.noarch
selinux-policy-3.13.1-60.el7_2.7.noarch
libselinux-2.2.2-6.el7.x86_64
ceph-selinux-10.2.2-98.gbb94997.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
nfs-utils-1.3.0-0.21.el7_2.x86_64

$ cat /lib/systemd/system/nfs-mountd.service 
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS

...the fact that it requires proc-fs-nfsd.mount would seem to mean that sunrpc.ko should also be plugged in before it can start.

#9 Updated by Jeff Layton about 1 year ago

I'm trying to get some clarification of what the application was doing when it got these AVC denials. In the meantime, I took a look at the code. These files are multiplexed over the network namespaces in the kernel. When the module is plugged in, the kernel will create these files for the init_net namespace.

Is it possible that mountd is being run within an different network namespace entirely that isn't yet set up properly? Seems like a stretch, but I don't see any way that these files wouldn't already exist otherwise.

#10 Updated by Greg Farnum about 1 year ago

  • Category set to Testing
  • Component(FS) qa-suite added

#11 Updated by Jeff Layton about 1 year ago

As Scott Mayhew pointed out, the version of nfs-utils that ships in RHEL7.2 uses fopen to open the channel file, and that will add an implicit O_CREAT. So these AVC messages are consistent with the files not being present at the time that the daemon tried to call into the kernel. I'm still a little unclear on how that could occur however.

#12 Updated by Jeff Layton about 1 year ago

The other thing of note is the logs seem to indicate that these hosts are running pretty bleeding-edge kernels -- 4.7.0-rc3. It's possible that this is a regression that crept in there.

#13 Updated by Ian Colle 11 months ago

  • Duplicated by Bug #17192: "SELinux denials" in knfs-master-testing-basic-smithi added

#14 Updated by John Spray 11 months ago

I notice a recent run that didn't have any of these failures, possibly the kernel got upgraded?
http://pulpito.ceph.com/teuthology-2016-09-17_17:35:03-knfs-master-testing-basic-mira/

#15 Updated by Jeff Layton 11 months ago

John Spray wrote:

I notice a recent run that didn't have any of these failures, possibly the kernel got upgraded?
http://pulpito.ceph.com/teuthology-2016-09-17_17:35:03-knfs-master-testing-basic-mira/

More likely just luck and the files were already present when mountd went to open them. FWIW, I think someone pointed out the problem upstream recently. The nfs-utils package in RHEL7 uses fopen() to open some files under /proc, and that passes in an implicit O_CREAT flag. SELinux is just catching that occurrence, but it wouldn't be allowed by the kernel anyway, so this shouldn't be causing any issues.

Upstream nfs-utils got a set of patches a year or two ago that changed it to use open() instead, and I think that would eliminate the issue there. We may want to have the RH NFS team backport those, but it'll probably be hard to convince them w/o a consistent reproducer. I'll look at that when I get the time.

#16 Updated by John Spray 9 months ago

  • Needs Doc set to No

From a recent run I see this is only happening on centos NFS servers, so I'm going to pin these tests to ubuntu (but leave this ticket open).
https://github.com/ceph/ceph-qa-suite/pull/1289

#17 Updated by Jeff Layton 8 months ago

Ok, sorry for the delay on this. Finally got around to opening a RHBZ:

https://bugzilla.redhat.com/show_bug.cgi?id=1409903

Maybe we'll be able to make something happen for 7.5.

#18 Updated by John Spray 7 months ago

<dgalloway> jcsp, jlayton all reachable miras and smithi have been updated to nfs-utils-1.3.0-0.35.el7.x86_64

I've pushed a branch that pins the knfs tests to centos instead of ubuntu, and doing a run to check it's passing now:
http://pulpito.ceph.com/jspray-2017-01-24_22:02:27-knfs-master-testing-basic-smithi/

#19 Updated by John Spray 7 months ago

  • Status changed from New to Need Review

Patch to unpin from ubuntu, waiting for that test to go through https://github.com/ceph/ceph/pull/13088

#20 Updated by Jeff Layton 7 months ago

I'll plan to leave this open for the next week or two and we can see if any failures crop up between now and then. If there are none, I'll plan to go ahead and close this.

#21 Updated by Jeff Layton 6 months ago

  • Status changed from Need Review to Resolved

I've not heard of any further reports of this since the new package went into production. I'm going to declare victory here and resolve this bug. Please reopen if it reappears.

Also available in: Atom PDF