Project

General

Profile

Bug #11255

nfs: mount failures on ceph-backed NFS share

Added by Greg Farnum over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
NFS (Linux Kernel)
Target version:
-
Start date:
03/27/2015
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

http://pulpito.ceph.com/teuthology-2015-03-25_23:10:02-knfs-master-testing-basic-multi/822373/

2015-03-26T20:49:50.194 INFO:teuthology.run_tasks:Running task knfsd...
2015-03-26T20:49:50.194 INFO:teuthology.task.knfsd:Exporting nfs server...
2015-03-26T20:49:50.194 DEBUG:teuthology.task.knfsd:Client client.0 config is {'options': ['rw', 'no_root_squash', 'async']}
2015-03-26T20:49:50.194 INFO:teuthology.task.knfsd:Exporting knfsd client.0 at ubuntu@plana77.front.sepia.ceph.com *:/home/ubuntu/cephtest/mnt.0 (rw,no_root_squash,async)...
2015-03-26T20:49:50.195 INFO:teuthology.orchestra.run.plana77:Running: 'sudo chmod 777 /home/ubuntu/cephtest/mnt.0'
2015-03-26T20:49:50.298 INFO:teuthology.task.knfsd:remote run ['sudo', 'exportfs', '-o', 'fsid=1230,rw,no_root_squash,async', '10.214.132.15:/home/ubuntu/cephtest/mnt.0', '10.214.132.18:/home/ubuntu/cephtest/mnt.0', '10.214.132.14:/home/ubuntu/cephtest/mnt.0', '10.214.133.39:/home/ubuntu/cephtest/mnt.0']
2015-03-26T20:49:50.298 INFO:teuthology.orchestra.run.plana77:Running: 'sudo exportfs -o fsid=1230,rw,no_root_squash,async 10.214.132.15:/home/ubuntu/cephtest/mnt.0 10.214.132.18:/home/ubuntu/cephtest/mnt.0 10.214.132.14:/home/ubuntu/cephtest/mnt.0 10.214.133.39:/home/ubuntu/cephtest/mnt.0'
2015-03-26T20:49:50.448 INFO:teuthology.run_tasks:Running task nfs...
2015-03-26T20:49:50.448 INFO:teuthology.task.nfs:Mounting nfs clients...
2015-03-26T20:49:50.448 DEBUG:teuthology.task.nfs:Client client.1 config is {'options': ['rw', 'hard', 'intr', 'nfsvers=4'], 'server': 'client.0'}
2015-03-26T20:49:50.449 INFO:teuthology.task.nfs:Mounting client.1 from client.0
2015-03-26T20:49:50.449 DEBUG:teuthology.task.nfs:mount -o rw,hard,intr,nfsvers=4 plana77.front.sepia.ceph.com:/home/ubuntu/cephtest/mnt.0 /home/ubuntu/cephtest/mnt.1
2015-03-26T20:49:50.449 INFO:teuthology.orchestra.run.plana64:Running: 'mkdir -- /home/ubuntu/cephtest/mnt.1'
2015-03-26T20:49:50.541 INFO:teuthology.orchestra.run.plana64:Running: 'sudo mount -o rw,hard,intr,nfsvers=4 plana77.front.sepia.ceph.com:/home/ubuntu/cephtest/mnt.0 /home/ubuntu/cephtest/mnt.1'
2015-03-26T20:49:51.360 INFO:teuthology.orchestra.run.plana64.stderr:mount.nfs: access denied by server while mounting plana77.front.sepia.ceph.com:/home/ubuntu/cephtest/mnt.0
2015-03-26T20:49:51.363 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/nfs.py", line 106, in task
    mnt
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 137, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana64 with status 32: 'sudo mount -o rw,hard,intr,nfsvers=4 plana77.front.sepia.ceph.com:/home/ubuntu/cephtest/mnt.0 /home/ubuntu/cephtest/mnt.1'

I couldn't find anything interesting in the logs, though. :(

History

#3 Updated by Greg Farnum over 4 years ago

Looking through the man pages, I see

Exporting to IP networks or DNS and NIS domains does not enable clients from these groups to access NFS immediately.  Rather, these sorts of exports are hints to rpc.mountd(8) to grant any mount requests from these clients.

I'm not quite sure if that means we maybe need to be doing some kind of blocking until the mountd is happy or something?

#4 Updated by Greg Farnum over 4 years ago

  • Subject changed from nfs: access denied to nfs: mount failures on ceph-backed NFS share

http://pulpito.ceph.com/teuthology-2015-04-29_23:10:02-knfs-next-testing-basic-multi/869428/

I also realized recently that (d'oh) error number 32 here is not actually Access Denied but just what the mount command returns to indicate "mount failure". Which again makes me think that the clients and servers are using different settings of some description...

#7 Updated by Greg Farnum about 4 years ago

Did you mean to reference https://github.com/ceph/teuthology/pull/554, Zheng?

#8 Updated by Greg Farnum about 4 years ago

  • Status changed from New to Need Review

#10 Updated by John Spray over 3 years ago

  • Status changed from Need Review to Resolved

The linked PR merged and this is not appearing currently in master.

Also available in: Atom PDF