Ceph » CephFS

teuthology-2014-06-09_23:02:09-knfs-master-testing-basic-plana/303822/
teuthology-2014-06-09_23:02:09-knfs-master-testing-basic-plana/303825/

I looked a little more closely and it doesn't appear to be teuthology being dumb at first glance. It's already cleaned up after NFS before trying to umount cephfs AFAICT.

http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-06_23:10:01-knfs-next-testing-basic-plana/346584/
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-04_23:10:02-knfs-master-testing-basic-plana/343740/

http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-07_23:10:01-knfs-master-testing-basic-plana/349023/
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-07_23:10:01-knfs-master-testing-basic-plana/349021/
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-07_23:10:01-knfs-master-testing-basic-plana/349026/
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-07_23:10:01-knfs-master-testing-basic-plana/349025/

http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-09_23:10:02-knfs-next-testing-basic-plana/353010/
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-09_23:10:02-knfs-next-testing-basic-plana/353009/

This is still showing up frequently. One example: http://qa-proxy.ceph.com/teuthology/teuthology-2014-09-07_23:10:01-knfs-next-testing-basic-multi/471638/

2014-09-09T10:14:15.908 INFO:tasks.workunit.client.1.plana29.stdout:6/999: dwrite d4/d5/d93/d6c/fe8 [0,1048576] 0
2014-09-09T10:14:15.908 INFO:teuthology.orchestra.run.plana29:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp'
2014-09-09T10:15:21.823 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1...
2014-09-09T10:15:21.823 INFO:teuthology.orchestra.run.plana29:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-09-09T10:15:21.842 DEBUG:teuthology.parallel:result is None
2014-09-09T10:15:21.842 INFO:teuthology.orchestra.run.plana29:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1'
2014-09-09T10:15:22.029 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1
2014-09-09T10:15:22.070 DEBUG:teuthology.run_tasks:Unwinding manager nfs
2014-09-09T10:15:22.070 INFO:teuthology.task.nfs:Unmounting nfs clients...
2014-09-09T10:15:22.070 DEBUG:teuthology.task.nfs:Unmounting nfs client client.1...
2014-09-09T10:15:22.070 INFO:teuthology.orchestra.run.plana29:Running: 'sudo umount /home/ubuntu/cephtest/mnt.1'
2014-09-09T10:15:22.186 INFO:teuthology.orchestra.run.plana29:Running: 'rmdir -- /home/ubuntu/cephtest/mnt.1'
2014-09-09T10:15:22.241 DEBUG:teuthology.run_tasks:Unwinding manager knfsd
2014-09-09T10:15:22.241 INFO:teuthology.task.knfsd:Unexporting nfs server...
2014-09-09T10:15:22.241 DEBUG:teuthology.task.knfsd:Unexporting client client.0...
2014-09-09T10:15:22.242 INFO:teuthology.orchestra.run.plana40:Running: 'sudo exportfs -au'
2014-09-09T10:15:22.289 DEBUG:teuthology.run_tasks:Unwinding manager kclient
2014-09-09T10:15:22.289 INFO:tasks.kclient:Unmounting kernel clients...
2014-09-09T10:15:22.289 INFO:teuthology.orchestra.run.plana40:Running: 'sudo umount /home/ubuntu/cephtest/mnt.0'
2014-09-09T10:15:22.381 INFO:teuthology.orchestra.run.plana40.stderr:umount: /home/ubuntu/cephtest/mnt.0: device is busy.
2014-09-09T10:15:22.382 INFO:teuthology.orchestra.run.plana40.stderr:        (In some cases useful info about processes that use
2014-09-09T10:15:22.382 INFO:teuthology.orchestra.run.plana40.stderr:         the device is found by lsof(8) or fuser(1))
2014-09-09T10:15:22.383 ERROR:teuthology.run_tasks:Manager failed: kclient
Traceback (most recent call last):

NFS was mounted async, and we're trying to unmount Ceph 47 ms after we remove the nfs export. I can't find in a quick skim anywhere if exportfs will block until everything is flushed or not....maybe we just need to wait a bit? Anybody know what the rules are, or a good way to check?

Is there any chance that just running a sync on the node prior to trying to "exportfs -au" might prevent this? I'm hesitant to just add it in case that masks something else we should be worried about, but given how I've not seen this failure when testing locally I don't know how else to check it besides just pushing it to teuthology. :/

I notice that if I execute 'service nfs stop' first, umounting cephfs always successes. 'service nfs stop' runs two commands: rpc.nfsd 0 and exportfs -f