Ceph » CephFS

This could be an issue with time sync on the nodes; check your clock drift. (That's the only issue I know of that we've run into with Hadoop.) If you're using the Hadoop/CephFS Filesystem you don't need to mount CephFS on the client nodes, btw.

If that's not the issue, you'll need to reproduce with mds logging turned on (probably debug mds = 20, debug ms = 1) and client logging (debug client = 20, debug ms = 1) and the admin socket enabled on the Hadoop nodes.
Once it hangs, see if you can get the mds to dump its cache ("ceph tell mds.0 dumpcache", I think) and gather the "mds_requests" and "dump_cache" output from the client admin sockets.

Greg Farnum wrote:

This could be an issue with time sync on the nodes; check your clock drift. (That's the only issue I know of that we've run into with Hadoop.) If you're using the Hadoop/CephFS Filesystem you don't need to mount CephFS on the client nodes, btw.

Double checked, all the clocks are in sync (NTP and all)

If that's not the issue, you'll need to reproduce with mds logging turned on (probably debug mds = 20, debug ms = 1) and client logging (debug client = 20, debug ms = 1) and the admin socket enabled on the Hadoop nodes.
Once it hangs, see if you can get the mds to dump its cache ("ceph tell mds.0 dumpcache", I think) and gather the "mds_requests" and "dump_cache" output from the client admin sockets.

I will do this tonight when the cluster is quiet, is there anything else I can grab at the same time ?

That should be enough to either diagnose it or realize we need to reproduce it locally.

Greg Farnum already knows this, but for reference

I spent a large part of today with debug logging on to try to catch this occurring without much success I think the debug logging slows everything down to the point at which it masks timing junk.

I am going to see if I can grab a mds dump when / if it happens again

Greg Bowyer wrote:

Greg Farnum wrote:

This could be an issue with time sync on the nodes; check your clock drift. (That's the only issue I know of that we've run into with Hadoop.) If you're using the Hadoop/CephFS Filesystem you don't need to mount CephFS on the client nodes, btw.

Double checked, all the clocks are in sync (NTP and all)

If that's not the issue, you'll need to reproduce with mds logging turned on (probably debug mds = 20, debug ms = 1) and client logging (debug client = 20, debug ms = 1) and the admin socket enabled on the Hadoop nodes.
Once it hangs, see if you can get the mds to dump its cache ("ceph tell mds.0 dumpcache", I think) and gather the "mds_requests" and "dump_cache" output from the client admin sockets.

I will do this tonight when the cluster is quiet, is there anything else I can grab at the same time ?

just enable admin socket (without enabling client debug and mds debug). When the hang occurs, dump client request and mds cache (ceph --admin-daemon /var/run/ceph/ceph-fuse.asok mds_requests, ceph mds tell 0 dumpcache /tmp/cachedump.mds0)