Ceph » CephFS

Hi,

we run cephfs (10.2.9 on Debian jessie; 108 OSDs on 9 nodes) as a scratch filesystem for a slurm cluster using IPoIB interconnect and Debian backports kernel (4.9.30).

Our cephfs kernel clients startet blocking on file system access. Logs show 'mds0: Behind on trimming' and slow requests to one osd (osd.049).
Replacing the disk of osd.049 didn't show any effect. Clust health is ok (besides d

'ceph daemon mds.cephmon1 dump_ops_in_flight' shows ops from client sessions which are no longer present according to 'ceph daemon mds.cephmon1 session ls'.

We observe constant traffic of ~200 Mbps on mds node and this OSD (osd.049).
Stopping the mds process ends the traffic.
Stopping OSD instance osd.049 shifts traffic to another OSD (osd.095).
ceph logs show slow requests even after stopping all clients.

Debug log on osd.049 show zillions of lines of a single pg (4.22e) of the cephfs_metadata pool which resides on OSDs [49, 95, 9].

2017-09-19 12:20:08.535383 7fd6b98c3700 20 osd.49 pg_epoch: 240725 pg[4.22e( v 240141'1432046 (239363'1429042,240141'1432046] local-les=240073 n=4848 ec=451 les/c/f 240073/240073/0 239916/240072/240072) [49,95,9] r=0 lpr=240072 crt=240129'1432044 lcod 240130'1432045 mlcod 240130'1432045 active+clean] Found key .chunk_4761369_head

ciao
Christian