Ceph » CephFS

While doing a simple copy of /usr with ceph-fuse and thrashing max_mds between 1 and 2, I got these errors from ceph-fuse:

2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x10000006ebc.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-28 06:30:45.687641 ctime=2018-04-21 00:43:13.684431 caps=- dirty_caps=Fx parents=0x10000006eab.head["Debian"] 0x3b91e2d080)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x10000006eac.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-28 06:30:45.687641 ctime=2018-04-21 00:43:13.171152 caps=- dirty_caps=Fx parents=0x10000006eab.head["Cygwin"] 0x3b91e8b180)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x100000070d4.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-28 06:30:45.627642 ctime=2018-04-21 00:43:19.574843 caps=- dirty_caps=Fx parents=0x10000006eab.head["openSUSE"] 0x3b930ce000)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x10000006db4.head(faked_ino=0 ref=2 ll_ref=0 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-28 06:30:45.607642 ctime=2018-04-21 00:43:09.100143 caps=- dirty_caps=Fx parents=0x10000006db3.head["scripts"] 0x3b91bd5080)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x10000000045.head(faked_ino=0 ref=4 ll_ref=18 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-09 18:00:35.636418 ctime=2018-04-21 00:53:01.063777 caps=- dirty_caps=Fx parents=0x10000000039.head["lib"] 0x3b73c3fb80)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x10000000043.head(faked_ino=0 ref=4 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-09 18:00:35.616419 ctime=2018-04-21 00:53:00.824885 caps=- dirty_caps=Fx parents=0x10000000039.head["dist"] 0x3b73c3f080)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x1000000004b.head(faked_ino=0 ref=4 ll_ref=18 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-09 18:00:35.652418 ctime=2018-04-21 00:53:01.314370 caps=- dirty_caps=Fx parents=0x10000000039.head["test"] 0x3b73cce000)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x100000000b4.head(faked_ino=0 ref=4 ll_ref=12 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-09 18:00:34.184445 ctime=2018-04-21 00:53:05.989777 caps=- dirty_caps=Fx parents=0x100000000ab.head[".nyc_output"] 0x3b73cbc680)
2018-04-21 00:53:20.635 7f2288e11700 -1 client.5313 remove_session_caps still has dirty|flushing caps on 0x100000000b7.head(faked_ino=0 ref=4 ll_ref=10 cap_refs={} open={} mode=40755 size=0/0 btime=0.000000 mtime=2018-03-09 18:00:34.188445 ctime=2018-04-21 00:53:06.075546 caps=- dirty_caps=Fx parents=0x100000000ab.head["test"] 0x3b73cbd700)

Seems we're not flushing caps completely when ranks are shutting down?