Ceph » CephFS

regarding q2:

I increased mds cache size to
mds cache size = 100000000
and it started in seconds.

I don't know how it's connected, but I believe it should be pointed in docs.

I think it's important to specify some kind of metrics so everyone could calculate memory utilization of specific cache size. For example, I have 12GB of RAM, so how big mds cache I can have?

Moreover, If It's so important to MDS then may be it should be connected with some backing storage like BerkeleyDB or something like this in order make system work even if RAM is small.

Actually, regarding initial ticket message. I think MDS goes in some kind of LOOP during start, when cache size is small and amount of files is high. I don't know how to explain in another way the time to start took two days and 5-10 seconds.

I mounted ceph root and counted amount of files and it's less than default cache size of 100000

s:/mnt/ceph# find | wc
  64397   64397 5936240

If the stuck startup is reproducible now (by lowering the cache size and restarting), a log with debug ms =1 and debug mds = 20 would tell us exactly what the problem is. It shouldn't get stuck like that.

Also, yes, allowing memory to be tuned more sanely would definitely be a good thing, and is on the list. :)

Log download link: http://pixeltram.com/ceph-mds.1.log.1.gz

Log is done when it was stuck last time. I stopped MDS, increased log level and started again.

Hi Ivan-

Looking at the log, it looks like all 3 times the MDS started up it came up within 5 seconds or so. Do you have, or can you capture, a log that covers the period where it wouldn't come up? (Or was that what you saw when you were generating this log?)

Thanks!

OK, I don't know what do you mean under "start" term. But actually, all the time MDS run with
debug ms =1 and debug mds = 20 noone client was able to connect it and it's much more than 5 seconds, you can find from log. For me, start is the time from launch till the clients are able to connect. So from the first launch with debugging until the third launch without, MDS was unavailable. Next, I increased mds cache size = 100000000 and it started to handle clients. That's what I mean.

During the previous settings launch it writes messages like this:

2013-03-10 13:53:42.737964 7f0d204f8700 10 mds.0.journal EMetaBlob.replay added [inode 100002b5ceb [2,head] /hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/database/user-elements.php auth v12 s=240 n(v0 b240 1=1+0) (iversion lock) cr={10839=0-4194304@1} 0x42eeba0]
2013-03-10 13:53:42.737973 7f0d204f8700 10 mds.0.journal EMetaBlob.replay added [dentry #1/hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/database/clone.php [2,head] auth NULL (dversion lock) v=10 inode=0 0x43a1950]
2013-03-10 13:53:42.737988 7f0d204f8700 10 mds.0.journal EMetaBlob.replay added [inode 100002b5cec [2,head] /hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/database/clone.php auth v10 s=1742 n(v0 b1742 1=1+0) (iversion lock) cr={10839=0-4194304@1} 0x42ef400]
2013-03-10 13:53:42.738002 7f0d204f8700 10 mds.0.journal EMetaBlob.replay dir 100002b5cee
2013-03-10 13:53:42.738005 7f0d204f8700 10 mds.0.cache force_dir_fragment * on [inode 100002b5cee [...2,head] /hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/change-password/ auth v224 f(v0 m2013-02-21 18:25:06.743720 2=2+0) n(v0 rc2013-02-21 18:25:06.743720 b712 3=2+1) (iversion lock) 0x42e54e0]
2013-03-10 13:53:42.738020 7f0d204f8700 10 mds.0.cache force_dir_fragment no frags under *
2013-03-10 13:53:42.738030 7f0d204f8700 10 mds.0.journal EMetaBlob.replay added dir [dir 100002b5cee /hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/change-password/ [2,head] auth v=0 cv=0/0 state=1073741824 f() n() hs=0+0,ss=0+0 0x41cdac8]
2013-03-10 13:53:42.738039 7f0d204f8700 10 mds.0.journal EMetaBlob.replay updated dir [dir 100002b5cee /hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/change-password/ [2,head] auth v=9 cv=0/0 state=1073741824 f(v0 m2013-02-21 18:25:06.743720 2=2+0) n(v0 rc2013-02-21 18:25:06.743720 b712 2=2+0) hs=0+0,ss=0+0 0x41cdac8]
2013-03-10 13:53:42.738065 7f0d204f8700 10 mds.0.journal EMetaBlob.replay added [dentry #1/hosting/cloud-1/configuration/opt/psa/admin/smb/application/resources/languages/zh-TW/components/forms/change-password/change.php [2,head] auth NULL (dversion lock) v=6 inode=0 0x43a1b28]

to log and doesn't handle clients.

And what's interesting all the time MDS server has incoming traffic of ~40MB/s, but no active clients. I found it after the flap of my NIC, and next found during every start until increased cache.

1) I don't use filesystem snapshots at all.
2) I really have 3 big directories with 40000 files total
3) Some days ago, I've tried to copy our corporate /home and mailboxes to Ceph with export to NFS and got a lot of hangs. I wrote about it in #3935 and as I think it could be only way to get 49436 directories, because current ceph fs doesn't have such amount of directories. After /home and mailboxes were removed from ceph, fs were umounted multiple times with clients reboots and so on.

If you will give me the glue to find something, I will do it for you for sure.

If you run "ceph mds 0 dumpcache <filename>" then the MDS will dump everything it has in cache to the filename you specify. If you can attach that here then we can see how many inodes it's got in cache now, and if there are still a bunch of stray inodes then that will contain clues about why.

root@node-04:~# ceph mds 0 dumpcache mds-0.dump
unrecognized command
root@tsk-vps-node-04:~# ceph -v
ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
root@node-04:~# 

Ok, I did

ceph mds tell 0 dumpcache /tmp/dump.txt

http://91.226.13.93/dump.txt.gz

This dump has 1063591 inodes in the cache, of which only 122104 are non-stray. That doesn't seem quite right.

I do see that on a spot-check all the stray inodes have their version locks held, but not apparently by a client; my guess is that we have a leak somewhere. Going to spend a little time looking into that now.