Ceph

I also wanted to add that I've instructed the pgs to scrub, deep-scrub, and repair, but the cluster never takes these actions. I'm not sure this is relevant, but I wanted to be thorough.

I need some additional information:
1) Please confirm via the admin socket that all of the osds are running the same version, and that version is the one you mentioned above.
2) What versions has this cluster run in the past?
3) Please push a branch matching that sha1 to somewhere I can fetch it.

In addition to the above information, can you add a recursive ls of the 33.151d collection on osd 47?

Also, are you using zfs?

1. All OSDs are running the same version -- confirmed (I can cut and paste the Ansible-y goodness if you'd like to verify yourself)
2. The cluster has previously run 0.94.3. It was upgraded from there to 0.94.6-2
3. https://github.com/bjfelton/as-ceph/tree/clc_hammer_patch
4. http://canada.os.ctl.io/osd-logs/33.151ds.tgz
5. Yes, we are using ZFS.

Are there no subdirectories in that directory? I need to see what subdirectories the files in that collection are in.

That github link gives a 404.

Sam,

My apologies. Here is an update, public link: https://github.com/bjfelton/ceph/tree/clc_hammer_patch

Also, there are no subdirectories:

root@osd07:/osd/47/current/33.151ds8_head# find . -type d
.
root@osd07:/osd/47/current/33.151ds8_head#

We have clusters paired together with one as the DR site. We have found that directory splitting on the OSDs causes significant performance problems due to our collocated journals, so we set one cluster in the pair to split on default boundaries and one to split much, much later in life. In our testing with ZFS, we didn't find this to be a problem, and this ensures we have a failover target whenever splits hit us.

It looks to me like HashIndex::list_by_hash is erroring out. We'll need more debugging there to figure out why.

https://github.com/athanatos/ceph/commit/6c9cffca444f7611bebd1be37e42bab2de12ba0b is based on v94.6 and adds some debugging (I have not even run this code, so use at your own risk). You'll want to cherry-pick it and run it on osd.47 and capture the same debugging levels on 47 and 424.

Sam,

I have built and deployed the patch, restarted OSDs 47 and 424, set debugging, downed 424, and let things settle. The following logs are the result:

http://osd-logs.canada.os.ctl.io/ceph-osd.47.log.20160816.tgz
http://osd-logs.canada.os.ctl.io/ceph-osd.424.log.20160816.tgz

I am reviewing as well to better understand the code path here, but your help is very much appreciated.

Brian

Also, I bet the primary (at least) is ok since the scan didn't error out. From that, I infer guessing that whatever happened to osd.47 (and I bet the target for all of the other stuck backfills) is unusual. You can probably work around the stall on osd.47 by using ceph-objectstore-tool to remove the partially backfilled shard for that pg on osd.47 and letting backfill restart. If the same problem happens again, then it should be relatively simple for you to modify the LFNIndex code to add debugging to work out why.

Sam,

First, I cannot thank you enough for your assistance here. The last bits of debugging pointed me in the right direction, and I've got my cluster in a healthy state again.

tl;dr -- there is no problem with Ceph or ZFS. Please close.

If you're still with me...

The problem was not ZFS. It is perfectly capable of handling Ceph's xattrs (both in normal cases and for lfn cases), although it does appear artificially limited here:

// max xattr value size
OPTION(filestore_max_xattr_value_size, OPT_U32, 0)      //Override
OPTION(filestore_max_xattr_value_size_xfs, OPT_U32, 64<<10)
OPTION(filestore_max_xattr_value_size_btrfs, OPT_U32, 64<<10)
// ext4 allows 4k xattrs total including some smallish extra fields and the
// keys.  We're allowing 2 512 inline attrs in addition some some filestore
// replay attrs.  After accounting for those, we still need to fit up to
// two attrs of this value.  That means we need this value to be around 1k
// to be safe.  This is hacky, but it's not worth complicating the code
// to work around ext4's total xattr limit.
OPTION(filestore_max_xattr_value_size_other, OPT_U32, 1<<10)

Since we've been running the clusters on ZFS for 18 or so months now, since we're currently storing hundreds of millions of objects, and since we're no strangers to disk replacements and backfills, it seemed unlikely that Ceph was not interacting properly with ZFS in most all cases. But the logs were certainly clear that we were missing an xattr here. So I did some digging. Using getfattr, I checked the file above on OSDs 47 and 424. On 424, I found a file with four attrs. On 47, however, the file was missing xattrs.

After taking a backup, I removed the file, cranked up debugging, and forced peering. And the file was no longer in the logs as problematic (although many others were). Now, at this point, I had two options -- do surgery on the files in the logs (i.e. remove them or manually set the attrs), or nuke the pg's contents on 47 and start the backfill over. Given the four other pgs in this state and the amount of time it would have taken to write the needed tooling, I stopped the OSD, took a backup of the contents, nuked everything except the __head* file, and let 'er rip.

Lather, rinse, repeat with the other four pgs, and a few hours later, I have a healthy cluster.

I'm not sure how I ended up with files with no xattrs, but I'd wager the thrashing caused by our collocated journal setup was the primary culprit. I'm not sure if being able to scrub/repair the pg would have solved this, but it's a moot point now.

Thanks again for your assistance.