Ceph » rbd

Could you provide a few more details on your setup:

- what kernel and distro?
- what fs are the files being imported on?
- does importing from stdin have the same problem? (e.g. rbd import --format 2 - pool/image < file)

Changing format wouldn't affect the import process, although with format 2 changing --stripe-unit and --stripe-count might.

Distro: Ubuntu quantal
Kernel: 3.7.10-030710-generic #201302271235 SMP Wed Feb 27 17:36:27 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux (from ubuntu mainline kernel repos)
Filesystem: XFS

I didn't try importing from stdin, but I can do that in the next few hours and report back.

Import from stdin seems to work fine. I tried it twice, both times it gave the correct checksum. Just to make sure I tried again using importing from the file and this returned again the same wrong md5sum as before. So probably some file hole handling bug?

Yes, this probably means fiemap is broken on xfs with that kernel. The image file you're importing wasn't just written, was it?
'rbd import' does do an fsync before the fiemap, but under heavy write workloads fiemap has still had problems on a just-written and fsync'd file before.

we should switch to using SEEK_HOLE and SEEK_DATA.. that is supposed to work regardless of sync state.

i thought, tho, that we were doing hole detection by looking for zeros? maybe not when reading from a file instead of stdin.

Yeah, I did that for stdin but left the file algorithm alone.

@Josh Jones: are you sure? I mean I'm using a stable kernel versuion, not an rc or any custom build. XFS is known for it's stability and all systems work stable, so I'd really be surprised if this is a xfs/ kernel issue. I googled a bit around but couldn't find any bug reports related to xfs and fiemap issues. Can you point me to any commits/ bug reports related to this? Regarding your question: the image was copied from another datacenter using rsync a few minutes before. But as I did the import several times (5 times at least) I'd expect that it's already been synced long before the last import. The host itself was completely idle at that time, so no (heavy) cpu/io load.

@Sage Weil: looking for zeros seems to be the best imo. I guess a lot of people copy files around and don't care about passing special sparse file options (for example to rsync) to their tools which are needed to preserve the holes and so they'd endup with lots more data in ceph than would actually be needed.

@Josh Jones To make sure it's no kernel bug I just upgraded this single host to the latest mainline stable kernel (3.8.2-030802-generic). Then i imported the file using "rbd import --format 2 vm.img clusterx-hdd/e0df798d-969e-405e-bd93-ba7da1353df9" and exported it again. The exported file has again the same wrong md5sum as with kernel 3.7.10.

I'm pretty sure it's fiemap since that's almost the only difference between importing from a file and importing from stdin.

Although xfs and ext* are generally pretty stable, everything has bugs. fiemap in particular has been known to cause issues in the past, and isn't really well-tested by anything that I'm aware of. We found this out the hard way before when it behaved incorrectly on all fses we tried under a heavy write load on the osds, so we don't use it on the osds anymore. SEEK_HOLE/SEEK_DATA is a newer interface that has tests in xfstests and should be less prone to races.

In the short term, it's easiest to do our own zero detection (which we already do for import from stdin), and use SEEK_HOLE/SEEK_DATA later.

Well you know the ceph sources better for sure. It's only that I'm quite surprised that such a grave bug (it causes corruption...) should be known but still be present in the latest stable kernel versions.

Anyway, probably it'd be a good idea to put a warning in the ceph docs (or even when running rados import ...) then?