Ceph » rbd

The information that should let us fully diagnose:

• set

  debug osd = 20
  debug filestore = 20
  debug ms = 1

and restart osds (and maybe rotate the logs).

• compile/prepare fiemap tool inside vm
• run the iotester tool to failure
• run fiemap on the bad file
• provide fiemap output and osd log files

The bug also does not seem to have any effect with the setting "filestore fiemap = false" in ceph.conf.

Are there multiple partitions or is LVM on the disk, or is the file system on the raw device?

Sage Weil wrote:

Are there multiple partitions or is LVM on the disk, or is the file system on the raw device?

There are multiple partitions, no LVM.

Guido Winkelmann wrote:

Sage Weil wrote:

Are there multiple partitions or is LVM on the disk, or is the file system on the raw device?

There are multiple partitions, no LVM.

Ah. In that case, you need to add the offset (in bytes) to the beginning of the file system's partition to that physical offset from fiemap. e.g. take the 'fdisk /dev/foo -l' Start sector value * 512, then recalculate the rbd block number and object name...

Sage Weil wrote:

Guido Winkelmann wrote:

Sage Weil wrote:

Are there multiple partitions or is LVM on the disk, or is the file system on the raw device?

There are multiple partitions, no LVM.

Ah. In that case, you need to add the offset (in bytes) to the beginning of the file system's partition to that physical offset from fiemap. e.g. take the 'fdisk /dev/foo -l' Start sector value * 512, then recalculate the rbd block number and object name...

That gives me rbd block 000000000a5b instead then.This still does no seem to conatin the correct data.

Status update:

I tried modifying the iotester so that it would work directly on the block device, in the hopes I can sidestep all those problems with filesystems and partitions in between.

This setup has so far not been able to reproduce the bug. Maybe it's because I'm filling the volume strictly sequentially and completely now, before I verify anything. I suppose I will modify it even more to do some random reads and writes.

Let's try a different tack: I pushed a osd-verify-sparse-read-holes branch to ceph.git (based on 0.47.2) that reads the fiemap-reported holes and verifies they are filled with zeros. If they are not, it puts a warning in the cluster log ($mon_data/log.err and 'ceph -w' output). Enable this in your ceph.conf:

osd verify sparse read holes = true

Restart the osds, and reproduce the corruption, then check for errors like:

2012-06-14 12:42:02.081373 osd.0 [ERR] 2.5_head 9b62f76d/rb.0.1.000000000000/head//2 sparse-read found data in hole 109839~135168

in your log.

Just a bit of context: rbd without caching does a 'sparse-read' operation, which uses FIEMAP to determine which parts of the object are allocated on disk, and then only reads and returns those portions to the client. I suspect that there is a FIEMAP bug in the kernel/file system you are using (we've seen such bugs in ext4, but not recently), or in our code that calls fiemap...

Sage Weil wrote:

Just a bit of context: rbd without caching does a 'sparse-read' operation, which uses FIEMAP to determine which parts of the object are allocated on disk, and then only reads and returns those portions to the client. I suspect that there is a FIEMAP bug in the kernel/file system you are using (we've seen such bugs in ext4, but not recently), or in our code that calls fiemap...

I would expect test_librbd_fsx to do a pretty good job uncovering those problems.

Oh, which leads to another idea: can you build test_librbd_fsx, and run that against your cluster? Something like

./test_librbd_fsx poolname imagename

(where imagename doesn't exist). It would be interesting to see if that runs into problems!

Just so this doesn't fall of the table, can this be considered resolved with the changed filestore fiemap default? Or are there open questions on this that RBD users should still be concerned about?