Ceph » rbd

Could you provide your fio job file / config to verify the issue?

Hi Guys,

This is all on the fio side. From what I remember, when you are doing sequential writes and specify multiple "--name" parameters or numjobs > 1 with the librbd engine in fio, each process will write starting at the beginning of the RBD volume. This has various effects, including what looks like a disparity between OSD journal writes and backend data store writes (since early writes to the same blcok can be thrown away if there is a new one). In reality it's not a problem (unless you are benchmarking journal devices!), but it caught us off-guard when we first saw it.

Normally in fio, if you are targeting files or even block devices with --name using sync or libaio you'll write to different offsets just by virtue of the file system allocating different extents (or maybe just because you are targeting different devices). With the librbd engine you don't get this, but one way around it is to just use a single --name/numjobs parameter with a higher iodepth. This may not be exactly the same though (Danny, any thoughts on how much this matters as far as fio goes? Will having fewer fio processes with higher iodepth change the behaviour?)

If the writes are the same size as the RBD objects, we can kind of cheat and use random writes to simulate sequential writes (they are all randomly distributed objects anyway!). In any event, none of this will actually affect users doing real work, just folks that are using the librbd engine with fio and increase numjobs or specify multiple name parameters.

@Mark: I have to take a look at fio for that. Is this all about sequential writes only? Do you see a different behavior between multiple jobs and increasing the iodepth instead?