Ceph » bluestore

I think the issue is caused by high fragmentation of some onode(s). Looks like there are more than 64K logical non-contiguous extents in such onode(s).

Is this an rbd pool? Does it have compression enabled?

You might want to set debug- bluestore to 2/15 for crashing osds and try to trigger the issue once again. Resulting log might prove the above hypothesis and provide some more insights, e.g. the crashing onode name. Please note that this setting can have negatively impact the performance.

If my hypothesis is correct then I can see the only way to work around the issue without patching and rebuilding the code - export-remove and import PG which keeps "bad" onodes back to its replica OSDs using ceph-objectstore-tool.Given the provided osd.146 log the first candidate for this operation would be PG 2.3. This operation should "defragment" the objects and hence work around the issue.

Dear Igor, many thx!

This is a cephfs pool [1]. Compression was enabled and then I disabled it as soon as there no benefit for data store.
Meantime I was able to run broken osd.27 and I were trying to migrate PGs from it.
I see your point and as soon as the problem returns (it will return definitely) I will update the issue.

By the way If I am be able to split PG NUM from 128 to 256 will it change the high fragmentation and solve the problem? Looks like it should?

Best regards,
Victor Kotlyar

[1]
```
pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 13461 flags hashpspool stripe_width 0 compression_algorithm lz4 compression_mode none application cephfs
```

Hi Victor,
first of all I'd like to confirm my analysis above - I managed to reproduce the issue in my lab.

So the issue occurs when an onode keeps more than 64K non-contiguous extents - for my repro I wrote extents in the following manner (offset~length): 0~1, 4096~1, 8196~1...... up to offset = 4096 * 65536.

IMO PG split wouldn't fix that as this would rather trigger onode remapping/backfilling which is issues faulty read op. The key point is that there are two types of reads from ObjectStore:
1. reading continuous extent using offset,length pair
2. reading object content in a scattered manner using a set of intervals.

PG remap uses the second one coupled by onode's fiemap (i.e. list of non-empty extents). Hence it gets a huge list of intervals to read which results in the assertion - kernel I/O stack is unable to process that long list in a single shot.

On the other hand PG export/import uses read option 1. which will read an onode from offset 0 to its size as a whole including all the gaps. Just a single piece of data. And then write it back in the same manner - with no gaps. Hence (hopefully) working around the issue. Potentially this can cause some increase in required space to keep that PG but I doubt it would be a large raise.

Victor Kotlyar wrote:

Dear Igor, many thx!

This is a cephfs pool [1]. Compression was enabled and then I disabled it as soon as there no benefit for data store.
Meantime I was able to run broken osd.27 and I were trying to migrate PGs from it.
I see your point and as soon as the problem returns (it will return definitely) I will update the issue.

By the way If I am be able to split PG NUM from 128 to 256 will it change the high fragmentation and solve the problem? Looks like it should?

Best regards,
Victor Kotlyar

[1]
```
pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 13461 flags hashpspool stripe_width 0 compression_algorithm lz4 compression_mode none application cephfs
```

Hi Igor, many thx!
Ok I will back with results when hit this issue again. It may takes several days. It is at least a second PG I observed to crush in our CEPH cluster.

Cheers,
Victor

Hi Victor,
thanks for the update. Indeed it confirms my hypothesis - there was very fragmented onode keeping 4-byte data extents every 8 bytes. Here is the relevant log snippet showing data layout in it:

readv 2.3_head #2:c1173368:::10000403d26.00000000:head# fiemap [224~4,232~4,240~4,248~4,256~4,264~4,272~4,280~4,288~4,296~4,304~4,312~4,320~4 ...

Which I presume is caused by a very specific user writing pattern.

The only way to find out additional "bad" objects which comes to my mind is running deep (--deep 1 switch) fsck using ceph-bluestore-tool for every OSD backing this pool. Coupled with setting bluestore_fsck_read_bytes_cap config parameter to a pretty large value, e.g. 2GB.

This will trigger object reads using 2GB data block and thus it will hopefully reveal (by exposing the same assertion) the problem for other objects if any. Might take a while though as deep fsck scans and reads the content for all the objects at an OSD.

Hi Igor,
thank you very much!

Ok, looks like we have to wait for a fix.

Cheers
Victor