Ceph » bluestore

This might be a duplicate of #53590.
As per my comment in that ticket - could you please collect and share "free-dump" output?

I am not sure if I did it right:
- scale the rook operator down
- edit the deployment
-- command: sleep 3600
-- remove liveness probe
- exec into the OSD pod and fire the command "ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-7 --command free-dump"


2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluefs _check_new_allocations invalid extent 1: 0xff2a0f000~81000: duplicate reference, ino 3507
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluefs mount failed to replay log: (14) Bad address
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_bluefs failed bluefs mount: (14) Bad address
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db failed to prepare db environment:
error from cold_open: (5) Input/output error
command terminated with exit code 1


Jan Tilsner wrote:

I am not sure if I did it right:
- scale the rook operator down
- edit the deployment
-- command: sleep 3600
-- remove liveness probe
- exec into the OSD pod and fire the command "ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-7 --command free-dump"


2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluefs _check_new_allocations invalid extent 1: 0xff2a0f000~81000: duplicate reference, ino 3507
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluefs mount failed to replay log: (14) Bad address
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_bluefs failed bluefs mount: (14) Bad address
2021-12-14T14:58:16.883+0000 7fa9046ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db failed to prepare db environment:
error from cold_open: (5) Input/output error
command terminated with exit code 1


No, something is definitely wrong with this attempt. Wrong device access or something... Unfortunately I'm unable to advise anything in this case due to pretty limited experience with Rook deployments.

Sorry for the lates response, Jan and I have been looking into the issue together.

We were unable to retrieve the ceph-bluestore-tool --command free-dump output from the initially mentioned OSDs and clusters. This did not seem to be caused by Rook,but interestingly went away after creating brand new OSDs. The brand new OSDs are crashing with the same error as above but with the new crashing OSDs we are able to get the `free-dump` output, e.g.,:

# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ --command free-dump > free-dump
2022-01-27T13:12:54.482+0000 7fd928c4a3c0 -1 asok(0x55c99cec4000) AdminSocket: request '[{"prefix": "bluestore allocator dump bluefs-db"}]' not defined
failure querying 'bluefs-db'
...

Would OSD logs with a higher verbosity level help as well?

Hi Alexander,
thanks for the dump, finally I've got a chance to take a look. Sorry for the delay.

I was hoping the dump would be collected for the same OSD which was suffering the outage. Which IIUC isn't the case.
Given the dump is for a different OSD I can't say what space BlueFS attempted to allocate. And hence I'm unable to try that allocation against your allocator snapshot..
So if possible please collect both free-dump and startup log for the same OSD unmodified between probing.

But actually I'm pretty sure the root cause is the lack of contiguous 64K extents which BlueFS tries to allocate. Which in turn is caused by high disk utilization/fragmentation and the fact that BlueStore uses 4K chunks for user data while BlueFS relies on 64K allocation. Once these allocation policies are mixed on the same volume larger chunks might become unavailable at some point. Even before out-of-free-space state is reached...

If my preliminary analysis is valid (which I'm 99% sure) you might want to start using additional standalone WAL/DB volume for your OSD even if physically it collocates the same disk. This workaround would provide less harmful fragmentation on DB volume - in fact BlueFS would be able to allocate space until there is some free space at DB volume. And even beyond that since it can spill data over to a main device. In the latter case the issue might reappear though so it's important to have DB volume large enough - which rather means some space wasting...
We're currently thinking about the allocation strategies redesign but that's evidently not available atm. Hence the workaround above.

Hi Igor

thank you for your input.
You are correct, the logs and the dump are not from the same OSD, they only share the same error. Sorry for that, but we had to repair the cluster at some time.

Your theory about the high utilization might be right, but is not the case in our clusters: we have less than 25% usage.
High fragmentation could be an issue indeed, as the volumes are permanently created, used and deleted in those clusters.

Speaking about the 4K / 64K allocation:
If we add bluefs_shared_alloc_size=4096 to our ceph.conf for the OSDs, they start up and run normal for hours. The OSD then crashes with the following error message and restarts:

debug -4> 2022-02-18T11:26:20.074+0000 7f98000f6700 5 bluestore(/var/lib/ceph/osd/ceph-9) _kv_sync_thread utilization: idle 9.130752563s of 10.114350319s, submitted: 326
debug -3> 2022-02-18T11:26:20.534+0000 7f98000f6700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, uint64_t, uint64_t)' thread 7f98000f6700 time 2022-02-18T11:26:20.526317+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2571: FAILED ceph_assert(bl.length() <= runway)

We discussed putting the DB/WAL in a standalone volume, but that is not something that ROOK supports (at least not on the same physical disk).

Setting the OSD config to 4096 is a workaround that keeps our Clusters going - but it clearly isn't ideal.
Is there a timeline for the rethinking/redesign of the allocation strategy?

Hey Jan,

as for high utilization - both of your OSDs I'm aware of are at more than 80% space utilization:
- from attached osd-2.log (capacity vs. free) :
debug -4> 2021-11-15T12:49:14.556+0000 7f86cebf1080 1 bluefs _allocate unable to allocate 0x90000 on bdev 1, allocator name block, allocator type hybrid, capacity 0x31ffc00000, block size 0x1000, free 0x5d008c000, fragmentation 0.211058, allocated 0x0
i.e. ~25GB of 200 free

- from free-dump output:
"capacity": 214744170496,
Free:66b5fb000 - that's what I got after free dump replay in my lab.
27GB of 210 GB

Setting bluefs-shared-alloc-size to 4K is generally looks like the proper solution but unfortunately it's not properly tested and hence apparently not 100% reliable. I'm aware about at least one more case where it didn't work. Not to mention about its performance impact...

And I'm sorry I wasn't aware about that ROOK limitation, good to know.

As for the redesign timeline - we're currently at the early steps discussing how to proceed. So it's hard to tell anything specific, sorry again.

Just an addendum to my previous comment.
From free-dump analysis the amount of free space available for 64K allocations is:
Available(strict align): 0x120000

The same for 16K allocations:
Available(strict align): 0x2daf84000

So may be try bluefs_shared_alloc_size=16384 ?
But the same concerns as with 4K to be applied. Plus this is still vulnerable to high fragmentation - no guarantee that it wouldn't become worse with the given access pattern...

https://pad.ceph.com/p/bluefs_enospc

https://github.com/ceph/ceph/pull/48854 merged