Ceph » bluestore

https://bugzilla.redhat.com/show_bug.cgi?id=2064219

Kaleb,
it would be nice to get OSD startup log with debug-bluestore set to 20.

Thanks,
Igor

Log with debug bluestore = 20 put at https://pipebreaker.pl/ceph-osd.1.log.zstd (26MiB)
Compressed, as it grown to 700MB in few seconds.

Thanks, Tomasz!

Unfortunately AvlAllocator has got assertion before logging the presumably broken extent.
Could you please switch to bitmap allocator and collect another log?
Relevant command to do the switch:
ceph config set osd.N bluestore_allocator bitmap

Here it is: https://pipebreaker.pl/ceph-osd.1.bitmap.log.zstd

So it looks like allocation map recovery doesn't properly handle disk sizes which are misaligned with bluestore allocation unit (which is apparently 0x10000 in this case).
From the following log snippet one can see an attempt to mark space beyond disk size as free which causes the assertion. Reported disk size is 0x4a7f851000 while extent being marked for free is 0x3b49110000~f36750000 (i.e. ending offset = 0x4a7f860000

Log snippet:

2022-03-16T12:00:01.116+0100 7f4a93c39180 1 bdev(0x5583418e5800 /var/lib/ceph/osd/ceph-1/block) open size 319967006720 (0x4a7f851000, 298 GiB)
block_size 4096 (4 KiB) rotational discard not supported

-2> 2022-03-16T12:00:24.678+0100 7f4a93c39180 10 fbmap_alloc 0x5583419bca20 init_add_free 0x3b49110000~f36750000
-1> 2022-03-16T12:00:24.692+0100 7f4a93c39180 -1 /builddir/build/BUILD/ceph-17.1.0/src/os/bluestore/BitmapAllocator.cc: In function 'virtual
void BitmapAllocator::init_add_free(uint64_t, uint64_t)' thread 7f4a93c39180 time 2022-03-16T12:00:24.678789+0100
/builddir/build/BUILD/ceph-17.1.0/src/os/bluestore/BitmapAllocator.cc: 74: FAILED ceph_assert(offs + l <= (uint64_t)device_size)

I'm going to submit a patch shortly to fix that shortly.

After additional investigation I realized that this has already been fixed in master.
Pending backport to Quincy: https://github.com/ceph/ceph/pull/45342 (https://tracker.ceph.com/issues/54523)

In fact the following modification is apparently sufficient to fix the issue:

- SimpleBitmap sbmap(cct, div_round_up(bdev->get_size(), min_alloc_size));
+ SimpleBitmap sbmap(cct, (bdev->get_size()/ min_alloc_size));

Tomacz, Kaleb,
broken OSDs should start up fine after you update Ceph binaries with the mentioned patch.

Hence you need to wait for https://github.com/ceph/ceph/pull/45342 to be merged and new Quincy release is published. Alternatively you might want to inject the changes yourself and build new binaries...

Thanks, that also explains why one OSD survived upgrade. It's on LVM volume (the rest are on raw disks), so the it is probably aligned.