Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2024-03-28T03:28:48ZCeph
Redmine bluestore - Support #65190 (New): Can ceph with SPDK driver has to be connected with local NVMe d...https://tracker.ceph.com/issues/651902024-03-28T03:28:48ZAlice Wang
<p>Hi everyone,<br />I am using ceph 18.2.2 to create a cluster with NVMe device under SPDK driver.<br />When the device is connect to the host, there is a PCIe number, such as 0000:3c:00.0, then I can start the cluster by the below command</p>
<p>vstart.sh --new -X --localhost --bluestore-spdk "trtype:PCIe traddr:0000:3c:00.0" --bluestore</p>
<p>The above is OK. But<br />If Ceph is using the SPDK engine, does it have to have a local PCIe device?<br />If the PCIe device is connected to host through NVMe-oF, can we still use bluestore-spdk to start the cluster with NVMe-oF device?<br />If it is the malloc bdev created by SPDK, can we still use bluestore-spdk to start the cluster with malloc bdev?</p>
<p>Thank you.</p> bluestore - Bug #64966 (Fix Under Review): OSDs crash | Assert error | KernelDevice::aio_submit |...https://tracker.ceph.com/issues/649662024-03-18T12:54:25ZVictor Kotlyar
<p>Dear all,<br />we faced a strange errors.</p>
<p>When one of osd died a cluster started to remap\bacfilling but than for one pool group (2.3 `3 replica) its primary osds started to crush on an "Assert error" [1] one by one making PG down.<br />When I disable backfill on the CEPH cluster they are back to normal but the cluster stays with undersized+degraded PGs.</p>
<p>I have attached this PG dump. And comments for the file: osd.27 - broken disk, osd.191&osd.146 alive disks, backfill started to osd.213 when 191&146 went to down loop state by "Assert error" <br />I have attached the log near crush from osd.146. The time of "Assert error" Mar 18 16:35:11</p>
<p>This cluster is Ansible setup in docker on Ubuntu 20.04 with kernel '5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux' on SAS disks. Whole disk a single bluestore OSD device over LVM.</p>
<p>Note that there was remapping process before osd.27 went down.</p>
<p>Is there any way to avoid such assert error?<br />Any help would be appreciated.</p>
<p>Best regards,<br />Victor Kotlyar</p>
<p>p.s. I also have added some, as I think, useful info. No anomaly logs from the system\kernel side were observed.</p>
<p>[1]<br /><pre>
Mar 18 16:35:11 storage-3-5 docker[2710998]: ceph-osd: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/blk/kernel/KernelDevice.cc:837: virtual void KernelDevice::aio_submit(IOContext*): Assertion `pending <= std::numeric_limits<uint16_t>::max()' failed.
Mar 18 16:35:11 storage-3-5 docker[2710998]: *** Caught signal (Aborted) **
Mar 18 16:35:11 storage-3-5 docker[2710998]: in thread 7f42ea171700 thread_name:tp_osd_tp
Mar 18 16:35:11 storage-3-5 docker[2710998]: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
</pre></p> bluestore - Backport #64936 (In Progress): quincy: BlueStore/DeferredWriteTest.NewData/3 is brokenhttps://tracker.ceph.com/issues/649362024-03-14T17:30:15ZBackport Botbluestore - Backport #64928 (In Progress): reef: KeyValueDB/KVTest.RocksDB_estimate_size tests fa...https://tracker.ceph.com/issues/649282024-03-14T13:32:10ZBackport Botbluestore - Backport #64860 (In Progress): reef: ObjectStore/StoreTest.BluestoreRepairGlobalStat...https://tracker.ceph.com/issues/648602024-03-12T10:57:43ZBackport Botbluestore - Backport #64858 (In Progress): quincy: ObjectStore/StoreTest.BluestoreRepairGlobalSt...https://tracker.ceph.com/issues/648582024-03-12T10:57:29ZBackport Botbluestore - Support #64702 (New): how to create spdk backend osdhttps://tracker.ceph.com/issues/647022024-03-05T02:33:29ZJianyun Cheng
<p>Hi, there</p>
<p>I'm trying to create spdk backend osd, but it cant' success. My operations like this:</p>
<p>1. build ceph 18.2.1 with -DWITH_SPDK=ON option<br />2. config spdk env by running<br /><pre><code class="c syntaxhl"><span class="CodeRay"><span class="preprocessor">#sudo</span> spdk/scripts/setup.sh
<span class="octal">0000</span>:<span class="octal">00</span>:<span class="float">0</span>e<span class="float">.0</span> (<span class="integer">8</span><span class="float">0</span>ee <span class="float">4</span>e56): nvme -> uio_pci_generic
</span></code></pre></p>
<p>3. my ceph.conf which like this:<br /><pre><code class="text syntaxhl"><span class="CodeRay">[global]
plugin dir = lib
erasure code dir = lib
fsid = dca4b307-dfbf-4bd5-9358-028c6ad23b85
mon_initial_members = localhost
mon_host = 127.0.0.1
#public_network = 10.234.46.0/24
#cluster_network = 10.234.46.0/24
auth_cluster_required = none
auth service_required = none
auth_client_required = none
cephx_sign_messages = false
auth_supported = none
debug_bluestore = 20/20
[osd]
bluestore = true
#bluestore_block_db_path = ""
#bluestore_block_db_size = 0
#bluestore_block_wal_path = ""
#bluestore_block_wal_size = 0
#bluestore_block_size = 0
#bluestore_allocator=bitmap
#bluefs_allocator=bitmap
#osd_memory_target = 10737418240
#bluestore_cache_size = 7516192768
[osd.0]
host = local
osd data = /var/lib/ceph/osd/ceph-0
bluestore_block_path = spdk:trtype:PCIe traddr:0000:00:0e.0
</span></code></pre></p>
<p>4. create osd by running the command<br /><pre><code class="c syntaxhl"><span class="CodeRay">$ sudo ./bin/ceph-osd -i <span class="integer">0</span> --mkfs --osd-uuid dca4b307-dfbf-<span class="integer">4</span>bd5-<span class="integer">9358</span>-<span class="integer">028</span>c6ad23b85 -c ceph.conf --no-mon-config
Starting SPDK v20<span class="float">.07</span> git sha1 <span class="integer">1</span>a527e501 / DPDK <span class="integer">2</span><span class="float">0</span><span class="float">.05</span><span class="float">.0</span> initialization...
[ DPDK EAL parameters: nvme-device-manager --no-shconf -c <span class="hex">0x1</span> -m <span class="integer">512</span> --log-level=lib.eal:<span class="integer">6</span> --log-level=lib.cryptodev:<span class="integer">5</span> --log-level=user1:<span class="integer">6</span> --iova-mode=pa --base-virtaddr=<span class="hex">0x200000000000</span> --match-allocations --file-prefix=spdk_pid1474 ]
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
EAL: Invalid NUMA socket, <span class="keyword">default</span> to <span class="integer">0</span>
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.829</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.845</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.849</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.857</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.861</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.873</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.885</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.885</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">7</span><span class="float">.885</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.013</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.185</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.189</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.189</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.189</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.269</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.417</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.421</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.433</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.437</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.445</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
<span class="integer">2024</span>-<span class="octal">03</span>-<span class="octal">05</span>T10:<span class="integer">21</span>:<span class="integer">5</span><span class="float">8</span><span class="float">.457</span>+<span class="integer">0800</span> <span class="float">7f</span><span class="integer">0</span><span class="float">4f</span><span class="integer">8</span>d75b40 -<span class="integer">1</span> bluestore(/var/lib/ceph/osd/ceph-<span class="integer">0</span>/block) _read_bdev_label unable to decode label at offset <span class="integer">0</span>: End of buffer [buffer:<span class="integer">2</span>]
</span></code></pre></p>
<p>After execute the command, /var/lib/ceph/osd/ceph-0 looks like this</p>
<pre><code class="text syntaxhl"><span class="CodeRay"># ll /var/lib/ceph/osd/ceph-0/
total 72
drwxr-xr-x 2 root root 4096 Mar 5 10:21 ./
drwxr-xr-x 3 root root 4096 Mar 1 10:16 ../
-rw------- 1 root root 8 Mar 5 10:21 bfm_blocks
-rw------- 1 root root 4 Mar 5 10:21 bfm_blocks_per_key
-rw------- 1 root root 5 Mar 5 10:21 bfm_bytes_per_block
-rw------- 1 root root 12 Mar 5 10:21 bfm_size
lrwxrwxrwx 1 root root 36 Mar 5 10:21 block -> 'spdk:trtype:PCIe traddr:0000:00:0e.0'
-rw------- 1 root root 2 Mar 5 10:21 bluefs
-rw------- 1 root root 37 Mar 5 10:21 ceph_fsid
-rw------- 1 root root 92 Mar 5 10:21 ceph_version_when_created
-rw------- 1 root root 28 Mar 5 10:21 created_at
-rw-r--r-- 1 root root 37 Mar 5 10:21 fsid
-rw------- 1 root root 8 Mar 5 10:21 kv_backend
-rw------- 1 root root 21 Mar 5 10:21 magic
-rw------- 1 root root 4 Mar 5 10:21 mkfs_done
-rw------- 1 root root 6 Mar 5 10:21 ready
-rw-r--r-- 1 root root 31 Mar 5 10:21 'spdk:trtype:PCIe traddr:0000:00:0e.0'
-rw------- 1 root root 10 Mar 5 10:21 type
-rw------- 1 root root 2 Mar 5 10:21 whoami
</span></code></pre>
<p>After debug code, I found the issue should be _write_bdev_label and _read_bdev_label don't execute on the right device. As there is no detail spdk configuration material, did I miss anything?</p>
<p>Thanks</p> bluestore - Backport #64648 (In Progress): squid: BlueStore/DeferredWriteTest.NewData/3 is brokenhttps://tracker.ceph.com/issues/646482024-02-29T16:14:26ZBackport Botbluestore - Backport #64647 (In Progress): reef: BlueStore/DeferredWriteTest.NewData/3 is brokenhttps://tracker.ceph.com/issues/646472024-02-29T16:14:18ZBackport Botbluestore - Backport #64592 (New): quincy: BlueFS: l_bluefs_log_compactions is counted twice in s...https://tracker.ceph.com/issues/645922024-02-27T09:42:56ZBackport Botbluestore - Backport #64591 (New): squid: BlueFS: l_bluefs_log_compactions is counted twice in sy...https://tracker.ceph.com/issues/645912024-02-27T09:42:46ZBackport Botbluestore - Backport #64590 (New): reef: BlueFS: l_bluefs_log_compactions is counted twice in syn...https://tracker.ceph.com/issues/645902024-02-27T09:42:39ZBackport Botbluestore - Bug #64567 (New): Expanding main device might cause leaked extents.https://tracker.ceph.com/issues/645672024-02-26T13:47:21ZIgor Fedotovigor.fedotov@croit.io
<p>This happens when freelist manager is in bitmap(not-null) mode and original block device size is not aligned with the manager's "per-key" block size.<br />E.g. having original block device size equal to 10240000000 and expanding it by 1G to <br />11313741824 bytes would cause fsck to complain on leaked extents.</p>
<p>This can be treated by ceph-bluestore-tool repair command though.</p> bluestore - Bug #64533 (Pending Backport): BlueFS: l_bluefs_log_compactions is counted twice in s...https://tracker.ceph.com/issues/645332024-02-22T06:53:35Zlinke wang
<p>When sync log compaction, in BlueFS::_compact_log_sync_LNF_LD, l_bluefs_log_compactions is first counted in _rewrite_log_and_layout_sync_LNF_LD.<br />After _rewrite_log_and_layout_sync_LNF_LD, l_bluefs_log_compactions is second counted.</p>
<p>@void BlueFS::_compact_log_sync_LNF_LD()
{<br /> dout(10) << <i>func</i> << dendl;<br /> uint8_t prefer_bdev;
{<br /> std::lock_guard ll(log.lock);<br /> prefer_bdev =<br /> vselector->select_prefer_bdev(log.writer->file->vselector_hint);<br /> }<br /> <ins>_rewrite_log_and_layout_sync_LNF_LD</ins>(true, //here is first counted<br /> BDEV_DB,<br /> prefer_bdev,<br /> prefer_bdev,<br /> 0,<br /> super.memorized_layout);<br /> <ins>logger->inc(l_bluefs_log_compactions)</ins>; // here is second counted<br />}<br />@</p> bluestore - Bug #64511 (Fix Under Review): kv/RocksDBStore: rocksdb_cf_compact_on_deletion has no...https://tracker.ceph.com/issues/645112024-02-20T21:25:22ZJoshua Baergen
<p>This setting is applied via update_column_family_options(), which is called for each configured CF but not for the default one (at least for upgraded systems that have not been resharded - I could be misunderstanding the flow for resharded/new systems). This is a big deal for upgraded systems since the bulk of rocksdb data lives in the default CF.</p>