Ceph » Orchestrator

I got the same problem.
Fistly tried to upgrade from 17.2.6 to 17.2.7.
Secondly reverted from 17.2.7 to 17.2.6 and upgraded to 18.2.1.
18.2.1 has the same error.
So I temperary muted this error:

ceph health mute CEPHADM_REFRESH_FAILED

I hope fix will come fast.

Harry Coin wrote:

A bug reported long ago, apparently fixed, has re-appeared when migrating within Quincy to 17.2.7

On Tue, 7 Nov 2023, Harry G Coin wrote:

These repeat for every host, only after upgrading from prev release Quincy to 17.2.7. As a result, the cluster is always warned, never indicates healthy.

I'm hitting this error, too.

"/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition
/usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part'
/usr/bin/docker: stderr KeyError: 'TYPE'

Variable names indicate usage of BLKID. It seems that `blkid` usually returns TYPE="something", but I have devices without TYPE:

/dev/mapper/data-4d323729--8fec--42c6--a1da--bacdea89fb37.disk0_data: PTUUID="c2901603-fae8-45cb-86fe-13d02e6b6dc6" PTTYPE="gpt"
/dev/mapper/data-8d485122--d8ca--4e11--85bb--3f795a4e31e9.disk0_data: PTUUID="2bc7a15e" PTTYPE="dos"
/dev/drbd3: PTUUID="2bc7a15e" PTTYPE="dos"

Maybe this indicates why the key is missing?

Please tell me if there is anything I can do to find the root cause.

Thanks, Sascha.

Detail when it appeared previously, but only in tests and not in production: https://tracker.ceph.com/issues/56573

Remains in 18.2.1

Remains in 18.2.2

Vadym Kukharenko wrote in #note-1:

I got the same problem.
Fistly tried to upgrade from 17.2.6 to 17.2.7.
Secondly reverted from 17.2.7 to 17.2.6 and upgraded to 18.2.1.
18.2.1 has the same error.
So I temperary muted this error:
[...]

I hope fix will come fast.

Harry Coin wrote:

A bug reported long ago, apparently fixed, has re-appeared when migrating within Quincy to 17.2.7

On Tue, 7 Nov 2023, Harry G Coin wrote:

These repeat for every host, only after upgrading from prev release Quincy to 17.2.7. As a result, the cluster is always warned, never indicates healthy.

I'm hitting this error, too.

"/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 482, in is_partition
/usr/bin/docker: stderr return self.blkid_api['TYPE'] == 'part'
/usr/bin/docker: stderr KeyError: 'TYPE'

Variable names indicate usage of BLKID. It seems that `blkid` usually returns TYPE="something", but I have devices without TYPE:

/dev/mapper/data-4d323729--8fec--42c6--a1da--bacdea89fb37.disk0_data: PTUUID="c2901603-fae8-45cb-86fe-13d02e6b6dc6" PTTYPE="gpt"
/dev/mapper/data-8d485122--d8ca--4e11--85bb--3f795a4e31e9.disk0_data: PTUUID="2bc7a15e" PTTYPE="dos"
/dev/drbd3: PTUUID="2bc7a15e" PTTYPE="dos"

Maybe this indicates why the key is missing?

Please tell me if there is anything I can do to find the root cause.

Thanks, Sascha.

Detail when it appeared previously, but only in tests and not in production: https://tracker.ceph.com/issues/56573

Fix didn't come yet, so I found workaround.
I use nvme disk for fast pools and this errror related to nvme disks.
So I removed osd and made zap and create osd again.
Before zapping disk, you must make lvremove for this disk on needed host.