Ceph » mgr » Dashboard

OSD host details and overview do not show any data anymore¶

This is due to a regression introduced in https://github.com/ceph/ceph/pull/41221, which was meant to fix an matching problem if multiple OSDs use a single device (NVMe).

Environment¶

• Prometheus
  • node_disk_writes_completed_total metric
  • ceph_disk_occupation metric
  • possibly other metrics as well, which are not all required to reproduce the issue

How reproducible¶

Steps:

1. Use the following two metrics to perform a query

ceph_disk_occupation{ceph_daemon="osd.99",db_device="/dev/dm-7",device="/dev/dm-1",instance="foo.ceph",job="ceph-mgr"} 1.0
node_disk_writes_completed_total{device="dm-1",instance="foo.ceph:9100",job="node-exporter"} 93809050.0

1. Perform query (does not yield a result)

Note that I removed the filtering for a particular instance from the queries.

label_replace(
 (
 irate(node_disk_writes_completed{}[5m]) or
 irate(node_disk_writes_completed_total{}[5m])
 ),
 "instance",
 "$1",
 "instance",
 "([^:.]*).*" 
)
* on(instance, device, ceph_daemon) group_left
 label_replace(
 label_replace(
 ceph_disk_occupation,
 "device",
 "$1",
 "device",
 "/dev/(.*)" 
 ),
 "instance",
 "$1",
 "instance",
 "([^:.]*).*" 
 )

1. Perform adapted query (yields result)

label_replace(
 (
 irate(node_disk_writes_completed{}[5m]) or
 irate(node_disk_writes_completed_total{}[5m])
 ),
 "instance",
 "$1",
 "instance",
 "([^:.]*).*" 
)
* on(instance, device) group_left(ceph_daemon)
 label_replace(
 label_replace(
 ceph_disk_occupation,
 "device",
 "$1",
 "device",
 "/dev/(.*)" 
 ),
 "instance",
 "$1",
 "instance",
 "([^:.]*).*" 
 )

Returns

{ceph_daemon="osd.99", device="dm-1", instance="foo", job="node-exporter"}

Additional info¶

According to the Prometheus documentation, the labels declared in parentheses directly following an `on` are used to join two metrics. As the query was adapted in a previous PR to additionally join on the `ceph_daemon` label, I cannot explain how this would have worked other than if both sides had the `ceph_daemon` label present. However, as `ceph_disk_occupation` is only a metadata metric that is seemingly used to get that label from one side to the other, I cannot explain how this works.