Ceph » RADOS

If cluster is partially upgraded and on a particular host :

In baremetal environment, all daemons on host shows same version even though some daemons are restart and some are yet to be restarted.
In containerized environment, all daemons on host inherits last service daemon ceph version as host version.

Assume if ceph mon/mgr container is upgraded on host which is co-hosting osd, for all daemons the ceph version in prometheus metrics is reportedly same whereas ideally it should be depends on ceph version on which daemon is running.

For example (in containerized env), if we have upgraded mgr on testnode046 from ceph version 16.2.7-82.el8cp to ceph version 16.2.7-88.el8cp
the prometheus metrics reports below metadata information for mon and mgr

# TYPE ceph_mon_metadata untyped
...
ceph_mon_metadata{ceph_daemon="mon.testnode046",hostname="testnode046",public_addr="10.0.0.46",rank="1",ceph_version="ceph version 16.2.7-82.el8cp (f62bcd07da5ce2cb1e10c300819e14ccf8f2104b) pacific (stable)"} 1.0
..
# HELP ceph_mgr_metadata MGR metadata
# TYPE ceph_mgr_metadata gauge
ceph_mgr_metadata{ceph_daemon="mgr.testnode046",hostname="testnode046",ceph_version="ceph version 16.2.7-82.el8cp (f62bcd07da5ce2cb1e10c300819e14ccf8f2104b) pacific (stable)"} 1.0

which should be :

# TYPE ceph_mon_metadata untyped
...
ceph_mon_metadata{ceph_daemon="mon.testnode046",hostname="testnode046",public_addr="10.0.0.46",rank="1",ceph_version="ceph version 16.2.7-82.el8cp (f62bcd07da5ce2cb1e10c300819e14ccf8f2104b) pacific (stable)"} 1.0
..
# HELP ceph_mgr_metadata MGR metadata
# TYPE ceph_mgr_metadata gauge
ceph_mgr_metadata{ceph_daemon="mgr.testnode046",hostname="testnode046",ceph_version="ceph version 16.2.7-88.el8cp (40253637381fbc86b3bffdaa2dddbedda58aed44) pacific (stable)"} 1.0

This applies to OSD daemons as well.