Ceph » RADOS

There is no printing information in this code, but it should be the min epoch taken from here.
@epoch_t OSDMonitor::get_min_last_epoch_clean() const {
auto floor = last_epoch_clean.get_lower_bound(osdmap);
// also scan osd epochs
// don't trim past the oldest reported osd epoch
for (auto [osd, epoch] : osd_epochs) {
if (epoch < floor) {
floor = epoch;
}
}
return floor;
}@

Another phenomenon is that the number of epochs in OSDMAP has been growing rapidly, perhaps because all the newly added epochs in the future are not normal clean, resulting in the last minute epoch clean not being updated？

hi Matan,
thank you for your reply. Yes, I checked the ceph report and there are some osds in the output and their "osd_epochs" is 237271.
Some of these OSDs are in the current cluster, and some have been deleted a long time ago, which is strange. Restarting these osds could not bring the ceph monitor back to normal, so we finally restarted the ceph leader monitor.
I saw the patch you submitted and I hope it can solve this problem. If you have any other questions, feel free to communicate. I have recorded the cluster information when the exception occurred. Thank you very much.

Matan Breizman wrote:

Thank you for the report!
Can you please share the `osd_epochs` section from ceph report?

In the ceph report, some clusters have deleted OSDs that do not exist, but the report will show it.
Is it related to this submission? https://github.com/ceph/ceph/pull/44303

In the ceph report, some clusters have deleted OSDs that do not exist, but the report will show it.

The issue is that when the OSD was marked as out it was erased from "osd_epochs".
However, the OSD may have continued to send beacons and therefore get added back to the map as a stale OSD (Since it won't be erased again).

Restarting the monitor worked because a new "osd_epochs" map was created, without the removed OSDs.
Alternatively, re-adding and marking the OSD as out would perhaps trigger the erasure from the map again.

Is it related to this submission? https://github.com/ceph/ceph/pull/44303

Not exactly, PR#44303 is (mostly) a cleanup since the the erasure which was removed is redundant.
Thank you again for the tracker.

Matan Breizman wrote:

In the ceph report, some clusters have deleted OSDs that do not exist, but the report will show it.

The issue is that when the OSD was marked as out it was erased from "osd_epochs".
However, the OSD may have continued to send beacons and therefore get added back to the map as a stale OSD (Since it won't be erased again).

Restarting the monitor worked because a new "osd_epochs" map was created, without the removed OSDs.
Alternatively, re-adding and marking the OSD as out would perhaps trigger the erasure from the map again.

Is it related to this submission? https://github.com/ceph/ceph/pull/44303

Not exactly, PR#44303 is (mostly) a cleanup since the the erasure which was removed is redundant.
Thank you again for the tracker.

thanks (: