Ceph

Regarding

# rados list-inconsistent-obj 2.87       
No scrub information available for pg 2.87
error 2: (2) No such file or directory

this seems to be this bug which was closed as "Can't reproduce" immediately 7 years ago, with many people reproducing it afterwards:

https://tracker.ceph.com/issues/15781

Could somebody reopen it?

Some more troubles in the related page https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/#pgs-inconsistent

There is only one consistent state, but in the worst case, we could have different inconsistencies in multiple perspectives found in more than one objects.

Again this seems to refer to unintroduced terminology.

• What's a "perspective" found in more than one object?

You can repair the inconsistent placement group by executing:

ceph pg repair {placement-group-ID}

Which overwrites the bad copies with the authoritative ones. In most cases, Ceph is able to choose authoritative copies from all available replicas using some predefined criteria. But this does not always work. For example, the stored data digest could be missing, and the calculated digest will be ignored when choosing the authoritative copies. So, please use the above command with caution.

First it tells to try `ceph pg repair`. But then it also says "use it with caution". It describes in which case caution is needed, but it does not explain how I can check whether this case is occurring in my cluster. Which commands to run, what logs to check?

Currently it's a bit "this command may help or screw you, but we don't tell you how you to check which one it will be".

It would be also nice to say what one should do when one has a replicated pool.

I found one suggested answer here:

https://ceph-users.ceph.narkive.com/2OSM6EuW/scrub-error-how-does-ceph-pg-repair-work#post7

In the case that an inconsistent PG is caused by afailed disk read, you don't need to run ceph pg repair at all.
Instead, since your drive is bad, stop the osd process, mark that osd out.
After backfilling has completed and the PG is re-scrubbed, you will find it is consistent again.

In the same thread https://ceph-users.ceph.narkive.com/2OSM6EuW/scrub-error-how-does-ceph-pg-repair-work

somebody is asking another question that I think should be documented:

What impact does it have on the cluster when an OSD is in "inconsistent" state?

Does that mean that any IO to this pg is blocked until it is repaired?

There is no answer in the thread.

I would imagine that it should not have an impact given that there are other replicas of the data, but it would be good for the docs to explain it explicitly.

From the post above, I think it also should be made super clear what Ceph is going do by itself to get rid of the inconsistency, and when the user needs to do something.

I learned more info now.

The information in the "Repairing PG Inconsistencies" page is misleading.

In the thread

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/GFUPBQAPKNKFTV3MPCFQZPE6J2H2NFYC/#X34ZMOUO37UNTUWSGO2PN2MMCHUTXWFU

I discovered that the correct way to fix a PG inconsistency after replacing a failed disk is to simply ensure that a scrub runs.

The issue there was that the `ceph pg deep-scrub` I invoked was never scheduled because Ceph always picked some other scrub to do first on the relevant OSD.
Increasing `osd_max_scrubs` beyond 1 made it possible to force the scrub to start.

I conclude that most of the information online, including the Ceph docs, does not give the correct advice when recommending `ceph pg repair`.
Instead, the docs should make clear that a scrub will fix such issues without involvement of `ceph pg repair`.

I think the docs should be improved, because a disk failing and being replaced is an extremely common operation for storage cluster.

I would usually make a PR to improve them, but unfortunately I still don't understand 90% of the rest of this docs section, so I do not feel equipped for it.