Ceph » CephFS

Maybe also related: https://pulpito.ceph.com/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7612993

"2024-03-20T21:19:00.179790+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log

Patrick Donnelly wrote in #note-1:

Maybe also related: https://pulpito.ceph.com/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7612993

"2024-03-20T21:19:00.179790+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log

This job doesn't have the log in the ignore list.

Isn't this same as: https://tracker.ceph.com/issues/48562 ?

Venky Shankar wrote in #note-5:

Isn't this same as: https://tracker.ceph.com/issues/48562 ?

"object missing on disk" is not the issue here

Patrick Donnelly wrote:

https://pulpito.ceph.com/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7612910/

and many others.

More fallout after https://github.com/ceph/ceph/pull/55455 was merged.

This is the scrub error for the PARADOX item:

"raw_stats":{
    "checked":true,
    "passed":false,
    "read_ret_val":0,
    "ondisk_value.dirstat":"f(v0 m2024-03-20T19:09:04.770954+0000 25=25+0)",
    "ondisk_value.rstat":"n(v0 rc2024-03-20T19:09:04.804841+0000 b5230239 25=24+1)",
    "memory_value.dirstat":"f(v69 m2024-03-20T19:09:04.804841+0000 24=24+0)",
    "memory_value.rstat":"n(v7 rc2024-03-20T19:09:04.804841+0000 b5230239 25=24+1)",
    "error_str":"freshly-calculated rstats don't match existing ones" 
}

So the ondisk and memory dirstat (apart from the timestamp) seem to be off by 1

Milind Changire wrote in #note-6:

Venky Shankar wrote in #note-5:

Isn't this same as: https://tracker.ceph.com/issues/48562 ?

"object missing on disk" is not the issue here

Or at least the "object missing on disk" log doesn't show up in the cluster logs for the job listed in the description.

Milind Changire wrote in #note-8:

Milind Changire wrote in #note-6:

Venky Shankar wrote in #note-5:

Isn't this same as: https://tracker.ceph.com/issues/48562 ?

"object missing on disk" is not the issue here

Or at least the "object missing on disk" log doesn't show up in the cluster logs for the job listed in the description.

OK. So that was a separate issue which is fixed by adding the string to ignorelist. It got mentioned in https://tracker.ceph.com/issues/65020#note-1, so I inferred if this is the same problem.