Bug #58434
closedWidespread metadata corruption
0%
Description
One of our CephFS volumes has become corrupt, with user files being substituted for each other with no clear pattern; eg User A's ~/.ssh/id_rsa, when read, may return the contents of another user's id_rsa; User B's id_rsa may return a different user's ~/.profile, etc. The returned data sometimes includes the contents of multiple files, and is interspersed with null bytes in some cases, which leads me to believe Ceph is retrieving the wrong raw data blocks rather than simply swapping file handles. This is of course causing a major security incident.
Please let me know what debug data would be useful. Thanks.
Updated by Nathan Fish over 1 year ago
Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.
ACK. Please update the ticket when you have more information.
Updated by Venky Shankar over 1 year ago
Nathan Fish wrote:
Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.
ACK. Please update the ticket when you have more information.
Nathan, ping?
Updated by Nathan Fish about 1 year ago
Venky Shankar wrote:
Nathan Fish wrote:
Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.
ACK. Please update the ticket when you have more information.
Nathan, ping?
Sorry, didn't see your replies. A coworker has just tracked down the file overwrites to an old script in our account management system. I'm not sure how it managed to cause corruption looking exactly like a filesystem problem, but we can conclude it's not a CephFS issue.
Thanks.