Project

General

Profile

Actions

Bug #58434

closed

Widespread metadata corruption

Added by Nathan Fish about 1 year ago. Updated about 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Yes
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
fs, kcephfs
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

One of our CephFS volumes has become corrupt, with user files being substituted for each other with no clear pattern; eg User A's ~/.ssh/id_rsa, when read, may return the contents of another user's id_rsa; User B's id_rsa may return a different user's ~/.profile, etc. The returned data sometimes includes the contents of multiple files, and is interspersed with null bytes in some cases, which leads me to believe Ceph is retrieving the wrong raw data blocks rather than simply swapping file handles. This is of course causing a major security incident.

Please let me know what debug data would be useful. Thanks.

Actions #1

Updated by Nathan Fish about 1 year ago

Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.

ACK. Please update the ticket when you have more information.

Actions #2

Updated by Venky Shankar about 1 year ago

Nathan Fish wrote:

Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.

ACK. Please update the ticket when you have more information.

Nathan, ping?

Actions #3

Updated by Nathan Fish about 1 year ago

Venky Shankar wrote:

Nathan Fish wrote:

Further evidence indicates the issue may not be with Ceph; possibly a bug in account management. Still investigating.

ACK. Please update the ticket when you have more information.

Nathan, ping?

Sorry, didn't see your replies. A coworker has just tracked down the file overwrites to an old script in our account management system. I'm not sure how it managed to cause corruption looking exactly like a filesystem problem, but we can conclude it's not a CephFS issue.

Thanks.

Actions #4

Updated by Venky Shankar about 1 year ago

  • Status changed from New to Rejected
Actions

Also available in: Atom PDF