Bug #18037: leveldb corruption leads to "Operation not permitted not handled" and assert - Ceph - Ceph

Actions

Copy link

Bug #18037

closed

leveldb corruption leads to "Operation not permitted not handled" and assert

Added by Nathan Cutler over 7 years ago. Updated about 7 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Nathan Cutler

Category:

Target version:

% Done:

Source:

Tags:

Backport:

jewel,kraken

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

we have a jewel OSD that doesn't start due to leveldb corruption (looks like). We upped the debug levels and can see filestore emitting an odd message "Operation not permitted not handled", followed "unexpected error code" and a dump of the operation, followed by assert(0 == "unexpected error").

Detailed log attached.

Files

ceph-osd.14.log (141 KB) ceph-osd.14.log

Nathan Cutler, 11/24/2016 04:41 PM

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

ceph version 10.2.2

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Related to Bug #16257: Ceph random bug is killing osds (os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error") added

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Description updated (diff)

EPERM

Operation not permitted; only the owner of the file
(or other resource) or processes with special privileges
can perform the operation.

Hm. This OSD had XFS corruption which was repaired. Maybe the xfs_repair restored a file, but left it owned by root? I'll try "chown -R ceph.ceph /var/lib/osd/..." and report back.

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Here is the code: https://github.com/ceph/ceph/blob/jewel/src/os/filestore/FileStore.cc#L2880-L2899

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Status changed from New to In Progress
Assignee set to Nathan Cutler

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Status changed from In Progress to Fix Under Review
Backport set to jewel

This PR improves the error message: https://github.com/ceph/ceph/pull/12181

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

This OSD had XFS corruption which was repaired. Maybe the xfs_repair restored a file, but left it owned by root? I'll try "chown -R ceph.ceph /var/lib/osd/..." and report back.

I just got word that the recursive chown does not help.

Actions

Copy link