Project

General

Profile

Actions
Copy link

Bug #8453

closed

OSD crash "error (28) No space left on device not handled on operation 10 (31417988.0.5, or op 5, counting from 0)" but disk space is available

Added by Henrik Korkuc almost 10 years ago. Updated almost 10 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

hello,
tonight one of OSDs crashed with "error (28) No space left on device not handled on operation 10 (31417988.0.5, or op 5, counting from 0)". But according to monitoring disk space didn't run out.

After manual start OSD still crashes with same error (some integers differ) during journal reply. There are >100GB of free disk space, inode usage is about 2%.

I am attaching 2 log files:
ceph-osd.12.log.1.gz - original log with crash (at ~2:08)
ceph-osd.12.log.gz - log with "debug filestore = 15" in ceph.conf. Truncated before OSD launch.

I am using 0.80.1 (ubuntu 14.04 from proposed repo), OSD deployed with ceph-deploy. OSD was running with "filestore max sync interval = 18" setting

Files

Download all files
ceph-osd.12.log.gz (938 KB) ceph-osd.12.log.gz log with "debug filestore = 15" in ceph.conf. Truncated before OSD launch. Henrik Korkuc, 05/28/2014 03:11 AM
ceph-osd.12.log.1.gz (7.25 MB) ceph-osd.12.log.1.gz original log with crash (at ~2:08) Henrik Korkuc, 05/28/2014 03:11 AM
Actions
Copy link

Also available in: Atom PDF