Bug #24238: test gets ENOSPC from bluestore block device - CephFS - Ceph

Actions

Copy link

Bug #24238

closed

test gets ENOSPC from bluestore block device

Added by Patrick Donnelly almost 6 years ago. Updated over 5 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sage Weil

Category:

Target version:

Ceph - v14.0.0

% Done:

Source:

Q/A

Tags:

Backport:

mimic,luminous

Regression:

Severity:

4 - irritation

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

qa-suite

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2018-05-21T14:07:29.887 INFO:tasks.ceph.osd.2.ovh020.stderr:2018-05-21 14:07:29.865 7f4ff0340700 -1 bdev(0x560bb5744380 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-28 ((28) No space left on device)
2018-05-21T14:07:29.888 INFO:tasks.ceph.osd.2.ovh020.stderr:/build/ceph-13.1.0-148-gec1d6e7/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4ff0340700 time 2018-05-21 14:07:29.870240
2018-05-21T14:07:29.888 INFO:tasks.ceph.osd.2.ovh020.stderr:/build/ceph-13.1.0-148-gec1d6e7/src/os/bluestore/KernelDevice.cc: 463: FAILED assert(0 == "got unexpected error from aio_t::get_return_value. " "This may suggest HW issue. Please check your dmesg!")
2018-05-21T14:07:29.909 INFO:tasks.ceph.osd.2.ovh020.stderr: ceph version 13.1.0-148-gec1d6e7 (ec1d6e76fab51bc357eacee86d5da01866a30deb) mimic (rc)
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f4ffc49aac2]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 2: (()+0x2e1c87) [0x7f4ffc49ac87]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 3: (KernelDevice::_aio_thread()+0x506) [0x560bb2952ea6]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 4: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560bb29572dd]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: 5: (()+0x76ba) [0x7f4ffaf466ba]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: 6: (clone()+0x6d) [0x7f4ffa76f41d]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

From: /ceph/teuthology-archive/teuthology-2018-05-19_05:10:01-fs-mimic-distro-basic-ovh/2552550/teuthology.log

Note: there is probably a deeper problem here.

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Sage Weil almost 6 years ago

Project changed from bluestore to RADOS
Subject changed from KernelDevice: doesn't handle ENOSPC in async i/o? to osd: hit ENOSPC without stopping io
Status changed from New to 12

Actions

Copy link

Updated by Sage Weil almost 6 years ago

Project changed from RADOS to CephFS
Subject changed from osd: hit ENOSPC without stopping io to test gets ENOSPC from bluestore block device

The underlying block device was thinly provisioned and ran out of space. Asserting on ENOSPC from a block IO is expected behavior.

The test in this case probably isn't creating a large enough device for the test.

Actions

Copy link