Project

General

Profile

Bug #24238

test gets ENOSPC from bluestore block device

Added by Patrick Donnelly 7 months ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
05/23/2018
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
mimic,luminous
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
qa-suite
Labels (FS):
qa
Pull request ID:

Description

2018-05-21T14:07:29.887 INFO:tasks.ceph.osd.2.ovh020.stderr:2018-05-21 14:07:29.865 7f4ff0340700 -1 bdev(0x560bb5744380 /var/lib/ceph/osd/ceph-2/block) _aio_thread got r=-28 ((28) No space left on device)
2018-05-21T14:07:29.888 INFO:tasks.ceph.osd.2.ovh020.stderr:/build/ceph-13.1.0-148-gec1d6e7/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4ff0340700 time 2018-05-21 14:07:29.870240
2018-05-21T14:07:29.888 INFO:tasks.ceph.osd.2.ovh020.stderr:/build/ceph-13.1.0-148-gec1d6e7/src/os/bluestore/KernelDevice.cc: 463: FAILED assert(0 == "got unexpected error from aio_t::get_return_value. " "This may suggest HW issue. Please check your dmesg!")
2018-05-21T14:07:29.909 INFO:tasks.ceph.osd.2.ovh020.stderr: ceph version 13.1.0-148-gec1d6e7 (ec1d6e76fab51bc357eacee86d5da01866a30deb) mimic (rc)
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f4ffc49aac2]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 2: (()+0x2e1c87) [0x7f4ffc49ac87]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 3: (KernelDevice::_aio_thread()+0x506) [0x560bb2952ea6]
2018-05-21T14:07:29.910 INFO:tasks.ceph.osd.2.ovh020.stderr: 4: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560bb29572dd]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: 5: (()+0x76ba) [0x7f4ffaf466ba]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: 6: (clone()+0x6d) [0x7f4ffa76f41d]
2018-05-21T14:07:29.911 INFO:tasks.ceph.osd.2.ovh020.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

From: /ceph/teuthology-archive/teuthology-2018-05-19_05:10:01-fs-mimic-distro-basic-ovh/2552550/teuthology.log

Note: there is probably a deeper problem here.


Related issues

Duplicated by Ceph - Bug #24237: mon: monitors run out of space during snapshot workflow Duplicate 05/23/2018
Copied to fs - Backport #24758: mimic: test gets ENOSPC from bluestore block device Resolved
Copied to fs - Backport #24759: luminous: test gets ENOSPC from bluestore block device In Progress

History

#1 Updated by Sage Weil 7 months ago

  • Project changed from bluestore to RADOS
  • Subject changed from KernelDevice: doesn't handle ENOSPC in async i/o? to osd: hit ENOSPC without stopping io
  • Status changed from New to Verified

#2 Updated by Sage Weil 7 months ago

  • Project changed from RADOS to fs
  • Subject changed from osd: hit ENOSPC without stopping io to test gets ENOSPC from bluestore block device

The underlying block device was thinly provisioned and ran out of space. Asserting on ENOSPC from a block IO is expected behavior.

The test in this case probably isn't creating a large enough device for the test.

#3 Updated by Sage Weil 7 months ago

  • Status changed from Verified to Need Review

#4 Updated by Patrick Donnelly 7 months ago

  • Duplicated by Bug #24237: mon: monitors run out of space during snapshot workflow added

#5 Updated by Patrick Donnelly 7 months ago

  • Assignee set to Sage Weil
  • Target version changed from v13.2.0 to v14.0.0
  • Severity changed from 2 - major to 4 - irritation
  • Component(FS) qa-suite added
  • Labels (FS) qa added

#6 Updated by Patrick Donnelly 5 months ago

  • Status changed from Need Review to Pending Backport

#7 Updated by Patrick Donnelly 5 months ago

  • Copied to Backport #24758: mimic: test gets ENOSPC from bluestore block device added

#8 Updated by Patrick Donnelly 5 months ago

  • Copied to Backport #24759: luminous: test gets ENOSPC from bluestore block device added

Also available in: Atom PDF