Project

General

Profile

Bug #37797

radosbench tests hit ENOSPC

Added by Sage Weil about 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

2019-01-06T03:35:08.079 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _do_alloc_write failed to allocate 0x400000 allocated 0x 1b4000 min_alloc_size 0x4000 available 0x 0
2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _do_write _do_alloc_write failed with (28) No space left on device
2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 3, counting from 0)
2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) ENOSPC from bluestore, misconfigured cluster
2019-01-06T03:35:08.081 INFO:teuthology.orchestra.run.smithi174.stderr:Read #2:4fa7a8d7:::benchmark_data_smithi076_12105_object4576:head#
2019-01-06T03:35:08.085 INFO:tasks.ceph.osd.4.smithi031.stderr:/build/ceph-14.0.1-2331-g452e876/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7fef30317700 time 2019-01-06 03:35:08.080878
2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr:/build/ceph-14.0.1-2331-g452e876/src/os/bluestore/BlueStore.cc: 11025: abort()
2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: ceph version 14.0.1-2331-g452e876 (452e876a8f2a28b7210a9e7ef31636ddf5312d85) nautilus (dev)
2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xda) [0x82e848]
2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0xc0c) [0xe176cc]
...

/a/sage-2019-01-05_23:50:21-rados-wip-sage4-testing-2019-01-05-1233-distro-basic-smithi/3429150

Related issues

Copied to RADOS - Backport #38239: mimic: radosbench tests hit ENOSPC Resolved
Copied to RADOS - Backport #38240: luminous: radosbench tests hit ENOSPC Resolved

History

#1 Updated by Neha Ojha about 1 year ago

  • Status changed from 12 to Resolved

#2 Updated by Neha Ojha 12 months ago

  • Status changed from Resolved to Pending Backport
  • Backport set to mimic

looks like we are hitting this in mimic as well: /a/yuriw-2019-02-06_16:30:03-rados-wip-yuri4-testing-2019-02-05-1539-mimic-distro-basic-smithi/3557601/

2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) _do_write _do_alloc_write failed with (28) No space left on device
2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 2, counting from 0)
2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) ENOSPC from bluestore, misconfigured cluster
2019-02-06T18:05:40.053 INFO:tasks.ceph.osd.7.smithi135.stderr:/build/ceph-13.2.4-304-g5e6d1de/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f4f53232700 time 2019-02-06 18:05:40.053314
2019-02-06T18:05:40.053 INFO:tasks.ceph.osd.7.smithi135.stderr:/build/ceph-13.2.4-304-g5e6d1de/src/os/bluestore/BlueStore.cc: 10186: FAILED assert(0 == "unexpected error")
2019-02-06T18:05:40.054 INFO:tasks.ceph.osd.7.smithi135.stderr: ceph version 13.2.4-304-g5e6d1de (5e6d1de370e8238cc4f2b6c7fad087d0d8f1bbdd) mimic (stable)

#3 Updated by Neha Ojha 12 months ago

  • Backport changed from mimic to luminous,mimic

#4 Updated by Nathan Cutler 11 months ago

#5 Updated by Nathan Cutler 11 months ago

#6 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

#7 Updated by Neha Ojha 9 months ago

This one appeared again.
/a/nojha-2019-04-25_05:43:35-rados-wip-39441-distro-basic-smithi/3892141/

Also available in: Atom PDF