Bug #37797
radosbench tests hit ENOSPC
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2019-01-06T03:35:08.079 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _do_alloc_write failed to allocate 0x400000 allocated 0x 1b4000 min_alloc_size 0x4000 available 0x 0 2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _do_write _do_alloc_write failed with (28) No space left on device 2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 3, counting from 0) 2019-01-06T03:35:08.080 INFO:tasks.ceph.osd.4.smithi031.stderr:2019-01-06 03:35:08.076 7fef30317700 -1 bluestore(/var/lib/ceph/osd/ceph-4) ENOSPC from bluestore, misconfigured cluster 2019-01-06T03:35:08.081 INFO:teuthology.orchestra.run.smithi174.stderr:Read #2:4fa7a8d7:::benchmark_data_smithi076_12105_object4576:head# 2019-01-06T03:35:08.085 INFO:tasks.ceph.osd.4.smithi031.stderr:/build/ceph-14.0.1-2331-g452e876/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7fef30317700 time 2019-01-06 03:35:08.080878 2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr:/build/ceph-14.0.1-2331-g452e876/src/os/bluestore/BlueStore.cc: 11025: abort() 2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: ceph version 14.0.1-2331-g452e876 (452e876a8f2a28b7210a9e7ef31636ddf5312d85) nautilus (dev) 2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xda) [0x82e848] 2019-01-06T03:35:08.086 INFO:tasks.ceph.osd.4.smithi031.stderr: 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0xc0c) [0xe176cc] ...
/a/sage-2019-01-05_23:50:21-rados-wip-sage4-testing-2019-01-05-1233-distro-basic-smithi/3429150
Related issues
History
#1 Updated by Neha Ojha about 5 years ago
- Status changed from 12 to Resolved
#2 Updated by Neha Ojha about 5 years ago
- Status changed from Resolved to Pending Backport
- Backport set to mimic
looks like we are hitting this in mimic as well: /a/yuriw-2019-02-06_16:30:03-rados-wip-yuri4-testing-2019-02-05-1539-mimic-distro-basic-smithi/3557601/
2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) _do_write _do_alloc_write failed with (28) No space left on device 2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 2, counting from 0) 2019-02-06T18:05:40.052 INFO:tasks.ceph.osd.7.smithi135.stderr:2019-02-06 18:05:40.051 7f4f53232700 -1 bluestore(/var/lib/ceph/osd/ceph-7) ENOSPC from bluestore, misconfigured cluster 2019-02-06T18:05:40.053 INFO:tasks.ceph.osd.7.smithi135.stderr:/build/ceph-13.2.4-304-g5e6d1de/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f4f53232700 time 2019-02-06 18:05:40.053314 2019-02-06T18:05:40.053 INFO:tasks.ceph.osd.7.smithi135.stderr:/build/ceph-13.2.4-304-g5e6d1de/src/os/bluestore/BlueStore.cc: 10186: FAILED assert(0 == "unexpected error") 2019-02-06T18:05:40.054 INFO:tasks.ceph.osd.7.smithi135.stderr: ceph version 13.2.4-304-g5e6d1de (5e6d1de370e8238cc4f2b6c7fad087d0d8f1bbdd) mimic (stable)
#3 Updated by Neha Ojha about 5 years ago
- Backport changed from mimic to luminous,mimic
#4 Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38239: mimic: radosbench tests hit ENOSPC added
#5 Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38240: luminous: radosbench tests hit ENOSPC added
#6 Updated by Nathan Cutler about 5 years ago
- Status changed from Pending Backport to Resolved
#7 Updated by Neha Ojha almost 5 years ago
This one appeared again.
/a/nojha-2019-04-25_05:43:35-rados-wip-39441-distro-basic-smithi/3892141/