Project

General

Profile

Actions

Bug #14289

closed

"No space left on device" + "FAILED assert(0 == "unexpected error")" in rados-infernalis-distro-basic-openstack

Added by Yuri Weinstein over 8 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We probably need to change this test configuration to increase the space, and not sure why we still got the assert.

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-06_21:00:01-rados-infernalis-distro-basic-openstack/
Job: 61156
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-01-06_21:00:01-rados-infernalis-distro-basic-openstack/61156/teuthology.log

2016-01-07T02:03:07.004 INFO:tasks.ceph.osd.0.target085133.stderr:problem writing to /var/log/ceph/ceph-osd.0.log: (28) No space left on device
2016-01-07T02:03:07.012 INFO:tasks.ceph.osd.1.target085133.stderr:problem writing to /var/log/ceph/ceph-osd.1.log: (28) No space left on device
2016-01-07T02:03:07.015 INFO:tasks.ceph.osd.2.target085133.stderr:problem writing to /var/log/ceph/ceph-osd.2.log: (28) No space left on device
2016-01-07T02:03:07.021 INFO:tasks.ceph.osd.0.target085133.stderr:problem writing to /var/log/ceph/ceph-osd.0.log: (28) No space left on device
2016-01-07T02:03:07.023 INFO:tasks.ceph.osd.1.target085133.stderr:os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f2ddddda700 time 2016-01-07 02:03:06.898684
2016-01-07T02:03:07.023 INFO:tasks.ceph.osd.1.target085133.stderr:os/FileStore.cc: 2855: FAILED assert(0 == "unexpected error")
2016-01-07T02:03:07.024 INFO:tasks.ceph.osd.2.target085133.stderr:os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f740f2c8700 time 2016-01-07 02:03:06.872168
2016-01-07T02:03:07.024 INFO:tasks.ceph.osd.2.target085133.stderr:os/FileStore.cc: 2855: FAILED assert(0 == "unexpected error")
2016-01-07T02:03:07.025 INFO:tasks.ceph.osd.2.target085133.stderr: ceph version 9.2.0-27-g64fab4f (64fab4f080c5b7d79b51b8db8c71428b507efec5)
2016-01-07T02:03:07.025 INFO:tasks.ceph.osd.2.target085133.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f741ddf127b]
2016-01-07T02:03:07.026 INFO:tasks.ceph.osd.2.target085133.stderr: 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa50) [0x7f741daca060]
2016-01-07T02:03:07.026 INFO:tasks.ceph.osd.2.target085133.stderr: 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x7f741dad13c4]
2016-01-07T02:03:07.027 INFO:tasks.ceph.osd.2.target085133.stderr: 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x1a9) [0x7f741dad1589]
2016-01-07T02:03:07.027 INFO:tasks.ceph.osd.2.target085133.stderr: 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0x7f741dde2cf6]
2016-01-07T02:03:07.028 INFO:tasks.ceph.osd.2.target085133.stderr: 6: (ThreadPool::WorkThread::entry()+0x10) [0x7f741dde3bc0]
2016-01-07T02:03:07.028 INFO:tasks.ceph.osd.2.target085133.stderr: 7: (()+0x8182) [0x7f741c40f182]
2016-01-07T02:03:07.028 INFO:tasks.ceph.osd.2.target085133.stderr: 8: (clone()+0x6d) [0x7f741a75647d]
2016-01-07T02:03:07.028 INFO:tasks.ceph.osd.2.target085133.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open1 closed)

Related to Ceph - Bug #14242: "FileStore.cc: In function 'unsigned int" in rados-jewel-distro-basic-smithiCan't reproduce01/05/2016

Actions
Actions #1

Updated by Yuri Weinstein over 8 years ago

  • Related to Bug #14242: "FileStore.cc: In function 'unsigned int" in rados-jewel-distro-basic-smithi added
Actions #3

Updated by Sage Weil over 8 years ago

  • Status changed from New to Resolved
Actions #6

Updated by Samuel Just over 8 years ago

  • Priority changed from Normal to Urgent
Actions #7

Updated by Sage Weil about 8 years ago

  • Status changed from New to Resolved

we've shorteend the rados bench durations

Actions

Also available in: Atom PDF