Bug #13490: "FAILED assert(!m_filestore_fail_eio || r != -5)" in upgrade:hammer-x-infernalis-distro-basic-multi - Ceph - Ceph

Actions

Copy link

Bug #13490

closed

"FAILED assert(!m_filestore_fail_eio || r != -5)" in upgrade:hammer-x-infernalis-distro-basic-multi

Added by Yuri Weinstein over 8 years ago. Updated over 8 years ago.

Status:

Rejected

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

upgrade/hammer-x

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-10-11_17:02:07-upgrade:hammer-x-infernalis-distro-basic-multi/
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-11_17:02:07-upgrade:hammer-x-infernalis-distro-basic-multi/1101035/teuthology.log

2015-10-13T19:55:37.801 INFO:tasks.ceph.osd.12.mira103.stderr:2015-10-13 19:55:37.790590 7fd4f79b7700 -1 filestore(/var/lib/ceph/osd/ceph-12) FileStore::_do_copy_range: read error at 1679360~2521088, (5) Input/output error
2015-10-13T19:55:37.803 INFO:tasks.ceph.osd.12.mira103.stderr:os/FileStore.cc: In function 'int FileStore::_clone(coll_t, const ghobject_t&, const ghobject_t&, const SequencerPosition&)' thread 7fd4f79b7700 time 2015-10-13 19:55:37.790779
2015-10-13T19:55:37.803 INFO:tasks.ceph.osd.12.mira103.stderr:os/FileStore.cc: 3178: FAILED assert(!m_filestore_fail_eio || r != -5)
2015-10-13T19:55:37.803 INFO:tasks.ceph.osd.12.mira103.stderr: ceph version 0.94.3-290-g7f485ed (7f485ed5aa620fe982561663bf64356b7e2c38f2)
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc633b]
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 2: (FileStore::_clone(coll_t, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0xa4d) [0x91d23d]
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2628) [0x926108]
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x92abc4]
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x92ad4a]
2015-10-13T19:55:37.804 INFO:tasks.ceph.osd.12.mira103.stderr: 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xbb6f26]
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr: 7: (ThreadPool::WorkThread::entry()+0x10) [0xbb7fd0]
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr: 8: (()+0x8182) [0x7fd5058ec182]
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr: 9: (clone()+0x6d) [0x7fd503e5747d]
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr:2015-10-13 19:55:37.792993 7fd4f79b7700 -1 os/FileStore.cc: In function 'int FileStore::_clone(coll_t, const ghobject_t&, const ghobject_t&, const SequencerPosition&)' thread 7fd4f79b7700 time 2015-10-13 19:55:37.790779
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr:os/FileStore.cc: 3178: FAILED assert(!m_filestore_fail_eio || r != -5)
2015-10-13T19:55:37.805 INFO:tasks.ceph.osd.12.mira103.stderr:
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: ceph version 0.94.3-290-g7f485ed (7f485ed5aa620fe982561663bf64356b7e2c38f2)
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc633b]
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 2: (FileStore::_clone(coll_t, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0xa4d) [0x91d23d]
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 3: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2628) [0x926108]
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x92abc4]
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x92ad4a]
2015-10-13T19:55:37.806 INFO:tasks.ceph.osd.12.mira103.stderr: 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xbb6f26]
2015-10-13T19:55:37.807 INFO:tasks.ceph.osd.12.mira103.stderr: 7: (ThreadPool::WorkThread::entry()+0x10) [0xbb7fd0]
2015-10-13T19:55:37.807 INFO:tasks.ceph.osd.12.mira103.stderr: 8: (()+0x8182) [0x7fd5058ec182]
2015-10-13T19:55:37.807 INFO:tasks.ceph.osd.12.mira103.stderr: 9: (clone()+0x6d) [0x7fd503e5747d]
2015-10-13T19:55:37.807 INFO:tasks.ceph.osd.12.mira103.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.