Project

General

Profile

Actions

Bug #12518

closed

"FileStore.cc: 3544: FAILED assert(pos == end)" in upgrade:hammer-x-next-distro-basic-vps

Added by Yuri Weinstein over 8 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/hammer-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-07-28_17:10:15-upgrade:hammer-x-next-distro-basic-vps/
Job: 990369
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-07-28_17:10:15-upgrade:hammer-x-next-distro-basic-vps/990369/teuthology.log

2015-07-28T22:04:31.810 INFO:tasks.workunit.client.0.vpm124.stdout: i=0 exists [53528184~3058136,61005543~7354503,72571749~4156093,113873230~7438750,247461543~798271,283750509~1628623,389512452~3646416,404142445~6250462] wrote [53528184~3058136,61005543~7354503,72571749~4156093,113873230~7438750,247461543~798271,283750509~1628623,389512452~3646416,404142445~6250462]
2015-07-28T22:04:33.099 INFO:tasks.workunit.client.0.vpm124.stdout: i=1 exists [55021557~1564763,61005543~7354503,72571749~4156093,97834093~5738217,113873230~7438750,132029699~5000347,208166686~4067809,247461543~798271,275280667~1660402,283750509~1628623,296766401~2025142,362811622~467002,389512452~3646416,404142445~6250462] wrote [53528184~1493373,97834093~5738217,132029699~5000347,208166686~4067809,275280667~1660402,296766401~2025142,362811622~467002]
2015-07-28T22:04:34.055 INFO:tasks.ceph.osd.1.vpm099.stderr:2015-07-29 01:04:34.045988 7f61fd49c700 -1 filestore(/var/lib/ceph/osd/ceph-1) FileStore::_do_copy_range got short read result at 1441792 of fd 63 len 4194304
2015-07-28T22:04:34.097 INFO:tasks.ceph.osd.1.vpm099.stderr:os/FileStore.cc: In function 'int FileStore::_do_copy_range(int, int, uint64_t, uint64_t, uint64_t, bool)' thread 7f61fd49c700 time 2015-07-29 01:04:34.046010
2015-07-28T22:04:34.097 INFO:tasks.ceph.osd.1.vpm099.stderr:os/FileStore.cc: 3544: FAILED assert(pos == end)
2015-07-28T22:04:34.195 INFO:tasks.ceph.osd.1.vpm099.stderr:2015-07-29 01:04:34.182954 7f61fde9d700 -1 filestore(/var/lib/ceph/osd/ceph-1) FileStore::_do_copy_range got short read result at 0 of fd 164 len 1230370
2015-07-28T22:04:34.195 INFO:tasks.ceph.osd.1.vpm099.stderr:os/FileStore.cc: In function 'int FileStore::_do_copy_range(int, int, uint64_t, uint64_t, uint64_t, bool)' thread 7f61fde9d700 time 2015-07-29 01:04:34.182958
2015-07-28T22:04:34.196 INFO:tasks.ceph.osd.1.vpm099.stderr:os/FileStore.cc: 3544: FAILED assert(pos == end)
2015-07-28T22:04:34.266 INFO:tasks.ceph.osd.1.vpm099.stderr: ceph version 9.0.2-680-ge957872 (e95787216b8c84da98810eb5dccb09941ad9fa2f)
2015-07-28T22:04:34.266 INFO:tasks.ceph.osd.1.vpm099.stderr: 1: (FileStore::_do_copy_range(int, int, unsigned long, unsigned long, unsigned long, bool)+0x1523) [0x785723]
2015-07-28T22:04:34.266 INFO:tasks.ceph.osd.1.vpm099.stderr: 2: (GenericFileStoreBackend::clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x59) [0x976d49]
2015-07-28T22:04:34.266 INFO:tasks.ceph.osd.1.vpm099.stderr: 3: (FileStore::_do_clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x185) [0x77e405]
2015-07-28T22:04:34.266 INFO:tasks.ceph.osd.1.vpm099.stderr: 4: (FileStore::_clone(coll_t, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0x485) [0x7a8345]
2015-07-28T22:04:34.267 INFO:tasks.ceph.osd.1.vpm099.stderr: 5: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2153) [0x7b5e53]
2015-07-28T22:04:34.267 INFO:tasks.ceph.osd.1.vpm099.stderr: 6: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x7bd954]
2015-07-28T22:04:34.267 INFO:tasks.ceph.osd.1.vpm099.stderr: 7: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x297) [0x7bdc07]
2015-07-28T22:04:34.267 INFO:tasks.ceph.osd.1.vpm099.stderr: 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xbdde76]
2015-07-28T22:04:34.267 INFO:tasks.ceph.osd.1.vpm099.stderr: 9: (ThreadPool::WorkThread::entry()+0x10) [0xbdfa50]
2015-07-28T22:04:34.268 INFO:tasks.ceph.osd.1.vpm099.stderr: 10: /lib64/libpthread.so.0() [0x37656079d1]
2015-07-28T22:04:34.268 INFO:tasks.ceph.osd.1.vpm099.stderr: 11: (clone()+0x6d) [0x37652e88fd]
2015-07-28T22:04:34.268 INFO:tasks.ceph.osd.1.vpm099.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-07-28T22:04:34.268 INFO:tasks.ceph.osd.1.vpm099.stderr:2015-07-29 01:04:34.254840 7f61fd49c700 -1 os/FileStore.cc: In function 'int FileStore::_do_copy_range(int, int, uint64_t, uint64_t, uint64_t, bool)' thread 7f61fd49c700 time 2015-07-29 01:04:34.046010
2015-07-28T22:04:34.268 INFO:tasks.ceph.osd.1.vpm099.stderr:os/FileStore.cc: 3544: FAILED assert(pos == end)
2015-07-28T22:04:34.269 INFO:tasks.ceph.osd.1.vpm099.stderr:
2015-07-28T22:04:34.269 INFO:tasks.ceph.osd.1.vpm099.stderr: ceph version 9.0.2-680-ge957872 (e95787216b8c84da98810eb5dccb09941ad9fa2f)
2015-07-28T22:04:34.269 INFO:tasks.ceph.osd.1.vpm099.stderr: 1: (FileStore::_do_copy_range(int, int, unsigned long, unsigned long, unsigned long, bool)+0x1523) [0x785723]
2015-07-28T22:04:34.269 INFO:tasks.ceph.osd.1.vpm099.stderr: 2: (GenericFileStoreBackend::clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x59) [0x976d49]
2015-07-28T22:04:34.269 INFO:tasks.ceph.osd.1.vpm099.stderr: 3: (FileStore::_do_clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x185) [0x77e405]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 4: (FileStore::_clone(coll_t, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0x485) [0x7a8345]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 5: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2153) [0x7b5e53]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 6: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x7bd954]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 7: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x297) [0x7bdc07]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xbdde76]
2015-07-28T22:04:34.270 INFO:tasks.ceph.osd.1.vpm099.stderr: 9: (ThreadPool::WorkThread::entry()+0x10) [0xbdfa50]
2015-07-28T22:04:34.271 INFO:tasks.ceph.osd.1.vpm099.stderr: 10: /lib64/libpthread.so.0() [0x37656079d1]
2015-07-28T22:04:34.271 INFO:tasks.ceph.osd.1.vpm099.stderr: 11: (clone()+0x6d) [0x37652e88fd]
2015-07-28T22:04:34.271 INFO:tasks.ceph.osd.1.vpm099.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-07-28T22:04:34.271 INFO:tasks.ceph.osd.1.vpm099.stderr:
2015-07-28T22:04:34.271 INFO:tasks.ceph.osd.1.vpm099.stderr: ceph version 9.0.2-680-ge957872 (e95787216b8c84da98810eb5dccb09941ad9fa2f)
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 1: (FileStore::_do_copy_range(int, int, unsigned long, unsigned long, unsigned long, bool)+0x1523) [0x785723]
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 2: (GenericFileStoreBackend::clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x59) [0x976d49]
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 3: (FileStore::_do_clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x185) [0x77e405]
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 4: (FileStore::_clone(coll_t, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0x485) [0x7a8345]
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 5: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2153) [0x7b5e53]
2015-07-28T22:04:34.272 INFO:tasks.ceph.osd.1.vpm099.stderr: 6: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x7bd954]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: 7: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x297) [0x7bdc07]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xbdde76]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: 9: (ThreadPool::WorkThread::entry()+0x10) [0xbdfa50]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: 10: /lib64/libpthread.so.0() [0x37656079d1]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: 11: (clone()+0x6d) [0x37652e88fd]
2015-07-28T22:04:34.273 INFO:tasks.ceph.osd.1.vpm099.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions #1

Updated by Casey Bodley about 4 years ago

  • Status changed from New to Closed
Actions

Also available in: Atom PDF