Bug #11594
Updated by David Zafman about 9 years ago
error (24) Too many open files not handled on operation caused os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") during ceph-objectstore-tool --op import http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-11_16:26:54-powercycle-hammer-testing-basic-multi/885356/teuthology.log <pre> 2015-05-11T22:06:08.978 INFO:teuthology.orchestra.run.plana86.stdout:Write 1380eddd/10000002033.00000000/head//2 2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr:os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f57ec427700 time 2015-05-11 22:06:08.976258 2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr:os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") 2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr: ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b) 2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbce2cb] 2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa52) [0x996fa2] 2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99d634] 2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x99d7ba] 2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbbf5fe] 2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 6: (ThreadPool::WorkThread::entry()+0x10) [0xbc06a0] 2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: 7: (()+0x8182) [0x7f57f7147182] 2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: 8: (clone()+0x6d) [0x7f57f3037efd] 2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2015-05-11T22:06:09.004 INFO:teuthology.orchestra.run.plana86.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion' 2015-05-11T22:06:09.004 INFO:teuthology.orchestra.run.plana86.stderr:*** Caught signal (Aborted) ** </pre> From ubuntu@teuthology:/a/teuthology-2015-05-11_16:26:54-powercycle-hammer-testing-basic-multi/885356/remote/plana86/log/objectstore_tool.7644.log.gz: <pre> -9> 2015-05-11 22:06:08.976048 7f57ec427700 10 journal op_apply_start 12518 open_ops 0 -> 1 -8> 2015-05-11 22:06:08.976054 7f57ec427700 5 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x582d1d0 seq 12518 osr(default 0x530a9f8)/0x530a9f8 start -7> 2015-05-11 22:06:08.976059 7f57ec427700 10 filestore(/var/lib/ceph/osd/ceph-0) _do_transaction on 0x7ffc3a52c110 -6> 2015-05-11 22:06:08.976069 7f57ec427700 15 filestore(/var/lib/ceph/osd/ceph-0) touch 2.5_head/1380eddd/10000002033.00000000/head//2 -5> 2015-05-11 22:06:08.976146 7f57ec427700 10 filestore(/var/lib/ceph/osd/ceph-0) error opening file /var/lib/ceph/osd/ceph-0/current/2.5_head/DIR_D/10000002033.00000000__head_1380EDDD__2 with flags=66: (24) Too many open files -4> 2015-05-11 22:06:08.976170 7f57ec427700 0 filestore(/var/lib/ceph/osd/ceph-0) error (24) Too many open files not handled on operation 0x54e6c60 (12518.0.0, or op 0, counting from 0) -3> 2015-05-11 22:06:08.976175 7f57ec427700 0 filestore(/var/lib/ceph/osd/ceph-0) unexpected error code -2> 2015-05-11 22:06:08.976177 7f57ec427700 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: { "ops": [ { "op_num": 0, "op_name": "touch", "collection": "2.5_head", "oid": "1380eddd\/10000002033.00000000\/head\/\/2" }, { "op_num": 1, "op_name": "write", "collection": "2.5_head", "oid": "1380eddd\/10000002033.00000000\/head\/\/2", "length": 1257, "offset": 0, "bufferlist length": 1257 }, { "op_num": 2, "op_name": "setattrs", "collection": "2.5_head", "oid": "1380eddd\/10000002033.00000000\/head\/\/2", "attr_lens": { "_": 255, "snapset": 31 } }, { "op_num": 3, "op_name": "omap_setheader", "collection": "2.5_head", "oid": "1380eddd\/10000002033.00000000\/head\/\/2", "header_length": "0" } ] } -1> 2015-05-11 22:06:08.976255 7f57ec427700 -1 dump_open_fds unable to open /proc/self/fd 0> 2015-05-11 22:06:08.979780 7f57ec427700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f57ec427700 time 2015-05-11 22:06:08.976258 os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbce2cb] 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa52) [0x996fa2] 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99d634] 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x99d7ba] 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbbf5fe] 6: (ThreadPool::WorkThread::entry()+0x10) [0xbc06a0] 7: (()+0x8182) [0x7f57f7147182] 8: (clone()+0x6d) [0x7f57f3037efd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. </pre>