Ceph

error (24) Too many open files not handled on operation caused os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") during ceph-objectstore-tool --op import

http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-11_16:26:54-powercycle-hammer-testing-basic-multi/885356/teuthology.log

2015-05-11T22:06:08.978 INFO:teuthology.orchestra.run.plana86.stdout:Write 1380eddd/10000002033.00000000/head//2
2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr:os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f57ec427700 time 2015-05-11 22:06:08.976258
2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr:os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")
2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr: ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b)
2015-05-11T22:06:08.982 INFO:teuthology.orchestra.run.plana86.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbce2cb]
2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa52) [0x996fa2]
2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99d634]
2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x99d7ba]
2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbbf5fe]
2015-05-11T22:06:08.983 INFO:teuthology.orchestra.run.plana86.stderr: 6: (ThreadPool::WorkThread::entry()+0x10) [0xbc06a0]
2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: 7: (()+0x8182) [0x7f57f7147182]
2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: 8: (clone()+0x6d) [0x7f57f3037efd]
2015-05-11T22:06:08.984 INFO:teuthology.orchestra.run.plana86.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-05-11T22:06:09.004 INFO:teuthology.orchestra.run.plana86.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-05-11T22:06:09.004 INFO:teuthology.orchestra.run.plana86.stderr:*** Caught signal (Aborted) **

From ubuntu@teuthology:/a/teuthology-2015-05-11_16:26:54-powercycle-hammer-testing-basic-multi/885356/remote/plana86/log/objectstore_tool.7644.log.gz:

    -9> 2015-05-11 22:06:08.976048 7f57ec427700 10 journal op_apply_start 12518 open_ops 0 -> 1
    -8> 2015-05-11 22:06:08.976054 7f57ec427700  5 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x582d1d0 seq 12518 osr(default 0x530a9f8)/0x530a9f8 start
    -7> 2015-05-11 22:06:08.976059 7f57ec427700 10 filestore(/var/lib/ceph/osd/ceph-0) _do_transaction on 0x7ffc3a52c110
    -6> 2015-05-11 22:06:08.976069 7f57ec427700 15 filestore(/var/lib/ceph/osd/ceph-0) touch 2.5_head/1380eddd/10000002033.00000000/head//2
    -5> 2015-05-11 22:06:08.976146 7f57ec427700 10 filestore(/var/lib/ceph/osd/ceph-0) error opening file /var/lib/ceph/osd/ceph-0/current/2.5_head/DIR_D/10000002033.00000000__head_1380EDDD__2 with flags=66: (24) Too many open files
    -4> 2015-05-11 22:06:08.976170 7f57ec427700  0 filestore(/var/lib/ceph/osd/ceph-0)  error (24) Too many open files not handled on operation 0x54e6c60 (12518.0.0, or op 0, counting from 0)
    -3> 2015-05-11 22:06:08.976175 7f57ec427700  0 filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
    -2> 2015-05-11 22:06:08.976177 7f57ec427700  0 filestore(/var/lib/ceph/osd/ceph-0)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "touch",
            "collection": "2.5_head",
            "oid": "1380eddd\/10000002033.00000000\/head\/\/2" 
        },
        {
            "op_num": 1,
            "op_name": "write",
            "collection": "2.5_head",
            "oid": "1380eddd\/10000002033.00000000\/head\/\/2",
            "length": 1257,
            "offset": 0,
            "bufferlist length": 1257
        },
        {
            "op_num": 2,
            "op_name": "setattrs",
            "collection": "2.5_head",
            "oid": "1380eddd\/10000002033.00000000\/head\/\/2",
            "attr_lens": {
                "_": 255,
                "snapset": 31
            }
        },
        {
            "op_num": 3,
            "op_name": "omap_setheader",
            "collection": "2.5_head",
            "oid": "1380eddd\/10000002033.00000000\/head\/\/2",
            "header_length": "0" 
        }
    ]
}

    -1> 2015-05-11 22:06:08.976255 7f57ec427700 -1 dump_open_fds unable to open /proc/self/fd
     0> 2015-05-11 22:06:08.979780 7f57ec427700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f57ec427700 time 2015-05-11 22:06:08.976258
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

 ceph version 0.94.1-111-gb69fb89 (b69fb89122d6f989152a29124cc7ed54b5e4d43b)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbce2cb]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa52) [0x996fa2]
 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99d634]
 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x16a) [0x99d7ba]
 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbbf5fe]
 6: (ThreadPool::WorkThread::entry()+0x10) [0xbc06a0]
 7: (()+0x8182) [0x7f57f7147182]
 8: (clone()+0x6d) [0x7f57f3037efd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.