Ceph » bluestore

A similar ENOSPC issue appeared in failed and dead jobs in 14.2.17 nautilus-p2p testing: https://pulpito.ceph.com/teuthology-2021-03-06_03:49:44-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/ and https://pulpito.ceph.com/yuriw-2021-03-08_19:14:19-upgrade:nautilus-p2p-nautilus-distro-basic-smithi

The only pattern was it was only with hybrid allocatore. Some of them crashed in earlier versions (e.g. 14.2.2) before reaching a point where they'd be using the hybrid allocator though. Others got all the way to 14.2.16.

2021-03-08 20:31:18.245 7ff70053f700 -1 bluestore(/var/lib/ceph/osd/ceph-0) _do_alloc_write failed to allocate 0x4000 allocated 0x 0 min_alloc_size 0x4000 available 0x 0
2021-03-08 20:31:18.245 7ff70053f700 -1 bluestore(/var/lib/ceph/osd/ceph-0) _do_write _do_alloc_write failed with (28) No space left on device
2021-03-08 20:31:18.245 7ff70053f700 -1 bluestore(/var/lib/ceph/osd/ceph-0) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 0, counting from 0)
2021-03-08 20:31:18.245 7ff70053f700 -1 bluestore(/var/lib/ceph/osd/ceph-0) ENOSPC from bluestore, misconfigured cluster
2021-03-08 20:31:18.245 7ff70053f700  0 _dump_transaction transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "write",
            "collection": "meta",
            "oid": "#-1:4b4b4e63:::inc_osdmap.880:0#",
            "length": 1284,
            "offset": 0,
            "bufferlist length": 1284
        },
        {
            "op_num": 1,
            "op_name": "write",
            "collection": "meta",
            "oid": "#-1:a6032935:::osdmap.880:0#",
            "length": 6040,
            "offset": 0,
            "bufferlist length": 6040
        },
        {
            "op_num": 2,
            "op_name": "write",
            "collection": "meta",
            "oid": "#-1:46e858f0:::pg_num_history:head#",
            "length": 406,
            "offset": 0,
            "bufferlist length": 406
        },
        {
            "op_num": 3,
            "op_name": "write",
            "collection": "meta",
            "oid": "#-1:7b3f43c4:::osd_superblock:0#",
            "length": 538,
            "offset": 0,
            "bufferlist length": 538
        }
    ]
}
2021-03-08 20:31:18.324 7ff6ec990700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16-432-g9b829698/rpm/el7/BUILD/ceph-14.2.16-432-g9b829698/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7ff6ec990700 time 2021-03-08 20:31:18.274039
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16-432-g9b829698/rpm/el7/BUILD/ceph-14.2.16-432-g9b829698/src/os/bluestore/BlueStore.cc: 12384: ceph_abort_msg("unexpected error")

 ceph version 14.2.16-432-g9b82969 (9b829698f3d6269668c9402131bbfba81df6df0f) nautilus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x55e75423bb22]
 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0xc97) [0x55e754794d77]
 3: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x55e7547ac250]
 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55e754516394]
 5: (ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xc4b) [0x55e75460cc5b]
 6: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x257) [0x55e75461b267]
 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x55e75452cc8a]
 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x55e7544dcdb3]
 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x55e75431c462]
 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55e7545abc02]
 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55e75433796f]
 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55e7548f3956]
 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e7548f6470]
 14: (()+0x7ea5) [0x7ff713182ea5]
 15: (clone()+0x6d) [0x7ff71204596d]