Ceph

Hi,

We have a cluster with 3 hosts, each with 12 10TB disks. We are about to add 3 new servers with a similar configuration. We added in one server, and then activated an OSD, the system started to rebalance, we added a second disk. That evening the second of the two new disks started to flap. Ceph is now reporting:

[root@ceph4 ceph]# ceph status
cluster:
id: e474ffde-40fd-44c2-aa4d-753dd4dd0761
health: HEALTH_ERR
1 clients failing to respond to capability release
1 MDSs report slow metadata IOs
1 MDSs report slow requests
1 MDSs behind on trimming
21375776/244346433 objects misplaced (8.748%)
Reduced data availability: 4 pgs inactive, 4 pgs incomplete
Degraded data redundancy (low space): 32 pgs backfill_toofull

services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3
    mgr: ceph2(active), standbys: ceph3, ceph1
    mds: cephfs-nan-1/1/1 up  {0=ceph3=up:active}, 2 up:standby
    osd: 38 osds: 37 up, 37 in; 233 remapped pgs

data:
    pools:   2 pools, 1280 pgs
    objects: 81.45 M objects, 86 TiB
    usage:   155 TiB used, 181 TiB / 337 TiB avail
    pgs:     0.313% pgs not active
             21375776/244346433 objects misplaced (8.748%)
             1047 active+clean
             194  active+remapped+backfill_wait
             32   active+remapped+backfill_wait+backfill_toofull
             4    remapped+incomplete
             3    active+remapped+backfilling

io:
    client:   511 B/s wr, 0 op/s rd, 0 op/s wr
    recovery: 26 MiB/s, 22 objects/s

[root@ceph4 ceph]#

The clients are hanging, presumably as they try to access objects in this PG.

[root@ceph4 ceph]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 21370460/244347825 objects misplaced (8.746%); Reduced data availability: 4 pgs inactive, 4 pgs incomplete; Degraded data redundancy (low space): 32 pgs backfill_toofull
MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release
mdsceph3(mds.0): Client connect failing to respond to capability release client_id: 834174
MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs
mdsceph3(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 4915 secs
MDS_SLOW_REQUEST 1 MDSs report slow requests
mdsceph3(mds.0): 5 slow requests are blocked > 30 secs
MDS_TRIM 1 MDSs behind on trimming
mdsceph3(mds.0): Behind on trimming (875/128) max_segments: 128, num_segments: 875
OBJECT_MISPLACED 21370460/244347825 objects misplaced (8.746%)
PG_AVAILABILITY Reduced data availability: 4 pgs inactive, 4 pgs incomplete
pg 5.fc is remapped+incomplete, acting [6,2147483647,27] (reducing pool data_ec_nan min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 5.292 is remapped+incomplete, acting [2147483647,13,18] (reducing pool data_ec_nan min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 5.2f8 is remapped+incomplete, acting [31,12,2147483647] (reducing pool data_ec_nan min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 5.33e is remapped+incomplete, acting [7,24,2147483647] (reducing pool data_ec_nan min_size from 3 may help; search ceph.com/docs for 'incomplete')
PG_DEGRADED_FULL Degraded data redundancy (low space): 32 pgs backfill_toofull
pg 5.8 is active+remapped+backfill_wait+backfill_toofull, acting [31,15,3]
pg 5.11 is active+remapped+backfill_wait+backfill_toofull, acting [23,10,30]
pg 5.42 is active+remapped+backfill_wait+backfill_toofull, acting [6,33,25]
pg 5.4f is active+remapped+backfill_wait+backfill_toofull, acting [21,7,26]
pg 5.69 is active+remapped+backfill_wait+backfill_toofull, acting [24,15,4]
pg 5.81 is active+remapped+backfill_wait+backfill_toofull, acting [24,34,13]
pg 5.101 is active+remapped+backfill_wait+backfill_toofull, acting [35,11,23]
pg 5.11b is active+remapped+backfill_wait+backfill_toofull, acting [27,15,23]
pg 5.129 is active+remapped+backfill_wait+backfill_toofull, acting [15,30,3]
pg 5.14f is active+remapped+backfill_wait+backfill_toofull, acting [5,16,14]
pg 5.168 is active+remapped+backfill_wait+backfill_toofull, acting [23,35,8]
pg 5.17e is active+remapped+backfill_wait+backfill_toofull, acting [19,26,7]
pg 5.18d is active+remapped+backfill_wait+backfill_toofull, acting [1,25,34]
pg 5.197 is active+remapped+backfill_wait+backfill_toofull, acting [0,19,32]
pg 5.199 is active+remapped+backfill_wait+backfill_toofull, acting [1,29,21]
pg 5.1b8 is active+remapped+backfill_wait+backfill_toofull, acting [21,0,32]
pg 5.1d3 is active+remapped+backfill_wait+backfill_toofull, acting [35,7,17]
pg 5.21f is active+remapped+backfill_wait+backfill_toofull, acting [2,13,30]
pg 5.246 is active+remapped+backfill_wait+backfill_toofull, acting [4,7,20]
pg 5.24c is active+remapped+backfill_wait+backfill_toofull, acting [11,17,34]
pg 5.255 is active+remapped+backfill_wait+backfill_toofull, acting [13,25,34]
pg 5.269 is active+remapped+backfill_wait+backfill_toofull, acting [13,29,23]
pg 5.29b is active+remapped+backfill_wait+backfill_toofull, acting [3,4,15]
pg 5.2ba is active+remapped+backfill_wait+backfill_toofull, acting [14,34,3]
pg 5.2c9 is active+remapped+backfill_wait+backfill_toofull, acting [34,18,6]
pg 5.2ef is active+remapped+backfill_wait+backfill_toofull, acting [35,15,21]
pg 5.302 is active+remapped+backfill_wait+backfill_toofull, acting [35,17,7]
pg 5.315 is active+remapped+backfill_wait+backfill_toofull, acting [7,26,21]
pg 5.374 is active+remapped+backfill_wait+backfill_toofull, acting [20,28,11]
pg 5.37c is active+remapped+backfill_wait+backfill_toofull, acting [23,8,5]
pg 6.c8 is active+remapped+backfill_wait+backfill_toofull, acting [0,25,29]
pg 6.fa is active+remapped+backfill_wait+backfill_toofull, acting [28,7,20]
[root@ceph4 ceph]#

We are trying to workout how to get this back into a functional state.

I tried to restart osd.37, but it just crashed out again, the log is long, part of it reports:

ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
1: (()+0x911e70) [0x56195a1f6e70]
2: (()+0xf5d0) [0x7f95f12465d0]
3: (gsignal()+0x37) [0x7f95f0267207]
4: (abort()+0x148) [0x7f95f02688f8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x242) [0x7f95f46be2b2]
6: (()+0x25a337) [0x7f95f46be337]
7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>, boost::intrusive_ptr<BlueStore::Onode>, BlueStore::WriteContext*)+0x2c80) [0x56195a0e9760]
8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x29f) [0x56195a0f59df]
9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0xc6) [0x56195a0f66d6]
10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0xcb6) [0x56195a0f9056]
11: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x531) [0x56195a0fb5f1]
12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x561959e53644]
13: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&)+0x957) [0x561959f92737]
14: (ECBackend::try_reads_to_commit()+0x72c) [0x561959f9ab1c]
15: (ECBackend::check_ops()+0x1c) [0x561959fa161c]
16: (ECBackend::start_rmw(ECBackend::Op*, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&)+0x7d2) [0x561959faa0b2]
17: (ECBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x372) [0x561959fab8d2]
18: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd62) [0x561959df97c2]
19: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xee2) [0x561959e39d52]
20: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3589) [0x561959e3db09]
21: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xc99) [0x561959e41c09]
22: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) [0x561959c9bc87]
23: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x561959f148c2]
24: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x592) [0x561959cbbae2]
25: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) [0x7f95f46c3ec3]
26: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f95f46c4ab0]
27: (()+0x7dd5) [0x7f95f123edd5]
28: (clone()+0x6d) [0x7f95f032eead]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Should I mark the osd as lost? The documentation makes it sound like this is a very final action.

I'm surprised things have hung, I thought we had a certain level of redundancy with our pools.

Chris