Bug #50462: OSDs crash in osd/osd_types.cc: FAILED ceph_assert(clone_overlap.count(clone)) - RADOS - Ceph

Actions

#1

Updated by Martin Steinigen about 3 years ago

Sorry for formatting that bad

The issue started on luminous and it looked like an instance of https://tracker.ceph.com/issues/23030, so we decided to upgrade to latest luminous.
the initial ceph -s output

ceph-s ceph-s

root@m1r1~ ceph -s
cluster:
id: 590c3f41-aadf-4fe6-ac78-aec96146e5e4
health: HEALTH_WARN
norebalance flag(s) set
516275/16254420 objects misplaced (3.176%)
Degraded data redundancy: 935/16254420 objects degraded (0.006%), 2 pgs degraded, 1 pg undersized
1 slow requests are blocked > 32 sec. Implicated osds 34

services:
    mon: 3 daemons, quorum m1r1,m2r1,m3r1
    mgr: m3r1(active), standbys: m2r1, m1r1
    osd: 60 osds: 60 up, 60 in; 219 remapped pgs
         flags norebalance
    rgw: 1 daemon active

data:
    pools:   10 pools, 2688 pgs
    objects: 5.42M objects, 20.7TiB
    usage:   64.3TiB used, 44.6TiB / 109TiB avail
    pgs:     935/16254420 objects degraded (0.006%)
             516275/16254420 objects misplaced (3.176%)
             2467 active+clean
             214  active+remapped+backfill_wait
             2    active+remapped+backfilling
             2    active+recovery_wait+remapped
             1    active+undersized+degraded+remapped+backfill_wait
             1    active+recovery_wait
             1    active+recovery_wait+degraded

io:
    client:   32.4MiB/s rd, 0B/s wr, 220op/s rd, 114op/s wr

initial osd errors:

assertion assertion

Jan 15 11:32:31 m64r1 systemd¹: Starting Ceph object storage daemon osd.8...
Jan 15 11:32:31 m64r1 systemd¹: Started Ceph object storage daemon osd.8.
Jan 15 11:32:31 m64r1 ceph-osd^3521940: starting osd.8 at - osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Jan 15 11:32:34 m64r1 ceph-osd^3521940: 2021-01-15 11:32:34.575518 7fee122dfe00 -1 journal do_read_entry(8876077056): bad header magic
Jan 15 11:32:37 m64r1 ceph-osd^3521940: 2021-01-15 11:32:37.984227 7fee122dfe00 -1 osd.8 241145 log_to_monitors {default=true}
Jan 15 11:32:39 m64r1 ceph-osd^3521940: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd^3521940: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2021-01-15 11:32:39.722708 7fedde45f700 -1 /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd^3521940: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd^3521940: -5943> 2021-01-15 11:32:34.575518 7fee122dfe00 -1 journal do_read_entry(8876077056): bad header magic
Jan 15 11:32:39 m64r1 ceph-osd^3521940: -5366> 2021-01-15 11:32:37.984227 7fee122dfe00 -1 osd.8 241145 log_to_monitors {default=true}
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 0> 2021-01-15 11:32:39.722708 7fedde45f700 -1 /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd^3521940: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd^3521940: * Caught signal (Aborted) *
Jan 15 11:32:39 m64r1 ceph-osd^3521940: in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2021-01-15 11:32:39.761726 7fedde45f700 -1 Caught signal (Aborted)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 0> 2021-01-15 11:32:39.761726 7fedde45f700 -1 Caught signal (Aborted) *
Jan 15 11:32:39 m64r1 ceph-osd^3521940: in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd^3521940: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: 17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd^3521940: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

That did not fix the issue, update to mimic and then to nautilus.
The upgrades went smooth, but the affected placement group is still broken.
“The symptom is that whenever an OSD touches the placement group for replication/recovery, it will fail with an assertion:

assertion assertion

OSD 39

Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 12: (()+0x76ba) [0x7f8118dbc6ba]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 13: (clone()+0x6d) [0x7f81183c34dd]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 0> 2021-03-06 14:32:27.878 7f80e2fee700 -1 ** Caught signal (Aborted) *
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: in thread 7f80e2fee700 thread_name:tp_osd_tp
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 1: (()+0x11390) [0x7f8118dc6390]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 2: (gsignal()+0x38) [0x7f81182f1438]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 3: (abort()+0x16a) [0x7f81182f303a]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 15: (()+0x76ba) [0x7f8118dbc6ba]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: 16: (clone()+0x6d) [0x7f81183c34dd]
Mar 06 14:32:28 m62r1 ceph-osd²⁴⁸³⁶: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:33:53 m62r1 systemd¹: Stopped Ceph object storage daemon osd.39.
Mar 06 14:33:53 m62r1 systemd¹: Starting Ceph object storage daemon osd.39...
Mar 06 14:33:54 m62r1 systemd¹: Started Ceph object storage daemon osd.39.

OSD 17

Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: /build/ceph-14.2.16/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fb1e2c14700 time 2021-03-06 14:32:02.411755
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2021-03-06 14:32:02.413 7fb1e2c14700 -1 /build/ceph-14.2.16/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fb1e2c14700 time 2021-03-06 14:32:02.411755
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: * Caught signal (Aborted) *
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2021-03-06 14:32:02.421 7fb1e2c14700 -1 Caught signal (Aborted)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 0> 2021-03-06 14:32:02.421 7fb1e2c14700 -1 Caught signal (Aborted)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 0> 2021-03-06 14:32:02.421 7fb1e2c14700 -1 Caught signal (Aborted) *
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: 16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd²²²⁶²: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:34:12 m65r1 systemd¹: Stopped Ceph object storage daemon osd.17.

this has forced to set norecovery on the cluster in order to at least have some availability (because this stops OSDs from crashing due to touching the pg during syncs), but that is no solution.

Actions

Copy link

#2

Updated by Martin Steinigen about 3 years ago

finaly the correct format

The issue started on luminous and it looked like an instance of https://tracker.ceph.com/issues/23030, so we decided to upgrade to latest luminous.
the initial ceph -s output

ceph-s ceph-s


root@m1r1~ ceph -s
  cluster:
    id:     590c3f41-aadf-4fe6-ac78-aec96146e5e4
    health: HEALTH_WARN
            norebalance flag(s) set
            516275/16254420 objects misplaced (3.176%)
            Degraded data redundancy: 935/16254420 objects degraded (0.006%), 2 pgs degraded, 1 pg undersized
            1 slow requests are blocked > 32 sec. Implicated osds 34

  services:
    mon: 3 daemons, quorum m1r1,m2r1,m3r1
    mgr: m3r1(active), standbys: m2r1, m1r1
    osd: 60 osds: 60 up, 60 in; 219 remapped pgs
         flags norebalance
    rgw: 1 daemon active

  data:
    pools:   10 pools, 2688 pgs
    objects: 5.42M objects, 20.7TiB
    usage:   64.3TiB used, 44.6TiB / 109TiB avail
    pgs:     935/16254420 objects degraded (0.006%)
             516275/16254420 objects misplaced (3.176%)
             2467 active+clean
             214  active+remapped+backfill_wait
             2    active+remapped+backfilling
             2    active+recovery_wait+remapped
             1    active+undersized+degraded+remapped+backfill_wait
             1    active+recovery_wait
             1    active+recovery_wait+degraded

  io:
    client:   32.4MiB/s rd, 0B/s wr, 220op/s rd, 114op/s wr

initial osd errors:

assertion assertion


Jan 15 11:32:31 m64r1 systemd[1]: Starting Ceph object storage daemon osd.8...
Jan 15 11:32:31 m64r1 systemd[1]: Started Ceph object storage daemon osd.8.
Jan 15 11:32:31 m64r1 ceph-osd[3521940]: starting osd.8 at - osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Jan 15 11:32:34 m64r1 ceph-osd[3521940]: 2021-01-15 11:32:34.575518 7fee122dfe00 -1 journal do_read_entry(8876077056): bad header magic
Jan 15 11:32:37 m64r1 ceph-osd[3521940]: 2021-01-15 11:32:37.984227 7fee122dfe00 -1 osd.8 241145 log_to_monitors {default=true}
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: 2021-01-15 11:32:39.722708 7fedde45f700 -1 /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  -5943> 2021-01-15 11:32:34.575518 7fee122dfe00 -1 journal do_read_entry(8876077056): bad header magic
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  -5366> 2021-01-15 11:32:37.984227 7fee122dfe00 -1 osd.8 241145 log_to_monitors {default=true}
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:      0> 2021-01-15 11:32:39.722708 7fedde45f700 -1 /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fedde45f700 time 2021-01-15 11:32:39.715549
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: /build/ceph-12.2.12/src/osd/PrimaryLogPG.cc: 10596: FAILED assert(head_obc)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5645f1d1ba02]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: *** Caught signal (Aborted) **
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]: 2021-01-15 11:32:39.761726 7fedde45f700 -1 *** Caught signal (Aborted) **
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:      0> 2021-01-15 11:32:39.761726 7fedde45f700 -1 *** Caught signal (Aborted) **
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  in thread 7fedde45f700 thread_name:tp_osd_tp
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  1: (()+0xaa2044) [0x5645f1cd8044]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  2: (()+0x11390) [0x7fee105bb390]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  3: (gsignal()+0x38) [0x7fee0f556428]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  4: (abort()+0x16a) [0x7fee0f55802a]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5645f1d1bb8e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0xc99) [0x5645f1892619]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  7: (PrimaryLogPG::maybe_kick_recovery(hobject_t const&)+0x39e) [0x5645f189339e]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  8: (PrimaryLogPG::maybe_force_recovery()+0x208) [0x5645f1897258]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3089) [0x5645f18cdbf9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5645f1887db9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5645f16f53e9]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5645f19ad267]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x5645f1723737]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5645f1d207f4]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5645f1d23830]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  16: (()+0x76ba) [0x7fee105b16ba]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  17: (clone()+0x6d) [0x7fee0f62841d]
Jan 15 11:32:39 m64r1 ceph-osd[3521940]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

That did not fix the issue, update to mimic and then to nautilus.
The upgrades went smooth, but the affected placement group is still broken.
“The symptom is that whenever an OSD touches the placement group for replication/recovery, it will fail with an assertion:

assertion assertion


OSD 39 

Mar 06 14:32:28 m62r1 ceph-osd[24836]: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  12: (()+0x76ba) [0x7f8118dbc6ba]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  13: (clone()+0x6d) [0x7f81183c34dd]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:      0> 2021-03-06 14:32:27.878 7f80e2fee700 -1 *** Caught signal (Aborted) **
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  in thread 7f80e2fee700 thread_name:tp_osd_tp
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  1: (()+0x11390) [0x7f8118dc6390]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  2: (gsignal()+0x38) [0x7f81182f1438]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  3: (abort()+0x16a) [0x7f81182f303a]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  15: (()+0x76ba) [0x7f8118dbc6ba]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  16: (clone()+0x6d) [0x7f81183c34dd]
Mar 06 14:32:28 m62r1 ceph-osd[24836]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:33:53 m62r1 systemd[1]: Stopped Ceph object storage daemon osd.39.
Mar 06 14:33:53 m62r1 systemd[1]: Starting Ceph object storage daemon osd.39...
Mar 06 14:33:54 m62r1 systemd[1]: Started Ceph object storage daemon osd.39.

OSD 17 

Mar 06 14:32:02 m65r1 ceph-osd[22262]: /build/ceph-14.2.16/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fb1e2c14700 time 2021-03-06 14:32:02.411755
Mar 06 14:32:02 m65r1 ceph-osd[22262]: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]: 2021-03-06 14:32:02.413 7fb1e2c14700 -1 /build/ceph-14.2.16/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fb1e2c14700 time 2021-03-06 14:32:02.411755
Mar 06 14:32:02 m65r1 ceph-osd[22262]: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]: *** Caught signal (Aborted) **
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]: 2021-03-06 14:32:02.421 7fb1e2c14700 -1 *** Caught signal (Aborted) **
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:32:02 m65r1 ceph-osd[22262]: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:      0> 2021-03-06 14:32:02.421 7fb1e2c14700 -1 *** Caught signal (Aborted) **
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:32:02 m65r1 ceph-osd[22262]: /build/ceph-14.2.16/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846d7e]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:      0> 2021-03-06 14:32:02.421 7fb1e2c14700 -1 *** Caught signal (Aborted) **
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  in thread 7fb1e2c14700 thread_name:tp_osd_tp
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  1: (()+0x11390) [0x7fb209b47390]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  2: (gsignal()+0x38) [0x7fb209072428]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  3: (abort()+0x16a) [0x7fb20907402a]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x846dcf]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846f59]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  6: (SnapSet::get_clone_bytes(snapid_t) const+0xd2) [0xb2fad2]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2d0) [0xa50ff0]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x163b) [0xa8493b]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0xec2) [0xa88d02]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x366) [0x8c9986]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0xb83599]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x8e81dd]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xf014ac]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xf04670]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  15: (()+0x76ba) [0x7fb209b3d6ba]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  16: (clone()+0x6d) [0x7fb20914441d]
Mar 06 14:32:02 m65r1 ceph-osd[22262]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Mar 06 14:34:12 m65r1 systemd[1]: Stopped Ceph object storage daemon osd.17.

this has forced to set norecovery on the cluster in order to at least have some availability (because this stops OSDs from crashing due to touching the pg during syncs), but that is no solution.

Actions

Copy link Download all files

#3

Updated by Ana Aviles about 3 years ago

File ceph-bluestore-tool.fsck.out ceph-bluestore-tool.fsck.out added
File osd21.log osd21.log added

We run into the same assert over and over on one OSD. We were upgrading from luminous to nautilus ceph version 14.2.20 and had a mix of filestore and bluestore OSDs. The problems were always on bluestore OSDs.

During the upgrade two OSDs were crashing for different reasons causing some PGs to be down. OSD wouldn't come up stopping on this assert. In the end we had to force_recreate the PGs to allow client I/O to continue working and tried to restore as much data as we could afterwards. That was a difficult phase because exporting the inactive PGs with `ceph-objectstore-tool` would fail with ` export_files error -5 ` and wouldn't export the whole PG.
Trying to fix the OSD consistency with the `ceph-bluestore-tool` gave a core dump, I attach the output.

I add here logs we gathered for the assert in case they are of any help. I attach the log with default debug level here, and I uploaded the log with debug level 20 with this id: b11325b3-0f3c-4af5-b349-64fe912bf984

Actions

Copy link

#4

Updated by Sage Weil about 3 years ago

Project changed from Ceph to RADOS
Category deleted (~~OSD~~)
Priority changed from Normal to High

Actions

Copy link

#5

Updated by Igor Fedotov almost 3 years ago

Just to note:
IMO ceph-bluestore-tool crash is caused by a bag in AvlAllocator and is a duplicate of https://tracker.ceph.com/issues/50555. Hence going to be fixed in 14.2.22
But it looks completely irrelevant to the original crash reported in this ticket.

Actions

Copy link

#6

Updated by Neha Ojha almost 3 years ago

Subject changed from OSDs crashing while PG operations to OSDs crash in osd/osd_types.cc: FAILED ceph_assert(clone_overlap.count(clone))

Actions

Copy link

#7

Updated by Neha Ojha over 2 years ago

Status changed from New to Won't Fix - EOL

Please feel free to reopen if you see the issue in a recent version of Ceph.

Actions

Copy link

#8

Updated by Justin Mammarella almost 2 years ago

We are seeing this bug in Nautilus 14.2.15 to 14.2.22 replicated pool.

Two of our osds are stuck in a crash loop trying to backfill. We have disabled backfills for now and are looking at removing the offending object from the pool. If that fails we will try and manually backfill to the target osd (479).

    -6> 2022-05-23 11:41:33.251 7f15245c1700 5 osd.262 pg_epoch: 685308 pg[6.244a( v 685307'35436378 (685280'35433366,685307'35436378] local-lis/les=685306/685307 n=14244 ec=418948/2707 lis/c 685306/683632 les/c/f 685307/683636/0 685305/685306/685306) [262,146,479]/[262,146] backfill=[479] r=0 lpr=685306 pi=[683632,685306)/3 rops=1 crt=685307'35436378 lcod 685307'35436377 mlcod 685307'35436377 active+undersized+degraded+remapped+backfilling mbc={255={}} trimq=[5ce7~1,5e9b~1,5ef5~1,6039~1,8509~1,9f87~1,9f8f~1,9f91~1,9f97~1,9f99~1,9f9b~1,9f9d~1]] backfill_pos is 6:52245a01:::rbd_data.8a845579e2a9e3.0000000000b06c49:head

    -1> 2022-05-23 11:41:33.327 7f15245c1700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7f15245c1700 time 2022-05-23 11:41:33.325202
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/osd_types.cc: 5450: FAILED ceph_assert(clone_overlap.count(clone))

ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55a68b6f9393]
2: (()+0x4da55b) [0x55a68b6f955b]
3: (SnapSet::get_clone_bytes(snapid_t) const+0xc2) [0x55a68ba15b62]
4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x28c) [0x55a68b947b2c]
5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0xf65) [0x55a68b976a05]
6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x114c) [0x55a68b97a82c]
7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2ff) [0x55a68b7d9fcf]
8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a68ba69fd9]
9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55a68b7f5d1f]
10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55a68bdb4006]
11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a68bdb6b20]
12: (()+0x7ea5) [0x7f1546778ea5]
13: (clone()+0x6d) [0x7f154563b9fd]

     0> 2022-05-23 11:41:33.331 7f15245c1700 -1 ** Caught signal (Aborted) *
in thread 7f15245c1700 thread_name:tp_osd_tp

ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
1: (()+0xf630) [0x7f1546780630]
2: (gsignal()+0x37) [0x7f15455733d7]
3: (abort()+0x148) [0x7f1545574ac8]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55a68b6f93e2]
5: (()+0x4da55b) [0x55a68b6f955b]
6: (SnapSet::get_clone_bytes(snapid_t) const+0xc2) [0x55a68ba15b62]
7: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x28c) [0x55a68b947b2c]
8: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0xf65) [0x55a68b976a05]
9: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x114c) [0x55a68b97a82c]
10: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x2ff) [0x55a68b7d9fcf]
11: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a68ba69fd9]
12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55a68b7f5d1f]
13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55a68bdb4006]
14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a68bdb6b20]
15: (()+0x7ea5) [0x7f1546778ea5]
16: (clone()+0x6d) [0x7f154563b9fd]

Actions

Copy link

#9

Updated by hoan nv over 1 year ago

Justin Mammarella wrote:

We are seeing this bug in Nautilus 14.2.15 to 14.2.22 replicated pool.

Two of our osds are stuck in a crash loop trying to backfill. We have disabled backfills for now and are looking at removing the offending object from the pool. If that fails we will try and manually backfill to the target osd (479).

How you manually backfill to the target osd ?
Can you show me command or guide.
Thanks

ceph-bluestore-tool.fsck.out (8.85 KB) ceph-bluestore-tool.fsck.out		Ana Aviles, 04/29/2021 09:46 AM
osd21.log (373 KB) osd21.log		Ana Aviles, 04/29/2021 09:49 AM

Project

General

Profile

Ceph » RADOS

Custom queries

Bug #50462

OSDs crash in osd/osd_types.cc: FAILED ceph_assert(clone_overlap.count(clone))

Updated by Martin Steinigen about 3 years ago

Updated by Martin Steinigen about 3 years ago

Updated by Ana Aviles about 3 years ago

Updated by Sage Weil about 3 years ago

Updated by Igor Fedotov almost 3 years ago

Updated by Neha Ojha almost 3 years ago

Updated by Neha Ojha over 2 years ago

Updated by Justin Mammarella almost 2 years ago

Updated by hoan nv over 1 year ago