Ceph » RADOS

Seen in a non-upgrade test with description:

rados/upgrade/jewel-x-singleton/{0-cluster/{openstack.yaml start.yaml}
1-jewel-install/jewel.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml
4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml}
5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-luminous.yaml
8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_latest.yaml
thrashosds-health.yaml}

http://pulpito.ceph.com/dzafman-2019-12-04_19:38:10-rados-wip-zafman-test2-luminous-distro-basic-smithi/4569076

   -19> 2019-12-05 13:13:40.630093 7f48afe79700 -1 /build/ceph-12.2.12-762-g6461a41/src/osd/PGLog.cc: In function 'void PGLog::IndexedLog::trim(CephContext*, eversion_t, std::set<eversion_t>*, std::set<std::__cxx11::basic_string<char> >*, eversion_t*)' thread 7f48afe79700 time 2019-12-05 13:13:40.609408
/build/ceph-12.2.12-762-g6461a41/src/osd/PGLog.cc: 53: FAILED assert(s <= can_rollback_to)

 ceph version 12.2.12-762-g6461a41 (6461a411b766622060e5df0433dc3ce79eb1889f) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x557024c18832]
 2: (PGLog::IndexedLog::trim(CephContext*, eversion_t, std::set<eversion_t, std::less<eversion_t>, std::allocator<eversion_t> >*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, eversion_t*)+0x1553) [0x5570247272d3]
 3: (PGLog::trim(eversion_t, pg_info_t&, bool)+0x23b) [0x55702472757b]
 4: (OSD::handle_pg_trim(boost::intrusive_ptr<OpRequest>)+0x3c3) [0x5570245d91d3]
 5: (OSD::dispatch_op(boost::intrusive_ptr<OpRequest>)+0x61) [0x557024610c11]
 6: (OSD::_dispatch(Message*)+0x389) [0x5570246117d9]
 7: (OSD::ms_dispatch(Message*)+0x87) [0x557024611b27]
 8: (DispatchQueue::entry()+0xf4a) [0x557024efd1fa]
 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x557024cac8ad]
 10: (()+0x76ba) [0x7f48c5fe56ba]
 11: (clone()+0x6d) [0x7f48c505c41d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

David Zafman wrote:

Seen in a non-upgrade test:

This is an upgrade test: "rados/upgrade/jewel-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-luminous.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_latest.yaml thrashosds-health.yaml}"

http://pulpito.ceph.com/dzafman-2019-12-04_19:38:10-rados-wip-zafman-test2-luminous-distro-basic-smithi/4569076

[...]

Oops. I think the more significant issue is that short_pg_log.yaml isn't involved.

Hi all,
what does "jewel to luminous split upgrades" and "boundary conditions" mean?

We're currently in the middle of upgrade process from 10.2.10/10.2.11 to 12.2.12. We faced with this bug at osd first start after package upgrade. After most of the cluster upgraded some already upgraded osd daemons started failing with the same error message.

Can we do something to prevent this fails during upgrade?
Will osd daemons continue failing after upgrade is finished?

For future references.
If I understand right: it seems that this happens during recovery when pg gets trim command and it is placed on OSDs with different versions (10.2.12 and 12.2.12 in our case). It might not happen at all on a small cluster. Faster upgrade fixed the problem.