Actions
Bug #44532
closednautilus: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) in PGLog::IndexedLog::add()
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/yuriw-2020-03-07_18:26:25-rados-wip-yuri8-testing-2020-03-06-2005-nautilus-distro-basic-smithi/
Job: 4835427
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2020-03-07_18:26:25-rados-wip-yuri8-testing-2020-03-06-2005-nautilus-distro-basic-smithi/4835427/teuthology.log
2020-03-08T03:10:24.141 INFO:tasks.ceph.osd.2:Restarting daemon
2020-03-08T03:10:24.141 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2
2020-03-08T03:10:24.143 INFO:tasks.ceph.osd.2:Started
2020-03-08T03:10:24.144 INFO:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight
2020-03-08T03:10:24.251 INFO:teuthology.orchestra.run.smithi188.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2020-03-08T03:10:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 22
2020-03-08T03:10:24.257 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight']
2020-03-08T03:10:24.787 INFO:tasks.ceph.osd.2.smithi188.stderr:2020-03-08 03:10:24.790 7f47dc8b4a80 -1 Falling back to public interface
2020-03-08T03:10:26.534 INFO:tasks.ceph.osd.2.smithi188.stderr:2020-03-08 03:10:26.537 7f47dc8b4a80 -1 osd.2 19 log_to_monitors {default=true}
2020-03-08T03:10:27.936 INFO:tasks.ceph.osd.2.smithi188.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f47b7f33700 time 2020-03-08 03:10:27.941071
2020-03-08T03:10:27.937 INFO:tasks.ceph.osd.2.smithi188.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: 519: FAILED ceph_assert(head.version == 0 || e.version.version > head.version)
2020-03-08T03:10:27.938 INFO:tasks.ceph.osd.2.smithi188.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f47b5f2f700 time 2020-03-08 03:10:27.942301
2020-03-08T03:10:27.938 INFO:tasks.ceph.osd.2.smithi188.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: 519: FAILED ceph_assert(head.version == 0 || e.version.version > head.version)
2020-03-08T03:10:27.940 INFO:tasks.ceph.osd.2.smithi188.stderr: ceph version 14.2.8-60-ga872021 (a8720218618e7c608c814c369c16739ada778bfd) nautilus (stable)
2020-03-08T03:10:27.940 INFO:tasks.ceph.osd.2.smithi188.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5606f82379c1]
2020-03-08T03:10:27.940 INFO:tasks.ceph.osd.2.smithi188.stderr: 2: (()+0x4cab89) [0x5606f8237b89]
2020-03-08T03:10:27.940 INFO:tasks.ceph.osd.2.smithi188.stderr: 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, bool, std::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)14, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0x5cd) [0x5606f8453a4d]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 4: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xdea) [0x5606f8448cba]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 5: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x5606f83ab454]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 6: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x5606f83ee5cb]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 7: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x5606f843ce85]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 8: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x5606f840097d]
2020-03-08T03:10:27.941 INFO:tasks.ceph.osd.2.smithi188.stderr: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x5606f833d374]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 10: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x5606f85a5df1]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x5606f8331fff]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5606f88d64e6]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5606f88d9000]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 14: (()+0x7ea5) [0x7f47d9910ea5]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr: 15: (clone()+0x6d) [0x7f47d87d48cd]
2020-03-08T03:10:27.942 INFO:tasks.ceph.osd.2.smithi188.stderr:*** Caught signal (Aborted) **
2020-03-08T03:10:27.943 INFO:tasks.ceph.osd.2.smithi188.stderr: in thread 7f47b7f33700 thread_name:tp_osd_tp
2020-03-08T03:10:27.943 INFO:tasks.ceph.osd.2.smithi188.stderr:2020-03-08 03:10:27.943 7f47b7f33700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f47b5f2f700 time 2020-03-08 03:10:27.942301
2020-03-08T03:10:27.943 INFO:tasks.ceph.osd.2.smithi188.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8-60-ga872021/rpm/el7/BUILD/ceph-14.2.8-60-ga872021/src/osd/PGLog.h: 519: FAILED ceph_assert(head.version == 0 || e.version.version > head.version)
2020-03-08T03:10:27.943 INFO:tasks.ceph.osd.2.smithi188.stderr:
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: ceph version 14.2.8-60-ga872021 (a8720218618e7c608c814c369c16739ada778bfd) nautilus (stable)
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5606f82379c1]
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 2: (()+0x4cab89) [0x5606f8237b89]
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, bool, std::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)14, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0x5cd) [0x5606f8453a4d]
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 4: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xdea) [0x5606f8448cba]
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 5: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x5606f83ab454]
2020-03-08T03:10:27.944 INFO:tasks.ceph.osd.2.smithi188.stderr: 6: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x5606f83ee5cb]
2020-03-08T03:10:27.945 INFO:tasks.ceph.osd.2.smithi188.stderr: 7: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x5606f843ce85]
2020-03-08T03:10:27.945 INFO:tasks.ceph.osd.2.smithi188.stderr: 8: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x5606f840097d]
2020-03-08T03:10:27.945 INFO:tasks.ceph.osd.2.smithi188.stderr: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x5606f833d374]
2020-03-08T03:10:27.945 INFO:tasks.ceph.osd.2.smithi188.stderr: 10: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x5606f85a5df1]
2020-03-08T03:10:27.945 INFO:tasks.ceph.osd.2.smithi188.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x5606f8331fff]
2020-03-08T03:10:27.946 INFO:tasks.ceph.osd.2.smithi188.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5606f88d64e6]
2020-03-08T03:10:27.946 INFO:tasks.ceph.osd.2.smithi188.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5606f88d9000]
2020-03-08T03:10:27.946 INFO:tasks.ceph.osd.2.smithi188.stderr: 14: (()+0x7ea5) [0x7f47d9910ea5]
2020-03-08T03:10:27.946 INFO:tasks.ceph.osd.2.smithi188.stderr: 15: (clone()+0x6d) [0x7f47d87d48cd]
2020-03-08T03:10:27.946 INFO:tasks.ceph.osd.2.smithi188.stderr:
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: ceph version 14.2.8-60-ga872021 (a8720218618e7c608c814c369c16739ada778bfd) nautilus (stable)
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5606f82379c1]
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: 2: (()+0x4cab89) [0x5606f8237b89]
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, bool, std::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)14, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0x5cd) [0x5606f8453a4d]
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: 4: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xdea) [0x5606f8448cba]
2020-03-08T03:10:27.947 INFO:tasks.ceph.osd.2.smithi188.stderr: 5: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x5606f83ab454]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 6: (PG::RecoveryState::Stray::react(MLogRec const&)+0x22b) [0x5606f83ee5cb]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 7: (boost::statechart::simple_state<PG::RecoveryState::Stray, PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xa5) [0x5606f843ce85]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 8: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x5606f840097d]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 9: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x5606f833d374]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 10: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x5606f85a5df1]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x5606f8331fff]
2020-03-08T03:10:27.948 INFO:tasks.ceph.osd.2.smithi188.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5606f88d64e6]
2020-03-08T03:10:27.949 INFO:tasks.ceph.osd.2.smithi188.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5606f88d9000]
2020-03-08T03:10:27.949 INFO:tasks.ceph.osd.2.smithi188.stderr: 14: (()+0x7ea5) [0x7f47d9910ea5]
2020-03-08T03:10:27.949 INFO:tasks.ceph.osd.2.smithi188.stderr: 15: (clone()+0x6d) [0x7f47d87d48cd]
Updated by Neha Ojha about 4 years ago
- Project changed from Ceph to RADOS
- Subject changed from "519: FAILED ceph_assert(head.version == 0 || e.version.version > head.version)" in rados to nautilus: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) in PGLog::IndexedLog::add()
Updated by xie xingguo about 4 years ago
- Status changed from New to Pending Backport
- Backport set to octopus,nautilus
- Pull request ID set to 33910
Updated by Nathan Cutler about 4 years ago
- Copied to Backport #44841: nautilus: nautilus: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) in PGLog::IndexedLog::add() added
Updated by Nathan Cutler about 4 years ago
- Copied to Backport #44842: octopus: nautilus: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) in PGLog::IndexedLog::add() added
Updated by Nathan Cutler almost 4 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions