Bug #36271
Updated by Patrick Donnelly over 5 years ago
<pre>
2018-10-01T12:36:38.744 INFO:tasks.ceph.osd.5.smithi161.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-3783-g1b501c5/rpm/el7/BUILD/ceph-14.0.0-3783-g1b501c5/src/common/interval_map.h: In function 'void interval_map<K, V, S>::insert(K, K, V&&) [with K = long unsigned int; V = ceph::buffer::list; S = bl_split_merge]' thread 7f9cedef2700 time 2018-10-01 12:36:38.751518
2018-10-01T12:36:38.744 INFO:tasks.ceph.osd.5.smithi161.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-3783-g1b501c5/rpm/el7/BUILD/ceph-14.0.0-3783-g1b501c5/src/common/interval_map.h: 161: FAILED ceph_assert(len > 0)
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: ceph version 14.0.0-3783-g1b501c5 (1b501c50886d0c90deaa251e65053ddd901e360b) nautilus (dev)
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x153) [0x55a9ef36a83f]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55a9ef36aa06]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 3: (CallClientContexts::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0xe17) [0x55a9ef7d4b17]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 4: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x8c) [0x55a9ef7a92fc]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcb2) [0x55a9ef7c3be2]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x9f) [0x55a9ef7c464f]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x87) [0x55a9ef69d127]
2018-10-01T12:36:38.747 INFO:tasks.ceph.osd.5.smithi161.stderr: 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x6c9) [0x55a9ef64d1f9]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55a9ef496af9]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55a9ef71f832]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x681) [0x55a9ef4aca91]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3e6) [0x55a9efac7286]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a9efacea20]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 14: (()+0x7dd5) [0x7f9d0ccc0dd5]
2018-10-01T12:36:38.748 INFO:tasks.ceph.osd.5.smithi161.stderr: 15: (clone()+0x6d) [0x7f9d0bb89b3d]
</pre>
From: /ceph/teuthology-archive/pdonnell-2018-10-01_03:19:12-multimds-wip-pdonnell-testing-20181001.011252-distro-basic-smithi/3090388/teuthology.log
Logs and core dumps available (now thanks to some QA fixes!).
Looks like #14056 is back.