Project

General

Profile

Bug #50192

FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))

Added by Neha Ojha almost 3 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-04-03T12:45:16.005 INFO:tasks.ceph.osd.6.smithi183.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-2788-g0d634484/rpm/el8/BUILD/ceph-17.0.0-2788-g0d634484/src/osd/PrimaryLogPG.cc: 11504: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: ceph version 17.0.0-2788-g0d634484 (0d6344840d6904e667a339b95a33d5effbffaf9f) quincy (dev)
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x6b006a]
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 2: ceph-osd(+0x5a8272) [0x6b0272]
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 3: (PrimaryLogPG::get_object_context(hobject_t const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const*)+0x6ff) [0x8c857f]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 4: (PrimaryLogPG::get_manifest_ref_count(std::shared_ptr<ObjectContext>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x33d) [0x8ca30d]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 5: (cls_get_manifest_ref_count(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x5d) [0x766e9d]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 6: /usr/lib64/rados-classes/libcls_cas.so(+0xbd14) [0x1b9b5d14]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 7: ceph-osd(+0x743ffc) [0x84bffc]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5e) [0x84c36e]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0x17e1) [0x906541]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x177) [0x919b27]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x31d) [0x91bc0d]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2ddc) [0x92559c]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd1c) [0x92c79c]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x7b8a79]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0xa12718]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x7d5cd8]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0xe36fc4]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0xe39c64]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 19: (Thread::_entry_func(void*)+0xd) [0xe28acd]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 20: /lib64/libpthread.so.0(+0x814a) [0x6cc014a]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 21: clone()

rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}

/a/kchai-2021-04-03_11:44:43-rados-wip-kefu-testing-2021-04-03-1318-distro-basic-smithi/6017847

reference: https://tracker.ceph.com/issues/49427#note-11


Related issues

Copied to RADOS - Backport #50274: pacific: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) Resolved

History

#1 Updated by Neha Ojha almost 3 years ago

  • Backport set to pacific

#2 Updated by Neha Ojha almost 3 years ago

  • Status changed from Fix Under Review to Pending Backport

#3 Updated by Backport Bot almost 3 years ago

  • Copied to Backport #50274: pacific: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) added

#4 Updated by Sridhar Seshasayee over 2 years ago

Observed on master:
/a/sseshasa-2021-06-01_08:27:04-rados-wip-sseshasa-testing-objs-test-2-distro-basic-smithi/6145024/

Unfortunately there are no osd logs or cores.

#5 Updated by Kefu Chai over 2 years ago

  • Status changed from Pending Backport to New

#6 Updated by Kefu Chai over 2 years ago

  • Status changed from New to Pending Backport

#7 Updated by Neha Ojha about 2 years ago

Needs a pacific backport, showed up in pacific

2021-11-28T17:39:29.135 INFO:tasks.ceph.osd.5.smithi140.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6-681-gfdc003bc/rpm/el8/BUILD/ceph-16.2.6-681-gfdc003bc/src/osd/PrimaryLogPG.cc: 11473: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))
2021-11-28T17:39:29.209 INFO:tasks.ceph.osd.7.smithi140.stderr:2021-11-28T17:39:29.208+0000 14a4f700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7  (PID: 77729) UID: 0
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: ceph version 16.2.6-681-gfdc003bc (fdc003bc12f1b2443c4596eeacb32cf62e806970) pacific (stable)
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x672b7e]
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 2: ceph-osd(+0x56ad98) [0x672d98]
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 3: (PrimaryLogPG::get_object_context(hobject_t const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const*)+0x6ff) [0x88db2f]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 4: (PrimaryLogPG::get_manifest_ref_count(std::shared_ptr<ObjectContext>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x33d) [0x88f8bd]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 5: (cls_get_manifest_ref_count(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x5d) [0x7287bd]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 6: /usr/lib64/rados-classes/libcls_cas.so(+0xbdc4) [0x23774dc4]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 7: ceph-osd(+0x70920c) [0x81120c]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5e) [0x81157e]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0x17e1) [0x8cbcc1]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x177) [0x8df2a7]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x326) [0x8e1396]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2de2) [0x8ead52]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd1c) [0x8f209c]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x77a8e9]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x9d8838]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x79ad48]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0xe0c5b4]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0xe0f254]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0xc80014a]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 20: clone()

/a/yuriw-2021-11-28_15:46:54-rados-pacific-16.2.7_RC1-distro-default-smithi/6532080

#9 Updated by Laura Flores about 2 years ago

/a/yuriw-2022-01-06_15:50:38-rados-wip-yuri8-testing-2022-01-05-1411-distro-default-smithi/6599338

#10 Updated by Laura Flores about 2 years ago

/a/yuriw-2022-01-27_15:09:25-rados-wip-yuri6-testing-2022-01-26-1547-distro-default-smithi/6644223

#11 Updated by Yuri Weinstein about 2 years ago

Myoungwon Oh wrote:

https://github.com/ceph/ceph/pull/44181

merged

#12 Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed

#13 Updated by Konstantin Shalygin over 1 year ago

  • Status changed from Pending Backport to Resolved
  • Tags deleted (backport_processed)

Also available in: Atom PDF