Bug #50192: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) - RADOS - Ceph

Actions

Copy link

Bug #50192

closed

FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))

Added by Neha Ojha about 3 years ago. Updated over 1 year ago.

Status:

Resolved

Priority:

Normal

Assignee:

Myoungwon Oh

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

40606

Crash signature (v1):

Crash signature (v2):

Description

2021-04-03T12:45:16.005 INFO:tasks.ceph.osd.6.smithi183.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-2788-g0d634484/rpm/el8/BUILD/ceph-17.0.0-2788-g0d634484/src/osd/PrimaryLogPG.cc: 11504: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: ceph version 17.0.0-2788-g0d634484 (0d6344840d6904e667a339b95a33d5effbffaf9f) quincy (dev)
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x6b006a]
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 2: ceph-osd(+0x5a8272) [0x6b0272]
2021-04-03T12:45:16.102 INFO:tasks.ceph.osd.6.smithi183.stderr: 3: (PrimaryLogPG::get_object_context(hobject_t const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const*)+0x6ff) [0x8c857f]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 4: (PrimaryLogPG::get_manifest_ref_count(std::shared_ptr<ObjectContext>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x33d) [0x8ca30d]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 5: (cls_get_manifest_ref_count(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x5d) [0x766e9d]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 6: /usr/lib64/rados-classes/libcls_cas.so(+0xbd14) [0x1b9b5d14]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 7: ceph-osd(+0x743ffc) [0x84bffc]
2021-04-03T12:45:16.103 INFO:tasks.ceph.osd.6.smithi183.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5e) [0x84c36e]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0x17e1) [0x906541]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x177) [0x919b27]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x31d) [0x91bc0d]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2ddc) [0x92559c]
2021-04-03T12:45:16.104 INFO:tasks.ceph.osd.6.smithi183.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd1c) [0x92c79c]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x7b8a79]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0xa12718]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x7d5cd8]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0xe36fc4]
2021-04-03T12:45:16.105 INFO:tasks.ceph.osd.6.smithi183.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0xe39c64]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 19: (Thread::_entry_func(void*)+0xd) [0xe28acd]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 20: /lib64/libpthread.so.0(+0x814a) [0x6cc014a]
2021-04-03T12:45:16.106 INFO:tasks.ceph.osd.6.smithi183.stderr: 21: clone()

rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}

/a/kchai-2021-04-03_11:44:43-rados-wip-kefu-testing-2021-04-03-1318-distro-basic-smithi/6017847

reference: https://tracker.ceph.com/issues/49427#note-11

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Neha Ojha about 3 years ago

Backport set to pacific

Actions

Copy link

Updated by Neha Ojha about 3 years ago

Status changed from Fix Under Review to Pending Backport

Actions

Copy link

Updated by Backport Bot about 3 years ago

Copied to Backport #50274: pacific: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) added

Actions

Copy link

Updated by Sridhar Seshasayee almost 3 years ago

Observed on master:
/a/sseshasa-2021-06-01_08:27:04-rados-wip-sseshasa-testing-objs-test-2-distro-basic-smithi/6145024/

Unfortunately there are no osd logs or cores.

Actions

Copy link

Updated by Kefu Chai almost 3 years ago

Status changed from Pending Backport to New

Actions

Copy link

Updated by Kefu Chai almost 3 years ago

Status changed from New to Pending Backport

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Needs a pacific backport, showed up in pacific

2021-11-28T17:39:29.135 INFO:tasks.ceph.osd.5.smithi140.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6-681-gfdc003bc/rpm/el8/BUILD/ceph-16.2.6-681-gfdc003bc/src/osd/PrimaryLogPG.cc: 11473: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))
2021-11-28T17:39:29.209 INFO:tasks.ceph.osd.7.smithi140.stderr:2021-11-28T17:39:29.208+0000 14a4f700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7  (PID: 77729) UID: 0
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: ceph version 16.2.6-681-gfdc003bc (fdc003bc12f1b2443c4596eeacb32cf62e806970) pacific (stable)
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x672b7e]
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 2: ceph-osd(+0x56ad98) [0x672d98]
2021-11-28T17:39:29.230 INFO:tasks.ceph.osd.5.smithi140.stderr: 3: (PrimaryLogPG::get_object_context(hobject_t const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const*)+0x6ff) [0x88db2f]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 4: (PrimaryLogPG::get_manifest_ref_count(std::shared_ptr<ObjectContext>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x33d) [0x88f8bd]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 5: (cls_get_manifest_ref_count(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x5d) [0x7287bd]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 6: /usr/lib64/rados-classes/libcls_cas.so(+0xbdc4) [0x23774dc4]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 7: ceph-osd(+0x70920c) [0x81120c]
2021-11-28T17:39:29.231 INFO:tasks.ceph.osd.5.smithi140.stderr: 8: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5e) [0x81157e]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 9: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0x17e1) [0x8cbcc1]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 10: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x177) [0x8df2a7]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x326) [0x8e1396]
2021-11-28T17:39:29.232 INFO:tasks.ceph.osd.5.smithi140.stderr: 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2de2) [0x8ead52]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd1c) [0x8f209c]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x77a8e9]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 15: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x9d8838]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x79ad48]
2021-11-28T17:39:29.233 INFO:tasks.ceph.osd.5.smithi140.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0xe0c5b4]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0xe0f254]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0xc80014a]
2021-11-28T17:39:29.234 INFO:tasks.ceph.osd.5.smithi140.stderr: 20: clone()

/a/yuriw-2021-11-28_15:46:54-rados-pacific-16.2.7_RC1-distro-default-smithi/6532080

Actions

Copy link