Project

General

Profile

Actions
Copy link

Bug #50806

closed

osd/PrimaryLogPG.cc: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))

Added by Neha Ojha almost 3 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Myoungwon Oh
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
41373
Crash signature (v1):
Crash signature (v2):

Description

2021-05-13T21:33:56.923 INFO:tasks.ceph.osd.6.smithi165.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-4115-g859d11fe/rpm/el8/BUILD/ceph-17.0.0-4115-g859d11fe/src/osd/PrimaryLogPG.cc: In function 'ObjectContextRef PrimaryLogPG::get_object_context(const hobject_t&, bool, const std::map<std::__cxx11::basic_string<char>, ceph::buffer::v15_2_0::list>*)' thread 7f1ad379a700 time 2021-05-13T21:33:56.898667+0000
2021-05-13T21:33:56.924 INFO:tasks.ceph.osd.6.smithi165.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-4115-g859d11fe/rpm/el8/BUILD/ceph-17.0.0-4115-g859d11fe/src/osd/PrimaryLogPG.cc: 11771: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))
2021-05-13T21:33:56.924 INFO:tasks.ceph.osd.6.smithi165.stderr: ceph version 17.0.0-4115-g859d11fe (859d11fe4eb7ccd44aee091e7fec489a444a9301) quincy (dev)
2021-05-13T21:33:56.924 INFO:tasks.ceph.osd.6.smithi165.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55aa8440f584]
2021-05-13T21:33:56.924 INFO:tasks.ceph.osd.6.smithi165.stderr: 2: ceph-osd(+0x5b078c) [0x55aa8440f78c]
2021-05-13T21:33:56.925 INFO:tasks.ceph.osd.6.smithi165.stderr: 3: (PrimaryLogPG::get_object_context(hobject_t const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const*)+0x6ff) [0x55aa84624cef]
2021-05-13T21:33:56.925 INFO:tasks.ceph.osd.6.smithi165.stderr: 4: (PrimaryLogPG::get_adjacent_clones(std::shared_ptr<ObjectContext>, std::shared_ptr<ObjectContext>&, std::shared_ptr<ObjectContext>&)+0xc5) [0x55aa84625f45]
2021-05-13T21:33:56.925 INFO:tasks.ceph.osd.6.smithi165.stderr: 5: (PrimaryLogPG::get_manifest_ref_count(std::shared_ptr<ObjectContext>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::intrusive_ptr<OpRequest>)+0x433) [0x55aa846467d3]
2021-05-13T21:33:56.925 INFO:tasks.ceph.osd.6.smithi165.stderr: 6: (cls_get_manifest_ref_count(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x76) [0x55aa844c0856]
2021-05-13T21:33:56.925 INFO:tasks.ceph.osd.6.smithi165.stderr: 7: /usr/lib64/rados-classes/libcls_cas.so(+0xb814) [0x7f1ae6752814]
2021-05-13T21:33:56.926 INFO:tasks.ceph.osd.6.smithi165.stderr: 8: ceph-osd(+0x74782c) [0x55aa845a682c]
2021-05-13T21:33:56.926 INFO:tasks.ceph.osd.6.smithi165.stderr: 9: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5e) [0x55aa845a6b9e]
2021-05-13T21:33:56.926 INFO:tasks.ceph.osd.6.smithi165.stderr: 10: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0x1771) [0x55aa84664e21]
2021-05-13T21:33:56.926 INFO:tasks.ceph.osd.6.smithi165.stderr: 11: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x177) [0x55aa84678467]
2021-05-13T21:33:56.927 INFO:tasks.ceph.osd.6.smithi165.stderr: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x31d) [0x55aa8467a54d]
2021-05-13T21:33:56.927 INFO:tasks.ceph.osd.6.smithi165.stderr: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2dcf) [0x55aa84683eff]
2021-05-13T21:33:56.927 INFO:tasks.ceph.osd.6.smithi165.stderr: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd1c) [0x55aa8468b0fc]
2021-05-13T21:33:56.927 INFO:tasks.ceph.osd.6.smithi165.stderr: 15: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x55aa84513cc9]
2021-05-13T21:33:56.927 INFO:tasks.ceph.osd.6.smithi165.stderr: 16: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x55aa847752b8]
2021-05-13T21:33:56.928 INFO:tasks.ceph.osd.6.smithi165.stderr: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xa58) [0x55aa84531188]
2021-05-13T21:33:56.928 INFO:tasks.ceph.osd.6.smithi165.stderr: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55aa84bbc624]
2021-05-13T21:33:56.928 INFO:tasks.ceph.osd.6.smithi165.stderr: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55aa84bbd9c4]
2021-05-13T21:33:56.928 INFO:tasks.ceph.osd.6.smithi165.stderr: 20: (Thread::_entry_func(void*)+0xd) [0x55aa84ba36ed]
2021-05-13T21:33:56.928 INFO:tasks.ceph.osd.6.smithi165.stderr: 21: /lib64/libpthread.so.0(+0x814a) [0x7f1af7fad14a]
2021-05-13T21:33:56.929 INFO:tasks.ceph.osd.6.smithi165.stderr: 22: clone()

/a/nojha-2021-05-13_20:07:18-rados-wip-41317-distro-basic-smithi/6114075

Related issues 2 (0 open — 2 closed)

Related to RADOS - Bug #49427: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))ResolvedMyoungwon Oh

Actions
Copied to RADOS - Backport #50893: pacific: osd/PrimaryLogPG.cc: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))ResolvedMyoungwon OhActions
Actions
Copy link
 #1

Updated by Deepika Upadhyay almost 3 years ago

additional relevant log: 

2021-05-17T11:13:43.167 INFO:tasks.ceph.osd.5.smithi150.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.3-90-g02a5217e/rpm/el8/BUILD/ceph-16.2.3-90-g02a5217e/src/osd/PrimaryLogPG.cc: 11462: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT))

/ceph/teuthology-archive/ideepika-2021-05-17_10:16:28-rados-wip-yuri2-testing-2021-05-11-1032-pacific-distro-basic-smithi/6118166/teuthology.log

Actions
Copy link
 #2

Updated by Neha Ojha almost 3 years ago

  • Related to Bug #49427: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) added
Actions
Copy link
 #3

Updated by Neha Ojha almost 3 years ago

  • Assignee set to Myoungwon Oh
Actions
Copy link
 #5

Updated by Neha Ojha almost 3 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 41373
Actions
Copy link
 #6

Updated by Neha Ojha almost 3 years ago

Myoungwon Oh wrote:

https://github.com/ceph/ceph/pull/41373

how far back should we backport this?

Actions
Copy link
 #7

Updated by Myoungwon Oh almost 3 years ago

  • Backport set to pacific

I think pacific.

Actions
Copy link
 #8

Updated by Kefu Chai almost 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #9

Updated by Backport Bot almost 3 years ago

  • Copied to Backport #50893: pacific: osd/PrimaryLogPG.cc: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soid) || (it_objects != recovery_state.get_pg_log().get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) added
Actions
Copy link
 #10

Updated by Sridhar Seshasayee almost 3 years ago

Observed on master:
/a/sseshasa-2021-06-01_08:27:04-rados-wip-sseshasa-testing-objs-test-2-distro-basic-smithi/6145024/

Unfortunately there are no osd logs or cores.

Actions
Copy link
 #12

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions
Copy link
 #13

Updated by Konstantin Shalygin over 1 year ago

  • Status changed from Pending Backport to Resolved
  • Tags deleted (backport_processed)
Actions
Copy link

Also available in: Atom PDF