Project

General

Profile

Bug #54742

crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)

Added by Telemetry Bot 11 months ago. Updated 6 months ago.

Status:
Pending Backport
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
gc backport_processed
Backport:
octopus pacific quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

cccf4e2c72e1f2034e1b59190d332599cb03a876fac92ea478e0098435b238d3


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596

Sanitized backtrace:

    RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    RGWRados::Object::complete_atomic_modification(DoutPrefixProvider const*)
    RGWRados::Object::Write::_do_write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, bool, bool, void*, optional_yield)
    RGWRados::Object::Write::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, optional_yield)
    rgw::sal::RGWRadosObject::RadosWriteOp::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, optional_yield)
    rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, optional_yield)
    RGWRados::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw_placement_rule const&, RGWRados::Object::Read&, long, rgw::sal::RGWObject*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)
    rgw::sal::RGWRadosObject::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw::sal::RGWObject*, unsigned short, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)
    RGWRados::rewrite_obj(RGWBucketInfo&, rgw::sal::RGWObject*, DoutPrefixProvider const*, optional_yield)

Crash dump sample:
{
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1f967ec140]",
        "(RGWGC::send_chain(cls_rgw_obj_chain&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc4) [0x5592a0841824]",
        "(RGWRados::Object::complete_atomic_modification(DoutPrefixProvider const*)+0xc9) [0x5592a0595089]",
        "(RGWRados::Object::Write::_do_write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, optional_yield)+0x9c8) [0x5592a05a5ab8]",
        "(RGWRados::Object::Write::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, optional_yield)+0x4c0) [0x5592a05a6d60]",
        "(rgw::sal::RGWRadosObject::RadosWriteOp::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, optional_yield)+0x1b) [0x5592a066f23b]",
        "(rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, optional_yield)+0x2f8) [0x5592a0560b78]",
        "(RGWRados::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw_placement_rule const&, RGWRados::Object::Read&, long, rgw::sal::RGWObject*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)+0x629) [0x5592a05ac2a9]",
        "(rgw::sal::RGWRadosObject::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw::sal::RGWObject*, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)+0x4ee) [0x5592a067795e]",
        "(RGWRados::rewrite_obj(RGWBucketInfo&, rgw::sal::RGWObject*, DoutPrefixProvider const*, optional_yield)+0x518) [0x5592a057bbb8]",
        "main()",
        "__libc_start_main()",
        "_start()" 
    ],
    "ceph_version": "16.2.7",
    "crash_id": "2022-02-10T03:31:38.014448Z_aa8125f0-6c6e-46a1-9ed3-be6f9f8814ee",
    "entity_name": "client.11dda3ee264a6f5da69afbe0d0b88503ddbddc81",
    "os_id": "11",
    "os_name": "Debian GNU/Linux 11 (bullseye)",
    "os_version": "11 (bullseye)",
    "os_version_id": "11",
    "process_name": "radosgw-admin",
    "stack_sig": "cccf4e2c72e1f2034e1b59190d332599cb03a876fac92ea478e0098435b238d3",
    "timestamp": "2022-02-10T03:31:38.014448Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.10.0-9-amd64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Debian 5.10.70-1 (2021-09-30)" 
}


Related issues

Related to rgw - Bug #54167: segfault in radosgw-admin bucket rewrite --bucket Resolved
Related to rgw - Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Duplicate
Copied to rgw - Backport #55226: octopus: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) New
Copied to rgw - Backport #55227: quincy: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) In Progress
Copied to rgw - Backport #55228: pacific: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) In Progress

History

#1 Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.7 added

#2 Updated by Casey Bodley 10 months ago

  • Related to Bug #54167: segfault in radosgw-admin bucket rewrite --bucket added

#3 Updated by Casey Bodley 10 months ago

  • Assignee set to Pritha Srivastava

#4 Updated by Pritha Srivastava 10 months ago

@Casey: Does this crash occur, when a radosgw-admin bucket rewrite command is executed? I am unable to access http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596, is there anyone I can get in touch with to get more details on this bug?

#5 Updated by Casey Bodley 10 months ago

Pritha Srivastava wrote:

@Casey: Does this crash occur, when a radosgw-admin bucket rewrite command is executed?

yes, https://tracker.ceph.com/issues/54167 tracks that one

I am unable to access http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596, is there anyone I can get in touch with to get more details on this bug?

i don't think we can get much detail outside of these backtraces

#6 Updated by Yaarit Hatuka 10 months ago

Hi Pritha,

To access the dashboard you need:
  1. access to the sepia lab
  2. to be a member of the Ceph organization in GitHub.

Once you have these, you can sign in to the dashboard with GitHub.

The dashboard lets you see all crash instances of this crash signature, the distribution of these instances by version, metadata of the anonymized clusters that experienced these crashes, and in case applicable, the contact details of the operators of these clusters (sometimes users choose to identify themself).

#7 Updated by Pritha Srivastava 10 months ago

  • Pull request ID set to 45760

#8 Updated by Casey Bodley 10 months ago

  • Status changed from New to Pending Backport
  • Tags set to gc
  • Backport set to octopus pacific quincy

#9 Updated by Backport Bot 10 months ago

  • Copied to Backport #55226: octopus: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#10 Updated by Backport Bot 10 months ago

  • Copied to Backport #55227: quincy: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#11 Updated by Backport Bot 10 months ago

  • Copied to Backport #55228: pacific: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#12 Updated by Casey Bodley 7 months ago

  • Related to Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#13 Updated by Backport Bot 6 months ago

  • Tags changed from gc to gc backport_processed

Also available in: Atom PDF