Project

General

Profile

Bug #54742

crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)

Added by Telemetry Bot about 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Target version:
-
% Done:

100%

Source:
Telemetry
Tags:
gc backport_processed
Backport:
octopus pacific quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

cccf4e2c72e1f2034e1b59190d332599cb03a876fac92ea478e0098435b238d3


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596

Sanitized backtrace:

    RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    RGWRados::Object::complete_atomic_modification(DoutPrefixProvider const*)
    RGWRados::Object::Write::_do_write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, bool, bool, void*, optional_yield)
    RGWRados::Object::Write::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, optional_yield)
    rgw::sal::RGWRadosObject::RadosWriteOp::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, optional_yield)
    rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, optional_yield)
    RGWRados::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw_placement_rule const&, RGWRados::Object::Read&, long, rgw::sal::RGWObject*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >&, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)
    rgw::sal::RGWRadosObject::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw::sal::RGWObject*, unsigned short, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)
    RGWRados::rewrite_obj(RGWBucketInfo&, rgw::sal::RGWObject*, DoutPrefixProvider const*, optional_yield)

Crash dump sample:
{
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1f967ec140]",
        "(RGWGC::send_chain(cls_rgw_obj_chain&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc4) [0x5592a0841824]",
        "(RGWRados::Object::complete_atomic_modification(DoutPrefixProvider const*)+0xc9) [0x5592a0595089]",
        "(RGWRados::Object::Write::_do_write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, bool, void*, optional_yield)+0x9c8) [0x5592a05a5ab8]",
        "(RGWRados::Object::Write::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, optional_yield)+0x4c0) [0x5592a05a6d60]",
        "(rgw::sal::RGWRadosObject::RadosWriteOp::write_meta(DoutPrefixProvider const*, unsigned long, unsigned long, optional_yield)+0x1b) [0x5592a066f23b]",
        "(rgw::putobj::AtomicObjectProcessor::complete(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rgw_zone_set*, bool*, optional_yield)+0x2f8) [0x5592a0560b78]",
        "(RGWRados::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw_placement_rule const&, RGWRados::Object::Read&, long, rgw::sal::RGWObject*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)+0x629) [0x5592a05ac2a9]",
        "(rgw::sal::RGWRadosObject::copy_obj_data(RGWObjectCtx&, rgw::sal::RGWBucket*, rgw::sal::RGWObject*, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, DoutPrefixProvider const*, optional_yield)+0x4ee) [0x5592a067795e]",
        "(RGWRados::rewrite_obj(RGWBucketInfo&, rgw::sal::RGWObject*, DoutPrefixProvider const*, optional_yield)+0x518) [0x5592a057bbb8]",
        "main()",
        "__libc_start_main()",
        "_start()" 
    ],
    "ceph_version": "16.2.7",
    "crash_id": "2022-02-10T03:31:38.014448Z_aa8125f0-6c6e-46a1-9ed3-be6f9f8814ee",
    "entity_name": "client.11dda3ee264a6f5da69afbe0d0b88503ddbddc81",
    "os_id": "11",
    "os_name": "Debian GNU/Linux 11 (bullseye)",
    "os_version": "11 (bullseye)",
    "os_version_id": "11",
    "process_name": "radosgw-admin",
    "stack_sig": "cccf4e2c72e1f2034e1b59190d332599cb03a876fac92ea478e0098435b238d3",
    "timestamp": "2022-02-10T03:31:38.014448Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.10.0-9-amd64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Debian 5.10.70-1 (2021-09-30)" 
}


Related issues

Related to rgw - Bug #54167: segfault in radosgw-admin bucket rewrite --bucket Resolved
Duplicated by rgw - Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Duplicate
Copied to rgw - Backport #55226: octopus: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Rejected
Copied to rgw - Backport #55227: quincy: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Resolved
Copied to rgw - Backport #55228: pacific: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) Resolved

History

#1 Updated by Telemetry Bot about 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.7 added

#2 Updated by Casey Bodley almost 2 years ago

  • Related to Bug #54167: segfault in radosgw-admin bucket rewrite --bucket added

#3 Updated by Casey Bodley almost 2 years ago

  • Assignee set to Pritha Srivastava

#4 Updated by Pritha Srivastava almost 2 years ago

@Casey: Does this crash occur, when a radosgw-admin bucket rewrite command is executed? I am unable to access http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596, is there anyone I can get in touch with to get more details on this bug?

#5 Updated by Casey Bodley almost 2 years ago

Pritha Srivastava wrote:

@Casey: Does this crash occur, when a radosgw-admin bucket rewrite command is executed?

yes, https://tracker.ceph.com/issues/54167 tracks that one

I am unable to access http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1bb7087131c2a460e20d577039a2a23e09f8afa768dc38a8dc28d4bda9cc8596, is there anyone I can get in touch with to get more details on this bug?

i don't think we can get much detail outside of these backtraces

#6 Updated by Yaarit Hatuka almost 2 years ago

Hi Pritha,

To access the dashboard you need:
  1. access to the sepia lab
  2. to be a member of the Ceph organization in GitHub.

Once you have these, you can sign in to the dashboard with GitHub.

The dashboard lets you see all crash instances of this crash signature, the distribution of these instances by version, metadata of the anonymized clusters that experienced these crashes, and in case applicable, the contact details of the operators of these clusters (sometimes users choose to identify themself).

#7 Updated by Pritha Srivastava almost 2 years ago

  • Pull request ID set to 45760

#8 Updated by Casey Bodley almost 2 years ago

  • Status changed from New to Pending Backport
  • Tags set to gc
  • Backport set to octopus pacific quincy

#9 Updated by Backport Bot almost 2 years ago

  • Copied to Backport #55226: octopus: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#10 Updated by Backport Bot almost 2 years ago

  • Copied to Backport #55227: quincy: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#11 Updated by Backport Bot almost 2 years ago

  • Copied to Backport #55228: pacific: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#12 Updated by Casey Bodley over 1 year ago

  • Related to Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

#13 Updated by Backport Bot over 1 year ago

  • Tags changed from gc to gc backport_processed

#14 Updated by Konstantin Shalygin about 1 year ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100

#15 Updated by Yaarit Hatuka 11 months ago

  • Related to deleted (Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))

#16 Updated by Yaarit Hatuka 11 months ago

  • Duplicated by Bug #56792: crash: RGWGC::send_chain(cls_rgw_obj_chain&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) added

Also available in: Atom PDF