Bug #55765: RGW Segmentation fault when requested resource includes a colon - rgw - Ceph

Actions

Copy link

Bug #55765

closed

RGW Segmentation fault when requested resource includes a colon

Added by Thomas Mertz almost 2 years ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

Adam Emerson

Target version:

% Done:

Source:

Tags:

iam backport_processed

Backport:

quincy, pacific

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v16.2.7, Ceph - v16.2.9

ceph-qa-suite:

Pull request ID:

47025

Crash signature (v1):

169298290c02b280707bc0349d28d55011b0412fff276ae7a4a9fb1ecb6b7653
185823fb9cb2ebec1d3f2d996e9b430f4f462cb8457d0b428a80d668b627a93b
3d8bd0ab19b12dacd44b0317148da8e88c43c00daf88b40957cff16d03e92725
dcde260ceb1b6979ac945709c0b63748ee0035ccffe960951d4c89432d1fcdc1

Crash signature (v2):

ff6e74fd7658aea648955e8dc49b28eb325b135256cf92394ebfccc32c8dbf10

Description

Hello,
We found that rgw crashing when receiving specific requests

image: quay.io/ceph/ceph:v16.2.9-20220519
version: 16.2.9-0

request exemple :

GET https://storage.example.org/:
GET https://storage.example.org/https:///example.com/%2f..

rgw logs:

debug 2022-05-25T11:32:01.748146233+02:00    -11> 2022-05-25T09:32:01.678+0000 7f8fdc412700  1 ====== starting new request req=0x7f90f0912630 =====
debug    -10> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s initializing for trans_id = tx0000090aa4432f2202fac-00628df791-8944b7-ch-gva-d3
debug     -9> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s getting op 0
debug     -8> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying requester
debug     -7> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket normalizing buckets and tenants
debug     -6> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket init permissions
debug     -5> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket recalculating target
debug     -4> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket reading permissions2022-05-25T11:32:01.748164980+02:00 
debug     -3> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket init op
debug     -2> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying op mask
debug     -1> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying op permissions
debug      0> 2022-05-25T09:32:01.678+0000 7f8fdc412700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f8fdc412700 thread_name:radosgw

 ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f90e500ace0]
 2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7f90efcb2d82]
 3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7f90efd035d2]
 4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7f90efd04403]
 5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7f90efed7eae]
 6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7f90efb8400b]
 7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7f90efb881c1]
 8: /lib64/libradosgw.so.2(+0x43d640) [0x7f90efb07640]
 9: /lib64/libradosgw.so.2(+0x43ef6a) [0x7f90efb08f6a]
 10: make_fcontext()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

not all requests containing colon are affected but it seems to be the thing, example without problem :

GET https://storage.example.org/fffff:fffff

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Casey Bodley almost 2 years ago

Assignee set to Adam Emerson
Tags set to iam

Actions

Copy link

Updated by Thomas Mertz almost 2 years ago

Hello, we find the same behavior with the corresponding hex code (%3A,%3a)

GET https://storage.example.org//https%3a//example.com/%2e%2e%2f

Actions

Copy link

Updated by J. Eric Ivancich almost 2 years ago

Subject changed from RGW Segmentation fault when requested ressource including colon to RGW Segmentation fault when requested resource includes a colon

Actions

Copy link

Updated by Adam Emerson almost 2 years ago

Assignee changed from Adam Emerson to Pritha Srivastava

Actions

Copy link

Updated by Casey Bodley almost 2 years ago

Assignee changed from Pritha Srivastava to Adam Emerson
Priority changed from Normal to High

Actions

Copy link

Updated by Casey Bodley almost 2 years ago

Has duplicate Bug #56248: crash: rgw::ARN::ARN(rgw_bucket const&) added

Actions

Copy link

Updated by Adam Emerson almost 2 years ago

Status changed from New to Fix Under Review
Pull request ID set to 47025

Actions

Copy link

Updated by Adam Emerson almost 2 years ago

Status changed from Fix Under Review to Pending Backport
Backport set to quincy, pacific

Actions

Copy link

Updated by Backport Bot almost 2 years ago

Copied to Backport #56585: quincy: RGW Segmentation fault when requested resource includes a colon added

Actions

Copy link

#10

Updated by Backport Bot almost 2 years ago

Copied to Backport #56586: pacific: RGW Segmentation fault when requested resource includes a colon added

Actions

Copy link

#11

Updated by Telemetry Bot over 1 year ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v16.2.9 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=ff6e74fd7658aea648955e8dc49b28eb325b135256cf92394ebfccc32c8dbf10

Sanitized backtrace:

    rgw::ARN::ARN(rgw_bucket const&)
    verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)
    verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)
    RGWListBucket::verify_permission(optional_yield)
    rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)
    process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)
    make_fcontext()

Crash dump sample:

{
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7fb6ed3eece0]",
        "(rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fb6f8096d82]",
        "(verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7fb6f80e75d2]",
        "(verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7fb6f80e8403]",
        "(RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fb6f82bbeae]",
        "(rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7fb6f7f6800b]",
        "(process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fb6f7f6c1c1]",
        "/lib64/libradosgw.so.2(+0x43d640) [0x7fb6f7eeb640]",
        "/lib64/libradosgw.so.2(+0x43ef6a) [0x7fb6f7eecf6a]",
        "make_fcontext()" 
    ],
    "ceph_version": "16.2.9",
    "crash_id": "2022-07-06T06:50:37.857068Z_08d6f7d1-ed37-4e2e-9dfd-84b44d96c882",
    "entity_name": "client.75b24a6517f8f1a939bbac4dca059daf1137dd8c",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "radosgw",
    "stack_sig": "3d8bd0ab19b12dacd44b0317148da8e88c43c00daf88b40957cff16d03e92725",
    "timestamp": "2022-07-06T06:50:37.857068Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-193.10.el8.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Thu Apr 23 17:33:15 UTC 2020" 
}

Actions

Copy link

#12