Project

General

Profile

Bug #55765

RGW Segmentation fault when requested resource includes a colon

Added by Thomas Mertz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Adam Emerson
Target version:
-
% Done:

0%

Source:
Tags:
iam backport_processed
Backport:
quincy, pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v16.2.7, Ceph - v16.2.9
ceph-qa-suite:
Pull request ID:
47025
Crash signature (v1):

169298290c02b280707bc0349d28d55011b0412fff276ae7a4a9fb1ecb6b7653
185823fb9cb2ebec1d3f2d996e9b430f4f462cb8457d0b428a80d668b627a93b
3d8bd0ab19b12dacd44b0317148da8e88c43c00daf88b40957cff16d03e92725
dcde260ceb1b6979ac945709c0b63748ee0035ccffe960951d4c89432d1fcdc1

Crash signature (v2):

ff6e74fd7658aea648955e8dc49b28eb325b135256cf92394ebfccc32c8dbf10

Description

Hello,
We found that rgw crashing when receiving specific requests

image: quay.io/ceph/ceph:v16.2.9-20220519
version: 16.2.9-0

request exemple :

GET https://storage.example.org/:
GET https://storage.example.org/https:///example.com/%2f..

rgw logs:

debug 2022-05-25T11:32:01.748146233+02:00    -11> 2022-05-25T09:32:01.678+0000 7f8fdc412700  1 ====== starting new request req=0x7f90f0912630 =====
debug    -10> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s initializing for trans_id = tx0000090aa4432f2202fac-00628df791-8944b7-ch-gva-d3
debug     -9> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s getting op 0
debug     -8> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying requester
debug     -7> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket normalizing buckets and tenants
debug     -6> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket init permissions
debug     -5> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket recalculating target
debug     -4> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket reading permissions2022-05-25T11:32:01.748164980+02:00 
debug     -3> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket init op
debug     -2> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying op mask
debug     -1> 2022-05-25T09:32:01.678+0000 7f8fdc412700  2 req 10424219273103683500 0.000000000s s3:list_bucket verifying op permissions
debug      0> 2022-05-25T09:32:01.678+0000 7f8fdc412700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f8fdc412700 thread_name:radosgw

 ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f90e500ace0]
 2: (rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7f90efcb2d82]
 3: (verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7f90efd035d2]
 4: (verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7f90efd04403]
 5: (RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7f90efed7eae]
 6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7f90efb8400b]
 7: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7f90efb881c1]
 8: /lib64/libradosgw.so.2(+0x43d640) [0x7f90efb07640]
 9: /lib64/libradosgw.so.2(+0x43ef6a) [0x7f90efb08f6a]
 10: make_fcontext()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

not all requests containing colon are affected but it seems to be the thing, example without problem :

GET https://storage.example.org/fffff:fffff

Related issues

Duplicated by rgw - Bug #56248: crash: rgw::ARN::ARN(rgw_bucket const&) Duplicate
Copied to rgw - Backport #56585: quincy: RGW Segmentation fault when requested resource includes a colon Resolved
Copied to rgw - Backport #56586: pacific: RGW Segmentation fault when requested resource includes a colon Resolved

History

#1 Updated by Casey Bodley over 1 year ago

  • Assignee set to Adam Emerson
  • Tags set to iam

#2 Updated by Thomas Mertz over 1 year ago

Hello, we find the same behavior with the corresponding hex code (%3A,%3a)

GET https://storage.example.org//https%3a//example.com/%2e%2e%2f

#3 Updated by J. Eric Ivancich over 1 year ago

  • Subject changed from RGW Segmentation fault when requested ressource including colon to RGW Segmentation fault when requested resource includes a colon

#4 Updated by Adam Emerson over 1 year ago

  • Assignee changed from Adam Emerson to Pritha Srivastava

#5 Updated by Casey Bodley about 1 year ago

  • Assignee changed from Pritha Srivastava to Adam Emerson
  • Priority changed from Normal to High

#6 Updated by Casey Bodley about 1 year ago

  • Duplicated by Bug #56248: crash: rgw::ARN::ARN(rgw_bucket const&) added

#7 Updated by Adam Emerson about 1 year ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 47025

#8 Updated by Adam Emerson about 1 year ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to quincy, pacific

#9 Updated by Backport Bot about 1 year ago

  • Copied to Backport #56585: quincy: RGW Segmentation fault when requested resource includes a colon added

#10 Updated by Backport Bot about 1 year ago

  • Copied to Backport #56586: pacific: RGW Segmentation fault when requested resource includes a colon added

#11 Updated by Telemetry Bot about 1 year ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.9 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=ff6e74fd7658aea648955e8dc49b28eb325b135256cf92394ebfccc32c8dbf10

Sanitized backtrace:

    rgw::ARN::ARN(rgw_bucket const&)
    verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)
    verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)
    RGWListBucket::verify_permission(optional_yield)
    rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)
    process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)
    make_fcontext()

Crash dump sample:
{
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7fb6ed3eece0]",
        "(rgw::ARN::ARN(rgw_bucket const&)+0x42) [0x7fb6f8096d82]",
        "(verify_bucket_permission(DoutPrefixProvider const*, perm_state_base*, rgw_bucket const&, RGWAccessControlPolicy*, RGWAccessControlPolicy*, boost::optional<rgw::IAM::Policy> const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, std::vector<rgw::IAM::Policy, std::allocator<rgw::IAM::Policy> > const&, unsigned long)+0xa2) [0x7fb6f80e75d2]",
        "(verify_bucket_permission(DoutPrefixProvider const*, req_state*, unsigned long)+0x83) [0x7fb6f80e8403]",
        "(RGWListBucket::verify_permission(optional_yield)+0x12e) [0x7fb6f82bbeae]",
        "(rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x81b) [0x7fb6f7f6800b]",
        "(process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x2891) [0x7fb6f7f6c1c1]",
        "/lib64/libradosgw.so.2(+0x43d640) [0x7fb6f7eeb640]",
        "/lib64/libradosgw.so.2(+0x43ef6a) [0x7fb6f7eecf6a]",
        "make_fcontext()" 
    ],
    "ceph_version": "16.2.9",
    "crash_id": "2022-07-06T06:50:37.857068Z_08d6f7d1-ed37-4e2e-9dfd-84b44d96c882",
    "entity_name": "client.75b24a6517f8f1a939bbac4dca059daf1137dd8c",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "radosgw",
    "stack_sig": "3d8bd0ab19b12dacd44b0317148da8e88c43c00daf88b40957cff16d03e92725",
    "timestamp": "2022-07-06T06:50:37.857068Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-193.10.el8.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Thu Apr 23 17:33:15 UTC 2020" 
}

#12 Updated by Yuri Weinstein about 1 year ago

  • Crash signature (v1) updated (diff)

#14 Updated by Backport Bot about 1 year ago

  • Tags changed from iam to iam backport_processed

#15 Updated by Adam Emerson about 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF