Ceph » rgw

After update from octopus v15.2.15 to pacific v16.2.7 rgw with only s3website api enabled is randomly crashing:

strange, this looks like the same crash from https://tracker.ceph.com/issues/51491, and its pacific backport https://tracker.ceph.com/issues/52468 says it was fixed in 16.2.7

I did some tests, tried to curl known fixed issues, like double ending slashes etc, with no rgw crash. Also fresh new installed rgw are crashing, so this issue was not caused by upgrade. Crashes occur about 20 times per day, at random hours, everytime producing same output like posted before. I'm open to further testing if you provide me what to check.

Is that enough? previous lines are just osd spams

33> 2022-02-04T08:36:56.009+0100 7f5968b77700 10 monclient: tick
   -32> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 HTTP_ACCEPT=*/*
   -31> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 HTTP_ACCEPT_ENCODING=deflate, gzip, br
   -30> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 HTTP_HOST=s3web.exea.pl
   -29> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 HTTP_USER_AGENT=curl/7.74.0
   -28> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 HTTP_VERSION=1.1
   -27> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 REMOTE_ADDR=10.5.11.142
   -26> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 REQUEST_METHOD=GET
   -25> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 REQUEST_URI=///installer.php
   -24> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 SCRIPT_URI=///installer.php
   -23> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 SERVER_PORT=8080
   -22> 2022-02-04T08:36:56.600+0100 7f58c422e700  1 ====== starting new request req=0x7f5859f55620 =====
   -21> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s initializing for trans_id = tx00000a1a69712f979177e-0061fcd798-1ef828a1-default
   -20> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s rgw api priority: s3=-1 s3website=1
   -19> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s host=s3web.exea.pl
   -18> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s subdomain= domain=s3web.exea.pl in_hosted_domain=1 in_hosted_domain_s3website=1
   -17> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s final domain/bucket subdomain= domain=s3web.exea.pl in_hosted_domain=1 in_hosted_domain_s3website=1 s>info.domain=s3web.exea.pl s->info.request_uri=///installer.php
   16> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s get_handler handler=33RGWHandler_REST_Service_S3Website
   -15> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s handler=33RGWHandler_REST_Service_S3Website
   -14> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s getting op 0
   -13> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1
   -12> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s s3:get_obj op=28RGWGetObj_ObjStore_S3Website
   -11> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s s3:get_obj verifying requester
   -10> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
    -9> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
    -8> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access
    -7> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access
    -6> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s s3:get_obj normalizing buckets and tenants
    -5> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s s>object=/installer.php s->bucket=
    -4> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s s3:get_obj init permissions
    -3> 2022-02-04T08:36:56.600+0100 7f58c422e700 20 req 11648163593991034750 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user
    -2> 2022-02-04T08:36:56.600+0100 7f58c422e700  2 req 11648163593991034750 0.000000000s s3:get_obj recalculating target
    -1> 2022-02-04T08:36:56.600+0100 7f58c422e700 10 req 11648163593991034750 0.000000000s retarget Starting retarget
     0> 2022-02-04T08:36:56.603+0100 7f58c422e700 -1 * Caught signal (Segmentation fault) *
 in thread 7f58c422e700 thread_name:radosgw
 ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f598f30cc20]
 2: (RGWHandler_REST_S3Website::retarget(RGWOp, RGWOp, optional_yield)+0x174) [0x7f599a385c24]
 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0xf0a) [0x7f5999f2435a]
 4: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, int*)+0x2891) [0x7f5999f27e21]
 5: /lib64/libradosgw.so.2(+0x4b1b63) [0x7f5999e77b63]
 6: /lib64/libradosgw.so.2(+0x4b3604) [0x7f5999e79604]
 7: /lib64/libradosgw.so.2(+0x4b386e) [0x7f5999e7986e]
 8: make_fcontext()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.