Bug #57518: RadosLuaManager::get_script is crashing during realm reload in lua background context - rgw - Ceph

Actions

Copy link

Bug #57518

closed

RadosLuaManager::get_script is crashing during realm reload in lua background context

Added by Yuval Lifshitz over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

Yuval Lifshitz

Target version:

% Done:

Source:

Development

Tags:

lua

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

48066

Crash signature (v1):

Crash signature (v2):

Description

see this run: http://qa-proxy.ceph.com/teuthology/yuvalif-2022-09-13_07:01:40-rgw:multisite-wip-yuval-fix-56572-distro-default-smithi/7030426/teuthology.log
backtrace:

in thread 7f2bb02d4700 thread_name:lua_background

 ceph version 17.0.0-14918-g840d0541 (840d05418c459144c92fb2a90d74a2ded29610da) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f2ce5651420]
 2: (RGWSI_SysObj::get_obj(rgw_raw_obj const&)+0x17) [0x56430213d707]
 3: (rgw_get_system_obj(RGWSI_SysObj*, rgw_pool const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, optional_yield, DoutPrefixProvider const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional<obj_version>, bool)+0x150) [0x564302611ce0]
 4: (rgw::sal::RadosLuaManager::get_script(DoutPrefixProvider const*, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x19d) [0x5643025fbc9d]
 5: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x75) [0x5643027188e5]
 6: (rgw::lua::Background::read_script()+0x9f) [0x564302723cff]
 7: (rgw::lua::Background::run()+0x6e4) [0x564302724cd4]
 8: /usr/lib/ceph/libceph-common.so.2(+0xa81eb4) [0x7f2ce6994eb4]
 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f2ce5645609]
 10: clone()

Actions

Copy link

Updated by Yuval Lifshitz over 1 year ago

log:

 -42> 2022-09-13T07:39:11.488+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -41> 2022-09-13T07:39:11.492+0000 7f2ce1746700 10 monclient: get_auth_request con 0x56430a758800 auth_method 0
   -40> 2022-09-13T07:39:11.492+0000 7f2ce0f45700 10 monclient: get_auth_request con 0x56430a2ab800 auth_method 0
   -39> 2022-09-13T07:39:11.492+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=46
   -38> 2022-09-13T07:39:11.492+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -37> 2022-09-13T07:39:11.492+0000 7f2ce0744700 10 monclient: get_auth_request con 0x56430919f000 auth_method 0
   -36> 2022-09-13T07:39:11.492+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=110
   -35> 2022-09-13T07:39:11.492+0000 7f2bb52de700 20 rgw realm reloader: realm  test-realm 66b6449a-c3fe-4ef0-92a4-9a5a70b14beb
   -34> 2022-09-13T07:39:11.492+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -33> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=110
   -32> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -31> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=10
   -30> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -29> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=1983
   -28> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -27> 2022-09-13T07:39:11.496+0000 7f2ce1746700 10 monclient: get_auth_request con 0x564308dd6c00 auth_method 0
   -26> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=46
   -25> 2022-09-13T07:39:11.496+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -24> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=0 bl.length=868
   -23> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 zone a2 found
   -22> 2022-09-13T07:39:11.500+0000 7f2bb52de700  4 rgw realm reloader: Realm:     test-realm           (66b6449a-c3fe-4ef0-92a4-9a5a70b14beb)
   -21> 2022-09-13T07:39:11.500+0000 7f2bb52de700  4 rgw realm reloader: ZoneGroup: a                    (a)
   -20> 2022-09-13T07:39:11.500+0000 7f2bb52de700  4 rgw realm reloader: Zone:      a2                   (87edf136-8bbb-4624-93bf-a8ef895d3872)
   -19> 2022-09-13T07:39:11.500+0000 7f2bb52de700  4 rgw realm reloader: using period configuration: 9a07cae8-bae8-4bf1-beea-b675819fa277:6
   -18> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 period zonegroup name a
   -17> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: using current period zonegroup a
   -16> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -15> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=-2 bl.length=0
   -14> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
   -13> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=-2 bl.length=0
   -12> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: init(): pipe_map (zone=87edf136-8bbb-4624-93bf-a8ef895d3872 bucket=*): adding potential pipe: {id=all,s={b=*,z=},d={b=*,z=}}
   -11> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=87edf136-8bbb-4624-93bf-a8ef895d3872,az=0},d={b=*,z=e33a5623-53b7-4fbc-b7d6-e5733ed433a2,az=0}}
   -10> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
    -9> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=-2 bl.length=0
    -8> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados->read ofs=0 len=0
    -7> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: rados_obj.operate() r=-2 bl.length=0
    -6> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: init(): pipe_map (zone=e33a5623-53b7-4fbc-b7d6-e5733ed433a2 bucket=*): adding potential pipe: {id=all,s={b=*,z=},d={b=*,z=}}
    -5> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: reflect(): flow manager (bucket=): adding source pipe: {s={b=*,z=87edf136-8bbb-4624-93bf-a8ef895d3872,az=0},d={b=*,z=e33a5623-53b7-4fbc-b7d6-e5733ed433a2,az=0}}
    -4> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: reflect(): flow manager (bucket=): adding dest pipe: {s={b=*,z=87edf136-8bbb-4624-93bf-a8ef895d3872,az=0},d={b=*,z=e33a5623-53b7-4fbc-b7d6-e5733ed433a2,az=0}}
    -3> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: started sync module instance, tier type = 
    -2> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: generating connection object for zone a1 id e33a5623-53b7-4fbc-b7d6-e5733ed433a2
    -1> 2022-09-13T07:39:11.500+0000 7f2bb52de700 20 rgw realm reloader: started zone id=87edf136-8bbb-4624-93bf-a8ef895d3872 (name=a2) with tier type = 
     0> 2022-09-13T07:39:11.500+0000 7f2bb02d4700 -1 *** Caught signal (Segmentation fault) **

Actions

Copy link