Project

General

Profile

Bug #44240

rgw: segmentation fault RGWBulkUploadOp_ObjStore_SWIFT::send_response calling RGWFormatter_Plain

Added by Eric Ivancich 4 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Found during teuthology rgw suite. Link: http://qa-proxy.ceph.com/teuthology/ivancich-2020-02-20_21:35:10-rgw-wip-listing-w-namespaces-distro-basic-smithi/4784951/remote/smithi045/crash/posted/2020-02-20T21%3A55%3A33.035055Z_ca7dc832-ba37-4e57-a0ad-5e2395e7522f/

0> 2020-02-20T21:55:33.029+0000 7f57ebe97700 -1 ** Caught signal (Segmentation fault) *
in thread 7f57ebe97700 thread_name:radosgw
ceph version 15.1.0-908-ge3780fa (e3780fa87839ba0b157725ed36103e410d324d43) octopus (rc)
1: (()+0x3ef20) [0x7f59a5d2df20]
2: (()+0x18e5a1) [0x7f59a5e7d5a1]
3: (IO_vfprintf()+0x2143) [0x7f59a5d4c4d3]
4: (
_vsnprintf_chk()+0xa9) [0x7f59a5e21169]
5: (RGWFormatter_Plain::write_data(char const*, ...)+0x102) [0x7f59a6618492]
6: (RGWFormatter_Plain::dump_value_int(std::basic_string_view<char, std::char_traits<char> >, char const*, ...)+0x192) [0x7f59a6618952]
7: (RGWBulkUploadOp_ObjStore_SWIFT::send_response()+0x163) [0x7f59a64d4e43]
8: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x14e2) [0x7f59a64a44a2]
9: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x336c) [0x7f59a64a85cc]
10: (()+0x347e1d) [0x7f59a6427e1d]
11: (()+0x348e14) [0x7f59a6428e14]
12: (()+0x348f48) [0x7f59a6428f48]
13: (make_fcontext()+0x2f) [0x7f59a6a53bcf]
182> 2020-02-20T21:55:32.665+0000 7f58066cc700  1 ====== starting new request req=0x7f59902c2870 =====
-181> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s initializing for trans_id = tx00000000000000000002e-005e4f0054-1087-default
-180> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s getting op 4
-179> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata verifying requester
-178> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata normalizing buckets and tenants
-177> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata init permissions
-176> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata recalculating target
-175> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata reading permissions
-174> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata init op
-173> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata verifying op mask
-172> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata verifying op permissions
-171> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 req 46 0s swift:put_account_metadata Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=b2ebbe5bfb8449a79f8bf9ce9990ee2c, acct_name=tempest-AccountQuotasTest-1146143091, perm_mask=15, is_admin=1) mask=50
-170> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Searching permissions for uid=b2ebbe5bfb8449a79f8bf9ce9990ee2c
-169> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Permissions for user not found
-168> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Searching permissions for uid=b2ebbe5bfb8449a79f8bf9ce9990ee2c$b2ebbe5bfb8449a79f8bf9ce9990ee2c
-167> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Permissions for user not found
-166> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Searching permissions for group=1 mask=50
-165> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Permissions for group not found
-164> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Searching permissions for group=2 mask=50
-163> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 Permissions for group not found
-162> 2020-02-20T21:55:32.665+0000 7f58066cc700 5 req 46 0s swift:put_account_metadata -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=b2ebbe5bfb8449a79f8bf9ce9990ee2c, acct_name=tempest-AccountQuotasTest-1146143091, perm_mask=15, is_admin=1), owner=d9c74da931e74336966620f17ccb8709, perm=0
-161> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 overriding permissions due to admin operation
-160> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata verifying op params
-159> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata pre-executing
-158> 2020-02-20T21:55:32.665+0000 7f58066cc700 2 req 46 0s swift:put_account_metadata executing
-157> 2020-02-20T21:55:32.669+0000 7f57ffebf700 2 req 46 0.004000000s swift:put_account_metadata completing
-156> 2020-02-20T21:55:32.669+0000 7f57ffebf700 5 nothing to log for operation
-155> 2020-02-20T21:55:32.669+0000 7f57ffebf700 2 req 46 0.004000000s swift:put_account_metadata op status=1902
-154> 2020-02-20T21:55:32.669+0000 7f57ffebf700 2 req 46 0.004000000s swift:put_account_metadata http status=204
-153> 2020-02-20T21:55:32.669+0000 7f57ffebf700 1 ====== req done req=0x7f59902c2870 op status=1902 http_status=204 latency=0.004000000s ======
-152> 2020-02-20T21:55:32.669+0000 7f5802ec5700 1 ====== starting new request req=0x7f59902c2870 =====
-151> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s initializing for trans_id = tx00000000000000000002f-005e4f0054-1087-default
-150> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s getting op 1
-149> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj verifying requester
-148> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj normalizing buckets and tenants
-147> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj init permissions
-146> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj recalculating target
-145> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj reading permissions
-144> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj init op
-143> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj verifying op mask
-142> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj verifying op permissions
-141> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 req 47 0s swift:put_obj Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0) mask=50
-140> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709
-139> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Found permission: 15
-138> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709$d9c74da931e74336966620f17ccb8709
-137> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Permissions for user not found
-136> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Searching permissions for group=1 mask=50
-135> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Permissions for group not found
-134> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Searching permissions for group=2 mask=50
-133> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 Permissions for group not found
-132> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 req 47 0s swift:put_obj -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0), owner=d9c74da931e74336966620f17ccb8709, perm=2
-131> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj verifying op params
-130> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj pre-executing
-129> 2020-02-20T21:55:32.669+0000 7f5802ec5700 2 req 47 0s swift:put_obj executing
-128> 2020-02-20T21:55:32.669+0000 7f5802ec5700 5 req 47 0s swift:put_obj NOTICE: call to do_aws4_auth_completion
-127> 2020-02-20T21:55:32.673+0000 7f57fbeb7700 2 req 47 0.004000000s swift:put_obj completing
-126> 2020-02-20T21:55:32.673+0000 7f57fbeb7700 2 req 47 0.004000000s swift:put_obj op status=1900
-125> 2020-02-20T21:55:32.673+0000 7f57fbeb7700 2 req 47 0.004000000s swift:put_obj http status=201
-124> 2020-02-20T21:55:32.673+0000 7f57fbeb7700 1 ====== req done req=0x7f59902c2870 op status=1900 http_status=201 latency=0.004000000s ======
-123> 2020-02-20T21:55:32.673+0000 7f57fdebb700 1 ====== starting new request req=0x7f59902c2870 =====
-122> 2020-02-20T21:55:32.673+0000 7f57fdebb700 2 req 48 0s initializing for trans_id = tx000000000000000000030-005e4f0054-1087-default
-121> 2020-02-20T21:55:32.673+0000 7f57fdebb700 2 req 48 0s getting op 4
-120> 2020-02-20T21:55:32.673+0000 7f57fdebb700 2 req 48 0s swift:put_account_metadata verifying requester
-119> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata normalizing buckets and tenants
-118> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata init permissions
-117> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata recalculating target
-116> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata reading permissions
-115> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata init op
-114> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata verifying op mask
-113> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata verifying op permissions
-112> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 req 48 0.004000000s swift:put_account_metadata Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=b2ebbe5bfb8449a79f8bf9ce9990ee2c, acct_name=tempest-AccountQuotasTest-1146143091, perm_mask=15, is_admin=1) mask=50
-111> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Searching permissions for uid=b2ebbe5bfb8449a79f8bf9ce9990ee2c
-110> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Permissions for user not found
-109> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Searching permissions for uid=b2ebbe5bfb8449a79f8bf9ce9990ee2c$b2ebbe5bfb8449a79f8bf9ce9990ee2c
-108> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Permissions for user not found
-107> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Searching permissions for group=1 mask=50
-106> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Permissions for group not found
-105> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Searching permissions for group=2 mask=50
-104> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 Permissions for group not found
-103> 2020-02-20T21:55:32.677+0000 7f57fdebb700 5 req 48 0.004000000s swift:put_account_metadata -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=b2ebbe5bfb8449a79f8bf9ce9990ee2c, acct_name=tempest-AccountQuotasTest-1146143091, perm_mask=15, is_admin=1), owner=d9c74da931e74336966620f17ccb8709, perm=0
-102> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 overriding permissions due to admin operation
-101> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata verifying op params
-100> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata pre-executing
-99> 2020-02-20T21:55:32.677+0000 7f57fdebb700 2 req 48 0.004000000s swift:put_account_metadata executing
-98> 2020-02-20T21:55:32.677+0000 7f57f86b0700 2 req 48 0.004000000s swift:put_account_metadata completing
-97> 2020-02-20T21:55:32.677+0000 7f57f86b0700 5 nothing to log for operation
-96> 2020-02-20T21:55:32.677+0000 7f57f86b0700 2 req 48 0.004000000s swift:put_account_metadata op status=1902
-95> 2020-02-20T21:55:32.677+0000 7f57f86b0700 2 req 48 0.004000000s swift:put_account_metadata http status=204
-94> 2020-02-20T21:55:32.677+0000 7f57f86b0700 1 ====== req done req=0x7f59902c2870 op status=1902 http_status=204 latency=0.004000000s ======
-93> 2020-02-20T21:55:32.677+0000 7f57faeb5700 1 ====== starting new request req=0x7f59902c2870 =====
-92> 2020-02-20T21:55:32.677+0000 7f57faeb5700 2 req 49 0s initializing for trans_id = tx000000000000000000031-005e4f0054-1087-default
-91> 2020-02-20T21:55:32.677+0000 7f57faeb5700 2 req 49 0s getting op 0
-90> 2020-02-20T21:55:32.677+0000 7f57faeb5700 2 req 49 0s swift:list_bucket verifying requester
-89> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket normalizing buckets and tenants
-88> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket init permissions
-87> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket recalculating target
-86> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket reading permissions
-85> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket init op
-84> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket verifying op mask
-83> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket verifying op permissions
-82> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 req 49 0.003999999s swift:list_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0) mask=49
-81> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709
-80> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Found permission: 15
-79> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709$d9c74da931e74336966620f17ccb8709
-78> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Permissions for user not found
-77> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Searching permissions for group=1 mask=49
-76> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Permissions for group not found
-75> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Searching permissions for group=2 mask=49
-74> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 Permissions for group not found
-73> 2020-02-20T21:55:32.681+0000 7f57faeb5700 5 req 49 0.003999999s swift:list_bucket -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0), owner=d9c74da931e74336966620f17ccb8709, perm=1
-72> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket verifying op params
-71> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket pre-executing
-70> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket executing
-69> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket completing
-68> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket op status=0
-67> 2020-02-20T21:55:32.681+0000 7f57faeb5700 2 req 49 0.003999999s swift:list_bucket http status=200
-66> 2020-02-20T21:55:32.681+0000 7f57faeb5700 1 ====== req done req=0x7f59902c2870 op status=0 http_status=200 latency=0.003999999s ======
-65> 2020-02-20T21:55:32.681+0000 7f57f5eab700 1 ====== starting new request req=0x7f59902c2870 =====
-64> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s initializing for trans_id = tx000000000000000000032-005e4f0054-1087-default
-63> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s getting op 2
-62> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj verifying requester
-61> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj normalizing buckets and tenants
-60> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj init permissions
-59> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj recalculating target
-58> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj reading permissions
-57> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj init op
-56> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj verifying op mask
-55> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj verifying op permissions
-54> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 req 50 0s swift:delete_obj Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0) mask=50
-53> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709
-52> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Found permission: 15
-51> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Searching permissions for uid=d9c74da931e74336966620f17ccb8709$d9c74da931e74336966620f17ccb8709
-50> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Permissions for user not found
-49> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Searching permissions for group=1 mask=50
-48> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Permissions for group not found
-47> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Searching permissions for group=2 mask=50
-46> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 Permissions for group not found
-45> 2020-02-20T21:55:32.681+0000 7f57f5eab700 5 req 50 0s swift:delete_obj -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(d9c74da931e74336966620f17ccb8709) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=d9c74da931e74336966620f17ccb8709, acct_name=tempest-AccountQuotasTest-1107054290, perm_mask=15, is_admin=0), owner=d9c74da931e74336966620f17ccb8709, perm=2
-44> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj verifying op params
-43> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj pre-executing
-42> 2020-02-20T21:55:32.681+0000 7f57f5eab700 2 req 50 0s swift:delete_obj executing
-41> 2020-02-20T21:55:32.685+0000 7f57eee9d700 2 req 50 0.004000000s swift:delete_obj completing
-40> 2020-02-20T21:55:32.685+0000 7f57eee9d700 2 req 50 0.004000000s swift:delete_obj op status=0
-39> 2020-02-20T21:55:32.685+0000 7f57eee9d700 2 req 50 0.004000000s swift:delete_obj http status=204
-38> 2020-02-20T21:55:32.685+0000 7f57eee9d700 1 ====== req done req=0x7f59902c2870 op status=0 http_status=204 latency=0.004000000s ======
-37> 2020-02-20T21:55:33.013+0000 7f581aef5700 0 req 40 2.359999948s swift:bulk_upload validated token: tempest-BulkTest-111146302:tempest-BulkTest-111146302 expires: 1582239330
-36> 2020-02-20T21:55:33.017+0000 7f581aef5700 0 req 40 2.363999948s swift:bulk_upload NOTICE: couldn't map swift user f176a3367e1b4debb745477e665ef391
-35> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload normalizing buckets and tenants
-34> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload init permissions
-33> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload recalculating target
-32> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload reading permissions
-31> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload init op
-30> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload verifying op mask
-29> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload verifying op permissions
-28> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 req 40 2.363999948s swift:bulk_upload Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(f176a3367e1b4debb745477e665ef391) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=f176a3367e1b4debb745477e665ef391, acct_name=tempest-BulkTest-111146302, perm_mask=15, is_admin=0) mask=50
-27> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Searching permissions for uid=f176a3367e1b4debb745477e665ef391
-26> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Found permission: 15
-25> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Searching permissions for uid=f176a3367e1b4debb745477e665ef391$f176a3367e1b4debb745477e665ef391
-24> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Permissions for user not found
-23> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Searching permissions for group=1 mask=50
-22> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Permissions for group not found
-21> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Searching permissions for group=2 mask=50
-20> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 Permissions for group not found
-19> 2020-02-20T21:55:33.017+0000 7f581aef5700 5 req 40 2.363999948s swift:bulk_upload -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(f176a3367e1b4debb745477e665ef391) > rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=f176a3367e1b4debb745477e665ef391, acct_name=tempest-BulkTest-111146302, perm_mask=15, is_admin=0), owner=f176a3367e1b4debb745477e665ef391, perm=2
-18> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload verifying op params
-17> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload pre-executing
-16> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload executing
-15> 2020-02-20T21:55:33.017+0000 7f581aef5700 2 req 40 2.363999948s swift:bulk_upload handling regular directory
-14> 2020-02-20T21:55:33.025+0000 7f57ece99700 2 req 40 2.371999948s swift:bulk_upload handling regular file
-13> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 req 40 2.371999948s swift:bulk_upload Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier(f176a3367e1b4debb745477e665ef391) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=f176a3367e1b4debb745477e665ef391, acct_name=tempest-BulkTest-111146302, perm_mask=15, is_admin=0) mask=50
-12> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Searching permissions for uid=f176a3367e1b4debb745477e665ef391
-11> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Found permission: 15
-10> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Searching permissions for uid=f176a3367e1b4debb745477e665ef391$f176a3367e1b4debb745477e665ef391
-9> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Permissions for user not found
-8> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Searching permissions for group=1 mask=50
-7> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Permissions for group not found
-6> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Searching permissions for group=2 mask=50
-5> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 Permissions for group not found
-4> 2020-02-20T21:55:33.025+0000 7f57ece99700 5 req 40 2.371999948s swift:bulk_upload -
Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier(f176a3367e1b4debb745477e665ef391) -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=f176a3367e1b4debb745477e665ef391, acct_name=tempest-BulkTest-111146302, perm_mask=15, is_admin=0), owner=f176a3367e1b4debb745477e665ef391, perm=2
-3> 2020-02-20T21:55:33.029+0000 7f57ebe97700 2 req 40 2.375999948s swift:bulk_upload an empty block
-2> 2020-02-20T21:55:33.029+0000 7f57ebe97700 2 req 40 2.375999948s swift:bulk_upload an empty block
-1> 2020-02-20T21:55:33.029+0000 7f57ebe97700 2 req 40 2.375999948s swift:bulk_upload completing
0> 2020-02-20T21:55:33.029+0000 7f57ebe97700 -1 ** Caught signal (Segmentation fault) *
in thread 7f57ebe97700 thread_name:radosgw

History

#1 Updated by Matthew Oliver 4 months ago

I'll give swift bulk upload a whirl and see if I can reproduce locally.

#2 Updated by Nathan Cutler 3 months ago

Link to teuthology run Eric is referring to: http://pulpito.ceph.com/ivancich-2020-02-20_21:35:10-rgw-wip-listing-w-namespaces-distro-basic-smithi/

@Eric - do you know if this is reproducible on the octopus branch?

#3 Updated by Eric Ivancich 3 months ago

Nathan Cutler wrote:

Link to teuthology run Eric is referring to: http://pulpito.ceph.com/ivancich-2020-02-20_21:35:10-rgw-wip-listing-w-namespaces-distro-basic-smithi/

@Eric - do you know if this is reproducible on the octopus branch?

Sorry, I haven't tried, @Nathan.

#4 Updated by Matthew Oliver 3 months ago

OK, tracked down the branch that is in this issue. It's sitting on ceph-ci. When I pull it, build ceph, run vstart and try and bulk upload using the swift API I do indeed get the segfault. So finally managed to recreate.

I couldn't on master or on octopus, so is definitely related to this branch. Next to find the PR that created it :)

Also now that I've recreated it, will now delve deeper.

Also available in: Atom PDF