Actions
Bug #62000
openrgw crashed on latest ceph version 17.2.6 quincy
Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
07/13/2023
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
On latest ceph version 17.2.6 quincy (stable) I got persistent error with crash radosgw process on all runned rgw. I have two rgw but it crashed simultaneously with a minimum load on the servers, while the radosgw process constantly consumes ~100%. We run rgw on "CentOS Linux release 8.5.2111" and "AlmaLinux release 8.8 (Sapphire Caracal)" therefore, I do not associate this with the operation of servers or operating systems. Example logs:
-34> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 1 ====== starting new request req=0x7f6e8a39c710 =====
-33> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s initializing for trans_id = tx00000b105ba9e9a29d015-0064a3fd8d-5c7a0-eu-west-1
-32> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s getting op 0
-31> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s s3:get_obj verifying requester
-30> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj normalizing buckets and tenants
-29> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj init permissions
-28> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj recalculating target
-27> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj reading permissions
-26> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 0 req 12755806709950828565 0.002999922s s3:get_obj WARNING: couldn't find acl header for object, generating default
-25> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj init op
-24> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op mask
-23> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op permissions
-22> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0) mask=49
-21> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for uid=6016-5
-20> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Found permission: 15
-19> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for group=1 mask=49
-18> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Permissions for group not found
-17> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for group=2 mask=49
-16> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Permissions for group not found
-15> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0), owner=6016-5, perm=1
-14> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op params
-13> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj pre-executing
-12> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj check rate limiting
-11> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj executing
-10> 2023-07-04T11:07:57.236+0000 7f6eee4e6700 -1 *** Caught signal (Aborted) **
in thread 7f6eee4e6700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7f70c9e8fcf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f70c8e7b09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f70c8e8153c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7f70c8e80559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f70c885fb03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7f70cc373c5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7f70cc46b48a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f70c8eadb13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7f70c9e851ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-9> 2023-07-04T11:07:57.306+0000 7f706c7e2700 5 req 6774614862144446470 0.150996074s s3:put_obj NOTICE: call to do_aws4_auth_completion
-8> 2023-07-04T11:07:57.306+0000 7f706c7e2700 5 req 6774614862144446470 0.150996074s s3:put_obj NOTICE: call to do_aws4_auth_completion
-7> 2023-07-04T11:07:57.314+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard[120]: failed to take lease
-6> 2023-07-04T11:07:57.347+0000 7f6f795fc700 2 req 6774614862144446470 0.191995010s s3:put_obj completing
-5> 2023-07-04T11:07:57.348+0000 7f6f795fc700 2 req 6774614862144446470 0.192994997s s3:put_obj op status=0
-4> 2023-07-04T11:07:57.348+0000 7f6f795fc700 2 req 6774614862144446470 0.192994997s s3:put_obj http status=200
-3> 2023-07-04T11:07:57.348+0000 7f6f795fc700 1 ====== req done req=0x7f6e8a41d710 op status=0 http_status=200 latency=0.192994997s ======
-2> 2023-07-04T11:07:57.348+0000 7f6f795fc700 1 beast: 0x7f6e8a41d710: [IPv6 address] - 6016-5 [04/Jul/2023:11:07:57.155 +0000] "PUT /owncloud-prod/urn%3Aoid%3A2376416?partNumber=11&uploadId=2~sX-2sT0iBoilw73U4ziIIXNCeOPgniT HTTP/1.1" 200 5242880 - "aws-sdk-php/3.134.8 Guzzle/5.3.1 curl/7.29.0 PHP/7.4.24" - latency=0.192994997s
-1> 2023-07-04T11:07:57.392+0000 7f70a4a7a700 10 monclient: tick
0> 2023-07-04T11:07:57.765+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard[119]: failed to take lease
On a second:
-34> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 1 ====== starting new request req=0x7f6e8a39c710 =====
-33> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s initializing for trans_id = tx00000b105ba9e9a29d015-0064a3fd8d-5c7a0-eu-west-1
-32> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s getting op 0
-31> 2023-07-04T11:07:57.228+0000 7f6eee4e6700 2 req 12755806709950828565 0.000000000s s3:get_obj verifying requester
-30> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj normalizing buckets and tenants
-29> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj init permissions
-28> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj recalculating target
-27> 2023-07-04T11:07:57.229+0000 7f6eee4e6700 2 req 12755806709950828565 0.000999974s s3:get_obj reading permissions
-26> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 0 req 12755806709950828565 0.002999922s s3:get_obj WARNING: couldn't find acl header for object, generating default
-25> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj init op
-24> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op mask
-23> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op permissions
-22> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0) mask=49
-21> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for uid=6016-5
-20> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Found permission: 15
-19> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for group=1 mask=49
-18> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Permissions for group not found
-17> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Searching permissions for group=2 mask=49
-16> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj Permissions for group not found
-15> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 5 req 12755806709950828565 0.002999922s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0), owner=6016-5, perm=1
-14> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj verifying op params
-13> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj pre-executing
-12> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj check rate limiting
-11> 2023-07-04T11:07:57.231+0000 7f6eee4e6700 2 req 12755806709950828565 0.002999922s s3:get_obj executing
-10> 2023-07-04T11:07:57.236+0000 7f6eee4e6700 -1 *** Caught signal (Aborted) **
in thread 7f6eee4e6700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7f70c9e8fcf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f70c8e7b09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7f70c8e8153c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7f70c8e80559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f70c885fb03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7f70cc373c5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7f70cc46b48a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7f70c8eadb13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7f70c9e851ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-9> 2023-07-04T11:07:57.306+0000 7f706c7e2700 5 req 6774614862144446470 0.150996074s s3:put_obj NOTICE: call to do_aws4_auth_completion
-8> 2023-07-04T11:07:57.306+0000 7f706c7e2700 5 req 6774614862144446470 0.150996074s s3:put_obj NOTICE: call to do_aws4_auth_completion
-7> 2023-07-04T11:07:57.314+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard[120]: failed to take lease
-6> 2023-07-04T11:07:57.347+0000 7f6f795fc700 2 req 6774614862144446470 0.191995010s s3:put_obj completing
-5> 2023-07-04T11:07:57.348+0000 7f6f795fc700 2 req 6774614862144446470 0.192994997s s3:put_obj op status=0
-4> 2023-07-04T11:07:57.348+0000 7f6f795fc700 2 req 6774614862144446470 0.192994997s s3:put_obj http status=200
-3> 2023-07-04T11:07:57.348+0000 7f6f795fc700 1 ====== req done req=0x7f6e8a41d710 op status=0 http_status=200 latency=0.192994997s ======
-2> 2023-07-04T11:07:57.348+0000 7f6f795fc700 1 beast: 0x7f6e8a41d710: [IPv6 address] - 6016-5 [04/Jul/2023:11:07:57.155 +0000] "PUT /owncloud-prod/urn%3Aoid%3A2376416?partNumber=11&uploadId=2~sX-2sT0iBoilw73U4ziIIXNCeOPgniT HTTP/1.1" 200 5242880 - "aws-sdk-php/3.134.8 Guzzle/5.3.1 curl/7.29.0 PHP/7.4.24" - latency=0.192994997s
-1> 2023-07-04T11:07:57.392+0000 7f70a4a7a700 10 monclient: tick
0> 2023-07-04T11:07:57.765+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard[119]: failed to take lease
We are running the latest version 17.2.6 on all mds, mgr, mon, osd, rgw nodes. I tried changing the default rgw_thread_pool_size up and down - but that didn't work.
Files
Updated by Casey Bodley 10 months ago
0> 2023-07-04T11:07:57.765+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard119: failed to take lease
since this is the last thing logged, i assume the crash is coming from multisite sync. do you see the same line in all of these crashes?
Updated by Oleksii Yermak 10 months ago
Yes.
-134> 2023-07-13T00:40:39.080+0000 7fe61c0dc700 1 ====== req done req=0x7fe5f508d710 op status=0 http_status=200 latency=0.005999846s ======
-133> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Searching permissions for group=1 mask=49
-132> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Permissions for group not found
-131> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Searching permissions for group=2 mask=49
-130> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Permissions for group not found
-129> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0), owner=6016-5, perm=1
-128> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj verifying op params
-127> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj pre-executing
-126> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj check rate limiting
-125> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj executing
-124> 2023-07-13T00:40:39.080+0000 7fe61c0dc700 1 beast: 0x7fe5f508d710: 2a03:afc0:2:703:: - 6016-7 [13/Jul/2023:00:40:39.074 +0000] "HEAD /owncloud-data/urn%3Aoid%3A1410078 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.005999846s
-123> 2023-07-13T00:40:39.080+0000 7fe666971700 -1 *** Caught signal (Aborted) **
in thread 7fe666971700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe6b47ffcf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7fe6b37eb09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7fe6b37f153c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7fe6b37f0559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7fe6b31cfb03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7fe6b6ce3c5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7fe6b6ddb48a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7fe6b381db13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7fe6b47f51ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
On other:
-243> 2023-07-13T00:40:39.064+0000 7fdb9788d700 1 ====== req done req=0x7fdb1d717710 op status=0 http_status=200 latency=0.003000056s ======
-242> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj completing
-241> 2023-07-13T00:40:39.064+0000 7fdb9788d700 1 beast: 0x7fdb1d717710: 2a03:afc0:2:703:: - 6016-5 [13/Jul/2023:00:40:39.061 +0000] "HEAD /owncloud-prod/urn%3Aoid%3A1410068 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.003000056s
-240> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj op status=0
-239> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj http status=200
-238> 2023-07-13T00:40:39.064+0000 7fdb337c5700 1 ====== req done req=0x7fdb1d798710 op status=0 http_status=200 latency=0.003000056s ======
-237> 2023-07-13T00:40:39.064+0000 7fdb337c5700 1 beast: 0x7fdb1d798710: 2a03:afc0:2:703:: - 6016-7 [13/Jul/2023:00:40:39.061 +0000] "HEAD /owncloud-data/urn%3Aoid%3A141003 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.003000056s
-236> 2023-07-13T00:40:39.065+0000 7fdb5600a700 -1 *** Caught signal (Aborted) **
in thread 7fdb5600a700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdbdcf0acf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7fdbdbef609b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7fdbdbefc53c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7fdbdbefb559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7fdbdb8dab03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7fdbdf3eec5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7fdbdf4e648a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7fdbdbf28b13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7fdbdcf001ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Oleksii Yermak 10 months ago
Yes.
-134> 2023-07-13T00:40:39.080+0000 7fe61c0dc700 1 ====== req done req=0x7fe5f508d710 op status=0 http_status=200 latency=0.005999846s ======
-133> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Searching permissions for group=1 mask=49
-132> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Permissions for group not found
-131> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Searching permissions for group=2 mask=49
-130> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj Permissions for group not found
-129> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 5 req 15695308673855493371 0.003999897s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=6016-5, acct_name=owncloud-prod, subuser=, perm_mask=15, is_admin=0), owner=6016-5, perm=1
-128> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj verifying op params
-127> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj pre-executing
-126> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj check rate limiting
-125> 2023-07-13T00:40:39.080+0000 7fe5fe0a0700 2 req 15695308673855493371 0.003999897s s3:get_obj executing
-124> 2023-07-13T00:40:39.080+0000 7fe61c0dc700 1 beast: 0x7fe5f508d710: [IPv6 address] - 6016-7 [13/Jul/2023:00:40:39.074 +0000] "HEAD /owncloud-data/urn%3Aoid%3A1410078 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.005999846s
-123> 2023-07-13T00:40:39.080+0000 7fe666971700 -1 *** Caught signal (Aborted) **
in thread 7fe666971700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7fe6b47ffcf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7fe6b37eb09b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7fe6b37f153c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7fe6b37f0559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7fe6b31cfb03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7fe6b6ce3c5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7fe6b6ddb48a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7fe6b381db13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7fe6b47f51ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
On other
-243> 2023-07-13T00:40:39.064+0000 7fdb9788d700 1 ====== req done req=0x7fdb1d717710 op status=0 http_status=200 latency=0.003000056s ======
-242> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj completing
-241> 2023-07-13T00:40:39.064+0000 7fdb9788d700 1 beast: 0x7fdb1d717710: [IPv6 address] - 6016-5 [13/Jul/2023:00:40:39.061 +0000] "HEAD /owncloud-prod/urn%3Aoid%3A1410068 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.003000056s
-240> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj op status=0
-239> 2023-07-13T00:40:39.064+0000 7fdb337c5700 2 req 17126751749223441550 0.003000056s s3:get_obj http status=200
-238> 2023-07-13T00:40:39.064+0000 7fdb337c5700 1 ====== req done req=0x7fdb1d798710 op status=0 http_status=200 latency=0.003000056s ======
-237> 2023-07-13T00:40:39.064+0000 7fdb337c5700 1 beast: 0x7fdb1d798710: [IPv6 address] - 6016-7 [13/Jul/2023:00:40:39.061 +0000] "HEAD /owncloud-data/urn%3Aoid%3A141003 HTTP/1.1" 200 0 - "rclone/v1.57.0" - latency=0.003000056s
-236> 2023-07-13T00:40:39.065+0000 7fdb5600a700 -1 *** Caught signal (Aborted) **
in thread 7fdb5600a700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdbdcf0acf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7fdbdbef609b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7fdbdbefc53c]
6: /lib64/libstdc++.so.6(+0x95559) [0x7fdbdbefb559]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7fdbdb8dab03]
9: _Unwind_Resume()
10: /lib64/libradosgw.so.2(+0x538c5b) [0x7fdbdf3eec5b]
11: /lib64/libradosgw.so.2(+0x63048a) [0x7fdbdf4e648a]
12: /lib64/libstdc++.so.6(+0xc2b13) [0x7fdbdbf28b13]
13: /lib64/libpthread.so.0(+0x81ca) [0x7fdbdcf001ca]
14: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Oleksii Yermak 9 months ago
Casey Bodley wrote:
0> 2023-07-04T11:07:57.765+0000 7f709f26f700 5 RGW-SYNC:data:sync:shard119: failed to take lease
since this is the last thing logged, i assume the crash is coming from multisite sync. do you see the same line in all of these crashes?
We added yet one node on a server with large amount of memory (128G), but the situation has not changed. Attached files have full logs.
rgw.eu-west-1.osd-hdd-3.mkoupt osd-hdd-3 *:8000 running (2h) 3m ago 3w 535M - 17.2.6 0dc91bca92c2 ed24128c3aea
rgw.eu-west-1.osd-hdd-4.fjsmgl osd-hdd-4 *:8000 running (2h) 37s ago 21M 470M - 17.2.6 0dc91bca92c2 6bbbbcab6e9e
rgw.eu-west-1.osd-hdd-6.lgkvkz osd-hdd-6 *:8000 running (2h) 6m ago 11M 413M - 17.2.6 0dc91bca92c2 0ccc34c3c7ac
Updated by hoan nv about 1 month ago
I found some important logs.
It not contain object
"name": "test1.zip",
"size": 722930283,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "HN-01-000021-infra-s3-lDmLtd",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "HN-01-000021-infra-s3-lDmLtd",
"grant": {
"type": {
"type": 0
},
"id": "HN-01-000021-infra-s3-lDmLtd",
"email": "",
"permission": {
"flags": 15
},
"name": "HN-01-000021-infra-s3-lDmLtd",
"group": 0,
"url_spec": ""
}
}
]
},
"owner": {
"id": "HN-01-000021-infra-s3-lDmLtd",
"display_name": "HN-01-000021-infra-s3-lDmLtd"
}
},
"etag": "05d2c577273b679bc75f0979edeabf42-7",
"tag": "6d581034-c3e2-4185-b896-71f93654bf3a.69090321.5099803845479849043",
"manifest": {
"objs": [],
"obj_size": 722930283,
"explicit_objs": "false",
"head_size": 0,
"max_head_size": 0,
"prefix": "test1.zip.2~190lQfEOBHMWnkYPuX3qzJX-n-_1D7l",
"rules": [
{
"key": 0,
"val": {
"start_part_num": 1,
"start_ofs": 0,
"part_size": 104857600,
"stripe_max_size": 4194304,
"override_prefix": ""
}
},
{
"key": 629145600,
"val": {
"start_part_num": 7,
"start_ofs": 629145600,
"part_size": 93784683,
"stripe_max_size": 4194304,
............
"attrs": {
"user.rgw.content_type": "application/zip",
"user.rgw.pg_ver": "<D9>\u0001",
"user.rgw.source_zone": "M<F8>eA",
"user.rgw.tail_tag": "6d581034-c3e2-4185-b896-71f93654bf3a.69090321.5099803845479849043",
"user.rgw.x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"user.rgw.x-amz-date": "20240222T103030Z",
"user.rgw.x-amz-meta-mtime": "1708597748.178"
}
radosgw-admin object stat --bucket=from1 --object=index1.txt
{
"name": "index1.txt",
"size": 0,
"tag": "",
"attrs": {
"user.rgw.manifest": "",
"user.rgw.olh.idtag": "gaxtbv5q4yxeu42vl8fx00qvftq3on1v",
"user.rgw.olh.info": "\u0001\u0001",
"user.rgw.olh.ver": "4"
}
}
cd59c574d682b54a80dc5bcfa4f6770fa7faa7f67d643575220be7ec60878418
-90> 2024-03-27T08:53:14.092+0000 7f883ed4b700 15 req 2631107773312189319 0.003999961s s3:get_obj server signature=75584fd0fba5bcd21c95559a6e16cb7fb3e4a17829761e137bd2b6df02cf2a72
-89> 2024-03-27T08:53:14.092+0000 7f883ed4b700 15 req 2631107773312189319 0.003999961s s3:get_obj client signature=75584fd0fba5bcd21c95559a6e16cb7fb3e4a17829761e137bd2b6df02cf2a72
-88> 2024-03-27T08:53:14.092+0000 7f883ed4b700 15 req 2631107773312189319 0.003999961s s3:get_obj compare=0
-87> 2024-03-27T08:53:14.092+0000 7f883ed4b700 20 req 2631107773312189319 0.003999961s s3:get_obj rgw::auth::s3::LocalEngine granted access
-86> 2024-03-27T08:53:14.092+0000 7f883ed4b700 20 req 2631107773312189319 0.003999961s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access
-85> 2024-03-27T08:53:14.092+0000 7f883ed4b700 2 req 2631107773312189319 0.003999961s s3:get_obj normalizing buckets and tenants
-84> 2024-03-27T08:53:14.092+0000 7f883ed4b700 10 req 2631107773312189319 0.003999961s s->object=index1.txt s->bucket=from1_is_deleted_1711348231
-83> 2024-03-27T08:53:14.092+0000 7f883ed4b700 2 req 2631107773312189319 0.003999961s s3:get_obj init permissions
-82> 2024-03-27T08:53:14.092+0000 7f883ed4b700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: rctx=0x7f884f58b1a0 obj=stag01.rgw.meta:root:from1_is_deleted_1711348231 state=0x5623fad2d960 s->prefetch_data=0
-81> 2024-03-27T08:53:14.092+0000 7f883ed4b700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+root+from1_is_deleted_1711348231 : miss
-80> 2024-03-27T08:53:14.092+0000 7f883dd49700 10 req 2631107773312189319 0.003999961s s3:get_obj cache put: name=stag01.rgw.meta+root+from1_is_deleted_1711348231 info.flags=0x16
-79> 2024-03-27T08:53:14.092+0000 7f883dd49700 10 req 2631107773312189319 0.003999961s s3:get_obj adding stag01.rgw.meta+root+from1_is_deleted_1711348231 to cache LRU end
-78> 2024-03-27T08:53:14.092+0000 7f883dd49700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42
-77> 2024-03-27T08:53:14.092+0000 7f883dd49700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: s->obj_tag was set empty
-76> 2024-03-27T08:53:14.092+0000 7f883dd49700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+root+from1_is_deleted_1711348231 : type miss (requested=0x11, cached=0x16)
-75> 2024-03-27T08:53:14.092+0000 7f883dd49700 20 req 2631107773312189319 0.003999961s s3:get_obj rados->read ofs=0 len=0
-74> 2024-03-27T08:53:14.092+0000 7f883d548700 20 req 2631107773312189319 0.003999961s s3:get_obj rados_obj.operate() r=0 bl.length=251
-73> 2024-03-27T08:53:14.092+0000 7f883d548700 10 req 2631107773312189319 0.003999961s s3:get_obj cache put: name=stag01.rgw.meta+root+from1_is_deleted_1711348231 info.flags=0x11
-72> 2024-03-27T08:53:14.092+0000 7f883d548700 10 req 2631107773312189319 0.003999961s s3:get_obj moving stag01.rgw.meta+root+from1_is_deleted_1711348231 to cache LRU end
-71> 2024-03-27T08:53:14.092+0000 7f883d548700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: rctx=0x7f884f58b1a0 obj=stag01.rgw.meta:root:.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 state=0x5623fad2d720 s->prefetch_data=0
-70> 2024-03-27T08:53:14.092+0000 7f883d548700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 : miss
-69> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj cache put: name=stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 info.flags=0x16
-68> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj adding stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 to cache LRU end
-67> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42
-66> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=user.rgw.acl bl.length()=291
-65> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=user.rgw.cors bl.length()=91
-64> 2024-03-27T08:53:14.092+0000 7f883cd47700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: s->obj_tag was set empty
-63> 2024-03-27T08:53:14.092+0000 7f883cd47700 20 req 2631107773312189319 0.003999961s s3:get_obj Read xattr: user.rgw.acl
-62> 2024-03-27T08:53:14.092+0000 7f883cd47700 20 req 2631107773312189319 0.003999961s s3:get_obj Read xattr: user.rgw.cors
-61> 2024-03-27T08:53:14.092+0000 7f883cd47700 20 req 2631107773312189319 0.003999961s s3:get_obj Read xattr: user.rgw.idtag
-60> 2024-03-27T08:53:14.092+0000 7f883cd47700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 : type miss (requested=0x13, cached=0x16)
-59> 2024-03-27T08:53:14.092+0000 7f883cd47700 20 req 2631107773312189319 0.003999961s s3:get_obj rados->read ofs=0 len=0
-58> 2024-03-27T08:53:14.092+0000 7f883c546700 20 req 2631107773312189319 0.003999961s s3:get_obj rados_obj.operate() r=0 bl.length=410
-57> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj cache put: name=stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 info.flags=0x13
-56> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj moving stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1 to cache LRU end
-55> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42
-54> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=user.rgw.acl bl.length()=291
-53> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=user.rgw.cors bl.length()=91
-52> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj chain_cache_entry: cache_locator=stag01.rgw.meta+root+.bucket.meta.from1_is_deleted_1711348231:6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1
-51> 2024-03-27T08:53:14.092+0000 7f883c546700 15 req 2631107773312189319 0.003999961s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>user-periodically-delete-bucket</ID><DisplayName>user periodically delete bucket</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>user-periodically-de
lete-bucket</ID><DisplayName>user periodically delete bucket</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
-50> 2024-03-27T08:53:14.092+0000 7f883c546700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: rctx=0x7f884f58acc8 obj=stag01.rgw.meta:users.uid:user-periodically-delete-bucket state=0x5623fad2d720 s->prefetch_data=0
-49> 2024-03-27T08:53:14.092+0000 7f883c546700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+users.uid+user-periodically-delete-bucket : type miss (requested=0x16, cached=0x13)
-48> 2024-03-27T08:53:14.092+0000 7f883b544700 10 req 2631107773312189319 0.003999961s s3:get_obj cache put: name=stag01.rgw.meta+users.uid+user-periodically-delete-bucket info.flags=0x16
-47> 2024-03-27T08:53:14.092+0000 7f883b544700 10 req 2631107773312189319 0.003999961s s3:get_obj moving stag01.rgw.meta+users.uid+user-periodically-delete-bucket to cache LRU end
-46> 2024-03-27T08:53:14.092+0000 7f883b544700 10 req 2631107773312189319 0.003999961s s3:get_obj updating xattr: name=ceph.objclass.version bl.length()=42
-45> 2024-03-27T08:53:14.092+0000 7f883b544700 20 req 2631107773312189319 0.003999961s s3:get_obj get_system_obj_state: s->obj_tag was set empty
-44> 2024-03-27T08:53:14.092+0000 7f883b544700 20 req 2631107773312189319 0.003999961s s3:get_obj Read xattr: user.rgw.idtag
-43> 2024-03-27T08:53:14.092+0000 7f883b544700 10 req 2631107773312189319 0.003999961s s3:get_obj cache get: name=stag01.rgw.meta+users.uid+user-periodically-delete-bucket : hit (requested=0x13, cached=0x17)
-42> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj recalculating target
-41> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj reading permissions
-40> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: rctx=0x7f884f58baa0 obj=from1_is_deleted_1711348231:index1.txt state=0x5623f6058de8 s->prefetch_data=1
-39> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj WARNING: blocking librados call
-38> 2024-03-27T08:53:14.096+0000 7f883b544700 10 req 2631107773312189319 0.007999922s s3:get_obj manifest: total_size = 12
-37> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: setting s->obj_tag to 6d581034-c3e2-4185-b896-71f93654bf3a.62975741.14247299702682078524
-36> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state_impl: setting s->olh_tag to gaxtbv5q4yxeu42vl8fx00qvftq3on1v
-35> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: rctx=0x7f884f58baa0 obj=from1:index.txt state=0x5623f5ca0de8 s->prefetch_data=0
-34> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj WARNING: blocking librados call
-33> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: s->obj_tag was set empty
-32> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state_impl: setting s->olh_tag to gaxtbv5q4yxeu42vl8fx00qvftq3on1v
-31> 2024-03-27T08:53:14.096+0000 7f883b544700 0 req 2631107773312189319 0.007999922s s3:get_obj WARNING: couldn't find acl header for object, generating default
-30> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_system_obj_state: rctx=0x7f884f58aeb8 obj=stag01.rgw.meta:users.uid:user-periodically-delete-bucket state=0x5623fad2d720 s->prefetch_data=0
-29> 2024-03-27T08:53:14.096+0000 7f883b544700 10 req 2631107773312189319 0.007999922s s3:get_obj cache get: name=stag01.rgw.meta+users.uid+user-periodically-delete-bucket : hit (requested=0x16, cached=0x17)
-28> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_system_obj_state: s->obj_tag was set empty
-27> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr: user.rgw.idtag
-26> 2024-03-27T08:53:14.096+0000 7f883b544700 10 req 2631107773312189319 0.007999922s s3:get_obj cache get: name=stag01.rgw.meta+users.uid+user-periodically-delete-bucket : hit (requested=0x13, cached=0x17)
-25> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj init op
-24> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj verifying op mask
-23> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj required_mask= 1 user.op_mask=7
-22> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj verifying op permissions
-21> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj -- Getting permissions begin with perm_mask=49
-20> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=user-periodically-delete-bucket, acct_name=user periodically delete bucket, subuser=, perm_mask=15, is_admin=0) mask=49
-19> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Searching permissions for uid=user-periodically-delete-bucket
-18> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Found permission: 15
-17> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Searching permissions for group=1 mask=49
-16> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Permissions for group not found
-15> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Searching permissions for group=2 mask=49
-14> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj Permissions for group not found
-13> 2024-03-27T08:53:14.096+0000 7f883b544700 5 req 2631107773312189319 0.007999922s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=user-periodically-delete-bucket, acct_name=user periodically delete bucket, subuser=, perm_mask=15, is_admin=0), owner=user-periodically-delete-bucket, perm=1
-12> 2024-03-27T08:53:14.096+0000 7f883b544700 10 req 2631107773312189319 0.007999922s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=user-periodically-delete-bucket, acct_name=user periodically delete bucket, subuser=, perm_mask=15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
-11> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj verifying op params
-10> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj pre-executing
-9> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj check rate limiting
-8> 2024-03-27T08:53:14.096+0000 7f883b544700 2 req 2631107773312189319 0.007999922s s3:get_obj executing
-7> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: rctx=0x7f884f58baa0 obj=from1_is_deleted_1711348231:index1.txt state=0x5623f6058de8 s->prefetch_data=1
-6> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj get_obj_state: rctx=0x7f884f58baa0 obj=from1:index.txt state=0x5623f5ca0de8 s->prefetch_data=0
-5> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr rgw_rados: user.rgw.idtag
-4> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr rgw_rados: user.rgw.manifest
-3> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr rgw_rados: user.rgw.olh.idtag
-2> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr rgw_rados: user.rgw.olh.info
-1> 2024-03-27T08:53:14.096+0000 7f883b544700 20 req 2631107773312189319 0.007999922s s3:get_obj Read xattr rgw_rados: user.rgw.olh.ver
0> 2024-03-27T08:53:14.100+0000 7f883b544700 -1 *** Caught signal (Aborted) **
in thread 7f883b544700 thread_name:radosgw
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7f887d1b5cf0]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7f887c1a109b]
5: /lib64/libstdc++.so.6(+0x9654c) [0x7f887c1a754c]
6: /lib64/libstdc++.so.6(+0x95569) [0x7f887c1a6569]
7: __gxx_personality_v0()
8: /lib64/libgcc_s.so.1(+0x10b03) [0x7f887bb85b03]
9: _Unwind_Resume()
Log to procedure this bug
2024-01-29T04:16:39.882+0000 7fbccaf7d700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:39.878 +0000] "HEAD /from1 HTTP/1.1" 404 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.003999947s 2024-01-29T04:16:40.590+0000 7fbcc977a700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:40.590 +0000] "HEAD /from1 HTTP/1.1" 404 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:16:40.626+0000 7fbcc7f77700 1 beast: 0x7fbbe67a8710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:40.606 +0000] "PUT /from1 HTTP/1.1" 200 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.019999733s 2024-01-29T04:16:40.662+0000 7fbcc5772700 1 beast: 0x7fbbe65a4710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:40.654 +0000] "PUT /from1?acl HTTP/1.1" 200 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.007999894s 2024-01-29T04:16:49.114+0000 7fbc6b6be700 1 beast: 0x7fbbe6db4710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:49.114 +0000] "HEAD /to1 HTTP/1.1" 404 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:16:49.826+0000 7fbc4ee85700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:49.826 +0000] "HEAD /to1 HTTP/1.1" 404 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:16:49.862+0000 7fbc66eb5700 1 beast: 0x7fbbe66a6710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:49.842 +0000] "PUT /to1 HTTP/1.1" 200 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.019999737s 2024-01-29T04:16:49.898+0000 7fbc5969a700 1 beast: 0x7fbbe66a6710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:16:49.890 +0000] "PUT /to1?acl HTTP/1.1" 200 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.007999895s 2024-01-29T04:17:15.238+0000 7fbcd178a700 1 beast: 0x7fbbe68aa710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:17:15.238 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 340 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:17:15.358+0000 7fbbf55d2700 1 beast: 0x7fbbe68aa710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:17:15.354 +0000] "GET /from1/error.html.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.003999951s 2024-01-29T04:17:18.217+0000 7fbcc9f7b700 1 beast: 0x7fbbe6c31710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:17:18.213 +0000] "GET /from1?cors HTTP/1.1" 404 230 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.003999950s 2024-01-29T04:17:18.265+0000 7fbcbc760700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:17:18.241 +0000] "POST /from1/error.html.txt?uploads HTTP/1.1" 200 250 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.023999702s 2024-01-29T04:17:25.621+0000 7fbc4ae7d700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - user-sync [29/Jan/2024:04:17:25.621 +0000] "GET /admin/log/?type=bucket-index&bucket-instance=from1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1%3A6&info&rgwx-zonegroup=ab7d724b-c43a-47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 2 00 97 - - - latency=0.000000000s 2024-01-29T04:17:25.669+0000 7fbc4e684700 1 beast: 0x7fbbe6a2d710: 172.19.33.202 - user-sync [29/Jan/2024:04:17:25.669 +0000] "GET /admin/log?bucket-instance=from1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1%3A6&format=json&marker=00000000001.657268.6&type=bucket-index&rgwx-zonegroup=ab7d724b-c43a- 47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 2 - - - latency=0.000000000s 2024-01-29T04:18:11.237+0000 7fbc7dee3700 1 beast: 0x7fbbe67a8710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:11.237 +0000] "GET /from1/index.html.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:12.585+0000 7fbc88ef9700 1 beast: 0x7fbbe67a8710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:12.585 +0000] "GET /from1?cors HTTP/1.1" 200 379 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:13.417+0000 7fbc6f6c6700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:13.405 +0000] "PUT /from1/index.html.txt?uploadId=2~JEcemMk3ORmr0HnjDogOWqY-5MekJn7&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=FIFV71XB3EOWXWRRZYVD%2F2024 0129%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240129T041813Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=a670d3f96de8fcdb31b72455117a68f3764f53a894817159b2cac2b642727565 HTTP/1.1" 200 12 "https://console-stg-iaas.fptcloud.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537 .36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" - latency=0.011999859s 2024-01-29T04:18:14.649+0000 7fbc61eab700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:14.649 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 569 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:23.837+0000 7fbc62ead700 1 beast: 0x7fbbe6db4710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:23.837 +0000] "GET /from1/index.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:23.981+0000 7fbc4ee85700 1 beast: 0x7fbbe6db4710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:23.981 +0000] "GET /from1/index.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:24.665+0000 7fbc5368e700 1 beast: 0x7fbbe69ac710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:24.657 +0000] "PUT /from1/index.txt HTTP/1.1" 200 240 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.007999869s 2024-01-29T04:18:24.685+0000 7fbc4a67c700 1 beast: 0x7fbbe69ac710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:24.673 +0000] "DELETE /from1/index.html.txt HTTP/1.1" 204 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.011999802s 2024-01-29T04:18:25.397+0000 7fbc15e13700 1 beast: 0x7fbbe6db4710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:25.393 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 564 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.003999934s 2024-01-29T04:18:25.837+0000 7fbc8c700700 1 beast: 0x7fbbe6c31710: 172.19.33.202 - user-sync [29/Jan/2024:04:18:25.837 +0000] "GET /from1?rgwx-bucket-instance=from1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1%3A5&versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=a b7d724b-c43a-47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 880 - - - latency=0.000000000s 2024-01-29T04:18:25.885+0000 7fbcab73e700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - user-sync [29/Jan/2024:04:18:25.881 +0000] "GET /admin/log?bucket-instance=from1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1%3A5&format=json&marker=00000000008.1438550.7&type=bucket-index&rgwx-zonegroup=ab7d724b-c43a -47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 2 - - - latency=0.003999933s 2024-01-29T04:18:31.021+0000 7fbcbbf5f700 1 beast: 0x7fbbe67a8710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:31.017 +0000] "GET /to1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 338 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-g eneric Botocore/1.21.65 Resource" - latency=0.003999933s 2024-01-29T04:18:35.256+0000 7fbca9f3b700 1 beast: 0x7fbbe6cb2710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:35.240 +0000] "DELETE /from1/index.txt HTTP/1.1" 204 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.015999738s 2024-01-29T04:18:35.948+0000 7fbc4ee85700 1 beast: 0x7fbbe6cb2710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:35.944 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 340 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.003999935s 2024-01-29T04:18:45.624+0000 7fbc36e55700 1 beast: 0x7fbbe6b2f710: 172.19.33.202 - user-sync [29/Jan/2024:04:18:45.624 +0000] "GET /admin/log/?type=bucket-index&bucket-instance=to1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.2%3A5&info&rgwx-zonegroup=ab7d724b-c43a-47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 97 - - - latency=0.000000000s 2024-01-29T04:18:45.668+0000 7fbbfb5de700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - user-sync [29/Jan/2024:04:18:45.668 +0000] "GET /to1/index.txt?rgwx-zonegroup=ab7d724b-c43a-47cb-8db3-bcc8fc9a5c71&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1" 304 0 - - - latency=0.000000000s 2024-01-29T04:18:45.860+0000 7fbce87b8700 1 beast: 0x7fbbe6b2f710: 172.19.33.202 - user-sync [29/Jan/2024:04:18:45.860 +0000] "GET /admin/log?bucket-instance=from1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.1%3A5&format=json&marker=00000000010.1438554.7&type=bucket-index&rgwx-zonegroup=ab7d724b-c43a -47cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 2 - - - latency=0.000000000s 2024-01-29T04:18:48.028+0000 7fbc0bdff700 1 beast: 0x7fbbe6aae710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:48.028 +0000] "GET /to1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 562 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-g eneric Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:51.328+0000 7fbbf45d0700 1 beast: 0x7fbbe6aae710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:51.324 +0000] "GET /to1/index.txt HTTP/1.1" 200 12 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:53.928+0000 7fbcc1f6b700 1 beast: 0x7fbbe6aae710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:53.928 +0000] "GET /from1/index.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:18:55.904+0000 7fbcbc760700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:55.884 +0000] "DELETE /to1/index.txt HTTP/1.1" 204 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.019999675s 2024-01-29T04:18:57.164+0000 7fbcd7796700 1 beast: 0x7fbbe6d33710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:18:57.160 +0000] "GET /to1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 338 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-g eneric Botocore/1.21.65 Resource" - latency=0.003999935s 2024-01-29T04:19:01.580+0000 7fbca372e700 1 beast: 0x7fbbe6cb2710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:01.576 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 564 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.003999935s 2024-01-29T04:19:05.696+0000 7fbc27e37700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - user-sync [29/Jan/2024:04:19:05.692 +0000] "GET /admin/log?bucket-instance=to1%3A6d581034-c3e2-4185-b896-71f93654bf3a.62984511.2%3A5&format=json&marker=00000000003.818535.7&type=bucket-index&rgwx-zonegroup=ab7d724b-c43a-47 cb-8db3-bcc8fc9a5c71 HTTP/1.1" 200 2 - - - latency=0.003999936s 2024-01-29T04:19:11.416+0000 7fbc50e89700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:11.416 +0000] "GET /from1/index1.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:19:11.560+0000 7fbc4ee85700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:11.556 +0000] "GET /from1/index1.txt HTTP/1.1" 404 216 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.003999936s 2024-01-29T04:19:12.248+0000 7fbc5969a700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:12.244 +0000] "GET /from1/index.txt?acl HTTP/1.1" 200 517 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.003999936s 2024-01-29T04:19:12.308+0000 7fbc9f726700 1 beast: 0x7fbbe6c31710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:12.292 +0000] "DELETE /from1/index.txt HTTP/1.1" 204 0 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167-generic Botocore/1.21.65 Resource" - latency=0.015999744s 2024-01-29T04:19:13.632+0000 7fbc36654700 1 beast: 0x7fbbe6bb0710: 172.19.33.202 - HN-01-000021-infra-s3-lDmLtd [29/Jan/2024:04:19:13.632 +0000] "GET /from1?list-type=2&delimiter=%2F&max-keys=25&continuation-token=&prefix=&encoding-type=url HTTP/1.1" 200 565 - "Boto3/1.18.39 Python/3.9.18 Linux/5.4.0-167 -generic Botocore/1.21.65 Resource" - latency=0.000000000s 2024-01-29T04:19:20.344+0000 7fbc1e624700 0 req 1985322065026226698 0.003999936s s3:get_obj WARNING: couldn't find acl header for object, generating default 9: _Unwind_Resume()
Actions