Project

General

Profile

Actions
Copy link

Bug #50139

closed

RadosGW can't start when upgrade to Pacific (16.2.0) from Octopus (15.2.10)

Added by Prince Inamatus about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have Ceph cluster that implement with Octopus (15.2.10).
When I upgrade with APT Install. The others services was working.
But RadosGW (16.2.0) can't start and error. Now I add New Instance that
served with RadosGW (15.2.10), It's work. I think RadosGW (16.2.0) must be
problem with upgrade process. When i ever upgrade from 15.2.0-15.2.10 in
Octopus Release this problem never come.

Files

gdb.txt (253 KB) gdb.txt Backtrace Prince Inamatus, 04/06/2021 02:05 PM

Related issues 1 (0 open — 1 closed)

Related to rgw - Bug #50218: Segmentation fault in radosgw_Main()ClosedMark Kogan

Actions
Actions
Copy link
 #1

Updated by Casey Bodley about 3 years ago

  • Status changed from New to Need More Info
  • Priority changed from Normal to High

can you share the error you saw? full debug logs (--debug-rgw 20 --debug-ms 1) from radosgw would be really helpful

Actions
Copy link
 #2

Updated by Prince Inamatus about 3 years ago

Casey Bodley wrote:

can you share the error you saw? full debug logs (--debug-rgw 20 --debug-ms 1) from radosgw would be really helpful

Here's my debug

epoch 23
fsid 41661ff3-5e9c-46b0-b73e-3d7bc65394d5
last_changed 2021-04-05T02:41:12.827224+0700
created 2019-05-27T10:13:46.265569+0700
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:10.11.165.111:3300/0,v1:10.11.165.111:6789/0] mon.ceph-1
1: [v2:10.11.165.113:3300/0,v1:10.11.165.113:6789/0] mon.ceph-3
2: [v2:10.11.165.112:3300/0,v1:10.11.165.112:6789/0] mon.ceph-2

243> 2021-04-05T22:18:59.155+0700 7f4841d492c0 10 monclient: _renew_subs
  -242> 2021-04-05T22:18:59.155+0700 7f4841d492c0 10 monclient: _send_mon_message to mon.ceph-1 at v2:10.11.165.111:3300/0
  -241> 2021-04-05T22:18:59.155+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:3300/0,v1:10.11.165.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561f777d1e0 con 0x5561f7619c00
  240> 2021-04-05T22:18:59.155+0700 7f4841d492c0  1 librados: init done
  -239> 2021-04-05T22:18:59.155+0700 7f4841d492c0  5 asok(0x5561f691c000) register_command cr dump hook 0x5561f6814dd8
  -238> 2021-04-05T22:18:59.159+0700 7f480f987700  1 - 10.11.165.111:0/856779288 <== mon.0 v2:10.11.165.111:3300/0 4 ==== mgrmap(e 9765984) v1 ==== 107669+0+0 (secure 0 0 0) 0x5561f7479600 con 0x5561f7619c00
  237> 2021-04-05T22:18:59.163+0700 7f480f987700  4 mgrc handle_mgr_map Got map version 9765984
  -236> 2021-04-05T22:18:59.163+0700 7f480f987700  4 mgrc handle_mgr_map Active mgr is now [v2:10.11.165.111:6810/2738794394,v1:10.11.165.111:6811/2738794394]
  -235> 2021-04-05T22:18:59.163+0700 7f480f987700  4 mgrc reconnect Starting new session with [v2:10.11.165.111:6810/2738794394,v1:10.11.165.111:6811/2738794394]
  -234> 2021-04-05T22:18:59.163+0700 7f480f987700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6810/2738794394,v1:10.11.165.111:6811/2738794394] conn(0x5561f78c6000 0x5561f78ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  233> 2021-04-05T22:18:59.163+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6810/2738794394,v1:10.11.165.111:6811/2738794394] conn(0x5561f78c6000 0x5561f78ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  232> 2021-04-05T22:18:59.163+0700 7f48299bb700 10 monclient: get_auth_request con 0x5561f78c6000 auth_method 0
  -231> 2021-04-05T22:18:59.163+0700 7f480f987700  1 - 10.11.165.111:0/856779288 <== mon.0 v2:10.11.165.111:3300/0 5 ==== osd_map(4991709..4991709 src has 4990961..4991709) v4 ==== 46358+0+0 (secure 0 0 0) 0x5561f6818540 con 0x5561f7619c00
  230> 2021-04-05T22:18:59.163+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6810/2738794394,v1:10.11.165.111:6811/2738794394] conn(0x5561f78c6000 0x5561f78ce000 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 rx=0x5561f7626300 tx=0x5561f753bea0).ready entity=mgr.10075455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  229> 2021-04-05T22:18:59.163+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] conn(0x5561f78c6c00 0x5561f7782800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  228> 2021-04-05T22:18:59.163+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] -- osd_op(unknown.0.0:1 8.1 8:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77401c0 con 0x5561f78c6c00
  227> 2021-04-05T22:18:59.163+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] conn(0x5561f78c6c00 0x5561f7782800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  226> 2021-04-05T22:18:59.163+0700 7f482a9bd700 10 monclient: get_auth_request con 0x5561f78c6c00 auth_method 0
  -225> 2021-04-05T22:18:59.167+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] conn(0x5561f78c6c00 0x5561f7782800 crc :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  224> 2021-04-05T22:18:59.167+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.1 v2:10.11.165.111:6804/2539437 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v4991709'11721298 uv11721278 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c6c00
  -223> 2021-04-05T22:18:59.167+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] conn(0x5561f78c7000 0x5561f7783700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  222> 2021-04-05T22:18:59.167+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:2 8.0 8:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773e000 con 0x5561f78c7000
  221> 2021-04-05T22:18:59.167+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] conn(0x5561f78c7000 0x5561f7783700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  220> 2021-04-05T22:18:59.167+0700 7f48299bb700 10 monclient: get_auth_request con 0x5561f78c7000 auth_method 0
  -219> 2021-04-05T22:18:59.171+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] conn(0x5561f78c7000 0x5561f7783700 crc :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  218> 2021-04-05T22:18:59.171+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v4991709'10025112 uv10025090 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
  -217> 2021-04-05T22:18:59.171+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] conn(0x5561f78c7400 0x5561f78d0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  216> 2021-04-05T22:18:59.171+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:3 8.3 8:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773f680 con 0x5561f78c7400
  215> 2021-04-05T22:18:59.171+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] conn(0x5561f78c7400 0x5561f78d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  214> 2021-04-05T22:18:59.171+0700 7f482a9bd700 10 monclient: get_auth_request con 0x5561f78c7400 auth_method 0
  -213> 2021-04-05T22:18:59.175+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] conn(0x5561f78c7400 0x5561f78d0800 crc :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.49 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  212> 2021-04-05T22:18:59.179+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v4991709'7984838 uv7984607 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -211> 2021-04-05T22:18:59.179+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] conn(0x5561f78c7800 0x5561f78d1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  210> 2021-04-05T22:18:59.179+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:4 8.4 8:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7740940 con 0x5561f78c7800
  209> 2021-04-05T22:18:59.179+0700 7f482a1bc700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] conn(0x5561f78c7800 0x5561f78d1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  208> 2021-04-05T22:18:59.179+0700 7f482a1bc700 10 monclient: get_auth_request con 0x5561f78c7800 auth_method 0
  -207> 2021-04-05T22:18:59.179+0700 7f482a1bc700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] conn(0x5561f78c7800 0x5561f78d1700 crc :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.46 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  206> 2021-04-05T22:18:59.183+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 1 ==== osd_op_reply(4 gc.3 [create,call,call,call] v4991709'12811729 uv12811411 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
  -205> 2021-04-05T22:18:59.183+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:5 8.3 8:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773e3c0 con 0x5561f78c7400
  204> 2021-04-05T22:18:59.211+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 2 ==== osd_op_reply(5 gc.4 [create,call,call,call] v4991709'7984839 uv7984609 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -203> 2021-04-05T22:18:59.211+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] -- osd_op(unknown.0.0:6 8.1 8:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773fe00 con 0x5561f78c6c00
  202> 2021-04-05T22:18:59.215+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.1 v2:10.11.165.111:6804/2539437 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v4991709'11721299 uv11720982 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c6c00
  -201> 2021-04-05T22:18:59.215+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] conn(0x5561f78c7c00 0x5561f7612a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  200> 2021-04-05T22:18:59.215+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] -- osd_op(unknown.0.0:7 8.6 8:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773e780 con 0x5561f78c7c00
  199> 2021-04-05T22:18:59.215+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] conn(0x5561f78c7c00 0x5561f7612a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  198> 2021-04-05T22:18:59.215+0700 7f48299bb700 10 monclient: get_auth_request con 0x5561f78c7c00 auth_method 0
  -197> 2021-04-05T22:18:59.215+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] conn(0x5561f78c7c00 0x5561f7612a00 crc :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.26 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  196> 2021-04-05T22:18:59.219+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.26 v2:10.11.165.113:6808/2273872 1 ==== osd_op_reply(7 gc.6 [create,call,call,call] v4991709'10332584 uv10332277 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7c00
  -195> 2021-04-05T22:18:59.219+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] conn(0x5561f78c8000 0x5561f7624300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  194> 2021-04-05T22:18:59.219+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:8 8.5 8:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77410c0 con 0x5561f78c8000
  193> 2021-04-05T22:18:59.219+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] conn(0x5561f78c8000 0x5561f7624300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  192> 2021-04-05T22:18:59.219+0700 7f482a9bd700 10 monclient: get_auth_request con 0x5561f78c8000 auth_method 0
  -191> 2021-04-05T22:18:59.219+0700 7f482a9bd700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] conn(0x5561f78c8000 0x5561f7624300 crc :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.28 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  190> 2021-04-05T22:18:59.223+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 1 ==== osd_op_reply(8 gc.7 [create,call,call,call] v4991709'11109701 uv11109420 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
  -189> 2021-04-05T22:18:59.223+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] conn(0x5561f78c8400 0x5561f7702f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  188> 2021-04-05T22:18:59.223+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] -- osd_op(unknown.0.0:9 8.2 8:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773ef00 con 0x5561f78c8400
  187> 2021-04-05T22:18:59.223+0700 7f482a1bc700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] conn(0x5561f78c8400 0x5561f7702f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  186> 2021-04-05T22:18:59.223+0700 7f482a1bc700 10 monclient: get_auth_request con 0x5561f78c8400 auth_method 0
  -185> 2021-04-05T22:18:59.223+0700 7f482a1bc700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] conn(0x5561f78c8400 0x5561f7702f00 crc :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.40 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  184> 2021-04-05T22:18:59.227+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.40 v2:10.11.165.114:6807/2251489 1 ==== osd_op_reply(9 gc.8 [create,call,call,call] v4991709'11008730 uv11008418 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c8400
  -183> 2021-04-05T22:18:59.227+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:10 8.4 8:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7741c00 con 0x5561f78c7800
  182> 2021-04-05T22:18:59.231+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 2 ==== osd_op_reply(10 gc.9 [create,call,call,call] v4991709'12811730 uv12811413 ondisk = 125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
  -181> 2021-04-05T22:18:59.231+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:11 8.3 8:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776dc00 con 0x5561f78c7400
  180> 2021-04-05T22:18:59.231+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v4991709'7984840 uv7984759 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -179> 2021-04-05T22:18:59.231+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:12 8.3 8:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776ba40 con 0x5561f78c7400
  178> 2021-04-05T22:18:59.235+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 4 ==== osd_op_reply(12 gc.11 [create,call,call,call] v4991709'7984841 uv7984761 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -177> 2021-04-05T22:18:59.235+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:13 8.5 8:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776cd00 con 0x5561f78c8000
  176> 2021-04-05T22:18:59.239+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 2 ==== osd_op_reply(13 gc.12 [create,call,call,call] v4991709'11109702 uv11109627 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
  -175> 2021-04-05T22:18:59.239+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:14 8.0 8:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776ab40 con 0x5561f78c7000
  174> 2021-04-05T22:18:59.243+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 2 ==== osd_op_reply(14 gc.13 [create,call,call,call] v4991709'10025113 uv10024999 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
  -173> 2021-04-05T22:18:59.243+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:15 8.4 8:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776be00 con 0x5561f78c7800
  172> 2021-04-05T22:18:59.243+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 3 ==== osd_op_reply(15 gc.14 [create,call,call,call] v4991709'12811731 uv12811693 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
  -171> 2021-04-05T22:18:59.243+0700 7f4841d492c0  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] conn(0x5561f78c8800 0x5561f7723400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
  170> 2021-04-05T22:18:59.243+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:16 8.7 8:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776a3c0 con 0x5561f78c8800
  169> 2021-04-05T22:18:59.243+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] conn(0x5561f78c8800 0x5561f7723400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
  168> 2021-04-05T22:18:59.243+0700 7f48299bb700 10 monclient: get_auth_request con 0x5561f78c8800 auth_method 0
  -167> 2021-04-05T22:18:59.247+0700 7f48299bb700  1 --2 10.11.165.111:0/856779288 >> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] conn(0x5561f78c8800 0x5561f7723400 crc :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
  166> 2021-04-05T22:18:59.247+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 1 ==== osd_op_reply(16 gc.15 [create,call,call,call] v4991709'8993443 uv8993366 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
  -165> 2021-04-05T22:18:59.247+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:17 8.3 8:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776d0c0 con 0x5561f78c7400
  164> 2021-04-05T22:18:59.251+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v4991709'7984842 uv7984805 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -163> 2021-04-05T22:18:59.251+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] -- osd_op(unknown.0.0:18 8.6 8:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776d840 con 0x5561f78c7c00
  162> 2021-04-05T22:18:59.255+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.26 v2:10.11.165.113:6808/2273872 2 ==== osd_op_reply(18 gc.17 [create,call,call,call] v4991709'10332585 uv10332536 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7c00
  -161> 2021-04-05T22:18:59.255+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:19 8.5 8:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776c1c0 con 0x5561f78c8000
  160> 2021-04-05T22:18:59.255+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 3 ==== osd_op_reply(19 gc.18 [create,call,call,call] v4991709'11109703 uv11109666 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
  -159> 2021-04-05T22:18:59.255+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:20 8.4 8:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776c580 con 0x5561f78c7800
  158> 2021-04-05T22:18:59.259+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 4 ==== osd_op_reply(20 gc.19 [create,call,call,call] v4991709'12811732 uv12811409 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
  -157> 2021-04-05T22:18:59.259+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:21 8.7 8:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776a000 con 0x5561f78c8800
  156> 2021-04-05T22:18:59.263+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 2 ==== osd_op_reply(21 gc.20 [create,call,call,call] v4991709'8993444 uv8993193 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
  -155> 2021-04-05T22:18:59.263+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] -- osd_op(unknown.0.0:22 8.2 8:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776b680 con 0x5561f78c8400
  154> 2021-04-05T22:18:59.263+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.40 v2:10.11.165.114:6807/2251489 2 ==== osd_op_reply(22 gc.21 [create,call,call,call] v4991709'11008731 uv11008655 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c8400
  -153> 2021-04-05T22:18:59.263+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] -- osd_op(unknown.0.0:23 8.1 8:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776d480 con 0x5561f78c6c00
  152> 2021-04-05T22:18:59.267+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.1 v2:10.11.165.111:6804/2539437 3 ==== osd_op_reply(23 gc.22 [create,call,call,call] v4991709'11721300 uv11721226 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c6c00
  -151> 2021-04-05T22:18:59.267+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:24 8.0 8:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776c940 con 0x5561f78c7000
  150> 2021-04-05T22:18:59.271+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 3 ==== osd_op_reply(24 gc.23 [create,call,call,call] v4991709'10025114 uv10025028 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
  -149> 2021-04-05T22:18:59.271+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:25 8.3 8:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7766000 con 0x5561f78c7400
  148> 2021-04-05T22:18:59.271+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 6 ==== osd_op_reply(25 gc.24 [create,call,call,call] v4991709'7984843 uv7984783 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  -147> 2021-04-05T22:18:59.271+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] -- osd_op(unknown.0.0:26 8.1 8:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7768580 con 0x5561f78c6c00
  146> 2021-04-05T22:18:59.275+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.1 v2:10.11.165.111:6804/2539437 4 ==== osd_op_reply(26 gc.25 [create,call,call,call] v4991709'11721301 uv11721230 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c6c00
  -145> 2021-04-05T22:18:59.275+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:27 8.7 8:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7768d00 con 0x5561f78c8800
  144> 2021-04-05T22:18:59.279+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 3 ==== osd_op_reply(27 gc.26 [create,call,call,call] v4991709'8993445 uv8993388 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
  -143> 2021-04-05T22:18:59.279+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.114:6807/2251489,v1:10.11.165.114:6810/2251489] -- osd_op(unknown.0.0:28 8.2 8:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7767680 con 0x5561f78c8400
  142> 2021-04-05T22:18:59.279+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.40 v2:10.11.165.114:6807/2251489 3 ==== osd_op_reply(28 gc.27 [create,call,call,call] v4991709'11008732 uv11008416 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c8400
  -141> 2021-04-05T22:18:59.279+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:29 8.5 8:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7767e00 con 0x5561f78c8000
  140> 2021-04-05T22:18:59.283+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 4 ==== osd_op_reply(29 gc.28 [create,call,call,call] v4991709'11109704 uv11109418 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
  -139> 2021-04-05T22:18:59.283+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:30 8.7 8:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7768940 con 0x5561f78c8800
  138> 2021-04-05T22:18:59.283+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 4 ==== osd_op_reply(30 gc.29 [create,call,call,call] v4991709'8993446 uv8993197 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
  -137> 2021-04-05T22:18:59.287+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:31 8.0 8:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7766f00 con 0x5561f78c7000
  136> 2021-04-05T22:18:59.287+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 4 ==== osd_op_reply(31 gc.30 [create,call,call,call] v4991709'10025115 uv10024798 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
  -135> 2021-04-05T22:18:59.287+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:32 8.7 8:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77681c0 con 0x5561f78c8800
  134> 2021-04-05T22:18:59.291+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 5 ==== osd_op_reply(32 gc.31 [create,call,call,call] v4991709'8993447 uv8993429 ondisk = 125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
  -133> 2021-04-05T22:18:59.291+0700 7f480d983700  2 garbage collection: garbage collection: start
  -132> 2021-04-05T22:18:59.291+0700 7f480d983700 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=1
  -131> 2021-04-05T22:18:59.291+0700 7f480d182700  2 object expiration: start
  -130> 2021-04-05T22:18:59.291+0700 7f480d182700 20 processing shard = obj_delete_at_hint.0000000000
  -129> 2021-04-05T22:18:59.291+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:33 8.3 8:c8333aab:gc::gc.11:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7774f00 con 0x5561f78c7400
  128> 2021-04-05T22:18:59.291+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:34 8.0 8:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77752c0 con 0x5561f78c7000
  127> 2021-04-05T22:18:59.291+0700 7f480c180700 20 reqs_thread_entry: start
  -126> 2021-04-05T22:18:59.291+0700 7f4841d492c0  5 asok(0x5561f691c000) register_command sync trace show hook 0x5561f68241c0
  -125> 2021-04-05T22:18:59.291+0700 7f4841d492c0  5 asok(0x5561f691c000) register_command sync trace history hook 0x5561f68241c0
  -124> 2021-04-05T22:18:59.291+0700 7f4841d492c0  5 asok(0x5561f691c000) register_command sync trace active hook 0x5561f68241c0
  -123> 2021-04-05T22:18:59.291+0700 7f4841d492c0  5 asok(0x5561f691c000) register_command sync trace active_short hook 0x5561f68241c0
  -122> 2021-04-05T22:18:59.291+0700 7f4841d492c0  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:838 8.4 8:3b29b93f:::bilog.trim:head [watch watch cookie 93879251495168] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7766780 con 0x5561f7618400
  121> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 7 ==== osd_op_reply(33 gc.11 [call] v4991709'7984844 uv7984844 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  120> 2021-04-05T22:18:59.295+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 5 ==== osd_op_reply(34 obj_delete_at_hint.0000000000 [call] v4991709'10025116 uv10025116 ondisk = 0) v8 ==== 173+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
  119> 2021-04-05T22:18:59.295+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:35 8.3 8:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7775680 con 0x5561f78c7400
  118> 2021-04-05T22:18:59.295+0700 7f480d182700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:839 8.0 8:1a734c59:::obj_delete_at_hint.0000000000:head [call timeindex.list in=43b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7776580 con 0x5561f7617c00
  117> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 8 ==== osd_op_reply(35 gc.11 [call out=15b] v0'0 uv7984844 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  116> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 <== osd.3 v2:10.11.165.111:6808/2539455 123 ==== osd_op_reply(839 obj_delete_at_hint.0000000000 [call out=15b] v0'0 uv10025116 ondisk = 0) v8 ==== 173+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f7617c00
  115> 2021-04-05T22:18:59.295+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
  -114> 2021-04-05T22:18:59.295+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:36 8.3 8:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f77761c0 con 0x5561f78c7400
  113> 2021-04-05T22:18:59.295+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:37 8.0 8:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7777840 con 0x5561f78c7000
  112> 2021-04-05T22:18:59.295+0700 7f48299bb700  1 - 10.11.165.111:0/2427470127 <== osd.46 v2:10.11.165.115:6816/2183272 77 ==== osd_op_reply(838 bilog.trim [watch watch cookie 93879251495168] v4991709'12811733 uv12811351 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f7618400
  111> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 9 ==== osd_op_reply(36 gc.11 [call out=24b] v0'0 uv7984844 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  110> 2021-04-05T22:18:59.295+0700 7f4841d492c0 10 trim: Watching bilog.trim
  -109> 2021-04-05T22:18:59.295+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:38 8.3 8:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7777c00 con 0x5561f78c7400
  108> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 10 ==== osd_op_reply(38 gc.11 [call out=15b] v0'0 uv7984844 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  107> 2021-04-05T22:18:59.295+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE
  -106> 2021-04-05T22:18:59.295+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:39 8.3 8:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7775e00 con 0x5561f78c7400
  105> 2021-04-05T22:18:59.295+0700 7f480917a700 20 reqs_thread_entry: start
  -104> 2021-04-05T22:18:59.295+0700 7f4808979700 20 cr:s=0x5561f686a800:op=0x5561f7813800:20MetaMasterTrimPollCR: operate()
  -103> 2021-04-05T22:18:59.295+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 11 ==== osd_op_reply(39 gc.11 [call out=15b] v0'0 uv7984844 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
  102> 2021-04-05T22:18:59.295+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
  -101> 2021-04-05T22:18:59.295+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6840/2183343,v1:10.11.165.115:6841/2183343] -- osd_op(unknown.0.0:40 8.3 8:c8333aab:gc::gc.11:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7776940 con 0x5561f78c7400
  100> 2021-04-05T22:18:59.295+0700 7f4808979700 20 run: stack=0x5561f686a800 is io blocked
   -99> 2021-04-05T22:18:59.295+0700 7f4808979700 20 cr:s=0x5561f686a6c0:op=0x5561f747d500:17DataLogTrimPollCR: operate()
   -98> 2021-04-05T22:18:59.295+0700 7f4808979700 20 run: stack=0x5561f686a6c0 is io blocked
   -97> 2021-04-05T22:18:59.295+0700 7f4808979700 20 cr:s=0x5561f68688c0:op=0x5561f751b800:16BucketTrimPollCR: operate()
   -96> 2021-04-05T22:18:59.295+0700 7f4808979700 20 run: stack=0x5561f68688c0 is io blocked
   -95> 2021-04-05T22:18:59.295+0700 7f4807977700 20 reqs_thread_entry: start
   -94> 2021-04-05T22:18:59.295+0700 7f4805172700  5 lifecycle: schedule life cycle next start time: Mon Apr  5 17:00:00 2021
   -93> 2021-04-05T22:18:59.295+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 6 ==== osd_op_reply(37 obj_delete_at_hint.0000000000 [call] v4991709'10025117 uv10025117 ondisk = 0) v8 ==== 173+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
   92> 2021-04-05T22:18:59.299+0700 7f480d182700 20 processing shard = obj_delete_at_hint.0000000001
   -91> 2021-04-05T22:18:59.299+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:41 8.7 8:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776af00 con 0x5561f78c8800
   90> 2021-04-05T22:18:59.299+0700 7f480316e700  5 lifecycle: schedule life cycle next start time: Mon Apr  5 17:00:00 2021
   -89> 2021-04-05T22:18:59.299+0700 7f480116a700  5 lifecycle: schedule life cycle next start time: Mon Apr  5 17:00:00 2021
   -88> 2021-04-05T22:18:59.299+0700 7f4800969700 20 BucketsSyncThread: start
   -87> 2021-04-05T22:18:59.299+0700 7f4841d492c0 20 init_complete bucket index max shards: 0
   -86> 2021-04-05T22:18:59.299+0700 7f4800168700 20 UserSyncThread: start
   -85> 2021-04-05T22:18:59.299+0700 7f47ff967700 20 processing logshard = reshard.0000000000
   -84> 2021-04-05T22:18:59.299+0700 7f4800168700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.112:6832/24146,v1:10.11.165.112:6833/24146] conn(0x5561f78c9400 0x5561f7625700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
   83> 2021-04-05T22:18:59.299+0700 7f4800168700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.112:6832/24146,v1:10.11.165.112:6833/24146] -- osd_op(unknown.0.0:840 7.0 7:00000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f776a780 con 0x5561f78c9400
   82> 2021-04-05T22:18:59.299+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:42 8.7 8:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776ab40 con 0x5561f78c8800
   81> 2021-04-05T22:18:59.299+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.49 v2:10.11.165.115:6840/2183343 12 ==== osd_op_reply(40 gc.11 [call] v4991709'7984845 uv7984845 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c7400
   80> 2021-04-05T22:18:59.299+0700 7f480d983700 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=1
   -79> 2021-04-05T22:18:59.299+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:43 8.5 8:ab569f81:gc::gc.12:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f776a3c0 con 0x5561f78c8000
   78> 2021-04-05T22:18:59.299+0700 7f4841d492c0 10 Started notification manager with: 1 workers
   -77> 2021-04-05T22:18:59.299+0700 7f482a9bd700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.112:6832/24146,v1:10.11.165.112:6833/24146] conn(0x5561f78c9400 0x5561f7625700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
   -76> 2021-04-05T22:18:59.299+0700 7f4841d492c0 20 RGW hostnames: 
   -75> 2021-04-05T22:18:59.299+0700 7f4841d492c0 20 RGW S3website hostnames: 
   -74> 2021-04-05T22:18:59.299+0700 7f482a9bd700 10 monclient: get_auth_request con 0x5561f78c9400 auth_method 0
   -73> 2021-04-05T22:18:59.299+0700 7f47fe965700 20 INFO: next queues processing will happen at: Mon Apr  5 22:19:29 2021
72> 2021-04-05T22:18:59.299+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 6 ==== osd_op_reply(41 obj_delete_at_hint.0000000001 [call] v4991709'8993448 uv8993448 ondisk = 0) v8 ==== 173+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
   71> 2021-04-05T22:18:59.299+0700 7f480d182700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:841 8.7 8:f1983ba0:::obj_delete_at_hint.0000000001:head [call timeindex.list in=43b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f773e000 con 0x5561f7617000
   70> 2021-04-05T22:18:59.299+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 7 ==== osd_op_reply(42 reshard.0000000000 [call] v4991709'8993449 uv8993449 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
   69> 2021-04-05T22:18:59.299+0700 7f482a9bd700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.112:6832/24146,v1:10.11.165.112:6833/24146] conn(0x5561f78c9400 0x5561f7625700 crc :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.18 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
   68> 2021-04-05T22:18:59.299+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 <== osd.7 v2:10.11.165.111:6818/2539443 73 ==== osd_op_reply(841 obj_delete_at_hint.0000000001 [call out=15b] v0'0 uv8993448 ondisk = 0) v8 ==== 173+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f7617000
   67> 2021-04-05T22:18:59.299+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:44 8.7 8:e9187ab8:reshard::reshard.0000000000:head [call rgw.reshard_list in=29b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7767a40 con 0x5561f78c8800
   66> 2021-04-05T22:18:59.299+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:45 8.7 8:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f773e780 con 0x5561f78c8800
   65> 2021-04-05T22:18:59.299+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 8 ==== osd_op_reply(44 reshard.0000000000 [call out=11b] v0'0 uv8993449 ondisk = 0) v8 ==== 162+0+11 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
   64> 2021-04-05T22:18:59.299+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6818/2539443,v1:10.11.165.111:6822/2539443] -- osd_op(unknown.0.0:46 8.7 8:e9187ab8:reshard::reshard.0000000000:head [call lock.unlock in=54b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7769c00 con 0x5561f78c8800
   63> 2021-04-05T22:18:59.299+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 5 ==== osd_op_reply(43 gc.12 [call] v4991709'11109705 uv11109705 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   62> 2021-04-05T22:18:59.303+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:47 8.5 8:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776d840 con 0x5561f78c8000
   61> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 <== osd.18 v2:10.11.165.112:6832/24146 1 ==== osd_op_reply(840  [pgnls start_epoch 4991709 out=200b] v4991629'207 uv207 ondisk = 1) v8 ==== 144+0+200 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c9400
   60> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.111:6836/2539459,v1:10.11.165.111:6837/2539459] -- osd_op(unknown.0.0:842 7.4 7:20000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f776d840 con 0x5561f688f000
   59> 2021-04-05T22:18:59.303+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 9 ==== osd_op_reply(45 obj_delete_at_hint.0000000001 [call] v4991709'8993450 uv8993450 ondisk = 0) v8 ==== 173+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
   58> 2021-04-05T22:18:59.303+0700 7f480d182700 20 processing shard = obj_delete_at_hint.0000000002
   -57> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 6 ==== osd_op_reply(47 gc.12 [call out=15b] v0'0 uv11109705 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   56> 2021-04-05T22:18:59.303+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] -- osd_op(unknown.0.0:48 8.6 8:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7769840 con 0x5561f78c7c00
   55> 2021-04-05T22:18:59.303+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
   -54> 2021-04-05T22:18:59.303+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:49 8.5 8:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776a000 con 0x5561f78c8000
   53> 2021-04-05T22:18:59.303+0700 7f4841d492c0  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:50 8.5 8:a60f9269:lc::lua_package_allowlist:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7766b40 con 0x5561f78c8000
   52> 2021-04-05T22:18:59.303+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.7 v2:10.11.165.111:6818/2539443 10 ==== osd_op_reply(46 reshard.0000000000 [call] v4991709'8993451 uv8993451 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c8800
   51> 2021-04-05T22:18:59.303+0700 7f47ff967700 20 finish processing logshard = reshard.0000000000 , ret = 0
   -50> 2021-04-05T22:18:59.303+0700 7f47ff967700 20 processing logshard = reshard.0000000001
   -49> 2021-04-05T22:18:59.303+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:51 8.4 8:2c0c0c23:reshard::reshard.0000000001:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7774f00 con 0x5561f78c7800
   48> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 7 ==== osd_op_reply(49 gc.12 [call out=24b] v0'0 uv11109705 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   47> 2021-04-05T22:18:59.303+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:52 8.5 8:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776c940 con 0x5561f78c8000
   46> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 8 ==== osd_op_reply(50 lua_package_allowlist [omap-get-keys] v0'0 uv0 ondisk = 2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   -45> 2021-04-05T22:18:59.303+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 <== osd.0 v2:10.11.165.111:6836/2539459 9 ==== osd_op_reply(842  [pgnls start_epoch 4991709 out=296b] v4991709'149221 uv149221 ondisk = 1) v8 ==== 144+0+296 (crc 0 0 0) 0x5561f75c7680 con 0x5561f688f000
   44> 2021-04-05T22:18:59.303+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:843 7.2 7:40000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f7774f00 con 0x5561f7617c00
   43> 2021-04-05T22:18:59.303+0700 7f4841d492c0  0 framework: beast
   -42> 2021-04-05T22:18:59.303+0700 7f4841d492c0  0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt
   -41> 2021-04-05T22:18:59.303+0700 7f4841d492c0  0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key
   -40> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 9 ==== osd_op_reply(52 gc.12 [call out=15b] v0'0 uv11109705 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   39> 2021-04-05T22:18:59.303+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE
   -38> 2021-04-05T22:18:59.303+0700 7f4841d492c0  0 starting handler: beast
   -37> 2021-04-05T22:18:59.303+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:53 8.5 8:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f773fa40 con 0x5561f78c8000
   36> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 10 ==== osd_op_reply(53 gc.12 [call out=15b] v0'0 uv11109705 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   35> 2021-04-05T22:18:59.303+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
   -34> 2021-04-05T22:18:59.303+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6836/2274027,v1:10.11.165.113:6837/2274027] -- osd_op(unknown.0.0:54 8.5 8:ab569f81:gc::gc.12:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77761c0 con 0x5561f78c8000
   33> 2021-04-05T22:18:59.303+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.26 v2:10.11.165.113:6808/2273872 3 ==== osd_op_reply(48 obj_delete_at_hint.0000000002 [call] v4991709'10332586 uv10332586 ondisk = 0) v8 ==== 173+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7c00
   32> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 <== osd.3 v2:10.11.165.111:6808/2539455 124 ==== osd_op_reply(843  [pgnls start_epoch 4991709 out=283b] v4950218'10691 uv10691 ondisk = 1) v8 ==== 144+0+283 (crc 0 0 0) 0x5561f75c7440 con 0x5561f7617c00
   31> 2021-04-05T22:18:59.303+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 5 ==== osd_op_reply(51 reshard.0000000001 [call] v4991709'12811734 uv12811734 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
   30> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.114:6830/2251511,v1:10.11.165.114:6837/2251511] conn(0x5561f78c9800 0x5561f79eca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect
   29> 2021-04-05T22:18:59.303+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.114:6830/2251511,v1:10.11.165.114:6837/2251511] -- osd_op(unknown.0.0:844 7.6 7:60000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f77761c0 con 0x5561f78c9800
   28> 2021-04-05T22:18:59.303+0700 7f480d182700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] -- osd_op(unknown.0.0:845 8.6 8:703a9e16:::obj_delete_at_hint.0000000002:head [call timeindex.list in=43b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f77690c0 con 0x5561f7618000
   27> 2021-04-05T22:18:59.307+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:55 8.4 8:2c0c0c23:reshard::reshard.0000000001:head [call rgw.reshard_list in=29b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7775680 con 0x5561f78c7800
   26> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.114:6830/2251511,v1:10.11.165.114:6837/2251511] conn(0x5561f78c9800 0x5561f79eca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0
   25> 2021-04-05T22:18:59.307+0700 7f482a1bc700 10 monclient: get_auth_request con 0x5561f78c9800 auth_method 0
   -24> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 <== osd.26 v2:10.11.165.113:6808/2273872 119 ==== osd_op_reply(845 obj_delete_at_hint.0000000002 [call out=15b] v0'0 uv10332586 ondisk = 0) v8 ==== 173+0+15 (crc 0 0 0) 0x5561f75c7680 con 0x5561f7618000
   23> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 6 ==== osd_op_reply(55 reshard.0000000001 [call out=11b] v0'0 uv12811734 ondisk = 0) v8 ==== 162+0+11 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
   22> 2021-04-05T22:18:59.307+0700 7f480d182700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.113:6808/2273872,v1:10.11.165.113:6809/2273872] -- osd_op(unknown.0.0:56 8.6 8:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7768d00 con 0x5561f78c7c00
   21> 2021-04-05T22:18:59.307+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.115:6816/2183272,v1:10.11.165.115:6817/2183272] -- osd_op(unknown.0.0:57 8.4 8:2c0c0c23:reshard::reshard.0000000001:head [call lock.unlock in=54b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7776580 con 0x5561f78c7800
   20> 2021-04-05T22:18:59.307+0700 7f482a9bd700  1 - 10.11.165.111:0/856779288 <== osd.28 v2:10.11.165.113:6836/2274027 11 ==== osd_op_reply(54 gc.12 [call] v4991709'11109706 uv11109706 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x5561f75c7440 con 0x5561f78c8000
   19> 2021-04-05T22:18:59.307+0700 7f480d983700 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=1
   -18> 2021-04-05T22:18:59.307+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:58 8.0 8:0667bb13:gc::gc.13:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f7777c00 con 0x5561f78c7000
   17> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 --2 10.11.165.111:0/2427470127 >> [v2:10.11.165.114:6830/2251511,v1:10.11.165.114:6837/2251511] conn(0x5561f78c9800 0x5561f79eca00 crc :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.33 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
   16> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 <== osd.33 v2:10.11.165.114:6830/2251511 1 ==== osd_op_reply(844  [pgnls start_epoch 4991709 out=182b] v4991709'361484 uv361484 ondisk = 1) v8 ==== 144+0+182 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c9800
   15> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.115:6804/2183313,v1:10.11.165.115:6805/2183313] -- osd_op(unknown.0.0:846 7.1 7:80000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f77761c0 con 0x5561f7616000
   14> 2021-04-05T22:18:59.307+0700 7f482a1bc700  1 - 10.11.165.111:0/856779288 <== osd.46 v2:10.11.165.115:6816/2183272 7 ==== osd_op_reply(57 reshard.0000000001 [call] v4991709'12811735 uv12811735 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x5561f75c7680 con 0x5561f78c7800
   13> 2021-04-05T22:18:59.307+0700 7f47ff967700 20 finish processing logshard = reshard.0000000001 , ret = 0
   -12> 2021-04-05T22:18:59.307+0700 7f47ff967700 20 processing logshard = reshard.0000000002
   -11> 2021-04-05T22:18:59.307+0700 7f47ff967700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6804/2539437,v1:10.11.165.111:6805/2539437] -- osd_op(unknown.0.0:59 8.1 8:856b9b26:reshard::reshard.0000000002:head [call lock.lock in=70b] snapc 0=[] ondisk+write+known_if_redirected e4991709) v8 -- 0x5561f77401c0 con 0x5561f78c6c00
   10> 2021-04-05T22:18:59.307+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 7 ==== osd_op_reply(58 gc.13 [call] v4991709'10025118 uv10025118 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
    9> 2021-04-05T22:18:59.307+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:60 8.0 8:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7775e00 con 0x5561f78c7000
    8> 2021-04-05T22:18:59.311+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 <== osd.53 v2:10.11.165.115:6804/2183313 4 ==== osd_op_reply(846  [pgnls start_epoch 4991709 out=350b] v4932503'1297 uv1297 ondisk = 1) v8 ==== 144+0+350 (crc 0 0 0) 0x5561f75c7440 con 0x5561f7616000
    7> 2021-04-05T22:18:59.311+0700 7f482a9bd700  1 - 10.11.165.111:0/2427470127 --> [v2:10.11.165.114:6808/2251458,v1:10.11.165.114:6812/2251458] -- osd_op(unknown.0.0:847 7.5 7:a0000000:users.uid:::head [pgnls start_epoch 4991709 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e4991709) v8 -- 0x5561f77401c0 con 0x5561f7617400
    6> 2021-04-05T22:18:59.311+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 8 ==== osd_op_reply(60 gc.13 [call out=15b] v0'0 uv10025118 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
    5> 2021-04-05T22:18:59.311+0700 7f480d983700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
    -4> 2021-04-05T22:18:59.311+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:61 8.0 8:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f7776940 con 0x5561f78c7000
    3> 2021-04-05T22:18:59.311+0700 7f48299bb700  1 - 10.11.165.111:0/856779288 <== osd.3 v2:10.11.165.111:6808/2539455 9 ==== osd_op_reply(61 gc.13 [call out=24b] v0'0 uv10025118 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x5561f75c6b40 con 0x5561f78c7000
    2> 2021-04-05T22:18:59.311+0700 7f480d983700  1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:62 8.0 8:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776b2c0 con 0x5561f78c7000
    1> 2021-04-05T22:18:59.311+0700 7f482a1bc700  1 - 10.11.165.111:0/2427470127 <== osd.30 v2:10.11.165.114:6808/2251458 3 ==== osd_op_reply(847  [pgnls start_epoch 4991709 out=215b] v4991707'29288 uv29288 ondisk = 1) v8 ==== 144+0+215 (crc 0 0 0) 0x5561f75c7680 con 0x5561f7617400
     0> 2021-04-05T22:18:59.311+0700 7f4841d492c0 -1 ** Caught signal (Segmentation fault) *
 in thread 7f4841d492c0 thread_name:radosgw
Actions
Copy link
 #3

Updated by Enrico Kern about 3 years ago

Hello,

i have a similar issue with none cephadm cluster. Upgraded from latest octopus to pacific via the ubuntu packages in the repo.

After upgrade all existing radosgw stil running octopus crashed and startup does not work with permission denied errors. Also manual startup only shows up an error:

/usr/bin/radosgw -f --cluster ceph --name client.rgw.stor01-fs --setuser ceph --setgroup ceph
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 int rgw::cls::fifo::get_meta(librados::v14_2_0::IoCtx&, const string&, std::optional<rados::cls::fifo::objv>, rados::cls::fifo::info*, uint32_t*, uint32_t*, uint64_t, optional_yield, bool):105 fifo::op::GET_META failed r=-1 tid=0
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, bool):884 get_meta failed: r=-1
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 static int RGWDataChangesFIFO::exists(ceph::common::CephContext*, librados::v14_2_0::Rados*, const rgw_pool&, bool*, bool*): unable to open FIFO: default.rgw.log/data_log.0: (1) Operation not permitted
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when checking for existing FIFO datalog backend: (1) Operation not permitted
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-1
2021-04-06T12:40:17.878+0200 7fe772a0fb40 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Operation not permitted
2021-04-06T12:40:17.898+0200 7fe772a0fb40 -1 Couldn't init storage provider (RADOS)

similar permission denied errors come up with OSDs, they also do not start with systemd (but start manual).

Actions
Copy link
 #4

Updated by Casey Bodley about 3 years ago

Actions
Copy link
 #5

Updated by Casey Bodley about 3 years ago

  • Status changed from Need More Info to In Progress
Actions
Copy link
 #6

Updated by Casey Bodley about 3 years ago

Prince Inamatus wrote:

2> 2021-04-05T22:18:59.311+0700 7f480d983700 1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:62 8.0 8:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776b2c0 con 0x5561f78c7000
1> 2021-04-05T22:18:59.311+0700 7f482a1bc700 1 - 10.11.165.111:0/2427470127 <== osd.30 v2:10.11.165.114:6808/2251458 3 ==== osd_op_reply(847 [pgnls start_epoch 4991709 out=215b] v4991707'29288 uv29288 ondisk = 1) v8 ==== 144+0+215 (crc 0 0 0) 0x5561f75c7680 con 0x5561f7617400
0> 2021-04-05T22:18:59.311+0700 7f4841d492c0 -1 ** Caught signal (Segmentation fault) *
in thread 7f4841d492c0 thread_name:radosgw

ok, this is a crash and may not be related to the RGWDataChangesFIFO errors. you should see another 'Caught signal (Segmentation fault)' earlier in the log with a backtrace. are you able to share that too?

Actions
Copy link
 #7

Updated by Prince Inamatus about 3 years ago

Casey Bodley wrote:

Prince Inamatus wrote:

2> 2021-04-05T22:18:59.311+0700 7f480d983700 1 - 10.11.165.111:0/856779288 --> [v2:10.11.165.111:6808/2539455,v1:10.11.165.111:6809/2539455] -- osd_op(unknown.0.0:62 8.0 8:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected e4991709) v8 -- 0x5561f776b2c0 con 0x5561f78c7000
1> 2021-04-05T22:18:59.311+0700 7f482a1bc700 1 - 10.11.165.111:0/2427470127 <== osd.30 v2:10.11.165.114:6808/2251458 3 ==== osd_op_reply(847 [pgnls start_epoch 4991709 out=215b] v4991707'29288 uv29288 ondisk = 1) v8 ==== 144+0+215 (crc 0 0 0) 0x5561f75c7680 con 0x5561f7617400
0> 2021-04-05T22:18:59.311+0700 7f4841d492c0 -1 ** Caught signal (Segmentation fault) *
in thread 7f4841d492c0 thread_name:radosgw

ok, this is a crash and may not be related to the RGWDataChangesFIFO errors. you should see another 'Caught signal (Segmentation fault)' earlier in the log with a backtrace. are you able to share that too?

็I'm not sure to do that. Can you suggest me to that?

Now I upload my all log that debug ms = 1 and debug rgw = 20
with google drive link because it's size more than i can upload.

https://drive.google.com/file/d/1awweMvrC75nRR4eKgApxGT0TgJuSUDAG/view?usp=sharing

Actions
Copy link
 #8

Updated by Prince Inamatus about 3 years ago

Actions
Copy link
 #9

Updated by Casey Bodley about 3 years ago

  • Related to Bug #50218: Segmentation fault in radosgw_Main() added
Actions
Copy link
 #10

Updated by Casey Bodley about 3 years ago

  • Related to deleted (Feature #50104: Online switch of datalog format)
Actions
Copy link
 #11

Updated by Casey Bodley about 3 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF