Project

General

Profile

Actions
Copy link

Bug #50756

open

Haproxy with keepalive crashes RGW when object name lenght > 1024 bytes

Added by Aleksandr Rudenko almost 3 years ago. Updated almost 3 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
Or Friedmann
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.15
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

AWS S3 specification required object names lenght <= 1024 bytes and RGW follows this spec.

But we have problem if RGW is behind haproxy (http mode with keepalive). In this case request with name >1024 bytes crashes RGW for ~30 secs. During this 30 secs RGW can't respond on other requests.

Example:

>>> import boto3
>>> import botocore.config
>>> config = botocore.config.Config(s3={"addressing_style": "path"});
>>>
>>>
>>>
>>> session = boto3.Session(aws_access_key_id="access", aws_secret_access_key="secret")
>>> client = session.client("s3", use_ssl=True, verify=False, endpoint_url="https://s3.domain.com", config=config)

>>> client.put_object(Bucket="test1", Key="a" * 2000, Body=b"test"

Response is normal:

botocore.exceptions.ClientError: An error occurred (InvalidObjectName) when calling the PutObject operation: Unknown

But next another request will fail:

>>> client.list_buckets()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ruden/envs/boto3/.venv/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/ruden/envs/boto3/.venv/lib/python2.7/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred () when calling the ListBuckets operation:

In RGW logs (debug_rgw 20) we can't see any errors but we can see that RGW will complete first request only after some time (10-30 secs).

Haproxy default options (for all RGW frontends):

defaults
    mode http
    log global

    option httplog
    option http-ignore-probes
    option redispatch

    retries 1
    timeout queue 30s
    timeout connect 10s
    timeout client 30s
    timeout server 360s
    timeout http-request 5s
    timeout http-keep-alive 10s

We can't see this problem with:

option forceclose

Actions
Copy link
 #1

Updated by Or Friedmann almost 3 years ago

  • Assignee set to Or Friedmann
Actions
Copy link
 #2

Updated by Or Friedmann almost 3 years ago

  • Status changed from New to Need More Info

I tried to reproduce it on my machine without success (it works fine in my machine), Can you please share logs from RGW with debug_rgw=20 and debug_ms=1 + haproxy logs?

Thank you

Actions
Copy link
 #3

Updated by Aleksandr Rudenko almost 3 years ago

Or Friedmann wrote:

I tried to reproduce it on my machine without success (it works fine in my machine), Can you please share logs from RGW with debug_rgw=20 and debug_ms=1 + haproxy logs?

Thank you

RGW log with problem request:
2021-06-15 18:48:42.943 7f8ccdfd6700 20 CONTENT_LENGTH=4
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_ACCEPT_ENCODING=identity
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential={secret}20210615/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=3875d3edb3003775239a01632be9d620124ea3c18628c4e4cf0b95a466e24d38
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_CONTENT_MD5=CY9rzUYh03PK3k6DJie09g==
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_EXPECT=100-continue
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_HOST={secret}
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_USER_AGENT=Boto3/1.17.35 Python/2.7.16 Darwin/20.4.0 Botocore/1.20.35
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_VERSION=1.1
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_X_AMZ_CONTENT_SHA256=9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_X_AMZ_DATE=20210615T154842Z
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_X_FORWARDED_FOR={secret}
2021-06-15 18:48:42.943 7f8ccdfd6700 20 HTTP_X_FORWARDED_PROTO=http
2021-06-15 18:48:42.943 7f8ccdfd6700 20 REMOTE_ADDR=192.168.36.115
2021-06-15 18:48:42.943 7f8ccdfd6700 20 REQUEST_METHOD=PUT
2021-06-15 18:48:42.943 7f8ccdfd6700 20 REQUEST_URI=/test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2021-06-15 18:48:42.943 7f8ccdfd6700 20 SCRIPT_URI=/test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2021-06-15 18:48:42.943 7f8ccdfd6700 20 SERVER_PORT=7480
2021-06-15 18:48:42.943 7f8ccdfd6700  1 ====== starting new request req=0x7f8ccdfcf7f0 =====
2021-06-15 18:48:42.943 7f8ccdfd6700  2 req 4641 0.000s initializing for trans_id = tx000000000000000001221-0060c8cbda-d7a2-default
2021-06-15 18:48:42.943 7f8ccdfd6700 10 rgw api priority: s3=2 s3website=-1
2021-06-15 18:48:42.943 7f8ccdfd6700 10 host={secret}
2021-06-15 18:48:42.943 7f8ccdfd6700 20 subdomain= domain={secret} in_hosted_domain=1 in_hosted_domain_s3website=0
2021-06-15 18:48:42.943 7f8ccdfd6700 20 final domain/bucket subdomain= domain={secret} in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain={secret} s->info.request_uri=/test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2021-06-15 18:48:42.943 7f8ccdfd6700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2021-06-15 18:48:42.943 7f8ccdfd6700 10 meta>> HTTP_X_AMZ_DATE
2021-06-15 18:48:42.943 7f8ccdfd6700 10 x>> x-amz-content-sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
2021-06-15 18:48:42.943 7f8ccdfd6700 10 x>> x-amz-date:20210615T154842Z
2021-06-15 18:48:42.943 7f8ccdfd6700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2021-06-15 18:48:42.943 7f8ccdfd6700 10 handler=22RGWHandler_REST_Obj_S3
2021-06-15 18:48:42.943 7f8ccdfd6700  2 req 4641 0.000s getting op 1
2021-06-15 18:48:42.943 7f8ccdfd6700 10 op=21RGWPutObj_ObjStore_S3
2021-06-15 18:48:42.943 7f8ccdfd6700  2 req 4641 0.000s s3:put_obj verifying requester
2021-06-15 18:48:42.943 7f8ccdfd6700 20 req 4641 0.000s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2021-06-15 18:48:42.943 7f8ccdfd6700 20 req 4641 0.000s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2021-06-15 18:48:42.943 7f8ccdfd6700 20 req 4641 0.000s s3:put_obj rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2021-06-15 18:48:42.943 7f8ccdfd6700 20 req 4641 0.000s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2021-06-15 18:48:42.943 7f8ccdfd6700 10 v4 signature format = 3875d3edb3003775239a01632be9d620124ea3c18628c4e4cf0b95a466e24d38
2021-06-15 18:48:42.943 7f8ccdfd6700 10 v4 credential format = {secret}/20210615/us-east-1/s3/aws4_request
2021-06-15 18:48:42.943 7f8ccdfd6700 10 access key id = {secret}
2021-06-15 18:48:42.943 7f8ccdfd6700 10 credential scope = 20210615/us-east-1/s3/aws4_request
2021-06-15 18:48:42.943 7f8ccdfd6700 10 canonical headers format = content-md5:CY9rzUYh03PK3k6DJie09g==
host:{secret}
x-amz-content-sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
x-amz-date:20210615T154842Z

2021-06-15 18:48:42.943 7f8ccdfd6700 10 payload request hash = 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
2021-06-15 18:48:42.944 7f8ccdfd6700 10 canonical request = PUT
/test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

content-md5:CY9rzUYh03PK3k6DJie09g==
host:{secret}
x-amz-content-sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
x-amz-date:20210615T154842Z

content-md5;host;x-amz-content-sha256;x-amz-date
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
2021-06-15 18:48:42.944 7f8ccdfd6700 10 canonical request hash = f640ce3e505a7ada1528690883552fb33b7a125413d06f59048c277ca2764ec1
2021-06-15 18:48:42.944 7f8ccdfd6700 10 string to sign = AWS4-HMAC-SHA256
20210615T154842Z
20210615/us-east-1/s3/aws4_request
f640ce3e505a7ada1528690883552fb33b7a125413d06f59048c277ca2764ec1
2021-06-15 18:48:42.944 7f8ccdfd6700 10 delaying v4 auth
2021-06-15 18:48:42.944 7f8ccdfd6700 10 date_k    = f4bbef1fbb7449dec52703fb152b380427985f03b9d54a2324f50063cea24c39
2021-06-15 18:48:42.944 7f8ccdfd6700 10 region_k  = f2dd98f1cac2a874ad7cc1c147e87ff833209b15810b0596d58c1f48815b43e5
2021-06-15 18:48:42.944 7f8ccdfd6700 10 service_k = fefcbde4fae01c02252f328b3c105d4d8afa13bb4fd70421fc3276382fb38fa6
2021-06-15 18:48:42.944 7f8ccdfd6700 10 signing_k = 00bbc41272afe7f83123ba6c76bb853e4b30327bbacdfa7d271e58f486c1cdf5
2021-06-15 18:48:42.944 7f8ccdfd6700 10 generated signature = 3875d3edb3003775239a01632be9d620124ea3c18628c4e4cf0b95a466e24d38
2021-06-15 18:48:42.944 7f8ccdfd6700 15 req 4641 0.001s s3:put_obj string_to_sign=AWS4-HMAC-SHA256
20210615T154842Z
20210615/us-east-1/s3/aws4_request
f640ce3e505a7ada1528690883552fb33b7a125413d06f59048c277ca2764ec1
2021-06-15 18:48:42.944 7f8ccdfd6700 15 req 4641 0.001s s3:put_obj server signature=3875d3edb3003775239a01632be9d620124ea3c18628c4e4cf0b95a466e24d38
2021-06-15 18:48:42.944 7f8ccdfd6700 15 req 4641 0.001s s3:put_obj client signature=3875d3edb3003775239a01632be9d620124ea3c18628c4e4cf0b95a466e24d38
2021-06-15 18:48:42.944 7f8ccdfd6700 15 req 4641 0.001s s3:put_obj compare=0
2021-06-15 18:48:42.944 7f8ccdfd6700 20 req 4641 0.001s s3:put_obj rgw::auth::s3::LocalEngine granted access
2021-06-15 18:48:42.944 7f8ccdfd6700 20 req 4641 0.001s s3:put_obj rgw::auth::s3::AWSAuthStrategy granted access
2021-06-15 18:48:42.944 7f8ccdfd6700  2 req 4641 0.001s s3:put_obj normalizing buckets and tenants
2021-06-15 18:48:42.944 7f8ccdfd6700 10 s->object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa s->bucket=test1
2021-06-15 18:48:42.944 7f8ccdfd6700 10 failed to run post-auth init
2021-06-15 18:48:42.944 7f8ccdfd6700 20 op->ERRORHANDLER: err_no=-2001 new_err_no=-2001
2021-06-15 18:48:42.944 7f8ccdfd6700  2 req 4641 0.001s s3:put_obj op status=0
2021-06-15 18:48:42.944 7f8ccdfd6700  2 req 4641 0.001s s3:put_obj http status=400
2021-06-15 18:48:42.944 7f8ccdfd6700  1 ====== req done req=0x7f8ccdfcf7f0 op status=0 http_status=400 latency=0.00100002s ======
2021-06-15 18:48:42.944 7f8ccdfd6700 20 process_request() returned -2001

2021-06-15 18:48:45.148 7f8ce5805700  1 -- 192.168.36.115:0/1670253716 --> [v2:192.168.36.115:6800/62151,v1:192.168.36.115:6801/62151] -- mgrreport(rgw.a +0-0 packed 294 status=1) v8 -- 0x557dcf665180 con 0x557dcf661c00

====> TIMEOUT <====

2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568299 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf520580 con 0x557dcc4a5400
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568300 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf4c9b80 con 0x557dcc4cc000
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568301 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b62c0 con 0x557dcc4cc000
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568302 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b7b80 con 0x557dcc2a8800
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568303 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b6000 con 0x557dcf571400
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568304 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b7340 con 0x557dcc419c00
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568305 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b7080 con 0x557dcc418c00
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568306 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61fb80 con 0x557dcc418c00
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcf4f3c00 con 0x557dcc2a8800
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dd0e18a80 con 0x557dcc418c00
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dd0e18540 con 0x557dcc419c00
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dcc4f68c0 con 0x557dcf571400
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dcf5e7500 con 0x557dcc4cc000
2021-06-15 18:48:47.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dcc4f7340 con 0x557dcc4a5400
2021-06-15 18:48:47.512 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195375 ==== osd_op_reply(1568299 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc435900 con 0x557dcc4a5400
2021-06-15 18:48:47.512 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413213 ==== osd_op_reply(1568300 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dd0e3ca00 con 0x557dcc4cc000
2021-06-15 18:48:47.512 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413214 ==== osd_op_reply(1568301 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dd0e3ca00 con 0x557dcc4cc000
2021-06-15 18:48:47.512 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195956 ==== osd_op_reply(1568302 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dd0e3ca00 con 0x557dcc2a8800
2021-06-15 18:48:47.513 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195535 ==== osd_op_reply(1568304 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc435900 con 0x557dcc419c00
2021-06-15 18:48:47.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174481 ==== osd_op_reply(1568303 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcf6a9900 con 0x557dcf571400
2021-06-15 18:48:47.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390926 ==== osd_op_reply(1568305 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:48:47.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390927 ==== osd_op_reply(1568306 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:48:51.900 7f8ce4002700  2 RGWDataChangesLog::ChangesRenewThread: start
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568307 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61eb00 con 0x557dcc4a5400
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568308 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61f8c0 con 0x557dcc4cc000
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568309 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61e2c0 con 0x557dcc4cc000
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568310 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61e840 con 0x557dcc2a8800
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568311 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61f600 con 0x557dcf571400
2021-06-15 18:48:52.512 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568312 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61e000 con 0x557dcc419c00
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568313 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfd8c0 con 0x557dcc418c00
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568314 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfc000 con 0x557dcc418c00
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcf5e7180 con 0x557dcc2a8800
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dcf4f21c0 con 0x557dcc418c00
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dd0e19a40 con 0x557dcc419c00
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dd0e19180 con 0x557dcf571400
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dd0e19340 con 0x557dcc4cc000
2021-06-15 18:48:52.513 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dd0e18700 con 0x557dcc4a5400
2021-06-15 18:48:52.513 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195376 ==== osd_op_reply(1568307 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc435900 con 0x557dcc4a5400
2021-06-15 18:48:52.513 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195957 ==== osd_op_reply(1568310 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dd0e3ca00 con 0x557dcc2a8800
2021-06-15 18:48:52.513 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413215 ==== osd_op_reply(1568308 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dd0e3ca00 con 0x557dcc4cc000
2021-06-15 18:48:52.513 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413216 ==== osd_op_reply(1568309 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dd0e3ca00 con 0x557dcc4cc000
2021-06-15 18:48:52.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390928 ==== osd_op_reply(1568313 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:48:52.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390929 ==== osd_op_reply(1568314 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:48:52.513 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195536 ==== osd_op_reply(1568312 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc435900 con 0x557dcc419c00
2021-06-15 18:48:52.513 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174482 ==== osd_op_reply(1568311 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcf6a9900 con 0x557dcf571400
2021-06-15 18:48:55.149 7f8ce5805700  1 -- 192.168.36.115:0/1670253716 --> [v2:192.168.36.115:6800/62151,v1:192.168.36.115:6801/62151] -- mgrreport(rgw.a +0-0 packed 294 status=1) v8 -- 0x557dcc4b1c00 con 0x557dcf661c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568315 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfd600 con 0x557dcc4a5400
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568316 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfd340 con 0x557dcc4cc000
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568317 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfdb80 con 0x557dcc4cc000
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568318 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcc48c000 con 0x557dcc2a8800
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568319 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf55fb80 con 0x557dcf571400
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568320 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf671600 con 0x557dcc419c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568321 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf671340 con 0x557dcc418c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568322 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf670840 con 0x557dcc418c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcf4f2fc0 con 0x557dcc2a8800
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dcf4f2700 con 0x557dcc418c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dcf4f2000 con 0x557dcc419c00
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dcf4f2a80 con 0x557dcf571400
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dcf4f2540 con 0x557dcc4cc000
2021-06-15 18:48:57.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dcf4f2c40 con 0x557dcc4a5400
2021-06-15 18:48:57.514 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195377 ==== osd_op_reply(1568315 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc4a6280 con 0x557dcc4a5400
2021-06-15 18:48:57.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413217 ==== osd_op_reply(1568316 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:48:57.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413218 ==== osd_op_reply(1568317 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:48:57.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195958 ==== osd_op_reply(1568318 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dcf679400 con 0x557dcc2a8800
2021-06-15 18:48:57.515 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174483 ==== osd_op_reply(1568319 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcf6a9900 con 0x557dcf571400
2021-06-15 18:48:57.515 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195537 ==== osd_op_reply(1568320 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc4a6280 con 0x557dcc419c00
2021-06-15 18:48:57.515 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390930 ==== osd_op_reply(1568321 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:48:57.515 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390931 ==== osd_op_reply(1568322 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:49:02.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568323 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf6702c0 con 0x557dcc4a5400
2021-06-15 18:49:02.514 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568324 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf671b80 con 0x557dcc4cc000
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568325 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf670000 con 0x557dcc4cc000
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568326 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf56e580 con 0x557dcc2a8800
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568327 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf56eb00 con 0x557dcf571400
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568328 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf663080 con 0x557dcc419c00
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568329 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf6638c0 con 0x557dcc418c00
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568330 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5222c0 con 0x557dcc418c00
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcf4f3180 con 0x557dcc2a8800
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dcf4f2e00 con 0x557dcc418c00
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dcf4f3500 con 0x557dcc419c00
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dcb619c00 con 0x557dcf571400
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dcb619500 con 0x557dcc4cc000
2021-06-15 18:49:02.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dcb618e00 con 0x557dcc4a5400
2021-06-15 18:49:02.515 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195378 ==== osd_op_reply(1568323 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc4a6280 con 0x557dcc4a5400
2021-06-15 18:49:02.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413219 ==== osd_op_reply(1568324 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:02.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413220 ==== osd_op_reply(1568325 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:02.515 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195959 ==== osd_op_reply(1568326 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dcf679400 con 0x557dcc2a8800
2021-06-15 18:49:02.515 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390932 ==== osd_op_reply(1568329 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:49:02.515 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195538 ==== osd_op_reply(1568328 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc4a6280 con 0x557dcc419c00
2021-06-15 18:49:02.515 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390933 ==== osd_op_reply(1568330 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcf6a9900 con 0x557dcc418c00
2021-06-15 18:49:02.516 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174484 ==== osd_op_reply(1568327 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcf6a9900 con 0x557dcf571400
2021-06-15 18:49:05.149 7f8ce5805700  1 -- 192.168.36.115:0/1670253716 --> [v2:192.168.36.115:6800/62151,v1:192.168.36.115:6801/62151] -- mgrreport(rgw.a +0-0 packed 294 status=1) v8 -- 0x557dcc4b1180 con 0x557dcf661c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568331 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd01cab00 con 0x557dcc4a5400
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568332 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf4c8580 con 0x557dcc4cc000
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568333 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61eb00 con 0x557dcc4cc000
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568334 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61e000 con 0x557dcc2a8800
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568335 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61edc0 con 0x557dcf571400
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568336 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61fb80 con 0x557dcc419c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568337 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b7340 con 0x557dcc418c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568338 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b6000 con 0x557dcc418c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcb618fc0 con 0x557dcc2a8800
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dcb619880 con 0x557dcc418c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dcb619340 con 0x557dcc419c00
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dcb619180 con 0x557dcf571400
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dcc4f6fc0 con 0x557dcc4cc000
2021-06-15 18:49:07.515 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dcc4f6380 con 0x557dcc4a5400
2021-06-15 18:49:07.516 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195379 ==== osd_op_reply(1568331 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc434000 con 0x557dcc4a5400
2021-06-15 18:49:07.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413221 ==== osd_op_reply(1568332 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:07.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413222 ==== osd_op_reply(1568333 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:07.516 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195539 ==== osd_op_reply(1568336 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc434000 con 0x557dcc419c00
2021-06-15 18:49:07.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195960 ==== osd_op_reply(1568334 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dcf679400 con 0x557dcc2a8800
2021-06-15 18:49:07.516 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174485 ==== osd_op_reply(1568335 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcc4a8280 con 0x557dcf571400
2021-06-15 18:49:07.516 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390934 ==== osd_op_reply(1568337 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcc4a8280 con 0x557dcc418c00
2021-06-15 18:49:07.516 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390935 ==== osd_op_reply(1568338 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcc4a8280 con 0x557dcc418c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568339 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf5b7080 con 0x557dcc4a5400
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568340 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf671600 con 0x557dcc4cc000
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568341 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf6702c0 con 0x557dcc4cc000
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568342 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf671080 con 0x557dcc2a8800
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568343 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf670dc0 con 0x557dcf571400
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568344 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf663080 con 0x557dcc419c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568345 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf663600 con 0x557dcc418c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568346 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd01ca840 con 0x557dcc418c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- ping magic: 0 v1 -- 0x557dcc4f6700 con 0x557dcc2a8800
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- ping magic: 0 v1 -- 0x557dcc4f61c0 con 0x557dcc418c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- ping magic: 0 v1 -- 0x557dcc4f6e00 con 0x557dcc419c00
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- ping magic: 0 v1 -- 0x557dcc4f7880 con 0x557dcf571400
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- ping magic: 0 v1 -- 0x557dcf5e6a80 con 0x557dcc4cc000
2021-06-15 18:49:12.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- ping magic: 0 v1 -- 0x557dcf5e6700 con 0x557dcc4a5400
2021-06-15 18:49:12.516 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.54 v1:192.168.36.115:6827/65464 195380 ==== osd_op_reply(1568339 notify.2 [watch ping cookie 93998786927488 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1687988650 0 0) 0x557dcc434000 con 0x557dcc4a5400
2021-06-15 18:49:12.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413223 ==== osd_op_reply(1568340 notify.4 [watch ping cookie 93998786907264 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 13415694 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:12.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.57 v1:192.168.36.105:6801/838646 413224 ==== osd_op_reply(1568341 notify.5 [watch ping cookie 93998786909184 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 2484848961 0 0) 0x557dcf679400 con 0x557dcc4cc000
2021-06-15 18:49:12.516 7f8cf181d700  1 -- 192.168.36.115:0/447973967 <== osd.64 v1:192.168.36.116:6825/3479628 195540 ==== osd_op_reply(1568344 notify.1 [watch ping cookie 93998786923008 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 108520527 0 0) 0x557dcc434000 con 0x557dcc419c00
2021-06-15 18:49:12.516 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.63 v1:192.168.36.116:6817/3478994 174486 ==== osd_op_reply(1568343 notify.6 [watch ping cookie 93998786368512 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3618329070 0 0) 0x557dcc4a8280 con 0x557dcf571400
2021-06-15 18:49:12.516 7f8cf081b700  1 -- 192.168.36.115:0/447973967 <== osd.58 v1:192.168.36.105:6809/839263 195961 ==== osd_op_reply(1568342 notify.0 [watch ping cookie 93998786921088 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3091539535 0 0) 0x557dcf679400 con 0x557dcc2a8800
2021-06-15 18:49:12.517 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390936 ==== osd_op_reply(1568345 notify.3 [watch ping cookie 93998786905344 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 3393825668 0 0) 0x557dcc4a8280 con 0x557dcc418c00
2021-06-15 18:49:12.517 7f8cf101c700  1 -- 192.168.36.115:0/447973967 <== osd.73 v1:192.168.36.117:6817/3354277 390937 ==== osd_op_reply(1568346 notify.7 [watch ping cookie 93998786367232 gen 1] v0'0 uv12 ondisk = 0) v8 ==== 152+0+0 (unknown 1216738826 0 0) 0x557dcc4a8280 con 0x557dcc418c00

====> REQUEST END <===== about 30 sec

2021-06-15 18:49:12.975 7f8ccdfd6700  1 civetweb: 0x557dcc602000: 192.168.36.115 - - [15/Jun/2021:18:48:42 +0300] "PUT /test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa HTTP/1.1" 400 426 - Boto3/1.17.35 Python/2.7.16 Darwin/20.4.0 Botocore/1.20.35

2021-06-15 18:49:13.901 7f8ce4002700  2 RGWDataChangesLog::ChangesRenewThread: start
2021-06-15 18:49:15.149 7f8ce5805700  1 -- 192.168.36.115:0/1670253716 --> [v2:192.168.36.115:6800/62151,v1:192.168.36.115:6801/62151] -- mgrreport(rgw.a +0-0 packed 294 status=1) v8 -- 0x557dcf5c4e00 con 0x557dcf661c00
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.115:6827/65464 -- osd_op(unknown.0.0:1568347 2.15 2:a93a5511:::notify.2:head [watch ping cookie 93998786927488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf520580 con 0x557dcc4a5400
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568348 2.d 2:b4812045:::notify.4:head [watch ping cookie 93998786907264 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf520dc0 con 0x557dcc4cc000
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6801/838646 -- osd_op(unknown.0.0:1568349 2.3 2:c609908c:::notify.5:head [watch ping cookie 93998786909184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dd0dfd600 con 0x557dcc4cc000
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.105:6809/839263 -- osd_op(unknown.0.0:1568350 2.1f 2:f95f44c2:::notify.0:head [watch ping cookie 93998786921088 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61e000 con 0x557dcc2a8800
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6817/3478994 -- osd_op(unknown.0.0:1568351 2.14 2:2b04a3e9:::notify.6:head [watch ping cookie 93998786368512 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf61eb00 con 0x557dcf571400
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.116:6825/3479628 -- osd_op(unknown.0.0:1568352 2.0 2:05bf5b68:::notify.1:head [watch ping cookie 93998786923008 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf4c8580 con 0x557dcc419c00
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568353 2.e 2:7759931f:::notify.3:head [watch ping cookie 93998786905344 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf56e580 con 0x557dcc418c00
2021-06-15 18:49:17.516 7f8cec012700  1 -- 192.168.36.115:0/447973967 --> v1:192.168.36.117:6817/3354277 -- osd_op(unknown.0.0:1568354 2.9 2:93e5b521:::notify.7:head [watch ping cookie 93998786367232 gen 1] snapc 0=[] ondisk+write+known_if_redirected e257) v8 -- 0x557dcf670000 con 0x557dcc418c00

haproxy access log:

{secret}:3686 [15/Jun/2021:19:02:07.599] rgw rgw/{secret} 0/0/0/1/61032 400 426 - - cD-- 1/1/0/0/0 0/0 {4||{secret}||Boto3/1.17.35 Python/2.7.16 Darwin/20.4.0 Botoco|AWS4-HMAC-SHA256 Credential={secret}/20210615/us-} {tx000000000000000001242-0060c8ceff-d7a2-default} "PUT /test1/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Actions
Copy link

Also available in: Atom PDF