Bug #40170
openradosgw response 500 error while putting object
0%
Description
ceph -v
ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable)
2019-06-05 14:33:09.562 7fc7c0a23700 20 CONTENT_LENGTH=4194304
2019-06-05 14:33:09.562 7fc7c0a23700 20 CONTENT_TYPE=image/jpeg
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_ACCEPT=*/*
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_AUTHORIZATION=AWS CKYNA0RVPZ430QW688DB:Zdxoxg6xhftHwol21ze27da9pLY=
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_DATE=Wed, 05 Jun 2019 06:33:09 GMT
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_EXPECT=100-continue
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_HOST=nru-hangye.server02
2019-06-05 14:33:09.562 7fc7c0a23700 20 HTTP_VERSION=1.1
2019-06-05 14:33:09.562 7fc7c0a23700 20 REMOTE_ADDR=172.16.18.213
2019-06-05 14:33:09.562 7fc7c0a23700 20 REQUEST_METHOD=PUT
2019-06-05 14:33:09.562 7fc7c0a23700 20 REQUEST_URI=/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 20 SCRIPT_URI=/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 20 SERVER_PORT=7480
2019-06-05 14:33:09.562 7fc7c0a23700 1 ====== starting new request req=0x7fc7c0a1cc80 =====
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s::PUT /12370800010000019401/49735::initializing for trans_id = tx00000000000000000000e-005cf76225-213bfe-jinhua-zone3
2019-06-05 14:33:09.562 7fc7c0a23700 10 rgw api priority: s3=5 s3website=4
2019-06-05 14:33:09.562 7fc7c0a23700 10 host=nru-hangye.server02
2019-06-05 14:33:09.562 7fc7c0a23700 20 subdomain=nru-hangye domain=server02 in_hosted_domain=1 in_hosted_domain_s3website=0
2019-06-05 14:33:09.562 7fc7c0a23700 20 final domain/bucket subdomain=nru-hangye domain=server02 in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=server02 s->info.request_uri=/nru-hangye/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2019-06-05 14:33:09.562 7fc7c0a23700 10 handler=22RGWHandler_REST_Obj_S3
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735::getting op 1
2019-06-05 14:33:09.562 7fc7c0a23700 10 op=21RGWPutObj_ObjStore_S3
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:verifying requester
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2019-06-05 14:33:09.562 7fc7c0a23700 10 get_canon_resource(): dest=/nru-hangye/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 10 string_to_sign:
PUT
image/jpeg
Wed, 05 Jun 2019 06:33:09 GMT
/nru-hangye/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 15 string_to_sign=PUT
image/jpeg
Wed, 05 Jun 2019 06:33:09 GMT
/nru-hangye/12370800010000019401/49735
2019-06-05 14:33:09.562 7fc7c0a23700 15 server signature=Zdxoxg6xhftHwol21ze27da9pLY=
2019-06-05 14:33:09.562 7fc7c0a23700 15 client signature=Zdxoxg6xhftHwol21ze27da9pLY=
2019-06-05 14:33:09.562 7fc7c0a23700 15 compare=0
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::s3::LocalEngine granted access
2019-06-05 14:33:09.562 7fc7c0a23700 20 rgw::auth::s3::AWSAuthStrategy granted access
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:normalizing buckets and tenants
2019-06-05 14:33:09.562 7fc7c0a23700 10 s->object=12370800010000019401/49735 s->bucket=nru-hangye
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:init permissions
2019-06-05 14:33:09.562 7fc7c0a23700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>jh-hangye</ID><DisplayName>jh-hangye</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>jh-hangye</ID><DisplayName>jh-hangye</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:recalculating target
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:reading permissions
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:init op
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:verifying op mask
2019-06-05 14:33:09.562 7fc7c0a23700 20 required_mask= 2 user.op_mask=7
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:verifying op permissions
2019-06-05 14:33:09.562 7fc7c0a23700 20 -- Getting permissions begin with perm_mask=50
2019-06-05 14:33:09.562 7fc7c0a23700 5 Searching permissions for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=jh-hangye, acct_name=jh-hangye, subuser=, perm_mask=15, is_admin=0) mask=50 Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=jh-hangye, acct_name=jh-hangye, subuser=, perm_mask=15, is_admin=0), owner=jh-hangye, perm=2
2019-06-05 14:33:09.562 7fc7c0a23700 5 Searching permissions for uid=jh-hangye
2019-06-05 14:33:09.562 7fc7c0a23700 5 Found permission: 15
2019-06-05 14:33:09.562 7fc7c0a23700 5 Searching permissions for group=1 mask=50
2019-06-05 14:33:09.562 7fc7c0a23700 5 Found permission: 1
2019-06-05 14:33:09.562 7fc7c0a23700 5 Searching permissions for group=2 mask=50
2019-06-05 14:33:09.562 7fc7c0a23700 5 Permissions for group not found
2019-06-05 14:33:09.562 7fc7c0a23700 5 -
2019-06-05 14:33:09.562 7fc7c0a23700 10 identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=jh-hangye, acct_name=jh-hangye, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:verifying op params
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:pre-executing
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:executing
2019-06-05 14:33:09.562 7fc7c0a23700 20 required alignment=12288
2019-06-05 14:33:09.562 7fc7c0a23700 20 max_chunk_size=4190208
2019-06-05 14:33:09.562 7fc7c0a23700 20 get_data() returned ret=-5
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:completing
2019-06-05 14:33:09.562 7fc7c0a23700 0 WARNING: set_req_state_err err_no=5 resorting to 500
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:op status=-5
2019-06-05 14:33:09.562 7fc7c0a23700 2 req 14:0s:s3:PUT /12370800010000019401/49735:put_obj:http status=500
2019-06-05 14:33:09.562 7fc7c0a23700 1 ====== req done req=0x7fc7c0a1cc80 op status=-5 http_status=500 ======
2019-06-05 14:33:09.562 7fc7c0a23700 20 process_request() returned -5
2019-06-05 14:33:09.562 7fc7c0a23700 1 civetweb: 0x55a9fec18000: 172.16.18.213 - - [05/Jun/2019:14:33:09 +0800] "PUT /12370800010000019401/49735 HTTP/1.1" 500 491 - -
Updated by Casey Bodley almost 5 years ago
- Status changed from New to Need More Info
What s3 client are you using? Is this happening for all PUT requests? The "20 get_data() returned ret=-5" error implies that civetweb failed trying to read data from the socket.
Updated by liang sibin almost 5 years ago
Casey Bodley wrote:
What s3 client are you using? Is this happening for all PUT requests? The "20 get_data() returned ret=-5" error implies that civetweb failed trying to read data from the socket.
i use cpp client, this is happening for all PUT requests
Updated by Casey Bodley almost 5 years ago
liang sibin wrote:
Casey Bodley wrote:
What s3 client are you using? Is this happening for all PUT requests? The "20 get_data() returned ret=-5" error implies that civetweb failed trying to read data from the socket.
i use cpp client, this is happening for all PUT requests
do you see similar issues with other s3 clients? it sounds like your c++ code may be closing the socket after sending headers?