Project

General

Profile

Actions
Copy link

Bug #64549

open

uncaught exception from AWSv4ComplMulti during java AWS4Test.testMultipartUploadWithPauseAWS4

Added by Casey Bodley 2 months ago. Updated 2 months ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Casey Bodley
Target version:
-
% Done:

0%

Source:
Tags:
sigv4 streaming backport_processed
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
55727
Crash signature (v1):
Crash signature (v2):

Description

from http://qa-proxy.ceph.com/teuthology/cbodley-2024-02-23_03:36:00-rgw-wip-cbodley-testing-distro-default-smithi/7571577/teuthology.log

2024-02-23T06:43:51.585 INFO:teuthology.orchestra.run.smithi142.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED
2024-02-23T06:43:51.784 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-02-23T06:43:51.784 INFO:teuthology.orchestra.run.smithi142.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED
2024-02-23T06:43:52.045 INFO:tasks.rgw.client.0.smithi142.stdout:terminate called after throwing an instance of 'std::system_error'
2024-02-23T06:43:52.045 INFO:tasks.rgw.client.0.smithi142.stdout:  what():  No such file or directory
2024-02-23T06:43:52.045 INFO:tasks.rgw.client.0.smithi142.stdout:*** Caught signal (Aborted) **
2024-02-23T06:43:52.046 INFO:tasks.rgw.client.0.smithi142.stdout: in thread 7f13e44d3640 thread_name:io_context_pool
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: ceph version 19.0.0-1668-g1fb7413e (1fb7413ec44552fe7422e97fed5ef3fe1d54b7c4) squid (dev)
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f147cc54db0]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 2: /lib64/libc.so.6(+0xa154c) [0x7f147cca154c]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 3: raise()
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 4: abort()
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 5: /lib64/libstdc++.so.6(+0xa1a01) [0x7f147d0a1a01]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 6: /lib64/libstdc++.so.6(+0xad37c) [0x7f147d0ad37c]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 7: /lib64/libstdc++.so.6(+0xad3e7) [0x7f147d0ad3e7]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 8: /lib64/libstdc++.so.6(+0xad36f) [0x7f147d0ad36f]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 9: radosgw(+0xd72fe6) [0x560774b14fe6]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 10: radosgw(+0x3da231) [0x56077417c231]
2024-02-23T06:43:52.047 INFO:tasks.rgw.client.0.smithi142.stdout: 11: /lib64/libstdc++.so.6(+0xdb924) [0x7f147d0db924]
2024-02-23T06:43:52.048 INFO:tasks.rgw.client.0.smithi142.stdout: 12: /lib64/libc.so.6(+0x9f802) [0x7f147cc9f802]
2024-02-23T06:43:52.048 INFO:tasks.rgw.client.0.smithi142.stdout: 13: /lib64/libc.so.6(+0x3f450) [0x7f147cc3f450]

   -18> 2024-02-23T06:43:52.044+0000 7f13e44d3640 16 AWSv4ComplMulti: declared signature=3d5bfd22b1cdd1bf51a2defa3a0438887d85140888579220534d7eab3621f885
AWSv4ComplMulti: calculated signature=3d5bfd22b1cdd1bf51a2defa3a0438887d85140888579220534d7eab3621f885
   -17> 2024-02-23T06:43:52.044+0000 7f13e44d3640 16 AWSv4ComplMulti: prev_chunk_signature=6a3ce38fb39e417815aa3b8c98d10f5339daa63cf55578c141876d6d9b85cac6
   -16> 2024-02-23T06:43:52.044+0000 7f13e44d3640 10 final chunk signature = 84d39cd104a69c1abf694949b8436f6fc69d3780df8a22e6ec2502ed9dc02e99
prev_chunk_signature was 3d5bfd22b1cdd1bf51a2defa3a0438887d85140888579220534d7eab3621f885
   -15> 2024-02-23T06:43:52.044+0000 7f13e44d3640  4 failed to read body: partial message
   -14> 2024-02-23T06:43:52.044+0000 7f1389c1e640 20 req 3976677133140425564 0.028999418s s3:put_obj Update meta: 7548481d-b1fb-47f2-93fa-498ec3ee60c4.4568.47__multipart_key1.2~0HiOGFamjCinPcrnHrcBcF-DGMD2_S1.meta part part.00000046 prefix key1.2~0HiOGFamjCinPcrnHrcBcF-DGMD2_S1 return 0
   -13> 2024-02-23T06:43:52.044+0000 7f1389c1e640  2 req 3976677133140425564 0.028999418s s3:put_obj completing
   -12> 2024-02-23T06:43:52.044+0000 7f1389c1e640 10 req 3976677133140425564 0.028999418s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
   -11> 2024-02-23T06:43:52.045+0000 7f13fcd04640 20 req 15763953508659836247 0.025999477s s3:put_obj Update meta: 7548481d-b1fb-47f2-93fa-498ec3ee60c4.4568.47__multipart_key1.2~0HiOGFamjCinPcrnHrcBcF-DGMD2_S1.meta part part.00000048 prefix key1.2~0HiOGFamjCinPcrnHrcBcF-DGMD2_S1 return 0
   -10> 2024-02-23T06:43:52.045+0000 7f1389c1e640  2 req 3976677133140425564 0.029999398s s3:put_obj op status=0
    -9> 2024-02-23T06:43:52.045+0000 7f13fcd04640  2 req 15763953508659836247 0.025999477s s3:put_obj completing
    -8> 2024-02-23T06:43:52.045+0000 7f1389c1e640  2 req 3976677133140425564 0.029999398s s3:put_obj http status=200
    -7> 2024-02-23T06:43:52.045+0000 7f1389c1e640  1 ====== req done req=0x7f134a0aa4a0 op status=0 http_status=200 latency=0.029999398s ======
    -6> 2024-02-23T06:43:52.045+0000 7f13fcd04640 10 req 15763953508659836247 0.025999477s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
    -5> 2024-02-23T06:43:52.045+0000 7f1389c1e640  1 beast: 0x7f134a0aa4a0: 172.21.15.142 - tester.client.0 [23/Feb/2024:06:43:52.015 +0000] "PUT /test-1d07bc81-2f52-4c45-b60f-4b88f546cd5a45/key1?uploadId=2%7E0HiOGFamjCinPcrnHrcBcF-DGMD2_S1&partNumber=46 HTTP/1.1" 200 262144 - "aws-sdk-java/1.11.549 Linux/5.14.0-419.el9.x86_64 OpenJDK_64-Bit_Server_VM/11.0.18+10-LTS java/11.0.18 vendor/Red_Hat,_Inc. com.amazonaws.services.s3.transfer.TransferManager_multipart/1.11.549" - latency=0.029999398s
    -4> 2024-02-23T06:43:52.045+0000 7f13fcd04640  2 req 15763953508659836247 0.025999477s s3:put_obj op status=0
    -3> 2024-02-23T06:43:52.045+0000 7f13fcd04640  2 req 15763953508659836247 0.025999477s s3:put_obj http status=200
    -2> 2024-02-23T06:43:52.045+0000 7f13fcd04640  1 ====== req done req=0x7f134a0294a0 op status=0 http_status=200 latency=0.025999477s ======
    -1> 2024-02-23T06:43:52.045+0000 7f13fcd04640  1 beast: 0x7f134a0294a0: 172.21.15.142 - tester.client.0 [23/Feb/2024:06:43:52.019 +0000] "PUT /test-1d07bc81-2f52-4c45-b60f-4b88f546cd5a45/key1?uploadId=2%7E0HiOGFamjCinPcrnHrcBcF-DGMD2_S1&partNumber=48 HTTP/1.1" 200 262144 - "aws-sdk-java/1.11.549 Linux/5.14.0-419.el9.x86_64 OpenJDK_64-Bit_Server_VM/11.0.18+10-LTS java/11.0.18 vendor/Red_Hat,_Inc. com.amazonaws.services.s3.transfer.TransferManager_multipart/1.11.549" - latency=0.025999477s
     0> 2024-02-23T06:43:52.047+0000 *7f13e44d3640* -1 *** Caught signal (Aborted) **
 in thread 7f13e44d3640 thread_name:io_context_pool

 ceph version 19.0.0-1668-g1fb7413e (1fb7413ec44552fe7422e97fed5ef3fe1d54b7c4) squid (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f147cc54db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f147cca154c]
 3: raise()
 4: abort()
 5: /lib64/libstdc++.so.6(+0xa1a01) [0x7f147d0a1a01]
 6: /lib64/libstdc++.so.6(+0xad37c) [0x7f147d0ad37c]
 7: /lib64/libstdc++.so.6(+0xad3e7) [0x7f147d0ad3e7]
 8: /lib64/libstdc++.so.6(+0xad36f) [0x7f147d0ad36f]
 9: radosgw(+0xd72fe6) [0x560774b14fe6]
 10: radosgw(+0x3da231) [0x56077417c231]
 11: /lib64/libstdc++.so.6(+0xdb924) [0x7f147d0db924]
 12: /lib64/libc.so.6(+0x9f802) [0x7f147cc9f802]
 13: /lib64/libc.so.6(+0x3f450) [0x7f147cc3f450]

Related issues 1 (0 open — 1 closed)

Copied to rgw - Backport #64661: squid: uncaught exception from AWSv4ComplMulti during java AWS4Test.testMultipartUploadWithPauseAWS4ResolvedCasey BodleyActions
Actions
Copy link
 #1

Updated by Casey Bodley 2 months ago 

   -15> 2024-02-23T06:43:52.044+0000 7f13e44d3640  4 failed to read body: partial message
...
2024-02-23T06:43:52.045 INFO:tasks.rgw.client.0.smithi142.stdout:terminate called after throwing an instance of 'std::system_error'
2024-02-23T06:43:52.045 INFO:tasks.rgw.client.0.smithi142.stdout:  what():  No such file or directory

this ENOENT error code actually corresponds to boost::beast::http::error::partial_message=2

according to https://stackoverflow.com/questions/66140059/why-does-boost-beast-give-me-a-partial-message-exception

This happens because the message being parsed wasn't complete. A typical reason for it is when the content-length header is wrong, or the sender abandons the connection prematurely.

Actions
Copy link
 #2

Updated by Casey Bodley 2 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Casey Bodley
  • Pull request ID set to 55727
Actions
Copy link
 #3

Updated by Casey Bodley 2 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #4

Updated by Backport Bot 2 months ago

  • Copied to Backport #64661: squid: uncaught exception from AWSv4ComplMulti during java AWS4Test.testMultipartUploadWithPauseAWS4 added
Actions
Copy link
 #5

Updated by Backport Bot 2 months ago

  • Tags changed from sigv4 streaming to sigv4 streaming backport_processed
Actions
Copy link

Also available in: Atom PDF