Project

General

Profile

Actions
Copy link

Bug #64841

open

java_s3tests: testObjectCreateBadExpectMismatch failure

Added by Casey Bodley about 2 months ago. Updated 4 days ago.

Status:
Triaged
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
java 100-continue
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ex. http://qa-proxy.ceph.com/teuthology/cbodley-2024-03-10_14:50:40-rgw-wip-cbodley2-testing-distro-default-smithi/7589576/teuthology.log

2024-03-10T16:40:31.654 INFO:teuthology.orchestra.run.smithi060.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch STARTED
2024-03-10T16:40:32.455 INFO:teuthology.orchestra.run.smithi060.stdout:
2024-03-10T16:40:32.455 INFO:teuthology.orchestra.run.smithi060.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch FAILED
2024-03-10T16:40:32.455 INFO:teuthology.orchestra.run.smithi060.stdout:    com.amazonaws.services.s3.model.AmazonS3Exception at ObjectTest.java:525

scanning the rgw log http://qa-proxy.ceph.com/teuthology/cbodley-2024-03-10_14:50:40-rgw-wip-cbodley2-testing-distro-default-smithi/7589576/remote/smithi060/log/rgw.ceph.client.0.log.gz i see this sequence of requests:

2024-03-10T16:40:31.653+0000 CreateBucket test-c2dd4bb1-7d2b-431d-a7b0-200c96d8349313
2024-03-10T16:40:32.445+0000 status ok

2024-03-10T16:40:33.409+0000 ListObjectVersions
2024-03-10T16:40:34.465+0000 status ok

2024-03-10T16:40:34.469+0000 a4d1c640  1 failed to read header: bad method

2024-03-10T16:40:34.885+0000 ListObjectVersions
2024-03-10T16:40:35.645+0000 status ok

2024-03-10T16:40:35.649+0000 DeleteBucket

that "bad method" error corresponds to the test's PutObject request that's supposed to pass `Expect: 200`. the "bad method" error comes from beast's http parser when it's trying to read the next request from a connection. i assume this means we either read too many bytes from the previous request, or too few

there are several other occurrances of this "bad method" error, but none led to test failures. scanning the rgw log of successful java_s3tests runs, i don't see any "bad method" errors

Related issues 1 (0 open — 1 closed)

Related to rgw - Bug #58286: Subsequent request fails after PutObject to non-existing bucketResolvedCasey Bodley

Actions
Actions
Copy link
 #1

Updated by Casey Bodley about 1 month ago

saw a different java test fail in https://qa-proxy.ceph.com/teuthology/cbodley-2024-03-19_01:03:50-rgw-wip-cbodley-testing-distro-default-smithi/7610160/teuthology.log:

suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 FAILED
    com.amazonaws.services.s3.model.AmazonS3Exception at AWS4Test.java:168

that rgw log also contains the failed to read header: bad method errors

Actions
Copy link
 #2

Updated by Casey Bodley about 1 month ago

another in https://qa-proxy.ceph.com/teuthology/cbodley-2024-03-19_16:47:35-rgw-wip-rgw-account-topic-distro-default-smithi/7610962/teuthology.log

suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 FAILED
    com.amazonaws.services.s3.model.AmazonS3Exception at AWS4Test.java:168

Actions
Copy link
 #3

Updated by Casey Bodley about 1 month ago

  • Priority changed from Urgent to Immediate
Actions
Copy link
 #4

Updated by Ali Maredia about 1 month ago

After doing 10 runs locally of the java s3-tests I am not able to reproduce this issue.

- Ali

Actions
Copy link
 #5

Updated by Casey Bodley about 1 month ago

  • Assignee set to Ali Maredia
Actions
Copy link
 #6

Updated by Ali Maredia 20 days ago

After several dozen runs of java-s3tests locally I was able to see what Casey was seeing in his logs in one run. The same "bad method" error happened twice after 2 ListBuckets calls early on in a run of the tests and then never again.

In all the runs I did locally there were no test failures.

Actions
Copy link
 #7

Updated by Ali Maredia 10 days ago

After having radosgw under valgrind and running the java s3tests I was able to reproduce the "failed to read header: bad method" issue in the radosgw log.

All of the instances of this in my log happened after put requests that returned 404s. There were only 6 instances of this in the whole log of an entire java s3tests run. No failures were seen.

Actions
Copy link
 #8

Updated by Ali Maredia 10 days ago

Here is a snippet with two of those "bad method" statements from the log I referenced in the last comment.

https://paste.sh/2VisQRhp#v0EBF8pNjGjrSrK_WeZ9vreA

Actions
Copy link
 #9

Updated by Casey Bodley 10 days ago

thanks Ali, that's super helpful. i came across https://tracker.ceph.com/issues/58286 which looks like the exact same thing - a failed to read header: bad method error after a PutObject request with Expect: 100-continue that fails with 404

https://tracker.ceph.com/issues/58286 was fixed for reef, i wonder why it's come back

Actions
Copy link
 #10

Updated by Casey Bodley 10 days ago

  • Related to Bug #58286: Subsequent request fails after PutObject to non-existing bucket added
Actions
Copy link
 #11

Updated by Casey Bodley 10 days ago

i tried running the python reproducer from https://tracker.ceph.com/issues/58286, but it doesn't reproduce the bad method error any more. so that's fixed, but apparently java-s3tests is doing something similar but different

Actions
Copy link
 #12

Updated by Casey Bodley 4 days ago

  • Status changed from New to Triaged
  • Assignee deleted (Ali Maredia)
  • Priority changed from Immediate to High
  • Tags changed from java to java 100-continue
Actions
Copy link

Also available in: Atom PDF