Project

General

Profile

Actions
Copy link

Bug #23531

closed

s3a/2.8.0 fs.contract failure Seek/Rename/ComplexDirActions/RecursiveRootListing/EmptyRootDirNonRecursive

Added by Vasu Kulkarni about 6 years ago. Updated over 4 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
Matt Benjamin
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Failing tests are. from fs.contract:
Seek/Rename/ComplexDirActions/RecursiveRootListing/EmptyRootDirNonRecursive/ListEmptyRootDir

Ignore failure on HugeFile/Scale tests, I think we need to increase timeout and I have updated the script to 1200 seconds

Logs: http://qa-proxy.ceph.com/teuthology/vasu-2018-03-30_16:12:32-rgw-luminous-distro-basic-ovh/2336681/teuthology.log

a)

2018-03-30T16:54:39.117 INFO:teuthology.orchestra.run.ovh061.stdout:testSeekFile(org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek) Time elapsed: 0.66 sec <<< ERROR!
2018-03-30T16:54:39.117 INFO:teuthology.orchestra.run.ovh061.stdout:java.io.FileNotFoundException: Reopen at position 63 on s3a://s3atest/fork-1/test/seekfile.txt: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: tx00000000000000000061a-005abe6bcd-1078-default), S3 Extended Request ID: 1078-default-default

b)
org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
2018-03-30T16:54:56.150 INFO:teuthology.orchestra.run.ovh061.stdout:testRenameWithNonEmptySubDir(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename) Time elapsed: 6.632 sec <<< ERROR!

c)

2018-03-30T16:55:05.387 INFO:teuthology.orchestra.run.ovh061.stdout:Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 38.095 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
2018-03-30T16:55:05.387 INFO:teuthology.orchestra.run.ovh061.stdout:testComplexDirActions(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus) Time elapsed: 11.405 sec <<< FAILURE!
2018-03-30T16:55:05.387 INFO:teuthology.orchestra.run.ovh061.stdout:java.lang.AssertionError: listStatus(): file count in 1 directory and 0 files expected:<4> but was:<0>

d)
org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir

2018-03-30T16:55:09.770 INFO:teuthology.orchestra.run.ovh061.stdout:testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 6.972 sec  <<< FAILURE!

e)
2018-03-30T16:55:09.773 INFO:teuthology.orchestra.run.ovh061.stdout:testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 1.075 sec  <<< FAILURE!

f)
2018-03-30T16:55:09.776 INFO:teuthology.orchestra.run.ovh061.stdout:testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 28.363 sec  <<< FAILURE!
Actions
Copy link
 #1

Updated by Casey Bodley about 6 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #2

Updated by Matt Benjamin about 6 years ago

  • Status changed from New to In Progress
  • Assignee set to Matt Benjamin
Actions
Copy link
 #3

Updated by Casey Bodley about 6 years ago

  • Related to Bug #22986: hadoop-s3a jobs failing with EPERM added
Actions
Copy link
 #4

Updated by Yehuda Sadeh almost 6 years ago

Casey, Vasu, can this one be closed?

Actions
Copy link
 #5

Updated by Vasu Kulkarni almost 6 years ago

  • Related to deleted (Bug #22986: hadoop-s3a jobs failing with EPERM)
Actions
Copy link
 #6

Updated by Vasu Kulkarni almost 6 years ago

I think the related link was for different reason, this is still a genuine issue with s3a tests.

Actions
Copy link
 #7

Updated by Casey Bodley about 5 years ago

  • Assignee changed from Matt Benjamin to Pritha Srivastava
Actions
Copy link
 #8

Updated by Pritha Srivastava about 5 years ago

I ran the following tests on hadoop master against ceph master and all of them passed:
mvn -Dtest=ITestS3AContractSeek test
mvn -Dtest=ITestS3AContractRename test
mvn -Dtest=ITestS3AContractRootDir test

Actions
Copy link
 #9

Updated by Pritha Srivastava about 5 years ago

An update: mvn -Dtest=ITestS3AContractGetFileStatus#testComplexDirActions test fails, looking into that.

Actions
Copy link
 #10

Updated by Pritha Srivastava about 5 years ago

Pritha Srivastava wrote:

An update: mvn -Dtest=ITestS3AContractGetFileStatus#testComplexDirActions test fails, looking into that.

The test fails because by default a call is made to listObjectsv2 which is not implemented in RGW currently. So setting the list version to 1 makes the test pass. This test can be re-run once we have listObjectsV2 in our code base.

We need to set the following property in hadoop/hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml:

<property>
<name>fs.s3a.list.version</name>
<value>1</value>
</property>

Actions
Copy link
 #11

Updated by Matt Benjamin about 5 years ago

vasu, is this effectively an indication of a bug in our upstream site-config.xml?

Matt

Actions
Copy link
 #12

Updated by Casey Bodley about 5 years ago

  • Assignee changed from Pritha Srivastava to Vasu Kulkarni

looking for confirmation that this is fixed upstream - if so, can we update the version that's run in teuthology?

Actions
Copy link
 #13

Updated by Matt Benjamin almost 5 years ago

  • Assignee changed from Vasu Kulkarni to Matt Benjamin
Actions
Copy link
 #14

Updated by Casey Bodley over 4 years ago

  • Status changed from In Progress to Won't Fix
Actions
Copy link

Also available in: Atom PDF