Bug #1656: Hadoop client unit test failures - CephFS - Ceph

Actions

Copy link

Bug #1656

closed

Hadoop client unit test failures

Added by Noah Watkins over 12 years ago. Updated about 5 years ago.

Status:

Won't Fix

Priority:

Low

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Hadoop/Java

Labels (FS):

Java/Hadoop

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The Ceph Hadoop File System passes nearly all its tests except a few. I've included the test log below that shows the failed tests. I do not have time at the moment to work on fixing these but wanted to record them some where.

Testcase: testWorkingDirectory took 0.282 sec
Testcase: testMkdirs took 0.048 sec
Testcase: testMkdirsFailsForSubdirectoryOfExistingFile took 0.043 sec
        FAILED
Should throw IOException.
junit.framework.AssertionFailedError: Should throw IOException.
        at org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile(FileSystemContractBaseTest.java:129)

Testcase: testGetFileStatusThrowsExceptionForNonExistentFile took 0.023 sec
Testcase: testListStatusReturnsNullForNonExistentFile took 0.02 sec
        Caused an ERROR
null
java.io.FileNotFoundException
        at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
        at org.apache.hadoop.fs.FileSystemContractBaseTest.testListStatusReturnsNullForNonExistentFile(FileSystemContractBaseTest.java:157)

Testcase: testListStatus took 0.045 sec
Testcase: testWriteReadAndDeleteEmptyFile took 0.036 sec
Testcase: testWriteReadAndDeleteHalfABlock took 0.038 sec
Testcase: testWriteReadAndDeleteOneBlock took 0.04 sec
Testcase: testWriteReadAndDeleteOneAndAHalfBlocks took 0.038 sec
Testcase: testWriteReadAndDeleteTwoBlocks took 0.045 sec
Testcase: testOverwrite took 0.035 sec
Testcase: testWriteInNonExistentDirectory took 0.025 sec
Testcase: testDeleteNonExistentFile took 0.014 sec
Testcase: testDeleteRecursively took 0.019 sec
Testcase: testDeleteEmptyDirectory took 0.014 sec
Testcase: testRenameNonExistentPath took 0.012 sec
Testcase: testRenameFileMoveToNonExistentDirectory took 0.015 sec
Testcase: testRenameFileMoveToExistingDirectory took 0.014 sec
Testcase: testRenameFileAsExistingFile took 0.019 sec
Testcase: testRenameFileAsExistingDirectory took 0.015 sec
Testcase: testRenameDirectoryMoveToNonExistentDirectory took 0.011 sec
Testcase: testRenameDirectoryMoveToExistingDirectory took 0.02 sec
Testcase: testRenameDirectoryAsExistingFile took 0.014 sec
Testcase: testRenameDirectoryAsExistingDirectory took 0.018 sec
Testcase: testInputStreamClosedTwice took 0.017 sec
Testcase: testOutputStreamClosedTwice took 0.012 sec

Actions

Copy link

Updated by Greg Farnum over 12 years ago

What versions of the systems were you running when these failed?

I don't remember how they're set up but they might just be invalid tests for the branch they were run on, or I might have broken something when switching back to the 0.20 branch, or some other similar problem.

Actions

Copy link

Updated by Noah Watkins over 12 years ago

This was hadoop-0.20.205.0 with the latest Ceph master branch.

It looked like the patch in src/client/hadoop was out of date with respect to your recent updates, as well as some paths that have changed.

When you originally made the patch was it against a standard release tarball, or the hadoop-common source (e.g. from git or svn)? I am just about done making a patch that applies to the standard release tarball that should work well while other cleanup efforts continue.

Actions

Copy link

Updated by Greg Farnum over 12 years ago

I believe the patch was made against the then-current svn 0.21 branch (which is now very dead). I pushed changes to the Java code into master yesterday afternoon that are the newest and most up-to-date code I've produced, which is against the 0.20 branch API (or at least my approximation of it).

You can also look at my personal git clone at https://github.com/gregsfortytwo/hadoop-common/tree/ceph-testing -- this contains the necessary Hadoop config changes (plus a couple site-specific ones on top). I don't know if this is an easier-to-digest format or not.

Actions

Copy link

Updated by Noah Watkins over 12 years ago

Alright, so I think at this point I'd like to see two patches:

1) A patch against the downloadable tarball (much easier for people setup than from source, especially for students I'm working with this quarter)
2) A patch for the source tree (hadoop-common) for upstream.

I've used your updated code in master from today to produce (1). Producing (2) is also trivial, but maybe less important in the short term as cleanup continues? Do you want to keep (1) and an updated copy of (2) in the Ceph tree for easier distribution?

One thing I am unaware of at this point is if new features are even being excepted into 0.20 branch, or if anything new must be targetted at the to-be-released 0.23 trunk.

Actions

Copy link