Bug #1656
closed
Hadoop client unit test failures
Added by Noah Watkins over 12 years ago.
Updated about 5 years ago.
Component(FS):
Hadoop/Java
Description
The Ceph Hadoop File System passes nearly all its tests except a few. I've included the test log below that shows the failed tests. I do not have time at the moment to work on fixing these but wanted to record them some where.
Testcase: testWorkingDirectory took 0.282 sec
Testcase: testMkdirs took 0.048 sec
Testcase: testMkdirsFailsForSubdirectoryOfExistingFile took 0.043 sec
FAILED
Should throw IOException.
junit.framework.AssertionFailedError: Should throw IOException.
at org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile(FileSystemContractBaseTest.java:129)
Testcase: testGetFileStatusThrowsExceptionForNonExistentFile took 0.023 sec
Testcase: testListStatusReturnsNullForNonExistentFile took 0.02 sec
Caused an ERROR
null
java.io.FileNotFoundException
at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
at org.apache.hadoop.fs.FileSystemContractBaseTest.testListStatusReturnsNullForNonExistentFile(FileSystemContractBaseTest.java:157)
Testcase: testListStatus took 0.045 sec
Testcase: testWriteReadAndDeleteEmptyFile took 0.036 sec
Testcase: testWriteReadAndDeleteHalfABlock took 0.038 sec
Testcase: testWriteReadAndDeleteOneBlock took 0.04 sec
Testcase: testWriteReadAndDeleteOneAndAHalfBlocks took 0.038 sec
Testcase: testWriteReadAndDeleteTwoBlocks took 0.045 sec
Testcase: testOverwrite took 0.035 sec
Testcase: testWriteInNonExistentDirectory took 0.025 sec
Testcase: testDeleteNonExistentFile took 0.014 sec
Testcase: testDeleteRecursively took 0.019 sec
Testcase: testDeleteEmptyDirectory took 0.014 sec
Testcase: testRenameNonExistentPath took 0.012 sec
Testcase: testRenameFileMoveToNonExistentDirectory took 0.015 sec
Testcase: testRenameFileMoveToExistingDirectory took 0.014 sec
Testcase: testRenameFileAsExistingFile took 0.019 sec
Testcase: testRenameFileAsExistingDirectory took 0.015 sec
Testcase: testRenameDirectoryMoveToNonExistentDirectory took 0.011 sec
Testcase: testRenameDirectoryMoveToExistingDirectory took 0.02 sec
Testcase: testRenameDirectoryAsExistingFile took 0.014 sec
Testcase: testRenameDirectoryAsExistingDirectory took 0.018 sec
Testcase: testInputStreamClosedTwice took 0.017 sec
Testcase: testOutputStreamClosedTwice took 0.012 sec
What versions of the systems were you running when these failed?
I don't remember how they're set up but they might just be invalid tests for the branch they were run on, or I might have broken something when switching back to the 0.20 branch, or some other similar problem.
This was hadoop-0.20.205.0 with the latest Ceph master branch.
It looked like the patch in src/client/hadoop was out of date with respect to your recent updates, as well as some paths that have changed.
When you originally made the patch was it against a standard release tarball, or the hadoop-common source (e.g. from git or svn)? I am just about done making a patch that applies to the standard release tarball that should work well while other cleanup efforts continue.
I believe the patch was made against the then-current svn 0.21 branch (which is now very dead). I pushed changes to the Java code into master yesterday afternoon that are the newest and most up-to-date code I've produced, which is against the 0.20 branch API (or at least my approximation of it).
You can also look at my personal git clone at https://github.com/gregsfortytwo/hadoop-common/tree/ceph-testing -- this contains the necessary Hadoop config changes (plus a couple site-specific ones on top). I don't know if this is an easier-to-digest format or not.
Alright, so I think at this point I'd like to see two patches:
1) A patch against the downloadable tarball (much easier for people setup than from source, especially for students I'm working with this quarter)
2) A patch for the source tree (hadoop-common) for upstream.
I've used your updated code in master from today to produce (1). Producing (2) is also trivial, but maybe less important in the short term as cleanup continues? Do you want to keep (1) and an updated copy of (2) in the Ceph tree for easier distribution?
One thing I am unaware of at this point is if new features are even being excepted into 0.20 branch, or if anything new must be targetted at the to-be-released 0.23 trunk.
Sounds good to me -- which patches we want to keep in the tree are probably a management decision but I'm happy to put them in. :)
If you're not going to use (2) let's not bother until we see what changes we make based on your feedback and our own plans going forward.
- Translation missing: en.field_position set to 29
- Translation missing: en.field_position deleted (
29)
- Translation missing: en.field_position set to 14
- Translation missing: en.field_position deleted (
16)
- Translation missing: en.field_position set to 1
- Translation missing: en.field_story_points set to 1
- Translation missing: en.field_position deleted (
1)
- Translation missing: en.field_position set to 1
- Translation missing: en.field_position deleted (
1)
- Translation missing: en.field_position set to 18
- Translation missing: en.field_position deleted (
14)
- Translation missing: en.field_position set to 30
- Project changed from Ceph to CephFS
- Category deleted (
20)
- Component(FS) Hadoop/Java added
- Priority changed from Normal to Low
- Status changed from New to Won't Fix
I'm told that there is a newer hdfs test suite that we would adopt if refreshing the hdfs support, so this ticket is no longer relevant.
- Category deleted (
48)
- Labels (FS) Java/Hadoop added
Also available in: Atom
PDF