Cleanup #814
closedhadoop: refactor hadoop shim in terms of java libceph bindings
Added by Sage Weil about 13 years ago. Updated about 5 years ago.
0%
Description
Refactor the hadoop code in terms of generic Java bindings for libceph, instead of mixing the SWIG crap in with the actual functionality.
Updated by Greg Farnum about 13 years ago
I'm not sure what you're after here -- you mean you want Java bindings for librados, and then the Hadoop patches should be updated to use those?
It's been a while, but from what I recall I was deliberately pretty careful to place Ceph functionality in the C code as much as possible so that we could update the interface and stuff without updating the Hadoop code (under the hope that it would eventually get merged into their codebase where we'd have more trouble getting at it). I suspect that maintaining the CephFS' ambivalence about how anything in libceph is actually used is something we want to preserve.
Updated by Sage Weil about 13 years ago
I mean that, at the end of the day, we should probably have:
- a java Ceph interface binding that is identical to the interface exposed by libceph (something that looks more or less like posix, with whatever changes/additions make sense from a generic utility point of view). This should be useful to any Java thing that wants to interact with a ceph file system (not librados).
- a hadoop class that uses the generic java Ceph bindings, and any hadoop-specific weirdness.
Updated by Noah Watkins about 13 years ago
Sage,
I see the general benefits of a Java-based libceph interface, but what are your long-term plans for Hadoop over Ceph? This sounds like there is room for a version that is not built on top of the kernel client, but the kernel client version is quite stable, and the tiny shim for locality information is fully integrated with the existing Hadoop native-code library.
Thanks,
Noah
Updated by Sage Weil about 13 years ago
- Translation missing: en.field_position set to 531
Updated by Sage Weil about 13 years ago
- Target version changed from v0.26 to 12
- Translation missing: en.field_position deleted (
531) - Translation missing: en.field_position set to 10
Updated by Sage Weil about 13 years ago
- Translation missing: en.field_position deleted (
44) - Translation missing: en.field_position set to 1
- Translation missing: en.field_position changed from 1 to 644
Updated by Sage Weil over 12 years ago
- Target version changed from v0.37 to v0.36
Updated by Sage Weil over 12 years ago
- Target version changed from v0.36 to v0.37
- Translation missing: en.field_position deleted (
762) - Translation missing: en.field_position set to 1
Updated by Sage Weil over 12 years ago
- Target version deleted (
v0.37) - Translation missing: en.field_position deleted (
4) - Translation missing: en.field_position set to 2
Updated by Sage Weil over 12 years ago
- Translation missing: en.field_position deleted (
12) - Translation missing: en.field_position set to 34
Updated by Sage Weil over 12 years ago
- Translation missing: en.field_position deleted (
34) - Translation missing: en.field_position set to 14
Updated by Sage Weil over 12 years ago
Updated by Sage Weil almost 12 years ago
- Status changed from New to In Progress
- Translation missing: en.field_position deleted (
336) - Translation missing: en.field_position set to 335
Updated by Sage Weil over 11 years ago
- Project changed from Ceph to CephFS
- Category deleted (
20)
Updated by Sage Weil over 11 years ago
- Translation missing: en.field_position deleted (
463) - Translation missing: en.field_position set to 1
Updated by Sage Weil over 11 years ago
- Target version set to v0.54b
- Translation missing: en.field_position deleted (
1) - Translation missing: en.field_position set to 3
Updated by Sage Weil over 11 years ago
- Status changed from In Progress to Resolved
- Translation missing: en.field_position deleted (
12) - Translation missing: en.field_position set to 12
Updated by Patrick Donnelly about 5 years ago
- Category deleted (
48) - Labels (FS) Java/Hadoop added