Project

General

Profile

Cleanup #814

hadoop: refactor hadoop shim in terms of java libceph bindings

Added by Sage Weil almost 11 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Tags:
Backport:
Reviewed:
Affected Versions:
Component(FS):
Hadoop/Java
Labels (FS):
Java/Hadoop
Pull request ID:

Description

Refactor the hadoop code in terms of generic Java bindings for libceph, instead of mixing the SWIG crap in with the actual functionality.

History

#1 Updated by Greg Farnum almost 11 years ago

I'm not sure what you're after here -- you mean you want Java bindings for librados, and then the Hadoop patches should be updated to use those?

It's been a while, but from what I recall I was deliberately pretty careful to place Ceph functionality in the C code as much as possible so that we could update the interface and stuff without updating the Hadoop code (under the hope that it would eventually get merged into their codebase where we'd have more trouble getting at it). I suspect that maintaining the CephFS' ambivalence about how anything in libceph is actually used is something we want to preserve.

#2 Updated by Sage Weil almost 11 years ago

I mean that, at the end of the day, we should probably have:
- a java Ceph interface binding that is identical to the interface exposed by libceph (something that looks more or less like posix, with whatever changes/additions make sense from a generic utility point of view). This should be useful to any Java thing that wants to interact with a ceph file system (not librados).
- a hadoop class that uses the generic java Ceph bindings, and any hadoop-specific weirdness.

#3 Updated by Noah Watkins almost 11 years ago

Sage,

I see the general benefits of a Java-based libceph interface, but what are your long-term plans for Hadoop over Ceph? This sounds like there is room for a version that is not built on top of the kernel client, but the kernel client version is quite stable, and the tiny shim for locality information is fully integrated with the existing Hadoop native-code library.

Thanks,
Noah

#4 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_position set to 531

#5 Updated by Sage Weil almost 11 years ago

  • Target version changed from v0.26 to 12
  • translation missing: en.field_position deleted (531)
  • translation missing: en.field_position set to 10

#6 Updated by Sage Weil over 10 years ago

  • translation missing: en.field_position deleted (44)
  • translation missing: en.field_position set to 1
  • translation missing: en.field_position changed from 1 to 644

#7 Updated by Sage Weil over 10 years ago

  • Target version changed from 12 to 19

#8 Updated by Sage Weil over 10 years ago

  • Target version changed from 19 to v0.37

#9 Updated by Sage Weil over 10 years ago

  • Target version changed from v0.37 to v0.36

#10 Updated by Sage Weil over 10 years ago

  • Target version changed from v0.36 to v0.37
  • translation missing: en.field_position deleted (762)
  • translation missing: en.field_position set to 1

#11 Updated by Sage Weil over 10 years ago

  • Target version deleted (v0.37)
  • translation missing: en.field_position deleted (4)
  • translation missing: en.field_position set to 2

#12 Updated by Sage Weil over 10 years ago

  • translation missing: en.field_position deleted (12)
  • translation missing: en.field_position set to 34

#13 Updated by Sage Weil over 10 years ago

  • translation missing: en.field_position deleted (34)
  • translation missing: en.field_position set to 14

#15 Updated by Sage Weil over 9 years ago

  • Status changed from New to In Progress
  • translation missing: en.field_position deleted (336)
  • translation missing: en.field_position set to 335

#16 Updated by Sage Weil over 9 years ago

  • Project changed from Ceph to CephFS
  • Category deleted (20)

#17 Updated by Sage Weil over 9 years ago

  • translation missing: en.field_position deleted (463)
  • translation missing: en.field_position set to 1

#18 Updated by Sage Weil over 9 years ago

  • Category set to 48

#19 Updated by Sage Weil over 9 years ago

  • Target version set to v0.54b
  • translation missing: en.field_position deleted (1)
  • translation missing: en.field_position set to 3

#20 Updated by Sage Weil about 9 years ago

  • Status changed from In Progress to Resolved
  • translation missing: en.field_position deleted (12)
  • translation missing: en.field_position set to 12

#21 Updated by Greg Farnum over 5 years ago

  • Component(FS) Hadoop/Java added

#22 Updated by Patrick Donnelly almost 3 years ago

  • Category deleted (48)
  • Labels (FS) Java/Hadoop added

Also available in: Atom PDF