Project

General

Profile

Feature #48513

uses librgw2 to directly access the rados cluster for hadoop

Added by chunsong feng about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

Hadoop S3 accesses Ceph through the object gateway. The radosgw daemon process needs to be deployed. Messages are sent from Hadoop to radosgw and then to the rados cluster. In the new solution, Hadoop uses librgw2 to directly access the rados cluster, avoiding radosgw forwarding and improving performance.

History

#1 Updated by chunsong feng about 1 month ago

The adapter invokes the librgw2 interface through JNI. The following interfaces need to be implemented:
static const JNINativeMethod gMethods[] = { {"native_initialize", "()V", (void*)native_initialize}, {"native_ceph_create", "(Ljava/lang/String;)I", (void*)native_ceph_create}, {"native_ceph_mount",
"(Lcom/ceph/rgw/CephRgwAdapter;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)J",
(void*)native_ceph_mount}, {"native_ceph_unmount", "(J)I", (void*)native_ceph_unmount}, {"native_ceph_release", "(J)I", (void*)native_ceph_release}, {"native_ceph_statfs", "(JJLjava/lang/String;Lcom/ceph/rgw/CephStatVFS;)I", (void*)native_ceph_statfs}, {"native_ceph_listdir",
"(JJLjava/lang/String;Lcom/ceph/rgw/CephRgwAdapter$ListDirHandler;)I",
(void*)native_ceph_listdir}, {"native_ceph_unlink", "(JJLjava/lang/String;)I", (void*)native_ceph_unlink}, {"native_ceph_rename",
"(JJLjava/lang/String;Ljava/lang/String;JLjava/lang/String;Ljava/lang/String;)I",
(void*)native_ceph_rename}, {"native_ceph_mkdirs", "(JJLjava/lang/String;Ljava/lang/String;I)Z", (void*)native_ceph_mkdirs}, {"native_ceph_lstat", "(JJLjava/lang/String;Lcom/ceph/rgw/CephStat;)I", (void*)native_ceph_lstat}, {"native_ceph_setattr", "(JJLjava/lang/String;Lcom/ceph/rgw/CephStat;I)I", (void*)native_ceph_setattr}, {"native_ceph_open", "(JJLjava/lang/String;II)J", (void*)native_ceph_open}, {"native_ceph_close", "(JJ)I", (void*)native_ceph_close}, {"native_ceph_read", "(JJJ[BJ)J", (void*)native_ceph_read}, {"native_ceph_write", "(JJJ[BJ)J", (void*)native_ceph_write}, {"native_ceph_fsync", "(JJZ)I", (void*)native_ceph_fsync},
};

#2 Updated by Igor Fedotov about 1 month ago

  • Project changed from bluestore to rgw

Also available in: Atom PDF