Skip to content

Support for RDMA Atomics + Support for user-defined initiator and responder resources on connect/accept #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Support for RDMA Atomics + Support for user-defined initiator and responder resources on connect/accept #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
14ee342
Added support for user-defined responder and initiator resources on c…
TaranovK Mar 31, 2020
0e7f403
added support for atomics and made a simple test
TaranovK Mar 31, 2020
402d8a5
added support for atomics and made a simple test
TaranovK Apr 5, 2020
f96a00c
added fully functioning query device
TaranovK Apr 6, 2020
759e282
added setting max connparams for various endpoints
TaranovK Apr 21, 2020
09a75a0
bug in old disni connect/accept.
Jun 6, 2020
642ed59
bumped the version
TaranovK Feb 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions libdisni/src/verbs/com_ibm_disni_verbs_impl_NativeDispatcher.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,41 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1connect(
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _connectV2
* Signature: (JJ)V
*/
JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1connectV2(
Copy link
Contributor

@PepperJo PepperJo Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the idea of multiple connect implementations. Why not only keep V2 and expose query_device to Java. This solves multiple problems at once: 1) setting initiator/responder resource without knowing what the max are 2) two implementations of the same function

Copy link
Author

@TaranovK TaranovK Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. However, since the libdisni is shipped separately, it may force some developers to recompile their java code which uses the library.

Copy link
Contributor

@PepperJo PepperJo Apr 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should be ok. Developers can always choose to use an old version of the library if needed.

JNIEnv *env, jobject obj, jlong id, jlong param) {
struct rdma_cm_id *cm_listen_id = NULL;
struct rdma_conn_param *conn_param = NULL;

cm_listen_id = (struct rdma_cm_id *)id;
conn_param = (struct rdma_conn_param *)param;

if (cm_listen_id != NULL && conn_param!=NULL) {
int ret = rdma_connect(cm_listen_id, conn_param);
if (ret == 0) {
log("j2c::connect: ret %i, guid %" PRIu64 "\n", ret,
ibv_get_device_guid(cm_listen_id->verbs->device));
} else {
log("j2c::connect: rdma_connect failed\n");
JNU_ThrowIOExceptionWithLastError(env,
"j2c::connect: rdma_connect failed");
}
} else {
if(cm_listen_id == NULL){
log("j2c:connect: cm_listen_id null\n");
JNU_ThrowIOException(env, "j2c:connect: cm_listen_id null\n");
} else {
log("j2c:connect: conn_param null\n");
JNU_ThrowIOException(env, "j2c:connect: conn_param null\n");
}
}
}


/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _accept
Expand Down Expand Up @@ -463,6 +498,40 @@ JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1accept(
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _acceptV2
* Signature: (JJ)V
*/
JNIEXPORT void JNICALL Java_com_ibm_disni_verbs_impl_NativeDispatcher__1acceptV2(
Copy link
Contributor

@PepperJo PepperJo Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment to connectV2

JNIEnv *env, jobject obj, jlong id, jlong param) {
struct rdma_cm_id *cm_listen_id = NULL;
struct rdma_conn_param *conn_param = NULL;

cm_listen_id = (struct rdma_cm_id *)id;
conn_param = (struct rdma_conn_param *)param;

if (cm_listen_id != NULL && conn_param!=NULL) {
int ret = rdma_accept(cm_listen_id, conn_param);
if (ret == 0) {
log("j2c::connect: ret %i, guid %" PRIu64 "\n", ret,
ibv_get_device_guid(cm_listen_id->verbs->device));
} else {
log("j2c::connect: rdma_connect failed\n");
JNU_ThrowIOExceptionWithLastError(env,
"j2c::connect: rdma_connect failed");
}
} else {
if(cm_listen_id == NULL){
log("j2c:connect: cm_listen_id null\n");
JNU_ThrowIOException(env, "j2c:connect: cm_listen_id null\n");
} else {
log("j2c:connect: conn_param null\n");
JNU_ThrowIOException(env, "j2c:connect: conn_param null\n");
}
}
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _ackCmEvent
Expand Down Expand Up @@ -823,6 +892,32 @@ Java_com_ibm_disni_verbs_impl_NativeDispatcher__1queryOdpSupport(JNIEnv *env,
return ret;
}

/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _queryDevice
* Signature: (JJ)I
*/
JNIEXPORT jint JNICALL
Java_com_ibm_disni_verbs_impl_NativeDispatcher__1queryDevice(JNIEnv *env,
jobject obj,
jlong id,
jlong addr) {
jint ret = -1;
struct ibv_context *context = (struct ibv_context *)id;

struct ibv_device_attr *dev_attr = (struct ibv_device_attr *)addr;
ret = ibv_query_device(context, dev_attr);

if(ret != 0) {
log("j2c::queryDevice: ibv_query_device failed, error %s\n",
strerror(ret));
ret = -1;
JNU_ThrowIOExceptionWithLastError(env, "j2c::queryDevice: ibv_query_device failed");
}
return ret;
}


/*
* Class: com_ibm_disni_verbs_impl_NativeDispatcher
* Method: _expPrefetchMr
Expand Down
29 changes: 29 additions & 0 deletions libdisni/src/verbs/com_ibm_disni_verbs_impl_NativeDispatcher.h
100755 → 100644
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<groupId>com.ibm.disni</groupId>
<artifactId>disni</artifactId>
<packaging>jar</packaging>
<version>2.1</version>
<version>2.2</version>
<name>disni</name>
<description>DiSNI (Direct Storage and Networking Interface) is a Java library for direct storage and networking access from userpace.</description>
<url>http://github.com/zrlio/disni</url>
Expand Down
4 changes: 4 additions & 0 deletions src/main/java/com/ibm/disni/RdmaEndpointGroup.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,10 @@ public RdmaConnParam getConnParam() {
return connParam;
}

public void setConnParam(RdmaConnParam connParam) {
this.connParam = connParam;
}

public synchronized void close() throws IOException, InterruptedException {
logger.info("shutting down group");
if (closed.get()){
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/com/ibm/disni/verbs/IbvContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,6 @@ public IbvCQ createCQ(IbvCompChannel compChannel, int ncqe, int comp_vector) thr
}

public int queryOdpSupport() throws IOException { return verbs.queryOdpSupport(this); }

public IbvDeviceAttr queryDevice() throws IOException { return verbs.queryDevice(this); }
}
Loading