Skip to content

prov/shm: make smr buffer pool allocations cache line aligned #11432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+8 −3
Open

prov/shm: make smr buffer pool allocations cache line aligned #11432

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions prov/shm/src/smr.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,11 @@
#ifndef _SMR_H_
#define _SMR_H_

/*
* Set alignment to aarch64 and x86 cache line size.
*/
#define SHM_SMR_BUFPOOL_ALIGNMENT (64)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove shm prefix

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aingerson thanks for the prompt review. I'm running some internal benchmarks, will try to collect data that I can share here.

remove shm prefix

:) I started with SMR_BUFPOOL_ALIGNMENT name and in the last minute I've added SHM prefix to make it provider specific. sure, I will remove the prefix it to match the other definitions related to smr.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snadampal SMR is the prefix for the shm provider. It is used instead of SHM to avoid any potential conflict with UNIX/Linux IPC "shm".


struct smr_env {
size_t sar_threshold;
int disable_cma;
Expand Down
6 changes: 3 additions & 3 deletions prov/shm/src/smr_ep.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1265,7 +1265,7 @@ static int smr_create_pools(struct smr_ep *ep, struct fi_info *info)
int ret;

ret = ofi_bufpool_create(&ep->cmd_ctx_pool, sizeof(struct smr_cmd_ctx),
16, 0, info->rx_attr->size,
SHM_SMR_BUFPOOL_ALIGNMENT, 0, info->rx_attr->size,
OFI_BUFPOOL_NO_TRACK);
if (ret)
goto err;
Expand All @@ -1276,13 +1276,13 @@ static int smr_create_pools(struct smr_ep *ep, struct fi_info *info)

ret = ofi_bufpool_create(&ep->unexp_buf_pool,
sizeof(struct smr_unexp_buf),
16, 0, 4, OFI_BUFPOOL_NO_TRACK);
SHM_SMR_BUFPOOL_ALIGNMENT, 0, 4, OFI_BUFPOOL_NO_TRACK);
if (ret)
goto free2;

ret = ofi_bufpool_create(&ep->pend_buf_pool,
sizeof(struct smr_pend_entry),
16, 0, 4, OFI_BUFPOOL_NO_TRACK);
SHM_SMR_BUFPOOL_ALIGNMENT, 0, 4, OFI_BUFPOOL_NO_TRACK);
if (ret)
goto free1;

Expand Down