ch4/progress: fine-tune progress throttle #7638
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The effect of tuning `MPIR_CVAR_CH4_PROGRESS_THROTTLE_MIN_PROCS` using `usleep` as the throttling method. At above 89 processes, the time of alltoallv quickly increases due to progress starvation.
|
Sometime, especially during troubleshooting, we may get confused. Print MPICH version string with MPIR_CVAR_DEBUG_SUMMARY as a simple way to verify.
When the no_progress_counter reach MPIR_CVAR_CH4_PROGRESS_THROTTLE_NO_PROGRESS_COUNT, it enters the throttle state and it should remain in this state untile progress are made. Resetting the counter removes the process from throttle when the need to throttle still exist.
hzhou
force-pushed
the
2510_progress_throttle
branch
from
66d1393 to
f78a270
Compare
Only perform progress throttle when minimum of MPIR_CVAR_CH4_PROGRESS_THROTTLE_MIN_PROCS enter the throttle state.
Add alternative method to throttle, e.g. sched_yield or PAUSE, and use MPIR_CVAR_CH4_PROGRESS_THROTTLE_NUM_PAUSES to control the amount of throttle.
hzhou
force-pushed
the
2510_progress_throttle
branch
from
f78a270 to
c41032d
Compare
|
Small comment on |
Good point |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Some system may encounter progress starvation when multiple processes on a node enter busy polling progress loop and NIC is unable to update the event due to PCIe atomic contention issue. Previously, we added a progress throttle using
usleep. It is effective but may be too blunt to negatively impact normal performance.This PR adds a few more knobs to fine-tune the progress throttle.
MPIR_CVAR_CH4_PROGRESS_THROTTLE_MIN_PROCSto only throttle when sufficient number of processes enter the throttle stateMPIR_CVAR_CH4_PROGRESS_THROTTLE_NUM_PAUSESif we choose to usethread_yield()in stead ofusleep.[skip warnings]
Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.