Cherry-pick ROCm/rocm-systems@65b769e #2374
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This patch fixes a deadlock in ActiveSignal when profiling and graph packet batching is enabled, that is introduced after ROCm/rocm-systems#1354.
Technical Details
Added a timestamp check to the signal insertion logic. Previously, a new profiling signal was only created when the existing signal was still active (signal > 0). Added a second condition signal_list_[temp_id]->ts_ == ts to handle the case where the slot is already associated with the current timestamp.
Test Plan
Tested with the pytorch job from https://ontrack-internal.amd.com/browse/SWDEV-569101 and verified that it resolves the hang
Test Result
Resolved hang in the reproducer job from https://ontrack-internal.amd.com/browse/SWDEV-569101
Submission Checklist