Potential hangup on AMD gpus with SDMA

[MartinGirard]

I've been running simulations on MI300A GPUs. Occasionally, simulations simply hangup with no errors, sometimes after 10s of millions of MD steps. Attaching a debugger shows that it is stuck in a synchronization, i.e. waiting for device to finish.

My compute center has advised to set HSA_ENABLE_SDMA=0. The small tests I ran seem to indicate that this solves the problem, but since the bug is stochastic, this makes diagnostics very complicated.

Has this issue been seen elsewhere?

[joaander]

We have not seen this problem on Frontier.