ROCm + MIOpen integration

[mihirparadkar]

As in #173 , it would be great to support non-NVIDIA graphics hardware to the best of its abilities. For AMD GPUs, the official libraries for deep learning computation are ROCm + MIOpen, which have APIs deliberately designed to be similar to CUDA and CuDNN respectively.

Furthermore, MIOpen also has an OpenCL backend for AMD hardware https://github.com/ROCmSoftwarePlatform/MIOpen

[ViralBShah]

Is there an actual use case you have on AMD hardware, or this is more of a generally good thing to have? I think the latter is always good, but curious to see who's using AMD and other hardware and what for.

[davidbp]

I feel most people is using Nvidia for the lack of software, not because AMD hardware is not capable.
A simple benchmark:

http://blog.gpueater.com/en/2018/03/07/00003_tech_flops_benchmark_1/

Allowing Flux users to use different GPU hardware would be a huge addition (most libraries don't allow it). A nice usecase would be software meant to run in realtime in embedded devices (realtime object detection for example). AMD apus are quite capable (but developing code that takes advantage of integrated GPUs is quite costly at the moment).

[vchuravy]

A ROCm codegen backend ala CUDAnative.jl is feasible, in the meantime ClArrays uses a transpiling approach to OpenCL https://github.com/JuliaGPU/CLArrays.jl, but NNLib.jl is missing integration with MIOpen.

This definitively falls under help would be appreciated.

[MikeInnes]

It would probably be fairly easy to prototype something simple in the vein of Flux's current CUDNN wrappers; e.g. wait for CLArrays to be loaded, check if MIOpen is available, and define some methods to ccall it. Defining something simple like softmax should be straightforward and I'd be very happy to see a PR :)

[mihirparadkar]

I tried to prototype on one of GPUEater's Radeon Vega 56 machines, but it seemed to have issues loading the OpenCL driver. A few minutes of debugging and searching suggests that this is related to issues with ROCm itself on these GPUs, so I'll put this on hold for now.

[MikeInnes]

Closing in favour of #173.

[jpsamaroo]

Now that Julia's AMDGPU/ROCm/HSA support is (mostly) functional [1][2], having Flux support ROCm+MIOpen directly is a very real possibility. I personally hope to have Flux working before the end of the summer if all goes well.

[1] https://github.com/JuliaGPU/AMDGPUnative.jl
[2] https://github.com/jpsamaroo/HSARuntime.jl

[elouie]

How is this progress? A lot of students in ML programs have Macbooks with AMD GPUs that support ROCm, but can't use parallelization on their devices due to lack of support outside Keras.

[vchuravy]

Macbooks with AMD GPUs that support ROCm

Mac sadly doesn't support ROCm and has deprecated support for OpenCL. Keras seems to be using PlaidML which in turn uses OpenCL, but given that Apple has discontinued support for OpenCL we don't have a huge incentive to work on supporting it.