Performance of llama.cpp on AMD ROCm (HIP) #15021
Replies: 10 comments 19 replies
-
RX 7800 XT (Sapphire Pulse 280W)ggml_cuda_init: found 1 ROCm devices:
build: 00131d6 (6031) ggml_vulkan: Found 1 Vulkan devices:
build: baad948 (6056) Notes:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: e725a1a (6104) |
Beta Was this translation helpful? Give feedback.
-
|
Happy to replicate: ggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) On Linux |
Beta Was this translation helpful? Give feedback.
-
RX 7600 XTggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) Running on Linux 6.12.32, mainline amdgpu, ROCm 6.4.1. ggml_vulkan: Found 1 Vulkan devices:
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov , the 7600 XT actually has a 128 bit memory bus. |
Beta Was this translation helpful? Give feedback.
-
|
AMD MI60. Happy to contribute.
I will post FA=1 and vulkan results once I have time during the weekend. |
Beta Was this translation helpful? Give feedback.
-
MI100Using ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -sm none -mg 0
build: 9c35706 (6060) I'm running Ubuntu 24.04.2 and ROCm 6.4.1 |
Beta Was this translation helpful? Give feedback.
-
|
I expected it to be faster than RX 7800 XT because of HBM2... Have you tried to launch with a single device only? |
Beta Was this translation helpful? Give feedback.
-
AMD Instinct MI300Xroot@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 2bf3fbf (6069) Ref: #14640 |
Beta Was this translation helpful? Give feedback.
-
|
I'm just referring to the rocWMMA flag from the build instructions: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip
It should work for CDNA too but we have only tested with our RDNA3 cards (7900 XTX) and saw huge performance jumps in PP with FA on: #10879 (reply in thread) Please try it out because 1/3rd the performance in PP with FA on is just... strange at best |
Beta Was this translation helpful? Give feedback.
-
|
With root@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: ee3a9fc (6090) |
Beta Was this translation helpful? Give feedback.
-
|
Performance is in the middle between RTX 4090 and RTX 5090. |
Beta Was this translation helpful? Give feedback.
-
|
So rocWMMA does work for CDNA in FA :) |
Beta Was this translation helpful? Give feedback.
-
|
not very well |
Beta Was this translation helpful? Give feedback.
-
Pro V620Why does FA slow down the V620 so much? Been a question I've been trying to answer for a while now.
build: 03d4698 (6074) Linux, ROCm 6.4.1 ( will try upgrading soon) |
Beta Was this translation helpful? Give feedback.
-
|
@samteezy Can you please run per each device PRO V620/Pro WX 3200 and ROCm only backend? |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov The numbers come out the same. Forcing root@llama:~# /root/llama-builds/llama.cpp/bin/llama-bench -m /mnt/models/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -mg 0 -sm none
build: 5c0eb5e (6075) |
Beta Was this translation helpful? Give feedback.
-
Powercolor Hellhound RX 7900 XTX (400W power limit)Opensuse tumbleweed system with rocm packages from
build: 5c0eb5e (6075) Sapphire Nitro 7900 XTX (400W power limit)In a different PC unfortunately because these GPUs are too chonky to fit in a regular case
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
Possibly. Any given Nitro+ should beat any given Red Devil by about 4% if all things were perfect. Red Devil has the second highest tier of power settings in the firmware, Nitro+ has the highest tier, and these two cards have the best stock air coolers and usually decent factory paste jobs; any given Nitro+ should beat any given Hellhound by about 6%. That Nitro+ is 3.9% faster than the Hellhound, that seems okay. My Red Devil is 4.7% faster than the Nitro+, that seems weird. |
Beta Was this translation helpful? Give feedback.
-
|
@rohan-sircar The power limit that slider controls is PPT, it is hard limited to +15% by the firmware on the card. Only the Aqua OC can reach 400W+15%=460W. The Red Devil is 315W+15%=362W, the Nitro+ is 339W+15%=389W, the Merc 310 is 327W+15%=376W, the Hellhound apparently is 303W+15%=348W. So, I'm reasonably sure you're not setting what you think you're setting. I can't go to 400W, and neither can you (not without hardware mods, at least). |
Beta Was this translation helpful? Give feedback.
-
|
BTW, a friend of mine that uses Linux recommended https://github.com/ilya-zlobintsev/LACT and has confirmed that it shows PPT/TGP on that slider. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I'm familiar with lact! Infact that's what I use to make the power limit increase persist between reboots. Here's a screenshot from it: screenshot from nvtop: output of It is kinda rare to push it hard enough to reach those power levels though, here's the reading while benching Qwen3 32B. It doesn't reach this level while benching the llama2 7B so my hypothesis was wrong anyway I guess... Sorry if I'm misunderstanding/misinterpreting things! |
Beta Was this translation helpful? Give feedback.
-
|
Actual power usage becomes a different problem on RDNA3. My card does not reach maximum power while running llama.cpp, and seems to be held back by VRAM bandwidth. If you want to see actual max power usage, go try throwing a simple test like Furmark at it and see what you reach. For example: At +15%, I have a PPT limit of 362W, with Furmark running, I can reach a PPT/TGP of 362W, and a TBP of 432W. llama.cpp peaks at 362W PPT/TGP, but spends most of the benchmark around 310W PPT/TGP. |
Beta Was this translation helpful? Give feedback.
-
Powercolor Red Devil 7900XTXAdrenalin 25.8.1 just came out, so time to test again
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
build: 2572689 (6099) Still lower than the historical highs on May 26th (3599 and 3743), and a loss and a win against July 22nd (3529 and 3598). |
Beta Was this translation helpful? Give feedback.
-
|
RX 7900 XTX (ASUS TUF)
build: 6c7e9a5 (6118) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Performance of llama.cpp on Apple Silicon M-series, Performance of llama.cpp on Nvidia CUDA and Performance of llama.cpp with Vulkan, but for ROCm! I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our ROCm(HIP) releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and ROCm info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device I'll prioritize newer commits with substantial ROCm updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
ROCm Scoreboard for Llama 2 7B, Q4_0 (no FA)
ROCm Scoreboard for Llama 2 7B, Q4_0 (with FA)
More detailed test
The main idea of this test is to show a decrease in performance with increasing size.
Beta Was this translation helpful? Give feedback.
All reactions