Performance of llama.cpp on AMD ROCm (HIP) #15021
Replies: 19 comments 34 replies
-
RX 7800 XT (Sapphire Pulse 280W)ggml_cuda_init: found 1 ROCm devices:
build: 00131d6 (6031) ggml_vulkan: Found 1 Vulkan devices:
build: baad948 (6056) Notes:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: e725a1a (6104) |
Beta Was this translation helpful? Give feedback.
-
|
Happy to replicate: ggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) On Linux |
Beta Was this translation helpful? Give feedback.
-
RX 7600 XTggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) Running on Linux 6.12.32, mainline amdgpu, ROCm 6.4.1. ggml_vulkan: Found 1 Vulkan devices:
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov , the 7600 XT actually has a 128 bit memory bus. |
Beta Was this translation helpful? Give feedback.
-
|
AMD MI60. Happy to contribute.
I will post FA=1 and vulkan results once I have time during the weekend. |
Beta Was this translation helpful? Give feedback.
-
MI100Using ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -sm none -mg 0
build: 9c35706 (6060) I'm running Ubuntu 24.04.2 and ROCm 6.4.1 |
Beta Was this translation helpful? Give feedback.
-
|
I expected it to be faster than RX 7800 XT because of HBM2... Have you tried to launch with a single device only? |
Beta Was this translation helpful? Give feedback.
-
|
Bandwidth utilization is still fairly low on the gcn/cdna parts (gcn/cdna = same thing for tg). |
Beta Was this translation helpful? Give feedback.
-
AMD Instinct MI300Xroot@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 2bf3fbf (6069) Ref: #14640 |
Beta Was this translation helpful? Give feedback.
-
|
I'm just referring to the rocWMMA flag from the build instructions: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip
It should work for CDNA too but we have only tested with our RDNA3 cards (7900 XTX) and saw huge performance jumps in PP with FA on: #10879 (reply in thread) Please try it out because 1/3rd the performance in PP with FA on is just... strange at best |
Beta Was this translation helpful? Give feedback.
-
|
With root@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: ee3a9fc (6090) |
Beta Was this translation helpful? Give feedback.
-
|
Performance is in the middle between RTX 4090 and RTX 5090. |
Beta Was this translation helpful? Give feedback.
-
|
So rocWMMA does work for CDNA in FA :) |
Beta Was this translation helpful? Give feedback.
-
|
not very well |
Beta Was this translation helpful? Give feedback.
-
Pro V620Why does FA slow down the V620 so much? Been a question I've been trying to answer for a while now.
build: 03d4698 (6074) Linux, ROCm 6.4.1 ( will try upgrading soon) |
Beta Was this translation helpful? Give feedback.
-
|
@samteezy Can you please run per each device PRO V620/Pro WX 3200 and ROCm only backend? |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov The numbers come out the same. Forcing root@llama:~# /root/llama-builds/llama.cpp/bin/llama-bench -m /mnt/models/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -mg 0 -sm none
build: 5c0eb5e (6075) |
Beta Was this translation helpful? Give feedback.
-
Powercolor Hellhound RX 7900 XTX (400W power limit)Opensuse tumbleweed system with rocm packages from
build: 5c0eb5e (6075) Sapphire Nitro 7900 XTX (400W power limit)In a different PC unfortunately because these GPUs are too chonky to fit in a regular case
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
Possibly. Any given Nitro+ should beat any given Red Devil by about 4% if all things were perfect. Red Devil has the second highest tier of power settings in the firmware, Nitro+ has the highest tier, and these two cards have the best stock air coolers and usually decent factory paste jobs; any given Nitro+ should beat any given Hellhound by about 6%. That Nitro+ is 3.9% faster than the Hellhound, that seems okay. My Red Devil is 4.7% faster than the Nitro+, that seems weird. |
Beta Was this translation helpful? Give feedback.
-
|
@rohan-sircar The power limit that slider controls is PPT, it is hard limited to +15% by the firmware on the card. Only the Aqua OC can reach 400W+15%=460W. The Red Devil is 315W+15%=362W, the Nitro+ is 339W+15%=389W, the Merc 310 is 327W+15%=376W, the Hellhound apparently is 303W+15%=348W. So, I'm reasonably sure you're not setting what you think you're setting. I can't go to 400W, and neither can you (not without hardware mods, at least). |
Beta Was this translation helpful? Give feedback.
-
|
BTW, a friend of mine that uses Linux recommended https://github.com/ilya-zlobintsev/LACT and has confirmed that it shows PPT/TGP on that slider. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I'm familiar with lact! Infact that's what I use to make the power limit increase persist between reboots. Here's a screenshot from it: screenshot from nvtop: output of It is kinda rare to push it hard enough to reach those power levels though, here's the reading while benching Qwen3 32B. It doesn't reach this level while benching the llama2 7B so my hypothesis was wrong anyway I guess... Sorry if I'm misunderstanding/misinterpreting things! |
Beta Was this translation helpful? Give feedback.
-
|
Actual power usage becomes a different problem on RDNA3. My card does not reach maximum power while running llama.cpp, and seems to be held back by VRAM bandwidth. If you want to see actual max power usage, go try throwing a simple test like Furmark at it and see what you reach. For example: At +15%, I have a PPT limit of 362W, with Furmark running, I can reach a PPT/TGP of 362W, and a TBP of 432W. llama.cpp peaks at 362W PPT/TGP, but spends most of the benchmark around 310W PPT/TGP. |
Beta Was this translation helpful? Give feedback.
-
Powercolor Red Devil 7900XTXAdrenalin 25.8.1 just came out, so time to test again
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
build: 2572689 (6099) Still lower than the historical highs on May 26th (3599 and 3743), and a loss and a win against July 22nd (3529 and 3598). |
Beta Was this translation helpful? Give feedback.
-
|
RX 7900 XTX (ASUS TUF)
build: 6c7e9a5 (6118) |
Beta Was this translation helpful? Give feedback.
-
RX 6800 (16GB 203W)ROCm 6.3.4 on Ubuntu 24.04 in a Docker container
build: 79c1160 (6123) Bonus benchmarksI ran these to compare ROCm versions on various models. Obviously the results are specific to my RX 6800 and shouldn't be used to make any judgments about ROCm performance in general, especially on RDNA3 and later gpus. I use 6.3.4 because I don't care about LLama 3 8B. Note how fast the new MoE models are - gpt-oss-20B even at Q6_K_XL is faster than this 7B Q4_0 model. (Do make sure that you have a fixed version because the original gpt-oss releases had some issues - I used https://huggingface.co/unsloth/gpt-oss-20b-GGUF). ROCm 6.3.4
ROCm 6.4.3
|
Beta Was this translation helpful? Give feedback.
-
|
RX 7900 XTX (ASUS TUF a bit overclocked for 100 mhz for core and VRAM) ./build/bin/llama-bench -m /home/vk/Downloads/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 648ebcd (6146) |
Beta Was this translation helpful? Give feedback.
-
|
RX 6900 XT AMD Reference Card (Stock clocks) Debian Testing llama.cpp version: gguf-v0.17.1-386-gfd1234cb
|
Beta Was this translation helpful? Give feedback.
-
|
@tdjb Results are pretty low, can you re-test using llama.cpp standalone without Docker? |
Beta Was this translation helpful? Give feedback.
-
|
Just a quick test, installed the llama.cpp build from Debian sid (was surprised to even find a build to be honest), which appears to be b5882 and the results came in quite similar. I tried the benchmark on both of my devices, as one is on a slower PCIe 4x slot, the results below are from the faster run. Why do you think the 6900 XT should perform better?
Happy to run further tests. |
Beta Was this translation helpful? Give feedback.
-
|
It should be 10% better on my understanding, according to specs: RX 6900 XT and RX 6800 XT |
Beta Was this translation helpful? Give feedback.
-
|
the rdna2 results are mostly surprisingly high, given the hardware capabilities, not low. |
Beta Was this translation helpful? Give feedback.
-
|
GigaByte R9700
|
Beta Was this translation helpful? Give feedback.
-
|
So 7900 XTX has better performance, that's sad. Also weird that Vulkan performs way worse on pp than ROCm but better on tg. |
Beta Was this translation helpful? Give feedback.
-
|
We'll see when someone a bit more experienced gives it a shot. My benchmarks are about as vanilla as it gets. Threw it in an unraid server (12700k and 128GB DDR4-2133), made docker images and ran benchmarks. Many of the 7900 XTX results are baremetal, have factory overclock or are manually overclocked, installed additional drivers, and/or have raised power limits. I bet someone will beat my benchmarks shortly. |
Beta Was this translation helpful? Give feedback.
-
there is zero reason to expect 9070(xt) to perform better than the xtx |
Beta Was this translation helpful? Give feedback.
-
Radeon RX 9070 (non-XT)
build: 65349f2 (6183) I tried to enable the use of rocwmma with Still surprising that these numbers are better than the 9070 XT. |
Beta Was this translation helpful? Give feedback.
-
|
EDIT: see my comment below I bought 2x MI50 32GB VRAM from Alibaba and for some reason I'm getting really poor performance on them... No idea why, even Vega 64 beats it and it's way slower than someone's elses MI50 16GB.
build: 21c17b5 (3)
build: 21c17b5 (3) |
Beta Was this translation helpful? Give feedback.
-
|
Found issue, my Vega RX 64 was crashing/locking up and that's why I had disabled power management
build: 21c17b5 (3)
build: 21c17b5 (3) But weird thing is that now my Vega RX 64 with "not disabled power management" performs worse... |
Beta Was this translation helpful? Give feedback.
-
|
Can you please run long one? It is interesting if large memory can help with degradation. |
Beta Was this translation helpful? Give feedback.
-
|
I have dissembled my system so I won't be able to test it for a while. But someone else from MI50 Discord run it |
Beta Was this translation helpful? Give feedback.
-
|
it is interesting how Vulkan out performs ROCm by a large margin when flash attention is on. FA is a big gain, so ig i'll be running vulkan over ROCm lol. I was looking to buy 2 (maybe 3) mi50's bc hardware specs seem really good on paper for such a low price, but from all of the benchmarks I've seen it seems like the software just isn't there yet to really squeeze out all the performance it has to offer. seems like 20% is still left on the table. Hopefully the support only gets better, but they are old cards, so i doubt it. |
Beta Was this translation helpful? Give feedback.
-
RX 9070 XT (Powercolor Red Devil)OS: Ubuntu 24.04 ROCm 6.4.3
build: 043fb27 (6264) Vulkan
build: c9a24fb (6262) |
Beta Was this translation helpful? Give feedback.
-
|
Vulkan with FA enabled has impressive pp and TG gains. What version of vulkan are you using? I see it is radv but is this the latest vulkan driver? I haven't seen such a speed up in MI50 cards with vulkan yet. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
This is becasue wmma fattn is currently disabled on gfx12 until rocm 7.0 (or when force enabled) |
Beta Was this translation helpful? Give feedback.
-
I installed the AMD drivers at the same time I installed ROCm 6.4.3 |
Beta Was this translation helpful? Give feedback.
-
|
With rocwmma_fattn, thermals at 50C
build: d82f6aa (6321) |
Beta Was this translation helpful? Give feedback.
-
|
With rocwmma_fattn, thermals at 51.9C
build: 9c979f865 (6248) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Performance of llama.cpp on Apple Silicon M-series, Performance of llama.cpp on Nvidia CUDA and Performance of llama.cpp with Vulkan, but for ROCm! I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our ROCm(HIP) releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and ROCm info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device I'll prioritize newer commits with substantial ROCm updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
ROCm Scoreboard for Llama 2 7B, Q4_0 (no FA)
ROCm Scoreboard for Llama 2 7B, Q4_0 (with FA)
More detailed test
The main idea of this test is to show a decrease in performance with increasing size.
Beta Was this translation helpful? Give feedback.
All reactions