cjw/amd-r9700-ai-toolboxes
1
0
Fork 0
Code Issues Pull Requests Actions 6 Packages Projects Releases Wiki Activity

docs: streamline README with a concise quick start guide and remove detailed usage details.

Browse Source
This commit is contained in:
Donato Capitella
2025-12-06 10:29:56 +00:00
parent cef614d895
commit a179482018
3 changed files with 90 additions and 575 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+90 -292
View File
@@ -1,348 +1,146 @@
# AMD R9700 Llama.cpp Toolboxes
This project provides pre-built containers (“toolboxes”) for running LLMs on **AMD Radeon AI PRO R9700** GPUs (gfx1201). Toolbx is the standard developer container system in Fedora (and now works on Ubuntu, openSUSE, Arch, etc).
This project provides pre-built containers (“toolboxes”) for running LLMs on **AMD Radeon AI PRO R9700** GPUs (gfx1201). It uses `toolbox` (standard on Fedora, available on Ubuntu, Arch, etc.) to run `llama.cpp` with full GPU acceleration (Vulkan or ROCm) without messing up your host system.
## 🚨 Updates — 2025-11-30
## 🚀 Quick Start
- Released new toolboxes for ROCm 7 that track the nightly builds, these are now called `nightly`.
- Updated and extended benchmakrs across all llama.cpp backend configurations, and included bennchmarks over RPC (two nodes) and long context (32k) -> [Interactive Benchmark Viewer](https://kyuz0.github.io/amd-r9700-ai-toolboxes/)
### 1. Create a Toolbox
**Which backend to choose?**
* **Vulkan (RADV)**: Recommended for **stability**. It works reliably with almost all models.
* **ROCm**: Recommended for **maximum performance**.
* *Note*: Multiple ROCm versions are available (e.g., 6.4.4, 7.1, 7.9). Performance can vary significantly depending on the model architecture (e.g., Llama vs. Qwen). **Check the [Benchmarks](https://kyuz0.github.io/amd-r9700-ai-toolboxes/)** to find the best version for your model.
## Table of Contents
- [Quick Answers (Read This First)](#quick-answers-read-this-first)
1. [Llama.cpp Compiled for Every Backend](#1-llamacpp-compiled-for-every-backend)
1.1 [Supported Container Images](#11-supported-container-images)
2. [Quickest Usage Example](#2-quickest-usage-example)
2.1 [Creating the toolboxes with GPU access](#21-creating-the-toolboxes-with-gpu-access)
2.2 [Running models inside the toolboxes](#22-running-models-inside-the-toolboxes)
2.3 [Downloading GGUF Models from HuggingFace](#23-downloading-gguf-models-from-huggingface)
3. [Performance Benchmarks](#3-performance-benchmarks)
4. [Memory Planning & VRAM Estimator](#4-memory-planning--vram-estimator)
5. [Building Containers Locally](#5-building-containers-locally)
6. [Host Configuration](#6-host-configuration)
6.1 [Test Configuration](#61-test-configuration)
6.2 [Kernel Parameters (tested on Fedora 42)](#62-kernel-parameters-tested-on-fedora-42)
6.3 [Ubuntu 24.04](#63-ubuntu-2404)
7. [More Documentation](#7-more-documentation)
8. [References](#8-references)
## Quick Answers (Read This First)
### How do I get a toolbox up and running?
**Command — Create Vulkan (RADV) toolbox**
```sh
**Option A: Vulkan (RADV) [Recommended]**
```bash
toolbox create llama-vulkan-radv \
--image docker.io/kyuz0/amd-r9700-toolboxes:vulkan-radv \
-- --device /dev/dri --group-add video --security-opt seccomp=unconfined
```
**Command — Create ROCm toolbox (6.4.4/7.1/7.9/7-nightly)**
```sh
**Option B: ROCm (7.1)**
```bash
toolbox create llama-rocm-7.1-rocwmma \
--image docker.io/kyuz0/amd-r9700-toolboxes:rocm-7.1-rocwmma \
-- --device /dev/dri --device /dev/kfd \
--group-add video --group-add render --group-add sudo --security-opt seccomp=unconfined
```
**Command — Enter the toolbox shell**
> **Ubuntu Users**: `toolbox` may have issues with GPU access. Use [Distrobox](https://github.com/89luca89/distrobox) instead. See [Detailed Guide](#ubuntu-users-distrobox) below.
```sh
### 2. Enter the Toolbox
```bash
toolbox enter llama-vulkan-radv
# or: toolbox enter llama-rocm-7.1-rocwmma
```
**Command — List detected GPUs (inside the toolbox)**
### 3. Download a Model
```sh
llama-cli --list-devices
```
### How do I download weights for a model?
**Command — Download a GGUF shard from Hugging Face**
**Option A: Manual Download (Recommended)**
Use the `hf` tool to download the model GGUF files to a local directory.
```bash
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
# Download to models/qwen3-coder-30B-A3B/
HF_HUB_ENABLE_HF_TRANSFER=1 hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
--local-dir models/qwen3-coder-30B-A3B/
--local-dir .
```
`HF_HUB_ENABLE_HF_TRANSFER=1` turns on the Rust-based accelerated downloader (`pip install hf-transfer`).
**Multi-shard Models:**
If a model is split into multiple files (e.g., `00001-of-00005.gguf`), you must download **all** shards to the same folder. The command above ensures all parts are downloaded.
### How do I run llama-server (and llama-cli) with a model?
> [!NOTE]
> The old `huggingface-cli` is deprecated. Use the modern `hf` tool (part of `huggingface_hub`).
Flash attention and no-memory-map **must** be enabled or R9700 will crawl/crash.
**Command — Run llama-server with flash attention + no-mmap**
```sh
llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
-c 8192 -ngl 999 -fa 1 --no-mmap
```
**Command — Run llama-cli with the same essentials**
```sh
llama-cli --no-mmap -ngl 999 -fa 1 -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
-p "Write a R9700 toolkit haiku."
```
### How do I keep the toolboxes updated?
**Command — Refresh every toolbox**
**Option B: Automatic Download (via llama.cpp)**
`llama.cpp` can automatically download models from the Hugging Face Hub to its internal cache (`~/.cache/huggingface/hub`).
```bash
./refresh-toolboxes.sh all
# Automatically download and run
llama-cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF -hf-file BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf ...
```
*Note: We prefer Option A (dedicated folder) to keep things organized, but Option B is great for quick tests.*
**Command — Refresh specific toolboxes**
### 4. Run a Model
> [!TIP]
> You should **always** use `-fa 1` (Flash Attention). This significantly improves performance and memory utilization on the R9700.
Use **`llama-cli`** for running models directly in your terminal—ideal for quick tests, benchmarking, or chatting without leaving the shell.
Use **`llama-server`** to start an OpenAI-compatible API server. This allows you to connect third-party UIs (like Open WebUI), use the built-in web interface, or build your own applications using standard libraries.
**Run it (CLI Chat):**
```bash
./refresh-toolboxes.sh llama-vulkan-radv llama-rocm-7.1-rocwmma
```
## 1. Llama.cpp Compiled for Every Backend
This project uses [Llama.cpp](https://github.com/ggerganov/llama.cpp), a high-performance inference engine for running local LLMs (large language models) on CPUs and GPUs. Llama.cpp is open source, extremely fast, and is the only engine supporting all key backends for AMD R9700: Vulkan (RADV, AMDVLK) and ROCm/HIP
* **Vulkan** is a cross-platform, low-level graphics and compute API. Llama.cpp can use Vulkan for GPU inference with either the open Mesa RADV driver or AMD's "official" open AMDVLK driver. This is the most stable and supported option for AMD CPUs at the moment.
* **ROCm** is AMD's open-source answer to CUDA: a GPU compute stack for machine learning and HPC. With ROCm, you can run Llama.cpp on AMD GPUs in a way similar to how CUDA works on NVIDIA - this is not the most stable/mature, but recently it's been getting better.
### 1.1 Supported Container Images
You can check the containers on DockerHub: https://hub.docker.com/r/kyuz0/amd-r9700-toolboxes/tags.
| Container Tag | Backend/Stack | Purpose / Notes |
| ------------------------------ | -------------------------------------- | --------------- |
| `vulkan-amdvlk` | Vulkan (AMDVLK) | Fastest backend—AMD open-source driver. ≤2 GiB single buffer allocation limit, some large models won't load. |
| `vulkan-radv` | Vulkan (Mesa RADV) | Most stable and compatible. Recommended for most users and all models. |
| `rocm-6.4.4` | ROCm 6.4.4 (HIP) + hipBLASLt* | Latest stable build for ROCm 6.4.4, performs very well with most model architectures/quants. |
| `rocm-6.4.4-rocwmma` | ROCm 6.4.4 + ROCWMMA + hipBLASLt* | 6.4.4 with ROCWMMA enabled for better flash attention on RDNA3+/CDNA. |
| `rocm-7.1` | ROCm 7.1 GA (HIP) + hipBLASLt* | Current GA release for ROCm 7.x; improved scheduler and hipBLASLt kernels. |
| `rocm-7.1-rocwmma` | ROCm 7.1 GA + ROCWMMA + hipBLASLt* | 7.1 with ROCWMMA for maximum flash-attention throughput. |
| `rocm-7.9` | ROCm 7.9 (HIP) + hipBLASLt* | Used to be the release candidate for ROCm 7.9.0 (hence the `rc` tag in the name), now released. |
| `rocm-7.9-rocwmma` | ROCm 7.9 + ROCWMMA + hipBLASLt* | 7.9.0 build with ROCWMMA—useful for early flash-attention validation. |
| `rocm-7-nightly` | ROCm 7 Nightly (“7rc-alpha”) + hipBLASLt* | Tracks ROCm 7 nightly (alpha) preview with bleeding-edge patches. |
| `rocm-7-nightly-rocwmma` | ROCm 7 Nightly + ROCWMMA + hipBLASLt* | Same nightly/alpha stack with ROCWMMA tuned for flash attention. |
\* All these toolboxes export `ROCBLAS_USE_HIPBLASLT=1` because it historically delivered better performance and stability, altough this might not be the case any more.
> These containers are **automatically** rebuilt whenever the Llama.cpp master branch is updated, ensuring you get the latest bug fixes and new model support. The easiest way to update to the newest versions is by running the `refresh-toolboxes.sh` [script below](#211-toolbox-refresh-script-automatic-updates).
>
> Legacy images `rocm-6.4.2` and `rocm-6.4.3` are still on Docker Hub for reproducibility but are intentionally excluded from the active list above. Prefer `rocm-6.4.4+` or any `rocm-7.x` tag unless you must bisect an old regression.
---
## 2. Quickest Usage Example
### 2.1 Creating the toolboxes with GPU access
To use Llama.cpp with hardware acceleration inside a toolbox container, you must expose the right GPU device nodes from your host. The exact flags depend on the backend.
#### Command — Create Vulkan (RADV/AMDVLK) toolbox
```sh
toolbox create llama-vulkan-radv \
--image docker.io/kyuz0/amd-r9700-toolboxes:vulkan-radv \
-- --device /dev/dri --group-add video --security-opt seccomp=unconfined
```
*Only `/dev/dri` is required for Vulkan. Make sure your user is in the `video` group.*
#### Command — Create ROCm toolbox (swap the tag for 6.4.4, 7.1, 7.9, 7-nightly…)
```sh
toolbox create llama-rocm-7.1-rocwmma \
--image docker.io/kyuz0/amd-r9700-toolboxes:rocm-7.1-rocwmma \
-- --device /dev/dri --device /dev/kfd \
--group-add video --group-add render --group-add sudo --security-opt seccomp=unconfined
```
*ROCm needs both `/dev/dri` and `/dev/kfd`, plus the `video`, `render`, and sometimes `sudo` groups for full compute access. Swap `rocm-7.1-rocwmma` for any other active ROCm tag (6.4.4, 7.9, 7-nightly, etc.).*
> **Note:**
>
> * `--device /dev/dri` provides graphics/video device nodes.
> * `--device /dev/kfd` is required for ROCm compute.
> * Extra groups (`video`, `render`, `sudo`) may be required for full access to GPU nodes and compute features, especially with ROCm.
> * Use `--security-opt seccomp=unconfined` to avoid seccomp sandbox issues (needed for some GPU syscalls).
### 2.1.1 Ubuntu users
Ubuntu’s `toolbox` package still breaks GPU access, so follow gyhor’s [issue comment](https://github.com/kyuz0/amd-r9700-toolboxes/issues/16#issuecomment-3582028864) and use [Distrobox](https://github.com/89luca89/distrobox) instead:
```sh
distrobox create -n llama-rocm-7.1 \
--image docker.io/kyuz0/amd-r9700-toolboxes:rocm-7.1-rocwmma \
--additional-flags "--device /dev/kfd --device /dev/dri --group-add video --group-add render --security-opt seccomp=unconfined"
distrobox enter llama-rocm-7.1
llama-cli --list-devices
```
### 2.1.2 Toolbox Refresh Script (Automatic Updates)
To pull the latest container images and recreate toolboxes cleanly, use the provided script:
#### 📦 `refresh-toolboxes.sh`
```bash
./refresh-toolboxes.sh all
```
This will:
1. Delete existing toolboxes (if any)
2. Pull the latest images from DockerHub
3. Recreate each toolbox with correct GPU access flags
You can also refresh just one or more toolboxes:
```bash
./refresh-toolboxes.sh llama-vulkan-radv llama-rocm-7.1-rocwmma
```
### 2.2 Running models inside the toolboxes
#### Command — Enter the toolbox shell
```sh
toolbox enter llama-vulkan-radv
```
*This drops you into a shell inside the toolbox using your regular user account. The container shares your host home directory—anything in `$HOME` is accessible and writable inside the toolbox, so treat it like your host shell.*
#### Command — Confirm Llama.cpp sees your GPU
```sh
llama-cli --list-devices
```
Run this inside the toolbox to verify RADV/AMDVLK/ROCm devices are visible before loading a multi-gigabyte model.
> ⚠️ Always pass **flash attention** and **no-memory-map** flags when running on R9700. `llama-server` and `llama-cli` both expect `-fa 1 --no-mmap`. Skipping either tanks performance or triggers kernel crashes because of the giant unified memory aperture.
#### Command — Run llama-cli with flash attention + no-mmap
```sh
llama-cli --no-mmap -ngl 999 -fa 1 \
llama-cli -ngl 999 -fa 1 \
-m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
-p "Write a R9700 toolkit haiku."
```
- `-ngl 999` forces every layer onto the GPU.
- `-fa 1` turns on flash attention; omit it and throughput collapses.
- `--no-mmap` keeps allocations in unified memory rather than trying to memory-map multi-gigabyte files.
#### Command — Run llama-server with flash attention + no-mmap
```sh
llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
-c 8192 -ngl 999 -fa 1 --no-mmap
```
Adjust `-c` for context length and never drop `-fa 1 --no-mmap`.
## 2.3 Downloading GGUF Models from HuggingFace
Most Llama.cpp-compatible models are on [HuggingFace](https://huggingface.co/models?format=gguf). Filter for **GGUF** format, and try to pick Unsloth quantizations—they work great and are actively updated: https://huggingface.co/unsloth.
Download using the Hugging Face CLI. For example, to get the first shard of Qwen3 Coder 30B BF16 (https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF):
#### Command — Download a GGUF shard with `huggingface-cli`
**Or run as Server (API + Web UI):**
```bash
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
--local-dir models/qwen3-coder-30B-A3B/
llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
-c 8192 -ngl 999 -fa 1
```
`HF_HUB_ENABLE_HF_TRANSFER=1` uses a Rust-based package that enables faster download (install from [Pypi](https://pypi.org/project/hf-transfer/)).
## 3. Performance Benchmarks
🌐 Interactive exploration of the latest benchmark runs: [Interactie Benchmark Viewer](https://kyuz0.github.io/amd-r9700-ai-toolboxes/)
## 4. Memory Planning & VRAM Estimator
Running large language models locally requires estimating **total VRAM required**—not just for the model weights, but also for the "context" (number of active tokens) and extra overhead.
Use `gguf-vram-estimator.py` to check exactly how much memory you need for a given `.gguf` model and target context length. Example output:
```
$ gguf-vram-estimator.py models/llama-4-scout-17b-16e/Q4_K_XL/Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL-00001-of-00002.gguf --contexts 4096 32768 1048576
--- Model 'Llama-4-Scout-17B-16E-Instruct' ---
Max Context: 10,485,760 tokens
Model Size: 57.74 GiB
Incl. Overhead: 2.00 GiB
--- Memory Footprint Estimation ---
Context Size | Context Memory | Est. Total VRAM
---------------------------------------------------
4,096 | 1.88 GiB | 61.62 GiB
32,768 | 15.06 GiB | 74.80 GiB
1,048,576 | 49.12 GiB | 108.87 GiB
```
With Q4\_K quantization, **Llama-4-Scout 17B** can reach a 1M token context and still fit within a 128GB system, but... **it will be extremely slow to process such a long context**: see benchmarks (e.g. \~200 tokens/sec for prompt processing). Processing a 1M token context may take hours.
Contrast: Qwen3-235B Q3\_K (quantized, 97GiB model):
```
$ gguf-vram-estimator.py models/qwen3-235B-Q3_K-XL/UD-Q3_K_XL/Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL-00001-of-00003.gguf --contexts 65536 131072 262144
--- Memory Footprint Estimation ---
Context Size | Context Memory | Est. Total VRAM
---------------------------------------------------
65,536 | 11.75 GiB | 110.75 GiB
131,072 | 23.50 GiB | 122.50 GiB
262,144 | 47.00 GiB | 146.00 GiB
```
For Qwen3-235B, **128GB RAM allows you to run with context up to \~130k tokens.**
* The estimator lets you plan ahead and avoid out-of-memory errors when loading or using models.
* For more examples and a breakdown of VRAM components, see [docs/vram-estimator.md](docs/vram-estimator.md).
---
## 5. Building Containers Locally
## 📖 Detailed Guide
Pre-built toolbox container images are published on Docker Hub for immediate use. If you wish to build the containers yourself (for example, to customize packages or rebuild with a different llama.cpp version), see:
### Managing Toolboxes
Full instructions: [docs/building.md](docs/building.md).
#### Ubuntu Users (Distrobox)
If you are on Ubuntu, use Distrobox to ensure proper GPU access:
```bash
distrobox create -n llama-rocm-7.1 \
--image docker.io/kyuz0/amd-r9700-toolboxes:rocm-7.1-rocwmma \
--additional-flags "--device /dev/kfd --device /dev/dri --group-add video --group-add render --security-opt seccomp=unconfined"
distrobox enter llama-rocm-7.1
```
---
#### Updating Toolboxes
To pull the latest images and recreate your toolboxes (useful when Llama.cpp updates):
```bash
# Refresh all toolboxes
./refresh-toolboxes.sh all
## 6. Host Configuration
# Or refresh specific ones
./refresh-toolboxes.sh llama-vulkan-radv llama-rocm-7.1-rocwmma
```
This should work on any R9700. For a complete list of available hardware, see: [R9700 Hardware Database](https://r9700-homelab.d7.wtf/Hardware)
## 📦 Architecture & Containers
### 6.1 Test Configuration
### Backends
* **Vulkan**: Cross-platform, very stable.
* **RADV (Mesa)**: Best compatibility.
* **AMDVLK**: Official AMD driver. Faster in some cases but has a strict 2GB single buffer limit (some large models won't load).
* **ROCm**: AMD's compute stack (CUDA-like).
* **ROCWMMA**: Variants with optimizations for RDNA3 matrix cores (better Flash Attention).
| | |
| ----------------- | --------------------------------------------- |
| **Test Machine** | HP Z2 Mini G1a |
| **CPU** | AMD Radeon AI PRO R9700 |
| **System Memory** | 64 GB RAM (Host) |
| **GPU Memory** | 32 GB GDDR6 |
| **Host OS** | Fedora 42, kernel 6.15.6-200.fc42.x86\_86\_64 |
### Supported Container Images
Images are hosted on [Docker Hub](https://hub.docker.com/r/kyuz0/amd-r9700-toolboxes/tags) and automatically rebuilt on Llama.cpp updates.
| Tag | Backend | Notes |
| :--- | :--- | :--- |
| `vulkan-radv` | Vulkan (Mesa) | **Recommended**. Most stable. |
| `vulkan-amdvlk` | Vulkan (AMDVLK) | Fast, but strict memory limits. |
| `rocm-7.1-rocwmma` | ROCm 7.1 + ROCWMMA | Good balance of features and stability. |
| `rocm-7.9-rocwmma` | ROCm 7.9.0 + ROCWMMA | Newer ROCm release. |
| `rocm-7-nightly-rocwmma` | ROCm 7 Nightly | Bleeding edge. |
| `rocm-6.4.4` | ROCm 6.4.4 | Stable release. |
## ⚡ Performance & Planning
## 7. More Documentation
### Benchmarks
Check the [Interactive Benchmark Viewer](https://kyuz0.github.io/amd-r9700-ai-toolboxes/) or [docs/benchmarks.md](docs/benchmarks.md) to see performance numbers.
* [docs/benchmarks.md](docs/benchmarks.md): Full benchmark logs, model list, parsed results
* [docs/vram-estimator.md](docs/vram-estimator.md): Memory planning, practical example runs
* [docs/building.md](docs/building.md): Local build, toolbox customization, advanced use
### VRAM Estimator
Use the included script to estimate memory usage for models + context. This helps avoid OOM errors.
```bash
gguf-vram-estimator.py models/my-model.gguf --contexts 4096 32768
```
See [docs/vram-estimator.md](docs/vram-estimator.md) for more details.
## 8. References
## References
* The main reference for AMD R9700 home labs, by deseven (there's also a Discord server): [https://r9700-homelab.d7.wtf/](https://r9700-homelab.d7.wtf/)
* Most comprehesive repostiry of test builds for R9700 by lhl -> [https://github.com/lhl/r9700-testing/tree/main](https://github.com/lhl/strix-halo-testing/tree/main)
* Ubuntu 24.04 configuration
[https://github.com/technigmaai/technigmaai-wiki/wiki/AMD-Ryzen-AI-Max--395:-GTT--Memory-Step%E2%80%90by%E2%80%90Step-Instructions-(Ubuntu-24.04)](https://github.com/technigmaai/technigmaai-wiki/wiki/AMD-Ryzen-AI-Max--395:-GTT--Memory-Step%E2%80%90by%E2%80%90Step-Instructions-%28Ubuntu-24.04%29)
* [Llama.cpp GitHub Repository](https://github.com/ggerganov/llama.cpp)
* [AMD RDNA™ 4 Architecture](https://www.amd.com/en/products/graphics/rdna-architecture.html)
docs/benchmarks.md
-164
View File
@@ -1,164 +0,0 @@
# AMD Strix Halo — llama.cpp Toolboxes (Benchmarks)
**Interactive results:** https://kyuz0.github.io/amd-strix-halo-toolboxes/
## Table of Contents
- [Benchmark methodology](#benchmark-methodology)
- [Summary of current dataset (Flash Attention ON)](#summary-of-current-dataset-flash-attention-on)
- [Placement counts](#placement-counts)
- [Pairwise head-to-head wins](#pairwise-head-to-head-wins)
- [Average ranks](#average-ranks)
- [Analyses by feature](#analyses-by-feature)
- [Impact of Flash Attention](#impact-of-flash-attention)
- [Impact of ROCWMMA](#impact-of-rocwmma)
- [Impact of hipBLASLt](#impact-of-hipblaslt)
- [Vulkan: AMDVLK vs RADV](#vulkan-amdvlk-vs-radv)
- [Recommendations](#recommendations)
- [Winner calculation](#winner-calculation)
---
## Benchmark methodology
- **pp512** — prompt processing throughput (tokens/sec, prefill)
- **tg128** — token generation throughput (tokens/sec, interactive)
- Each backend tested twice per model: `-fa 0` and `-fa 1`
- Winners per model/test are **margin-aware**; multiple winners are possible when mean±σ overlap
- Built from the same llama.cpp commit for consistency
**Backends in this dataset:** ROCm 7 RC + ROCWMMA + hipBLASLt, ROCm 7 RC (hipBLASLt), ROCm 7 RC (hipBLASLt OFF), ROCm 7 RC + ROCWMMA (hipBLASLt OFF), ROCm 6.4.4 (hipBLASLt), ROCm 6.4.4 (hipBLASLt OFF), ROCm 6.4.4 + ROCWMMA (hipBLASLt), ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF), Vulkan AMDVLK, Vulkan RADV
**ROCm 7 hipBLASLt policy:** Toolboxes ship with **hipBLASLt enabled** by default (`ROCBLAS_USE_HIPBLASLT=1`). The benchmark script also runs **hipBLASLt OFF** variants (`-hblt0`) to measure its effect.
---
## Summary of current dataset (Flash Attention ON)
### Placement counts
**Prompt Processing (pp512)**
| Backend | 1st | 2nd | 3rd |
| --- | ---: | ---: | ---: |
| ROCm 6.4.4 (hipBLASLt) | 6 | 2 | 2 |
| Vulkan AMDVLK | 6 | 1 | 0 |
| ROCm 6.4.4 (hipBLASLt OFF) | 3 | 2 | 3 |
| Vulkan RADV | 1 | 2 | 0 |
| ROCm 7 RC (hipBLASLt) | 1 | 1 | 1 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 0 | 5 | 4 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt) | 0 | 4 | 2 |
| ROCm 7 RC (hipBLASLt OFF) | 0 | 0 | 2 |
| ROCm 7 RC + ROCWMMA + hipBLASLt | 0 | 0 | 3 |
**Token Generation (tg128)**
| Backend | 1st | 2nd | 3rd |
| --- | ---: | ---: | ---: |
| Vulkan RADV | 10 | 1 | 2 |
| Vulkan AMDVLK | 3 | 10 | 0 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 2 | 3 | 7 |
| ROCm 6.4.4 (hipBLASLt) | 1 | 4 | 3 |
| ROCm 6.4.4 (hipBLASLt OFF) | 1 | 3 | 5 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt) | 1 | 2 | 6 |
| ROCm 7 RC (hipBLASLt) | 1 | 0 | 1 |
| ROCm 7 RC (hipBLASLt OFF) | 0 | 1 | 1 |
| ROCm 7 RC + ROCWMMA + hipBLASLt | 0 | 1 | 1 |
| ROCm 7 RC + ROCWMMA (hipBLASLt OFF) | 0 | 1 | 1 |
### Pairwise head-to-head wins
For any model+quant where both backends succeeded, this counts who was faster (ties when equal).
| Comparison | Test | A wins | B wins | Ties | Total |
| --- | --- | ---: | ---: | ---: | ---: |
| ROCm 7 RC + ROCWMMA + hipBLASLt vs Vulkan AMDVLK | pp512 | 9 | 7 | 0 | 16 |
| ROCm 7 RC + ROCWMMA + hipBLASLt vs Vulkan AMDVLK | tg128 | 2 | 14 | 0 | 16 |
| ROCm 7 RC + ROCWMMA + hipBLASLt vs Vulkan RADV | pp512 | 14 | 3 | 0 | 17 |
| ROCm 7 RC + ROCWMMA + hipBLASLt vs Vulkan RADV | tg128 | 4 | 12 | 1 | 17 |
| Vulkan AMDVLK vs Vulkan RADV | pp512 | 12 | 4 | 0 | 16 |
| Vulkan AMDVLK vs Vulkan RADV | tg128 | 5 | 11 | 0 | 16 |
### Average ranks
**Prompt Processing (pp512)**
| Backend | Avg Rank (↓ is better) |
| --- | ---: |
| Vulkan AMDVLK | 1.14 |
| ROCm 6.4.4 (hipBLASLt) | 1.6 |
| Vulkan RADV | 1.67 |
| ROCm 6.4.4 (hipBLASLt OFF) | 2.0 |
| ROCm 7 RC (hipBLASLt) | 2.0 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt) | 2.33 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 2.44 |
| ROCm 7 RC (hipBLASLt OFF) | 3.0 |
| ROCm 7 RC + ROCWMMA + hipBLASLt | 3.0 |
**Token Generation (tg128)**
| Backend | Avg Rank (↓ is better) |
| --- | ---: |
| Vulkan RADV | 1.38 |
| Vulkan AMDVLK | 1.77 |
| ROCm 7 RC (hipBLASLt) | 2.0 |
| ROCm 6.4.4 (hipBLASLt) | 2.25 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 2.42 |
| ROCm 6.4.4 (hipBLASLt OFF) | 2.44 |
| ROCm 7 RC + ROCWMMA + hipBLASLt | 2.5 |
| ROCm 7 RC (hipBLASLt OFF) | 2.5 |
| ROCm 7 RC + ROCWMMA (hipBLASLt OFF) | 2.5 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt) | 2.56 |
---
## Analyses by feature
### Impact of Flash Attention
Median % change when **Flash Attention ON vs OFF**, paired by model+quant, per backend:
| Backend | pp512 Δ% (median, min..max, n) | tg128 Δ% (median, min..max, n) |
| --- | --- | --- |
| ROCm 7 RC + ROCWMMA + hipBLASLt | 11.4% (4.2..34.1), n=17 | -0.5% (-8.8..0.8), n=17 |
| ROCm 7 RC (hipBLASLt) | 11.7% (-23.0..25.6), n=14 | -1.1% (-8.7..1.0), n=14 |
| ROCm 7 RC (hipBLASLt OFF) | 6.8% (2.1..18.4), n=15 | -0.8% (-9.0..0.5), n=15 |
| ROCm 7 RC + ROCWMMA (hipBLASLt OFF) | 6.3% (-5.5..17.4), n=16 | -0.8% (-15.1..0.6), n=16 |
| ROCm 6.4.4 (hipBLASLt) | 8.3% (5.6..20.8), n=17 | 0.8% (-3.0..2.6), n=17 |
| ROCm 6.4.4 (hipBLASLt OFF) | 7.2% (-0.5..19.5), n=17 | 1.1% (-2.9..2.7), n=17 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt) | 7.1% (5.0..19.9), n=17 | 0.9% (-2.8..2.8), n=17 |
| ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 6.5% (2.7..18.6), n=17 | 1.1% (-2.7..3.4), n=17 |
| Vulkan AMDVLK | 1.3% (-10.8..27.8), n=16 | -1.2% (-6.8..0.1), n=16 |
| Vulkan RADV | 4.8% (-0.5..20.1), n=17 | -0.1% (-2.1..2.0), n=17 |
### Impact of ROCWMMA
| Context | Test | Compared Envs | Pairs | Median Δ% |
| --- | --- | --- | ---: | ---: |
| ROCm 7 RC (hipBLASLt) | pp512 | ROCm 7 RC + ROCWMMA + hipBLASLt vs ROCm 7 RC (hipBLASLt) | 15 | -0.0% |
| ROCm 7 RC (hipBLASLt) | tg128 | ROCm 7 RC + ROCWMMA + hipBLASLt vs ROCm 7 RC (hipBLASLt) | 15 | 0.0% |
| ROCm 7 RC (hipBLASLt OFF) | pp512 | ROCm 7 RC + ROCWMMA (hipBLASLt OFF) vs ROCm 7 RC (hipBLASLt OFF) | 17 | -0.2% |
| ROCm 7 RC (hipBLASLt OFF) | tg128 | ROCm 7 RC + ROCWMMA (hipBLASLt OFF) vs ROCm 7 RC (hipBLASLt OFF) | 17 | 0.0% |
| ROCm 6.4.4 (hipBLASLt) | pp512 | ROCm 6.4.4 + ROCWMMA (hipBLASLt) vs ROCm 6.4.4 (hipBLASLt) | 17 | -0.4% |
| ROCm 6.4.4 (hipBLASLt) | tg128 | ROCm 6.4.4 + ROCWMMA (hipBLASLt) vs ROCm 6.4.4 (hipBLASLt) | 17 | 0.0% |
| ROCm 6.4.4 (hipBLASLt OFF) | pp512 | ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) vs ROCm 6.4.4 (hipBLASLt OFF) | 17 | -0.5% |
| ROCm 6.4.4 (hipBLASLt OFF) | tg128 | ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) vs ROCm 6.4.4 (hipBLASLt OFF) | 17 | -0.1% |
### Impact of hipBLASLt
| Context | Test | Compared Envs | Pairs | Median Δ% |
| --- | --- | --- | ---: | ---: |
| ROCm 7 RC (no ROCWMMA) | pp512 | ROCm 7 RC (hipBLASLt) vs ROCm 7 RC (hipBLASLt OFF) | 15 | -0.2% |
| ROCm 7 RC (no ROCWMMA) | tg128 | ROCm 7 RC (hipBLASLt) vs ROCm 7 RC (hipBLASLt OFF) | 15 | 0.0% |
| ROCm 7 RC + ROCWMMA | pp512 | ROCm 7 RC + ROCWMMA + hipBLASLt vs ROCm 7 RC + ROCWMMA (hipBLASLt OFF) | 17 | -0.1% |
| ROCm 7 RC + ROCWMMA | tg128 | ROCm 7 RC + ROCWMMA + hipBLASLt vs ROCm 7 RC + ROCWMMA (hipBLASLt OFF) | 17 | 0.0% |
| ROCm 6.4.4 (no ROCWMMA) | pp512 | ROCm 6.4.4 (hipBLASLt) vs ROCm 6.4.4 (hipBLASLt OFF) | 17 | 0.0% |
| ROCm 6.4.4 (no ROCWMMA) | tg128 | ROCm 6.4.4 (hipBLASLt) vs ROCm 6.4.4 (hipBLASLt OFF) | 17 | 0.0% |
| ROCm 6.4.4 + ROCWMMA | pp512 | ROCm 6.4.4 + ROCWMMA (hipBLASLt) vs ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 17 | -0.3% |
| ROCm 6.4.4 + ROCWMMA | tg128 | ROCm 6.4.4 + ROCWMMA (hipBLASLt) vs ROCm 6.4.4 + ROCWMMA (hipBLASLt OFF) | 17 | 0.0% |
### Vulkan: AMDVLK vs RADV
Head-to-head wins with selected Flash Attention filter:
| Test | AMDVLK wins | RADV wins | Ties | Total |
| --- | ---: | ---: | ---: | ---: |
| pp512 | 12 | 4 | 0 | 16 |
| tg128 | 5 | 11 | 0 | 16 |
---
## Recommendations
- **Fastest prompt processing:** Vulkan AMDVLK, ROCm 6.4.4 (hipBLASLt) (most 1st-place finishes with selected Flash Attention filter).
- **Fastest token generation:** Vulkan RADV (most 1st-place finishes with selected Flash Attention filter).
- **Balanced choice:** Vulkan AMDVLK (consistently near the top across PP/TG).
---
## Winner calculation
A backend is counted as a winner if its mean throughput is within the best backend’s pooled ± error margin for that model/test type. This treats results within measurement noise as ties instead of false losses.
docs/docker-compose-how-to.md
-119
View File
@@ -1,119 +0,0 @@
## How to use docker-compose instead of toolbox
## Table of Contents
1. [Vulkan AMDVLK](#1-vulkanamdvlk)
2. [ROCm-6.4.4+ROCWMMA](#2-rocm-644-rocwmma)
## 1. Vulkan(AMDVLK)
1. Select applicable backend Dockerfile from repo. Example:
https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/toolboxes/Dockerfile.vulkan-amdvlk
2. In the build file, change shell command to:
```
# shell
CMD ["/bin/bash", "-c", "llama-server --host $HOST --port $PORT -c $CONTEXT_LENGTH --temp $TEMPERATURE --jinja --no-mmap -ngl $NGL -fa $FA -m $MODEL_PATH"]
```
3. Build container with:
```
docker build -f Dockerfile.vulkan-amdvlk -t vulkan-amdvlk:1.0 .
```
4. Download your model files to a directory. We will mount this from the container. I use:
```
/mnt/models
```
5. Create your docker compose, using this template. Change the ports and paths as needed.
```
services:
gpt-oss-120b:
container_name: gpt-oss-120b
image: vulkan-amdvlk:1.0
ports:
- "8069:8069"
volumes:
- /mnt/models:/mnt/models
devices:
- "/dev/dri:/dev/dri"
privileged: true
restart: unless-stopped
environment:
- HOST=0.0.0.0
- PORT=8069
- CONTEXT_LENGTH=120000
- TEMPERATURE=0.0
- MODEL_PATH=/mnt/models/gpt-oss-120b-UD-Q4_K_XL/gpt-oss-120b-UD-Q4_K_XL-00001-of-00002.gguf
- NGL=999
- FA=on
```
6. Start as usual.
```
docker compose up -d
```
## 2. ROCm-6.4.4-ROCWMMA
1. Select applicable backend Dockerfile from repo. Example:
https://github.com/kyuz0/amd-strix-halo-toolboxes/blob/main/toolboxes/Dockerfile.rocm-6.4.4-rocwmma
3. In the build file, change shell command to:
```
# shell
CMD ["/bin/bash", "-c", "llama-server --host $HOST --port $PORT -c $CONTEXT_LENGTH --temp $TEMPERATURE --jinja --no-mmap -ngl $NGL -fa $FA -m $MODEL_PATH"]
```
3. Build container with:
```
docker build -f Dockerfile.rocm-6.4.4-rocwmma -t rocm-6.4.4-rocwmma:1.0 .
```
4. Download your model files to a directory. We will mount this from the container. I use:
```
/mnt/models
```
5. Create your docker compose, using this template. Change the ports and paths as needed.
```
services:
gpt-oss-120b:
container_name: gpt-oss-120b
image: rocm-6.4.4-rocwmma:1.0
ports:
- "8069:8069"
volumes:
- /mnt/models:/mnt/models
devices:
- "/dev/dri:/dev/dri"
- "/dev/kfd:/dev/kfd"
privileged: true
restart: unless-stopped
environment:
- HOST=0.0.0.0
- PORT=8069
- CONTEXT_LENGTH=120000
- TEMPERATURE=0.0
- MODEL_PATH=/mnt/models/gpt-oss-120b-UD-Q4_K_XL/gpt-oss-120b-UD-Q4_K_XL-00001-of-00002.gguf
- NGL=999
- FA=on
```
6. Start as usual.
```
docker compose up -d
```