# AMD Radeon 9700 AI PRO (gfx1201) — vLLM Toolbox/Container
An **fedora-based** Docker/Podman container that is **Toolbx-compatible** (usable as a Fedora toolbox) for serving LLMs with **vLLM** on **AMD Radeon R9700 (gfx1201)**. Built on the TheRock nightly builds for ROCM.
See [TUNING.md](TUNING.md) for a guide on how to enable undervolting and raise the power limit on AMD R9700 cards on Linux to improve performance and efficiency.
### 🆕 Update: Comparison of Attention Backends (Triton vs ROCm)
*Added Support for ROCm Native Attention Backend*
I have added the ability to switch between the default **Triton** backend and the experimental **ROCm** native backend for attention operations. This provides you with more flexibility to optimize for stability or throughput depending on your specific model and workload.
| **Triton** (Default) | ✅ **High** | 🔸 Good | Works with all tested models |
| **ROCm** | ⚠️ **Experimental** | 🚀 **Highest** | May fail with complex architectures |
**Key Differences:**
- **Triton**: The safe choice. It uses the Triton compiler to generate kernels and is the standard for vLLM on AMD.
- **ROCm**: Uses composable kernel based attention. In my benchmarks, this often yields higher throughput (tokens/sec) but can be less stable, leading to crashes or "invalid graph" errors on some newer models.
**How to Use:**
1.**Easy Mode**: Select the backend in the `start-vllm` wizard (Item 5 in the menu).
2.**Manual Mode**: Export the following environment variables before running `vllm serve`:
```bash
export VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1
export VLLM_USE_TRITON_FLASH_ATTN=0
```
---
## 1) Toolbx vs Docker/Podman
The `kyuz0/vllm-therock-gfx1201:latest` image can be used both as:
* **Fedora Toolbx (recommended for development):** Toolbx shares your **HOME** and user, so models/configs live on the host. Great for iterating quickly while keeping the host clean.
* **Docker/Podman (recommended for deployment/perf):** Use for running vLLM as a service (host networking, IPC tuning, etc.). Always **mount a host directory** for model weights so they stay outside the container.
---
## 2) Quickstart — Fedora Toolbx
Create a toolbox that exposes the GPU and relaxes seccomp to avoid ROCm syscall issues:
--group-add video --group-add render --security-opt seccomp=unconfined
```
Enter it:
```bash
toolbox enter vllm-r9700
```
**Model storage:** Models are downloaded to `~/.cache/huggingface` by default. This directory is shared with the host if you created the toolbox correctly, so downloads persist.
### Serving a Model (Easiest Way)
The toolbox includes a TUI wizard called **`start-vllm`** which includes pre-configured models and handles launch flags. It also allows you to select the experimental **ROCm attention backend**. This is the easiest way to get started.
```bash
# if your weights live on disk instead of on HuggingFace, point the
# launcher at the directory that contains the model folders. the
# script will look for a subdirectory matching the repo ID and use it
# when launching.
export LOCAL_MODEL_DIR=/workspace/models
start-vllm
# you can also just run the CLI yourself and pass the path directly:
