90 lines
2.9 KiB
Markdown
90 lines
2.9 KiB
Markdown
|
---
|
|||
|
|
|
|||
|
|
## docs/vram-estimator.md
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 1. Memory Planning with `gguf-vram-estimator.py`
|
|||
|
|
|
|||
|
|
Estimating memory requirements is critical when running large models on Strix Halo (or any GPU with limited RAM). It's not enough to check just the model file size: context length and runtime overheads matter.
|
|||
|
|
|
|||
|
|
This repo provides a tool, **`gguf-vram-estimator.py`**, which reads a `.gguf` model and prints the estimated VRAM needed for different context sizes.
|
|||
|
|
|
|||
|
|
**Why?**
|
|||
|
|
|
|||
|
|
* Helps decide what fits on 32GB, 64GB, 128GB, etc—especially with multi-shard models or large quantized files.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 2. Usage
|
|||
|
|
|
|||
|
|
Make sure you have the estimator script (in `tools/`):
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
gguf-vram-estimator.py <path-to-model.gguf>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
* Supply one or more context lengths to get the corresponding VRAM footprint.
|
|||
|
|
* Handles multi-shard and single-shard models.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. Examples
|
|||
|
|
|
|||
|
|
### 3.1 Llama-4-Scout 17B Q4\_K\_XL, up to 1M tokens
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
$ gguf-vram-estimator.py models/llama-4-scout-17b-16e/Q4_K_XL/Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL-00001-of-00002.gguf --contexts 4096 32768 1048576
|
|||
|
|
|
|||
|
|
--- Model 'Llama-4-Scout-17B-16E-Instruct' ---
|
|||
|
|
Max Context: 10,485,760 tokens
|
|||
|
|
Model Size: 57.74 GiB
|
|||
|
|
Incl. Overhead: 2.00 GiB
|
|||
|
|
|
|||
|
|
--- Memory Footprint Estimation ---
|
|||
|
|
Context Size | Context Memory | Est. Total VRAM
|
|||
|
|
---------------------------------------------------
|
|||
|
|
4,096 | 1.88 GiB | 61.62 GiB
|
|||
|
|
32,768 | 15.06 GiB | 74.80 GiB
|
|||
|
|
1,048,576 | 49.12 GiB | 108.87 GiB
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
* **Takeaway:**
|
|||
|
|
|
|||
|
|
* Q4\_K quantization allows for a huge context in 128GB, but *processing 1M tokens will be extremely slow* (see benchmark: 200 tokens/sec prompt processing ⇒ almost 1.5 hours for a full 1M context fill).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
### 3.2 Qwen3-235B Q3\_K XL, high context
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
$ gguf-vram-estimator.py models/qwen3-235B-Q3_K-XL/UD-Q3_K_XL/Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL-00001-of-00003.gguf --contexts 65536 131072 262144
|
|||
|
|
|
|||
|
|
--- Memory Footprint Estimation ---
|
|||
|
|
Context Size | Context Memory | Est. Total VRAM
|
|||
|
|
---------------------------------------------------
|
|||
|
|
65,536 | 11.75 GiB | 110.75 GiB
|
|||
|
|
131,072 | 23.50 GiB | 122.50 GiB
|
|||
|
|
262,144 | 47.00 GiB | 146.00 GiB
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
* **Takeaway:**
|
|||
|
|
|
|||
|
|
* With 128GB, you can go up to \~130k context on this Qwen 235B quantized model.
|
|||
|
|
* If you go higher, you will OOM—even before context reaches the model's max.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. Notes
|
|||
|
|
|
|||
|
|
* “Est. Total VRAM” is the minimum you’ll need for the model + context, but does not include OS, other processes, or toolbox/container overhead—leave a margin.
|
|||
|
|
* For detailed methodology or custom scenarios, check the script source.
|
|||
|
|
* Benchmark speed for large context sizes is often the real bottleneck—see `docs/benchmarks.md` for real throughput figures.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. Related
|
|||
|
|
|
|||
|
|
* Main README section [Memory Planning & VRAM Estimator](../Readme#4--memory-planning--vram-estimator)
|
|||
|
|
* [docs/benchmarks.md](benchmarks.md) for full speed/compat charts
|