2026-03-29 05:06:19 +08:00
|
|
|
|
# ROCm vLLM 容器化推理项目
|
|
|
|
|
|
|
|
|
|
|
|
基于镜像 `rocm/vllm:rocm7.12.0_gfx120X-all_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` 的 Python 推理服务，适配双 AMD R9700 32G GPU。
|
|
|
|
|
|
|
|
|
|
|
|
## 项目目标
|
|
|
|
|
|
|
|
|
|
|
|
- 提供可容器化部署的模型推理 API
|
|
|
|
|
|
- 使用 vLLM 在双 GPU 上进行张量并行推理
|
|
|
|
|
|
- 提供健康检查、鉴权和参数化配置能力
|
|
|
|
|
|
|
|
|
|
|
|
## 目录结构
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
|
.
|
|
|
|
|
|
├── app
|
|
|
|
|
|
│ ├── config.py
|
|
|
|
|
|
│ ├── engine.py
|
|
|
|
|
|
│ ├── main.py
|
|
|
|
|
|
│ └── schemas.py
|
|
|
|
|
|
├── .dockerignore
|
|
|
|
|
|
├── .env.example
|
|
|
|
|
|
├── docker-compose.yml
|
|
|
|
|
|
├── Dockerfile
|
|
|
|
|
|
└── requirements.txt
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 接口说明
|
|
|
|
|
|
|
|
|
|
|
|
### 1) 健康检查
|
|
|
|
|
|
|
|
|
|
|
|
- 方法：`GET /health`
|
|
|
|
|
|
- 返回：服务状态与当前模型名
|
|
|
|
|
|
|
|
|
|
|
|
### 2) 文本生成
|
|
|
|
|
|
|
|
|
|
|
|
- 方法：`POST /v1/generate`
|
|
|
|
|
|
- Header：可选 `x-api-key`（配置 `API_KEY` 后必填）
|
|
|
|
|
|
- 请求体：
|
|
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
|
{
|
|
|
|
|
|
"prompt": "请介绍一下ROCm和vLLM的关系",
|
|
|
|
|
|
"max_tokens": 256,
|
|
|
|
|
|
"temperature": 0.7,
|
|
|
|
|
|
"top_p": 0.95,
|
|
|
|
|
|
"repetition_penalty": 1.0,
|
|
|
|
|
|
"stop": ["</s>"]
|
|
|
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 配置项
|
|
|
|
|
|
|
|
|
|
|
|
可通过 `.env` 配置，建议先复制 `.env.example`：
|
|
|
|
|
|
|
|
|
|
|
|
- `MODEL_NAME`：模型名称或本地路径
|
|
|
|
|
|
- `TENSOR_PARALLEL_SIZE`：张量并行数，双卡建议为 `2`
|
|
|
|
|
|
- `GPU_MEMORY_UTILIZATION`：单卡显存利用率，建议 `0.90~0.95`
|
|
|
|
|
|
- `MAX_MODEL_LEN`：模型最大上下文长度
|
|
|
|
|
|
- `MAX_NUM_SEQS`：并发序列数量
|
|
|
|
|
|
- `DTYPE`：精度类型，默认 `bfloat16`
|
|
|
|
|
|
- `API_KEY`：可选接口访问密钥
|
|
|
|
|
|
|
|
|
|
|
|
## 部署步骤
|
|
|
|
|
|
|
|
|
|
|
|
1. 准备环境变量：
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
cp .env.example .env
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
2. 修改 `.env` 中的 `MODEL_NAME` 与资源参数。
|
|
|
|
|
|
|
|
|
|
|
|
3. 构建并启动容器：
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
docker compose up -d --build
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
4. 验证服务：
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
curl http://localhost:8000/health
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 推理请求示例
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
curl -X POST "http://localhost:8000/v1/generate" \
|
|
|
|
|
|
-H "Content-Type: application/json" \
|
|
|
|
|
|
-d "{\"prompt\":\"请用三句话介绍大模型推理优化\",\"max_tokens\":128,\"temperature\":0.7}"
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 双 AMD R9700 调优建议
|
|
|
|
|
|
|
|
|
|
|
|
- 首选 `TENSOR_PARALLEL_SIZE=2`
|
|
|
|
|
|
- 首次部署建议设置 `GPU_MEMORY_UTILIZATION=0.90`，稳定后再调高
|
|
|
|
|
|
- 若模型较大且吞吐压力高，可逐步调低 `MAX_MODEL_LEN` 或 `MAX_NUM_SEQS`
|
|
|
|
|
|
- 确保宿主机已正确安装 ROCm 驱动并暴露 `/dev/kfd` 与 `/dev/dri`
|
2026-03-29 04:54:28 +08:00
|
|
|
|
|
