468 lines
11 KiB
Markdown
468 lines
11 KiB
Markdown
# RustFS 对象存储集成说明
|
||
|
||
> 文档定位：**RustFS / 对象存储集成的专题说明文档**。
|
||
> 本文解释对象存储侧的接口与集成思路，不作为当前部署入口文档。当前部署方式见 [../../DEPLOYMENT.md](../../DEPLOYMENT.md) 与 [../../deployment/LINUX_SETUP.md](../../deployment/LINUX_SETUP.md)，当前状态见 [../../STATUS.md](../../STATUS.md)。
|
||
## 架构概述
|
||
|
||
本项目采用 **RustFS** 作为对象存储和 **PostgreSQL** 作为元数据存储的双层存储架构。
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ 应用层 (FastAPI) │
|
||
└──────────────────────┬──────────────────────────────────────┘
|
||
│
|
||
┌──────────────┴──────────────┐
|
||
│ │
|
||
┌───────▼────────┐ ┌─────────▼─────────┐
|
||
│ PostgreSQL │ │ RustFS │
|
||
│ (元数据) │ │ (对象存储) │
|
||
│ │ │ │
|
||
│ - users │ │ - stp-files │
|
||
│ - stp_files │ │ - geometry │
|
||
│ - geometry_data│ │ - mold-cavities │
|
||
│ - mold_cavity │ │ - html-files │
|
||
│ - features │ │ - user-files │
|
||
│ - logs │ │ │
|
||
└────────────────┘ └──────────────────┘
|
||
```
|
||
|
||
## RustFS API 端点
|
||
|
||
假设 RustFS 运行在 `http://localhost:8080`，需要实现以下 REST API：
|
||
|
||
### 1. 健康检查
|
||
```
|
||
GET /health
|
||
返回: 200 OK
|
||
```
|
||
|
||
### 2. 初始化上传
|
||
```
|
||
POST /api/v1/upload/init
|
||
Content-Type: application/json
|
||
Authorization: Bearer <api_key>
|
||
|
||
请求体:
|
||
{
|
||
"namespace": "moldinsight/stp-files",
|
||
"key": "stp-files/abc123.stp",
|
||
"file_size": 1234567,
|
||
"file_hash": "sha256_hash",
|
||
"metadata": {
|
||
"original_filename": "model.stp",
|
||
"user_id": "1"
|
||
}
|
||
}
|
||
|
||
响应:
|
||
{
|
||
"upload_id": "unique-upload-id",
|
||
"upload_url": "https://rustfs/upload/xyz",
|
||
"expires_at": "2024-01-01T00:00:00Z"
|
||
}
|
||
```
|
||
|
||
### 3. 上传文件内容
|
||
```
|
||
PUT <upload_url>
|
||
Content-Type: application/octet-stream
|
||
Content-Length: 1234567
|
||
X-File-Hash: sha256_hash
|
||
|
||
请求体: <文件二进制数据>
|
||
|
||
响应: 201 Created
|
||
```
|
||
|
||
### 4. 完成上传
|
||
```
|
||
POST /api/v1/upload/complete
|
||
Content-Type: application/json
|
||
Authorization: Bearer <api_key>
|
||
|
||
请求体:
|
||
{
|
||
"upload_id": "unique-upload-id",
|
||
"namespace": "moldinsight/stp-files",
|
||
"key": "stp-files/abc123.stp"
|
||
}
|
||
|
||
响应:
|
||
{
|
||
"object_key": "moldinsight/stp-files/abc123.stp",
|
||
"etag": "d41d8cd98f00b204e9800998ecf8427e",
|
||
"created_at": "2024-01-01T00:00:00Z"
|
||
}
|
||
```
|
||
|
||
### 5. 上传 JSON（小文件，直接上传）
|
||
```
|
||
PUT /api/v1/objects/<namespace>/<key>
|
||
Content-Type: application/json
|
||
Authorization: Bearer <api_key>
|
||
X-File-Hash: sha256_hash
|
||
|
||
请求体: <JSON字符串>
|
||
|
||
响应: 201 Created
|
||
{
|
||
"object_key": "moldinsight/geometry/abc123.json",
|
||
"etag": "d41d8cd98f00b204e9800998ecf8427e",
|
||
"created_at": "2024-01-01T00:00:00Z"
|
||
}
|
||
```
|
||
|
||
### 6. 下载文件
|
||
```
|
||
GET /api/v1/objects/<namespace>/<key>
|
||
Authorization: Bearer <api_key>
|
||
|
||
响应: <文件二进制数据>
|
||
Content-Type: <原上传时的内容类型>
|
||
Content-Length: <文件大小>
|
||
ETag: <文件etag>
|
||
```
|
||
|
||
### 7. 获取文件信息
|
||
```
|
||
GET /api/v1/objects/<namespace>/<key>/info
|
||
Authorization: Bearer <api_key>
|
||
|
||
响应:
|
||
{
|
||
"object_key": "moldinsight/stp-files/abc123.stp",
|
||
"namespace": "moldinsight/stp-files",
|
||
"file_size": 1234567,
|
||
"file_hash": "sha256_hash",
|
||
"content_type": "application/octet-stream",
|
||
"created_at": "2024-01-01T00:00:00Z",
|
||
"last_modified": "2024-01-01T00:00:00Z",
|
||
"metadata": {
|
||
"original_filename": "model.stp",
|
||
"user_id": "1"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 8. 删除文件
|
||
```
|
||
DELETE /api/v1/objects/<namespace>/<key>
|
||
Authorization: Bearer <api_key>
|
||
|
||
响应: 204 No Content
|
||
```
|
||
|
||
### 9. 列出文件
|
||
```
|
||
GET /api/v1/buckets/<namespace>/objects?prefix=stp-files
|
||
Authorization: Bearer <api_key>
|
||
|
||
响应:
|
||
{
|
||
"namespace": "moldinsight/stp-files",
|
||
"prefix": "stp-files",
|
||
"objects": [
|
||
{
|
||
"object_key": "moldinsight/stp-files/abc123.stp",
|
||
"file_size": 1234567,
|
||
"file_hash": "sha256_hash",
|
||
"created_at": "2024-01-01T00:00:00Z"
|
||
},
|
||
...
|
||
],
|
||
"is_truncated": false,
|
||
"next_marker": null
|
||
}
|
||
```
|
||
|
||
### 10. 生成预签名 URL
|
||
```
|
||
POST /api/v1/presigned-url
|
||
Content-Type: application/json
|
||
Authorization: Bearer <api_key>
|
||
|
||
请求体:
|
||
{
|
||
"namespace": "moldinsight/stp-files",
|
||
"key": "stp-files/abc123.stp",
|
||
"expires": 3600,
|
||
"method": "GET"
|
||
}
|
||
|
||
响应:
|
||
{
|
||
"url": "https://rustfs/objects/moldinsight/stp-files/abc123.stp?signature=xyz&expires=123",
|
||
"expires_at": "2024-01-01T01:00:00Z"
|
||
}
|
||
```
|
||
|
||
### 11. 存储统计
|
||
```
|
||
GET /api/v1/stats
|
||
Authorization: Bearer <api_key>
|
||
|
||
响应:
|
||
{
|
||
"total_objects": 1234,
|
||
"total_size": 1234567890,
|
||
"namespace_stats": {
|
||
"moldinsight/stp-files": {
|
||
"object_count": 100,
|
||
"total_size": 123456789
|
||
},
|
||
"moldinsight/geometry": {
|
||
"object_count": 200,
|
||
"total_size": 234567890
|
||
},
|
||
...
|
||
}
|
||
}
|
||
```
|
||
|
||
## 存储桶（命名空间）
|
||
|
||
| 命名空间 | 用途 | 存储内容 |
|
||
|---------|------|---------|
|
||
| `moldinsight/stp-files` | STP/STEP文件 | 用户上传的原始3D模型文件 |
|
||
| `moldinsight/geometry` | 几何数据 | 几何分析结果的JSON数据 |
|
||
| `moldinsight/mold-cavities` | 模具型腔数据 | 模具设计的详细JSON数据 |
|
||
| `moldinsight/html` | HTML文件 | 生成的HTML报告文件 |
|
||
| `moldinsight/user-files` | 用户文件 | 其他用户上传的文件 |
|
||
|
||
## 快速开始
|
||
|
||
### 1. 配置环境变量
|
||
|
||
```bash
|
||
# 复制示例配置
|
||
cp .env.example .env
|
||
|
||
# 编辑 .env 文件
|
||
nano .env
|
||
```
|
||
|
||
设置 RustFS 相关配置：
|
||
```env
|
||
RUSTFS_ENDPOINT=http://localhost:8080
|
||
RUSTFS_API_KEY=your-rustfs-api-key
|
||
RUSTFS_TIMEOUT=30
|
||
RUSTFS_PRESIGNED_URL_EXPIRES=3600
|
||
```
|
||
|
||
### 2. 启动 RustFS 服务
|
||
|
||
假设你已经有 RustFS 服务，如果没有，可以按照以下方式启动：
|
||
|
||
```bash
|
||
# 使用 Docker（如果提供了 Docker 镜像）
|
||
docker run -d \
|
||
--name rustfs \
|
||
-p 8080:8080 \
|
||
-e RUSTFS_API_KEY=your-rustfs-api-key \
|
||
-e RUSTFS_STORAGE_PATH=/data \
|
||
-v rustfs_data:/data \
|
||
your-registry/rustfs:latest
|
||
|
||
# 或直接运行编译好的二进制文件
|
||
./rustfs-server --port 8080 --api-key your-rustfs-api-key --storage-path ./rustfs-data
|
||
```
|
||
|
||
### 3. 初始化存储
|
||
|
||
```bash
|
||
# 初始化 RustFS 连接
|
||
python src/storage/init_storage.py
|
||
```
|
||
|
||
### 4. 使用存储集成服务
|
||
|
||
```python
|
||
from services.storage_integration_rustfs import storage_integration
|
||
from database.database import db_manager
|
||
|
||
async def upload_file(file_path: str):
|
||
async with db_manager.get_session() as session:
|
||
stp_file = await storage_integration.save_stp_file(
|
||
session=session,
|
||
file_path=Path(file_path),
|
||
original_filename="model.stp",
|
||
user_id=1
|
||
)
|
||
print(f"文件已保存到 RustFS，ID: {stp_file.id}")
|
||
```
|
||
|
||
## Python 客户端使用
|
||
|
||
### 上传文件
|
||
|
||
```python
|
||
from storage.rustfs_storage import rustfs_manager
|
||
from pathlib import Path
|
||
|
||
# 先连接
|
||
await rustfs_manager.connect(
|
||
endpoint="http://localhost:8080",
|
||
api_key="your-api-key"
|
||
)
|
||
|
||
# 上传 STP 文件
|
||
result = await rustfs_manager.upload_file(
|
||
bucket_type='stp_files',
|
||
file_path=Path('model.stp'),
|
||
original_filename='model.stp'
|
||
)
|
||
print(f"上传成功: {result['object_key']}")
|
||
```
|
||
|
||
### 上传 JSON 数据
|
||
|
||
```python
|
||
geometry_data = {
|
||
"volume": 5061079.99,
|
||
"surface_area": 640037.28,
|
||
"bounding_box": {...}
|
||
}
|
||
|
||
result = await rustfs_manager.upload_json_data(
|
||
bucket_type='geometry_data',
|
||
json_data=geometry_data,
|
||
file_hash='sha256-hash'
|
||
)
|
||
print(f"JSON 上传成功: {result['object_key']}")
|
||
```
|
||
|
||
### 下载文件
|
||
|
||
```python
|
||
data = await rustfs_manager.download_file(
|
||
bucket_type='geometry_data',
|
||
object_key='geometry_data/abc123.json'
|
||
)
|
||
json_data = json.loads(data.decode('utf-8'))
|
||
print(json_data)
|
||
```
|
||
|
||
### 生成预签名 URL
|
||
|
||
```python
|
||
url = await rustfs_manager.generate_presigned_url(
|
||
bucket_type='stp_files',
|
||
object_key='stp-files/abc123.stp',
|
||
expires=3600 # 1小时
|
||
)
|
||
print(f"临时访问链接: {url}")
|
||
```
|
||
|
||
### 列出文件
|
||
|
||
```python
|
||
files = await rustfs_manager.list_files(
|
||
bucket_type='stp_files',
|
||
prefix='stp-files'
|
||
)
|
||
for f in files:
|
||
print(f"{f['object_key']}: {f['file_size']} bytes")
|
||
```
|
||
|
||
## RustFS 服务端实现参考
|
||
|
||
如果你需要实现 RustFS 服务端，以下是一个简单的 Rust 实现框架：
|
||
|
||
```rust
|
||
use actix_web::{web, App, HttpServer};
|
||
use serde::{Deserialize, Serialize};
|
||
use std::path::PathBuf;
|
||
|
||
#[derive(Deserialize)]
|
||
struct UploadInitRequest {
|
||
namespace: String,
|
||
key: String,
|
||
file_size: u64,
|
||
file_hash: String,
|
||
metadata: Option<serde_json::Value>,
|
||
}
|
||
|
||
#[derive(Serialize)]
|
||
struct UploadInitResponse {
|
||
upload_id: String,
|
||
upload_url: String,
|
||
expires_at: String,
|
||
}
|
||
|
||
async fn upload_init(
|
||
req: web::Json<UploadInitRequest>,
|
||
path: web::Data<PathBuf>
|
||
) -> impl web::Responder {
|
||
// 验证 API Key
|
||
// 生成 upload_id
|
||
// 返回上传 URL
|
||
web::Json(UploadInitResponse { ... })
|
||
}
|
||
|
||
#[actix_web::main]
|
||
async fn main() -> std::io::Result<()> {
|
||
HttpServer::new(|| {
|
||
App::new()
|
||
.route("/health", web::get().to(health_check))
|
||
.route("/api/v1/upload/init", web::post().to(upload_init))
|
||
// ... 其他路由
|
||
})
|
||
.bind("0.0.0.0:8080")?
|
||
.run()
|
||
.await
|
||
}
|
||
```
|
||
|
||
## 故障排除
|
||
|
||
### 连接 RustFS 失败
|
||
```
|
||
错误: RustFS 连接失败
|
||
解决:
|
||
1. 检查 RUSTFS_ENDPOINT 是否正确
|
||
2. 检查 RustFS 服务是否运行
|
||
3. 检查网络连接
|
||
4. 验证 API Key
|
||
```
|
||
|
||
### 文件上传失败
|
||
```
|
||
错误: RustFS 上传失败
|
||
解决:
|
||
1. 检查磁盘空间
|
||
2. 检查网络连接
|
||
3. 检查文件大小限制
|
||
4. 查看服务端日志
|
||
```
|
||
|
||
### 预签名 URL 失败
|
||
```
|
||
错误: RustFS 生成预签名URL失败
|
||
解决:
|
||
1. 检查 expires 参数是否有效
|
||
2. 确认服务端支持预签名 URL
|
||
3. 检查权限配置
|
||
```
|
||
|
||
## 性能优化建议
|
||
|
||
### 客户端
|
||
- 使用连接池（aiohttp 默认支持）
|
||
- 实现上传重试机制
|
||
- 对大文件使用分片上传
|
||
- 并行上传多个小文件
|
||
|
||
### 服务端
|
||
- 实现缓存层
|
||
- 支持范围请求（断点续传）
|
||
- 压缩存储
|
||
- CDN 分发静态文件
|
||
|
||
## 安全建议
|
||
|
||
1. **更改默认 API Key**：生产环境必须使用强密钥
|
||
2. **启用 HTTPS**：生产环境使用 TLS
|
||
3. **访问控制**：配置命名空间权限
|
||
4. **数据加密**：敏感数据加密存储
|
||
5. **日志监控**：记录所有访问和操作
|