feat: pluggable storage backends (S3, Azure, local) with admin UI
Implement 4-phase external storage backends architecture: Phase 1 - Foundation: - BlobStorageBackend trait (application/ports/blob_storage_ports.rs) - LocalBlobBackend: extracted all tokio::fs ops from DedupService - S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2) - DedupService refactored to use Arc<dyn BlobStorageBackend> Phase 2 - Admin Panel: - StorageSettingsService with DB persistence + env override - Storage tab in admin panel (backend selector, S3 form, provider presets) - GET/PUT/POST endpoints for storage settings + connection test - i18n keys (en/es) and BEM CSS Phase 3 - Migration: - MigrationBlobBackend decorator (dual-read: target-first + source fallback) - Background migration job with parallel transfers + progress tracking - Migration UI (progress bar, ETA, pause/resume/verify/complete) - 6 admin API endpoints for migration lifecycle Phase 4 - Enterprise Extras: - CachedBlobBackend: LRU disk cache for remote backends - EncryptedBlobBackend: AES-256-GCM at-rest encryption - AzureBlobBackend: Azure Blob Storage support - RetryBlobBackend: exponential backoff for transient errors - Decorator composition in DI: retry → encryption → cache All 223 tests passing, clippy clean, fmt verified.
This commit is contained in:
Generated
+1502
-92
File diff suppressed because it is too large
Load Diff
@@ -58,6 +58,14 @@ dashmap = "6"
|
||||
socket2 = { version = "0.6.2", features = ["all"] }
|
||||
urlencoding = "2.1.3"
|
||||
utoipa = { version = "5", features = ["axum_extras", "uuid", "chrono"] }
|
||||
aws-sdk-s3 = "1"
|
||||
aws-config = { version = "1", features = ["behavior-version-latest"] }
|
||||
aws-smithy-types = "1"
|
||||
azure_core = "0.21"
|
||||
azure_storage = "0.21"
|
||||
azure_storage_blobs = "0.21"
|
||||
aes-gcm = "0.10"
|
||||
lru = "0.12"
|
||||
|
||||
[features]
|
||||
default = []
|
||||
|
||||
@@ -0,0 +1,867 @@
|
||||
# External Storage Backends — Implementation Plan
|
||||
|
||||
> **Purpose**: This prompt provides Claude Code with full architectural context to implement pluggable blob storage backends (S3, Backblaze B2, MinIO, etc.) for OxiCloud, across 4 phases. All changes MUST respect the existing hexagonal architecture, BLAKE3 dedup system, and coding conventions defined in `CLAUDE.md`.
|
||||
|
||||
---
|
||||
|
||||
## Current Architecture Summary
|
||||
|
||||
### How blobs are stored today
|
||||
|
||||
- **Content-addressable**: Files hashed with BLAKE3 → stored at `.blobs/{2-char-prefix}/{hash}.blob`
|
||||
- **Dedup index**: PostgreSQL `storage.blobs` table (hash PK, ref_count, size, content_type)
|
||||
- **Write-first strategy**: Blob written to disk BEFORE PostgreSQL upsert (PG connection never held during disk I/O)
|
||||
- **Streaming reads**: 256 KB chunks via `tokio::fs::File` + `ReaderStream`
|
||||
- **Range support**: `AsyncSeekExt::seek()` + `file.take()` for HTTP Range requests
|
||||
|
||||
### Key files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `src/application/ports/dedup_ports.rs` | `DedupPort` trait — 12 methods, the hexagonal port |
|
||||
| `src/infrastructure/services/dedup_service.rs` | `DedupService` struct — sole implementation of `DedupPort` |
|
||||
| `src/common/di.rs` | `AppServiceFactory` — DI composition root, builds `DedupService` |
|
||||
| `src/common/config.rs` | `StorageConfig`, `AppConfig` — env var loading |
|
||||
| `src/interfaces/api/handlers/admin_handler.rs` | Admin API handlers (OIDC pattern to follow) |
|
||||
| `src/application/services/admin_settings_service.rs` | `AdminSettingsService` — runtime settings with env override |
|
||||
| `src/domain/repositories/settings_repository.rs` | `SettingsRepository` trait |
|
||||
| `src/infrastructure/repositories/pg/settings_pg_repository.rs` | PostgreSQL settings impl |
|
||||
| `static/admin.html` | Admin panel HTML (3 tabs: Dashboard, Users, OIDC) |
|
||||
| `static/js/views/admin/admin.js` | Admin panel JS logic |
|
||||
|
||||
### DedupService filesystem operations (candidates for extraction)
|
||||
|
||||
These are the EXACT `tokio::fs` calls inside `DedupService` that must be delegated to the new `BlobStorageBackend` trait:
|
||||
|
||||
```
|
||||
initialize() → fs::create_dir_all (blob_root, temp_root, 256 prefix dirs)
|
||||
store_from_file() → fs::metadata, fs::try_exists, fs::rename, fs::copy, fs::remove_file
|
||||
read_blob_stream() → File::open + ReaderStream
|
||||
read_blob_range_stream() → File::open + seek + take + ReaderStream
|
||||
blob_size() → fs::metadata
|
||||
remove_reference() → fs::remove_file (after PG commit)
|
||||
verify_integrity() → spawn_blocking with path checks
|
||||
blob_path() → PathBuf computation (sync)
|
||||
```
|
||||
|
||||
### DedupService PostgreSQL operations (stay in DedupService, untouched)
|
||||
|
||||
```
|
||||
store_from_file() → INSERT … ON CONFLICT … RETURNING ref_count
|
||||
blob_exists() → SELECT EXISTS from storage.blobs
|
||||
get_blob_metadata() → SELECT from storage.blobs
|
||||
add_reference() → UPDATE ref_count + 1
|
||||
remove_reference() → BEGIN TX → SELECT FOR UPDATE → DELETE if ref_count=0 → COMMIT
|
||||
get_stats() → SELECT COUNT, SUM from storage.blobs
|
||||
verify_integrity() → Streaming cursor SELECT from storage.blobs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Foundation (Backend Trait + Local + S3)
|
||||
|
||||
### Task 1.1: Create `BlobStorageBackend` trait
|
||||
|
||||
**File**: `src/application/ports/blob_storage_ports.rs` (NEW)
|
||||
|
||||
Create a minimal trait that abstracts ONLY raw byte I/O operations:
|
||||
|
||||
```rust
|
||||
use async_trait::async_trait;
|
||||
use bytes::Bytes;
|
||||
use futures::Stream;
|
||||
use std::path::Path;
|
||||
use std::pin::Pin;
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Health check result for storage backend connectivity
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct StorageHealthStatus {
|
||||
pub connected: bool,
|
||||
pub backend_type: String,
|
||||
pub message: String,
|
||||
/// Optional: available space in bytes (if backend reports it)
|
||||
pub available_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
|
||||
///
|
||||
/// Implementations: `LocalBlobBackend`, `S3BlobBackend`, etc.
|
||||
/// DedupService owns an `Arc<dyn BlobStorageBackend>` and delegates
|
||||
/// all filesystem/object-store operations through this trait.
|
||||
#[async_trait]
|
||||
pub trait BlobStorageBackend: Send + Sync + 'static {
|
||||
/// Initialize the backend (create directories, verify bucket access, etc.)
|
||||
async fn initialize(&self) -> Result<(), DomainError>;
|
||||
|
||||
/// Store a blob from a local temporary file.
|
||||
/// The backend MUST handle the case where the blob already exists (idempotent).
|
||||
/// Returns the number of bytes stored.
|
||||
async fn put_blob(&self, hash: &str, source_path: &Path) -> Result<u64, DomainError>;
|
||||
|
||||
/// Stream the full blob content.
|
||||
async fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
|
||||
|
||||
/// Stream a byte range of the blob (for HTTP Range requests).
|
||||
async fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
|
||||
|
||||
/// Delete a blob by hash. Must be idempotent (no error if already deleted).
|
||||
async fn delete_blob(&self, hash: &str) -> Result<(), DomainError>;
|
||||
|
||||
/// Check if a blob exists in the backend.
|
||||
async fn blob_exists(&self, hash: &str) -> Result<bool, DomainError>;
|
||||
|
||||
/// Get blob size in bytes without downloading content.
|
||||
async fn blob_size(&self, hash: &str) -> Result<u64, DomainError>;
|
||||
|
||||
/// Verify connectivity and permissions. Used by admin "Test Connection" button.
|
||||
async fn health_check(&self) -> Result<StorageHealthStatus, DomainError>;
|
||||
|
||||
/// Return the backend type name (for display in admin panel).
|
||||
fn backend_type(&self) -> &'static str;
|
||||
}
|
||||
```
|
||||
|
||||
Register in `src/application/ports/mod.rs` and `src/application/mod.rs`.
|
||||
|
||||
### Task 1.2: Create `LocalBlobBackend`
|
||||
|
||||
**File**: `src/infrastructure/services/local_blob_backend.rs` (NEW)
|
||||
|
||||
Extract ALL `tokio::fs` operations from `DedupService` into this struct. This is a **pure refactor** — zero behavior change.
|
||||
|
||||
```rust
|
||||
pub struct LocalBlobBackend {
|
||||
blob_root: PathBuf,
|
||||
temp_root: PathBuf,
|
||||
}
|
||||
```
|
||||
|
||||
Methods to implement from the trait, mapping from current DedupService code:
|
||||
|
||||
| Trait method | Source in DedupService | Key logic |
|
||||
|---|---|---|
|
||||
| `initialize()` | `DedupService::initialize()` lines 115-154 | Create `.blobs/`, `.dedup_temp/`, 256 prefix dirs |
|
||||
| `put_blob()` | `DedupService::store_from_file()` lines 196-253 | `fs::try_exists` → `fs::rename` (EXDEV fallback `fs::copy`) → cleanup source |
|
||||
| `get_blob_stream()` | `DedupService::read_blob_stream()` lines 306-323 | `File::open` → `ReaderStream::with_capacity(256KB)` |
|
||||
| `get_blob_range_stream()` | `DedupService::read_blob_range_stream()` lines 325-353 | `File::open` → `seek` → `take` → `ReaderStream` |
|
||||
| `delete_blob()` | `DedupService::remove_reference()` line ~469 | `fs::remove_file` |
|
||||
| `blob_exists()` | New (was inline `fs::try_exists`) | `fs::try_exists(blob_path)` |
|
||||
| `blob_size()` | `DedupService::blob_size()` lines 355-369 | `fs::metadata().len()` |
|
||||
| `health_check()` | New | Check blob_root is writable, return disk available via `statvfs` |
|
||||
| `backend_type()` | New | Return `"local"` |
|
||||
|
||||
Also add a public helper:
|
||||
```rust
|
||||
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||
let prefix = &hash[..2];
|
||||
self.blob_root.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
```
|
||||
|
||||
Register in `src/infrastructure/services/mod.rs`.
|
||||
|
||||
### Task 1.3: Refactor `DedupService` to use `BlobStorageBackend`
|
||||
|
||||
**File**: `src/infrastructure/services/dedup_service.rs` (MODIFY)
|
||||
|
||||
Changes:
|
||||
1. Add field: `backend: Arc<dyn BlobStorageBackend>`
|
||||
2. Remove fields: `blob_root: PathBuf`, `temp_root: PathBuf` (moved to `LocalBlobBackend`)
|
||||
3. Update constructor to accept `Arc<dyn BlobStorageBackend>` instead of `storage_root: &Path`
|
||||
4. Replace all direct `tokio::fs` calls with `self.backend.*` calls
|
||||
5. Keep `blob_path()` in DedupPort as a delegation: `self.backend.blob_path()` — BUT since `blob_path()` returns `PathBuf` and is used by thumbnails/caching services, consider adding it to the backend trait OR keeping a separate method. For S3 backends, this should return a virtual path or the method should be deprecated in favor of streaming.
|
||||
|
||||
**Critical**: `hash_file()` stays in `DedupService` (BLAKE3 hashing is NOT a backend concern — it always runs on local temp files before upload).
|
||||
|
||||
**Critical**: The write-first strategy is preserved:
|
||||
```
|
||||
1. hash_file() on local temp file
|
||||
2. self.backend.put_blob(hash, temp_path) ← backend moves/uploads
|
||||
3. INSERT INTO storage.blobs … ON CONFLICT ← PostgreSQL upsert
|
||||
```
|
||||
|
||||
**Critical**: `remove_reference()` flow preserved:
|
||||
```
|
||||
1. BEGIN TX → SELECT FOR UPDATE → check ref_count
|
||||
2. If ref_count == 1 → DELETE FROM storage.blobs → COMMIT
|
||||
3. self.backend.delete_blob(hash) ← after PG commit
|
||||
```
|
||||
|
||||
### Task 1.4: Create `S3BlobBackend`
|
||||
|
||||
**File**: `src/infrastructure/services/s3_blob_backend.rs` (NEW)
|
||||
|
||||
**Dependency to add to `Cargo.toml`**:
|
||||
```toml
|
||||
aws-sdk-s3 = "1"
|
||||
aws-config = { version = "1", features = ["behavior-version-latest"] }
|
||||
aws-smithy-types = "1" # For ByteStream
|
||||
```
|
||||
|
||||
> Note: `aws-sdk-s3` is the official AWS SDK for Rust. It's compatible with ALL S3-compatible services (Backblaze B2, MinIO, Cloudflare R2, DigitalOcean Spaces, Wasabi) via custom endpoint configuration.
|
||||
|
||||
```rust
|
||||
pub struct S3BlobBackend {
|
||||
client: aws_sdk_s3::Client,
|
||||
bucket: String,
|
||||
}
|
||||
```
|
||||
|
||||
**S3 key scheme**: Same as local — `{2-char-prefix}/{hash}.blob` (e.g., `a3/a3c5f2e8d1…blob`)
|
||||
|
||||
Method mapping:
|
||||
|
||||
| Trait method | S3 operation |
|
||||
|---|---|
|
||||
| `initialize()` | `head_bucket()` to verify bucket exists + permissions |
|
||||
| `put_blob()` | `put_object()` with `Body::from_path(source_path)`. Check existence with `head_object()` first for idempotency |
|
||||
| `get_blob_stream()` | `get_object()` → `.body.into_async_read()` → `ReaderStream` |
|
||||
| `get_blob_range_stream()` | `get_object().range(format!("bytes={start}-{end}"))` → stream |
|
||||
| `delete_blob()` | `delete_object()` (already idempotent in S3) |
|
||||
| `blob_exists()` | `head_object()` — 200 = true, 404 = false |
|
||||
| `blob_size()` | `head_object()` → `.content_length()` |
|
||||
| `health_check()` | `head_bucket()` + `list_objects_v2(max_keys=1)` |
|
||||
| `backend_type()` | Return `"s3"` |
|
||||
|
||||
**S3 Client construction**: Must support custom endpoints for non-AWS providers:
|
||||
|
||||
```rust
|
||||
impl S3BlobBackend {
|
||||
pub async fn new(config: &S3StorageConfig) -> Result<Self, DomainError> {
|
||||
let mut s3_config_builder = aws_sdk_s3::config::Builder::new()
|
||||
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
|
||||
.credentials_provider(
|
||||
aws_sdk_s3::config::Credentials::new(
|
||||
&config.access_key,
|
||||
&config.secret_key,
|
||||
None, None, "oxicloud",
|
||||
)
|
||||
)
|
||||
.behavior_version_latest();
|
||||
|
||||
if let Some(endpoint) = &config.endpoint_url {
|
||||
s3_config_builder = s3_config_builder
|
||||
.endpoint_url(endpoint)
|
||||
.force_path_style(config.force_path_style);
|
||||
}
|
||||
|
||||
let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());
|
||||
Ok(Self { client, bucket: config.bucket.clone() })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Task 1.5: Add storage backend configuration
|
||||
|
||||
**File**: `src/common/config.rs` (MODIFY)
|
||||
|
||||
Add to existing `StorageConfig`:
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum StorageBackendType {
|
||||
Local,
|
||||
S3,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct S3StorageConfig {
|
||||
pub endpoint_url: Option<String>, // OXICLOUD_S3_ENDPOINT_URL
|
||||
pub bucket: String, // OXICLOUD_S3_BUCKET
|
||||
pub region: String, // OXICLOUD_S3_REGION (default: "us-east-1")
|
||||
pub access_key: String, // OXICLOUD_S3_ACCESS_KEY
|
||||
pub secret_key: String, // OXICLOUD_S3_SECRET_KEY
|
||||
pub force_path_style: bool, // OXICLOUD_S3_FORCE_PATH_STYLE (default: false)
|
||||
}
|
||||
|
||||
// Add to existing StorageConfig:
|
||||
pub struct StorageConfig {
|
||||
pub root_dir: String, // existing
|
||||
pub chunk_size: usize, // existing
|
||||
pub parallel_threshold: usize, // existing
|
||||
pub trash_retention_days: u32, // existing
|
||||
pub max_upload_size: usize, // existing
|
||||
pub backend: StorageBackendType, // NEW — OXICLOUD_STORAGE_BACKEND (default: "local")
|
||||
pub s3: Option<S3StorageConfig>, // NEW — populated when backend=s3
|
||||
}
|
||||
```
|
||||
|
||||
**Env var loading** in `AppConfig::from_env()`:
|
||||
```
|
||||
OXICLOUD_STORAGE_BACKEND → "local" | "s3" (default: "local")
|
||||
OXICLOUD_S3_ENDPOINT_URL → Optional custom endpoint
|
||||
OXICLOUD_S3_BUCKET → Required when backend=s3
|
||||
OXICLOUD_S3_REGION → Default "us-east-1"
|
||||
OXICLOUD_S3_ACCESS_KEY → Required when backend=s3
|
||||
OXICLOUD_S3_SECRET_KEY → Required when backend=s3
|
||||
OXICLOUD_S3_FORCE_PATH_STYLE → Default false
|
||||
```
|
||||
|
||||
### Task 1.6: Wire backend selection in DI
|
||||
|
||||
**File**: `src/common/di.rs` (MODIFY)
|
||||
|
||||
In `create_core_services()`, replace the current DedupService construction:
|
||||
|
||||
```rust
|
||||
// Build storage backend based on config
|
||||
let blob_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
|
||||
StorageBackendType::Local => {
|
||||
Arc::new(LocalBlobBackend::new(&self.storage_path))
|
||||
}
|
||||
StorageBackendType::S3 => {
|
||||
let s3_config = self.config.storage.s3.as_ref()
|
||||
.expect("S3 config required when backend=s3");
|
||||
Arc::new(S3BlobBackend::new(s3_config).await?)
|
||||
}
|
||||
};
|
||||
|
||||
blob_backend.initialize().await?;
|
||||
|
||||
let dedup_service = Arc::new(DedupService::new(
|
||||
blob_backend.clone(),
|
||||
db_pool.clone(),
|
||||
maintenance_pool.clone(),
|
||||
));
|
||||
```
|
||||
|
||||
### Task 1.7: Handle `blob_path()` deprecation path
|
||||
|
||||
The `DedupPort::blob_path()` method returns a `PathBuf` and is used by:
|
||||
- Thumbnail generation service (needs local file access)
|
||||
- File content caching (moka cache)
|
||||
|
||||
For S3 backends, `blob_path()` has no meaning. Solutions:
|
||||
1. **For thumbnails**: Change thumbnail service to accept a `Stream` instead of a `PathBuf`, OR download to a temp file first
|
||||
2. **For caching**: Cache already works with streams
|
||||
3. Keep `blob_path()` on `DedupPort` but make it return `Option<PathBuf>` (None for remote backends) — consumers must handle the None case
|
||||
|
||||
Search for all callers of `blob_path()` and update them.
|
||||
|
||||
### Task 1.8: Tests
|
||||
|
||||
- Unit test `LocalBlobBackend` in isolation (mock filesystem with temp dirs)
|
||||
- Unit test `S3BlobBackend` with a mock S3 (use `aws-smithy-runtime` test utilities or `mockall`)
|
||||
- Integration test: `DedupService` with `LocalBlobBackend` must pass ALL existing tests unchanged (this proves the refactor is correct)
|
||||
- Add `#[cfg(test)]` inline tests in each new file following existing project convention
|
||||
|
||||
### Task 1.9: Pre-commit validation
|
||||
|
||||
```bash
|
||||
cargo fmt --all
|
||||
cargo clippy -- -D warnings
|
||||
cargo test --workspace
|
||||
```
|
||||
|
||||
ALL existing ~208 tests MUST pass. Zero regressions.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Admin Panel Configuration
|
||||
|
||||
### Task 2.1: Storage settings service
|
||||
|
||||
**File**: `src/application/services/storage_settings_service.rs` (NEW)
|
||||
|
||||
Follow the EXACT same pattern as `AdminSettingsService` for OIDC. Create `StorageSettingsService`:
|
||||
|
||||
```rust
|
||||
pub struct StorageSettingsService {
|
||||
settings_repo: Arc<SettingsPgRepository>,
|
||||
env_storage_config: StorageConfig, // from AppConfig at startup
|
||||
}
|
||||
```
|
||||
|
||||
**Methods** (follow OIDC pattern):
|
||||
|
||||
| Method | Purpose |
|
||||
|---|---|
|
||||
| `get_storage_settings()` | Load from DB (category: `"storage"`), mask secrets, mark env overrides |
|
||||
| `save_storage_settings(dto, user_id)` | Upsert each field to `admin_settings`, mark secrets with `is_secret: true` |
|
||||
| `test_storage_connection(dto)` | Build temporary backend from DTO config, call `health_check()`, return result |
|
||||
| `load_effective_storage_config()` | Merge: DB settings + env var overrides + defaults |
|
||||
| `get_env_overrides()` | Return list of `OXICLOUD_S3_*` / `OXICLOUD_STORAGE_*` env vars that are set |
|
||||
|
||||
**DB keys** (category: `"storage"`):
|
||||
|
||||
```
|
||||
storage.backend → "local" | "s3"
|
||||
storage.s3.endpoint_url → string (optional)
|
||||
storage.s3.bucket → string
|
||||
storage.s3.region → string
|
||||
storage.s3.access_key → string (is_secret: true)
|
||||
storage.s3.secret_key → string (is_secret: true)
|
||||
storage.s3.force_path_style → "true" | "false"
|
||||
```
|
||||
|
||||
### Task 2.2: Storage admin API endpoints
|
||||
|
||||
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
|
||||
|
||||
Add to `admin_routes()`:
|
||||
|
||||
```rust
|
||||
.route("/settings/storage", get(get_storage_settings))
|
||||
.route("/settings/storage", put(save_storage_settings))
|
||||
.route("/settings/storage/test", post(test_storage_connection))
|
||||
```
|
||||
|
||||
**Handler implementations** (follow OIDC handlers exactly):
|
||||
|
||||
| Handler | Method | Body | Response |
|
||||
|---|---|---|---|
|
||||
| `get_storage_settings` | GET | — | `StorageSettingsDto` (secrets masked, env_overrides listed) |
|
||||
| `save_storage_settings` | PUT | `SaveStorageSettingsDto` | `{ "message": "Storage settings saved" }` |
|
||||
| `test_storage_connection` | POST | `TestStorageConnectionDto` | `StorageTestResultDto { connected, message, available_bytes }` |
|
||||
|
||||
**DTOs** (add to `src/application/dtos/`):
|
||||
|
||||
```rust
|
||||
#[derive(Serialize)]
|
||||
pub struct StorageSettingsDto {
|
||||
pub backend: String, // "local" | "s3"
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
pub s3_access_key_set: bool, // masked — never send actual key
|
||||
pub s3_secret_key_set: bool, // masked — never send actual secret
|
||||
pub s3_force_path_style: bool,
|
||||
pub env_overrides: Vec<String>, // which fields are locked by env vars
|
||||
// Current stats
|
||||
pub current_backend: String,
|
||||
pub total_blobs: u64,
|
||||
pub total_bytes_stored: u64,
|
||||
pub dedup_ratio: f64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct SaveStorageSettingsDto {
|
||||
pub backend: String,
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
pub s3_access_key: Option<String>, // only sent if changed
|
||||
pub s3_secret_key: Option<String>, // only sent if changed
|
||||
pub s3_force_path_style: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct TestStorageConnectionDto {
|
||||
pub backend: String,
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
pub s3_access_key: Option<String>,
|
||||
pub s3_secret_key: Option<String>,
|
||||
pub s3_force_path_style: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct StorageTestResultDto {
|
||||
pub connected: bool,
|
||||
pub message: String,
|
||||
pub backend_type: String,
|
||||
pub available_bytes: Option<u64>,
|
||||
}
|
||||
```
|
||||
|
||||
### Task 2.3: Wire `StorageSettingsService` into DI
|
||||
|
||||
**File**: `src/common/di.rs` (MODIFY)
|
||||
|
||||
Add `StorageSettingsService` construction alongside `AdminSettingsService`. Add to `AppState`.
|
||||
|
||||
### Task 2.4: Admin Panel — Storage tab (HTML)
|
||||
|
||||
**File**: `static/admin.html` (MODIFY)
|
||||
|
||||
Add a 4th tab button after the OIDC tab:
|
||||
|
||||
```html
|
||||
<button class="admin-tab" id="tab-btn-storage">
|
||||
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
|
||||
</button>
|
||||
```
|
||||
|
||||
Add `tab-storage` content div with:
|
||||
|
||||
1. **Backend selector** — Radio buttons: Local Filesystem / S3-Compatible
|
||||
2. **S3 configuration form** (shown/hidden based on selector):
|
||||
- Provider preset dropdown (Amazon S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces, Wasabi, Custom)
|
||||
- Endpoint URL field
|
||||
- Bucket field
|
||||
- Region field
|
||||
- Access Key ID field
|
||||
- Secret Key field (password type)
|
||||
- Force Path Style checkbox
|
||||
- ENV badges on fields overridden by env vars (same as OIDC)
|
||||
3. **Test Connection button** → calls `POST /api/admin/settings/storage/test`
|
||||
4. **Save button** → calls `PUT /api/admin/settings/storage`
|
||||
5. **Current Status section**:
|
||||
- Active backend type
|
||||
- Total blobs / total size / dedup ratio (from `DedupStatsDto`)
|
||||
6. **Migration section** (Phase 3 — can be placeholder with "Coming soon")
|
||||
|
||||
### Task 2.5: Admin Panel — Storage tab (JS)
|
||||
|
||||
**File**: `static/js/views/admin/admin.js` (MODIFY)
|
||||
|
||||
Follow OIDC tab patterns:
|
||||
|
||||
```javascript
|
||||
// Provider presets
|
||||
const STORAGE_PRESETS = {
|
||||
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
|
||||
'backblaze': { endpoint: 's3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
|
||||
'cloudflare-r2': { endpoint: '{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
|
||||
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
|
||||
'digitalocean': { endpoint: '{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
|
||||
'wasabi': { endpoint: 's3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
|
||||
'custom': { endpoint: '', region: '', pathStyle: false },
|
||||
};
|
||||
```
|
||||
|
||||
Functions:
|
||||
- `loadStorage()` — `GET /api/admin/settings/storage` → populate form
|
||||
- `saveStorage()` — Collect form → `PUT /api/admin/settings/storage`
|
||||
- `testStorageConnection()` — Collect form → `POST /api/admin/settings/storage/test` → show result
|
||||
- `onPresetChange(preset)` — Auto-fill endpoint/region/pathStyle from preset
|
||||
- `toggleS3Form(visible)` — Show/hide S3 fields when backend radio changes
|
||||
|
||||
### Task 2.6: i18n keys
|
||||
|
||||
**Files**: `static/locales/*.json` (MODIFY at minimum `en.json` and `es.json`)
|
||||
|
||||
Add translation keys for all new UI labels:
|
||||
```
|
||||
admin.tab_storage
|
||||
admin.storage_backend
|
||||
admin.storage_local
|
||||
admin.storage_s3
|
||||
admin.storage_provider_preset
|
||||
admin.storage_endpoint_url
|
||||
admin.storage_bucket
|
||||
admin.storage_region
|
||||
admin.storage_access_key
|
||||
admin.storage_secret_key
|
||||
admin.storage_path_style
|
||||
admin.storage_test_connection
|
||||
admin.storage_test_success
|
||||
admin.storage_test_failure
|
||||
admin.storage_save
|
||||
admin.storage_saved
|
||||
admin.storage_current_status
|
||||
admin.storage_total_blobs
|
||||
admin.storage_total_size
|
||||
admin.storage_dedup_ratio
|
||||
admin.storage_migration
|
||||
admin.storage_migration_coming_soon
|
||||
```
|
||||
|
||||
### Task 2.7: CSS for storage tab
|
||||
|
||||
**File**: `static/css/admin.css` (MODIFY)
|
||||
|
||||
Add styles for:
|
||||
- `.storage-backend-selector` — Radio button group
|
||||
- `.storage-provider-presets` — Dropdown styling
|
||||
- `.storage-form` — Form fields (reuse existing OIDC form patterns)
|
||||
- `.storage-status-grid` — Stats display
|
||||
- BEM methodology, CSS custom properties for colors, no raw hex/rgb
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Migration Between Backends
|
||||
|
||||
### Task 3.1: `MigrationBlobBackend` wrapper
|
||||
|
||||
**File**: `src/infrastructure/services/migration_blob_backend.rs` (NEW)
|
||||
|
||||
A `BlobStorageBackend` decorator that enables zero-downtime migration between backends:
|
||||
|
||||
```rust
|
||||
pub struct MigrationBlobBackend {
|
||||
source: Arc<dyn BlobStorageBackend>, // old backend (read fallback)
|
||||
target: Arc<dyn BlobStorageBackend>, // new backend (primary for writes)
|
||||
state: Arc<RwLock<MigrationState>>,
|
||||
}
|
||||
|
||||
pub struct MigrationState {
|
||||
pub status: MigrationStatus,
|
||||
pub total_blobs: u64,
|
||||
pub migrated_blobs: u64,
|
||||
pub migrated_bytes: u64,
|
||||
pub failed_blobs: Vec<String>, // hashes that failed
|
||||
pub started_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
pub completed_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
}
|
||||
|
||||
pub enum MigrationStatus {
|
||||
Idle, // no migration in progress
|
||||
Running, // background job active
|
||||
Paused, // manually paused
|
||||
Completed, // all blobs migrated
|
||||
Failed, // unrecoverable error
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**:
|
||||
|
||||
| Operation | During Migration |
|
||||
|---|---|
|
||||
| `put_blob()` | Write to **target** only |
|
||||
| `get_blob_stream()` | Try **target** first → fallback to **source** (+ schedule lazy copy) |
|
||||
| `get_blob_range_stream()` | Same fallback strategy |
|
||||
| `delete_blob()` | Delete from **both** (best-effort on source) |
|
||||
| `blob_exists()` | Check **target** first, then **source** |
|
||||
| `blob_size()` | Check **target** first, then **source** |
|
||||
|
||||
### Task 3.2: Background migration job
|
||||
|
||||
**File**: `src/infrastructure/services/migration_job.rs` (NEW)
|
||||
|
||||
```rust
|
||||
pub async fn run_migration(
|
||||
source: Arc<dyn BlobStorageBackend>,
|
||||
target: Arc<dyn BlobStorageBackend>,
|
||||
pool: Arc<PgPool>,
|
||||
state: Arc<RwLock<MigrationState>>,
|
||||
concurrency: usize, // default: 4 parallel transfers
|
||||
bandwidth_limit: Option<u64>, // bytes/sec, None = unlimited
|
||||
) -> Result<(), DomainError>
|
||||
```
|
||||
|
||||
**Algorithm**:
|
||||
1. Query `SELECT hash, size FROM storage.blobs ORDER BY hash` with streaming cursor
|
||||
2. For each blob hash:
|
||||
a. Check if already exists in target (`target.blob_exists(hash)`)
|
||||
b. If not: stream from source → write to temp file → `target.put_blob(hash, temp)`
|
||||
c. Update `MigrationState` counters
|
||||
d. Respect bandwidth limit via `tokio::time::sleep` throttling
|
||||
3. Use `futures::stream::buffer_unordered(concurrency)` for parallel transfers
|
||||
4. On error: log, add to `failed_blobs`, continue (don't abort entire migration)
|
||||
|
||||
### Task 3.3: Migration API endpoints
|
||||
|
||||
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
|
||||
|
||||
Add to `admin_routes()`:
|
||||
|
||||
```rust
|
||||
.route("/storage/migration", get(get_migration_status))
|
||||
.route("/storage/migration/start", post(start_migration))
|
||||
.route("/storage/migration/pause", post(pause_migration))
|
||||
.route("/storage/migration/resume", post(resume_migration))
|
||||
.route("/storage/migration/complete", post(complete_migration))
|
||||
```
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `GET /migration` | Return `MigrationState` (status, progress, ETA) |
|
||||
| `POST /migration/start` | Begin background migration from current → configured backend |
|
||||
| `POST /migration/pause` | Pause the background job |
|
||||
| `POST /migration/resume` | Resume paused migration |
|
||||
| `POST /migration/complete` | Finalize: switch primary backend, optionally clean up source |
|
||||
|
||||
### Task 3.4: Migration UI in admin panel
|
||||
|
||||
In the Storage tab's Migration section:
|
||||
|
||||
1. **Start Migration button** (when backend config differs from active)
|
||||
2. **Progress bar**: `{migrated} / {total} blobs ({percent}%) — {bytes_migrated} transferred`
|
||||
3. **Estimated time remaining** (based on throughput)
|
||||
4. **Pause / Resume button**
|
||||
5. **Complete Migration button** (enabled only when 100% migrated)
|
||||
6. **Failed blobs list** (expandable, with retry button)
|
||||
7. **Status badge**: Idle / Running / Paused / Completed / Failed
|
||||
|
||||
### Task 3.5: Integrity verification post-migration
|
||||
|
||||
After migration completes (before `complete_migration`):
|
||||
1. Run `verify_integrity()` against the target backend
|
||||
2. Compare blob count in PG vs target backend
|
||||
3. Sample-verify N random blobs (download + BLAKE3 hash check)
|
||||
4. Show verification results in admin UI before allowing finalization
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Enterprise Extras
|
||||
|
||||
### Task 4.1: `CachedBlobBackend` — LRU local disk cache
|
||||
|
||||
**File**: `src/infrastructure/services/cached_blob_backend.rs` (NEW)
|
||||
|
||||
A `BlobStorageBackend` decorator for remote backends (S3, Azure) that caches hot blobs on local SSD:
|
||||
|
||||
```rust
|
||||
pub struct CachedBlobBackend {
|
||||
inner: Arc<dyn BlobStorageBackend>, // S3 backend
|
||||
cache_dir: PathBuf, // local cache directory
|
||||
max_cache_bytes: u64, // configurable limit
|
||||
index: Arc<RwLock<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
}
|
||||
|
||||
struct CacheEntry {
|
||||
size: u64,
|
||||
last_accessed: Instant,
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**:
|
||||
- **Reads**: Check local cache first → cache hit returns local file stream → cache miss downloads from inner backend, writes to cache, returns stream
|
||||
- **Writes**: `put_blob()` writes to inner backend AND local cache simultaneously
|
||||
- **Eviction**: LRU eviction when `current_size` exceeds `max_cache_bytes`
|
||||
- **Startup**: Scan cache directory to rebuild index
|
||||
|
||||
**Configuration** (admin panel):
|
||||
```
|
||||
storage.cache.enabled → "true" | "false"
|
||||
storage.cache.max_size_bytes → u64 (default: 50 GB)
|
||||
storage.cache.path → PathBuf (default: "{storage_root}/.cache")
|
||||
```
|
||||
|
||||
**Env vars**:
|
||||
```
|
||||
OXICLOUD_STORAGE_CACHE_ENABLED=true
|
||||
OXICLOUD_STORAGE_CACHE_MAX_SIZE=53687091200 # 50 GB
|
||||
OXICLOUD_STORAGE_CACHE_PATH=/fast-ssd/oxicloud-cache
|
||||
```
|
||||
|
||||
### Task 4.2: Client-side encryption (AES-256-GCM)
|
||||
|
||||
**File**: `src/infrastructure/services/encrypted_blob_backend.rs` (NEW)
|
||||
|
||||
Another `BlobStorageBackend` decorator that encrypts blobs before sending to the inner backend:
|
||||
|
||||
```rust
|
||||
pub struct EncryptedBlobBackend {
|
||||
inner: Arc<dyn BlobStorageBackend>,
|
||||
encryption_key: [u8; 32], // AES-256 key
|
||||
}
|
||||
```
|
||||
|
||||
**Dependency**: Add `aes-gcm = "0.10"` to Cargo.toml.
|
||||
|
||||
**Behavior**:
|
||||
- `put_blob()`: Read source → encrypt with AES-256-GCM (random 96-bit nonce prepended) → write encrypted to temp → `inner.put_blob(hash, encrypted_temp)`
|
||||
- `get_blob_stream()`: `inner.get_blob_stream()` → decrypt stream → return plaintext stream
|
||||
- **CRITICAL**: BLAKE3 hash is computed on the PLAINTEXT (before encryption), so dedup still works across encrypted backends
|
||||
- **Nonce storage**: Prepend 12-byte nonce to each encrypted blob (total overhead: 28 bytes per blob — 12 nonce + 16 GCM tag)
|
||||
|
||||
**Configuration** (admin panel):
|
||||
```
|
||||
storage.encryption.enabled → "true" | "false"
|
||||
storage.encryption.key → base64-encoded 32-byte key (is_secret: true)
|
||||
```
|
||||
|
||||
**Key generation**: Provide an admin API endpoint `POST /api/admin/settings/storage/generate-key` that generates a cryptographically secure key and returns it once (user must save it).
|
||||
|
||||
**WARNING in admin UI**: "If you lose the encryption key, all data in the storage backend is IRRECOVERABLY LOST. Back up this key securely."
|
||||
|
||||
### Task 4.3: Azure Blob Storage backend
|
||||
|
||||
**File**: `src/infrastructure/services/azure_blob_backend.rs` (NEW)
|
||||
|
||||
**Dependency**: `azure_storage_blobs = "0.21"`, `azure_storage = "0.21"`
|
||||
|
||||
Same `BlobStorageBackend` trait implementation targeting Azure Blob Storage:
|
||||
- Container = bucket equivalent
|
||||
- Blob key scheme: `{prefix}/{hash}.blob` (same as S3/local)
|
||||
- Authentication: Account Name + Account Key OR SAS token
|
||||
|
||||
**Configuration**:
|
||||
```
|
||||
OXICLOUD_AZURE_ACCOUNT_NAME
|
||||
OXICLOUD_AZURE_ACCOUNT_KEY
|
||||
OXICLOUD_AZURE_CONTAINER
|
||||
OXICLOUD_AZURE_SAS_TOKEN # alternative auth
|
||||
```
|
||||
|
||||
### Task 4.4: Bandwidth throttling & retry policies
|
||||
|
||||
Apply to all remote backends (S3, Azure):
|
||||
|
||||
**Throttling**:
|
||||
- Configurable upload/download bandwidth limit per-backend
|
||||
- Implemented via `tokio::time::sleep` between chunks
|
||||
- Admin configurable: `storage.s3.max_upload_bandwidth`, `storage.s3.max_download_bandwidth`
|
||||
|
||||
**Retry policy** (exponential backoff):
|
||||
```rust
|
||||
pub struct RetryPolicy {
|
||||
pub max_retries: u32, // default: 3
|
||||
pub initial_backoff_ms: u64, // default: 100
|
||||
pub max_backoff_ms: u64, // default: 10_000
|
||||
pub backoff_multiplier: f64, // default: 2.0
|
||||
}
|
||||
```
|
||||
|
||||
Wrap remote backend calls with retry logic for transient errors (network timeouts, 500s, 503s).
|
||||
|
||||
---
|
||||
|
||||
## Architecture Invariants (MUST be preserved)
|
||||
|
||||
1. **BLAKE3 hashing**: Always performed locally on temp files, never delegated to backend
|
||||
2. **PostgreSQL dedup index**: `storage.blobs` table remains the source of truth for ref_count, metadata
|
||||
3. **Write-first strategy**: Blob stored in backend BEFORE PostgreSQL upsert
|
||||
4. **Remove-after-commit**: Blob deleted from backend AFTER PostgreSQL transaction commits
|
||||
5. **Streaming reads**: All blob reads return `Pin<Box<dyn Stream>>`, never load full blob into memory
|
||||
6. **Zero framework deps in domain**: `BlobStorageBackend` trait lives in `application/ports/`, not infrastructure
|
||||
7. **DI via AppState**: All backends are `Arc`-wrapped and assembled in `common/di.rs`
|
||||
8. **Env var precedence**: `OXICLOUD_*` env vars always override DB-stored admin settings
|
||||
9. **Admin guard**: All storage admin endpoints require JWT with role `"admin"`
|
||||
10. **Existing tests**: All ~208 tests must pass after Phase 1 refactor
|
||||
|
||||
## Backend Decorator Composition
|
||||
|
||||
The backends compose as decorators. In `di.rs`, the assembly looks like:
|
||||
|
||||
```rust
|
||||
// Phase 1: Base backend
|
||||
let base_backend: Arc<dyn BlobStorageBackend> = match config {
|
||||
Local => Arc::new(LocalBlobBackend::new(...)),
|
||||
S3 => Arc::new(S3BlobBackend::new(...).await?),
|
||||
Azure => Arc::new(AzureBlobBackend::new(...).await?),
|
||||
};
|
||||
|
||||
// Phase 4: Optional encryption layer
|
||||
let backend = if encryption_enabled {
|
||||
Arc::new(EncryptedBlobBackend::new(base_backend, key))
|
||||
} else {
|
||||
base_backend
|
||||
};
|
||||
|
||||
// Phase 4: Optional cache layer (only for remote backends)
|
||||
let backend = if cache_enabled && !matches!(config, Local) {
|
||||
Arc::new(CachedBlobBackend::new(backend, cache_config))
|
||||
} else {
|
||||
backend
|
||||
};
|
||||
|
||||
// Phase 3: Optional migration wrapper
|
||||
let backend = if migration_in_progress {
|
||||
Arc::new(MigrationBlobBackend::new(old_backend, backend, state))
|
||||
} else {
|
||||
backend
|
||||
};
|
||||
|
||||
// Finally: DedupService uses the composed backend
|
||||
let dedup_service = Arc::new(DedupService::new(backend, pool, maintenance_pool));
|
||||
```
|
||||
|
||||
This decorator pattern means each feature (encryption, caching, migration) is:
|
||||
- Independently testable
|
||||
- Independently toggleable
|
||||
- Zero overhead when disabled
|
||||
- Composable in any order
|
||||
@@ -128,3 +128,97 @@ pub struct DashboardStatsDto {
|
||||
pub users_over_quota: i64,
|
||||
pub registration_enabled: bool,
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Storage Settings DTOs (Admin Panel)
|
||||
// ============================================================================
|
||||
|
||||
/// Current storage settings returned to admin UI (secrets masked)
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct StorageSettingsDto {
|
||||
/// Active backend type: "local" or "s3"
|
||||
pub backend: String,
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
/// True if an access key is configured (never reveals the actual value)
|
||||
pub s3_access_key_set: bool,
|
||||
/// True if a secret key is configured (never reveals the actual value)
|
||||
pub s3_secret_key_set: bool,
|
||||
pub s3_force_path_style: bool,
|
||||
/// Field names overridden by environment variables (read-only in UI)
|
||||
pub env_overrides: Vec<String>,
|
||||
// ── Current stats ──
|
||||
pub current_backend: String,
|
||||
pub total_blobs: u64,
|
||||
pub total_bytes_stored: u64,
|
||||
pub dedup_ratio: f64,
|
||||
}
|
||||
|
||||
/// Request body for saving storage settings from the admin panel
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct SaveStorageSettingsDto {
|
||||
pub backend: String,
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
/// Only update if provided and non-empty (None = keep existing)
|
||||
pub s3_access_key: Option<String>,
|
||||
/// Only update if provided and non-empty (None = keep existing)
|
||||
pub s3_secret_key: Option<String>,
|
||||
pub s3_force_path_style: Option<bool>,
|
||||
}
|
||||
|
||||
/// Request body for testing a storage connection
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct TestStorageConnectionDto {
|
||||
pub backend: String,
|
||||
pub s3_endpoint_url: Option<String>,
|
||||
pub s3_bucket: Option<String>,
|
||||
pub s3_region: Option<String>,
|
||||
pub s3_access_key: Option<String>,
|
||||
pub s3_secret_key: Option<String>,
|
||||
pub s3_force_path_style: Option<bool>,
|
||||
}
|
||||
|
||||
/// Result of a storage connection test
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct StorageTestResultDto {
|
||||
pub connected: bool,
|
||||
pub message: String,
|
||||
pub backend_type: String,
|
||||
pub available_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Migration DTOs (Admin Panel — Storage Migration)
|
||||
// ============================================================================
|
||||
|
||||
/// Migration progress returned by `GET /api/admin/storage/migration`.
|
||||
/// Re-exports the `MigrationState` shape for the admin UI.
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct MigrationStateDto {
|
||||
pub status: String,
|
||||
pub total_blobs: u64,
|
||||
pub migrated_blobs: u64,
|
||||
pub migrated_bytes: u64,
|
||||
pub failed_blobs: Vec<String>,
|
||||
pub started_at: Option<String>,
|
||||
pub completed_at: Option<String>,
|
||||
/// Estimated throughput in bytes/sec (for UI ETA calculation).
|
||||
pub throughput_bytes_per_sec: Option<f64>,
|
||||
}
|
||||
|
||||
/// Request body for `POST /api/admin/storage/migration/start`.
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct StartMigrationDto {
|
||||
/// How many blobs to copy in parallel (default: 4).
|
||||
pub concurrency: Option<usize>,
|
||||
}
|
||||
|
||||
/// Request body (empty) for `POST /api/admin/storage/migration/verify`.
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct VerifyMigrationDto {
|
||||
/// Number of random blobs to sample-check (default: 100).
|
||||
pub sample_size: Option<usize>,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
//! Blob Storage Backend Port — abstracts raw byte I/O for content-addressable storage.
|
||||
//!
|
||||
//! This trait decouples `DedupService` from any specific storage medium.
|
||||
//! Implementations include:
|
||||
//! - `LocalBlobBackend` — local filesystem (default)
|
||||
//! - `S3BlobBackend` — any S3-compatible service (AWS, Backblaze B2, MinIO, R2…)
|
||||
//!
|
||||
//! `DedupService` owns an `Arc<dyn BlobStorageBackend>` and delegates all
|
||||
//! byte-level I/O through this trait, keeping BLAKE3 hashing, ref-counting
|
||||
//! and PostgreSQL index logic in `DedupService` itself.
|
||||
|
||||
use bytes::Bytes;
|
||||
use futures::Stream;
|
||||
use serde::Serialize;
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Boxed future alias used by [`BlobStorageBackend`] to keep the trait dyn-compatible.
|
||||
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||
|
||||
/// Pinned boxed byte stream — the return type for blob reads.
|
||||
pub type BlobStream = Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>;
|
||||
|
||||
/// Health-check result returned by [`BlobStorageBackend::health_check`].
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct StorageHealthStatus {
|
||||
/// Whether the backend is reachable and functional.
|
||||
pub connected: bool,
|
||||
/// Human-readable backend identifier (e.g. `"local"`, `"s3"`).
|
||||
pub backend_type: String,
|
||||
/// Descriptive status message.
|
||||
pub message: String,
|
||||
/// Available space in bytes, if the backend can report it.
|
||||
pub available_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
|
||||
///
|
||||
/// Every method operates on a *hash key* that uniquely identifies a blob.
|
||||
/// The backend is responsible for mapping the hash to its own addressing
|
||||
/// scheme (filesystem path, S3 key, etc.).
|
||||
///
|
||||
/// Returns boxed futures so the trait is dyn-compatible (`Arc<dyn BlobStorageBackend>`).
|
||||
pub trait BlobStorageBackend: Send + Sync + 'static {
|
||||
/// Perform any one-time setup (create directories, verify bucket, etc.).
|
||||
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>>;
|
||||
|
||||
/// Store a blob from a local temporary file.
|
||||
///
|
||||
/// Must be **idempotent**: if the blob already exists the call succeeds
|
||||
/// without overwriting. Returns the number of bytes stored.
|
||||
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>>;
|
||||
|
||||
/// Stream the full blob content in chunks.
|
||||
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>>;
|
||||
|
||||
/// Stream a byte range of the blob (for HTTP Range requests / video seek).
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> BoxFut<'_, Result<BlobStream, DomainError>>;
|
||||
|
||||
/// Delete a blob by hash. Must be **idempotent** (no error if already gone).
|
||||
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>>;
|
||||
|
||||
/// Check if a blob exists in the backend.
|
||||
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>>;
|
||||
|
||||
/// Get blob size in bytes without downloading content.
|
||||
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>>;
|
||||
|
||||
/// Verify connectivity and permissions (used by the admin "Test Connection" button).
|
||||
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>>;
|
||||
|
||||
/// Return the backend type name for display (e.g. `"local"`, `"s3"`).
|
||||
fn backend_type(&self) -> &'static str;
|
||||
|
||||
/// Return the local filesystem path for a blob, if available.
|
||||
///
|
||||
/// Only meaningful for local-filesystem backends. Remote backends
|
||||
/// return `None`; callers that need a local file must stream + spool.
|
||||
fn local_blob_path(&self, hash: &str) -> Option<PathBuf>;
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
pub mod auth_ports;
|
||||
pub mod blob_storage_ports;
|
||||
pub mod cache_ports;
|
||||
pub mod calendar_ports;
|
||||
pub mod carddav_ports;
|
||||
|
||||
@@ -18,6 +18,7 @@ pub mod nextcloud_login_flow_service;
|
||||
pub mod recent_service;
|
||||
pub mod search_service;
|
||||
pub mod share_service;
|
||||
pub mod storage_settings_service;
|
||||
pub mod storage_usage_service;
|
||||
pub mod trash_service;
|
||||
pub mod wopi_lock_service;
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::application::dtos::settings_dto::{
|
||||
SaveStorageSettingsDto, StorageSettingsDto, StorageTestResultDto, TestStorageConnectionDto,
|
||||
};
|
||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use crate::common::config::{S3StorageConfig, StorageConfig};
|
||||
use crate::common::errors::{DomainError, ErrorKind};
|
||||
use crate::domain::repositories::settings_repository::SettingsRepository;
|
||||
use crate::infrastructure::repositories::pg::SettingsPgRepository;
|
||||
use crate::infrastructure::services::dedup_service::DedupService;
|
||||
use crate::infrastructure::services::s3_blob_backend::S3BlobBackend;
|
||||
|
||||
/// Storage settings service — manages storage backend configuration via the admin panel.
|
||||
///
|
||||
/// Configuration priority: **env vars > DB settings > defaults**.
|
||||
pub struct StorageSettingsService {
|
||||
settings_repo: Arc<SettingsPgRepository>,
|
||||
env_storage_config: StorageConfig,
|
||||
dedup_service: Arc<DedupService>,
|
||||
}
|
||||
|
||||
impl StorageSettingsService {
|
||||
pub fn new(
|
||||
settings_repo: Arc<SettingsPgRepository>,
|
||||
env_storage_config: StorageConfig,
|
||||
dedup_service: Arc<DedupService>,
|
||||
) -> Self {
|
||||
Self {
|
||||
settings_repo,
|
||||
env_storage_config,
|
||||
dedup_service,
|
||||
}
|
||||
}
|
||||
|
||||
/// Detect which storage fields are overridden by environment variables.
|
||||
fn get_env_overrides(&self) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
let vars = [
|
||||
("OXICLOUD_STORAGE_BACKEND", "backend"),
|
||||
("OXICLOUD_S3_ENDPOINT_URL", "s3_endpoint_url"),
|
||||
("OXICLOUD_S3_BUCKET", "s3_bucket"),
|
||||
("OXICLOUD_S3_REGION", "s3_region"),
|
||||
("OXICLOUD_S3_ACCESS_KEY", "s3_access_key"),
|
||||
("OXICLOUD_S3_SECRET_KEY", "s3_secret_key"),
|
||||
("OXICLOUD_S3_FORCE_PATH_STYLE", "s3_force_path_style"),
|
||||
];
|
||||
for (env_key, field_name) in &vars {
|
||||
if std::env::var(env_key).is_ok() {
|
||||
out.push(field_name.to_string());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Apply environment variable overrides on top of a config.
|
||||
fn apply_env_overrides(&self, config: &mut StorageConfig) {
|
||||
let e = &self.env_storage_config;
|
||||
if std::env::var("OXICLOUD_STORAGE_BACKEND").is_ok() {
|
||||
config.backend = e.backend.clone();
|
||||
}
|
||||
// S3 env overrides — only apply if S3 config exists in env
|
||||
if let Some(env_s3) = &e.s3 {
|
||||
let s3 = config.s3.get_or_insert_with(|| S3StorageConfig {
|
||||
endpoint_url: None,
|
||||
bucket: String::new(),
|
||||
region: "us-east-1".to_string(),
|
||||
access_key: String::new(),
|
||||
secret_key: String::new(),
|
||||
force_path_style: false,
|
||||
});
|
||||
if std::env::var("OXICLOUD_S3_ENDPOINT_URL").is_ok() {
|
||||
s3.endpoint_url = env_s3.endpoint_url.clone();
|
||||
}
|
||||
if std::env::var("OXICLOUD_S3_BUCKET").is_ok() {
|
||||
s3.bucket = env_s3.bucket.clone();
|
||||
}
|
||||
if std::env::var("OXICLOUD_S3_REGION").is_ok() {
|
||||
s3.region = env_s3.region.clone();
|
||||
}
|
||||
if std::env::var("OXICLOUD_S3_ACCESS_KEY").is_ok() {
|
||||
s3.access_key = env_s3.access_key.clone();
|
||||
}
|
||||
if std::env::var("OXICLOUD_S3_SECRET_KEY").is_ok() {
|
||||
s3.secret_key = env_s3.secret_key.clone();
|
||||
}
|
||||
if std::env::var("OXICLOUD_S3_FORCE_PATH_STYLE").is_ok() {
|
||||
s3.force_path_style = env_s3.force_path_style;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Load effective storage config: DB settings + env var overrides + defaults.
|
||||
pub async fn load_effective_storage_config(&self) -> Result<StorageConfig, DomainError> {
|
||||
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
|
||||
let d = StorageConfig::default();
|
||||
|
||||
let backend = db
|
||||
.get("storage.backend")
|
||||
.map(|v| match v.as_str() {
|
||||
"s3" => crate::common::config::StorageBackendType::S3,
|
||||
"azure" => crate::common::config::StorageBackendType::Azure,
|
||||
_ => crate::common::config::StorageBackendType::Local,
|
||||
})
|
||||
.unwrap_or(d.backend);
|
||||
|
||||
let s3 = {
|
||||
let bucket = db.get("storage.s3.bucket").cloned().unwrap_or_default();
|
||||
if bucket.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(S3StorageConfig {
|
||||
endpoint_url: db
|
||||
.get("storage.s3.endpoint_url")
|
||||
.cloned()
|
||||
.filter(|s| !s.is_empty()),
|
||||
bucket,
|
||||
region: db
|
||||
.get("storage.s3.region")
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "us-east-1".to_string()),
|
||||
access_key: db.get("storage.s3.access_key").cloned().unwrap_or_default(),
|
||||
secret_key: db.get("storage.s3.secret_key").cloned().unwrap_or_default(),
|
||||
force_path_style: db
|
||||
.get("storage.s3.force_path_style")
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(false),
|
||||
})
|
||||
}
|
||||
};
|
||||
|
||||
let mut config = StorageConfig {
|
||||
backend,
|
||||
s3,
|
||||
..self.env_storage_config.clone()
|
||||
};
|
||||
|
||||
self.apply_env_overrides(&mut config);
|
||||
Ok(config)
|
||||
}
|
||||
|
||||
/// Get storage settings for display in admin UI (secrets masked).
|
||||
pub async fn get_storage_settings(&self) -> Result<StorageSettingsDto, DomainError> {
|
||||
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
|
||||
|
||||
let has_access_key = db
|
||||
.get("storage.s3.access_key")
|
||||
.map(|s| !s.is_empty())
|
||||
.unwrap_or(false)
|
||||
|| std::env::var("OXICLOUD_S3_ACCESS_KEY")
|
||||
.map(|s| !s.is_empty())
|
||||
.unwrap_or(false);
|
||||
|
||||
let has_secret_key = db
|
||||
.get("storage.s3.secret_key")
|
||||
.map(|s| !s.is_empty())
|
||||
.unwrap_or(false)
|
||||
|| std::env::var("OXICLOUD_S3_SECRET_KEY")
|
||||
.map(|s| !s.is_empty())
|
||||
.unwrap_or(false);
|
||||
|
||||
let effective = self.load_effective_storage_config().await?;
|
||||
let stats = self.dedup_service.get_stats().await;
|
||||
let current_backend = self.dedup_service.backend().backend_type().to_string();
|
||||
|
||||
let backend_str = match effective.backend {
|
||||
crate::common::config::StorageBackendType::Local => "local",
|
||||
crate::common::config::StorageBackendType::S3 => "s3",
|
||||
crate::common::config::StorageBackendType::Azure => "azure",
|
||||
};
|
||||
|
||||
Ok(StorageSettingsDto {
|
||||
backend: backend_str.to_string(),
|
||||
s3_endpoint_url: effective.s3.as_ref().and_then(|s| s.endpoint_url.clone()),
|
||||
s3_bucket: effective.s3.as_ref().map(|s| s.bucket.clone()),
|
||||
s3_region: effective.s3.as_ref().map(|s| s.region.clone()),
|
||||
s3_access_key_set: has_access_key,
|
||||
s3_secret_key_set: has_secret_key,
|
||||
s3_force_path_style: effective.s3.as_ref().is_some_and(|s| s.force_path_style),
|
||||
env_overrides: self.get_env_overrides(),
|
||||
current_backend,
|
||||
total_blobs: stats.total_blobs,
|
||||
total_bytes_stored: stats.total_bytes_stored,
|
||||
dedup_ratio: stats.dedup_ratio,
|
||||
})
|
||||
}
|
||||
|
||||
/// Save storage settings to DB.
|
||||
pub async fn save_storage_settings(
|
||||
&self,
|
||||
dto: SaveStorageSettingsDto,
|
||||
updated_by: Uuid,
|
||||
) -> Result<(), DomainError> {
|
||||
let cat = "storage";
|
||||
let by = Some(updated_by);
|
||||
|
||||
self.settings_repo
|
||||
.set("storage.backend", &dto.backend, cat, false, by)
|
||||
.await?;
|
||||
|
||||
if let Some(ref v) = dto.s3_endpoint_url {
|
||||
self.settings_repo
|
||||
.set("storage.s3.endpoint_url", v, cat, false, by)
|
||||
.await?;
|
||||
}
|
||||
if let Some(ref v) = dto.s3_bucket {
|
||||
self.settings_repo
|
||||
.set("storage.s3.bucket", v, cat, false, by)
|
||||
.await?;
|
||||
}
|
||||
if let Some(ref v) = dto.s3_region {
|
||||
self.settings_repo
|
||||
.set("storage.s3.region", v, cat, false, by)
|
||||
.await?;
|
||||
}
|
||||
if let Some(ref v) = dto.s3_access_key
|
||||
&& !v.is_empty()
|
||||
{
|
||||
self.settings_repo
|
||||
.set("storage.s3.access_key", v, cat, true, by)
|
||||
.await?;
|
||||
}
|
||||
if let Some(ref v) = dto.s3_secret_key
|
||||
&& !v.is_empty()
|
||||
{
|
||||
self.settings_repo
|
||||
.set("storage.s3.secret_key", v, cat, true, by)
|
||||
.await?;
|
||||
}
|
||||
if let Some(v) = dto.s3_force_path_style {
|
||||
self.settings_repo
|
||||
.set(
|
||||
"storage.s3.force_path_style",
|
||||
&v.to_string(),
|
||||
cat,
|
||||
false,
|
||||
by,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
|
||||
tracing::info!("Storage settings saved by admin (backend={})", dto.backend);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test a storage connection by building a temporary backend and calling health_check().
|
||||
pub async fn test_storage_connection(
|
||||
&self,
|
||||
dto: TestStorageConnectionDto,
|
||||
) -> Result<StorageTestResultDto, DomainError> {
|
||||
match dto.backend.as_str() {
|
||||
"local" => {
|
||||
// Test local backend health via the current dedup service backend
|
||||
let status = self.dedup_service.backend().health_check().await?;
|
||||
Ok(StorageTestResultDto {
|
||||
connected: status.connected,
|
||||
message: status.message,
|
||||
backend_type: "local".to_string(),
|
||||
available_bytes: status.available_bytes,
|
||||
})
|
||||
}
|
||||
"s3" => {
|
||||
let bucket = dto.s3_bucket.as_deref().unwrap_or_default();
|
||||
if bucket.is_empty() {
|
||||
return Ok(StorageTestResultDto {
|
||||
connected: false,
|
||||
message: "S3 bucket name is required".to_string(),
|
||||
backend_type: "s3".to_string(),
|
||||
available_bytes: None,
|
||||
});
|
||||
}
|
||||
|
||||
// Build a temporary S3 backend from the DTO values,
|
||||
// falling back to existing DB/env config for missing fields.
|
||||
let effective = self.load_effective_storage_config().await.ok();
|
||||
let existing_s3 = effective.as_ref().and_then(|c| c.s3.as_ref());
|
||||
|
||||
let config = S3StorageConfig {
|
||||
endpoint_url: dto
|
||||
.s3_endpoint_url
|
||||
.clone()
|
||||
.or_else(|| existing_s3.and_then(|s| s.endpoint_url.clone())),
|
||||
bucket: bucket.to_string(),
|
||||
region: dto.s3_region.clone().unwrap_or_else(|| {
|
||||
existing_s3
|
||||
.map(|s| s.region.clone())
|
||||
.unwrap_or_else(|| "us-east-1".to_string())
|
||||
}),
|
||||
access_key: dto
|
||||
.s3_access_key
|
||||
.clone()
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| {
|
||||
existing_s3
|
||||
.map(|s| s.access_key.clone())
|
||||
.unwrap_or_default()
|
||||
}),
|
||||
secret_key: dto
|
||||
.s3_secret_key
|
||||
.clone()
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| {
|
||||
existing_s3
|
||||
.map(|s| s.secret_key.clone())
|
||||
.unwrap_or_default()
|
||||
}),
|
||||
force_path_style: dto
|
||||
.s3_force_path_style
|
||||
.unwrap_or_else(|| existing_s3.is_some_and(|s| s.force_path_style)),
|
||||
};
|
||||
|
||||
let backend = S3BlobBackend::new(&config);
|
||||
match backend.health_check().await {
|
||||
Ok(status) => Ok(StorageTestResultDto {
|
||||
connected: status.connected,
|
||||
message: status.message,
|
||||
backend_type: "s3".to_string(),
|
||||
available_bytes: status.available_bytes,
|
||||
}),
|
||||
Err(e) => Ok(StorageTestResultDto {
|
||||
connected: false,
|
||||
message: format!("Connection failed: {}", e),
|
||||
backend_type: "s3".to_string(),
|
||||
available_bytes: None,
|
||||
}),
|
||||
}
|
||||
}
|
||||
other => Err(DomainError::new(
|
||||
ErrorKind::InvalidInput,
|
||||
"Storage",
|
||||
format!("Unknown backend type: {}", other),
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -211,6 +211,127 @@ pub struct StorageConfig {
|
||||
/// Maximum upload file size in bytes (default: 10 GB).
|
||||
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
|
||||
pub max_upload_size: usize,
|
||||
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
|
||||
pub backend: StorageBackendType,
|
||||
/// S3-compatible backend configuration (used when `backend == S3`).
|
||||
pub s3: Option<S3StorageConfig>,
|
||||
/// Azure Blob Storage configuration (used when `backend == Azure`).
|
||||
pub azure: Option<AzureStorageConfig>,
|
||||
/// Local disk cache for remote backends.
|
||||
pub cache: BlobCacheConfig,
|
||||
/// Client-side encryption.
|
||||
pub encryption: EncryptionConfig,
|
||||
/// Retry policy for remote backends.
|
||||
pub retry: RetryConfig,
|
||||
}
|
||||
|
||||
/// Which blob storage backend to use.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub enum StorageBackendType {
|
||||
/// Local filesystem (default).
|
||||
#[default]
|
||||
Local,
|
||||
/// Any S3-compatible object store (AWS, Backblaze B2, R2, MinIO, …).
|
||||
S3,
|
||||
/// Azure Blob Storage.
|
||||
Azure,
|
||||
}
|
||||
|
||||
/// Configuration for an S3-compatible blob storage backend.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct S3StorageConfig {
|
||||
/// Custom endpoint URL (required for non-AWS providers).
|
||||
pub endpoint_url: Option<String>,
|
||||
/// S3 bucket name.
|
||||
pub bucket: String,
|
||||
/// AWS region (default: `us-east-1`).
|
||||
pub region: String,
|
||||
/// Access key ID.
|
||||
pub access_key: String,
|
||||
/// Secret access key.
|
||||
pub secret_key: String,
|
||||
/// Force path-style access (required for MinIO, R2, some providers).
|
||||
pub force_path_style: bool,
|
||||
}
|
||||
|
||||
/// Configuration for Azure Blob Storage.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AzureStorageConfig {
|
||||
/// Azure storage account name.
|
||||
pub account_name: String,
|
||||
/// Azure storage account key.
|
||||
pub account_key: String,
|
||||
/// Container name.
|
||||
pub container: String,
|
||||
/// Optional SAS token (alternative to account key).
|
||||
pub sas_token: Option<String>,
|
||||
}
|
||||
|
||||
/// LRU local disk cache configuration for remote blob backends.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BlobCacheConfig {
|
||||
/// Enable the LRU disk cache (only useful for remote backends).
|
||||
pub enabled: bool,
|
||||
/// Maximum cache size in bytes (default: 50 GB).
|
||||
pub max_size_bytes: u64,
|
||||
/// Cache directory path (default: `{root_dir}/.blob-cache`).
|
||||
pub cache_path: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for BlobCacheConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
max_size_bytes: 50 * 1024 * 1024 * 1024, // 50 GB
|
||||
cache_path: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Client-side encryption configuration.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct EncryptionConfig {
|
||||
/// Enable AES-256-GCM encryption for blobs at rest.
|
||||
pub enabled: bool,
|
||||
/// Base64-encoded 32-byte encryption key.
|
||||
pub key_base64: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for EncryptionConfig {
|
||||
#[allow(clippy::derivable_impls)]
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
key_base64: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Retry policy configuration for remote backends.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RetryConfig {
|
||||
/// Enable retry with exponential backoff.
|
||||
pub enabled: bool,
|
||||
/// Maximum number of retry attempts.
|
||||
pub max_retries: u32,
|
||||
/// Initial backoff in milliseconds.
|
||||
pub initial_backoff_ms: u64,
|
||||
/// Maximum backoff in milliseconds.
|
||||
pub max_backoff_ms: u64,
|
||||
/// Backoff multiplier.
|
||||
pub backoff_multiplier: f64,
|
||||
}
|
||||
|
||||
impl Default for RetryConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: true,
|
||||
max_retries: 3,
|
||||
initial_backoff_ms: 100,
|
||||
max_backoff_ms: 10_000,
|
||||
backoff_multiplier: 2.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for StorageConfig {
|
||||
@@ -227,6 +348,12 @@ impl Default for StorageConfig {
|
||||
parallel_threshold: 100 * 1024 * 1024, // 100 MB
|
||||
trash_retention_days: 30, // 30 days
|
||||
max_upload_size: MAX_UPLOAD_SIZE,
|
||||
backend: StorageBackendType::Local,
|
||||
s3: None,
|
||||
azure: None,
|
||||
cache: BlobCacheConfig::default(),
|
||||
encryption: EncryptionConfig::default(),
|
||||
retry: RetryConfig::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -828,6 +955,95 @@ impl AppConfig {
|
||||
config.storage.max_upload_size = val;
|
||||
}
|
||||
|
||||
// Storage backend selection
|
||||
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
|
||||
match backend.to_lowercase().as_str() {
|
||||
"s3" => config.storage.backend = StorageBackendType::S3,
|
||||
"azure" => config.storage.backend = StorageBackendType::Azure,
|
||||
_ => config.storage.backend = StorageBackendType::Local,
|
||||
}
|
||||
}
|
||||
|
||||
// S3-compatible storage configuration
|
||||
if config.storage.backend == StorageBackendType::S3 {
|
||||
let bucket = env::var("OXICLOUD_S3_BUCKET").unwrap_or_default();
|
||||
if bucket.is_empty() {
|
||||
tracing::warn!("OXICLOUD_STORAGE_BACKEND=s3 but OXICLOUD_S3_BUCKET is not set");
|
||||
}
|
||||
config.storage.s3 = Some(S3StorageConfig {
|
||||
endpoint_url: env::var("OXICLOUD_S3_ENDPOINT_URL").ok(),
|
||||
bucket,
|
||||
region: env::var("OXICLOUD_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
|
||||
access_key: env::var("OXICLOUD_S3_ACCESS_KEY").unwrap_or_default(),
|
||||
secret_key: env::var("OXICLOUD_S3_SECRET_KEY").unwrap_or_default(),
|
||||
force_path_style: env::var("OXICLOUD_S3_FORCE_PATH_STYLE")
|
||||
.map(|v| v.parse::<bool>().unwrap_or(false))
|
||||
.unwrap_or(false),
|
||||
});
|
||||
}
|
||||
|
||||
// Azure Blob Storage configuration
|
||||
if config.storage.backend == StorageBackendType::Azure {
|
||||
let container = env::var("OXICLOUD_AZURE_CONTAINER").unwrap_or_default();
|
||||
if container.is_empty() {
|
||||
tracing::warn!(
|
||||
"OXICLOUD_STORAGE_BACKEND=azure but OXICLOUD_AZURE_CONTAINER is not set"
|
||||
);
|
||||
}
|
||||
config.storage.azure = Some(AzureStorageConfig {
|
||||
account_name: env::var("OXICLOUD_AZURE_ACCOUNT_NAME").unwrap_or_default(),
|
||||
account_key: env::var("OXICLOUD_AZURE_ACCOUNT_KEY").unwrap_or_default(),
|
||||
container,
|
||||
sas_token: env::var("OXICLOUD_AZURE_SAS_TOKEN").ok(),
|
||||
});
|
||||
}
|
||||
|
||||
// Blob cache configuration
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_ENABLED") {
|
||||
config.storage.cache.enabled = v.parse::<bool>().unwrap_or(false);
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_MAX_SIZE")
|
||||
&& let Ok(bytes) = v.parse::<u64>()
|
||||
{
|
||||
config.storage.cache.max_size_bytes = bytes;
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_PATH") {
|
||||
config.storage.cache.cache_path = Some(v);
|
||||
}
|
||||
|
||||
// Encryption configuration
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_ENABLED") {
|
||||
config.storage.encryption.enabled = v.parse::<bool>().unwrap_or(false);
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_KEY") {
|
||||
config.storage.encryption.key_base64 = Some(v);
|
||||
}
|
||||
|
||||
// Retry configuration
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_ENABLED") {
|
||||
config.storage.retry.enabled = v.parse::<bool>().unwrap_or(true);
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_RETRIES")
|
||||
&& let Ok(n) = v.parse::<u32>()
|
||||
{
|
||||
config.storage.retry.max_retries = n;
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_INITIAL_BACKOFF_MS")
|
||||
&& let Ok(n) = v.parse::<u64>()
|
||||
{
|
||||
config.storage.retry.initial_backoff_ms = n;
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_BACKOFF_MS")
|
||||
&& let Ok(n) = v.parse::<u64>()
|
||||
{
|
||||
config.storage.retry.max_backoff_ms = n;
|
||||
}
|
||||
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_BACKOFF_MULTIPLIER")
|
||||
&& let Ok(n) = v.parse::<f64>()
|
||||
{
|
||||
config.storage.retry.backoff_multiplier = n;
|
||||
}
|
||||
|
||||
// OIDC configuration
|
||||
if let Ok(v) = env::var("OXICLOUD_OIDC_ENABLED") {
|
||||
config.oidc.enabled = v.parse::<bool>().unwrap_or(false);
|
||||
|
||||
+118
-1
@@ -2,10 +2,14 @@ use sqlx::PgPool;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use crate::common::config::StorageBackendType;
|
||||
use crate::infrastructure::db::DbPools;
|
||||
|
||||
use crate::application::services::admin_settings_service::AdminSettingsService;
|
||||
use crate::application::services::auth_application_service::AuthApplicationService;
|
||||
use crate::application::services::storage_settings_service::StorageSettingsService;
|
||||
use crate::infrastructure::services::migration_blob_backend::MigrationState;
|
||||
|
||||
use crate::application::ports::file_ports::FileUseCaseFactory;
|
||||
use crate::application::services::favorites_service::FavoritesService;
|
||||
@@ -153,10 +157,110 @@ impl AppServiceFactory {
|
||||
);
|
||||
image_transcode_service.initialize().await?;
|
||||
|
||||
// Build blob storage backend based on configuration
|
||||
let base_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
|
||||
StorageBackendType::S3 => {
|
||||
let s3_config = self
|
||||
.config
|
||||
.storage
|
||||
.s3
|
||||
.as_ref()
|
||||
.expect("S3 config required when OXICLOUD_STORAGE_BACKEND=s3");
|
||||
Arc::new(
|
||||
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3_config),
|
||||
)
|
||||
}
|
||||
StorageBackendType::Azure => {
|
||||
let az_config = self
|
||||
.config
|
||||
.storage
|
||||
.azure
|
||||
.as_ref()
|
||||
.expect("Azure config required when OXICLOUD_STORAGE_BACKEND=azure");
|
||||
Arc::new(
|
||||
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(
|
||||
az_config,
|
||||
),
|
||||
)
|
||||
}
|
||||
StorageBackendType::Local => Arc::new(
|
||||
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
|
||||
&self.storage_path,
|
||||
),
|
||||
),
|
||||
};
|
||||
|
||||
// Stack decorators: retry → encryption → cache (inner-to-outer)
|
||||
let mut blob_backend: Arc<dyn BlobStorageBackend> = base_backend;
|
||||
|
||||
// Retry decorator (for remote backends)
|
||||
if self.config.storage.retry.enabled
|
||||
&& self.config.storage.backend != StorageBackendType::Local
|
||||
{
|
||||
use crate::infrastructure::services::retry_blob_backend::{
|
||||
RetryBlobBackend, RetryPolicy,
|
||||
};
|
||||
let policy = RetryPolicy {
|
||||
max_retries: self.config.storage.retry.max_retries,
|
||||
initial_backoff: std::time::Duration::from_millis(
|
||||
self.config.storage.retry.initial_backoff_ms,
|
||||
),
|
||||
max_backoff: std::time::Duration::from_millis(
|
||||
self.config.storage.retry.max_backoff_ms,
|
||||
),
|
||||
backoff_multiplier: self.config.storage.retry.backoff_multiplier,
|
||||
};
|
||||
blob_backend = Arc::new(RetryBlobBackend::new(blob_backend, policy));
|
||||
tracing::info!("Blob storage retry decorator enabled");
|
||||
}
|
||||
|
||||
// Encryption decorator
|
||||
if self.config.storage.encryption.enabled {
|
||||
use crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend;
|
||||
let key_b64 = self
|
||||
.config
|
||||
.storage
|
||||
.encryption
|
||||
.key_base64
|
||||
.as_ref()
|
||||
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY required when encryption is enabled");
|
||||
let key_bytes =
|
||||
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, key_b64)
|
||||
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY must be valid base64");
|
||||
let key: [u8; 32] = key_bytes.try_into().expect(
|
||||
"OXICLOUD_STORAGE_ENCRYPTION_KEY must be exactly 32 bytes (base64 of 32 bytes)",
|
||||
);
|
||||
blob_backend = Arc::new(EncryptedBlobBackend::new(blob_backend, &key));
|
||||
tracing::info!("Blob storage encryption decorator enabled (AES-256-GCM)");
|
||||
}
|
||||
|
||||
// Cache decorator (for remote backends only)
|
||||
if self.config.storage.cache.enabled
|
||||
&& self.config.storage.backend != StorageBackendType::Local
|
||||
{
|
||||
use crate::infrastructure::services::cached_blob_backend::{
|
||||
BlobCacheConfig as CacheCfg, CachedBlobBackend,
|
||||
};
|
||||
let cache_path = self
|
||||
.config
|
||||
.storage
|
||||
.cache
|
||||
.cache_path
|
||||
.as_ref()
|
||||
.map(std::path::PathBuf::from)
|
||||
.unwrap_or_else(|| self.storage_path.join(".blob-cache"));
|
||||
let cfg = CacheCfg {
|
||||
cache_dir: cache_path,
|
||||
max_cache_bytes: self.config.storage.cache.max_size_bytes,
|
||||
};
|
||||
blob_backend = Arc::new(CachedBlobBackend::new(blob_backend, &cfg));
|
||||
tracing::info!("Blob storage LRU disk cache enabled");
|
||||
}
|
||||
|
||||
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
|
||||
let dedup_service = Arc::new(
|
||||
crate::infrastructure::services::dedup_service::DedupService::new(
|
||||
&self.storage_path,
|
||||
blob_backend,
|
||||
db_pool.clone(),
|
||||
maintenance_pool.clone(),
|
||||
),
|
||||
@@ -631,6 +735,8 @@ impl AppServiceFactory {
|
||||
auth_service: auth_services,
|
||||
nextcloud: nextcloud_services,
|
||||
admin_settings_service: None,
|
||||
storage_settings_service: None,
|
||||
migration_state: Arc::new(tokio::sync::RwLock::new(MigrationState::default())),
|
||||
trash_service,
|
||||
share_service,
|
||||
favorites_service,
|
||||
@@ -700,6 +806,15 @@ impl AppServiceFactory {
|
||||
|
||||
app_state.admin_settings_service = Some(admin_svc.clone());
|
||||
|
||||
// 9b-1b. Wire storage settings service (reuses same settings_repo)
|
||||
let storage_settings_svc = Arc::new(StorageSettingsService::new(
|
||||
settings_repo.clone(),
|
||||
self.config.storage.clone(),
|
||||
app_state.core.dedup_service.clone(),
|
||||
));
|
||||
app_state.storage_settings_service = Some(storage_settings_svc);
|
||||
tracing::info!("Storage settings service initialized");
|
||||
|
||||
// 9b-2. Log whether system needs first-time admin setup
|
||||
if !admin_svc.is_system_initialized().await {
|
||||
tracing::warn!("╔══════════════════════════════════════════════════════════╗");
|
||||
@@ -928,6 +1043,8 @@ pub struct AppState {
|
||||
pub auth_service: Option<AuthServices>,
|
||||
pub nextcloud: Option<NextcloudServices>,
|
||||
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
||||
pub storage_settings_service: Option<Arc<StorageSettingsService>>,
|
||||
pub migration_state: Arc<tokio::sync::RwLock<MigrationState>>,
|
||||
pub trash_service: Option<Arc<TrashService>>,
|
||||
pub share_service: Option<Arc<ShareService>>,
|
||||
pub favorites_service: Option<Arc<FavoritesService>>,
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
//! Azure Blob Storage Backend — stores blobs in an Azure Storage container.
|
||||
//!
|
||||
//! Authenticates via Account Name + Account Key (or SAS token).
|
||||
//! Blob key scheme mirrors local/S3: `{2-char-prefix}/{hash}.blob`.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
|
||||
use azure_storage::StorageCredentials;
|
||||
use azure_storage_blobs::prelude::*;
|
||||
use bytes::Bytes;
|
||||
use futures::StreamExt;
|
||||
use tokio::fs;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::common::config::AzureStorageConfig;
|
||||
use crate::domain::errors::{DomainError, ErrorKind};
|
||||
|
||||
/// Azure Blob Storage backend.
|
||||
pub struct AzureBlobBackend {
|
||||
container_client: ContainerClient,
|
||||
container_name: String,
|
||||
}
|
||||
|
||||
impl AzureBlobBackend {
|
||||
/// Build a new Azure backend from configuration.
|
||||
pub fn new(config: &AzureStorageConfig) -> Self {
|
||||
let credentials = if let Some(ref sas) = config.sas_token {
|
||||
StorageCredentials::sas_token(sas).expect("Invalid SAS token")
|
||||
} else {
|
||||
StorageCredentials::access_key(&config.account_name, config.account_key.clone())
|
||||
};
|
||||
|
||||
let container_client = ClientBuilder::new(&config.account_name, credentials)
|
||||
.container_client(&config.container);
|
||||
|
||||
Self {
|
||||
container_client,
|
||||
container_name: config.container.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the blob name for a given hash.
|
||||
fn blob_name(hash: &str) -> String {
|
||||
let prefix = &hash[0..2];
|
||||
format!("{prefix}/{hash}.blob")
|
||||
}
|
||||
|
||||
/// Get a `BlobClient` for a given hash.
|
||||
fn blob_client(&self, hash: &str) -> BlobClient {
|
||||
self.container_client.blob_client(Self::blob_name(hash))
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for AzureBlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
Box::pin(async move {
|
||||
// Verify container exists by getting its properties
|
||||
self.container_client.get_properties().await.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"Azure",
|
||||
format!("Cannot access container '{}': {}", self.container_name, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
tracing::info!(
|
||||
"Azure blob backend initialized: container={}",
|
||||
self.container_name
|
||||
);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
let source_path = source_path.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
|
||||
// Check if blob already exists (idempotent)
|
||||
if client.get_properties().await.is_ok() {
|
||||
let file_size = fs::metadata(&source_path)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"Azure",
|
||||
format!("Failed to stat source file: {e}"),
|
||||
)
|
||||
})?
|
||||
.len();
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
return Ok(file_size);
|
||||
}
|
||||
|
||||
// Read file and upload as block blob
|
||||
let data = fs::read(&source_path).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to read source: {e}"))
|
||||
})?;
|
||||
let file_size = data.len() as u64;
|
||||
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
Ok(file_size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
|
||||
let mut result_data: Vec<u8> = Vec::new();
|
||||
let mut stream = client.get().into_stream();
|
||||
|
||||
while let Some(response) = stream.next().await {
|
||||
let response = response.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Azure",
|
||||
format!("Failed to get blob {hash}: {e}"),
|
||||
)
|
||||
})?;
|
||||
let mut body = response.data;
|
||||
while let Some(chunk) = body.next().await {
|
||||
let chunk = chunk.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Stream read error: {e}"))
|
||||
})?;
|
||||
result_data.extend_from_slice(&chunk);
|
||||
}
|
||||
}
|
||||
|
||||
let stream: BlobStream = Box::pin(futures::stream::once(async move {
|
||||
Ok(Bytes::from(result_data))
|
||||
}));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
|
||||
let range = match end {
|
||||
Some(e) => azure_core::request_options::Range::new(start, e),
|
||||
None => azure_core::request_options::Range::new(start, u64::MAX),
|
||||
};
|
||||
|
||||
let mut result_data: Vec<u8> = Vec::new();
|
||||
let mut stream = client.get().range(range).into_stream();
|
||||
|
||||
while let Some(response) = stream.next().await {
|
||||
let response = response.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Azure",
|
||||
format!("Failed to get blob range {hash}: {e}"),
|
||||
)
|
||||
})?;
|
||||
let mut body = response.data;
|
||||
while let Some(chunk) = body.next().await {
|
||||
let chunk = chunk.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"Azure",
|
||||
format!("Stream range read error: {e}"),
|
||||
)
|
||||
})?;
|
||||
result_data.extend_from_slice(&chunk);
|
||||
}
|
||||
}
|
||||
|
||||
let stream: BlobStream = Box::pin(futures::stream::once(async move {
|
||||
Ok(Bytes::from(result_data))
|
||||
}));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
|
||||
// Azure delete is not fully idempotent — 404 is expected for missing blobs
|
||||
match client.delete().await {
|
||||
Ok(_) => Ok(()),
|
||||
Err(e) => {
|
||||
// If 404, treat as success (idempotent)
|
||||
let status = e.as_http_error().map(|h| h.status());
|
||||
if status == Some(azure_core::StatusCode::NotFound) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(DomainError::internal_error(
|
||||
"Azure",
|
||||
format!("Failed to delete blob {hash}: {e}"),
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
match client.get_properties().await {
|
||||
Ok(_) => Ok(true),
|
||||
Err(e) => {
|
||||
let status = e.as_http_error().map(|h| h.status());
|
||||
if status == Some(azure_core::StatusCode::NotFound) {
|
||||
Ok(false)
|
||||
} else {
|
||||
Err(DomainError::internal_error(
|
||||
"Azure",
|
||||
format!("Failed to check blob {hash}: {e}"),
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
let props = client.get_properties().await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Azure",
|
||||
format!("Failed to stat blob {hash}: {e}"),
|
||||
)
|
||||
})?;
|
||||
Ok(props.blob.properties.content_length)
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
Box::pin(async move {
|
||||
match self.container_client.get_properties().await {
|
||||
Ok(_) => Ok(StorageHealthStatus {
|
||||
connected: true,
|
||||
backend_type: "azure".to_string(),
|
||||
message: format!("Azure container '{}' is accessible", self.container_name),
|
||||
available_bytes: None,
|
||||
}),
|
||||
Err(e) => Ok(StorageHealthStatus {
|
||||
connected: false,
|
||||
backend_type: "azure".to_string(),
|
||||
message: format!(
|
||||
"Azure container '{}' is not accessible: {}",
|
||||
self.container_name, e
|
||||
),
|
||||
available_bytes: None,
|
||||
}),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"azure"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||
None
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,472 @@
|
||||
//! `CachedBlobBackend` — LRU local-disk cache decorator for remote blob backends.
|
||||
//!
|
||||
//! Wraps any `BlobStorageBackend` (typically S3 or Azure) and transparently
|
||||
//! caches hot blobs on a local SSD. Reads check the cache first; cache misses
|
||||
//! are fetched from the inner backend and written to the cache. Writes go to
|
||||
//! the inner backend AND the local cache simultaneously.
|
||||
//!
|
||||
//! Eviction is LRU based on a configurable maximum disk budget.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use lru::LruCache;
|
||||
use std::num::NonZeroUsize;
|
||||
use tokio::fs;
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Chunk size for streaming cached file reads (256 KB).
|
||||
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||
|
||||
// ── Configuration ──────────────────────────────────────────────────
|
||||
|
||||
/// Configuration for the LRU disk cache.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BlobCacheConfig {
|
||||
/// Directory where cached blobs are stored.
|
||||
pub cache_dir: PathBuf,
|
||||
/// Maximum total cache size in bytes.
|
||||
pub max_cache_bytes: u64,
|
||||
}
|
||||
|
||||
// ── Cache entry ────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct CacheEntry {
|
||||
size: u64,
|
||||
}
|
||||
|
||||
// ── CachedBlobBackend ──────────────────────────────────────────────
|
||||
|
||||
/// A `BlobStorageBackend` decorator that adds an LRU disk cache in front of
|
||||
/// a remote backend.
|
||||
pub struct CachedBlobBackend {
|
||||
inner: Arc<dyn BlobStorageBackend>,
|
||||
cache_dir: PathBuf,
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
}
|
||||
|
||||
impl CachedBlobBackend {
|
||||
/// Create a new cached backend wrapping `inner`.
|
||||
pub fn new(inner: Arc<dyn BlobStorageBackend>, config: &BlobCacheConfig) -> Self {
|
||||
Self {
|
||||
inner,
|
||||
cache_dir: config.cache_dir.clone(),
|
||||
max_cache_bytes: config.max_cache_bytes,
|
||||
// Capacity is essentially unbounded — eviction is by byte budget, not count.
|
||||
index: Arc::new(Mutex::new(LruCache::new(
|
||||
NonZeroUsize::new(1_000_000).unwrap(),
|
||||
))),
|
||||
current_size: Arc::new(AtomicU64::new(0)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Path where a blob is cached locally.
|
||||
fn cached_path(&self, hash: &str) -> PathBuf {
|
||||
let prefix = &hash[..2.min(hash.len())];
|
||||
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for CachedBlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let index = self.index.clone();
|
||||
let current_size = self.current_size.clone();
|
||||
Box::pin(async move {
|
||||
inner.initialize().await?;
|
||||
|
||||
// Create cache dir structure (256 prefix dirs)
|
||||
fs::create_dir_all(&cache_dir).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("mkdir cache_dir: {e}"))
|
||||
})?;
|
||||
|
||||
// Scan existing cache to rebuild index
|
||||
let mut total_bytes = 0u64;
|
||||
let mut idx = index.lock().await;
|
||||
if let Ok(mut read_dir) = fs::read_dir(&cache_dir).await {
|
||||
while let Ok(Some(prefix_entry)) = read_dir.next_entry().await {
|
||||
if !prefix_entry.path().is_dir() {
|
||||
continue;
|
||||
}
|
||||
if let Ok(mut sub_dir) = fs::read_dir(prefix_entry.path()).await {
|
||||
while let Ok(Some(entry)) = sub_dir.next_entry().await {
|
||||
let path = entry.path();
|
||||
if path.extension().and_then(|e| e.to_str()) == Some("blob")
|
||||
&& let Some(stem) = path.file_stem().and_then(|s| s.to_str())
|
||||
{
|
||||
let size = fs::metadata(&path).await.map(|m| m.len()).unwrap_or(0);
|
||||
idx.put(stem.to_string(), CacheEntry { size });
|
||||
total_bytes += size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
drop(idx);
|
||||
current_size.store(total_bytes, Ordering::Relaxed);
|
||||
tracing::info!(
|
||||
"Blob cache initialized: {} bytes in cache at {}",
|
||||
total_bytes,
|
||||
cache_dir.display()
|
||||
);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let source = source_path.to_path_buf();
|
||||
let self_ref = CachedRef {
|
||||
cache_dir: self.cache_dir.clone(),
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
// Write to inner backend
|
||||
let bytes = inner.put_blob(&hash, &source).await?;
|
||||
// Also cache locally (best-effort)
|
||||
let _ = self_ref.insert_into_cache_static(&hash, &source).await;
|
||||
Ok(bytes)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_string();
|
||||
let cached = self.cached_path(&hash);
|
||||
let index = self.index.clone();
|
||||
let inner = self.inner.clone();
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache
|
||||
{
|
||||
let mut idx = index.lock().await;
|
||||
if idx.get(&hash).is_some() {
|
||||
if let Ok(file) = fs::File::open(&cached).await {
|
||||
let stream: BlobStream =
|
||||
Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
|
||||
return Ok(stream);
|
||||
}
|
||||
// Cache entry stale — remove
|
||||
if let Some(entry) = idx.pop(&hash) {
|
||||
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch from inner, spool to cache
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let file = fs::File::open(&dest).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
|
||||
})?;
|
||||
let stream: BlobStream = Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_string();
|
||||
let cached = self.cached_path(&hash);
|
||||
let index = self.index.clone();
|
||||
let inner = self.inner.clone();
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
Box::pin(async move {
|
||||
// Try cache first
|
||||
{
|
||||
let mut idx = index.lock().await;
|
||||
if idx.get(&hash).is_some() {
|
||||
if let Ok(mut file) = fs::File::open(&cached).await {
|
||||
file.seek(std::io::SeekFrom::Start(start))
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("seek: {e}"))
|
||||
})?;
|
||||
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
|
||||
let limited = file.take(take_len);
|
||||
let stream: BlobStream =
|
||||
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
|
||||
return Ok(stream);
|
||||
}
|
||||
if let Some(entry) = idx.pop(&hash) {
|
||||
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch full blob into cache, then serve range
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let mut file = fs::File::open(&dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
|
||||
file.seek(std::io::SeekFrom::Start(start))
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("seek: {e}")))?;
|
||||
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
|
||||
let limited = file.take(take_len);
|
||||
let stream: BlobStream =
|
||||
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let cached = self.cached_path(&hash);
|
||||
let index = self.index.clone();
|
||||
let current_size = self.current_size.clone();
|
||||
Box::pin(async move {
|
||||
inner.delete_blob(&hash).await?;
|
||||
// Remove from cache
|
||||
let mut idx = index.lock().await;
|
||||
if let Some(entry) = idx.pop(&hash) {
|
||||
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||
}
|
||||
let _ = fs::remove_file(&cached).await;
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let index = self.index.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache first (fast)
|
||||
{
|
||||
let mut idx = index.lock().await;
|
||||
if idx.get(&hash).is_some() {
|
||||
return Ok(true);
|
||||
}
|
||||
}
|
||||
inner.blob_exists(&hash).await
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let index = self.index.clone();
|
||||
let cached = self.cached_path(&hash);
|
||||
Box::pin(async move {
|
||||
// Check cache
|
||||
{
|
||||
let mut idx = index.lock().await;
|
||||
if let Some(entry) = idx.get(&hash) {
|
||||
return Ok(entry.size);
|
||||
}
|
||||
}
|
||||
// Fallback to cached file on disk (in case index was lost)
|
||||
if let Ok(meta) = fs::metadata(&cached).await {
|
||||
return Ok(meta.len());
|
||||
}
|
||||
inner.blob_size(&hash).await
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
let inner = self.inner.clone();
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let current_size = self.current_size.clone();
|
||||
let max_bytes = self.max_cache_bytes;
|
||||
Box::pin(async move {
|
||||
let mut status = inner.health_check().await?;
|
||||
let used = current_size.load(Ordering::Relaxed);
|
||||
status.message = format!(
|
||||
"{} | Cache: {}/{} bytes used at {}",
|
||||
status.message,
|
||||
used,
|
||||
max_bytes,
|
||||
cache_dir.display()
|
||||
);
|
||||
status.backend_type = format!("cached({})", status.backend_type);
|
||||
Ok(status)
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"cached"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||
// If the blob is cached locally, return that path
|
||||
let path = self.cached_path(hash);
|
||||
if path.exists() { Some(path) } else { None }
|
||||
}
|
||||
}
|
||||
|
||||
// ── Helper struct for owned references in async closures ───────────
|
||||
|
||||
/// Cloneable set of cache internals — avoids borrow issues in boxed futures.
|
||||
struct CachedRef {
|
||||
cache_dir: PathBuf,
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
}
|
||||
|
||||
impl CachedRef {
|
||||
fn cached_path(&self, hash: &str) -> PathBuf {
|
||||
let prefix = &hash[..2.min(hash.len())];
|
||||
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
|
||||
async fn insert_into_cache_static(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Result<(), DomainError> {
|
||||
let dest = self.cached_path(hash);
|
||||
if let Some(parent) = dest.parent() {
|
||||
fs::create_dir_all(parent).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
|
||||
})?;
|
||||
}
|
||||
|
||||
let size = fs::metadata(source_path)
|
||||
.await
|
||||
.map(|m| m.len())
|
||||
.unwrap_or(0);
|
||||
|
||||
fs::copy(source_path, &dest).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("cache copy failed: {e}"))
|
||||
})?;
|
||||
|
||||
let mut idx = self.index.lock().await;
|
||||
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size }) {
|
||||
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
|
||||
}
|
||||
self.current_size.fetch_add(size, Ordering::Relaxed);
|
||||
|
||||
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
|
||||
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
|
||||
self.current_size
|
||||
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
|
||||
let evicted_path = self.cached_path(&evicted_hash);
|
||||
let _ = fs::remove_file(&evicted_path).await;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn fetch_and_cache_static(
|
||||
&self,
|
||||
hash: &str,
|
||||
inner: &dyn BlobStorageBackend,
|
||||
) -> Result<PathBuf, DomainError> {
|
||||
let stream = inner.get_blob_stream(hash).await?;
|
||||
|
||||
let dest = self.cached_path(hash);
|
||||
if let Some(parent) = dest.parent() {
|
||||
fs::create_dir_all(parent).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
|
||||
})?;
|
||||
}
|
||||
|
||||
let tmp = dest.with_extension("tmp");
|
||||
let mut file = fs::File::create(&tmp)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
|
||||
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut total = 0u64;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||
})?;
|
||||
total += bytes.len() as u64;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
drop(file);
|
||||
|
||||
fs::rename(&tmp, &dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
|
||||
|
||||
let mut idx = self.index.lock().await;
|
||||
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size: total }) {
|
||||
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
|
||||
}
|
||||
self.current_size.fetch_add(total, Ordering::Relaxed);
|
||||
|
||||
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
|
||||
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
|
||||
self.current_size
|
||||
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
|
||||
let evicted_path = self.cached_path(&evicted_hash);
|
||||
let _ = fs::remove_file(&evicted_path).await;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(dest)
|
||||
}
|
||||
}
|
||||
@@ -39,24 +39,22 @@ use sqlx::PgPool;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use tokio::fs::{self, File};
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt};
|
||||
use tokio_util::io::ReaderStream;
|
||||
use tokio::fs;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use crate::application::ports::dedup_ports::{
|
||||
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
||||
};
|
||||
use crate::domain::errors::{DomainError, ErrorKind};
|
||||
|
||||
/// Chunk size for streaming file reads (256 KB)
|
||||
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||
|
||||
/// Content-Addressable Storage Service (PostgreSQL-backed)
|
||||
///
|
||||
/// Delegates all byte-level I/O to a [`BlobStorageBackend`] implementation
|
||||
/// (local filesystem, S3, etc.) while keeping BLAKE3 hashing, ref-counting
|
||||
/// and the PostgreSQL dedup index here.
|
||||
pub struct DedupService {
|
||||
/// Root directory for blob storage on the filesystem
|
||||
blob_root: PathBuf,
|
||||
/// Root directory for temporary files during upload
|
||||
temp_root: PathBuf,
|
||||
/// Pluggable blob storage backend (local FS, S3, …).
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
/// PostgreSQL connection pool (dedup index in `storage.blobs`) — primary,
|
||||
/// used by request-path operations (store_from_file, etc.).
|
||||
pool: Arc<PgPool>,
|
||||
@@ -65,39 +63,19 @@ pub struct DedupService {
|
||||
maintenance_pool: Arc<PgPool>,
|
||||
}
|
||||
|
||||
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
|
||||
/// Avoids a `format!("{:02x}", i)` allocation on every iteration of `initialize()`.
|
||||
static HEX_PREFIXES: [&str; 256] = [
|
||||
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
|
||||
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
|
||||
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
|
||||
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
|
||||
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
|
||||
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
|
||||
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
|
||||
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
|
||||
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
|
||||
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
|
||||
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
|
||||
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
|
||||
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
|
||||
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
|
||||
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
|
||||
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
|
||||
];
|
||||
|
||||
impl DedupService {
|
||||
/// Create a new dedup service backed by PostgreSQL.
|
||||
///
|
||||
/// * `backend` — pluggable blob storage (local filesystem, S3, etc.).
|
||||
/// * `pool` — primary pool for request-path operations.
|
||||
/// * `maintenance_pool` — isolated pool for verify_integrity / garbage_collect.
|
||||
pub fn new(storage_root: &Path, pool: Arc<PgPool>, maintenance_pool: Arc<PgPool>) -> Self {
|
||||
let blob_root = storage_root.join(".blobs");
|
||||
let temp_root = storage_root.join(".dedup_temp");
|
||||
|
||||
pub fn new(
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
pool: Arc<PgPool>,
|
||||
maintenance_pool: Arc<PgPool>,
|
||||
) -> Self {
|
||||
Self {
|
||||
blob_root,
|
||||
temp_root,
|
||||
backend,
|
||||
pool,
|
||||
maintenance_pool,
|
||||
}
|
||||
@@ -106,6 +84,7 @@ impl DedupService {
|
||||
/// Creates a stub instance for testing — never hits PG or the filesystem.
|
||||
#[cfg(any(test, feature = "integration_tests"))]
|
||||
pub fn new_stub() -> Self {
|
||||
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||
let stub_pool = Arc::new(
|
||||
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
|
||||
.max_connections(1)
|
||||
@@ -113,29 +92,15 @@ impl DedupService {
|
||||
.unwrap(),
|
||||
);
|
||||
Self {
|
||||
blob_root: std::path::PathBuf::from("/tmp/oxicloud_stub_blobs"),
|
||||
temp_root: std::path::PathBuf::from("/tmp/oxicloud_stub_temp"),
|
||||
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
|
||||
pool: stub_pool.clone(),
|
||||
maintenance_pool: stub_pool,
|
||||
}
|
||||
}
|
||||
|
||||
/// Initialize the service (create blob directories on the filesystem).
|
||||
/// Initialize the service (delegate to backend + log stats from PG).
|
||||
pub async fn initialize(&self) -> Result<(), DomainError> {
|
||||
// Create directories
|
||||
fs::create_dir_all(&self.blob_root)
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
fs::create_dir_all(&self.temp_root)
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
|
||||
// Create hash prefix directories (00-ff)
|
||||
for prefix in &HEX_PREFIXES {
|
||||
fs::create_dir_all(self.blob_root.join(prefix))
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
}
|
||||
self.backend.initialize().await?;
|
||||
|
||||
// Log existing blob stats from PG
|
||||
let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||
@@ -150,7 +115,8 @@ impl DedupService {
|
||||
.unwrap_or(0);
|
||||
|
||||
tracing::info!(
|
||||
"Dedup service initialized (PostgreSQL-backed): {} blobs, {} bytes stored",
|
||||
"Dedup service initialized (backend={}): {} blobs, {} bytes stored",
|
||||
self.backend.backend_type(),
|
||||
count,
|
||||
total_bytes
|
||||
);
|
||||
@@ -158,12 +124,18 @@ impl DedupService {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Return a reference to the underlying blob storage backend.
|
||||
pub fn backend(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||
&self.backend
|
||||
}
|
||||
|
||||
// ── Path helpers ─────────────────────────────────────────────
|
||||
|
||||
/// Get the blob path for a given hash.
|
||||
/// Get the local blob path for a given hash (if the backend supports it).
|
||||
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||
let prefix = &hash[0..2];
|
||||
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
||||
self.backend
|
||||
.local_blob_path(hash)
|
||||
.unwrap_or_else(|| PathBuf::from(format!("remote://{}", hash)))
|
||||
}
|
||||
|
||||
// ── Hash helpers ─────────────────────────────────────────────
|
||||
@@ -193,9 +165,9 @@ impl DedupService {
|
||||
|
||||
/// Store content with deduplication (streaming from file).
|
||||
///
|
||||
/// **Write-first strategy**: the source file is moved/copied to the
|
||||
/// blob store *before* touching PostgreSQL, so the PG connection is
|
||||
/// never held during disk I/O.
|
||||
/// **Write-first strategy**: the source file is moved/uploaded to the
|
||||
/// blob backend *before* touching PostgreSQL, so the PG connection is
|
||||
/// never held during I/O.
|
||||
///
|
||||
/// If `pre_computed_hash` is `Some`, the file will NOT be re-read for
|
||||
/// BLAKE3 — saving one full sequential read (the biggest I/O win).
|
||||
@@ -205,13 +177,6 @@ impl DedupService {
|
||||
content_type: Option<String>,
|
||||
pre_computed_hash: Option<String>,
|
||||
) -> Result<DedupResultDto, DomainError> {
|
||||
let file_size = fs::metadata(source_path)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("Dedup", format!("Failed to get file metadata: {}", e))
|
||||
})?
|
||||
.len();
|
||||
|
||||
// Use pre-computed hash if available, otherwise calculate (streaming)
|
||||
let hash = match pre_computed_hash {
|
||||
Some(h) => h,
|
||||
@@ -220,44 +185,11 @@ impl DedupService {
|
||||
.map_err(DomainError::from)?,
|
||||
};
|
||||
|
||||
// ── Phase 1: Place blob in backend (NO PG connection held) ───
|
||||
let file_size = self.backend.put_blob(&hash, source_path).await?;
|
||||
|
||||
let blob_path = self.blob_path(&hash);
|
||||
|
||||
// ── Phase 1: Move/place blob on disk (NO PG connection held) ─
|
||||
//
|
||||
// If the blob file already exists on disk, the source is simply
|
||||
// deleted — the file content is identical by definition.
|
||||
if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||
// Blob already on disk — discard the source file
|
||||
let _ = fs::remove_file(source_path).await;
|
||||
} else {
|
||||
// Parent directory (xx/) guaranteed to exist — created by initialize()
|
||||
|
||||
// rename is atomic on the same filesystem. If source and blob
|
||||
// dirs live on different filesystems (rare), this falls back to
|
||||
// copy+delete which is slower but still correct.
|
||||
if let Err(e) = fs::rename(source_path, &blob_path).await {
|
||||
if e.raw_os_error() == Some(18) {
|
||||
// EXDEV: cross-device link — fall back to copy+delete
|
||||
fs::copy(source_path, &blob_path).await.map_err(|ce| {
|
||||
DomainError::internal_error(
|
||||
"Dedup",
|
||||
format!("Failed to copy file to blob store: {}", ce),
|
||||
)
|
||||
})?;
|
||||
let _ = fs::remove_file(source_path).await;
|
||||
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||
// Another writer may have placed the blob concurrently
|
||||
let _ = fs::remove_file(source_path).await;
|
||||
tracing::debug!("Blob file placed by concurrent writer: {}", e);
|
||||
} else {
|
||||
return Err(DomainError::internal_error(
|
||||
"Dedup",
|
||||
format!("Failed to move file to blob store: {}", e),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase 2: Single atomic upsert (~2-4 ms, no explicit TX) ─
|
||||
let ref_count: i32 = sqlx::query_scalar(
|
||||
"INSERT INTO storage.blobs (hash, size, ref_count, content_type)
|
||||
@@ -415,10 +347,9 @@ impl DedupService {
|
||||
DomainError::internal_error("Dedup", format!("Failed to commit: {}", e))
|
||||
})?;
|
||||
|
||||
// Delete blob file AFTER committing PG — the row is gone, so no
|
||||
// concurrent store_from_file can resurrect a reference to this hash.
|
||||
let blob_path = self.blob_path(hash);
|
||||
if let Err(e) = fs::remove_file(&blob_path).await {
|
||||
// Delete blob from backend AFTER committing PG — the row is gone,
|
||||
// so no concurrent store_from_file can resurrect a reference.
|
||||
if let Err(e) = self.backend.delete_blob(hash).await {
|
||||
tracing::warn!("Failed to delete blob file {}: {}", hash, e);
|
||||
}
|
||||
|
||||
@@ -449,31 +380,16 @@ impl DedupService {
|
||||
|
||||
// ── Read operations ──────────────────────────────────────────
|
||||
|
||||
/// Stream blob content in 64 KB chunks — constant memory (~64 KB per stream).
|
||||
///
|
||||
/// A 1 GB file uses the same ~64 KB as a 1 KB file.
|
||||
/// Stream blob content in chunks — constant memory usage.
|
||||
pub async fn read_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
let blob_path = self.blob_path(hash);
|
||||
let file = File::open(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to open blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(Box::pin(ReaderStream::with_capacity(
|
||||
file,
|
||||
STREAM_CHUNK_SIZE,
|
||||
)))
|
||||
self.backend.get_blob_stream(hash).await
|
||||
}
|
||||
|
||||
/// Stream a byte range of a blob — only reads the requested portion.
|
||||
///
|
||||
/// Uses seek + take so a 1 MB range request on a 1 GB file only reads 1 MB.
|
||||
pub async fn read_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
@@ -481,49 +397,12 @@ impl DedupService {
|
||||
end: Option<u64>,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
let blob_path = self.blob_path(hash);
|
||||
let mut file = File::open(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to open blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
// Seek to the start position
|
||||
file.seek(std::io::SeekFrom::Start(start))
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
|
||||
})?;
|
||||
|
||||
// If an end is specified, limit the read with take()
|
||||
if let Some(end_pos) = end {
|
||||
let limit = end_pos.saturating_sub(start);
|
||||
let limited = file.take(limit);
|
||||
Ok(Box::pin(ReaderStream::with_capacity(
|
||||
limited,
|
||||
STREAM_CHUNK_SIZE,
|
||||
)))
|
||||
} else {
|
||||
Ok(Box::pin(ReaderStream::with_capacity(
|
||||
file,
|
||||
STREAM_CHUNK_SIZE,
|
||||
)))
|
||||
}
|
||||
self.backend.get_blob_range_stream(hash, start, end).await
|
||||
}
|
||||
|
||||
/// Get the size of a blob without reading its content.
|
||||
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
|
||||
let blob_path = self.blob_path(hash);
|
||||
let meta = fs::metadata(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to stat blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(meta.len())
|
||||
self.backend.blob_size(hash).await
|
||||
}
|
||||
|
||||
// ── Statistics (computed from PG) ────────────────────────────
|
||||
@@ -596,56 +475,46 @@ impl DedupService {
|
||||
|
||||
// Flush when batch is full or we've exhausted the cursor
|
||||
if batch.len() >= VERIFY_CONCURRENCY || (is_done && !batch.is_empty()) {
|
||||
let blob_root = self.blob_root.clone();
|
||||
let backend = self.backend.clone();
|
||||
let current_batch =
|
||||
std::mem::replace(&mut batch, Vec::with_capacity(VERIFY_CONCURRENCY));
|
||||
|
||||
let issues: Vec<String> = stream::iter(current_batch)
|
||||
.map(move |(hash, expected_size)| {
|
||||
let blob_root = blob_root.clone();
|
||||
let backend = backend.clone();
|
||||
async move {
|
||||
let prefix = &hash[0..2];
|
||||
let blob_path = blob_root.join(prefix).join(format!("{}.blob", hash));
|
||||
|
||||
let mut issues = Vec::new();
|
||||
|
||||
// Single async metadata() replaces the previous
|
||||
// blocking .exists() + separate metadata() — one
|
||||
// stat() syscall instead of two, and non-blocking.
|
||||
let file_meta = match fs::metadata(&blob_path).await {
|
||||
Ok(m) => m,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
issues.push(format!("{}: file missing on disk", hash));
|
||||
return issues;
|
||||
// Check existence + size via backend
|
||||
match backend.blob_size(&hash).await {
|
||||
Ok(actual_size) => {
|
||||
if actual_size != expected_size as u64 {
|
||||
issues.push(format!(
|
||||
"{}: size mismatch (expected: {}, actual: {})",
|
||||
hash, expected_size, actual_size,
|
||||
));
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
issues.push(format!("{}: metadata error ({})", hash, e));
|
||||
Err(_) => {
|
||||
issues.push(format!("{}: blob missing in backend", hash));
|
||||
return issues;
|
||||
}
|
||||
};
|
||||
|
||||
// Check size
|
||||
if file_meta.len() != expected_size as u64 {
|
||||
issues.push(format!(
|
||||
"{}: size mismatch (expected: {}, actual: {})",
|
||||
hash,
|
||||
expected_size,
|
||||
file_meta.len(),
|
||||
));
|
||||
}
|
||||
|
||||
// Verify hash
|
||||
match Self::hash_file(&blob_path).await {
|
||||
Ok(actual_hash) => {
|
||||
if actual_hash != hash {
|
||||
issues.push(format!(
|
||||
"{}: hash mismatch (actual: {})",
|
||||
hash, actual_hash,
|
||||
));
|
||||
// Verify hash — only possible for local backends
|
||||
if let Some(blob_path) = backend.local_blob_path(&hash) {
|
||||
match Self::hash_file(&blob_path).await {
|
||||
Ok(actual_hash) => {
|
||||
if actual_hash != hash {
|
||||
issues.push(format!(
|
||||
"{}: hash mismatch (actual: {})",
|
||||
hash, actual_hash,
|
||||
));
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
issues.push(format!("{}: read error ({})", hash, e));
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
issues.push(format!("{}: read error ({})", hash, e));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -717,20 +586,20 @@ impl DedupService {
|
||||
// Also clean up any thumbnail files for these blob hashes
|
||||
// (thumbnails are keyed by blob_hash and live under
|
||||
// storage_root/.thumbnails/{icon,preview,large}/{hash}.jpg).
|
||||
let thumbnails_root = self
|
||||
.blob_root
|
||||
.parent()
|
||||
.unwrap_or(&self.blob_root)
|
||||
.join(".thumbnails");
|
||||
|
||||
for (hash, size) in &batch {
|
||||
let blob_path = self.blob_path(hash);
|
||||
if let Err(e) = fs::remove_file(&blob_path).await {
|
||||
if let Err(e) = self.backend.delete_blob(hash).await {
|
||||
tracing::warn!("Failed to delete orphan blob file {hash}: {e}");
|
||||
}
|
||||
// Remove associated thumbnail files (best-effort)
|
||||
for dir in &["icon", "preview", "large"] {
|
||||
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
|
||||
let _ = fs::remove_file(&thumb).await;
|
||||
// Remove associated thumbnail files (best-effort, always local)
|
||||
if let Some(blob_path) = self.backend.local_blob_path(hash)
|
||||
&& let Some(storage_root) = blob_path.ancestors().nth(3)
|
||||
{
|
||||
let thumbnails_root = storage_root.join(".thumbnails");
|
||||
for dir in &["icon", "preview", "large"] {
|
||||
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
|
||||
let _ = fs::remove_file(&thumb).await;
|
||||
}
|
||||
}
|
||||
total_bytes += *size as u64;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
//! `EncryptedBlobBackend` — AES-256-GCM encryption decorator for blob storage.
|
||||
//!
|
||||
//! Transparently encrypts blobs before they reach the inner backend and
|
||||
//! decrypts them on read. Each blob gets a random 96-bit nonce which is
|
||||
//! prepended to the ciphertext. The GCM authentication tag (16 bytes) is
|
||||
//! appended by the cipher.
|
||||
//!
|
||||
//! **IMPORTANT**: BLAKE3 hashing is performed on the *plaintext* by
|
||||
//! `DedupService` before this layer sees the blob, so content-addressable
|
||||
//! dedup still works correctly.
|
||||
//!
|
||||
//! Layout on disk/S3: `[12-byte nonce][ciphertext + 16-byte GCM tag]`
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
|
||||
use aes_gcm::aead::{Aead, KeyInit, OsRng};
|
||||
use aes_gcm::{AeadCore, Aes256Gcm, Nonce};
|
||||
use bytes::Bytes;
|
||||
use std::sync::Arc;
|
||||
use tokio::fs;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Nonce size for AES-256-GCM (96 bits = 12 bytes).
|
||||
const NONCE_SIZE: usize = 12;
|
||||
|
||||
/// `BlobStorageBackend` decorator that encrypts blobs at rest.
|
||||
pub struct EncryptedBlobBackend {
|
||||
inner: Arc<dyn BlobStorageBackend>,
|
||||
cipher: Aes256Gcm,
|
||||
}
|
||||
|
||||
impl EncryptedBlobBackend {
|
||||
/// Create a new encryption layer wrapping `inner`.
|
||||
///
|
||||
/// `key` must be exactly 32 bytes (AES-256).
|
||||
pub fn new(inner: Arc<dyn BlobStorageBackend>, key: &[u8; 32]) -> Self {
|
||||
let cipher = Aes256Gcm::new_from_slice(key).expect("AES-256 key must be 32 bytes");
|
||||
Self { inner, cipher }
|
||||
}
|
||||
|
||||
/// Generate a random 32-byte key suitable for AES-256.
|
||||
pub fn generate_key() -> [u8; 32] {
|
||||
use aes_gcm::aead::rand_core::RngCore;
|
||||
let mut key = [0u8; 32];
|
||||
OsRng.fill_bytes(&mut key);
|
||||
key
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for EncryptedBlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
self.inner.initialize()
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let source = source_path.to_path_buf();
|
||||
// Clone cipher key material (Aes256Gcm is not Send-safe to move across await)
|
||||
let cipher = self.cipher.clone();
|
||||
Box::pin(async move {
|
||||
// Read plaintext from source
|
||||
let plaintext = fs::read(&source).await.map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("read source: {e}"))
|
||||
})?;
|
||||
|
||||
// Encrypt: nonce || ciphertext (includes GCM tag)
|
||||
let nonce = Aes256Gcm::generate_nonce(&mut OsRng);
|
||||
let ciphertext = cipher.encrypt(&nonce, plaintext.as_ref()).map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("encrypt failed: {e}"))
|
||||
})?;
|
||||
|
||||
// Write encrypted blob to a temp file
|
||||
let tmp = source.with_extension("enc.tmp");
|
||||
let mut file = fs::File::create(&tmp).await.map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("create tmp: {e}"))
|
||||
})?;
|
||||
file.write_all(nonce.as_slice()).await.map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("write nonce: {e}"))
|
||||
})?;
|
||||
file.write_all(&ciphertext).await.map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("write ciphertext: {e}"))
|
||||
})?;
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Encryption", format!("flush: {e}")))?;
|
||||
drop(file);
|
||||
|
||||
let result = inner.put_blob(&hash, &tmp).await;
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
result
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let cipher = self.cipher.clone();
|
||||
Box::pin(async move {
|
||||
// Read entire encrypted blob (nonce + ciphertext) into memory for decryption
|
||||
let enc_stream = inner.get_blob_stream(&hash).await?;
|
||||
let encrypted = collect_stream(enc_stream).await?;
|
||||
|
||||
if encrypted.len() < NONCE_SIZE {
|
||||
return Err(DomainError::internal_error(
|
||||
"Encryption",
|
||||
"encrypted blob too short (missing nonce)",
|
||||
));
|
||||
}
|
||||
|
||||
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
|
||||
let nonce = Nonce::from_slice(nonce_bytes);
|
||||
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
|
||||
})?;
|
||||
|
||||
let stream: BlobStream =
|
||||
Box::pin(futures::stream::once(
|
||||
async move { Ok(Bytes::from(plaintext)) },
|
||||
));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
let cipher = self.cipher.clone();
|
||||
Box::pin(async move {
|
||||
// Must decrypt the full blob then slice the plaintext range
|
||||
let enc_stream = inner.get_blob_stream(&hash).await?;
|
||||
let encrypted = collect_stream(enc_stream).await?;
|
||||
|
||||
if encrypted.len() < NONCE_SIZE {
|
||||
return Err(DomainError::internal_error(
|
||||
"Encryption",
|
||||
"encrypted blob too short",
|
||||
));
|
||||
}
|
||||
|
||||
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
|
||||
let nonce = Nonce::from_slice(nonce_bytes);
|
||||
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
|
||||
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
|
||||
})?;
|
||||
|
||||
let start = start as usize;
|
||||
let end = end.map(|e| (e as usize) + 1).unwrap_or(plaintext.len());
|
||||
let end = end.min(plaintext.len());
|
||||
let start = start.min(end);
|
||||
|
||||
let slice = Bytes::from(plaintext[start..end].to_vec());
|
||||
let stream: BlobStream = Box::pin(futures::stream::once(async move { Ok(slice) }));
|
||||
Ok(stream)
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
self.inner.delete_blob(hash)
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
self.inner.blob_exists(hash)
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
// The stored size includes nonce + GCM tag overhead.
|
||||
// Return the *plaintext* size by subtracting overhead.
|
||||
let inner = self.inner.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
let encrypted_size = inner.blob_size(&hash).await?;
|
||||
// overhead = 12 (nonce) + 16 (GCM tag) = 28 bytes
|
||||
Ok(encrypted_size.saturating_sub(28))
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
let inner = self.inner.clone();
|
||||
Box::pin(async move {
|
||||
let mut status = inner.health_check().await?;
|
||||
status.backend_type = format!("encrypted({})", status.backend_type);
|
||||
status.message = format!("{} | Encryption: AES-256-GCM", status.message);
|
||||
Ok(status)
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"encrypted"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||
// Encrypted blobs cannot be served directly from disk
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Collect a byte stream into a single `Vec<u8>`.
|
||||
async fn collect_stream(stream: BlobStream) -> Result<Vec<u8>, DomainError> {
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut buf = Vec::new();
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk
|
||||
.map_err(|e| DomainError::internal_error("Encryption", format!("stream read: {e}")))?;
|
||||
buf.extend_from_slice(&bytes);
|
||||
}
|
||||
Ok(buf)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||
use tempfile::TempDir;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_encrypt_decrypt_roundtrip() {
|
||||
let tmp = TempDir::new().unwrap();
|
||||
let blob_dir = tmp.path().join("blobs");
|
||||
let local = Arc::new(LocalBlobBackend::new(&blob_dir));
|
||||
local.initialize().await.unwrap();
|
||||
|
||||
let key = EncryptedBlobBackend::generate_key();
|
||||
let encrypted = EncryptedBlobBackend::new(local, &key);
|
||||
|
||||
// Write a test blob
|
||||
let data = b"Hello, encrypted world!";
|
||||
let source = tmp.path().join("test.tmp");
|
||||
let mut f = fs::File::create(&source).await.unwrap();
|
||||
f.write_all(data).await.unwrap();
|
||||
f.flush().await.unwrap();
|
||||
drop(f);
|
||||
|
||||
let hash = "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890";
|
||||
encrypted.put_blob(hash, &source).await.unwrap();
|
||||
|
||||
// Read back via stream
|
||||
let stream = encrypted.get_blob_stream(hash).await.unwrap();
|
||||
let decrypted = collect_stream(stream).await.unwrap();
|
||||
assert_eq!(decrypted, data);
|
||||
|
||||
// Read range
|
||||
let range_stream = encrypted
|
||||
.get_blob_range_stream(hash, 7, Some(15))
|
||||
.await
|
||||
.unwrap();
|
||||
let range_data = collect_stream(range_stream).await.unwrap();
|
||||
assert_eq!(range_data, b"encrypted");
|
||||
|
||||
// Size should reflect plaintext
|
||||
let size = encrypted.blob_size(hash).await.unwrap();
|
||||
assert_eq!(size, data.len() as u64);
|
||||
|
||||
// Exists
|
||||
assert!(encrypted.blob_exists(hash).await.unwrap());
|
||||
|
||||
// Delete
|
||||
encrypted.delete_blob(hash).await.unwrap();
|
||||
assert!(!encrypted.blob_exists(hash).await.unwrap());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,277 @@
|
||||
//! Local Filesystem Blob Backend — stores blobs under `.blobs/{prefix}/{hash}.blob`.
|
||||
//!
|
||||
//! This is the default backend and a direct extraction of the filesystem I/O
|
||||
//! that previously lived inside `DedupService`.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use tokio::fs::{self, File};
|
||||
use tokio::io::AsyncSeekExt;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::domain::errors::{DomainError, ErrorKind};
|
||||
|
||||
/// Chunk size for streaming file reads (256 KB).
|
||||
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||
|
||||
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
|
||||
static HEX_PREFIXES: [&str; 256] = [
|
||||
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
|
||||
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
|
||||
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
|
||||
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
|
||||
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
|
||||
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
|
||||
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
|
||||
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
|
||||
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
|
||||
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
|
||||
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
|
||||
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
|
||||
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
|
||||
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
|
||||
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
|
||||
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
|
||||
];
|
||||
|
||||
/// Local filesystem blob backend.
|
||||
///
|
||||
/// Blobs are stored under `blob_root/{2-char-prefix}/{hash}.blob`.
|
||||
/// Temporary upload staging uses `temp_root/`.
|
||||
pub struct LocalBlobBackend {
|
||||
blob_root: PathBuf,
|
||||
temp_root: PathBuf,
|
||||
}
|
||||
|
||||
impl LocalBlobBackend {
|
||||
/// Create a new local backend rooted at `storage_root`.
|
||||
///
|
||||
/// Blob files go under `{storage_root}/.blobs/`, temp files under
|
||||
/// `{storage_root}/.dedup_temp/`.
|
||||
pub fn new(storage_root: &Path) -> Self {
|
||||
Self {
|
||||
blob_root: storage_root.join(".blobs"),
|
||||
temp_root: storage_root.join(".dedup_temp"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the filesystem path for a blob hash.
|
||||
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||
let prefix = &hash[0..2];
|
||||
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
||||
}
|
||||
|
||||
/// Return a reference to the blob root directory.
|
||||
pub fn blob_root(&self) -> &Path {
|
||||
&self.blob_root
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for LocalBlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
Box::pin(async move {
|
||||
fs::create_dir_all(&self.blob_root)
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
fs::create_dir_all(&self.temp_root)
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
|
||||
// Create the 256 hash-prefix directories (00-ff)
|
||||
for prefix in &HEX_PREFIXES {
|
||||
fs::create_dir_all(self.blob_root.join(prefix))
|
||||
.await
|
||||
.map_err(DomainError::from)?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
let source_path = source_path.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
|
||||
let file_size = fs::metadata(&source_path)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"Blob",
|
||||
format!("Failed to stat source file: {}", e),
|
||||
)
|
||||
})?
|
||||
.len();
|
||||
|
||||
// Idempotent: if blob already exists, just remove the source
|
||||
if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
return Ok(file_size);
|
||||
}
|
||||
|
||||
// Atomic rename (same filesystem). Falls back to copy+delete for
|
||||
// cross-device moves (EXDEV errno 18).
|
||||
if let Err(e) = fs::rename(&source_path, &blob_path).await {
|
||||
if e.raw_os_error() == Some(18) {
|
||||
// EXDEV — cross-device link
|
||||
fs::copy(&source_path, &blob_path).await.map_err(|ce| {
|
||||
DomainError::internal_error(
|
||||
"Blob",
|
||||
format!("Failed to copy file to blob store: {}", ce),
|
||||
)
|
||||
})?;
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||
// Concurrent writer placed the blob — discard our copy
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
tracing::debug!("Blob placed by concurrent writer: {}", e);
|
||||
} else {
|
||||
return Err(DomainError::internal_error(
|
||||
"Blob",
|
||||
format!("Failed to move file to blob store: {}", e),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
Ok(file_size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
let file = File::open(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to open blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
let mut file = File::open(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to open blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
file.seek(std::io::SeekFrom::Start(start))
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
|
||||
})?;
|
||||
|
||||
if let Some(end_pos) = end {
|
||||
use tokio::io::AsyncReadExt;
|
||||
let limit = end_pos.saturating_sub(start);
|
||||
let limited = file.take(limit);
|
||||
Ok(Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||
} else {
|
||||
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
match fs::remove_file(&blob_path).await {
|
||||
Ok(()) => Ok(()),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), // idempotent
|
||||
Err(e) => Err(DomainError::internal_error(
|
||||
"Blob",
|
||||
format!("Failed to delete blob {}: {}", hash, e),
|
||||
)),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
Ok(fs::try_exists(&blob_path).await.unwrap_or(false))
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let blob_path = self.blob_path(&hash);
|
||||
let meta = fs::metadata(&blob_path).await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Blob",
|
||||
format!("Failed to stat blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(meta.len())
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
Box::pin(async move {
|
||||
let writable = fs::metadata(&self.blob_root).await.is_ok();
|
||||
Ok(StorageHealthStatus {
|
||||
connected: writable,
|
||||
backend_type: "local".to_string(),
|
||||
message: if writable {
|
||||
"Local filesystem is accessible".to_string()
|
||||
} else {
|
||||
"Blob root directory is not accessible".to_string()
|
||||
},
|
||||
available_bytes: None,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"local"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||
Some(self.blob_path(hash))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
//! `MigrationBlobBackend` — decorator that enables zero-downtime migration
|
||||
//! between blob storage backends.
|
||||
//!
|
||||
//! During a migration the decorator writes to the **target** backend and reads
|
||||
//! from **target-first-then-source** (dual-read). A background job
|
||||
//! (see `migration_job.rs`) copies remaining blobs in the background.
|
||||
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::Serialize;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::common::errors::DomainError;
|
||||
|
||||
// ── Migration state ────────────────────────────────────────────────
|
||||
|
||||
/// Progress of an ongoing (or completed) backend migration.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct MigrationState {
|
||||
pub status: MigrationStatus,
|
||||
pub total_blobs: u64,
|
||||
pub migrated_blobs: u64,
|
||||
pub migrated_bytes: u64,
|
||||
pub failed_blobs: Vec<String>,
|
||||
pub started_at: Option<DateTime<Utc>>,
|
||||
pub completed_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
impl Default for MigrationState {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
status: MigrationStatus::Idle,
|
||||
total_blobs: 0,
|
||||
migrated_blobs: 0,
|
||||
migrated_bytes: 0,
|
||||
failed_blobs: Vec::new(),
|
||||
started_at: None,
|
||||
completed_at: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Status of the migration job.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum MigrationStatus {
|
||||
Idle,
|
||||
Running,
|
||||
Paused,
|
||||
Completed,
|
||||
Failed,
|
||||
}
|
||||
|
||||
// ── MigrationBlobBackend ───────────────────────────────────────────
|
||||
|
||||
/// A `BlobStorageBackend` decorator that proxies requests to a *source*
|
||||
/// (old) and *target* (new) backend, enabling live migration.
|
||||
pub struct MigrationBlobBackend {
|
||||
source: Arc<dyn BlobStorageBackend>,
|
||||
target: Arc<dyn BlobStorageBackend>,
|
||||
state: Arc<RwLock<MigrationState>>,
|
||||
}
|
||||
|
||||
impl MigrationBlobBackend {
|
||||
pub fn new(
|
||||
source: Arc<dyn BlobStorageBackend>,
|
||||
target: Arc<dyn BlobStorageBackend>,
|
||||
state: Arc<RwLock<MigrationState>>,
|
||||
) -> Self {
|
||||
Self {
|
||||
source,
|
||||
target,
|
||||
state,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn state(&self) -> &Arc<RwLock<MigrationState>> {
|
||||
&self.state
|
||||
}
|
||||
|
||||
pub fn source(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||
&self.source
|
||||
}
|
||||
|
||||
pub fn target(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||
&self.target
|
||||
}
|
||||
}
|
||||
|
||||
/// Boxed future alias (same as in the trait module).
|
||||
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||
|
||||
impl BlobStorageBackend for MigrationBlobBackend {
|
||||
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
Box::pin(async move {
|
||||
self.target.initialize().await?;
|
||||
// Source is already initialised; call anyway for idempotency.
|
||||
self.source.initialize().await?;
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
/// Writes go to **target** only.
|
||||
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
let path = source_path.to_path_buf();
|
||||
Box::pin(async move { self.target.put_blob(&hash, &path).await })
|
||||
}
|
||||
|
||||
/// Read from target first; fall back to source.
|
||||
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
match self.target.get_blob_stream(&hash).await {
|
||||
Ok(stream) => Ok(stream),
|
||||
Err(_) => self.source.get_blob_stream(&hash).await,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
match self.target.get_blob_range_stream(&hash, start, end).await {
|
||||
Ok(stream) => Ok(stream),
|
||||
Err(_) => self.source.get_blob_range_stream(&hash, start, end).await,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Delete from **both** backends (best-effort on source).
|
||||
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
self.target.delete_blob(&hash).await?;
|
||||
// Best-effort on source — ignore errors (blob may already be gone).
|
||||
let _ = self.source.delete_blob(&hash).await;
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
/// Exists in either backend.
|
||||
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
if self.target.blob_exists(&hash).await? {
|
||||
return Ok(true);
|
||||
}
|
||||
self.source.blob_exists(&hash).await
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
match self.target.blob_size(&hash).await {
|
||||
Ok(sz) => Ok(sz),
|
||||
Err(_) => self.source.blob_size(&hash).await,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>> {
|
||||
Box::pin(async move {
|
||||
let target_health = self.target.health_check().await?;
|
||||
let source_health = self.source.health_check().await?;
|
||||
Ok(StorageHealthStatus {
|
||||
connected: target_health.connected && source_health.connected,
|
||||
backend_type: format!(
|
||||
"migration({} → {})",
|
||||
source_health.backend_type, target_health.backend_type
|
||||
),
|
||||
message: format!(
|
||||
"Source: {} | Target: {}",
|
||||
source_health.message, target_health.message
|
||||
),
|
||||
available_bytes: target_health.available_bytes,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"migration"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||
// Prefer target, fall back to source.
|
||||
self.target
|
||||
.local_blob_path(hash)
|
||||
.or_else(|| self.source.local_blob_path(hash))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
//! Background migration job — copies blobs from a source backend to a target
|
||||
//! backend with configurable concurrency and progress tracking.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use futures::StreamExt;
|
||||
use serde::Serialize;
|
||||
use sqlx::PgPool;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use crate::common::errors::DomainError;
|
||||
use crate::infrastructure::services::migration_blob_backend::{MigrationState, MigrationStatus};
|
||||
|
||||
/// Run the migration: stream all blob hashes from `storage.blobs` and copy
|
||||
/// each one from `source` to `target`.
|
||||
///
|
||||
/// * The job respects `Paused` / `Failed` status in `state` — it will stop
|
||||
/// streaming when the status is no longer `Running`.
|
||||
/// * Errors on individual blobs are logged and collected in `failed_blobs`
|
||||
/// but do **not** abort the full run.
|
||||
/// * `concurrency` controls `buffer_unordered` parallelism (default: 4).
|
||||
pub async fn run_migration(
|
||||
source: Arc<dyn BlobStorageBackend>,
|
||||
target: Arc<dyn BlobStorageBackend>,
|
||||
pool: Arc<PgPool>,
|
||||
state: Arc<RwLock<MigrationState>>,
|
||||
concurrency: usize,
|
||||
) -> Result<(), DomainError> {
|
||||
// Count total blobs for progress tracking.
|
||||
let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||
.fetch_one(pool.as_ref())
|
||||
.await
|
||||
.unwrap_or(0);
|
||||
|
||||
{
|
||||
let mut s = state.write().await;
|
||||
s.status = MigrationStatus::Running;
|
||||
s.total_blobs = total as u64;
|
||||
s.migrated_blobs = 0;
|
||||
s.migrated_bytes = 0;
|
||||
s.failed_blobs.clear();
|
||||
s.started_at = Some(chrono::Utc::now());
|
||||
s.completed_at = None;
|
||||
}
|
||||
|
||||
// Stream all hashes+sizes with a cursor.
|
||||
let mut rows =
|
||||
sqlx::query_as::<_, (String, i64)>("SELECT hash, size FROM storage.blobs ORDER BY hash")
|
||||
.fetch(pool.as_ref());
|
||||
|
||||
// Collect all hashes first to avoid holding the cursor across awaits.
|
||||
let mut work: Vec<(String, i64)> = Vec::with_capacity(total as usize);
|
||||
while let Some(row) = rows.next().await {
|
||||
match row {
|
||||
Ok(r) => work.push(r),
|
||||
Err(e) => {
|
||||
tracing::warn!("Error fetching blob row during migration: {}", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Process in parallel chunks.
|
||||
let results = futures::stream::iter(work.into_iter().map(|(hash, size)| {
|
||||
let src = source.clone();
|
||||
let tgt = target.clone();
|
||||
let st = state.clone();
|
||||
async move {
|
||||
// Check if we should keep running.
|
||||
{
|
||||
let s = st.read().await;
|
||||
if s.status != MigrationStatus::Running {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Skip if already in target.
|
||||
match tgt.blob_exists(&hash).await {
|
||||
Ok(true) => {
|
||||
let mut s = st.write().await;
|
||||
s.migrated_blobs += 1;
|
||||
s.migrated_bytes += size as u64;
|
||||
return;
|
||||
}
|
||||
Ok(false) => {}
|
||||
Err(e) => {
|
||||
tracing::warn!("blob_exists check failed for {}: {}", hash, e);
|
||||
}
|
||||
}
|
||||
|
||||
// Copy: stream from source → temp file → put into target.
|
||||
if let Err(e) = copy_blob(&src, &tgt, &hash).await {
|
||||
tracing::warn!("Failed to migrate blob {}: {}", hash, e);
|
||||
let mut s = st.write().await;
|
||||
s.failed_blobs.push(hash);
|
||||
return;
|
||||
}
|
||||
|
||||
let mut s = st.write().await;
|
||||
s.migrated_blobs += 1;
|
||||
s.migrated_bytes += size as u64;
|
||||
}
|
||||
}))
|
||||
.buffer_unordered(concurrency)
|
||||
.collect::<Vec<()>>()
|
||||
.await;
|
||||
|
||||
drop(results);
|
||||
|
||||
// Finalize state.
|
||||
let mut s = state.write().await;
|
||||
if s.status == MigrationStatus::Running {
|
||||
if s.failed_blobs.is_empty() {
|
||||
s.status = MigrationStatus::Completed;
|
||||
} else {
|
||||
s.status = MigrationStatus::Failed;
|
||||
}
|
||||
s.completed_at = Some(chrono::Utc::now());
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"Migration finished: {}/{} blobs, {} failures",
|
||||
s.migrated_blobs,
|
||||
s.total_blobs,
|
||||
s.failed_blobs.len()
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy a single blob: stream from source → spool to temp file → put_blob into target.
|
||||
async fn copy_blob(
|
||||
source: &Arc<dyn BlobStorageBackend>,
|
||||
target: &Arc<dyn BlobStorageBackend>,
|
||||
hash: &str,
|
||||
) -> Result<(), DomainError> {
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
// Create a temp file to spool content.
|
||||
let tmp_dir = std::env::temp_dir().join("oxicloud-migration");
|
||||
tokio::fs::create_dir_all(&tmp_dir).await.map_err(|e| {
|
||||
DomainError::internal_error("Migration", format!("Failed to create temp dir: {}", e))
|
||||
})?;
|
||||
|
||||
let tmp_path = tmp_dir.join(format!("{}.tmp", hash));
|
||||
|
||||
// Stream from source.
|
||||
let stream = source.get_blob_stream(hash).await?;
|
||||
|
||||
// Write to temp file.
|
||||
let mut file = tokio::fs::File::create(&tmp_path).await.map_err(|e| {
|
||||
DomainError::internal_error("Migration", format!("Failed to create temp file: {}", e))
|
||||
})?;
|
||||
|
||||
let mut stream = std::pin::pin!(stream);
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("Migration", format!("Stream error: {}", e))
|
||||
})?;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Migration", format!("Write error: {}", e)))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Migration", format!("Flush error: {}", e)))?;
|
||||
drop(file);
|
||||
|
||||
// Put into target.
|
||||
target.put_blob(hash, &tmp_path).await?;
|
||||
|
||||
// Clean up temp file.
|
||||
let _ = tokio::fs::remove_file(&tmp_path).await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Verify migration integrity by comparing blob counts and sampling random hashes.
|
||||
pub async fn verify_migration(
|
||||
target: Arc<dyn BlobStorageBackend>,
|
||||
pool: Arc<PgPool>,
|
||||
sample_size: usize,
|
||||
) -> Result<VerificationResult, DomainError> {
|
||||
// 1. Count blobs in PG.
|
||||
let pg_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||
.fetch_one(pool.as_ref())
|
||||
.await
|
||||
.unwrap_or(0);
|
||||
|
||||
// 2. Verify sample of blobs exist in target.
|
||||
let sample_rows: Vec<(String, i64)> =
|
||||
sqlx::query_as("SELECT hash, size FROM storage.blobs ORDER BY random() LIMIT $1")
|
||||
.bind(sample_size as i64)
|
||||
.fetch_all(pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("Migration", format!("Sample query failed: {}", e))
|
||||
})?;
|
||||
|
||||
let mut missing = Vec::new();
|
||||
let mut size_mismatches = Vec::new();
|
||||
|
||||
for (hash, expected_size) in &sample_rows {
|
||||
match target.blob_exists(hash).await {
|
||||
Ok(false) => missing.push(hash.clone()),
|
||||
Err(e) => {
|
||||
tracing::warn!("blob_exists failed for {}: {}", hash, e);
|
||||
missing.push(hash.clone());
|
||||
}
|
||||
Ok(true) => {
|
||||
// Verify size matches.
|
||||
if let Ok(actual_size) = target.blob_size(hash).await
|
||||
&& actual_size != *expected_size as u64
|
||||
{
|
||||
size_mismatches.push(hash.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let passed = missing.is_empty() && size_mismatches.is_empty();
|
||||
|
||||
Ok(VerificationResult {
|
||||
pg_blob_count: pg_count as u64,
|
||||
sample_checked: sample_rows.len() as u64,
|
||||
missing_in_target: missing,
|
||||
size_mismatches,
|
||||
passed,
|
||||
})
|
||||
}
|
||||
|
||||
/// Result of a post-migration integrity check.
|
||||
#[derive(Debug, Clone, Serialize, serde::Deserialize)]
|
||||
pub struct VerificationResult {
|
||||
pub pg_blob_count: u64,
|
||||
pub sample_checked: u64,
|
||||
pub missing_in_target: Vec<String>,
|
||||
pub size_mismatches: Vec<String>,
|
||||
pub passed: bool,
|
||||
}
|
||||
@@ -1,18 +1,26 @@
|
||||
pub mod audio_metadata_service;
|
||||
pub mod azure_blob_backend;
|
||||
pub mod cached_blob_backend;
|
||||
pub mod chunked_upload_service;
|
||||
pub mod compression_service;
|
||||
pub mod dedup_service;
|
||||
pub mod encrypted_blob_backend;
|
||||
pub mod exif_service;
|
||||
pub mod file_content_cache;
|
||||
pub mod file_system_i18n_service;
|
||||
pub mod image_transcode_service;
|
||||
pub mod jwt_service;
|
||||
pub mod local_blob_backend;
|
||||
pub mod login_lockout_service;
|
||||
pub mod migration_blob_backend;
|
||||
pub mod migration_job;
|
||||
pub mod nextcloud_chunked_upload_service;
|
||||
pub mod oidc_service;
|
||||
pub mod password_hasher;
|
||||
pub mod path_resolver_service;
|
||||
pub mod path_service;
|
||||
pub mod retry_blob_backend;
|
||||
pub mod s3_blob_backend;
|
||||
pub mod thumbnail_service;
|
||||
#[cfg(test)]
|
||||
mod thumbnail_service_test;
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
//! `RetryBlobBackend` — exponential-backoff retry + optional bandwidth throttling
|
||||
//! decorator for remote blob backends.
|
||||
//!
|
||||
//! Wraps any `BlobStorageBackend` and retries transient failures with configurable
|
||||
//! exponential backoff. Optionally throttles upload/download bandwidth via
|
||||
//! inter-chunk sleeps.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
// ── Retry policy ───────────────────────────────────────────────────
|
||||
|
||||
/// Exponential backoff retry configuration.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RetryPolicy {
|
||||
/// Maximum number of retry attempts (0 = no retries).
|
||||
pub max_retries: u32,
|
||||
/// Initial backoff duration before the first retry.
|
||||
pub initial_backoff: Duration,
|
||||
/// Maximum backoff duration (capped).
|
||||
pub max_backoff: Duration,
|
||||
/// Multiplier applied to backoff after each attempt.
|
||||
pub backoff_multiplier: f64,
|
||||
}
|
||||
|
||||
impl Default for RetryPolicy {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_retries: 3,
|
||||
initial_backoff: Duration::from_millis(100),
|
||||
max_backoff: Duration::from_secs(10),
|
||||
backoff_multiplier: 2.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── RetryBlobBackend ───────────────────────────────────────────────
|
||||
|
||||
/// Decorator that retries failed backend operations with exponential backoff.
|
||||
pub struct RetryBlobBackend {
|
||||
inner: Arc<dyn BlobStorageBackend>,
|
||||
policy: RetryPolicy,
|
||||
}
|
||||
|
||||
impl RetryBlobBackend {
|
||||
pub fn new(inner: Arc<dyn BlobStorageBackend>, policy: RetryPolicy) -> Self {
|
||||
Self { inner, policy }
|
||||
}
|
||||
}
|
||||
|
||||
/// Execute an async closure with exponential backoff retry.
|
||||
async fn retry_async<F, Fut, T>(
|
||||
policy: &RetryPolicy,
|
||||
name: &str,
|
||||
mut f: F,
|
||||
) -> Result<T, DomainError>
|
||||
where
|
||||
F: FnMut() -> Fut,
|
||||
Fut: std::future::Future<Output = Result<T, DomainError>>,
|
||||
{
|
||||
let mut attempt = 0u32;
|
||||
let mut backoff = policy.initial_backoff;
|
||||
|
||||
loop {
|
||||
match f().await {
|
||||
Ok(v) => return Ok(v),
|
||||
Err(e) if attempt < policy.max_retries && is_retryable(&e) => {
|
||||
attempt += 1;
|
||||
tracing::warn!(
|
||||
"Retry {}/{} for {} after error: {} (backoff {:?})",
|
||||
attempt,
|
||||
policy.max_retries,
|
||||
name,
|
||||
e,
|
||||
backoff
|
||||
);
|
||||
tokio::time::sleep(backoff).await;
|
||||
let next =
|
||||
Duration::from_secs_f64(backoff.as_secs_f64() * policy.backoff_multiplier);
|
||||
backoff = next.min(policy.max_backoff);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Determine if an error is likely transient (network timeout, 5xx, etc.).
|
||||
fn is_retryable(err: &DomainError) -> bool {
|
||||
let msg = err.to_string().to_lowercase();
|
||||
msg.contains("timeout")
|
||||
|| msg.contains("connection")
|
||||
|| msg.contains("503")
|
||||
|| msg.contains("500")
|
||||
|| msg.contains("429")
|
||||
|| msg.contains("temporarily")
|
||||
|| msg.contains("broken pipe")
|
||||
|| msg.contains("reset by peer")
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for RetryBlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, "initialize", || {
|
||||
let inner = inner.clone();
|
||||
async move { inner.initialize().await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
let path = source_path.to_path_buf();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("put_blob({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
let path = path.clone();
|
||||
async move { inner.put_blob(&hash, &path).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("get_blob_stream({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
async move { inner.get_blob_stream(&hash).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("get_blob_range({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
async move { inner.get_blob_range_stream(&hash, start, end).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("delete_blob({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
async move { inner.delete_blob(&hash).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("blob_exists({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
async move { inner.blob_exists(&hash).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
let hash = hash.to_string();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, &format!("blob_size({hash})"), || {
|
||||
let inner = inner.clone();
|
||||
let hash = hash.clone();
|
||||
async move { inner.blob_size(&hash).await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
let inner = self.inner.clone();
|
||||
let policy = self.policy.clone();
|
||||
Box::pin(async move {
|
||||
retry_async(&policy, "health_check", || {
|
||||
let inner = inner.clone();
|
||||
async move { inner.health_check().await }
|
||||
})
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"retry"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||
self.inner.local_blob_path(hash)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
//! S3-Compatible Blob Backend — stores blobs in any S3-compatible object store.
|
||||
//!
|
||||
//! Supports AWS S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces,
|
||||
//! Wasabi, and any other service that implements the S3 API.
|
||||
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use tokio::fs;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use crate::common::config::S3StorageConfig;
|
||||
use crate::domain::errors::{DomainError, ErrorKind};
|
||||
|
||||
/// S3-compatible blob storage backend.
|
||||
///
|
||||
/// Blobs are stored as objects with key `{2-char-prefix}/{hash}.blob`,
|
||||
/// mirroring the local filesystem layout for consistency.
|
||||
pub struct S3BlobBackend {
|
||||
client: aws_sdk_s3::Client,
|
||||
bucket: String,
|
||||
}
|
||||
|
||||
impl S3BlobBackend {
|
||||
/// Build a new S3 backend from configuration.
|
||||
///
|
||||
/// Supports custom endpoints for non-AWS providers (Backblaze B2,
|
||||
/// MinIO, Cloudflare R2, etc.).
|
||||
pub fn new(config: &S3StorageConfig) -> Self {
|
||||
let credentials = aws_sdk_s3::config::Credentials::new(
|
||||
&config.access_key,
|
||||
&config.secret_key,
|
||||
None,
|
||||
None,
|
||||
"oxicloud",
|
||||
);
|
||||
|
||||
let mut builder = aws_sdk_s3::config::Builder::new()
|
||||
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
|
||||
.credentials_provider(credentials)
|
||||
.behavior_version_latest();
|
||||
|
||||
if let Some(ref endpoint) = config.endpoint_url {
|
||||
builder = builder.endpoint_url(endpoint);
|
||||
}
|
||||
|
||||
if config.force_path_style {
|
||||
builder = builder.force_path_style(true);
|
||||
}
|
||||
|
||||
let client = aws_sdk_s3::Client::from_conf(builder.build());
|
||||
|
||||
Self {
|
||||
client,
|
||||
bucket: config.bucket.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the S3 object key for a given hash.
|
||||
fn object_key(hash: &str) -> String {
|
||||
let prefix = &hash[0..2];
|
||||
format!("{}/{}.blob", prefix, hash)
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for S3BlobBackend {
|
||||
fn initialize(
|
||||
&self,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
Box::pin(async move {
|
||||
// Verify bucket exists and is accessible
|
||||
self.client
|
||||
.head_bucket()
|
||||
.bucket(&self.bucket)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Cannot access bucket '{}': {}", self.bucket, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
tracing::info!("S3 blob backend initialized: bucket={}", self.bucket);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn put_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
source_path: &Path,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
let source_path = source_path.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
// Check if object already exists (idempotent)
|
||||
let exists = self
|
||||
.client
|
||||
.head_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await
|
||||
.is_ok();
|
||||
|
||||
if exists {
|
||||
// Blob already in S3 — remove local source and return size
|
||||
let file_size = fs::metadata(&source_path)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to stat source file: {}", e),
|
||||
)
|
||||
})?
|
||||
.len();
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
return Ok(file_size);
|
||||
}
|
||||
|
||||
// Upload from local file
|
||||
let body = ByteStream::from_path(&source_path).await.map_err(|e| {
|
||||
DomainError::internal_error("S3", format!("Failed to read source file: {}", e))
|
||||
})?;
|
||||
|
||||
let file_size = fs::metadata(&source_path)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("S3", format!("Failed to stat source file: {}", e))
|
||||
})?
|
||||
.len();
|
||||
|
||||
self.client
|
||||
.put_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.body(body)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to upload blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
// Clean up local source after successful upload
|
||||
let _ = fs::remove_file(&source_path).await;
|
||||
|
||||
Ok(file_size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
let output = self
|
||||
.client
|
||||
.get_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"S3",
|
||||
format!("Failed to get blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
// Convert S3 ByteStream into a Stream<Item = Result<Bytes, io::Error>>
|
||||
// via AsyncRead adapter
|
||||
let reader = output.body.into_async_read();
|
||||
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||
{
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
let range = match end {
|
||||
Some(end_pos) => format!("bytes={}-{}", start, end_pos.saturating_sub(1)),
|
||||
None => format!("bytes={}-", start),
|
||||
};
|
||||
|
||||
let output = self
|
||||
.client
|
||||
.get_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.range(range)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"S3",
|
||||
format!("Failed to get blob range {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
let reader = output.body.into_async_read();
|
||||
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
|
||||
})
|
||||
}
|
||||
|
||||
fn delete_blob(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
// S3 DeleteObject is already idempotent (returns 204 even if not found)
|
||||
self.client
|
||||
.delete_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to delete blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_exists(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
match self
|
||||
.client
|
||||
.head_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
Ok(_) => Ok(true),
|
||||
Err(e) => {
|
||||
// Check if it's a 404 (not found) vs an actual error
|
||||
let service_err = e.into_service_error();
|
||||
if service_err.is_not_found() {
|
||||
Ok(false)
|
||||
} else {
|
||||
Err(DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to check blob {}: {}", hash, service_err),
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn blob_size(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
|
||||
let output = self
|
||||
.client
|
||||
.head_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"S3",
|
||||
format!("Failed to stat blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
Ok(output.content_length().unwrap_or(0) as u64)
|
||||
})
|
||||
}
|
||||
|
||||
fn health_check(
|
||||
&self,
|
||||
) -> Pin<
|
||||
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||
> {
|
||||
Box::pin(async move {
|
||||
match self.client.head_bucket().bucket(&self.bucket).send().await {
|
||||
Ok(_) => Ok(StorageHealthStatus {
|
||||
connected: true,
|
||||
backend_type: "s3".to_string(),
|
||||
message: format!("S3 bucket '{}' is accessible", self.bucket),
|
||||
available_bytes: None,
|
||||
}),
|
||||
Err(e) => Ok(StorageHealthStatus {
|
||||
connected: false,
|
||||
backend_type: "s3".to_string(),
|
||||
message: format!("S3 bucket '{}' is not accessible: {}", self.bucket, e),
|
||||
available_bytes: None,
|
||||
}),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"s3"
|
||||
}
|
||||
|
||||
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||
None // Remote backend — no local path
|
||||
}
|
||||
}
|
||||
@@ -8,8 +8,9 @@ use axum::{
|
||||
|
||||
use crate::application::dtos::settings_dto::{
|
||||
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, ListUsersQueryDto,
|
||||
SaveOidcSettingsDto, TestOidcConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
|
||||
UpdateUserRoleDto,
|
||||
MigrationStateDto, SaveOidcSettingsDto, SaveStorageSettingsDto, StartMigrationDto,
|
||||
TestOidcConnectionDto, TestStorageConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
|
||||
UpdateUserRoleDto, VerifyMigrationDto,
|
||||
};
|
||||
use crate::application::ports::auth_ports::TokenServicePort;
|
||||
use crate::common::di::AppState;
|
||||
@@ -24,6 +25,22 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
|
||||
.route("/settings/oidc", get(get_oidc_settings))
|
||||
.route("/settings/oidc", put(save_oidc_settings))
|
||||
.route("/settings/oidc/test", post(test_oidc_connection))
|
||||
// Storage settings
|
||||
.route("/settings/storage", get(get_storage_settings))
|
||||
.route("/settings/storage", put(save_storage_settings))
|
||||
.route("/settings/storage/test", post(test_storage_connection))
|
||||
// Storage migration
|
||||
.route("/storage/migration", get(get_migration_status))
|
||||
.route("/storage/migration/start", post(start_migration))
|
||||
.route("/storage/migration/pause", post(pause_migration))
|
||||
.route("/storage/migration/resume", post(resume_migration))
|
||||
.route("/storage/migration/complete", post(complete_migration))
|
||||
.route("/storage/migration/verify", post(verify_migration))
|
||||
// Encryption key generation
|
||||
.route(
|
||||
"/settings/storage/generate-key",
|
||||
post(generate_encryption_key),
|
||||
)
|
||||
.route("/settings/general", get(get_general_settings))
|
||||
// Dashboard / stats
|
||||
.route("/dashboard", get(get_dashboard_stats))
|
||||
@@ -148,6 +165,336 @@ async fn test_oidc_connection(
|
||||
Ok(Json(result))
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────
|
||||
// Storage settings handlers
|
||||
// ─────────────────────────────────────────────────────
|
||||
|
||||
/// GET /api/admin/settings/storage — get storage backend settings
|
||||
async fn get_storage_settings(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let svc = state
|
||||
.storage_settings_service
|
||||
.as_ref()
|
||||
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||
|
||||
let settings = svc
|
||||
.get_storage_settings()
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to load storage settings: {}", e)))?;
|
||||
|
||||
Ok(Json(settings))
|
||||
}
|
||||
|
||||
/// PUT /api/admin/settings/storage — save storage backend settings
|
||||
async fn save_storage_settings(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
Json(dto): Json<SaveStorageSettingsDto>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
let (user_id, _) = admin_guard(&state, &headers).await?;
|
||||
|
||||
let svc = state
|
||||
.storage_settings_service
|
||||
.as_ref()
|
||||
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||
|
||||
svc.save_storage_settings(dto, user_id)
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to save storage settings: {}", e)))?;
|
||||
|
||||
Ok((
|
||||
StatusCode::OK,
|
||||
Json(serde_json::json!({
|
||||
"message": "Storage settings saved successfully"
|
||||
})),
|
||||
))
|
||||
}
|
||||
|
||||
/// POST /api/admin/settings/storage/test — test storage backend connection
|
||||
async fn test_storage_connection(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
Json(dto): Json<TestStorageConnectionDto>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let svc = state
|
||||
.storage_settings_service
|
||||
.as_ref()
|
||||
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||
|
||||
let result = svc
|
||||
.test_storage_connection(dto)
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Storage connection test failed: {}", e)))?;
|
||||
|
||||
Ok(Json(result))
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────
|
||||
// Storage migration handlers
|
||||
// ─────────────────────────────────────────────────────
|
||||
|
||||
/// GET /api/admin/storage/migration — current migration progress
|
||||
async fn get_migration_status(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
admin_guard(&state, &headers).await?;
|
||||
let s = state.migration_state.read().await;
|
||||
Ok(Json(migration_state_to_dto(&s)))
|
||||
}
|
||||
|
||||
/// POST /api/admin/storage/migration/start — begin background migration
|
||||
async fn start_migration(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
Json(dto): Json<StartMigrationDto>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
// Check not already running.
|
||||
{
|
||||
let s = state.migration_state.read().await;
|
||||
if s.status == MigrationStatus::Running {
|
||||
return Err(AppError::bad_request("A migration is already running"));
|
||||
}
|
||||
}
|
||||
|
||||
let pool = state
|
||||
.db_pool
|
||||
.clone()
|
||||
.ok_or_else(|| AppError::internal_error("Database not available"))?;
|
||||
|
||||
let source = state.core.dedup_service.backend().clone();
|
||||
let svc = state
|
||||
.storage_settings_service
|
||||
.as_ref()
|
||||
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||
|
||||
// Build target backend from saved settings.
|
||||
let effective = svc
|
||||
.load_effective_storage_config()
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
|
||||
|
||||
let target = build_backend_from_config(&effective)
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
|
||||
target
|
||||
.initialize()
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
|
||||
|
||||
let concurrency = dto.concurrency.unwrap_or(4).clamp(1, 16);
|
||||
let migration_state = state.migration_state.clone();
|
||||
|
||||
// Spawn the background migration job.
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = crate::infrastructure::services::migration_job::run_migration(
|
||||
source,
|
||||
target,
|
||||
pool,
|
||||
migration_state,
|
||||
concurrency,
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Migration job error: {}", e);
|
||||
}
|
||||
});
|
||||
|
||||
Ok((
|
||||
StatusCode::OK,
|
||||
Json(serde_json::json!({ "message": "Migration started" })),
|
||||
))
|
||||
}
|
||||
|
||||
/// POST /api/admin/storage/migration/pause — pause running migration
|
||||
async fn pause_migration(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let mut s = state.migration_state.write().await;
|
||||
if s.status != MigrationStatus::Running {
|
||||
return Err(AppError::bad_request("No running migration to pause"));
|
||||
}
|
||||
s.status = MigrationStatus::Paused;
|
||||
Ok((
|
||||
StatusCode::OK,
|
||||
Json(serde_json::json!({ "message": "Migration paused" })),
|
||||
))
|
||||
}
|
||||
|
||||
/// POST /api/admin/storage/migration/resume — resume paused migration
|
||||
async fn resume_migration(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
// Set status back to Running — the background task checks on each blob.
|
||||
let mut s = state.migration_state.write().await;
|
||||
if s.status != MigrationStatus::Paused {
|
||||
return Err(AppError::bad_request("No paused migration to resume"));
|
||||
}
|
||||
s.status = MigrationStatus::Running;
|
||||
Ok((
|
||||
StatusCode::OK,
|
||||
Json(serde_json::json!({ "message": "Migration resumed" })),
|
||||
))
|
||||
}
|
||||
|
||||
/// POST /api/admin/storage/migration/complete — finalize migration
|
||||
async fn complete_migration(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let s = state.migration_state.read().await;
|
||||
if s.status != MigrationStatus::Completed {
|
||||
return Err(AppError::bad_request(
|
||||
"Migration must be completed (100%) before finalizing",
|
||||
));
|
||||
}
|
||||
drop(s);
|
||||
|
||||
// Mark as idle — the admin has acknowledged completion.
|
||||
let mut s = state.migration_state.write().await;
|
||||
s.status = MigrationStatus::Idle;
|
||||
|
||||
Ok((
|
||||
StatusCode::OK,
|
||||
Json(
|
||||
serde_json::json!({ "message": "Migration finalized. Restart the server to use the new backend." }),
|
||||
),
|
||||
))
|
||||
}
|
||||
|
||||
/// POST /api/admin/storage/migration/verify — run integrity check
|
||||
async fn verify_migration(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
Json(dto): Json<VerifyMigrationDto>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let pool = state
|
||||
.db_pool
|
||||
.clone()
|
||||
.ok_or_else(|| AppError::internal_error("Database not available"))?;
|
||||
|
||||
let svc = state
|
||||
.storage_settings_service
|
||||
.as_ref()
|
||||
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||
|
||||
let effective = svc
|
||||
.load_effective_storage_config()
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
|
||||
|
||||
let target = build_backend_from_config(&effective)
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
|
||||
target
|
||||
.initialize()
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
|
||||
|
||||
let sample_size = dto.sample_size.unwrap_or(100).clamp(1, 1000);
|
||||
|
||||
let result =
|
||||
crate::infrastructure::services::migration_job::verify_migration(target, pool, sample_size)
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Verification failed: {}", e)))?;
|
||||
|
||||
Ok(Json(result))
|
||||
}
|
||||
|
||||
/// Helper: convert MigrationState to DTO for JSON serialization.
|
||||
fn migration_state_to_dto(
|
||||
s: &crate::infrastructure::services::migration_blob_backend::MigrationState,
|
||||
) -> MigrationStateDto {
|
||||
let throughput = match (s.started_at, s.migrated_bytes) {
|
||||
(Some(start), bytes) if bytes > 0 => {
|
||||
let elapsed = chrono::Utc::now()
|
||||
.signed_duration_since(start)
|
||||
.num_seconds()
|
||||
.max(1) as f64;
|
||||
Some(bytes as f64 / elapsed)
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
|
||||
MigrationStateDto {
|
||||
status: format!("{:?}", s.status).to_lowercase(),
|
||||
total_blobs: s.total_blobs,
|
||||
migrated_blobs: s.migrated_blobs,
|
||||
migrated_bytes: s.migrated_bytes,
|
||||
failed_blobs: s.failed_blobs.clone(),
|
||||
started_at: s.started_at.map(|d| d.to_rfc3339()),
|
||||
completed_at: s.completed_at.map(|d| d.to_rfc3339()),
|
||||
throughput_bytes_per_sec: throughput,
|
||||
}
|
||||
}
|
||||
|
||||
/// POST /api/admin/settings/storage/generate-key — generate a random AES-256 key.
|
||||
async fn generate_encryption_key(
|
||||
State(state): State<Arc<AppState>>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
admin_guard(&state, &headers).await?;
|
||||
|
||||
let key =
|
||||
crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend::generate_key(
|
||||
);
|
||||
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
|
||||
|
||||
Ok(Json(serde_json::json!({
|
||||
"key": key_b64,
|
||||
"warning": "Store this key securely. If lost, encrypted data is IRRECOVERABLY LOST."
|
||||
})))
|
||||
}
|
||||
|
||||
/// Helper: build a BlobStorageBackend from StorageConfig.
|
||||
fn build_backend_from_config(
|
||||
config: &crate::common::config::StorageConfig,
|
||||
) -> Result<
|
||||
std::sync::Arc<dyn crate::application::ports::blob_storage_ports::BlobStorageBackend>,
|
||||
String,
|
||||
> {
|
||||
match config.backend {
|
||||
crate::common::config::StorageBackendType::Local => Ok(std::sync::Arc::new(
|
||||
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
|
||||
std::path::Path::new(&config.root_dir),
|
||||
),
|
||||
)),
|
||||
crate::common::config::StorageBackendType::S3 => {
|
||||
let s3 = config.s3.as_ref().ok_or("S3 config missing")?;
|
||||
Ok(std::sync::Arc::new(
|
||||
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3),
|
||||
))
|
||||
}
|
||||
crate::common::config::StorageBackendType::Azure => {
|
||||
let az = config.azure.as_ref().ok_or("Azure config missing")?;
|
||||
Ok(std::sync::Arc::new(
|
||||
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(az),
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// GET /api/admin/settings/general — system overview (backward compat)
|
||||
async fn get_general_settings(
|
||||
State(state): State<Arc<AppState>>,
|
||||
|
||||
@@ -53,6 +53,9 @@
|
||||
<button class="admin-tab" id="tab-btn-oidc">
|
||||
<i class="fas fa-key"></i> <span data-i18n="admin.tab_oidc">SSO / OIDC</span>
|
||||
</button>
|
||||
<button class="admin-tab" id="tab-btn-storage">
|
||||
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div id="tab-dashboard" class="tab-content active">
|
||||
@@ -424,6 +427,183 @@
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ════════ Storage tab ════════ -->
|
||||
<div id="tab-storage" class="tab-content">
|
||||
<div class="admin-card">
|
||||
<h2>
|
||||
<i class="fas fa-database"></i> <span data-i18n="admin.storage_title">Storage Backend</span>
|
||||
</h2>
|
||||
|
||||
<!-- Current status -->
|
||||
<div class="storage-status-grid">
|
||||
<div class="storage-stat">
|
||||
<span class="storage-stat__label" data-i18n="admin.storage_current_backend">Active Backend</span>
|
||||
<span class="storage-stat__value" id="storage-current-backend">—</span>
|
||||
</div>
|
||||
<div class="storage-stat">
|
||||
<span class="storage-stat__label" data-i18n="admin.storage_total_blobs">Total Blobs</span>
|
||||
<span class="storage-stat__value" id="storage-total-blobs">—</span>
|
||||
</div>
|
||||
<div class="storage-stat">
|
||||
<span class="storage-stat__label" data-i18n="admin.storage_total_size">Total Size</span>
|
||||
<span class="storage-stat__value" id="storage-total-size">—</span>
|
||||
</div>
|
||||
<div class="storage-stat">
|
||||
<span class="storage-stat__label" data-i18n="admin.storage_dedup_ratio">Dedup Ratio</span>
|
||||
<span class="storage-stat__value" id="storage-dedup-ratio">—</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Backend selector -->
|
||||
<div class="storage-backend-selector">
|
||||
<label class="storage-backend-option">
|
||||
<input type="radio" name="storage-backend" value="local" checked />
|
||||
<span class="storage-backend-option__card">
|
||||
<i class="fas fa-hdd"></i>
|
||||
<span data-i18n="admin.storage_local">Local Filesystem</span>
|
||||
</span>
|
||||
</label>
|
||||
<label class="storage-backend-option">
|
||||
<input type="radio" name="storage-backend" value="s3" />
|
||||
<span class="storage-backend-option__card">
|
||||
<i class="fas fa-cloud"></i>
|
||||
<span data-i18n="admin.storage_s3">S3-Compatible</span>
|
||||
</span>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<!-- S3 form (shown when S3 is selected) -->
|
||||
<div id="storage-s3-form" class="storage-form" style="display:none">
|
||||
<!-- Provider preset -->
|
||||
<div class="form-group">
|
||||
<label data-i18n="admin.storage_provider_preset">Provider Preset</label>
|
||||
<select id="storage-preset">
|
||||
<option value="custom" data-i18n="admin.storage_preset_custom">Custom</option>
|
||||
<option value="aws">Amazon S3</option>
|
||||
<option value="backblaze">Backblaze B2</option>
|
||||
<option value="cloudflare-r2">Cloudflare R2</option>
|
||||
<option value="minio">MinIO</option>
|
||||
<option value="digitalocean">DigitalOcean Spaces</option>
|
||||
<option value="wasabi">Wasabi</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<!-- Endpoint URL -->
|
||||
<div class="form-group">
|
||||
<label><span data-i18n="admin.storage_endpoint_url">Endpoint URL</span>
|
||||
<span id="badge-s3_endpoint_url"></span></label>
|
||||
<input type="url" id="storage-endpoint-url" placeholder="https://s3.amazonaws.com" />
|
||||
<small data-i18n="admin.storage_endpoint_hint">Leave empty for Amazon S3 default</small>
|
||||
</div>
|
||||
|
||||
<!-- Bucket -->
|
||||
<div class="form-group">
|
||||
<label><span data-i18n="admin.storage_bucket">Bucket</span>
|
||||
<span id="badge-s3_bucket"></span></label>
|
||||
<input type="text" id="storage-bucket" placeholder="my-oxicloud-bucket" />
|
||||
</div>
|
||||
|
||||
<!-- Region -->
|
||||
<div class="form-group">
|
||||
<label><span data-i18n="admin.storage_region">Region</span>
|
||||
<span id="badge-s3_region"></span></label>
|
||||
<input type="text" id="storage-region" placeholder="us-east-1" />
|
||||
</div>
|
||||
|
||||
<!-- Access Key -->
|
||||
<div class="form-group">
|
||||
<label><span data-i18n="admin.storage_access_key">Access Key ID</span>
|
||||
<span id="badge-s3_access_key"></span></label>
|
||||
<input type="text" id="storage-access-key" placeholder="AKIA..." />
|
||||
</div>
|
||||
|
||||
<!-- Secret Key -->
|
||||
<div class="form-group">
|
||||
<label><span data-i18n="admin.storage_secret_key">Secret Access Key</span>
|
||||
<span id="badge-s3_secret_key"></span></label>
|
||||
<input type="password" id="storage-secret-key" placeholder="Leave empty to keep current value" />
|
||||
<small id="storage-secret-hint" style="display:none">
|
||||
<i class="fas fa-check-circle secret-icon"></i>
|
||||
<span data-i18n="admin.storage_secret_configured">A secret key is already configured</span>
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<!-- Force Path Style -->
|
||||
<div class="toggle-row">
|
||||
<label data-i18n="admin.storage_path_style">Force Path Style</label>
|
||||
<label class="switch">
|
||||
<input type="checkbox" id="storage-path-style" /><span class="slider"></span>
|
||||
</label>
|
||||
</div>
|
||||
<small data-i18n="admin.storage_path_style_hint">Required for MinIO and some S3-compatible providers</small>
|
||||
</div>
|
||||
|
||||
<!-- Action buttons -->
|
||||
<div class="storage-actions">
|
||||
<button class="btn btn-secondary" id="btn-test-storage">
|
||||
<i class="fas fa-vial"></i> <span data-i18n="admin.storage_test_connection">Test Connection</span>
|
||||
</button>
|
||||
<button class="btn btn-primary" id="btn-save-storage">
|
||||
<i class="fas fa-save"></i> <span data-i18n="admin.storage_save">Save</span>
|
||||
</button>
|
||||
</div>
|
||||
<div id="storage-status" class="alert"></div>
|
||||
|
||||
<!-- Migration section -->
|
||||
<div class="storage-migration-section">
|
||||
<h3><i class="fas fa-exchange-alt"></i> <span data-i18n="admin.storage_migration">Backend Migration</span></h3>
|
||||
|
||||
<!-- Status badge -->
|
||||
<div class="migration-status-row">
|
||||
<span data-i18n="admin.migration_status_label">Status:</span>
|
||||
<span id="migration-status-badge" class="badge badge-migration badge-migration--idle">Idle</span>
|
||||
</div>
|
||||
|
||||
<!-- Progress bar (hidden when idle) -->
|
||||
<div id="migration-progress-section" style="display:none">
|
||||
<div class="migration-progress-bar">
|
||||
<div class="migration-progress-bar__fill" id="migration-progress-fill" style="width:0%"></div>
|
||||
</div>
|
||||
<div class="migration-progress-text">
|
||||
<span id="migration-progress-label">0 / 0 blobs (0%)</span>
|
||||
<span id="migration-throughput"></span>
|
||||
</div>
|
||||
<div class="migration-progress-text">
|
||||
<span id="migration-bytes-label"></span>
|
||||
<span id="migration-eta"></span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Failed blobs (expandable) -->
|
||||
<details id="migration-failed-section" style="display:none">
|
||||
<summary><i class="fas fa-exclamation-triangle"></i> <span id="migration-failed-count">0</span> <span data-i18n="admin.migration_failed_blobs">failed blobs</span></summary>
|
||||
<pre id="migration-failed-list" class="migration-failed-list"></pre>
|
||||
</details>
|
||||
|
||||
<!-- Action buttons -->
|
||||
<div class="migration-actions">
|
||||
<button class="btn btn-primary" id="btn-start-migration">
|
||||
<i class="fas fa-play"></i> <span data-i18n="admin.migration_start">Start Migration</span>
|
||||
</button>
|
||||
<button class="btn btn-secondary" id="btn-pause-migration" style="display:none">
|
||||
<i class="fas fa-pause"></i> <span data-i18n="admin.migration_pause">Pause</span>
|
||||
</button>
|
||||
<button class="btn btn-primary" id="btn-resume-migration" style="display:none">
|
||||
<i class="fas fa-play"></i> <span data-i18n="admin.migration_resume">Resume</span>
|
||||
</button>
|
||||
<button class="btn btn-secondary" id="btn-verify-migration" style="display:none">
|
||||
<i class="fas fa-check-double"></i> <span data-i18n="admin.migration_verify">Verify Integrity</span>
|
||||
</button>
|
||||
<button class="btn btn-primary" id="btn-complete-migration" style="display:none">
|
||||
<i class="fas fa-flag-checkered"></i> <span data-i18n="admin.migration_complete">Finalize</span>
|
||||
</button>
|
||||
</div>
|
||||
<div id="migration-verify-result" style="display:none"></div>
|
||||
<div id="migration-status-msg" class="alert" style="display:none"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -324,6 +324,7 @@ body {
|
||||
#oidc-form,
|
||||
#secret-hint,
|
||||
#password-warning,
|
||||
#storage-secret-hint,
|
||||
#quota-modal,
|
||||
#create-user-modal,
|
||||
#reset-pw-modal {
|
||||
@@ -938,3 +939,176 @@ details[open] summary {
|
||||
.btn-danger:hover {
|
||||
opacity: 0.88;
|
||||
}
|
||||
|
||||
/* ── Storage Tab ── */
|
||||
|
||||
.storage-status-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
|
||||
gap: 12px;
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
.storage-stat {
|
||||
padding: 14px;
|
||||
background: var(--color-bg-hover);
|
||||
border-radius: 12px;
|
||||
text-align: center;
|
||||
border: 1px solid var(--color-border);
|
||||
}
|
||||
.storage-stat__label {
|
||||
display: block;
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
color: var(--color-text-secondary);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
.storage-stat__value {
|
||||
display: block;
|
||||
font-size: 1.35rem;
|
||||
font-weight: 700;
|
||||
color: var(--color-text-heading);
|
||||
}
|
||||
|
||||
.storage-backend-selector {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
.storage-backend-option {
|
||||
flex: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
.storage-backend-option input[type="radio"] {
|
||||
display: none;
|
||||
}
|
||||
.storage-backend-option__card {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 18px 14px;
|
||||
border: 2px solid var(--color-border);
|
||||
border-radius: 12px;
|
||||
transition: border-color 0.2s, background 0.2s;
|
||||
text-align: center;
|
||||
font-weight: 500;
|
||||
}
|
||||
.storage-backend-option__card i {
|
||||
font-size: 1.5rem;
|
||||
color: var(--color-text-secondary);
|
||||
transition: color 0.2s;
|
||||
}
|
||||
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card {
|
||||
border-color: var(--color-accent);
|
||||
background: var(--color-accent-subtle);
|
||||
}
|
||||
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card i {
|
||||
color: var(--color-accent);
|
||||
}
|
||||
|
||||
.storage-form {
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.storage-actions {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
margin-top: 24px;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.storage-migration-section {
|
||||
margin-top: 32px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid var(--color-border);
|
||||
}
|
||||
.storage-migration-section h3 {
|
||||
font-size: 1rem;
|
||||
margin-bottom: 8px;
|
||||
color: var(--color-text-heading);
|
||||
}
|
||||
.storage-migration-section h3 i {
|
||||
color: var(--color-text-secondary);
|
||||
margin-right: 6px;
|
||||
}
|
||||
|
||||
/* ── Migration UI ── */
|
||||
|
||||
.migration-status-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 14px;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.badge-migration {
|
||||
padding: 3px 10px;
|
||||
border-radius: 10px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.4px;
|
||||
}
|
||||
.badge-migration--idle {
|
||||
background: var(--color-bg-hover);
|
||||
color: var(--color-text-secondary);
|
||||
}
|
||||
.badge-migration--running {
|
||||
background: var(--color-accent-subtle);
|
||||
color: var(--color-accent);
|
||||
}
|
||||
.badge-migration--paused {
|
||||
background: var(--color-warning-bg, #fff8e1);
|
||||
color: var(--color-warning-text, #b28704);
|
||||
}
|
||||
.badge-migration--completed {
|
||||
background: var(--color-success-bg, #e8f5e9);
|
||||
color: var(--color-success-text, #2e7d32);
|
||||
}
|
||||
.badge-migration--failed {
|
||||
background: var(--color-danger-bg, #fce4ec);
|
||||
color: var(--color-danger-text, #c62828);
|
||||
}
|
||||
|
||||
.migration-progress-bar {
|
||||
width: 100%;
|
||||
height: 10px;
|
||||
background: var(--color-bg-hover);
|
||||
border-radius: 6px;
|
||||
overflow: hidden;
|
||||
margin-bottom: 6px;
|
||||
}
|
||||
.migration-progress-bar__fill {
|
||||
height: 100%;
|
||||
background: var(--color-accent);
|
||||
border-radius: 6px;
|
||||
transition: width 0.3s ease;
|
||||
}
|
||||
|
||||
.migration-progress-text {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
font-size: 12px;
|
||||
color: var(--color-text-secondary);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.migration-failed-list {
|
||||
max-height: 150px;
|
||||
overflow-y: auto;
|
||||
font-size: 11px;
|
||||
background: var(--color-bg-hover);
|
||||
padding: 8px 12px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--color-border);
|
||||
margin-top: 6px;
|
||||
}
|
||||
|
||||
.migration-actions {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
margin-top: 16px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
@@ -138,6 +138,7 @@ function switchTab(name, el) {
|
||||
activeTabName = name;
|
||||
if (name === 'users') loadUsers();
|
||||
if (name === 'dashboard') loadDashboard();
|
||||
if (name === 'storage') loadStorage();
|
||||
}
|
||||
|
||||
async function loadDashboard() {
|
||||
@@ -693,6 +694,353 @@ async function saveOidcSettings() {
|
||||
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.save_btn'))}`;
|
||||
}
|
||||
|
||||
/* ── Storage tab ── */
|
||||
|
||||
const STORAGE_PRESETS = {
|
||||
'custom': { endpoint: '', region: '', pathStyle: false },
|
||||
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
|
||||
'backblaze': { endpoint: 'https://s3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
|
||||
'cloudflare-r2': { endpoint: 'https://{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
|
||||
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
|
||||
'digitalocean': { endpoint: 'https://{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
|
||||
'wasabi': { endpoint: 'https://s3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
|
||||
};
|
||||
|
||||
function toggleS3Form(visible) {
|
||||
if (visible) showElement('storage-s3-form');
|
||||
else hideElement('storage-s3-form');
|
||||
}
|
||||
|
||||
function onStoragePresetChange() {
|
||||
const preset = document.getElementById('storage-preset').value;
|
||||
const p = STORAGE_PRESETS[preset];
|
||||
if (!p) return;
|
||||
if (p.endpoint) document.getElementById('storage-endpoint-url').value = p.endpoint;
|
||||
if (p.region) document.getElementById('storage-region').value = p.region;
|
||||
document.getElementById('storage-path-style').checked = p.pathStyle;
|
||||
}
|
||||
|
||||
function showStorageStatus(msg, type) {
|
||||
const el = document.getElementById('storage-status');
|
||||
el.textContent = msg;
|
||||
el.className = `alert alert-${type}`;
|
||||
}
|
||||
|
||||
async function loadStorage() {
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/settings/storage`, {
|
||||
headers: headers(),
|
||||
credentials: 'same-origin'
|
||||
});
|
||||
if (!resp.ok) return;
|
||||
const s = await resp.json();
|
||||
|
||||
// Backend selector
|
||||
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
|
||||
r.checked = r.value === s.backend;
|
||||
});
|
||||
toggleS3Form(s.backend === 's3');
|
||||
|
||||
// S3 fields
|
||||
document.getElementById('storage-endpoint-url').value = s.s3_endpoint_url || '';
|
||||
document.getElementById('storage-bucket').value = s.s3_bucket || '';
|
||||
document.getElementById('storage-region').value = s.s3_region || '';
|
||||
document.getElementById('storage-access-key').value = '';
|
||||
document.getElementById('storage-secret-key').value = '';
|
||||
document.getElementById('storage-path-style').checked = s.s3_force_path_style;
|
||||
|
||||
// Secret hints
|
||||
if (s.s3_access_key_set) {
|
||||
document.getElementById('storage-access-key').placeholder = t('admin.storage_key_placeholder') || 'Leave empty to keep current value';
|
||||
}
|
||||
if (s.s3_secret_key_set) {
|
||||
showElement('storage-secret-hint');
|
||||
} else {
|
||||
hideElement('storage-secret-hint');
|
||||
}
|
||||
|
||||
// ENV badges
|
||||
(s.env_overrides || []).forEach((field) => {
|
||||
const badge = document.getElementById(`badge-${field}`);
|
||||
if (badge) badge.innerHTML = '<span class="badge badge-env">ENV</span>';
|
||||
});
|
||||
|
||||
// Status section
|
||||
document.getElementById('storage-current-backend').textContent = s.current_backend || '—';
|
||||
document.getElementById('storage-total-blobs').textContent = s.total_blobs != null ? s.total_blobs.toLocaleString() : '—';
|
||||
document.getElementById('storage-total-size').textContent = s.total_bytes_stored != null ? formatBytes(s.total_bytes_stored) : '—';
|
||||
document.getElementById('storage-dedup-ratio').textContent = s.dedup_ratio != null ? `${s.dedup_ratio.toFixed(2)}x` : '—';
|
||||
} catch (e) {
|
||||
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||
}
|
||||
|
||||
// Also load migration status
|
||||
loadMigrationStatus();
|
||||
}
|
||||
|
||||
async function saveStorageSettings() {
|
||||
const btn = document.getElementById('btn-save-storage');
|
||||
btn.disabled = true;
|
||||
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.saving'))}`;
|
||||
|
||||
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
|
||||
const body = {
|
||||
backend,
|
||||
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
|
||||
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
|
||||
s3_region: document.getElementById('storage-region').value.trim() || null,
|
||||
s3_access_key: document.getElementById('storage-access-key').value || null,
|
||||
s3_secret_key: document.getElementById('storage-secret-key').value || null,
|
||||
s3_force_path_style: document.getElementById('storage-path-style').checked
|
||||
};
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/settings/storage`, {
|
||||
method: 'PUT',
|
||||
headers: headers(),
|
||||
credentials: 'same-origin',
|
||||
body: JSON.stringify(body)
|
||||
});
|
||||
if (resp.ok) {
|
||||
showStorageStatus(t('admin.storage_saved') || 'Storage settings saved successfully', 'success');
|
||||
loadStorage();
|
||||
} else {
|
||||
const e = await resp.json().catch(() => ({}));
|
||||
showStorageStatus(`Error: ${e.message || resp.statusText}`, 'error');
|
||||
}
|
||||
} catch (e) {
|
||||
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||
}
|
||||
btn.disabled = false;
|
||||
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.storage_save') || 'Save')}`;
|
||||
}
|
||||
|
||||
async function testStorageConnection() {
|
||||
const btn = document.getElementById('btn-test-storage');
|
||||
btn.disabled = true;
|
||||
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.testing') || 'Testing...')}`;
|
||||
|
||||
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
|
||||
const body = {
|
||||
backend,
|
||||
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
|
||||
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
|
||||
s3_region: document.getElementById('storage-region').value.trim() || null,
|
||||
s3_access_key: document.getElementById('storage-access-key').value || null,
|
||||
s3_secret_key: document.getElementById('storage-secret-key').value || null,
|
||||
s3_force_path_style: document.getElementById('storage-path-style').checked
|
||||
};
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/settings/storage/test`, {
|
||||
method: 'POST',
|
||||
headers: headers(),
|
||||
credentials: 'same-origin',
|
||||
body: JSON.stringify(body)
|
||||
});
|
||||
const r = await resp.json();
|
||||
if (r.connected) {
|
||||
let msg = `${t('admin.storage_test_success') || 'Connection successful'} (${escapeHtml(r.backend_type)})`;
|
||||
if (r.available_bytes != null) msg += ` — ${formatBytes(r.available_bytes)} available`;
|
||||
showStorageStatus(msg, 'success');
|
||||
} else {
|
||||
showStorageStatus(`${t('admin.storage_test_failure') || 'Connection failed'}: ${escapeHtml(r.message)}`, 'error');
|
||||
}
|
||||
} catch (e) {
|
||||
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||
}
|
||||
btn.disabled = false;
|
||||
btn.innerHTML = `<i class="fas fa-vial"></i> ${escapeHtml(t('admin.storage_test_connection') || 'Test Connection')}`;
|
||||
}
|
||||
|
||||
/* ── Migration ── */
|
||||
|
||||
let migrationPollTimer = null;
|
||||
|
||||
function showMigrationMsg(msg, type) {
|
||||
const el = document.getElementById('migration-status-msg');
|
||||
el.textContent = msg;
|
||||
el.className = `alert alert-${type}`;
|
||||
el.style.display = '';
|
||||
}
|
||||
|
||||
function updateMigrationUI(m) {
|
||||
// Status badge
|
||||
const badge = document.getElementById('migration-status-badge');
|
||||
badge.textContent = (m.status || 'idle').charAt(0).toUpperCase() + (m.status || 'idle').slice(1);
|
||||
badge.className = `badge badge-migration badge-migration--${m.status || 'idle'}`;
|
||||
|
||||
const isActive = m.status === 'running' || m.status === 'paused';
|
||||
const isCompleted = m.status === 'completed';
|
||||
|
||||
// Progress section
|
||||
const progressSection = document.getElementById('migration-progress-section');
|
||||
progressSection.style.display = (isActive || isCompleted) ? '' : 'none';
|
||||
|
||||
if (m.total_blobs > 0) {
|
||||
const pct = Math.round((m.migrated_blobs / m.total_blobs) * 100);
|
||||
document.getElementById('migration-progress-fill').style.width = `${pct}%`;
|
||||
document.getElementById('migration-progress-label').textContent =
|
||||
`${m.migrated_blobs.toLocaleString()} / ${m.total_blobs.toLocaleString()} blobs (${pct}%)`;
|
||||
document.getElementById('migration-bytes-label').textContent =
|
||||
`${formatBytes(m.migrated_bytes)} transferred`;
|
||||
|
||||
if (m.throughput_bytes_per_sec && m.status === 'running') {
|
||||
document.getElementById('migration-throughput').textContent =
|
||||
`${formatBytes(Math.round(m.throughput_bytes_per_sec))}/s`;
|
||||
const remaining = m.total_blobs - m.migrated_blobs;
|
||||
if (remaining > 0 && m.throughput_bytes_per_sec > 0) {
|
||||
const avgBlobSize = m.migrated_bytes / Math.max(m.migrated_blobs, 1);
|
||||
const etaSecs = Math.round((remaining * avgBlobSize) / m.throughput_bytes_per_sec);
|
||||
const etaMin = Math.ceil(etaSecs / 60);
|
||||
document.getElementById('migration-eta').textContent =
|
||||
`~${etaMin} min remaining`;
|
||||
}
|
||||
} else {
|
||||
document.getElementById('migration-throughput').textContent = '';
|
||||
document.getElementById('migration-eta').textContent = '';
|
||||
}
|
||||
}
|
||||
|
||||
// Failed blobs section
|
||||
const failedSection = document.getElementById('migration-failed-section');
|
||||
if (m.failed_blobs && m.failed_blobs.length > 0) {
|
||||
failedSection.style.display = '';
|
||||
document.getElementById('migration-failed-count').textContent = m.failed_blobs.length;
|
||||
document.getElementById('migration-failed-list').textContent = m.failed_blobs.join('\n');
|
||||
} else {
|
||||
failedSection.style.display = 'none';
|
||||
}
|
||||
|
||||
// Button visibility
|
||||
document.getElementById('btn-start-migration').style.display =
|
||||
(!isActive && !isCompleted) ? '' : 'none';
|
||||
document.getElementById('btn-pause-migration').style.display =
|
||||
m.status === 'running' ? '' : 'none';
|
||||
document.getElementById('btn-resume-migration').style.display =
|
||||
m.status === 'paused' ? '' : 'none';
|
||||
document.getElementById('btn-verify-migration').style.display =
|
||||
isCompleted ? '' : 'none';
|
||||
document.getElementById('btn-complete-migration').style.display =
|
||||
isCompleted ? '' : 'none';
|
||||
}
|
||||
|
||||
async function loadMigrationStatus() {
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration`, {
|
||||
headers: headers(),
|
||||
credentials: 'same-origin'
|
||||
});
|
||||
if (!resp.ok) return;
|
||||
const m = await resp.json();
|
||||
updateMigrationUI(m);
|
||||
|
||||
// Auto-poll while running
|
||||
if (m.status === 'running') {
|
||||
if (!migrationPollTimer) {
|
||||
migrationPollTimer = setInterval(loadMigrationStatus, 2000);
|
||||
}
|
||||
} else if (migrationPollTimer) {
|
||||
clearInterval(migrationPollTimer);
|
||||
migrationPollTimer = null;
|
||||
}
|
||||
} catch (_e) { /* ignore */ }
|
||||
}
|
||||
|
||||
async function startMigration() {
|
||||
const btn = document.getElementById('btn-start-migration');
|
||||
btn.disabled = true;
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration/start`, {
|
||||
method: 'POST',
|
||||
headers: headers(),
|
||||
credentials: 'same-origin',
|
||||
body: JSON.stringify({ concurrency: 4 })
|
||||
});
|
||||
if (resp.ok) {
|
||||
showMigrationMsg(t('admin.migration_started') || 'Migration started', 'success');
|
||||
loadMigrationStatus();
|
||||
} else {
|
||||
const e = await resp.json().catch(() => ({}));
|
||||
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
|
||||
}
|
||||
} catch (e) {
|
||||
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
|
||||
}
|
||||
btn.disabled = false;
|
||||
}
|
||||
|
||||
async function pauseMigration() {
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration/pause`, {
|
||||
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||
});
|
||||
if (resp.ok) {
|
||||
showMigrationMsg(t('admin.migration_paused_msg') || 'Migration paused', 'success');
|
||||
loadMigrationStatus();
|
||||
}
|
||||
} catch (_e) { /* ignore */ }
|
||||
}
|
||||
|
||||
async function resumeMigration() {
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration/resume`, {
|
||||
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||
});
|
||||
if (resp.ok) {
|
||||
showMigrationMsg(t('admin.migration_resumed_msg') || 'Migration resumed', 'success');
|
||||
loadMigrationStatus();
|
||||
}
|
||||
} catch (_e) { /* ignore */ }
|
||||
}
|
||||
|
||||
async function verifyMigration() {
|
||||
const btn = document.getElementById('btn-verify-migration');
|
||||
btn.disabled = true;
|
||||
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.migration_verifying') || 'Verifying...')}`;
|
||||
const resultDiv = document.getElementById('migration-verify-result');
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration/verify`, {
|
||||
method: 'POST',
|
||||
headers: headers(),
|
||||
credentials: 'same-origin',
|
||||
body: JSON.stringify({ sample_size: 100 })
|
||||
});
|
||||
const r = await resp.json();
|
||||
resultDiv.style.display = '';
|
||||
if (r.passed) {
|
||||
resultDiv.innerHTML = `<div class="discovery-result ok"><strong><i class="fas fa-check-circle"></i> ${escapeHtml(t('admin.migration_verify_passed') || 'Verification passed')}</strong><p>${r.sample_checked} blobs checked, ${r.pg_blob_count} total in database</p></div>`;
|
||||
} else {
|
||||
const issues = [];
|
||||
if (r.missing_in_target.length) issues.push(`${r.missing_in_target.length} missing`);
|
||||
if (r.size_mismatches.length) issues.push(`${r.size_mismatches.length} size mismatches`);
|
||||
resultDiv.innerHTML = `<div class="discovery-result fail"><strong><i class="fas fa-times-circle"></i> ${escapeHtml(t('admin.migration_verify_failed') || 'Verification failed')}</strong><p>${issues.join(', ')}</p></div>`;
|
||||
}
|
||||
} catch (e) {
|
||||
resultDiv.style.display = '';
|
||||
resultDiv.innerHTML = `<div class="discovery-result fail"><i class="fas fa-times-circle"></i> Error: ${escapeHtml(e.message)}</div>`;
|
||||
}
|
||||
btn.disabled = false;
|
||||
btn.innerHTML = `<i class="fas fa-check-double"></i> ${escapeHtml(t('admin.migration_verify') || 'Verify Integrity')}`;
|
||||
}
|
||||
|
||||
async function completeMigration() {
|
||||
try {
|
||||
const resp = await fetch(`${API}/admin/storage/migration/complete`, {
|
||||
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||
});
|
||||
if (resp.ok) {
|
||||
showMigrationMsg(t('admin.migration_completed_msg') || 'Migration finalized. Restart the server to use the new backend.', 'success');
|
||||
loadMigrationStatus();
|
||||
} else {
|
||||
const e = await resp.json().catch(() => ({}));
|
||||
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
|
||||
}
|
||||
} catch (e) {
|
||||
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
|
||||
}
|
||||
}
|
||||
|
||||
async function init() {
|
||||
try {
|
||||
const me = await fetch(`${API}/auth/me`, {
|
||||
@@ -775,6 +1123,9 @@ document.getElementById('tab-btn-users').addEventListener('click', function () {
|
||||
document.getElementById('tab-btn-oidc').addEventListener('click', function () {
|
||||
switchTab('oidc', this);
|
||||
});
|
||||
document.getElementById('tab-btn-storage').addEventListener('click', function () {
|
||||
switchTab('storage', this);
|
||||
});
|
||||
|
||||
document.getElementById('ds-registration').addEventListener('change', function () {
|
||||
toggleRegistration(this.checked);
|
||||
@@ -797,3 +1148,20 @@ document.getElementById('cu-submit').addEventListener('click', submitCreateUser)
|
||||
|
||||
document.getElementById('btn-close-reset-pw').addEventListener('click', closeResetPasswordModal);
|
||||
document.getElementById('rp-submit').addEventListener('click', submitResetPassword);
|
||||
|
||||
/* ── Storage event listeners ── */
|
||||
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
|
||||
r.addEventListener('change', function () {
|
||||
toggleS3Form(this.value === 's3');
|
||||
});
|
||||
});
|
||||
document.getElementById('storage-preset').addEventListener('change', onStoragePresetChange);
|
||||
document.getElementById('btn-test-storage').addEventListener('click', testStorageConnection);
|
||||
document.getElementById('btn-save-storage').addEventListener('click', saveStorageSettings);
|
||||
|
||||
/* ── Migration event listeners ── */
|
||||
document.getElementById('btn-start-migration').addEventListener('click', startMigration);
|
||||
document.getElementById('btn-pause-migration').addEventListener('click', pauseMigration);
|
||||
document.getElementById('btn-resume-migration').addEventListener('click', resumeMigration);
|
||||
document.getElementById('btn-verify-migration').addEventListener('click', verifyMigration);
|
||||
document.getElementById('btn-complete-migration').addEventListener('click', completeMigration);
|
||||
|
||||
+44
-1
@@ -566,7 +566,50 @@
|
||||
"error_password_short": "Password must be at least 8 characters",
|
||||
"error_generic": "Failed",
|
||||
"error_network": "Network error: {{message}}",
|
||||
"error_create_user": "Failed to create user"
|
||||
"error_create_user": "Failed to create user",
|
||||
"tab_storage": "Storage",
|
||||
"storage_title": "Storage Backend",
|
||||
"storage_current_backend": "Active Backend",
|
||||
"storage_total_blobs": "Total Blobs",
|
||||
"storage_total_size": "Total Size",
|
||||
"storage_dedup_ratio": "Dedup Ratio",
|
||||
"storage_backend": "Backend Type",
|
||||
"storage_local": "Local Filesystem",
|
||||
"storage_s3": "S3-Compatible",
|
||||
"storage_provider_preset": "Provider Preset",
|
||||
"storage_preset_custom": "Custom",
|
||||
"storage_endpoint_url": "Endpoint URL",
|
||||
"storage_endpoint_hint": "Leave empty for Amazon S3 default",
|
||||
"storage_bucket": "Bucket",
|
||||
"storage_region": "Region",
|
||||
"storage_access_key": "Access Key ID",
|
||||
"storage_secret_key": "Secret Access Key",
|
||||
"storage_secret_configured": "A secret key is already configured",
|
||||
"storage_key_placeholder": "Leave empty to keep current value",
|
||||
"storage_path_style": "Force Path Style",
|
||||
"storage_path_style_hint": "Required for MinIO and some S3-compatible providers",
|
||||
"storage_test_connection": "Test Connection",
|
||||
"storage_test_success": "Connection successful",
|
||||
"storage_test_failure": "Connection failed",
|
||||
"storage_save": "Save",
|
||||
"storage_saved": "Storage settings saved successfully",
|
||||
"storage_migration": "Backend Migration",
|
||||
"storage_migration_coming_soon": "Backend migration will be available in a future update.",
|
||||
"migration_status_label": "Status:",
|
||||
"migration_start": "Start Migration",
|
||||
"migration_pause": "Pause",
|
||||
"migration_resume": "Resume",
|
||||
"migration_verify": "Verify Integrity",
|
||||
"migration_complete": "Finalize",
|
||||
"migration_started": "Migration started",
|
||||
"migration_paused_msg": "Migration paused",
|
||||
"migration_resumed_msg": "Migration resumed",
|
||||
"migration_completed_msg": "Migration finalized. Restart the server to use the new backend.",
|
||||
"migration_verifying": "Verifying…",
|
||||
"migration_verify_passed": "Verification passed",
|
||||
"migration_verify_failed": "Verification failed",
|
||||
"migration_failed_blobs": "failed blobs",
|
||||
"testing": "Testing…"
|
||||
},
|
||||
"profile": {
|
||||
"page_title": "Profile",
|
||||
|
||||
+44
-1
@@ -566,7 +566,50 @@
|
||||
"error_password_short": "La contraseña debe tener al menos 8 caracteres",
|
||||
"error_generic": "Error",
|
||||
"error_network": "Error de red: {{message}}",
|
||||
"error_create_user": "Error al crear usuario"
|
||||
"error_create_user": "Error al crear usuario",
|
||||
"tab_storage": "Almacenamiento",
|
||||
"storage_title": "Backend de Almacenamiento",
|
||||
"storage_current_backend": "Backend Activo",
|
||||
"storage_total_blobs": "Total de Blobs",
|
||||
"storage_total_size": "Tamaño Total",
|
||||
"storage_dedup_ratio": "Ratio de Dedup",
|
||||
"storage_backend": "Tipo de Backend",
|
||||
"storage_local": "Sistema de Archivos Local",
|
||||
"storage_s3": "Compatible con S3",
|
||||
"storage_provider_preset": "Proveedor Preconfigurado",
|
||||
"storage_preset_custom": "Personalizado",
|
||||
"storage_endpoint_url": "URL del Endpoint",
|
||||
"storage_endpoint_hint": "Dejar vacío para usar Amazon S3 por defecto",
|
||||
"storage_bucket": "Bucket",
|
||||
"storage_region": "Región",
|
||||
"storage_access_key": "Access Key ID",
|
||||
"storage_secret_key": "Secret Access Key",
|
||||
"storage_secret_configured": "Ya hay una clave secreta configurada",
|
||||
"storage_key_placeholder": "Dejar vacío para mantener el valor actual",
|
||||
"storage_path_style": "Forzar Path Style",
|
||||
"storage_path_style_hint": "Requerido para MinIO y algunos proveedores compatibles con S3",
|
||||
"storage_test_connection": "Probar Conexión",
|
||||
"storage_test_success": "Conexión exitosa",
|
||||
"storage_test_failure": "Conexión fallida",
|
||||
"storage_save": "Guardar",
|
||||
"storage_saved": "Configuración de almacenamiento guardada correctamente",
|
||||
"storage_migration": "Migración de Backend",
|
||||
"storage_migration_coming_soon": "La migración de backend estará disponible en una futura actualización.",
|
||||
"migration_status_label": "Estado:",
|
||||
"migration_start": "Iniciar Migración",
|
||||
"migration_pause": "Pausar",
|
||||
"migration_resume": "Reanudar",
|
||||
"migration_verify": "Verificar Integridad",
|
||||
"migration_complete": "Finalizar",
|
||||
"migration_started": "Migración iniciada",
|
||||
"migration_paused_msg": "Migración pausada",
|
||||
"migration_resumed_msg": "Migración reanudada",
|
||||
"migration_completed_msg": "Migración finalizada. Reinicia el servidor para usar el nuevo backend.",
|
||||
"migration_verifying": "Verificando…",
|
||||
"migration_verify_passed": "Verificación exitosa",
|
||||
"migration_verify_failed": "Verificación fallida",
|
||||
"migration_failed_blobs": "blobs fallidos",
|
||||
"testing": "Probando…"
|
||||
},
|
||||
"profile": {
|
||||
"page_title": "Perfil",
|
||||
|
||||
Reference in New Issue
Block a user