feat: pluggable storage backends (S3, Azure, local) with admin UI
Implement 4-phase external storage backends architecture: Phase 1 - Foundation: - BlobStorageBackend trait (application/ports/blob_storage_ports.rs) - LocalBlobBackend: extracted all tokio::fs ops from DedupService - S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2) - DedupService refactored to use Arc<dyn BlobStorageBackend> Phase 2 - Admin Panel: - StorageSettingsService with DB persistence + env override - Storage tab in admin panel (backend selector, S3 form, provider presets) - GET/PUT/POST endpoints for storage settings + connection test - i18n keys (en/es) and BEM CSS Phase 3 - Migration: - MigrationBlobBackend decorator (dual-read: target-first + source fallback) - Background migration job with parallel transfers + progress tracking - Migration UI (progress bar, ETA, pause/resume/verify/complete) - 6 admin API endpoints for migration lifecycle Phase 4 - Enterprise Extras: - CachedBlobBackend: LRU disk cache for remote backends - EncryptedBlobBackend: AES-256-GCM at-rest encryption - AzureBlobBackend: Azure Blob Storage support - RetryBlobBackend: exponential backoff for transient errors - Decorator composition in DI: retry → encryption → cache All 223 tests passing, clippy clean, fmt verified.
This commit is contained in:
Generated
+1502
-92
File diff suppressed because it is too large
Load Diff
@@ -58,6 +58,14 @@ dashmap = "6"
|
|||||||
socket2 = { version = "0.6.2", features = ["all"] }
|
socket2 = { version = "0.6.2", features = ["all"] }
|
||||||
urlencoding = "2.1.3"
|
urlencoding = "2.1.3"
|
||||||
utoipa = { version = "5", features = ["axum_extras", "uuid", "chrono"] }
|
utoipa = { version = "5", features = ["axum_extras", "uuid", "chrono"] }
|
||||||
|
aws-sdk-s3 = "1"
|
||||||
|
aws-config = { version = "1", features = ["behavior-version-latest"] }
|
||||||
|
aws-smithy-types = "1"
|
||||||
|
azure_core = "0.21"
|
||||||
|
azure_storage = "0.21"
|
||||||
|
azure_storage_blobs = "0.21"
|
||||||
|
aes-gcm = "0.10"
|
||||||
|
lru = "0.12"
|
||||||
|
|
||||||
[features]
|
[features]
|
||||||
default = []
|
default = []
|
||||||
|
|||||||
@@ -0,0 +1,867 @@
|
|||||||
|
# External Storage Backends — Implementation Plan
|
||||||
|
|
||||||
|
> **Purpose**: This prompt provides Claude Code with full architectural context to implement pluggable blob storage backends (S3, Backblaze B2, MinIO, etc.) for OxiCloud, across 4 phases. All changes MUST respect the existing hexagonal architecture, BLAKE3 dedup system, and coding conventions defined in `CLAUDE.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Architecture Summary
|
||||||
|
|
||||||
|
### How blobs are stored today
|
||||||
|
|
||||||
|
- **Content-addressable**: Files hashed with BLAKE3 → stored at `.blobs/{2-char-prefix}/{hash}.blob`
|
||||||
|
- **Dedup index**: PostgreSQL `storage.blobs` table (hash PK, ref_count, size, content_type)
|
||||||
|
- **Write-first strategy**: Blob written to disk BEFORE PostgreSQL upsert (PG connection never held during disk I/O)
|
||||||
|
- **Streaming reads**: 256 KB chunks via `tokio::fs::File` + `ReaderStream`
|
||||||
|
- **Range support**: `AsyncSeekExt::seek()` + `file.take()` for HTTP Range requests
|
||||||
|
|
||||||
|
### Key files
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|------|------|
|
||||||
|
| `src/application/ports/dedup_ports.rs` | `DedupPort` trait — 12 methods, the hexagonal port |
|
||||||
|
| `src/infrastructure/services/dedup_service.rs` | `DedupService` struct — sole implementation of `DedupPort` |
|
||||||
|
| `src/common/di.rs` | `AppServiceFactory` — DI composition root, builds `DedupService` |
|
||||||
|
| `src/common/config.rs` | `StorageConfig`, `AppConfig` — env var loading |
|
||||||
|
| `src/interfaces/api/handlers/admin_handler.rs` | Admin API handlers (OIDC pattern to follow) |
|
||||||
|
| `src/application/services/admin_settings_service.rs` | `AdminSettingsService` — runtime settings with env override |
|
||||||
|
| `src/domain/repositories/settings_repository.rs` | `SettingsRepository` trait |
|
||||||
|
| `src/infrastructure/repositories/pg/settings_pg_repository.rs` | PostgreSQL settings impl |
|
||||||
|
| `static/admin.html` | Admin panel HTML (3 tabs: Dashboard, Users, OIDC) |
|
||||||
|
| `static/js/views/admin/admin.js` | Admin panel JS logic |
|
||||||
|
|
||||||
|
### DedupService filesystem operations (candidates for extraction)
|
||||||
|
|
||||||
|
These are the EXACT `tokio::fs` calls inside `DedupService` that must be delegated to the new `BlobStorageBackend` trait:
|
||||||
|
|
||||||
|
```
|
||||||
|
initialize() → fs::create_dir_all (blob_root, temp_root, 256 prefix dirs)
|
||||||
|
store_from_file() → fs::metadata, fs::try_exists, fs::rename, fs::copy, fs::remove_file
|
||||||
|
read_blob_stream() → File::open + ReaderStream
|
||||||
|
read_blob_range_stream() → File::open + seek + take + ReaderStream
|
||||||
|
blob_size() → fs::metadata
|
||||||
|
remove_reference() → fs::remove_file (after PG commit)
|
||||||
|
verify_integrity() → spawn_blocking with path checks
|
||||||
|
blob_path() → PathBuf computation (sync)
|
||||||
|
```
|
||||||
|
|
||||||
|
### DedupService PostgreSQL operations (stay in DedupService, untouched)
|
||||||
|
|
||||||
|
```
|
||||||
|
store_from_file() → INSERT … ON CONFLICT … RETURNING ref_count
|
||||||
|
blob_exists() → SELECT EXISTS from storage.blobs
|
||||||
|
get_blob_metadata() → SELECT from storage.blobs
|
||||||
|
add_reference() → UPDATE ref_count + 1
|
||||||
|
remove_reference() → BEGIN TX → SELECT FOR UPDATE → DELETE if ref_count=0 → COMMIT
|
||||||
|
get_stats() → SELECT COUNT, SUM from storage.blobs
|
||||||
|
verify_integrity() → Streaming cursor SELECT from storage.blobs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Foundation (Backend Trait + Local + S3)
|
||||||
|
|
||||||
|
### Task 1.1: Create `BlobStorageBackend` trait
|
||||||
|
|
||||||
|
**File**: `src/application/ports/blob_storage_ports.rs` (NEW)
|
||||||
|
|
||||||
|
Create a minimal trait that abstracts ONLY raw byte I/O operations:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use bytes::Bytes;
|
||||||
|
use futures::Stream;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::pin::Pin;
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Health check result for storage backend connectivity
|
||||||
|
#[derive(Debug, Clone, serde::Serialize)]
|
||||||
|
pub struct StorageHealthStatus {
|
||||||
|
pub connected: bool,
|
||||||
|
pub backend_type: String,
|
||||||
|
pub message: String,
|
||||||
|
/// Optional: available space in bytes (if backend reports it)
|
||||||
|
pub available_bytes: Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
|
||||||
|
///
|
||||||
|
/// Implementations: `LocalBlobBackend`, `S3BlobBackend`, etc.
|
||||||
|
/// DedupService owns an `Arc<dyn BlobStorageBackend>` and delegates
|
||||||
|
/// all filesystem/object-store operations through this trait.
|
||||||
|
#[async_trait]
|
||||||
|
pub trait BlobStorageBackend: Send + Sync + 'static {
|
||||||
|
/// Initialize the backend (create directories, verify bucket access, etc.)
|
||||||
|
async fn initialize(&self) -> Result<(), DomainError>;
|
||||||
|
|
||||||
|
/// Store a blob from a local temporary file.
|
||||||
|
/// The backend MUST handle the case where the blob already exists (idempotent).
|
||||||
|
/// Returns the number of bytes stored.
|
||||||
|
async fn put_blob(&self, hash: &str, source_path: &Path) -> Result<u64, DomainError>;
|
||||||
|
|
||||||
|
/// Stream the full blob content.
|
||||||
|
async fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
|
||||||
|
|
||||||
|
/// Stream a byte range of the blob (for HTTP Range requests).
|
||||||
|
async fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
|
||||||
|
|
||||||
|
/// Delete a blob by hash. Must be idempotent (no error if already deleted).
|
||||||
|
async fn delete_blob(&self, hash: &str) -> Result<(), DomainError>;
|
||||||
|
|
||||||
|
/// Check if a blob exists in the backend.
|
||||||
|
async fn blob_exists(&self, hash: &str) -> Result<bool, DomainError>;
|
||||||
|
|
||||||
|
/// Get blob size in bytes without downloading content.
|
||||||
|
async fn blob_size(&self, hash: &str) -> Result<u64, DomainError>;
|
||||||
|
|
||||||
|
/// Verify connectivity and permissions. Used by admin "Test Connection" button.
|
||||||
|
async fn health_check(&self) -> Result<StorageHealthStatus, DomainError>;
|
||||||
|
|
||||||
|
/// Return the backend type name (for display in admin panel).
|
||||||
|
fn backend_type(&self) -> &'static str;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Register in `src/application/ports/mod.rs` and `src/application/mod.rs`.
|
||||||
|
|
||||||
|
### Task 1.2: Create `LocalBlobBackend`
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/local_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
Extract ALL `tokio::fs` operations from `DedupService` into this struct. This is a **pure refactor** — zero behavior change.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct LocalBlobBackend {
|
||||||
|
blob_root: PathBuf,
|
||||||
|
temp_root: PathBuf,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Methods to implement from the trait, mapping from current DedupService code:
|
||||||
|
|
||||||
|
| Trait method | Source in DedupService | Key logic |
|
||||||
|
|---|---|---|
|
||||||
|
| `initialize()` | `DedupService::initialize()` lines 115-154 | Create `.blobs/`, `.dedup_temp/`, 256 prefix dirs |
|
||||||
|
| `put_blob()` | `DedupService::store_from_file()` lines 196-253 | `fs::try_exists` → `fs::rename` (EXDEV fallback `fs::copy`) → cleanup source |
|
||||||
|
| `get_blob_stream()` | `DedupService::read_blob_stream()` lines 306-323 | `File::open` → `ReaderStream::with_capacity(256KB)` |
|
||||||
|
| `get_blob_range_stream()` | `DedupService::read_blob_range_stream()` lines 325-353 | `File::open` → `seek` → `take` → `ReaderStream` |
|
||||||
|
| `delete_blob()` | `DedupService::remove_reference()` line ~469 | `fs::remove_file` |
|
||||||
|
| `blob_exists()` | New (was inline `fs::try_exists`) | `fs::try_exists(blob_path)` |
|
||||||
|
| `blob_size()` | `DedupService::blob_size()` lines 355-369 | `fs::metadata().len()` |
|
||||||
|
| `health_check()` | New | Check blob_root is writable, return disk available via `statvfs` |
|
||||||
|
| `backend_type()` | New | Return `"local"` |
|
||||||
|
|
||||||
|
Also add a public helper:
|
||||||
|
```rust
|
||||||
|
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||||
|
let prefix = &hash[..2];
|
||||||
|
self.blob_root.join(prefix).join(format!("{hash}.blob"))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Register in `src/infrastructure/services/mod.rs`.
|
||||||
|
|
||||||
|
### Task 1.3: Refactor `DedupService` to use `BlobStorageBackend`
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/dedup_service.rs` (MODIFY)
|
||||||
|
|
||||||
|
Changes:
|
||||||
|
1. Add field: `backend: Arc<dyn BlobStorageBackend>`
|
||||||
|
2. Remove fields: `blob_root: PathBuf`, `temp_root: PathBuf` (moved to `LocalBlobBackend`)
|
||||||
|
3. Update constructor to accept `Arc<dyn BlobStorageBackend>` instead of `storage_root: &Path`
|
||||||
|
4. Replace all direct `tokio::fs` calls with `self.backend.*` calls
|
||||||
|
5. Keep `blob_path()` in DedupPort as a delegation: `self.backend.blob_path()` — BUT since `blob_path()` returns `PathBuf` and is used by thumbnails/caching services, consider adding it to the backend trait OR keeping a separate method. For S3 backends, this should return a virtual path or the method should be deprecated in favor of streaming.
|
||||||
|
|
||||||
|
**Critical**: `hash_file()` stays in `DedupService` (BLAKE3 hashing is NOT a backend concern — it always runs on local temp files before upload).
|
||||||
|
|
||||||
|
**Critical**: The write-first strategy is preserved:
|
||||||
|
```
|
||||||
|
1. hash_file() on local temp file
|
||||||
|
2. self.backend.put_blob(hash, temp_path) ← backend moves/uploads
|
||||||
|
3. INSERT INTO storage.blobs … ON CONFLICT ← PostgreSQL upsert
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical**: `remove_reference()` flow preserved:
|
||||||
|
```
|
||||||
|
1. BEGIN TX → SELECT FOR UPDATE → check ref_count
|
||||||
|
2. If ref_count == 1 → DELETE FROM storage.blobs → COMMIT
|
||||||
|
3. self.backend.delete_blob(hash) ← after PG commit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 1.4: Create `S3BlobBackend`
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/s3_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
**Dependency to add to `Cargo.toml`**:
|
||||||
|
```toml
|
||||||
|
aws-sdk-s3 = "1"
|
||||||
|
aws-config = { version = "1", features = ["behavior-version-latest"] }
|
||||||
|
aws-smithy-types = "1" # For ByteStream
|
||||||
|
```
|
||||||
|
|
||||||
|
> Note: `aws-sdk-s3` is the official AWS SDK for Rust. It's compatible with ALL S3-compatible services (Backblaze B2, MinIO, Cloudflare R2, DigitalOcean Spaces, Wasabi) via custom endpoint configuration.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct S3BlobBackend {
|
||||||
|
client: aws_sdk_s3::Client,
|
||||||
|
bucket: String,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**S3 key scheme**: Same as local — `{2-char-prefix}/{hash}.blob` (e.g., `a3/a3c5f2e8d1…blob`)
|
||||||
|
|
||||||
|
Method mapping:
|
||||||
|
|
||||||
|
| Trait method | S3 operation |
|
||||||
|
|---|---|
|
||||||
|
| `initialize()` | `head_bucket()` to verify bucket exists + permissions |
|
||||||
|
| `put_blob()` | `put_object()` with `Body::from_path(source_path)`. Check existence with `head_object()` first for idempotency |
|
||||||
|
| `get_blob_stream()` | `get_object()` → `.body.into_async_read()` → `ReaderStream` |
|
||||||
|
| `get_blob_range_stream()` | `get_object().range(format!("bytes={start}-{end}"))` → stream |
|
||||||
|
| `delete_blob()` | `delete_object()` (already idempotent in S3) |
|
||||||
|
| `blob_exists()` | `head_object()` — 200 = true, 404 = false |
|
||||||
|
| `blob_size()` | `head_object()` → `.content_length()` |
|
||||||
|
| `health_check()` | `head_bucket()` + `list_objects_v2(max_keys=1)` |
|
||||||
|
| `backend_type()` | Return `"s3"` |
|
||||||
|
|
||||||
|
**S3 Client construction**: Must support custom endpoints for non-AWS providers:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
impl S3BlobBackend {
|
||||||
|
pub async fn new(config: &S3StorageConfig) -> Result<Self, DomainError> {
|
||||||
|
let mut s3_config_builder = aws_sdk_s3::config::Builder::new()
|
||||||
|
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
|
||||||
|
.credentials_provider(
|
||||||
|
aws_sdk_s3::config::Credentials::new(
|
||||||
|
&config.access_key,
|
||||||
|
&config.secret_key,
|
||||||
|
None, None, "oxicloud",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
.behavior_version_latest();
|
||||||
|
|
||||||
|
if let Some(endpoint) = &config.endpoint_url {
|
||||||
|
s3_config_builder = s3_config_builder
|
||||||
|
.endpoint_url(endpoint)
|
||||||
|
.force_path_style(config.force_path_style);
|
||||||
|
}
|
||||||
|
|
||||||
|
let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());
|
||||||
|
Ok(Self { client, bucket: config.bucket.clone() })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 1.5: Add storage backend configuration
|
||||||
|
|
||||||
|
**File**: `src/common/config.rs` (MODIFY)
|
||||||
|
|
||||||
|
Add to existing `StorageConfig`:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub enum StorageBackendType {
|
||||||
|
Local,
|
||||||
|
S3,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct S3StorageConfig {
|
||||||
|
pub endpoint_url: Option<String>, // OXICLOUD_S3_ENDPOINT_URL
|
||||||
|
pub bucket: String, // OXICLOUD_S3_BUCKET
|
||||||
|
pub region: String, // OXICLOUD_S3_REGION (default: "us-east-1")
|
||||||
|
pub access_key: String, // OXICLOUD_S3_ACCESS_KEY
|
||||||
|
pub secret_key: String, // OXICLOUD_S3_SECRET_KEY
|
||||||
|
pub force_path_style: bool, // OXICLOUD_S3_FORCE_PATH_STYLE (default: false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add to existing StorageConfig:
|
||||||
|
pub struct StorageConfig {
|
||||||
|
pub root_dir: String, // existing
|
||||||
|
pub chunk_size: usize, // existing
|
||||||
|
pub parallel_threshold: usize, // existing
|
||||||
|
pub trash_retention_days: u32, // existing
|
||||||
|
pub max_upload_size: usize, // existing
|
||||||
|
pub backend: StorageBackendType, // NEW — OXICLOUD_STORAGE_BACKEND (default: "local")
|
||||||
|
pub s3: Option<S3StorageConfig>, // NEW — populated when backend=s3
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Env var loading** in `AppConfig::from_env()`:
|
||||||
|
```
|
||||||
|
OXICLOUD_STORAGE_BACKEND → "local" | "s3" (default: "local")
|
||||||
|
OXICLOUD_S3_ENDPOINT_URL → Optional custom endpoint
|
||||||
|
OXICLOUD_S3_BUCKET → Required when backend=s3
|
||||||
|
OXICLOUD_S3_REGION → Default "us-east-1"
|
||||||
|
OXICLOUD_S3_ACCESS_KEY → Required when backend=s3
|
||||||
|
OXICLOUD_S3_SECRET_KEY → Required when backend=s3
|
||||||
|
OXICLOUD_S3_FORCE_PATH_STYLE → Default false
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 1.6: Wire backend selection in DI
|
||||||
|
|
||||||
|
**File**: `src/common/di.rs` (MODIFY)
|
||||||
|
|
||||||
|
In `create_core_services()`, replace the current DedupService construction:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Build storage backend based on config
|
||||||
|
let blob_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
|
||||||
|
StorageBackendType::Local => {
|
||||||
|
Arc::new(LocalBlobBackend::new(&self.storage_path))
|
||||||
|
}
|
||||||
|
StorageBackendType::S3 => {
|
||||||
|
let s3_config = self.config.storage.s3.as_ref()
|
||||||
|
.expect("S3 config required when backend=s3");
|
||||||
|
Arc::new(S3BlobBackend::new(s3_config).await?)
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
blob_backend.initialize().await?;
|
||||||
|
|
||||||
|
let dedup_service = Arc::new(DedupService::new(
|
||||||
|
blob_backend.clone(),
|
||||||
|
db_pool.clone(),
|
||||||
|
maintenance_pool.clone(),
|
||||||
|
));
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 1.7: Handle `blob_path()` deprecation path
|
||||||
|
|
||||||
|
The `DedupPort::blob_path()` method returns a `PathBuf` and is used by:
|
||||||
|
- Thumbnail generation service (needs local file access)
|
||||||
|
- File content caching (moka cache)
|
||||||
|
|
||||||
|
For S3 backends, `blob_path()` has no meaning. Solutions:
|
||||||
|
1. **For thumbnails**: Change thumbnail service to accept a `Stream` instead of a `PathBuf`, OR download to a temp file first
|
||||||
|
2. **For caching**: Cache already works with streams
|
||||||
|
3. Keep `blob_path()` on `DedupPort` but make it return `Option<PathBuf>` (None for remote backends) — consumers must handle the None case
|
||||||
|
|
||||||
|
Search for all callers of `blob_path()` and update them.
|
||||||
|
|
||||||
|
### Task 1.8: Tests
|
||||||
|
|
||||||
|
- Unit test `LocalBlobBackend` in isolation (mock filesystem with temp dirs)
|
||||||
|
- Unit test `S3BlobBackend` with a mock S3 (use `aws-smithy-runtime` test utilities or `mockall`)
|
||||||
|
- Integration test: `DedupService` with `LocalBlobBackend` must pass ALL existing tests unchanged (this proves the refactor is correct)
|
||||||
|
- Add `#[cfg(test)]` inline tests in each new file following existing project convention
|
||||||
|
|
||||||
|
### Task 1.9: Pre-commit validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo clippy -- -D warnings
|
||||||
|
cargo test --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
ALL existing ~208 tests MUST pass. Zero regressions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Admin Panel Configuration
|
||||||
|
|
||||||
|
### Task 2.1: Storage settings service
|
||||||
|
|
||||||
|
**File**: `src/application/services/storage_settings_service.rs` (NEW)
|
||||||
|
|
||||||
|
Follow the EXACT same pattern as `AdminSettingsService` for OIDC. Create `StorageSettingsService`:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct StorageSettingsService {
|
||||||
|
settings_repo: Arc<SettingsPgRepository>,
|
||||||
|
env_storage_config: StorageConfig, // from AppConfig at startup
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Methods** (follow OIDC pattern):
|
||||||
|
|
||||||
|
| Method | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `get_storage_settings()` | Load from DB (category: `"storage"`), mask secrets, mark env overrides |
|
||||||
|
| `save_storage_settings(dto, user_id)` | Upsert each field to `admin_settings`, mark secrets with `is_secret: true` |
|
||||||
|
| `test_storage_connection(dto)` | Build temporary backend from DTO config, call `health_check()`, return result |
|
||||||
|
| `load_effective_storage_config()` | Merge: DB settings + env var overrides + defaults |
|
||||||
|
| `get_env_overrides()` | Return list of `OXICLOUD_S3_*` / `OXICLOUD_STORAGE_*` env vars that are set |
|
||||||
|
|
||||||
|
**DB keys** (category: `"storage"`):
|
||||||
|
|
||||||
|
```
|
||||||
|
storage.backend → "local" | "s3"
|
||||||
|
storage.s3.endpoint_url → string (optional)
|
||||||
|
storage.s3.bucket → string
|
||||||
|
storage.s3.region → string
|
||||||
|
storage.s3.access_key → string (is_secret: true)
|
||||||
|
storage.s3.secret_key → string (is_secret: true)
|
||||||
|
storage.s3.force_path_style → "true" | "false"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 2.2: Storage admin API endpoints
|
||||||
|
|
||||||
|
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
|
||||||
|
|
||||||
|
Add to `admin_routes()`:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
.route("/settings/storage", get(get_storage_settings))
|
||||||
|
.route("/settings/storage", put(save_storage_settings))
|
||||||
|
.route("/settings/storage/test", post(test_storage_connection))
|
||||||
|
```
|
||||||
|
|
||||||
|
**Handler implementations** (follow OIDC handlers exactly):
|
||||||
|
|
||||||
|
| Handler | Method | Body | Response |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `get_storage_settings` | GET | — | `StorageSettingsDto` (secrets masked, env_overrides listed) |
|
||||||
|
| `save_storage_settings` | PUT | `SaveStorageSettingsDto` | `{ "message": "Storage settings saved" }` |
|
||||||
|
| `test_storage_connection` | POST | `TestStorageConnectionDto` | `StorageTestResultDto { connected, message, available_bytes }` |
|
||||||
|
|
||||||
|
**DTOs** (add to `src/application/dtos/`):
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[derive(Serialize)]
|
||||||
|
pub struct StorageSettingsDto {
|
||||||
|
pub backend: String, // "local" | "s3"
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
pub s3_access_key_set: bool, // masked — never send actual key
|
||||||
|
pub s3_secret_key_set: bool, // masked — never send actual secret
|
||||||
|
pub s3_force_path_style: bool,
|
||||||
|
pub env_overrides: Vec<String>, // which fields are locked by env vars
|
||||||
|
// Current stats
|
||||||
|
pub current_backend: String,
|
||||||
|
pub total_blobs: u64,
|
||||||
|
pub total_bytes_stored: u64,
|
||||||
|
pub dedup_ratio: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub struct SaveStorageSettingsDto {
|
||||||
|
pub backend: String,
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
pub s3_access_key: Option<String>, // only sent if changed
|
||||||
|
pub s3_secret_key: Option<String>, // only sent if changed
|
||||||
|
pub s3_force_path_style: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub struct TestStorageConnectionDto {
|
||||||
|
pub backend: String,
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
pub s3_access_key: Option<String>,
|
||||||
|
pub s3_secret_key: Option<String>,
|
||||||
|
pub s3_force_path_style: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
pub struct StorageTestResultDto {
|
||||||
|
pub connected: bool,
|
||||||
|
pub message: String,
|
||||||
|
pub backend_type: String,
|
||||||
|
pub available_bytes: Option<u64>,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 2.3: Wire `StorageSettingsService` into DI
|
||||||
|
|
||||||
|
**File**: `src/common/di.rs` (MODIFY)
|
||||||
|
|
||||||
|
Add `StorageSettingsService` construction alongside `AdminSettingsService`. Add to `AppState`.
|
||||||
|
|
||||||
|
### Task 2.4: Admin Panel — Storage tab (HTML)
|
||||||
|
|
||||||
|
**File**: `static/admin.html` (MODIFY)
|
||||||
|
|
||||||
|
Add a 4th tab button after the OIDC tab:
|
||||||
|
|
||||||
|
```html
|
||||||
|
<button class="admin-tab" id="tab-btn-storage">
|
||||||
|
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
|
||||||
|
</button>
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `tab-storage` content div with:
|
||||||
|
|
||||||
|
1. **Backend selector** — Radio buttons: Local Filesystem / S3-Compatible
|
||||||
|
2. **S3 configuration form** (shown/hidden based on selector):
|
||||||
|
- Provider preset dropdown (Amazon S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces, Wasabi, Custom)
|
||||||
|
- Endpoint URL field
|
||||||
|
- Bucket field
|
||||||
|
- Region field
|
||||||
|
- Access Key ID field
|
||||||
|
- Secret Key field (password type)
|
||||||
|
- Force Path Style checkbox
|
||||||
|
- ENV badges on fields overridden by env vars (same as OIDC)
|
||||||
|
3. **Test Connection button** → calls `POST /api/admin/settings/storage/test`
|
||||||
|
4. **Save button** → calls `PUT /api/admin/settings/storage`
|
||||||
|
5. **Current Status section**:
|
||||||
|
- Active backend type
|
||||||
|
- Total blobs / total size / dedup ratio (from `DedupStatsDto`)
|
||||||
|
6. **Migration section** (Phase 3 — can be placeholder with "Coming soon")
|
||||||
|
|
||||||
|
### Task 2.5: Admin Panel — Storage tab (JS)
|
||||||
|
|
||||||
|
**File**: `static/js/views/admin/admin.js` (MODIFY)
|
||||||
|
|
||||||
|
Follow OIDC tab patterns:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Provider presets
|
||||||
|
const STORAGE_PRESETS = {
|
||||||
|
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
|
||||||
|
'backblaze': { endpoint: 's3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
|
||||||
|
'cloudflare-r2': { endpoint: '{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
|
||||||
|
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
|
||||||
|
'digitalocean': { endpoint: '{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
|
||||||
|
'wasabi': { endpoint: 's3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
|
||||||
|
'custom': { endpoint: '', region: '', pathStyle: false },
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Functions:
|
||||||
|
- `loadStorage()` — `GET /api/admin/settings/storage` → populate form
|
||||||
|
- `saveStorage()` — Collect form → `PUT /api/admin/settings/storage`
|
||||||
|
- `testStorageConnection()` — Collect form → `POST /api/admin/settings/storage/test` → show result
|
||||||
|
- `onPresetChange(preset)` — Auto-fill endpoint/region/pathStyle from preset
|
||||||
|
- `toggleS3Form(visible)` — Show/hide S3 fields when backend radio changes
|
||||||
|
|
||||||
|
### Task 2.6: i18n keys
|
||||||
|
|
||||||
|
**Files**: `static/locales/*.json` (MODIFY at minimum `en.json` and `es.json`)
|
||||||
|
|
||||||
|
Add translation keys for all new UI labels:
|
||||||
|
```
|
||||||
|
admin.tab_storage
|
||||||
|
admin.storage_backend
|
||||||
|
admin.storage_local
|
||||||
|
admin.storage_s3
|
||||||
|
admin.storage_provider_preset
|
||||||
|
admin.storage_endpoint_url
|
||||||
|
admin.storage_bucket
|
||||||
|
admin.storage_region
|
||||||
|
admin.storage_access_key
|
||||||
|
admin.storage_secret_key
|
||||||
|
admin.storage_path_style
|
||||||
|
admin.storage_test_connection
|
||||||
|
admin.storage_test_success
|
||||||
|
admin.storage_test_failure
|
||||||
|
admin.storage_save
|
||||||
|
admin.storage_saved
|
||||||
|
admin.storage_current_status
|
||||||
|
admin.storage_total_blobs
|
||||||
|
admin.storage_total_size
|
||||||
|
admin.storage_dedup_ratio
|
||||||
|
admin.storage_migration
|
||||||
|
admin.storage_migration_coming_soon
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 2.7: CSS for storage tab
|
||||||
|
|
||||||
|
**File**: `static/css/admin.css` (MODIFY)
|
||||||
|
|
||||||
|
Add styles for:
|
||||||
|
- `.storage-backend-selector` — Radio button group
|
||||||
|
- `.storage-provider-presets` — Dropdown styling
|
||||||
|
- `.storage-form` — Form fields (reuse existing OIDC form patterns)
|
||||||
|
- `.storage-status-grid` — Stats display
|
||||||
|
- BEM methodology, CSS custom properties for colors, no raw hex/rgb
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Migration Between Backends
|
||||||
|
|
||||||
|
### Task 3.1: `MigrationBlobBackend` wrapper
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/migration_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
A `BlobStorageBackend` decorator that enables zero-downtime migration between backends:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct MigrationBlobBackend {
|
||||||
|
source: Arc<dyn BlobStorageBackend>, // old backend (read fallback)
|
||||||
|
target: Arc<dyn BlobStorageBackend>, // new backend (primary for writes)
|
||||||
|
state: Arc<RwLock<MigrationState>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct MigrationState {
|
||||||
|
pub status: MigrationStatus,
|
||||||
|
pub total_blobs: u64,
|
||||||
|
pub migrated_blobs: u64,
|
||||||
|
pub migrated_bytes: u64,
|
||||||
|
pub failed_blobs: Vec<String>, // hashes that failed
|
||||||
|
pub started_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||||
|
pub completed_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub enum MigrationStatus {
|
||||||
|
Idle, // no migration in progress
|
||||||
|
Running, // background job active
|
||||||
|
Paused, // manually paused
|
||||||
|
Completed, // all blobs migrated
|
||||||
|
Failed, // unrecoverable error
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behavior**:
|
||||||
|
|
||||||
|
| Operation | During Migration |
|
||||||
|
|---|---|
|
||||||
|
| `put_blob()` | Write to **target** only |
|
||||||
|
| `get_blob_stream()` | Try **target** first → fallback to **source** (+ schedule lazy copy) |
|
||||||
|
| `get_blob_range_stream()` | Same fallback strategy |
|
||||||
|
| `delete_blob()` | Delete from **both** (best-effort on source) |
|
||||||
|
| `blob_exists()` | Check **target** first, then **source** |
|
||||||
|
| `blob_size()` | Check **target** first, then **source** |
|
||||||
|
|
||||||
|
### Task 3.2: Background migration job
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/migration_job.rs` (NEW)
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub async fn run_migration(
|
||||||
|
source: Arc<dyn BlobStorageBackend>,
|
||||||
|
target: Arc<dyn BlobStorageBackend>,
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
state: Arc<RwLock<MigrationState>>,
|
||||||
|
concurrency: usize, // default: 4 parallel transfers
|
||||||
|
bandwidth_limit: Option<u64>, // bytes/sec, None = unlimited
|
||||||
|
) -> Result<(), DomainError>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Algorithm**:
|
||||||
|
1. Query `SELECT hash, size FROM storage.blobs ORDER BY hash` with streaming cursor
|
||||||
|
2. For each blob hash:
|
||||||
|
a. Check if already exists in target (`target.blob_exists(hash)`)
|
||||||
|
b. If not: stream from source → write to temp file → `target.put_blob(hash, temp)`
|
||||||
|
c. Update `MigrationState` counters
|
||||||
|
d. Respect bandwidth limit via `tokio::time::sleep` throttling
|
||||||
|
3. Use `futures::stream::buffer_unordered(concurrency)` for parallel transfers
|
||||||
|
4. On error: log, add to `failed_blobs`, continue (don't abort entire migration)
|
||||||
|
|
||||||
|
### Task 3.3: Migration API endpoints
|
||||||
|
|
||||||
|
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
|
||||||
|
|
||||||
|
Add to `admin_routes()`:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
.route("/storage/migration", get(get_migration_status))
|
||||||
|
.route("/storage/migration/start", post(start_migration))
|
||||||
|
.route("/storage/migration/pause", post(pause_migration))
|
||||||
|
.route("/storage/migration/resume", post(resume_migration))
|
||||||
|
.route("/storage/migration/complete", post(complete_migration))
|
||||||
|
```
|
||||||
|
|
||||||
|
| Endpoint | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `GET /migration` | Return `MigrationState` (status, progress, ETA) |
|
||||||
|
| `POST /migration/start` | Begin background migration from current → configured backend |
|
||||||
|
| `POST /migration/pause` | Pause the background job |
|
||||||
|
| `POST /migration/resume` | Resume paused migration |
|
||||||
|
| `POST /migration/complete` | Finalize: switch primary backend, optionally clean up source |
|
||||||
|
|
||||||
|
### Task 3.4: Migration UI in admin panel
|
||||||
|
|
||||||
|
In the Storage tab's Migration section:
|
||||||
|
|
||||||
|
1. **Start Migration button** (when backend config differs from active)
|
||||||
|
2. **Progress bar**: `{migrated} / {total} blobs ({percent}%) — {bytes_migrated} transferred`
|
||||||
|
3. **Estimated time remaining** (based on throughput)
|
||||||
|
4. **Pause / Resume button**
|
||||||
|
5. **Complete Migration button** (enabled only when 100% migrated)
|
||||||
|
6. **Failed blobs list** (expandable, with retry button)
|
||||||
|
7. **Status badge**: Idle / Running / Paused / Completed / Failed
|
||||||
|
|
||||||
|
### Task 3.5: Integrity verification post-migration
|
||||||
|
|
||||||
|
After migration completes (before `complete_migration`):
|
||||||
|
1. Run `verify_integrity()` against the target backend
|
||||||
|
2. Compare blob count in PG vs target backend
|
||||||
|
3. Sample-verify N random blobs (download + BLAKE3 hash check)
|
||||||
|
4. Show verification results in admin UI before allowing finalization
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4 — Enterprise Extras
|
||||||
|
|
||||||
|
### Task 4.1: `CachedBlobBackend` — LRU local disk cache
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/cached_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
A `BlobStorageBackend` decorator for remote backends (S3, Azure) that caches hot blobs on local SSD:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct CachedBlobBackend {
|
||||||
|
inner: Arc<dyn BlobStorageBackend>, // S3 backend
|
||||||
|
cache_dir: PathBuf, // local cache directory
|
||||||
|
max_cache_bytes: u64, // configurable limit
|
||||||
|
index: Arc<RwLock<LruCache<String, CacheEntry>>>,
|
||||||
|
current_size: Arc<AtomicU64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct CacheEntry {
|
||||||
|
size: u64,
|
||||||
|
last_accessed: Instant,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Behavior**:
|
||||||
|
- **Reads**: Check local cache first → cache hit returns local file stream → cache miss downloads from inner backend, writes to cache, returns stream
|
||||||
|
- **Writes**: `put_blob()` writes to inner backend AND local cache simultaneously
|
||||||
|
- **Eviction**: LRU eviction when `current_size` exceeds `max_cache_bytes`
|
||||||
|
- **Startup**: Scan cache directory to rebuild index
|
||||||
|
|
||||||
|
**Configuration** (admin panel):
|
||||||
|
```
|
||||||
|
storage.cache.enabled → "true" | "false"
|
||||||
|
storage.cache.max_size_bytes → u64 (default: 50 GB)
|
||||||
|
storage.cache.path → PathBuf (default: "{storage_root}/.cache")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Env vars**:
|
||||||
|
```
|
||||||
|
OXICLOUD_STORAGE_CACHE_ENABLED=true
|
||||||
|
OXICLOUD_STORAGE_CACHE_MAX_SIZE=53687091200 # 50 GB
|
||||||
|
OXICLOUD_STORAGE_CACHE_PATH=/fast-ssd/oxicloud-cache
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 4.2: Client-side encryption (AES-256-GCM)
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/encrypted_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
Another `BlobStorageBackend` decorator that encrypts blobs before sending to the inner backend:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct EncryptedBlobBackend {
|
||||||
|
inner: Arc<dyn BlobStorageBackend>,
|
||||||
|
encryption_key: [u8; 32], // AES-256 key
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Dependency**: Add `aes-gcm = "0.10"` to Cargo.toml.
|
||||||
|
|
||||||
|
**Behavior**:
|
||||||
|
- `put_blob()`: Read source → encrypt with AES-256-GCM (random 96-bit nonce prepended) → write encrypted to temp → `inner.put_blob(hash, encrypted_temp)`
|
||||||
|
- `get_blob_stream()`: `inner.get_blob_stream()` → decrypt stream → return plaintext stream
|
||||||
|
- **CRITICAL**: BLAKE3 hash is computed on the PLAINTEXT (before encryption), so dedup still works across encrypted backends
|
||||||
|
- **Nonce storage**: Prepend 12-byte nonce to each encrypted blob (total overhead: 28 bytes per blob — 12 nonce + 16 GCM tag)
|
||||||
|
|
||||||
|
**Configuration** (admin panel):
|
||||||
|
```
|
||||||
|
storage.encryption.enabled → "true" | "false"
|
||||||
|
storage.encryption.key → base64-encoded 32-byte key (is_secret: true)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key generation**: Provide an admin API endpoint `POST /api/admin/settings/storage/generate-key` that generates a cryptographically secure key and returns it once (user must save it).
|
||||||
|
|
||||||
|
**WARNING in admin UI**: "If you lose the encryption key, all data in the storage backend is IRRECOVERABLY LOST. Back up this key securely."
|
||||||
|
|
||||||
|
### Task 4.3: Azure Blob Storage backend
|
||||||
|
|
||||||
|
**File**: `src/infrastructure/services/azure_blob_backend.rs` (NEW)
|
||||||
|
|
||||||
|
**Dependency**: `azure_storage_blobs = "0.21"`, `azure_storage = "0.21"`
|
||||||
|
|
||||||
|
Same `BlobStorageBackend` trait implementation targeting Azure Blob Storage:
|
||||||
|
- Container = bucket equivalent
|
||||||
|
- Blob key scheme: `{prefix}/{hash}.blob` (same as S3/local)
|
||||||
|
- Authentication: Account Name + Account Key OR SAS token
|
||||||
|
|
||||||
|
**Configuration**:
|
||||||
|
```
|
||||||
|
OXICLOUD_AZURE_ACCOUNT_NAME
|
||||||
|
OXICLOUD_AZURE_ACCOUNT_KEY
|
||||||
|
OXICLOUD_AZURE_CONTAINER
|
||||||
|
OXICLOUD_AZURE_SAS_TOKEN # alternative auth
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 4.4: Bandwidth throttling & retry policies
|
||||||
|
|
||||||
|
Apply to all remote backends (S3, Azure):
|
||||||
|
|
||||||
|
**Throttling**:
|
||||||
|
- Configurable upload/download bandwidth limit per-backend
|
||||||
|
- Implemented via `tokio::time::sleep` between chunks
|
||||||
|
- Admin configurable: `storage.s3.max_upload_bandwidth`, `storage.s3.max_download_bandwidth`
|
||||||
|
|
||||||
|
**Retry policy** (exponential backoff):
|
||||||
|
```rust
|
||||||
|
pub struct RetryPolicy {
|
||||||
|
pub max_retries: u32, // default: 3
|
||||||
|
pub initial_backoff_ms: u64, // default: 100
|
||||||
|
pub max_backoff_ms: u64, // default: 10_000
|
||||||
|
pub backoff_multiplier: f64, // default: 2.0
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Wrap remote backend calls with retry logic for transient errors (network timeouts, 500s, 503s).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Invariants (MUST be preserved)
|
||||||
|
|
||||||
|
1. **BLAKE3 hashing**: Always performed locally on temp files, never delegated to backend
|
||||||
|
2. **PostgreSQL dedup index**: `storage.blobs` table remains the source of truth for ref_count, metadata
|
||||||
|
3. **Write-first strategy**: Blob stored in backend BEFORE PostgreSQL upsert
|
||||||
|
4. **Remove-after-commit**: Blob deleted from backend AFTER PostgreSQL transaction commits
|
||||||
|
5. **Streaming reads**: All blob reads return `Pin<Box<dyn Stream>>`, never load full blob into memory
|
||||||
|
6. **Zero framework deps in domain**: `BlobStorageBackend` trait lives in `application/ports/`, not infrastructure
|
||||||
|
7. **DI via AppState**: All backends are `Arc`-wrapped and assembled in `common/di.rs`
|
||||||
|
8. **Env var precedence**: `OXICLOUD_*` env vars always override DB-stored admin settings
|
||||||
|
9. **Admin guard**: All storage admin endpoints require JWT with role `"admin"`
|
||||||
|
10. **Existing tests**: All ~208 tests must pass after Phase 1 refactor
|
||||||
|
|
||||||
|
## Backend Decorator Composition
|
||||||
|
|
||||||
|
The backends compose as decorators. In `di.rs`, the assembly looks like:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Phase 1: Base backend
|
||||||
|
let base_backend: Arc<dyn BlobStorageBackend> = match config {
|
||||||
|
Local => Arc::new(LocalBlobBackend::new(...)),
|
||||||
|
S3 => Arc::new(S3BlobBackend::new(...).await?),
|
||||||
|
Azure => Arc::new(AzureBlobBackend::new(...).await?),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Phase 4: Optional encryption layer
|
||||||
|
let backend = if encryption_enabled {
|
||||||
|
Arc::new(EncryptedBlobBackend::new(base_backend, key))
|
||||||
|
} else {
|
||||||
|
base_backend
|
||||||
|
};
|
||||||
|
|
||||||
|
// Phase 4: Optional cache layer (only for remote backends)
|
||||||
|
let backend = if cache_enabled && !matches!(config, Local) {
|
||||||
|
Arc::new(CachedBlobBackend::new(backend, cache_config))
|
||||||
|
} else {
|
||||||
|
backend
|
||||||
|
};
|
||||||
|
|
||||||
|
// Phase 3: Optional migration wrapper
|
||||||
|
let backend = if migration_in_progress {
|
||||||
|
Arc::new(MigrationBlobBackend::new(old_backend, backend, state))
|
||||||
|
} else {
|
||||||
|
backend
|
||||||
|
};
|
||||||
|
|
||||||
|
// Finally: DedupService uses the composed backend
|
||||||
|
let dedup_service = Arc::new(DedupService::new(backend, pool, maintenance_pool));
|
||||||
|
```
|
||||||
|
|
||||||
|
This decorator pattern means each feature (encryption, caching, migration) is:
|
||||||
|
- Independently testable
|
||||||
|
- Independently toggleable
|
||||||
|
- Zero overhead when disabled
|
||||||
|
- Composable in any order
|
||||||
@@ -128,3 +128,97 @@ pub struct DashboardStatsDto {
|
|||||||
pub users_over_quota: i64,
|
pub users_over_quota: i64,
|
||||||
pub registration_enabled: bool,
|
pub registration_enabled: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// Storage Settings DTOs (Admin Panel)
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// Current storage settings returned to admin UI (secrets masked)
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct StorageSettingsDto {
|
||||||
|
/// Active backend type: "local" or "s3"
|
||||||
|
pub backend: String,
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
/// True if an access key is configured (never reveals the actual value)
|
||||||
|
pub s3_access_key_set: bool,
|
||||||
|
/// True if a secret key is configured (never reveals the actual value)
|
||||||
|
pub s3_secret_key_set: bool,
|
||||||
|
pub s3_force_path_style: bool,
|
||||||
|
/// Field names overridden by environment variables (read-only in UI)
|
||||||
|
pub env_overrides: Vec<String>,
|
||||||
|
// ── Current stats ──
|
||||||
|
pub current_backend: String,
|
||||||
|
pub total_blobs: u64,
|
||||||
|
pub total_bytes_stored: u64,
|
||||||
|
pub dedup_ratio: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Request body for saving storage settings from the admin panel
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct SaveStorageSettingsDto {
|
||||||
|
pub backend: String,
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
/// Only update if provided and non-empty (None = keep existing)
|
||||||
|
pub s3_access_key: Option<String>,
|
||||||
|
/// Only update if provided and non-empty (None = keep existing)
|
||||||
|
pub s3_secret_key: Option<String>,
|
||||||
|
pub s3_force_path_style: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Request body for testing a storage connection
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct TestStorageConnectionDto {
|
||||||
|
pub backend: String,
|
||||||
|
pub s3_endpoint_url: Option<String>,
|
||||||
|
pub s3_bucket: Option<String>,
|
||||||
|
pub s3_region: Option<String>,
|
||||||
|
pub s3_access_key: Option<String>,
|
||||||
|
pub s3_secret_key: Option<String>,
|
||||||
|
pub s3_force_path_style: Option<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result of a storage connection test
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct StorageTestResultDto {
|
||||||
|
pub connected: bool,
|
||||||
|
pub message: String,
|
||||||
|
pub backend_type: String,
|
||||||
|
pub available_bytes: Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// Migration DTOs (Admin Panel — Storage Migration)
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
/// Migration progress returned by `GET /api/admin/storage/migration`.
|
||||||
|
/// Re-exports the `MigrationState` shape for the admin UI.
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct MigrationStateDto {
|
||||||
|
pub status: String,
|
||||||
|
pub total_blobs: u64,
|
||||||
|
pub migrated_blobs: u64,
|
||||||
|
pub migrated_bytes: u64,
|
||||||
|
pub failed_blobs: Vec<String>,
|
||||||
|
pub started_at: Option<String>,
|
||||||
|
pub completed_at: Option<String>,
|
||||||
|
/// Estimated throughput in bytes/sec (for UI ETA calculation).
|
||||||
|
pub throughput_bytes_per_sec: Option<f64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Request body for `POST /api/admin/storage/migration/start`.
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct StartMigrationDto {
|
||||||
|
/// How many blobs to copy in parallel (default: 4).
|
||||||
|
pub concurrency: Option<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Request body (empty) for `POST /api/admin/storage/migration/verify`.
|
||||||
|
#[derive(Debug, Serialize, Deserialize)]
|
||||||
|
pub struct VerifyMigrationDto {
|
||||||
|
/// Number of random blobs to sample-check (default: 100).
|
||||||
|
pub sample_size: Option<usize>,
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
//! Blob Storage Backend Port — abstracts raw byte I/O for content-addressable storage.
|
||||||
|
//!
|
||||||
|
//! This trait decouples `DedupService` from any specific storage medium.
|
||||||
|
//! Implementations include:
|
||||||
|
//! - `LocalBlobBackend` — local filesystem (default)
|
||||||
|
//! - `S3BlobBackend` — any S3-compatible service (AWS, Backblaze B2, MinIO, R2…)
|
||||||
|
//!
|
||||||
|
//! `DedupService` owns an `Arc<dyn BlobStorageBackend>` and delegates all
|
||||||
|
//! byte-level I/O through this trait, keeping BLAKE3 hashing, ref-counting
|
||||||
|
//! and PostgreSQL index logic in `DedupService` itself.
|
||||||
|
|
||||||
|
use bytes::Bytes;
|
||||||
|
use futures::Stream;
|
||||||
|
use serde::Serialize;
|
||||||
|
use std::future::Future;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Boxed future alias used by [`BlobStorageBackend`] to keep the trait dyn-compatible.
|
||||||
|
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||||
|
|
||||||
|
/// Pinned boxed byte stream — the return type for blob reads.
|
||||||
|
pub type BlobStream = Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>;
|
||||||
|
|
||||||
|
/// Health-check result returned by [`BlobStorageBackend::health_check`].
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct StorageHealthStatus {
|
||||||
|
/// Whether the backend is reachable and functional.
|
||||||
|
pub connected: bool,
|
||||||
|
/// Human-readable backend identifier (e.g. `"local"`, `"s3"`).
|
||||||
|
pub backend_type: String,
|
||||||
|
/// Descriptive status message.
|
||||||
|
pub message: String,
|
||||||
|
/// Available space in bytes, if the backend can report it.
|
||||||
|
pub available_bytes: Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
|
||||||
|
///
|
||||||
|
/// Every method operates on a *hash key* that uniquely identifies a blob.
|
||||||
|
/// The backend is responsible for mapping the hash to its own addressing
|
||||||
|
/// scheme (filesystem path, S3 key, etc.).
|
||||||
|
///
|
||||||
|
/// Returns boxed futures so the trait is dyn-compatible (`Arc<dyn BlobStorageBackend>`).
|
||||||
|
pub trait BlobStorageBackend: Send + Sync + 'static {
|
||||||
|
/// Perform any one-time setup (create directories, verify bucket, etc.).
|
||||||
|
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>>;
|
||||||
|
|
||||||
|
/// Store a blob from a local temporary file.
|
||||||
|
///
|
||||||
|
/// Must be **idempotent**: if the blob already exists the call succeeds
|
||||||
|
/// without overwriting. Returns the number of bytes stored.
|
||||||
|
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>>;
|
||||||
|
|
||||||
|
/// Stream the full blob content in chunks.
|
||||||
|
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>>;
|
||||||
|
|
||||||
|
/// Stream a byte range of the blob (for HTTP Range requests / video seek).
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> BoxFut<'_, Result<BlobStream, DomainError>>;
|
||||||
|
|
||||||
|
/// Delete a blob by hash. Must be **idempotent** (no error if already gone).
|
||||||
|
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>>;
|
||||||
|
|
||||||
|
/// Check if a blob exists in the backend.
|
||||||
|
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>>;
|
||||||
|
|
||||||
|
/// Get blob size in bytes without downloading content.
|
||||||
|
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>>;
|
||||||
|
|
||||||
|
/// Verify connectivity and permissions (used by the admin "Test Connection" button).
|
||||||
|
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>>;
|
||||||
|
|
||||||
|
/// Return the backend type name for display (e.g. `"local"`, `"s3"`).
|
||||||
|
fn backend_type(&self) -> &'static str;
|
||||||
|
|
||||||
|
/// Return the local filesystem path for a blob, if available.
|
||||||
|
///
|
||||||
|
/// Only meaningful for local-filesystem backends. Remote backends
|
||||||
|
/// return `None`; callers that need a local file must stream + spool.
|
||||||
|
fn local_blob_path(&self, hash: &str) -> Option<PathBuf>;
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
pub mod auth_ports;
|
pub mod auth_ports;
|
||||||
|
pub mod blob_storage_ports;
|
||||||
pub mod cache_ports;
|
pub mod cache_ports;
|
||||||
pub mod calendar_ports;
|
pub mod calendar_ports;
|
||||||
pub mod carddav_ports;
|
pub mod carddav_ports;
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ pub mod nextcloud_login_flow_service;
|
|||||||
pub mod recent_service;
|
pub mod recent_service;
|
||||||
pub mod search_service;
|
pub mod search_service;
|
||||||
pub mod share_service;
|
pub mod share_service;
|
||||||
|
pub mod storage_settings_service;
|
||||||
pub mod storage_usage_service;
|
pub mod storage_usage_service;
|
||||||
pub mod trash_service;
|
pub mod trash_service;
|
||||||
pub mod wopi_lock_service;
|
pub mod wopi_lock_service;
|
||||||
|
|||||||
@@ -0,0 +1,337 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use crate::application::dtos::settings_dto::{
|
||||||
|
SaveStorageSettingsDto, StorageSettingsDto, StorageTestResultDto, TestStorageConnectionDto,
|
||||||
|
};
|
||||||
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
|
use crate::common::config::{S3StorageConfig, StorageConfig};
|
||||||
|
use crate::common::errors::{DomainError, ErrorKind};
|
||||||
|
use crate::domain::repositories::settings_repository::SettingsRepository;
|
||||||
|
use crate::infrastructure::repositories::pg::SettingsPgRepository;
|
||||||
|
use crate::infrastructure::services::dedup_service::DedupService;
|
||||||
|
use crate::infrastructure::services::s3_blob_backend::S3BlobBackend;
|
||||||
|
|
||||||
|
/// Storage settings service — manages storage backend configuration via the admin panel.
|
||||||
|
///
|
||||||
|
/// Configuration priority: **env vars > DB settings > defaults**.
|
||||||
|
pub struct StorageSettingsService {
|
||||||
|
settings_repo: Arc<SettingsPgRepository>,
|
||||||
|
env_storage_config: StorageConfig,
|
||||||
|
dedup_service: Arc<DedupService>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl StorageSettingsService {
|
||||||
|
pub fn new(
|
||||||
|
settings_repo: Arc<SettingsPgRepository>,
|
||||||
|
env_storage_config: StorageConfig,
|
||||||
|
dedup_service: Arc<DedupService>,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
settings_repo,
|
||||||
|
env_storage_config,
|
||||||
|
dedup_service,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Detect which storage fields are overridden by environment variables.
|
||||||
|
fn get_env_overrides(&self) -> Vec<String> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
let vars = [
|
||||||
|
("OXICLOUD_STORAGE_BACKEND", "backend"),
|
||||||
|
("OXICLOUD_S3_ENDPOINT_URL", "s3_endpoint_url"),
|
||||||
|
("OXICLOUD_S3_BUCKET", "s3_bucket"),
|
||||||
|
("OXICLOUD_S3_REGION", "s3_region"),
|
||||||
|
("OXICLOUD_S3_ACCESS_KEY", "s3_access_key"),
|
||||||
|
("OXICLOUD_S3_SECRET_KEY", "s3_secret_key"),
|
||||||
|
("OXICLOUD_S3_FORCE_PATH_STYLE", "s3_force_path_style"),
|
||||||
|
];
|
||||||
|
for (env_key, field_name) in &vars {
|
||||||
|
if std::env::var(env_key).is_ok() {
|
||||||
|
out.push(field_name.to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Apply environment variable overrides on top of a config.
|
||||||
|
fn apply_env_overrides(&self, config: &mut StorageConfig) {
|
||||||
|
let e = &self.env_storage_config;
|
||||||
|
if std::env::var("OXICLOUD_STORAGE_BACKEND").is_ok() {
|
||||||
|
config.backend = e.backend.clone();
|
||||||
|
}
|
||||||
|
// S3 env overrides — only apply if S3 config exists in env
|
||||||
|
if let Some(env_s3) = &e.s3 {
|
||||||
|
let s3 = config.s3.get_or_insert_with(|| S3StorageConfig {
|
||||||
|
endpoint_url: None,
|
||||||
|
bucket: String::new(),
|
||||||
|
region: "us-east-1".to_string(),
|
||||||
|
access_key: String::new(),
|
||||||
|
secret_key: String::new(),
|
||||||
|
force_path_style: false,
|
||||||
|
});
|
||||||
|
if std::env::var("OXICLOUD_S3_ENDPOINT_URL").is_ok() {
|
||||||
|
s3.endpoint_url = env_s3.endpoint_url.clone();
|
||||||
|
}
|
||||||
|
if std::env::var("OXICLOUD_S3_BUCKET").is_ok() {
|
||||||
|
s3.bucket = env_s3.bucket.clone();
|
||||||
|
}
|
||||||
|
if std::env::var("OXICLOUD_S3_REGION").is_ok() {
|
||||||
|
s3.region = env_s3.region.clone();
|
||||||
|
}
|
||||||
|
if std::env::var("OXICLOUD_S3_ACCESS_KEY").is_ok() {
|
||||||
|
s3.access_key = env_s3.access_key.clone();
|
||||||
|
}
|
||||||
|
if std::env::var("OXICLOUD_S3_SECRET_KEY").is_ok() {
|
||||||
|
s3.secret_key = env_s3.secret_key.clone();
|
||||||
|
}
|
||||||
|
if std::env::var("OXICLOUD_S3_FORCE_PATH_STYLE").is_ok() {
|
||||||
|
s3.force_path_style = env_s3.force_path_style;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load effective storage config: DB settings + env var overrides + defaults.
|
||||||
|
pub async fn load_effective_storage_config(&self) -> Result<StorageConfig, DomainError> {
|
||||||
|
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
|
||||||
|
let d = StorageConfig::default();
|
||||||
|
|
||||||
|
let backend = db
|
||||||
|
.get("storage.backend")
|
||||||
|
.map(|v| match v.as_str() {
|
||||||
|
"s3" => crate::common::config::StorageBackendType::S3,
|
||||||
|
"azure" => crate::common::config::StorageBackendType::Azure,
|
||||||
|
_ => crate::common::config::StorageBackendType::Local,
|
||||||
|
})
|
||||||
|
.unwrap_or(d.backend);
|
||||||
|
|
||||||
|
let s3 = {
|
||||||
|
let bucket = db.get("storage.s3.bucket").cloned().unwrap_or_default();
|
||||||
|
if bucket.is_empty() {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
Some(S3StorageConfig {
|
||||||
|
endpoint_url: db
|
||||||
|
.get("storage.s3.endpoint_url")
|
||||||
|
.cloned()
|
||||||
|
.filter(|s| !s.is_empty()),
|
||||||
|
bucket,
|
||||||
|
region: db
|
||||||
|
.get("storage.s3.region")
|
||||||
|
.cloned()
|
||||||
|
.unwrap_or_else(|| "us-east-1".to_string()),
|
||||||
|
access_key: db.get("storage.s3.access_key").cloned().unwrap_or_default(),
|
||||||
|
secret_key: db.get("storage.s3.secret_key").cloned().unwrap_or_default(),
|
||||||
|
force_path_style: db
|
||||||
|
.get("storage.s3.force_path_style")
|
||||||
|
.and_then(|v| v.parse().ok())
|
||||||
|
.unwrap_or(false),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut config = StorageConfig {
|
||||||
|
backend,
|
||||||
|
s3,
|
||||||
|
..self.env_storage_config.clone()
|
||||||
|
};
|
||||||
|
|
||||||
|
self.apply_env_overrides(&mut config);
|
||||||
|
Ok(config)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get storage settings for display in admin UI (secrets masked).
|
||||||
|
pub async fn get_storage_settings(&self) -> Result<StorageSettingsDto, DomainError> {
|
||||||
|
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
|
||||||
|
|
||||||
|
let has_access_key = db
|
||||||
|
.get("storage.s3.access_key")
|
||||||
|
.map(|s| !s.is_empty())
|
||||||
|
.unwrap_or(false)
|
||||||
|
|| std::env::var("OXICLOUD_S3_ACCESS_KEY")
|
||||||
|
.map(|s| !s.is_empty())
|
||||||
|
.unwrap_or(false);
|
||||||
|
|
||||||
|
let has_secret_key = db
|
||||||
|
.get("storage.s3.secret_key")
|
||||||
|
.map(|s| !s.is_empty())
|
||||||
|
.unwrap_or(false)
|
||||||
|
|| std::env::var("OXICLOUD_S3_SECRET_KEY")
|
||||||
|
.map(|s| !s.is_empty())
|
||||||
|
.unwrap_or(false);
|
||||||
|
|
||||||
|
let effective = self.load_effective_storage_config().await?;
|
||||||
|
let stats = self.dedup_service.get_stats().await;
|
||||||
|
let current_backend = self.dedup_service.backend().backend_type().to_string();
|
||||||
|
|
||||||
|
let backend_str = match effective.backend {
|
||||||
|
crate::common::config::StorageBackendType::Local => "local",
|
||||||
|
crate::common::config::StorageBackendType::S3 => "s3",
|
||||||
|
crate::common::config::StorageBackendType::Azure => "azure",
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(StorageSettingsDto {
|
||||||
|
backend: backend_str.to_string(),
|
||||||
|
s3_endpoint_url: effective.s3.as_ref().and_then(|s| s.endpoint_url.clone()),
|
||||||
|
s3_bucket: effective.s3.as_ref().map(|s| s.bucket.clone()),
|
||||||
|
s3_region: effective.s3.as_ref().map(|s| s.region.clone()),
|
||||||
|
s3_access_key_set: has_access_key,
|
||||||
|
s3_secret_key_set: has_secret_key,
|
||||||
|
s3_force_path_style: effective.s3.as_ref().is_some_and(|s| s.force_path_style),
|
||||||
|
env_overrides: self.get_env_overrides(),
|
||||||
|
current_backend,
|
||||||
|
total_blobs: stats.total_blobs,
|
||||||
|
total_bytes_stored: stats.total_bytes_stored,
|
||||||
|
dedup_ratio: stats.dedup_ratio,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Save storage settings to DB.
|
||||||
|
pub async fn save_storage_settings(
|
||||||
|
&self,
|
||||||
|
dto: SaveStorageSettingsDto,
|
||||||
|
updated_by: Uuid,
|
||||||
|
) -> Result<(), DomainError> {
|
||||||
|
let cat = "storage";
|
||||||
|
let by = Some(updated_by);
|
||||||
|
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.backend", &dto.backend, cat, false, by)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if let Some(ref v) = dto.s3_endpoint_url {
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.s3.endpoint_url", v, cat, false, by)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
if let Some(ref v) = dto.s3_bucket {
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.s3.bucket", v, cat, false, by)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
if let Some(ref v) = dto.s3_region {
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.s3.region", v, cat, false, by)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
if let Some(ref v) = dto.s3_access_key
|
||||||
|
&& !v.is_empty()
|
||||||
|
{
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.s3.access_key", v, cat, true, by)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
if let Some(ref v) = dto.s3_secret_key
|
||||||
|
&& !v.is_empty()
|
||||||
|
{
|
||||||
|
self.settings_repo
|
||||||
|
.set("storage.s3.secret_key", v, cat, true, by)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
if let Some(v) = dto.s3_force_path_style {
|
||||||
|
self.settings_repo
|
||||||
|
.set(
|
||||||
|
"storage.s3.force_path_style",
|
||||||
|
&v.to_string(),
|
||||||
|
cat,
|
||||||
|
false,
|
||||||
|
by,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!("Storage settings saved by admin (backend={})", dto.backend);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Test a storage connection by building a temporary backend and calling health_check().
|
||||||
|
pub async fn test_storage_connection(
|
||||||
|
&self,
|
||||||
|
dto: TestStorageConnectionDto,
|
||||||
|
) -> Result<StorageTestResultDto, DomainError> {
|
||||||
|
match dto.backend.as_str() {
|
||||||
|
"local" => {
|
||||||
|
// Test local backend health via the current dedup service backend
|
||||||
|
let status = self.dedup_service.backend().health_check().await?;
|
||||||
|
Ok(StorageTestResultDto {
|
||||||
|
connected: status.connected,
|
||||||
|
message: status.message,
|
||||||
|
backend_type: "local".to_string(),
|
||||||
|
available_bytes: status.available_bytes,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
"s3" => {
|
||||||
|
let bucket = dto.s3_bucket.as_deref().unwrap_or_default();
|
||||||
|
if bucket.is_empty() {
|
||||||
|
return Ok(StorageTestResultDto {
|
||||||
|
connected: false,
|
||||||
|
message: "S3 bucket name is required".to_string(),
|
||||||
|
backend_type: "s3".to_string(),
|
||||||
|
available_bytes: None,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build a temporary S3 backend from the DTO values,
|
||||||
|
// falling back to existing DB/env config for missing fields.
|
||||||
|
let effective = self.load_effective_storage_config().await.ok();
|
||||||
|
let existing_s3 = effective.as_ref().and_then(|c| c.s3.as_ref());
|
||||||
|
|
||||||
|
let config = S3StorageConfig {
|
||||||
|
endpoint_url: dto
|
||||||
|
.s3_endpoint_url
|
||||||
|
.clone()
|
||||||
|
.or_else(|| existing_s3.and_then(|s| s.endpoint_url.clone())),
|
||||||
|
bucket: bucket.to_string(),
|
||||||
|
region: dto.s3_region.clone().unwrap_or_else(|| {
|
||||||
|
existing_s3
|
||||||
|
.map(|s| s.region.clone())
|
||||||
|
.unwrap_or_else(|| "us-east-1".to_string())
|
||||||
|
}),
|
||||||
|
access_key: dto
|
||||||
|
.s3_access_key
|
||||||
|
.clone()
|
||||||
|
.filter(|s| !s.is_empty())
|
||||||
|
.unwrap_or_else(|| {
|
||||||
|
existing_s3
|
||||||
|
.map(|s| s.access_key.clone())
|
||||||
|
.unwrap_or_default()
|
||||||
|
}),
|
||||||
|
secret_key: dto
|
||||||
|
.s3_secret_key
|
||||||
|
.clone()
|
||||||
|
.filter(|s| !s.is_empty())
|
||||||
|
.unwrap_or_else(|| {
|
||||||
|
existing_s3
|
||||||
|
.map(|s| s.secret_key.clone())
|
||||||
|
.unwrap_or_default()
|
||||||
|
}),
|
||||||
|
force_path_style: dto
|
||||||
|
.s3_force_path_style
|
||||||
|
.unwrap_or_else(|| existing_s3.is_some_and(|s| s.force_path_style)),
|
||||||
|
};
|
||||||
|
|
||||||
|
let backend = S3BlobBackend::new(&config);
|
||||||
|
match backend.health_check().await {
|
||||||
|
Ok(status) => Ok(StorageTestResultDto {
|
||||||
|
connected: status.connected,
|
||||||
|
message: status.message,
|
||||||
|
backend_type: "s3".to_string(),
|
||||||
|
available_bytes: status.available_bytes,
|
||||||
|
}),
|
||||||
|
Err(e) => Ok(StorageTestResultDto {
|
||||||
|
connected: false,
|
||||||
|
message: format!("Connection failed: {}", e),
|
||||||
|
backend_type: "s3".to_string(),
|
||||||
|
available_bytes: None,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
other => Err(DomainError::new(
|
||||||
|
ErrorKind::InvalidInput,
|
||||||
|
"Storage",
|
||||||
|
format!("Unknown backend type: {}", other),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -211,6 +211,127 @@ pub struct StorageConfig {
|
|||||||
/// Maximum upload file size in bytes (default: 10 GB).
|
/// Maximum upload file size in bytes (default: 10 GB).
|
||||||
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
|
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
|
||||||
pub max_upload_size: usize,
|
pub max_upload_size: usize,
|
||||||
|
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
|
||||||
|
pub backend: StorageBackendType,
|
||||||
|
/// S3-compatible backend configuration (used when `backend == S3`).
|
||||||
|
pub s3: Option<S3StorageConfig>,
|
||||||
|
/// Azure Blob Storage configuration (used when `backend == Azure`).
|
||||||
|
pub azure: Option<AzureStorageConfig>,
|
||||||
|
/// Local disk cache for remote backends.
|
||||||
|
pub cache: BlobCacheConfig,
|
||||||
|
/// Client-side encryption.
|
||||||
|
pub encryption: EncryptionConfig,
|
||||||
|
/// Retry policy for remote backends.
|
||||||
|
pub retry: RetryConfig,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Which blob storage backend to use.
|
||||||
|
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||||
|
pub enum StorageBackendType {
|
||||||
|
/// Local filesystem (default).
|
||||||
|
#[default]
|
||||||
|
Local,
|
||||||
|
/// Any S3-compatible object store (AWS, Backblaze B2, R2, MinIO, …).
|
||||||
|
S3,
|
||||||
|
/// Azure Blob Storage.
|
||||||
|
Azure,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Configuration for an S3-compatible blob storage backend.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct S3StorageConfig {
|
||||||
|
/// Custom endpoint URL (required for non-AWS providers).
|
||||||
|
pub endpoint_url: Option<String>,
|
||||||
|
/// S3 bucket name.
|
||||||
|
pub bucket: String,
|
||||||
|
/// AWS region (default: `us-east-1`).
|
||||||
|
pub region: String,
|
||||||
|
/// Access key ID.
|
||||||
|
pub access_key: String,
|
||||||
|
/// Secret access key.
|
||||||
|
pub secret_key: String,
|
||||||
|
/// Force path-style access (required for MinIO, R2, some providers).
|
||||||
|
pub force_path_style: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Configuration for Azure Blob Storage.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct AzureStorageConfig {
|
||||||
|
/// Azure storage account name.
|
||||||
|
pub account_name: String,
|
||||||
|
/// Azure storage account key.
|
||||||
|
pub account_key: String,
|
||||||
|
/// Container name.
|
||||||
|
pub container: String,
|
||||||
|
/// Optional SAS token (alternative to account key).
|
||||||
|
pub sas_token: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// LRU local disk cache configuration for remote blob backends.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct BlobCacheConfig {
|
||||||
|
/// Enable the LRU disk cache (only useful for remote backends).
|
||||||
|
pub enabled: bool,
|
||||||
|
/// Maximum cache size in bytes (default: 50 GB).
|
||||||
|
pub max_size_bytes: u64,
|
||||||
|
/// Cache directory path (default: `{root_dir}/.blob-cache`).
|
||||||
|
pub cache_path: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for BlobCacheConfig {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
enabled: false,
|
||||||
|
max_size_bytes: 50 * 1024 * 1024 * 1024, // 50 GB
|
||||||
|
cache_path: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Client-side encryption configuration.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct EncryptionConfig {
|
||||||
|
/// Enable AES-256-GCM encryption for blobs at rest.
|
||||||
|
pub enabled: bool,
|
||||||
|
/// Base64-encoded 32-byte encryption key.
|
||||||
|
pub key_base64: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for EncryptionConfig {
|
||||||
|
#[allow(clippy::derivable_impls)]
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
enabled: false,
|
||||||
|
key_base64: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Retry policy configuration for remote backends.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct RetryConfig {
|
||||||
|
/// Enable retry with exponential backoff.
|
||||||
|
pub enabled: bool,
|
||||||
|
/// Maximum number of retry attempts.
|
||||||
|
pub max_retries: u32,
|
||||||
|
/// Initial backoff in milliseconds.
|
||||||
|
pub initial_backoff_ms: u64,
|
||||||
|
/// Maximum backoff in milliseconds.
|
||||||
|
pub max_backoff_ms: u64,
|
||||||
|
/// Backoff multiplier.
|
||||||
|
pub backoff_multiplier: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for RetryConfig {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
enabled: true,
|
||||||
|
max_retries: 3,
|
||||||
|
initial_backoff_ms: 100,
|
||||||
|
max_backoff_ms: 10_000,
|
||||||
|
backoff_multiplier: 2.0,
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for StorageConfig {
|
impl Default for StorageConfig {
|
||||||
@@ -227,6 +348,12 @@ impl Default for StorageConfig {
|
|||||||
parallel_threshold: 100 * 1024 * 1024, // 100 MB
|
parallel_threshold: 100 * 1024 * 1024, // 100 MB
|
||||||
trash_retention_days: 30, // 30 days
|
trash_retention_days: 30, // 30 days
|
||||||
max_upload_size: MAX_UPLOAD_SIZE,
|
max_upload_size: MAX_UPLOAD_SIZE,
|
||||||
|
backend: StorageBackendType::Local,
|
||||||
|
s3: None,
|
||||||
|
azure: None,
|
||||||
|
cache: BlobCacheConfig::default(),
|
||||||
|
encryption: EncryptionConfig::default(),
|
||||||
|
retry: RetryConfig::default(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -828,6 +955,95 @@ impl AppConfig {
|
|||||||
config.storage.max_upload_size = val;
|
config.storage.max_upload_size = val;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Storage backend selection
|
||||||
|
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
|
||||||
|
match backend.to_lowercase().as_str() {
|
||||||
|
"s3" => config.storage.backend = StorageBackendType::S3,
|
||||||
|
"azure" => config.storage.backend = StorageBackendType::Azure,
|
||||||
|
_ => config.storage.backend = StorageBackendType::Local,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// S3-compatible storage configuration
|
||||||
|
if config.storage.backend == StorageBackendType::S3 {
|
||||||
|
let bucket = env::var("OXICLOUD_S3_BUCKET").unwrap_or_default();
|
||||||
|
if bucket.is_empty() {
|
||||||
|
tracing::warn!("OXICLOUD_STORAGE_BACKEND=s3 but OXICLOUD_S3_BUCKET is not set");
|
||||||
|
}
|
||||||
|
config.storage.s3 = Some(S3StorageConfig {
|
||||||
|
endpoint_url: env::var("OXICLOUD_S3_ENDPOINT_URL").ok(),
|
||||||
|
bucket,
|
||||||
|
region: env::var("OXICLOUD_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
|
||||||
|
access_key: env::var("OXICLOUD_S3_ACCESS_KEY").unwrap_or_default(),
|
||||||
|
secret_key: env::var("OXICLOUD_S3_SECRET_KEY").unwrap_or_default(),
|
||||||
|
force_path_style: env::var("OXICLOUD_S3_FORCE_PATH_STYLE")
|
||||||
|
.map(|v| v.parse::<bool>().unwrap_or(false))
|
||||||
|
.unwrap_or(false),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Azure Blob Storage configuration
|
||||||
|
if config.storage.backend == StorageBackendType::Azure {
|
||||||
|
let container = env::var("OXICLOUD_AZURE_CONTAINER").unwrap_or_default();
|
||||||
|
if container.is_empty() {
|
||||||
|
tracing::warn!(
|
||||||
|
"OXICLOUD_STORAGE_BACKEND=azure but OXICLOUD_AZURE_CONTAINER is not set"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
config.storage.azure = Some(AzureStorageConfig {
|
||||||
|
account_name: env::var("OXICLOUD_AZURE_ACCOUNT_NAME").unwrap_or_default(),
|
||||||
|
account_key: env::var("OXICLOUD_AZURE_ACCOUNT_KEY").unwrap_or_default(),
|
||||||
|
container,
|
||||||
|
sas_token: env::var("OXICLOUD_AZURE_SAS_TOKEN").ok(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Blob cache configuration
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_ENABLED") {
|
||||||
|
config.storage.cache.enabled = v.parse::<bool>().unwrap_or(false);
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_MAX_SIZE")
|
||||||
|
&& let Ok(bytes) = v.parse::<u64>()
|
||||||
|
{
|
||||||
|
config.storage.cache.max_size_bytes = bytes;
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_PATH") {
|
||||||
|
config.storage.cache.cache_path = Some(v);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Encryption configuration
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_ENABLED") {
|
||||||
|
config.storage.encryption.enabled = v.parse::<bool>().unwrap_or(false);
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_KEY") {
|
||||||
|
config.storage.encryption.key_base64 = Some(v);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retry configuration
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_ENABLED") {
|
||||||
|
config.storage.retry.enabled = v.parse::<bool>().unwrap_or(true);
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_RETRIES")
|
||||||
|
&& let Ok(n) = v.parse::<u32>()
|
||||||
|
{
|
||||||
|
config.storage.retry.max_retries = n;
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_INITIAL_BACKOFF_MS")
|
||||||
|
&& let Ok(n) = v.parse::<u64>()
|
||||||
|
{
|
||||||
|
config.storage.retry.initial_backoff_ms = n;
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_BACKOFF_MS")
|
||||||
|
&& let Ok(n) = v.parse::<u64>()
|
||||||
|
{
|
||||||
|
config.storage.retry.max_backoff_ms = n;
|
||||||
|
}
|
||||||
|
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_BACKOFF_MULTIPLIER")
|
||||||
|
&& let Ok(n) = v.parse::<f64>()
|
||||||
|
{
|
||||||
|
config.storage.retry.backoff_multiplier = n;
|
||||||
|
}
|
||||||
|
|
||||||
// OIDC configuration
|
// OIDC configuration
|
||||||
if let Ok(v) = env::var("OXICLOUD_OIDC_ENABLED") {
|
if let Ok(v) = env::var("OXICLOUD_OIDC_ENABLED") {
|
||||||
config.oidc.enabled = v.parse::<bool>().unwrap_or(false);
|
config.oidc.enabled = v.parse::<bool>().unwrap_or(false);
|
||||||
|
|||||||
+118
-1
@@ -2,10 +2,14 @@ use sqlx::PgPool;
|
|||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
|
use crate::common::config::StorageBackendType;
|
||||||
use crate::infrastructure::db::DbPools;
|
use crate::infrastructure::db::DbPools;
|
||||||
|
|
||||||
use crate::application::services::admin_settings_service::AdminSettingsService;
|
use crate::application::services::admin_settings_service::AdminSettingsService;
|
||||||
use crate::application::services::auth_application_service::AuthApplicationService;
|
use crate::application::services::auth_application_service::AuthApplicationService;
|
||||||
|
use crate::application::services::storage_settings_service::StorageSettingsService;
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::MigrationState;
|
||||||
|
|
||||||
use crate::application::ports::file_ports::FileUseCaseFactory;
|
use crate::application::ports::file_ports::FileUseCaseFactory;
|
||||||
use crate::application::services::favorites_service::FavoritesService;
|
use crate::application::services::favorites_service::FavoritesService;
|
||||||
@@ -153,10 +157,110 @@ impl AppServiceFactory {
|
|||||||
);
|
);
|
||||||
image_transcode_service.initialize().await?;
|
image_transcode_service.initialize().await?;
|
||||||
|
|
||||||
|
// Build blob storage backend based on configuration
|
||||||
|
let base_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
|
||||||
|
StorageBackendType::S3 => {
|
||||||
|
let s3_config = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.s3
|
||||||
|
.as_ref()
|
||||||
|
.expect("S3 config required when OXICLOUD_STORAGE_BACKEND=s3");
|
||||||
|
Arc::new(
|
||||||
|
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3_config),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
StorageBackendType::Azure => {
|
||||||
|
let az_config = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.azure
|
||||||
|
.as_ref()
|
||||||
|
.expect("Azure config required when OXICLOUD_STORAGE_BACKEND=azure");
|
||||||
|
Arc::new(
|
||||||
|
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(
|
||||||
|
az_config,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
StorageBackendType::Local => Arc::new(
|
||||||
|
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
|
||||||
|
&self.storage_path,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Stack decorators: retry → encryption → cache (inner-to-outer)
|
||||||
|
let mut blob_backend: Arc<dyn BlobStorageBackend> = base_backend;
|
||||||
|
|
||||||
|
// Retry decorator (for remote backends)
|
||||||
|
if self.config.storage.retry.enabled
|
||||||
|
&& self.config.storage.backend != StorageBackendType::Local
|
||||||
|
{
|
||||||
|
use crate::infrastructure::services::retry_blob_backend::{
|
||||||
|
RetryBlobBackend, RetryPolicy,
|
||||||
|
};
|
||||||
|
let policy = RetryPolicy {
|
||||||
|
max_retries: self.config.storage.retry.max_retries,
|
||||||
|
initial_backoff: std::time::Duration::from_millis(
|
||||||
|
self.config.storage.retry.initial_backoff_ms,
|
||||||
|
),
|
||||||
|
max_backoff: std::time::Duration::from_millis(
|
||||||
|
self.config.storage.retry.max_backoff_ms,
|
||||||
|
),
|
||||||
|
backoff_multiplier: self.config.storage.retry.backoff_multiplier,
|
||||||
|
};
|
||||||
|
blob_backend = Arc::new(RetryBlobBackend::new(blob_backend, policy));
|
||||||
|
tracing::info!("Blob storage retry decorator enabled");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Encryption decorator
|
||||||
|
if self.config.storage.encryption.enabled {
|
||||||
|
use crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend;
|
||||||
|
let key_b64 = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.encryption
|
||||||
|
.key_base64
|
||||||
|
.as_ref()
|
||||||
|
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY required when encryption is enabled");
|
||||||
|
let key_bytes =
|
||||||
|
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, key_b64)
|
||||||
|
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY must be valid base64");
|
||||||
|
let key: [u8; 32] = key_bytes.try_into().expect(
|
||||||
|
"OXICLOUD_STORAGE_ENCRYPTION_KEY must be exactly 32 bytes (base64 of 32 bytes)",
|
||||||
|
);
|
||||||
|
blob_backend = Arc::new(EncryptedBlobBackend::new(blob_backend, &key));
|
||||||
|
tracing::info!("Blob storage encryption decorator enabled (AES-256-GCM)");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cache decorator (for remote backends only)
|
||||||
|
if self.config.storage.cache.enabled
|
||||||
|
&& self.config.storage.backend != StorageBackendType::Local
|
||||||
|
{
|
||||||
|
use crate::infrastructure::services::cached_blob_backend::{
|
||||||
|
BlobCacheConfig as CacheCfg, CachedBlobBackend,
|
||||||
|
};
|
||||||
|
let cache_path = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.cache
|
||||||
|
.cache_path
|
||||||
|
.as_ref()
|
||||||
|
.map(std::path::PathBuf::from)
|
||||||
|
.unwrap_or_else(|| self.storage_path.join(".blob-cache"));
|
||||||
|
let cfg = CacheCfg {
|
||||||
|
cache_dir: cache_path,
|
||||||
|
max_cache_bytes: self.config.storage.cache.max_size_bytes,
|
||||||
|
};
|
||||||
|
blob_backend = Arc::new(CachedBlobBackend::new(blob_backend, &cfg));
|
||||||
|
tracing::info!("Blob storage LRU disk cache enabled");
|
||||||
|
}
|
||||||
|
|
||||||
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
|
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
|
||||||
let dedup_service = Arc::new(
|
let dedup_service = Arc::new(
|
||||||
crate::infrastructure::services::dedup_service::DedupService::new(
|
crate::infrastructure::services::dedup_service::DedupService::new(
|
||||||
&self.storage_path,
|
blob_backend,
|
||||||
db_pool.clone(),
|
db_pool.clone(),
|
||||||
maintenance_pool.clone(),
|
maintenance_pool.clone(),
|
||||||
),
|
),
|
||||||
@@ -631,6 +735,8 @@ impl AppServiceFactory {
|
|||||||
auth_service: auth_services,
|
auth_service: auth_services,
|
||||||
nextcloud: nextcloud_services,
|
nextcloud: nextcloud_services,
|
||||||
admin_settings_service: None,
|
admin_settings_service: None,
|
||||||
|
storage_settings_service: None,
|
||||||
|
migration_state: Arc::new(tokio::sync::RwLock::new(MigrationState::default())),
|
||||||
trash_service,
|
trash_service,
|
||||||
share_service,
|
share_service,
|
||||||
favorites_service,
|
favorites_service,
|
||||||
@@ -700,6 +806,15 @@ impl AppServiceFactory {
|
|||||||
|
|
||||||
app_state.admin_settings_service = Some(admin_svc.clone());
|
app_state.admin_settings_service = Some(admin_svc.clone());
|
||||||
|
|
||||||
|
// 9b-1b. Wire storage settings service (reuses same settings_repo)
|
||||||
|
let storage_settings_svc = Arc::new(StorageSettingsService::new(
|
||||||
|
settings_repo.clone(),
|
||||||
|
self.config.storage.clone(),
|
||||||
|
app_state.core.dedup_service.clone(),
|
||||||
|
));
|
||||||
|
app_state.storage_settings_service = Some(storage_settings_svc);
|
||||||
|
tracing::info!("Storage settings service initialized");
|
||||||
|
|
||||||
// 9b-2. Log whether system needs first-time admin setup
|
// 9b-2. Log whether system needs first-time admin setup
|
||||||
if !admin_svc.is_system_initialized().await {
|
if !admin_svc.is_system_initialized().await {
|
||||||
tracing::warn!("╔══════════════════════════════════════════════════════════╗");
|
tracing::warn!("╔══════════════════════════════════════════════════════════╗");
|
||||||
@@ -928,6 +1043,8 @@ pub struct AppState {
|
|||||||
pub auth_service: Option<AuthServices>,
|
pub auth_service: Option<AuthServices>,
|
||||||
pub nextcloud: Option<NextcloudServices>,
|
pub nextcloud: Option<NextcloudServices>,
|
||||||
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
||||||
|
pub storage_settings_service: Option<Arc<StorageSettingsService>>,
|
||||||
|
pub migration_state: Arc<tokio::sync::RwLock<MigrationState>>,
|
||||||
pub trash_service: Option<Arc<TrashService>>,
|
pub trash_service: Option<Arc<TrashService>>,
|
||||||
pub share_service: Option<Arc<ShareService>>,
|
pub share_service: Option<Arc<ShareService>>,
|
||||||
pub favorites_service: Option<Arc<FavoritesService>>,
|
pub favorites_service: Option<Arc<FavoritesService>>,
|
||||||
|
|||||||
@@ -0,0 +1,302 @@
|
|||||||
|
//! Azure Blob Storage Backend — stores blobs in an Azure Storage container.
|
||||||
|
//!
|
||||||
|
//! Authenticates via Account Name + Account Key (or SAS token).
|
||||||
|
//! Blob key scheme mirrors local/S3: `{2-char-prefix}/{hash}.blob`.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
|
||||||
|
use azure_storage::StorageCredentials;
|
||||||
|
use azure_storage_blobs::prelude::*;
|
||||||
|
use bytes::Bytes;
|
||||||
|
use futures::StreamExt;
|
||||||
|
use tokio::fs;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::common::config::AzureStorageConfig;
|
||||||
|
use crate::domain::errors::{DomainError, ErrorKind};
|
||||||
|
|
||||||
|
/// Azure Blob Storage backend.
|
||||||
|
pub struct AzureBlobBackend {
|
||||||
|
container_client: ContainerClient,
|
||||||
|
container_name: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AzureBlobBackend {
|
||||||
|
/// Build a new Azure backend from configuration.
|
||||||
|
pub fn new(config: &AzureStorageConfig) -> Self {
|
||||||
|
let credentials = if let Some(ref sas) = config.sas_token {
|
||||||
|
StorageCredentials::sas_token(sas).expect("Invalid SAS token")
|
||||||
|
} else {
|
||||||
|
StorageCredentials::access_key(&config.account_name, config.account_key.clone())
|
||||||
|
};
|
||||||
|
|
||||||
|
let container_client = ClientBuilder::new(&config.account_name, credentials)
|
||||||
|
.container_client(&config.container);
|
||||||
|
|
||||||
|
Self {
|
||||||
|
container_client,
|
||||||
|
container_name: config.container.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute the blob name for a given hash.
|
||||||
|
fn blob_name(hash: &str) -> String {
|
||||||
|
let prefix = &hash[0..2];
|
||||||
|
format!("{prefix}/{hash}.blob")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get a `BlobClient` for a given hash.
|
||||||
|
fn blob_client(&self, hash: &str) -> BlobClient {
|
||||||
|
self.container_client.blob_client(Self::blob_name(hash))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for AzureBlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
// Verify container exists by getting its properties
|
||||||
|
self.container_client.get_properties().await.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"Azure",
|
||||||
|
format!("Cannot access container '{}': {}", self.container_name, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
"Azure blob backend initialized: container={}",
|
||||||
|
self.container_name
|
||||||
|
);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
let source_path = source_path.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
|
||||||
|
// Check if blob already exists (idempotent)
|
||||||
|
if client.get_properties().await.is_ok() {
|
||||||
|
let file_size = fs::metadata(&source_path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to stat source file: {e}"),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
.len();
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
return Ok(file_size);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read file and upload as block blob
|
||||||
|
let data = fs::read(&source_path).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Azure", format!("Failed to read source: {e}"))
|
||||||
|
})?;
|
||||||
|
let file_size = data.len() as u64;
|
||||||
|
|
||||||
|
client.put_block_blob(data).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
Ok(file_size)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
|
||||||
|
let mut result_data: Vec<u8> = Vec::new();
|
||||||
|
let mut stream = client.get().into_stream();
|
||||||
|
|
||||||
|
while let Some(response) = stream.next().await {
|
||||||
|
let response = response.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to get blob {hash}: {e}"),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let mut body = response.data;
|
||||||
|
while let Some(chunk) = body.next().await {
|
||||||
|
let chunk = chunk.map_err(|e| {
|
||||||
|
DomainError::internal_error("Azure", format!("Stream read error: {e}"))
|
||||||
|
})?;
|
||||||
|
result_data.extend_from_slice(&chunk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let stream: BlobStream = Box::pin(futures::stream::once(async move {
|
||||||
|
Ok(Bytes::from(result_data))
|
||||||
|
}));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
|
||||||
|
let range = match end {
|
||||||
|
Some(e) => azure_core::request_options::Range::new(start, e),
|
||||||
|
None => azure_core::request_options::Range::new(start, u64::MAX),
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut result_data: Vec<u8> = Vec::new();
|
||||||
|
let mut stream = client.get().range(range).into_stream();
|
||||||
|
|
||||||
|
while let Some(response) = stream.next().await {
|
||||||
|
let response = response.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to get blob range {hash}: {e}"),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let mut body = response.data;
|
||||||
|
while let Some(chunk) = body.next().await {
|
||||||
|
let chunk = chunk.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"Azure",
|
||||||
|
format!("Stream range read error: {e}"),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
result_data.extend_from_slice(&chunk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let stream: BlobStream = Box::pin(futures::stream::once(async move {
|
||||||
|
Ok(Bytes::from(result_data))
|
||||||
|
}));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
|
||||||
|
// Azure delete is not fully idempotent — 404 is expected for missing blobs
|
||||||
|
match client.delete().await {
|
||||||
|
Ok(_) => Ok(()),
|
||||||
|
Err(e) => {
|
||||||
|
// If 404, treat as success (idempotent)
|
||||||
|
let status = e.as_http_error().map(|h| h.status());
|
||||||
|
if status == Some(azure_core::StatusCode::NotFound) {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(DomainError::internal_error(
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to delete blob {hash}: {e}"),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
match client.get_properties().await {
|
||||||
|
Ok(_) => Ok(true),
|
||||||
|
Err(e) => {
|
||||||
|
let status = e.as_http_error().map(|h| h.status());
|
||||||
|
if status == Some(azure_core::StatusCode::NotFound) {
|
||||||
|
Ok(false)
|
||||||
|
} else {
|
||||||
|
Err(DomainError::internal_error(
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to check blob {hash}: {e}"),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let client = self.blob_client(&hash);
|
||||||
|
let props = client.get_properties().await.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Azure",
|
||||||
|
format!("Failed to stat blob {hash}: {e}"),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
Ok(props.blob.properties.content_length)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
Box::pin(async move {
|
||||||
|
match self.container_client.get_properties().await {
|
||||||
|
Ok(_) => Ok(StorageHealthStatus {
|
||||||
|
connected: true,
|
||||||
|
backend_type: "azure".to_string(),
|
||||||
|
message: format!("Azure container '{}' is accessible", self.container_name),
|
||||||
|
available_bytes: None,
|
||||||
|
}),
|
||||||
|
Err(e) => Ok(StorageHealthStatus {
|
||||||
|
connected: false,
|
||||||
|
backend_type: "azure".to_string(),
|
||||||
|
message: format!(
|
||||||
|
"Azure container '{}' is not accessible: {}",
|
||||||
|
self.container_name, e
|
||||||
|
),
|
||||||
|
available_bytes: None,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"azure"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,472 @@
|
|||||||
|
//! `CachedBlobBackend` — LRU local-disk cache decorator for remote blob backends.
|
||||||
|
//!
|
||||||
|
//! Wraps any `BlobStorageBackend` (typically S3 or Azure) and transparently
|
||||||
|
//! caches hot blobs on a local SSD. Reads check the cache first; cache misses
|
||||||
|
//! are fetched from the inner backend and written to the cache. Writes go to
|
||||||
|
//! the inner backend AND the local cache simultaneously.
|
||||||
|
//!
|
||||||
|
//! Eviction is LRU based on a configurable maximum disk budget.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
|
||||||
|
use lru::LruCache;
|
||||||
|
use std::num::NonZeroUsize;
|
||||||
|
use tokio::fs;
|
||||||
|
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
|
||||||
|
use tokio::sync::Mutex;
|
||||||
|
use tokio_util::io::ReaderStream;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Chunk size for streaming cached file reads (256 KB).
|
||||||
|
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||||
|
|
||||||
|
// ── Configuration ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Configuration for the LRU disk cache.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct BlobCacheConfig {
|
||||||
|
/// Directory where cached blobs are stored.
|
||||||
|
pub cache_dir: PathBuf,
|
||||||
|
/// Maximum total cache size in bytes.
|
||||||
|
pub max_cache_bytes: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Cache entry ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
struct CacheEntry {
|
||||||
|
size: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── CachedBlobBackend ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// A `BlobStorageBackend` decorator that adds an LRU disk cache in front of
|
||||||
|
/// a remote backend.
|
||||||
|
pub struct CachedBlobBackend {
|
||||||
|
inner: Arc<dyn BlobStorageBackend>,
|
||||||
|
cache_dir: PathBuf,
|
||||||
|
max_cache_bytes: u64,
|
||||||
|
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||||
|
current_size: Arc<AtomicU64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CachedBlobBackend {
|
||||||
|
/// Create a new cached backend wrapping `inner`.
|
||||||
|
pub fn new(inner: Arc<dyn BlobStorageBackend>, config: &BlobCacheConfig) -> Self {
|
||||||
|
Self {
|
||||||
|
inner,
|
||||||
|
cache_dir: config.cache_dir.clone(),
|
||||||
|
max_cache_bytes: config.max_cache_bytes,
|
||||||
|
// Capacity is essentially unbounded — eviction is by byte budget, not count.
|
||||||
|
index: Arc::new(Mutex::new(LruCache::new(
|
||||||
|
NonZeroUsize::new(1_000_000).unwrap(),
|
||||||
|
))),
|
||||||
|
current_size: Arc::new(AtomicU64::new(0)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Path where a blob is cached locally.
|
||||||
|
fn cached_path(&self, hash: &str) -> PathBuf {
|
||||||
|
let prefix = &hash[..2.min(hash.len())];
|
||||||
|
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for CachedBlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let cache_dir = self.cache_dir.clone();
|
||||||
|
let index = self.index.clone();
|
||||||
|
let current_size = self.current_size.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
inner.initialize().await?;
|
||||||
|
|
||||||
|
// Create cache dir structure (256 prefix dirs)
|
||||||
|
fs::create_dir_all(&cache_dir).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("mkdir cache_dir: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Scan existing cache to rebuild index
|
||||||
|
let mut total_bytes = 0u64;
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if let Ok(mut read_dir) = fs::read_dir(&cache_dir).await {
|
||||||
|
while let Ok(Some(prefix_entry)) = read_dir.next_entry().await {
|
||||||
|
if !prefix_entry.path().is_dir() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Ok(mut sub_dir) = fs::read_dir(prefix_entry.path()).await {
|
||||||
|
while let Ok(Some(entry)) = sub_dir.next_entry().await {
|
||||||
|
let path = entry.path();
|
||||||
|
if path.extension().and_then(|e| e.to_str()) == Some("blob")
|
||||||
|
&& let Some(stem) = path.file_stem().and_then(|s| s.to_str())
|
||||||
|
{
|
||||||
|
let size = fs::metadata(&path).await.map(|m| m.len()).unwrap_or(0);
|
||||||
|
idx.put(stem.to_string(), CacheEntry { size });
|
||||||
|
total_bytes += size;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
drop(idx);
|
||||||
|
current_size.store(total_bytes, Ordering::Relaxed);
|
||||||
|
tracing::info!(
|
||||||
|
"Blob cache initialized: {} bytes in cache at {}",
|
||||||
|
total_bytes,
|
||||||
|
cache_dir.display()
|
||||||
|
);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let source = source_path.to_path_buf();
|
||||||
|
let self_ref = CachedRef {
|
||||||
|
cache_dir: self.cache_dir.clone(),
|
||||||
|
max_cache_bytes: self.max_cache_bytes,
|
||||||
|
index: self.index.clone(),
|
||||||
|
current_size: self.current_size.clone(),
|
||||||
|
};
|
||||||
|
Box::pin(async move {
|
||||||
|
// Write to inner backend
|
||||||
|
let bytes = inner.put_blob(&hash, &source).await?;
|
||||||
|
// Also cache locally (best-effort)
|
||||||
|
let _ = self_ref.insert_into_cache_static(&hash, &source).await;
|
||||||
|
Ok(bytes)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let cached = self.cached_path(&hash);
|
||||||
|
let index = self.index.clone();
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let cache_dir = self.cache_dir.clone();
|
||||||
|
let max_cache_bytes = self.max_cache_bytes;
|
||||||
|
let current_size = self.current_size.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Check cache
|
||||||
|
{
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if idx.get(&hash).is_some() {
|
||||||
|
if let Ok(file) = fs::File::open(&cached).await {
|
||||||
|
let stream: BlobStream =
|
||||||
|
Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
|
||||||
|
return Ok(stream);
|
||||||
|
}
|
||||||
|
// Cache entry stale — remove
|
||||||
|
if let Some(entry) = idx.pop(&hash) {
|
||||||
|
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cache miss — fetch from inner, spool to cache
|
||||||
|
let self_ref = CachedRef {
|
||||||
|
cache_dir,
|
||||||
|
max_cache_bytes,
|
||||||
|
index: index.clone(),
|
||||||
|
current_size: current_size.clone(),
|
||||||
|
};
|
||||||
|
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||||
|
let file = fs::File::open(&dest).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
|
||||||
|
})?;
|
||||||
|
let stream: BlobStream = Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let cached = self.cached_path(&hash);
|
||||||
|
let index = self.index.clone();
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let cache_dir = self.cache_dir.clone();
|
||||||
|
let max_cache_bytes = self.max_cache_bytes;
|
||||||
|
let current_size = self.current_size.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Try cache first
|
||||||
|
{
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if idx.get(&hash).is_some() {
|
||||||
|
if let Ok(mut file) = fs::File::open(&cached).await {
|
||||||
|
file.seek(std::io::SeekFrom::Start(start))
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("seek: {e}"))
|
||||||
|
})?;
|
||||||
|
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
|
||||||
|
let limited = file.take(take_len);
|
||||||
|
let stream: BlobStream =
|
||||||
|
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
|
||||||
|
return Ok(stream);
|
||||||
|
}
|
||||||
|
if let Some(entry) = idx.pop(&hash) {
|
||||||
|
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cache miss — fetch full blob into cache, then serve range
|
||||||
|
let self_ref = CachedRef {
|
||||||
|
cache_dir,
|
||||||
|
max_cache_bytes,
|
||||||
|
index: index.clone(),
|
||||||
|
current_size: current_size.clone(),
|
||||||
|
};
|
||||||
|
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||||
|
let mut file = fs::File::open(&dest)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
|
||||||
|
file.seek(std::io::SeekFrom::Start(start))
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("seek: {e}")))?;
|
||||||
|
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
|
||||||
|
let limited = file.take(take_len);
|
||||||
|
let stream: BlobStream =
|
||||||
|
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let cached = self.cached_path(&hash);
|
||||||
|
let index = self.index.clone();
|
||||||
|
let current_size = self.current_size.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
inner.delete_blob(&hash).await?;
|
||||||
|
// Remove from cache
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if let Some(entry) = idx.pop(&hash) {
|
||||||
|
current_size.fetch_sub(entry.size, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
let _ = fs::remove_file(&cached).await;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let index = self.index.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Check cache first (fast)
|
||||||
|
{
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if idx.get(&hash).is_some() {
|
||||||
|
return Ok(true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inner.blob_exists(&hash).await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let index = self.index.clone();
|
||||||
|
let cached = self.cached_path(&hash);
|
||||||
|
Box::pin(async move {
|
||||||
|
// Check cache
|
||||||
|
{
|
||||||
|
let mut idx = index.lock().await;
|
||||||
|
if let Some(entry) = idx.get(&hash) {
|
||||||
|
return Ok(entry.size);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Fallback to cached file on disk (in case index was lost)
|
||||||
|
if let Ok(meta) = fs::metadata(&cached).await {
|
||||||
|
return Ok(meta.len());
|
||||||
|
}
|
||||||
|
inner.blob_size(&hash).await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let cache_dir = self.cache_dir.clone();
|
||||||
|
let current_size = self.current_size.clone();
|
||||||
|
let max_bytes = self.max_cache_bytes;
|
||||||
|
Box::pin(async move {
|
||||||
|
let mut status = inner.health_check().await?;
|
||||||
|
let used = current_size.load(Ordering::Relaxed);
|
||||||
|
status.message = format!(
|
||||||
|
"{} | Cache: {}/{} bytes used at {}",
|
||||||
|
status.message,
|
||||||
|
used,
|
||||||
|
max_bytes,
|
||||||
|
cache_dir.display()
|
||||||
|
);
|
||||||
|
status.backend_type = format!("cached({})", status.backend_type);
|
||||||
|
Ok(status)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"cached"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||||
|
// If the blob is cached locally, return that path
|
||||||
|
let path = self.cached_path(hash);
|
||||||
|
if path.exists() { Some(path) } else { None }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Helper struct for owned references in async closures ───────────
|
||||||
|
|
||||||
|
/// Cloneable set of cache internals — avoids borrow issues in boxed futures.
|
||||||
|
struct CachedRef {
|
||||||
|
cache_dir: PathBuf,
|
||||||
|
max_cache_bytes: u64,
|
||||||
|
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||||
|
current_size: Arc<AtomicU64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CachedRef {
|
||||||
|
fn cached_path(&self, hash: &str) -> PathBuf {
|
||||||
|
let prefix = &hash[..2.min(hash.len())];
|
||||||
|
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn insert_into_cache_static(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Result<(), DomainError> {
|
||||||
|
let dest = self.cached_path(hash);
|
||||||
|
if let Some(parent) = dest.parent() {
|
||||||
|
fs::create_dir_all(parent).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
|
||||||
|
let size = fs::metadata(source_path)
|
||||||
|
.await
|
||||||
|
.map(|m| m.len())
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
|
fs::copy(source_path, &dest).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("cache copy failed: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let mut idx = self.index.lock().await;
|
||||||
|
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size }) {
|
||||||
|
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
self.current_size.fetch_add(size, Ordering::Relaxed);
|
||||||
|
|
||||||
|
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
|
||||||
|
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
|
||||||
|
self.current_size
|
||||||
|
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
|
||||||
|
let evicted_path = self.cached_path(&evicted_hash);
|
||||||
|
let _ = fs::remove_file(&evicted_path).await;
|
||||||
|
} else {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn fetch_and_cache_static(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
inner: &dyn BlobStorageBackend,
|
||||||
|
) -> Result<PathBuf, DomainError> {
|
||||||
|
let stream = inner.get_blob_stream(hash).await?;
|
||||||
|
|
||||||
|
let dest = self.cached_path(hash);
|
||||||
|
if let Some(parent) = dest.parent() {
|
||||||
|
fs::create_dir_all(parent).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
|
||||||
|
let tmp = dest.with_extension("tmp");
|
||||||
|
let mut file = fs::File::create(&tmp)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
|
||||||
|
|
||||||
|
use futures::StreamExt;
|
||||||
|
let mut stream = stream;
|
||||||
|
let mut total = 0u64;
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
let bytes = chunk.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||||
|
})?;
|
||||||
|
total += bytes.len() as u64;
|
||||||
|
file.write_all(&bytes)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||||
|
}
|
||||||
|
file.flush()
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||||
|
drop(file);
|
||||||
|
|
||||||
|
fs::rename(&tmp, &dest)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
|
||||||
|
|
||||||
|
let mut idx = self.index.lock().await;
|
||||||
|
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size: total }) {
|
||||||
|
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
self.current_size.fetch_add(total, Ordering::Relaxed);
|
||||||
|
|
||||||
|
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
|
||||||
|
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
|
||||||
|
self.current_size
|
||||||
|
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
|
||||||
|
let evicted_path = self.cached_path(&evicted_hash);
|
||||||
|
let _ = fs::remove_file(&evicted_path).await;
|
||||||
|
} else {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(dest)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -39,24 +39,22 @@ use sqlx::PgPool;
|
|||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::pin::Pin;
|
use std::pin::Pin;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use tokio::fs::{self, File};
|
use tokio::fs;
|
||||||
use tokio::io::{AsyncReadExt, AsyncSeekExt};
|
|
||||||
use tokio_util::io::ReaderStream;
|
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
use crate::application::ports::dedup_ports::{
|
use crate::application::ports::dedup_ports::{
|
||||||
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
||||||
};
|
};
|
||||||
use crate::domain::errors::{DomainError, ErrorKind};
|
use crate::domain::errors::{DomainError, ErrorKind};
|
||||||
|
|
||||||
/// Chunk size for streaming file reads (256 KB)
|
|
||||||
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
|
||||||
|
|
||||||
/// Content-Addressable Storage Service (PostgreSQL-backed)
|
/// Content-Addressable Storage Service (PostgreSQL-backed)
|
||||||
|
///
|
||||||
|
/// Delegates all byte-level I/O to a [`BlobStorageBackend`] implementation
|
||||||
|
/// (local filesystem, S3, etc.) while keeping BLAKE3 hashing, ref-counting
|
||||||
|
/// and the PostgreSQL dedup index here.
|
||||||
pub struct DedupService {
|
pub struct DedupService {
|
||||||
/// Root directory for blob storage on the filesystem
|
/// Pluggable blob storage backend (local FS, S3, …).
|
||||||
blob_root: PathBuf,
|
backend: Arc<dyn BlobStorageBackend>,
|
||||||
/// Root directory for temporary files during upload
|
|
||||||
temp_root: PathBuf,
|
|
||||||
/// PostgreSQL connection pool (dedup index in `storage.blobs`) — primary,
|
/// PostgreSQL connection pool (dedup index in `storage.blobs`) — primary,
|
||||||
/// used by request-path operations (store_from_file, etc.).
|
/// used by request-path operations (store_from_file, etc.).
|
||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
@@ -65,39 +63,19 @@ pub struct DedupService {
|
|||||||
maintenance_pool: Arc<PgPool>,
|
maintenance_pool: Arc<PgPool>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
|
|
||||||
/// Avoids a `format!("{:02x}", i)` allocation on every iteration of `initialize()`.
|
|
||||||
static HEX_PREFIXES: [&str; 256] = [
|
|
||||||
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
|
|
||||||
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
|
|
||||||
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
|
|
||||||
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
|
|
||||||
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
|
|
||||||
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
|
|
||||||
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
|
|
||||||
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
|
|
||||||
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
|
|
||||||
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
|
|
||||||
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
|
|
||||||
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
|
|
||||||
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
|
|
||||||
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
|
|
||||||
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
|
|
||||||
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
|
|
||||||
];
|
|
||||||
|
|
||||||
impl DedupService {
|
impl DedupService {
|
||||||
/// Create a new dedup service backed by PostgreSQL.
|
/// Create a new dedup service backed by PostgreSQL.
|
||||||
///
|
///
|
||||||
|
/// * `backend` — pluggable blob storage (local filesystem, S3, etc.).
|
||||||
/// * `pool` — primary pool for request-path operations.
|
/// * `pool` — primary pool for request-path operations.
|
||||||
/// * `maintenance_pool` — isolated pool for verify_integrity / garbage_collect.
|
/// * `maintenance_pool` — isolated pool for verify_integrity / garbage_collect.
|
||||||
pub fn new(storage_root: &Path, pool: Arc<PgPool>, maintenance_pool: Arc<PgPool>) -> Self {
|
pub fn new(
|
||||||
let blob_root = storage_root.join(".blobs");
|
backend: Arc<dyn BlobStorageBackend>,
|
||||||
let temp_root = storage_root.join(".dedup_temp");
|
pool: Arc<PgPool>,
|
||||||
|
maintenance_pool: Arc<PgPool>,
|
||||||
|
) -> Self {
|
||||||
Self {
|
Self {
|
||||||
blob_root,
|
backend,
|
||||||
temp_root,
|
|
||||||
pool,
|
pool,
|
||||||
maintenance_pool,
|
maintenance_pool,
|
||||||
}
|
}
|
||||||
@@ -106,6 +84,7 @@ impl DedupService {
|
|||||||
/// Creates a stub instance for testing — never hits PG or the filesystem.
|
/// Creates a stub instance for testing — never hits PG or the filesystem.
|
||||||
#[cfg(any(test, feature = "integration_tests"))]
|
#[cfg(any(test, feature = "integration_tests"))]
|
||||||
pub fn new_stub() -> Self {
|
pub fn new_stub() -> Self {
|
||||||
|
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||||
let stub_pool = Arc::new(
|
let stub_pool = Arc::new(
|
||||||
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
|
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
|
||||||
.max_connections(1)
|
.max_connections(1)
|
||||||
@@ -113,29 +92,15 @@ impl DedupService {
|
|||||||
.unwrap(),
|
.unwrap(),
|
||||||
);
|
);
|
||||||
Self {
|
Self {
|
||||||
blob_root: std::path::PathBuf::from("/tmp/oxicloud_stub_blobs"),
|
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
|
||||||
temp_root: std::path::PathBuf::from("/tmp/oxicloud_stub_temp"),
|
|
||||||
pool: stub_pool.clone(),
|
pool: stub_pool.clone(),
|
||||||
maintenance_pool: stub_pool,
|
maintenance_pool: stub_pool,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Initialize the service (create blob directories on the filesystem).
|
/// Initialize the service (delegate to backend + log stats from PG).
|
||||||
pub async fn initialize(&self) -> Result<(), DomainError> {
|
pub async fn initialize(&self) -> Result<(), DomainError> {
|
||||||
// Create directories
|
self.backend.initialize().await?;
|
||||||
fs::create_dir_all(&self.blob_root)
|
|
||||||
.await
|
|
||||||
.map_err(DomainError::from)?;
|
|
||||||
fs::create_dir_all(&self.temp_root)
|
|
||||||
.await
|
|
||||||
.map_err(DomainError::from)?;
|
|
||||||
|
|
||||||
// Create hash prefix directories (00-ff)
|
|
||||||
for prefix in &HEX_PREFIXES {
|
|
||||||
fs::create_dir_all(self.blob_root.join(prefix))
|
|
||||||
.await
|
|
||||||
.map_err(DomainError::from)?;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Log existing blob stats from PG
|
// Log existing blob stats from PG
|
||||||
let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||||
@@ -150,7 +115,8 @@ impl DedupService {
|
|||||||
.unwrap_or(0);
|
.unwrap_or(0);
|
||||||
|
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
"Dedup service initialized (PostgreSQL-backed): {} blobs, {} bytes stored",
|
"Dedup service initialized (backend={}): {} blobs, {} bytes stored",
|
||||||
|
self.backend.backend_type(),
|
||||||
count,
|
count,
|
||||||
total_bytes
|
total_bytes
|
||||||
);
|
);
|
||||||
@@ -158,12 +124,18 @@ impl DedupService {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Return a reference to the underlying blob storage backend.
|
||||||
|
pub fn backend(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||||
|
&self.backend
|
||||||
|
}
|
||||||
|
|
||||||
// ── Path helpers ─────────────────────────────────────────────
|
// ── Path helpers ─────────────────────────────────────────────
|
||||||
|
|
||||||
/// Get the blob path for a given hash.
|
/// Get the local blob path for a given hash (if the backend supports it).
|
||||||
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||||
let prefix = &hash[0..2];
|
self.backend
|
||||||
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
.local_blob_path(hash)
|
||||||
|
.unwrap_or_else(|| PathBuf::from(format!("remote://{}", hash)))
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Hash helpers ─────────────────────────────────────────────
|
// ── Hash helpers ─────────────────────────────────────────────
|
||||||
@@ -193,9 +165,9 @@ impl DedupService {
|
|||||||
|
|
||||||
/// Store content with deduplication (streaming from file).
|
/// Store content with deduplication (streaming from file).
|
||||||
///
|
///
|
||||||
/// **Write-first strategy**: the source file is moved/copied to the
|
/// **Write-first strategy**: the source file is moved/uploaded to the
|
||||||
/// blob store *before* touching PostgreSQL, so the PG connection is
|
/// blob backend *before* touching PostgreSQL, so the PG connection is
|
||||||
/// never held during disk I/O.
|
/// never held during I/O.
|
||||||
///
|
///
|
||||||
/// If `pre_computed_hash` is `Some`, the file will NOT be re-read for
|
/// If `pre_computed_hash` is `Some`, the file will NOT be re-read for
|
||||||
/// BLAKE3 — saving one full sequential read (the biggest I/O win).
|
/// BLAKE3 — saving one full sequential read (the biggest I/O win).
|
||||||
@@ -205,13 +177,6 @@ impl DedupService {
|
|||||||
content_type: Option<String>,
|
content_type: Option<String>,
|
||||||
pre_computed_hash: Option<String>,
|
pre_computed_hash: Option<String>,
|
||||||
) -> Result<DedupResultDto, DomainError> {
|
) -> Result<DedupResultDto, DomainError> {
|
||||||
let file_size = fs::metadata(source_path)
|
|
||||||
.await
|
|
||||||
.map_err(|e| {
|
|
||||||
DomainError::internal_error("Dedup", format!("Failed to get file metadata: {}", e))
|
|
||||||
})?
|
|
||||||
.len();
|
|
||||||
|
|
||||||
// Use pre-computed hash if available, otherwise calculate (streaming)
|
// Use pre-computed hash if available, otherwise calculate (streaming)
|
||||||
let hash = match pre_computed_hash {
|
let hash = match pre_computed_hash {
|
||||||
Some(h) => h,
|
Some(h) => h,
|
||||||
@@ -220,44 +185,11 @@ impl DedupService {
|
|||||||
.map_err(DomainError::from)?,
|
.map_err(DomainError::from)?,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// ── Phase 1: Place blob in backend (NO PG connection held) ───
|
||||||
|
let file_size = self.backend.put_blob(&hash, source_path).await?;
|
||||||
|
|
||||||
let blob_path = self.blob_path(&hash);
|
let blob_path = self.blob_path(&hash);
|
||||||
|
|
||||||
// ── Phase 1: Move/place blob on disk (NO PG connection held) ─
|
|
||||||
//
|
|
||||||
// If the blob file already exists on disk, the source is simply
|
|
||||||
// deleted — the file content is identical by definition.
|
|
||||||
if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
|
||||||
// Blob already on disk — discard the source file
|
|
||||||
let _ = fs::remove_file(source_path).await;
|
|
||||||
} else {
|
|
||||||
// Parent directory (xx/) guaranteed to exist — created by initialize()
|
|
||||||
|
|
||||||
// rename is atomic on the same filesystem. If source and blob
|
|
||||||
// dirs live on different filesystems (rare), this falls back to
|
|
||||||
// copy+delete which is slower but still correct.
|
|
||||||
if let Err(e) = fs::rename(source_path, &blob_path).await {
|
|
||||||
if e.raw_os_error() == Some(18) {
|
|
||||||
// EXDEV: cross-device link — fall back to copy+delete
|
|
||||||
fs::copy(source_path, &blob_path).await.map_err(|ce| {
|
|
||||||
DomainError::internal_error(
|
|
||||||
"Dedup",
|
|
||||||
format!("Failed to copy file to blob store: {}", ce),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
let _ = fs::remove_file(source_path).await;
|
|
||||||
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
|
||||||
// Another writer may have placed the blob concurrently
|
|
||||||
let _ = fs::remove_file(source_path).await;
|
|
||||||
tracing::debug!("Blob file placed by concurrent writer: {}", e);
|
|
||||||
} else {
|
|
||||||
return Err(DomainError::internal_error(
|
|
||||||
"Dedup",
|
|
||||||
format!("Failed to move file to blob store: {}", e),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── Phase 2: Single atomic upsert (~2-4 ms, no explicit TX) ─
|
// ── Phase 2: Single atomic upsert (~2-4 ms, no explicit TX) ─
|
||||||
let ref_count: i32 = sqlx::query_scalar(
|
let ref_count: i32 = sqlx::query_scalar(
|
||||||
"INSERT INTO storage.blobs (hash, size, ref_count, content_type)
|
"INSERT INTO storage.blobs (hash, size, ref_count, content_type)
|
||||||
@@ -415,10 +347,9 @@ impl DedupService {
|
|||||||
DomainError::internal_error("Dedup", format!("Failed to commit: {}", e))
|
DomainError::internal_error("Dedup", format!("Failed to commit: {}", e))
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
// Delete blob file AFTER committing PG — the row is gone, so no
|
// Delete blob from backend AFTER committing PG — the row is gone,
|
||||||
// concurrent store_from_file can resurrect a reference to this hash.
|
// so no concurrent store_from_file can resurrect a reference.
|
||||||
let blob_path = self.blob_path(hash);
|
if let Err(e) = self.backend.delete_blob(hash).await {
|
||||||
if let Err(e) = fs::remove_file(&blob_path).await {
|
|
||||||
tracing::warn!("Failed to delete blob file {}: {}", hash, e);
|
tracing::warn!("Failed to delete blob file {}: {}", hash, e);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -449,31 +380,16 @@ impl DedupService {
|
|||||||
|
|
||||||
// ── Read operations ──────────────────────────────────────────
|
// ── Read operations ──────────────────────────────────────────
|
||||||
|
|
||||||
/// Stream blob content in 64 KB chunks — constant memory (~64 KB per stream).
|
/// Stream blob content in chunks — constant memory usage.
|
||||||
///
|
|
||||||
/// A 1 GB file uses the same ~64 KB as a 1 KB file.
|
|
||||||
pub async fn read_blob_stream(
|
pub async fn read_blob_stream(
|
||||||
&self,
|
&self,
|
||||||
hash: &str,
|
hash: &str,
|
||||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||||
{
|
{
|
||||||
let blob_path = self.blob_path(hash);
|
self.backend.get_blob_stream(hash).await
|
||||||
let file = File::open(&blob_path).await.map_err(|e| {
|
|
||||||
DomainError::new(
|
|
||||||
ErrorKind::NotFound,
|
|
||||||
"Blob",
|
|
||||||
format!("Failed to open blob {}: {}", hash, e),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
Ok(Box::pin(ReaderStream::with_capacity(
|
|
||||||
file,
|
|
||||||
STREAM_CHUNK_SIZE,
|
|
||||||
)))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Stream a byte range of a blob — only reads the requested portion.
|
/// Stream a byte range of a blob — only reads the requested portion.
|
||||||
///
|
|
||||||
/// Uses seek + take so a 1 MB range request on a 1 GB file only reads 1 MB.
|
|
||||||
pub async fn read_blob_range_stream(
|
pub async fn read_blob_range_stream(
|
||||||
&self,
|
&self,
|
||||||
hash: &str,
|
hash: &str,
|
||||||
@@ -481,49 +397,12 @@ impl DedupService {
|
|||||||
end: Option<u64>,
|
end: Option<u64>,
|
||||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||||
{
|
{
|
||||||
let blob_path = self.blob_path(hash);
|
self.backend.get_blob_range_stream(hash, start, end).await
|
||||||
let mut file = File::open(&blob_path).await.map_err(|e| {
|
|
||||||
DomainError::new(
|
|
||||||
ErrorKind::NotFound,
|
|
||||||
"Blob",
|
|
||||||
format!("Failed to open blob {}: {}", hash, e),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
// Seek to the start position
|
|
||||||
file.seek(std::io::SeekFrom::Start(start))
|
|
||||||
.await
|
|
||||||
.map_err(|e| {
|
|
||||||
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
|
|
||||||
})?;
|
|
||||||
|
|
||||||
// If an end is specified, limit the read with take()
|
|
||||||
if let Some(end_pos) = end {
|
|
||||||
let limit = end_pos.saturating_sub(start);
|
|
||||||
let limited = file.take(limit);
|
|
||||||
Ok(Box::pin(ReaderStream::with_capacity(
|
|
||||||
limited,
|
|
||||||
STREAM_CHUNK_SIZE,
|
|
||||||
)))
|
|
||||||
} else {
|
|
||||||
Ok(Box::pin(ReaderStream::with_capacity(
|
|
||||||
file,
|
|
||||||
STREAM_CHUNK_SIZE,
|
|
||||||
)))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get the size of a blob without reading its content.
|
/// Get the size of a blob without reading its content.
|
||||||
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
|
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
|
||||||
let blob_path = self.blob_path(hash);
|
self.backend.blob_size(hash).await
|
||||||
let meta = fs::metadata(&blob_path).await.map_err(|e| {
|
|
||||||
DomainError::new(
|
|
||||||
ErrorKind::NotFound,
|
|
||||||
"Blob",
|
|
||||||
format!("Failed to stat blob {}: {}", hash, e),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
Ok(meta.len())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Statistics (computed from PG) ────────────────────────────
|
// ── Statistics (computed from PG) ────────────────────────────
|
||||||
@@ -596,56 +475,46 @@ impl DedupService {
|
|||||||
|
|
||||||
// Flush when batch is full or we've exhausted the cursor
|
// Flush when batch is full or we've exhausted the cursor
|
||||||
if batch.len() >= VERIFY_CONCURRENCY || (is_done && !batch.is_empty()) {
|
if batch.len() >= VERIFY_CONCURRENCY || (is_done && !batch.is_empty()) {
|
||||||
let blob_root = self.blob_root.clone();
|
let backend = self.backend.clone();
|
||||||
let current_batch =
|
let current_batch =
|
||||||
std::mem::replace(&mut batch, Vec::with_capacity(VERIFY_CONCURRENCY));
|
std::mem::replace(&mut batch, Vec::with_capacity(VERIFY_CONCURRENCY));
|
||||||
|
|
||||||
let issues: Vec<String> = stream::iter(current_batch)
|
let issues: Vec<String> = stream::iter(current_batch)
|
||||||
.map(move |(hash, expected_size)| {
|
.map(move |(hash, expected_size)| {
|
||||||
let blob_root = blob_root.clone();
|
let backend = backend.clone();
|
||||||
async move {
|
async move {
|
||||||
let prefix = &hash[0..2];
|
|
||||||
let blob_path = blob_root.join(prefix).join(format!("{}.blob", hash));
|
|
||||||
|
|
||||||
let mut issues = Vec::new();
|
let mut issues = Vec::new();
|
||||||
|
|
||||||
// Single async metadata() replaces the previous
|
// Check existence + size via backend
|
||||||
// blocking .exists() + separate metadata() — one
|
match backend.blob_size(&hash).await {
|
||||||
// stat() syscall instead of two, and non-blocking.
|
Ok(actual_size) => {
|
||||||
let file_meta = match fs::metadata(&blob_path).await {
|
if actual_size != expected_size as u64 {
|
||||||
Ok(m) => m,
|
issues.push(format!(
|
||||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
"{}: size mismatch (expected: {}, actual: {})",
|
||||||
issues.push(format!("{}: file missing on disk", hash));
|
hash, expected_size, actual_size,
|
||||||
return issues;
|
));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(_) => {
|
||||||
issues.push(format!("{}: metadata error ({})", hash, e));
|
issues.push(format!("{}: blob missing in backend", hash));
|
||||||
return issues;
|
return issues;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
// Check size
|
// Verify hash — only possible for local backends
|
||||||
if file_meta.len() != expected_size as u64 {
|
if let Some(blob_path) = backend.local_blob_path(&hash) {
|
||||||
issues.push(format!(
|
match Self::hash_file(&blob_path).await {
|
||||||
"{}: size mismatch (expected: {}, actual: {})",
|
Ok(actual_hash) => {
|
||||||
hash,
|
if actual_hash != hash {
|
||||||
expected_size,
|
issues.push(format!(
|
||||||
file_meta.len(),
|
"{}: hash mismatch (actual: {})",
|
||||||
));
|
hash, actual_hash,
|
||||||
}
|
));
|
||||||
|
}
|
||||||
// Verify hash
|
}
|
||||||
match Self::hash_file(&blob_path).await {
|
Err(e) => {
|
||||||
Ok(actual_hash) => {
|
issues.push(format!("{}: read error ({})", hash, e));
|
||||||
if actual_hash != hash {
|
|
||||||
issues.push(format!(
|
|
||||||
"{}: hash mismatch (actual: {})",
|
|
||||||
hash, actual_hash,
|
|
||||||
));
|
|
||||||
}
|
}
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
issues.push(format!("{}: read error ({})", hash, e));
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -717,20 +586,20 @@ impl DedupService {
|
|||||||
// Also clean up any thumbnail files for these blob hashes
|
// Also clean up any thumbnail files for these blob hashes
|
||||||
// (thumbnails are keyed by blob_hash and live under
|
// (thumbnails are keyed by blob_hash and live under
|
||||||
// storage_root/.thumbnails/{icon,preview,large}/{hash}.jpg).
|
// storage_root/.thumbnails/{icon,preview,large}/{hash}.jpg).
|
||||||
let thumbnails_root = self
|
|
||||||
.blob_root
|
|
||||||
.parent()
|
|
||||||
.unwrap_or(&self.blob_root)
|
|
||||||
.join(".thumbnails");
|
|
||||||
for (hash, size) in &batch {
|
for (hash, size) in &batch {
|
||||||
let blob_path = self.blob_path(hash);
|
if let Err(e) = self.backend.delete_blob(hash).await {
|
||||||
if let Err(e) = fs::remove_file(&blob_path).await {
|
|
||||||
tracing::warn!("Failed to delete orphan blob file {hash}: {e}");
|
tracing::warn!("Failed to delete orphan blob file {hash}: {e}");
|
||||||
}
|
}
|
||||||
// Remove associated thumbnail files (best-effort)
|
// Remove associated thumbnail files (best-effort, always local)
|
||||||
for dir in &["icon", "preview", "large"] {
|
if let Some(blob_path) = self.backend.local_blob_path(hash)
|
||||||
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
|
&& let Some(storage_root) = blob_path.ancestors().nth(3)
|
||||||
let _ = fs::remove_file(&thumb).await;
|
{
|
||||||
|
let thumbnails_root = storage_root.join(".thumbnails");
|
||||||
|
for dir in &["icon", "preview", "large"] {
|
||||||
|
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
|
||||||
|
let _ = fs::remove_file(&thumb).await;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
total_bytes += *size as u64;
|
total_bytes += *size as u64;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,298 @@
|
|||||||
|
//! `EncryptedBlobBackend` — AES-256-GCM encryption decorator for blob storage.
|
||||||
|
//!
|
||||||
|
//! Transparently encrypts blobs before they reach the inner backend and
|
||||||
|
//! decrypts them on read. Each blob gets a random 96-bit nonce which is
|
||||||
|
//! prepended to the ciphertext. The GCM authentication tag (16 bytes) is
|
||||||
|
//! appended by the cipher.
|
||||||
|
//!
|
||||||
|
//! **IMPORTANT**: BLAKE3 hashing is performed on the *plaintext* by
|
||||||
|
//! `DedupService` before this layer sees the blob, so content-addressable
|
||||||
|
//! dedup still works correctly.
|
||||||
|
//!
|
||||||
|
//! Layout on disk/S3: `[12-byte nonce][ciphertext + 16-byte GCM tag]`
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
|
||||||
|
use aes_gcm::aead::{Aead, KeyInit, OsRng};
|
||||||
|
use aes_gcm::{AeadCore, Aes256Gcm, Nonce};
|
||||||
|
use bytes::Bytes;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::fs;
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Nonce size for AES-256-GCM (96 bits = 12 bytes).
|
||||||
|
const NONCE_SIZE: usize = 12;
|
||||||
|
|
||||||
|
/// `BlobStorageBackend` decorator that encrypts blobs at rest.
|
||||||
|
pub struct EncryptedBlobBackend {
|
||||||
|
inner: Arc<dyn BlobStorageBackend>,
|
||||||
|
cipher: Aes256Gcm,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EncryptedBlobBackend {
|
||||||
|
/// Create a new encryption layer wrapping `inner`.
|
||||||
|
///
|
||||||
|
/// `key` must be exactly 32 bytes (AES-256).
|
||||||
|
pub fn new(inner: Arc<dyn BlobStorageBackend>, key: &[u8; 32]) -> Self {
|
||||||
|
let cipher = Aes256Gcm::new_from_slice(key).expect("AES-256 key must be 32 bytes");
|
||||||
|
Self { inner, cipher }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate a random 32-byte key suitable for AES-256.
|
||||||
|
pub fn generate_key() -> [u8; 32] {
|
||||||
|
use aes_gcm::aead::rand_core::RngCore;
|
||||||
|
let mut key = [0u8; 32];
|
||||||
|
OsRng.fill_bytes(&mut key);
|
||||||
|
key
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for EncryptedBlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
self.inner.initialize()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let source = source_path.to_path_buf();
|
||||||
|
// Clone cipher key material (Aes256Gcm is not Send-safe to move across await)
|
||||||
|
let cipher = self.cipher.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Read plaintext from source
|
||||||
|
let plaintext = fs::read(&source).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("read source: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Encrypt: nonce || ciphertext (includes GCM tag)
|
||||||
|
let nonce = Aes256Gcm::generate_nonce(&mut OsRng);
|
||||||
|
let ciphertext = cipher.encrypt(&nonce, plaintext.as_ref()).map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("encrypt failed: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Write encrypted blob to a temp file
|
||||||
|
let tmp = source.with_extension("enc.tmp");
|
||||||
|
let mut file = fs::File::create(&tmp).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("create tmp: {e}"))
|
||||||
|
})?;
|
||||||
|
file.write_all(nonce.as_slice()).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("write nonce: {e}"))
|
||||||
|
})?;
|
||||||
|
file.write_all(&ciphertext).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("write ciphertext: {e}"))
|
||||||
|
})?;
|
||||||
|
file.flush()
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Encryption", format!("flush: {e}")))?;
|
||||||
|
drop(file);
|
||||||
|
|
||||||
|
let result = inner.put_blob(&hash, &tmp).await;
|
||||||
|
let _ = fs::remove_file(&tmp).await;
|
||||||
|
result
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let cipher = self.cipher.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Read entire encrypted blob (nonce + ciphertext) into memory for decryption
|
||||||
|
let enc_stream = inner.get_blob_stream(&hash).await?;
|
||||||
|
let encrypted = collect_stream(enc_stream).await?;
|
||||||
|
|
||||||
|
if encrypted.len() < NONCE_SIZE {
|
||||||
|
return Err(DomainError::internal_error(
|
||||||
|
"Encryption",
|
||||||
|
"encrypted blob too short (missing nonce)",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
|
||||||
|
let nonce = Nonce::from_slice(nonce_bytes);
|
||||||
|
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let stream: BlobStream =
|
||||||
|
Box::pin(futures::stream::once(
|
||||||
|
async move { Ok(Bytes::from(plaintext)) },
|
||||||
|
));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let cipher = self.cipher.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Must decrypt the full blob then slice the plaintext range
|
||||||
|
let enc_stream = inner.get_blob_stream(&hash).await?;
|
||||||
|
let encrypted = collect_stream(enc_stream).await?;
|
||||||
|
|
||||||
|
if encrypted.len() < NONCE_SIZE {
|
||||||
|
return Err(DomainError::internal_error(
|
||||||
|
"Encryption",
|
||||||
|
"encrypted blob too short",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
|
||||||
|
let nonce = Nonce::from_slice(nonce_bytes);
|
||||||
|
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
|
||||||
|
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let start = start as usize;
|
||||||
|
let end = end.map(|e| (e as usize) + 1).unwrap_or(plaintext.len());
|
||||||
|
let end = end.min(plaintext.len());
|
||||||
|
let start = start.min(end);
|
||||||
|
|
||||||
|
let slice = Bytes::from(plaintext[start..end].to_vec());
|
||||||
|
let stream: BlobStream = Box::pin(futures::stream::once(async move { Ok(slice) }));
|
||||||
|
Ok(stream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
self.inner.delete_blob(hash)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
self.inner.blob_exists(hash)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
// The stored size includes nonce + GCM tag overhead.
|
||||||
|
// Return the *plaintext* size by subtracting overhead.
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
let encrypted_size = inner.blob_size(&hash).await?;
|
||||||
|
// overhead = 12 (nonce) + 16 (GCM tag) = 28 bytes
|
||||||
|
Ok(encrypted_size.saturating_sub(28))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
let mut status = inner.health_check().await?;
|
||||||
|
status.backend_type = format!("encrypted({})", status.backend_type);
|
||||||
|
status.message = format!("{} | Encryption: AES-256-GCM", status.message);
|
||||||
|
Ok(status)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"encrypted"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||||
|
// Encrypted blobs cannot be served directly from disk
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Collect a byte stream into a single `Vec<u8>`.
|
||||||
|
async fn collect_stream(stream: BlobStream) -> Result<Vec<u8>, DomainError> {
|
||||||
|
use futures::StreamExt;
|
||||||
|
let mut stream = stream;
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
let bytes = chunk
|
||||||
|
.map_err(|e| DomainError::internal_error("Encryption", format!("stream read: {e}")))?;
|
||||||
|
buf.extend_from_slice(&bytes);
|
||||||
|
}
|
||||||
|
Ok(buf)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||||
|
use tempfile::TempDir;
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_encrypt_decrypt_roundtrip() {
|
||||||
|
let tmp = TempDir::new().unwrap();
|
||||||
|
let blob_dir = tmp.path().join("blobs");
|
||||||
|
let local = Arc::new(LocalBlobBackend::new(&blob_dir));
|
||||||
|
local.initialize().await.unwrap();
|
||||||
|
|
||||||
|
let key = EncryptedBlobBackend::generate_key();
|
||||||
|
let encrypted = EncryptedBlobBackend::new(local, &key);
|
||||||
|
|
||||||
|
// Write a test blob
|
||||||
|
let data = b"Hello, encrypted world!";
|
||||||
|
let source = tmp.path().join("test.tmp");
|
||||||
|
let mut f = fs::File::create(&source).await.unwrap();
|
||||||
|
f.write_all(data).await.unwrap();
|
||||||
|
f.flush().await.unwrap();
|
||||||
|
drop(f);
|
||||||
|
|
||||||
|
let hash = "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890";
|
||||||
|
encrypted.put_blob(hash, &source).await.unwrap();
|
||||||
|
|
||||||
|
// Read back via stream
|
||||||
|
let stream = encrypted.get_blob_stream(hash).await.unwrap();
|
||||||
|
let decrypted = collect_stream(stream).await.unwrap();
|
||||||
|
assert_eq!(decrypted, data);
|
||||||
|
|
||||||
|
// Read range
|
||||||
|
let range_stream = encrypted
|
||||||
|
.get_blob_range_stream(hash, 7, Some(15))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let range_data = collect_stream(range_stream).await.unwrap();
|
||||||
|
assert_eq!(range_data, b"encrypted");
|
||||||
|
|
||||||
|
// Size should reflect plaintext
|
||||||
|
let size = encrypted.blob_size(hash).await.unwrap();
|
||||||
|
assert_eq!(size, data.len() as u64);
|
||||||
|
|
||||||
|
// Exists
|
||||||
|
assert!(encrypted.blob_exists(hash).await.unwrap());
|
||||||
|
|
||||||
|
// Delete
|
||||||
|
encrypted.delete_blob(hash).await.unwrap();
|
||||||
|
assert!(!encrypted.blob_exists(hash).await.unwrap());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,277 @@
|
|||||||
|
//! Local Filesystem Blob Backend — stores blobs under `.blobs/{prefix}/{hash}.blob`.
|
||||||
|
//!
|
||||||
|
//! This is the default backend and a direct extraction of the filesystem I/O
|
||||||
|
//! that previously lived inside `DedupService`.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use tokio::fs::{self, File};
|
||||||
|
use tokio::io::AsyncSeekExt;
|
||||||
|
use tokio_util::io::ReaderStream;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::domain::errors::{DomainError, ErrorKind};
|
||||||
|
|
||||||
|
/// Chunk size for streaming file reads (256 KB).
|
||||||
|
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
|
||||||
|
|
||||||
|
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
|
||||||
|
static HEX_PREFIXES: [&str; 256] = [
|
||||||
|
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
|
||||||
|
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
|
||||||
|
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
|
||||||
|
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
|
||||||
|
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
|
||||||
|
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
|
||||||
|
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
|
||||||
|
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
|
||||||
|
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
|
||||||
|
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
|
||||||
|
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
|
||||||
|
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
|
||||||
|
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
|
||||||
|
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
|
||||||
|
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
|
||||||
|
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
|
||||||
|
];
|
||||||
|
|
||||||
|
/// Local filesystem blob backend.
|
||||||
|
///
|
||||||
|
/// Blobs are stored under `blob_root/{2-char-prefix}/{hash}.blob`.
|
||||||
|
/// Temporary upload staging uses `temp_root/`.
|
||||||
|
pub struct LocalBlobBackend {
|
||||||
|
blob_root: PathBuf,
|
||||||
|
temp_root: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LocalBlobBackend {
|
||||||
|
/// Create a new local backend rooted at `storage_root`.
|
||||||
|
///
|
||||||
|
/// Blob files go under `{storage_root}/.blobs/`, temp files under
|
||||||
|
/// `{storage_root}/.dedup_temp/`.
|
||||||
|
pub fn new(storage_root: &Path) -> Self {
|
||||||
|
Self {
|
||||||
|
blob_root: storage_root.join(".blobs"),
|
||||||
|
temp_root: storage_root.join(".dedup_temp"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute the filesystem path for a blob hash.
|
||||||
|
pub fn blob_path(&self, hash: &str) -> PathBuf {
|
||||||
|
let prefix = &hash[0..2];
|
||||||
|
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return a reference to the blob root directory.
|
||||||
|
pub fn blob_root(&self) -> &Path {
|
||||||
|
&self.blob_root
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for LocalBlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
fs::create_dir_all(&self.blob_root)
|
||||||
|
.await
|
||||||
|
.map_err(DomainError::from)?;
|
||||||
|
fs::create_dir_all(&self.temp_root)
|
||||||
|
.await
|
||||||
|
.map_err(DomainError::from)?;
|
||||||
|
|
||||||
|
// Create the 256 hash-prefix directories (00-ff)
|
||||||
|
for prefix in &HEX_PREFIXES {
|
||||||
|
fs::create_dir_all(self.blob_root.join(prefix))
|
||||||
|
.await
|
||||||
|
.map_err(DomainError::from)?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
let source_path = source_path.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
|
||||||
|
let file_size = fs::metadata(&source_path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to stat source file: {}", e),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
.len();
|
||||||
|
|
||||||
|
// Idempotent: if blob already exists, just remove the source
|
||||||
|
if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
return Ok(file_size);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Atomic rename (same filesystem). Falls back to copy+delete for
|
||||||
|
// cross-device moves (EXDEV errno 18).
|
||||||
|
if let Err(e) = fs::rename(&source_path, &blob_path).await {
|
||||||
|
if e.raw_os_error() == Some(18) {
|
||||||
|
// EXDEV — cross-device link
|
||||||
|
fs::copy(&source_path, &blob_path).await.map_err(|ce| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to copy file to blob store: {}", ce),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
|
||||||
|
// Concurrent writer placed the blob — discard our copy
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
tracing::debug!("Blob placed by concurrent writer: {}", e);
|
||||||
|
} else {
|
||||||
|
return Err(DomainError::internal_error(
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to move file to blob store: {}", e),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(file_size)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
let file = File::open(&blob_path).await.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to open blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
let mut file = File::open(&blob_path).await.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to open blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
file.seek(std::io::SeekFrom::Start(start))
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if let Some(end_pos) = end {
|
||||||
|
use tokio::io::AsyncReadExt;
|
||||||
|
let limit = end_pos.saturating_sub(start);
|
||||||
|
let limited = file.take(limit);
|
||||||
|
Ok(Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||||
|
} else {
|
||||||
|
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
match fs::remove_file(&blob_path).await {
|
||||||
|
Ok(()) => Ok(()),
|
||||||
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), // idempotent
|
||||||
|
Err(e) => Err(DomainError::internal_error(
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to delete blob {}: {}", hash, e),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
Ok(fs::try_exists(&blob_path).await.unwrap_or(false))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let blob_path = self.blob_path(&hash);
|
||||||
|
let meta = fs::metadata(&blob_path).await.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"Blob",
|
||||||
|
format!("Failed to stat blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
Ok(meta.len())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
Box::pin(async move {
|
||||||
|
let writable = fs::metadata(&self.blob_root).await.is_ok();
|
||||||
|
Ok(StorageHealthStatus {
|
||||||
|
connected: writable,
|
||||||
|
backend_type: "local".to_string(),
|
||||||
|
message: if writable {
|
||||||
|
"Local filesystem is accessible".to_string()
|
||||||
|
} else {
|
||||||
|
"Blob root directory is not accessible".to_string()
|
||||||
|
},
|
||||||
|
available_bytes: None,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"local"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||||
|
Some(self.blob_path(hash))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,204 @@
|
|||||||
|
//! `MigrationBlobBackend` — decorator that enables zero-downtime migration
|
||||||
|
//! between blob storage backends.
|
||||||
|
//!
|
||||||
|
//! During a migration the decorator writes to the **target** backend and reads
|
||||||
|
//! from **target-first-then-source** (dual-read). A background job
|
||||||
|
//! (see `migration_job.rs`) copies remaining blobs in the background.
|
||||||
|
|
||||||
|
use std::future::Future;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use chrono::{DateTime, Utc};
|
||||||
|
use serde::Serialize;
|
||||||
|
use tokio::sync::RwLock;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::common::errors::DomainError;
|
||||||
|
|
||||||
|
// ── Migration state ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Progress of an ongoing (or completed) backend migration.
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct MigrationState {
|
||||||
|
pub status: MigrationStatus,
|
||||||
|
pub total_blobs: u64,
|
||||||
|
pub migrated_blobs: u64,
|
||||||
|
pub migrated_bytes: u64,
|
||||||
|
pub failed_blobs: Vec<String>,
|
||||||
|
pub started_at: Option<DateTime<Utc>>,
|
||||||
|
pub completed_at: Option<DateTime<Utc>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for MigrationState {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
status: MigrationStatus::Idle,
|
||||||
|
total_blobs: 0,
|
||||||
|
migrated_blobs: 0,
|
||||||
|
migrated_bytes: 0,
|
||||||
|
failed_blobs: Vec::new(),
|
||||||
|
started_at: None,
|
||||||
|
completed_at: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Status of the migration job.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum MigrationStatus {
|
||||||
|
Idle,
|
||||||
|
Running,
|
||||||
|
Paused,
|
||||||
|
Completed,
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── MigrationBlobBackend ───────────────────────────────────────────
|
||||||
|
|
||||||
|
/// A `BlobStorageBackend` decorator that proxies requests to a *source*
|
||||||
|
/// (old) and *target* (new) backend, enabling live migration.
|
||||||
|
pub struct MigrationBlobBackend {
|
||||||
|
source: Arc<dyn BlobStorageBackend>,
|
||||||
|
target: Arc<dyn BlobStorageBackend>,
|
||||||
|
state: Arc<RwLock<MigrationState>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MigrationBlobBackend {
|
||||||
|
pub fn new(
|
||||||
|
source: Arc<dyn BlobStorageBackend>,
|
||||||
|
target: Arc<dyn BlobStorageBackend>,
|
||||||
|
state: Arc<RwLock<MigrationState>>,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
source,
|
||||||
|
target,
|
||||||
|
state,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn state(&self) -> &Arc<RwLock<MigrationState>> {
|
||||||
|
&self.state
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn source(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||||
|
&self.source
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn target(&self) -> &Arc<dyn BlobStorageBackend> {
|
||||||
|
&self.target
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Boxed future alias (same as in the trait module).
|
||||||
|
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||||
|
|
||||||
|
impl BlobStorageBackend for MigrationBlobBackend {
|
||||||
|
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
self.target.initialize().await?;
|
||||||
|
// Source is already initialised; call anyway for idempotency.
|
||||||
|
self.source.initialize().await?;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Writes go to **target** only.
|
||||||
|
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let path = source_path.to_path_buf();
|
||||||
|
Box::pin(async move { self.target.put_blob(&hash, &path).await })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read from target first; fall back to source.
|
||||||
|
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
match self.target.get_blob_stream(&hash).await {
|
||||||
|
Ok(stream) => Ok(stream),
|
||||||
|
Err(_) => self.source.get_blob_stream(&hash).await,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
match self.target.get_blob_range_stream(&hash, start, end).await {
|
||||||
|
Ok(stream) => Ok(stream),
|
||||||
|
Err(_) => self.source.get_blob_range_stream(&hash, start, end).await,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Delete from **both** backends (best-effort on source).
|
||||||
|
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
self.target.delete_blob(&hash).await?;
|
||||||
|
// Best-effort on source — ignore errors (blob may already be gone).
|
||||||
|
let _ = self.source.delete_blob(&hash).await;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Exists in either backend.
|
||||||
|
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
if self.target.blob_exists(&hash).await? {
|
||||||
|
return Ok(true);
|
||||||
|
}
|
||||||
|
self.source.blob_exists(&hash).await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
match self.target.blob_size(&hash).await {
|
||||||
|
Ok(sz) => Ok(sz),
|
||||||
|
Err(_) => self.source.blob_size(&hash).await,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
let target_health = self.target.health_check().await?;
|
||||||
|
let source_health = self.source.health_check().await?;
|
||||||
|
Ok(StorageHealthStatus {
|
||||||
|
connected: target_health.connected && source_health.connected,
|
||||||
|
backend_type: format!(
|
||||||
|
"migration({} → {})",
|
||||||
|
source_health.backend_type, target_health.backend_type
|
||||||
|
),
|
||||||
|
message: format!(
|
||||||
|
"Source: {} | Target: {}",
|
||||||
|
source_health.message, target_health.message
|
||||||
|
),
|
||||||
|
available_bytes: target_health.available_bytes,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"migration"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||||
|
// Prefer target, fall back to source.
|
||||||
|
self.target
|
||||||
|
.local_blob_path(hash)
|
||||||
|
.or_else(|| self.source.local_blob_path(hash))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
//! Background migration job — copies blobs from a source backend to a target
|
||||||
|
//! backend with configurable concurrency and progress tracking.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use futures::StreamExt;
|
||||||
|
use serde::Serialize;
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use tokio::sync::RwLock;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
|
use crate::common::errors::DomainError;
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::{MigrationState, MigrationStatus};
|
||||||
|
|
||||||
|
/// Run the migration: stream all blob hashes from `storage.blobs` and copy
|
||||||
|
/// each one from `source` to `target`.
|
||||||
|
///
|
||||||
|
/// * The job respects `Paused` / `Failed` status in `state` — it will stop
|
||||||
|
/// streaming when the status is no longer `Running`.
|
||||||
|
/// * Errors on individual blobs are logged and collected in `failed_blobs`
|
||||||
|
/// but do **not** abort the full run.
|
||||||
|
/// * `concurrency` controls `buffer_unordered` parallelism (default: 4).
|
||||||
|
pub async fn run_migration(
|
||||||
|
source: Arc<dyn BlobStorageBackend>,
|
||||||
|
target: Arc<dyn BlobStorageBackend>,
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
state: Arc<RwLock<MigrationState>>,
|
||||||
|
concurrency: usize,
|
||||||
|
) -> Result<(), DomainError> {
|
||||||
|
// Count total blobs for progress tracking.
|
||||||
|
let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||||
|
.fetch_one(pool.as_ref())
|
||||||
|
.await
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
|
{
|
||||||
|
let mut s = state.write().await;
|
||||||
|
s.status = MigrationStatus::Running;
|
||||||
|
s.total_blobs = total as u64;
|
||||||
|
s.migrated_blobs = 0;
|
||||||
|
s.migrated_bytes = 0;
|
||||||
|
s.failed_blobs.clear();
|
||||||
|
s.started_at = Some(chrono::Utc::now());
|
||||||
|
s.completed_at = None;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stream all hashes+sizes with a cursor.
|
||||||
|
let mut rows =
|
||||||
|
sqlx::query_as::<_, (String, i64)>("SELECT hash, size FROM storage.blobs ORDER BY hash")
|
||||||
|
.fetch(pool.as_ref());
|
||||||
|
|
||||||
|
// Collect all hashes first to avoid holding the cursor across awaits.
|
||||||
|
let mut work: Vec<(String, i64)> = Vec::with_capacity(total as usize);
|
||||||
|
while let Some(row) = rows.next().await {
|
||||||
|
match row {
|
||||||
|
Ok(r) => work.push(r),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Error fetching blob row during migration: {}", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process in parallel chunks.
|
||||||
|
let results = futures::stream::iter(work.into_iter().map(|(hash, size)| {
|
||||||
|
let src = source.clone();
|
||||||
|
let tgt = target.clone();
|
||||||
|
let st = state.clone();
|
||||||
|
async move {
|
||||||
|
// Check if we should keep running.
|
||||||
|
{
|
||||||
|
let s = st.read().await;
|
||||||
|
if s.status != MigrationStatus::Running {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Skip if already in target.
|
||||||
|
match tgt.blob_exists(&hash).await {
|
||||||
|
Ok(true) => {
|
||||||
|
let mut s = st.write().await;
|
||||||
|
s.migrated_blobs += 1;
|
||||||
|
s.migrated_bytes += size as u64;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Ok(false) => {}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("blob_exists check failed for {}: {}", hash, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy: stream from source → temp file → put into target.
|
||||||
|
if let Err(e) = copy_blob(&src, &tgt, &hash).await {
|
||||||
|
tracing::warn!("Failed to migrate blob {}: {}", hash, e);
|
||||||
|
let mut s = st.write().await;
|
||||||
|
s.failed_blobs.push(hash);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut s = st.write().await;
|
||||||
|
s.migrated_blobs += 1;
|
||||||
|
s.migrated_bytes += size as u64;
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
.buffer_unordered(concurrency)
|
||||||
|
.collect::<Vec<()>>()
|
||||||
|
.await;
|
||||||
|
|
||||||
|
drop(results);
|
||||||
|
|
||||||
|
// Finalize state.
|
||||||
|
let mut s = state.write().await;
|
||||||
|
if s.status == MigrationStatus::Running {
|
||||||
|
if s.failed_blobs.is_empty() {
|
||||||
|
s.status = MigrationStatus::Completed;
|
||||||
|
} else {
|
||||||
|
s.status = MigrationStatus::Failed;
|
||||||
|
}
|
||||||
|
s.completed_at = Some(chrono::Utc::now());
|
||||||
|
}
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
"Migration finished: {}/{} blobs, {} failures",
|
||||||
|
s.migrated_blobs,
|
||||||
|
s.total_blobs,
|
||||||
|
s.failed_blobs.len()
|
||||||
|
);
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Copy a single blob: stream from source → spool to temp file → put_blob into target.
|
||||||
|
async fn copy_blob(
|
||||||
|
source: &Arc<dyn BlobStorageBackend>,
|
||||||
|
target: &Arc<dyn BlobStorageBackend>,
|
||||||
|
hash: &str,
|
||||||
|
) -> Result<(), DomainError> {
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
|
|
||||||
|
// Create a temp file to spool content.
|
||||||
|
let tmp_dir = std::env::temp_dir().join("oxicloud-migration");
|
||||||
|
tokio::fs::create_dir_all(&tmp_dir).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Migration", format!("Failed to create temp dir: {}", e))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let tmp_path = tmp_dir.join(format!("{}.tmp", hash));
|
||||||
|
|
||||||
|
// Stream from source.
|
||||||
|
let stream = source.get_blob_stream(hash).await?;
|
||||||
|
|
||||||
|
// Write to temp file.
|
||||||
|
let mut file = tokio::fs::File::create(&tmp_path).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("Migration", format!("Failed to create temp file: {}", e))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let mut stream = std::pin::pin!(stream);
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
let bytes = chunk.map_err(|e| {
|
||||||
|
DomainError::internal_error("Migration", format!("Stream error: {}", e))
|
||||||
|
})?;
|
||||||
|
file.write_all(&bytes)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Migration", format!("Write error: {}", e)))?;
|
||||||
|
}
|
||||||
|
file.flush()
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Migration", format!("Flush error: {}", e)))?;
|
||||||
|
drop(file);
|
||||||
|
|
||||||
|
// Put into target.
|
||||||
|
target.put_blob(hash, &tmp_path).await?;
|
||||||
|
|
||||||
|
// Clean up temp file.
|
||||||
|
let _ = tokio::fs::remove_file(&tmp_path).await;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Verify migration integrity by comparing blob counts and sampling random hashes.
|
||||||
|
pub async fn verify_migration(
|
||||||
|
target: Arc<dyn BlobStorageBackend>,
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
sample_size: usize,
|
||||||
|
) -> Result<VerificationResult, DomainError> {
|
||||||
|
// 1. Count blobs in PG.
|
||||||
|
let pg_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||||
|
.fetch_one(pool.as_ref())
|
||||||
|
.await
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
|
// 2. Verify sample of blobs exist in target.
|
||||||
|
let sample_rows: Vec<(String, i64)> =
|
||||||
|
sqlx::query_as("SELECT hash, size FROM storage.blobs ORDER BY random() LIMIT $1")
|
||||||
|
.bind(sample_size as i64)
|
||||||
|
.fetch_all(pool.as_ref())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("Migration", format!("Sample query failed: {}", e))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let mut missing = Vec::new();
|
||||||
|
let mut size_mismatches = Vec::new();
|
||||||
|
|
||||||
|
for (hash, expected_size) in &sample_rows {
|
||||||
|
match target.blob_exists(hash).await {
|
||||||
|
Ok(false) => missing.push(hash.clone()),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("blob_exists failed for {}: {}", hash, e);
|
||||||
|
missing.push(hash.clone());
|
||||||
|
}
|
||||||
|
Ok(true) => {
|
||||||
|
// Verify size matches.
|
||||||
|
if let Ok(actual_size) = target.blob_size(hash).await
|
||||||
|
&& actual_size != *expected_size as u64
|
||||||
|
{
|
||||||
|
size_mismatches.push(hash.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let passed = missing.is_empty() && size_mismatches.is_empty();
|
||||||
|
|
||||||
|
Ok(VerificationResult {
|
||||||
|
pg_blob_count: pg_count as u64,
|
||||||
|
sample_checked: sample_rows.len() as u64,
|
||||||
|
missing_in_target: missing,
|
||||||
|
size_mismatches,
|
||||||
|
passed,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result of a post-migration integrity check.
|
||||||
|
#[derive(Debug, Clone, Serialize, serde::Deserialize)]
|
||||||
|
pub struct VerificationResult {
|
||||||
|
pub pg_blob_count: u64,
|
||||||
|
pub sample_checked: u64,
|
||||||
|
pub missing_in_target: Vec<String>,
|
||||||
|
pub size_mismatches: Vec<String>,
|
||||||
|
pub passed: bool,
|
||||||
|
}
|
||||||
@@ -1,18 +1,26 @@
|
|||||||
pub mod audio_metadata_service;
|
pub mod audio_metadata_service;
|
||||||
|
pub mod azure_blob_backend;
|
||||||
|
pub mod cached_blob_backend;
|
||||||
pub mod chunked_upload_service;
|
pub mod chunked_upload_service;
|
||||||
pub mod compression_service;
|
pub mod compression_service;
|
||||||
pub mod dedup_service;
|
pub mod dedup_service;
|
||||||
|
pub mod encrypted_blob_backend;
|
||||||
pub mod exif_service;
|
pub mod exif_service;
|
||||||
pub mod file_content_cache;
|
pub mod file_content_cache;
|
||||||
pub mod file_system_i18n_service;
|
pub mod file_system_i18n_service;
|
||||||
pub mod image_transcode_service;
|
pub mod image_transcode_service;
|
||||||
pub mod jwt_service;
|
pub mod jwt_service;
|
||||||
|
pub mod local_blob_backend;
|
||||||
pub mod login_lockout_service;
|
pub mod login_lockout_service;
|
||||||
|
pub mod migration_blob_backend;
|
||||||
|
pub mod migration_job;
|
||||||
pub mod nextcloud_chunked_upload_service;
|
pub mod nextcloud_chunked_upload_service;
|
||||||
pub mod oidc_service;
|
pub mod oidc_service;
|
||||||
pub mod password_hasher;
|
pub mod password_hasher;
|
||||||
pub mod path_resolver_service;
|
pub mod path_resolver_service;
|
||||||
pub mod path_service;
|
pub mod path_service;
|
||||||
|
pub mod retry_blob_backend;
|
||||||
|
pub mod s3_blob_backend;
|
||||||
pub mod thumbnail_service;
|
pub mod thumbnail_service;
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod thumbnail_service_test;
|
mod thumbnail_service_test;
|
||||||
|
|||||||
@@ -0,0 +1,254 @@
|
|||||||
|
//! `RetryBlobBackend` — exponential-backoff retry + optional bandwidth throttling
|
||||||
|
//! decorator for remote blob backends.
|
||||||
|
//!
|
||||||
|
//! Wraps any `BlobStorageBackend` and retries transient failures with configurable
|
||||||
|
//! exponential backoff. Optionally throttles upload/download bandwidth via
|
||||||
|
//! inter-chunk sleeps.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
// ── Retry policy ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Exponential backoff retry configuration.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct RetryPolicy {
|
||||||
|
/// Maximum number of retry attempts (0 = no retries).
|
||||||
|
pub max_retries: u32,
|
||||||
|
/// Initial backoff duration before the first retry.
|
||||||
|
pub initial_backoff: Duration,
|
||||||
|
/// Maximum backoff duration (capped).
|
||||||
|
pub max_backoff: Duration,
|
||||||
|
/// Multiplier applied to backoff after each attempt.
|
||||||
|
pub backoff_multiplier: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for RetryPolicy {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
max_retries: 3,
|
||||||
|
initial_backoff: Duration::from_millis(100),
|
||||||
|
max_backoff: Duration::from_secs(10),
|
||||||
|
backoff_multiplier: 2.0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── RetryBlobBackend ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Decorator that retries failed backend operations with exponential backoff.
|
||||||
|
pub struct RetryBlobBackend {
|
||||||
|
inner: Arc<dyn BlobStorageBackend>,
|
||||||
|
policy: RetryPolicy,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RetryBlobBackend {
|
||||||
|
pub fn new(inner: Arc<dyn BlobStorageBackend>, policy: RetryPolicy) -> Self {
|
||||||
|
Self { inner, policy }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Execute an async closure with exponential backoff retry.
|
||||||
|
async fn retry_async<F, Fut, T>(
|
||||||
|
policy: &RetryPolicy,
|
||||||
|
name: &str,
|
||||||
|
mut f: F,
|
||||||
|
) -> Result<T, DomainError>
|
||||||
|
where
|
||||||
|
F: FnMut() -> Fut,
|
||||||
|
Fut: std::future::Future<Output = Result<T, DomainError>>,
|
||||||
|
{
|
||||||
|
let mut attempt = 0u32;
|
||||||
|
let mut backoff = policy.initial_backoff;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
match f().await {
|
||||||
|
Ok(v) => return Ok(v),
|
||||||
|
Err(e) if attempt < policy.max_retries && is_retryable(&e) => {
|
||||||
|
attempt += 1;
|
||||||
|
tracing::warn!(
|
||||||
|
"Retry {}/{} for {} after error: {} (backoff {:?})",
|
||||||
|
attempt,
|
||||||
|
policy.max_retries,
|
||||||
|
name,
|
||||||
|
e,
|
||||||
|
backoff
|
||||||
|
);
|
||||||
|
tokio::time::sleep(backoff).await;
|
||||||
|
let next =
|
||||||
|
Duration::from_secs_f64(backoff.as_secs_f64() * policy.backoff_multiplier);
|
||||||
|
backoff = next.min(policy.max_backoff);
|
||||||
|
}
|
||||||
|
Err(e) => return Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Determine if an error is likely transient (network timeout, 5xx, etc.).
|
||||||
|
fn is_retryable(err: &DomainError) -> bool {
|
||||||
|
let msg = err.to_string().to_lowercase();
|
||||||
|
msg.contains("timeout")
|
||||||
|
|| msg.contains("connection")
|
||||||
|
|| msg.contains("503")
|
||||||
|
|| msg.contains("500")
|
||||||
|
|| msg.contains("429")
|
||||||
|
|| msg.contains("temporarily")
|
||||||
|
|| msg.contains("broken pipe")
|
||||||
|
|| msg.contains("reset by peer")
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for RetryBlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, "initialize", || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
async move { inner.initialize().await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
let path = source_path.to_path_buf();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("put_blob({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
let path = path.clone();
|
||||||
|
async move { inner.put_blob(&hash, &path).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("get_blob_stream({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
async move { inner.get_blob_stream(&hash).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("get_blob_range({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
async move { inner.get_blob_range_stream(&hash, start, end).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("delete_blob({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
async move { inner.delete_blob(&hash).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("blob_exists({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
async move { inner.blob_exists(&hash).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
let hash = hash.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, &format!("blob_size({hash})"), || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
let hash = hash.clone();
|
||||||
|
async move { inner.blob_size(&hash).await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
let inner = self.inner.clone();
|
||||||
|
let policy = self.policy.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
retry_async(&policy, "health_check", || {
|
||||||
|
let inner = inner.clone();
|
||||||
|
async move { inner.health_check().await }
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"retry"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
|
||||||
|
self.inner.local_blob_path(hash)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,344 @@
|
|||||||
|
//! S3-Compatible Blob Backend — stores blobs in any S3-compatible object store.
|
||||||
|
//!
|
||||||
|
//! Supports AWS S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces,
|
||||||
|
//! Wasabi, and any other service that implements the S3 API.
|
||||||
|
|
||||||
|
use aws_sdk_s3::primitives::ByteStream;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use tokio::fs;
|
||||||
|
use tokio_util::io::ReaderStream;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_storage_ports::{
|
||||||
|
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||||
|
};
|
||||||
|
use crate::common::config::S3StorageConfig;
|
||||||
|
use crate::domain::errors::{DomainError, ErrorKind};
|
||||||
|
|
||||||
|
/// S3-compatible blob storage backend.
|
||||||
|
///
|
||||||
|
/// Blobs are stored as objects with key `{2-char-prefix}/{hash}.blob`,
|
||||||
|
/// mirroring the local filesystem layout for consistency.
|
||||||
|
pub struct S3BlobBackend {
|
||||||
|
client: aws_sdk_s3::Client,
|
||||||
|
bucket: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl S3BlobBackend {
|
||||||
|
/// Build a new S3 backend from configuration.
|
||||||
|
///
|
||||||
|
/// Supports custom endpoints for non-AWS providers (Backblaze B2,
|
||||||
|
/// MinIO, Cloudflare R2, etc.).
|
||||||
|
pub fn new(config: &S3StorageConfig) -> Self {
|
||||||
|
let credentials = aws_sdk_s3::config::Credentials::new(
|
||||||
|
&config.access_key,
|
||||||
|
&config.secret_key,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
"oxicloud",
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut builder = aws_sdk_s3::config::Builder::new()
|
||||||
|
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
|
||||||
|
.credentials_provider(credentials)
|
||||||
|
.behavior_version_latest();
|
||||||
|
|
||||||
|
if let Some(ref endpoint) = config.endpoint_url {
|
||||||
|
builder = builder.endpoint_url(endpoint);
|
||||||
|
}
|
||||||
|
|
||||||
|
if config.force_path_style {
|
||||||
|
builder = builder.force_path_style(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
let client = aws_sdk_s3::Client::from_conf(builder.build());
|
||||||
|
|
||||||
|
Self {
|
||||||
|
client,
|
||||||
|
bucket: config.bucket.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute the S3 object key for a given hash.
|
||||||
|
fn object_key(hash: &str) -> String {
|
||||||
|
let prefix = &hash[0..2];
|
||||||
|
format!("{}/{}.blob", prefix, hash)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobStorageBackend for S3BlobBackend {
|
||||||
|
fn initialize(
|
||||||
|
&self,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
// Verify bucket exists and is accessible
|
||||||
|
self.client
|
||||||
|
.head_bucket()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"S3",
|
||||||
|
format!("Cannot access bucket '{}': {}", self.bucket, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
tracing::info!("S3 blob backend initialized: bucket={}", self.bucket);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn put_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
source_path: &Path,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
let source_path = source_path.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
// Check if object already exists (idempotent)
|
||||||
|
let exists = self
|
||||||
|
.client
|
||||||
|
.head_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.is_ok();
|
||||||
|
|
||||||
|
if exists {
|
||||||
|
// Blob already in S3 — remove local source and return size
|
||||||
|
let file_size = fs::metadata(&source_path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"S3",
|
||||||
|
format!("Failed to stat source file: {}", e),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
.len();
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
return Ok(file_size);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload from local file
|
||||||
|
let body = ByteStream::from_path(&source_path).await.map_err(|e| {
|
||||||
|
DomainError::internal_error("S3", format!("Failed to read source file: {}", e))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let file_size = fs::metadata(&source_path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("S3", format!("Failed to stat source file: {}", e))
|
||||||
|
})?
|
||||||
|
.len();
|
||||||
|
|
||||||
|
self.client
|
||||||
|
.put_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.body(body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"S3",
|
||||||
|
format!("Failed to upload blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Clean up local source after successful upload
|
||||||
|
let _ = fs::remove_file(&source_path).await;
|
||||||
|
|
||||||
|
Ok(file_size)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
let output = self
|
||||||
|
.client
|
||||||
|
.get_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"S3",
|
||||||
|
format!("Failed to get blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Convert S3 ByteStream into a Stream<Item = Result<Bytes, io::Error>>
|
||||||
|
// via AsyncRead adapter
|
||||||
|
let reader = output.body.into_async_read();
|
||||||
|
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_blob_range_stream(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
start: u64,
|
||||||
|
end: Option<u64>,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
|
||||||
|
{
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
let range = match end {
|
||||||
|
Some(end_pos) => format!("bytes={}-{}", start, end_pos.saturating_sub(1)),
|
||||||
|
None => format!("bytes={}-", start),
|
||||||
|
};
|
||||||
|
|
||||||
|
let output = self
|
||||||
|
.client
|
||||||
|
.get_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.range(range)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"S3",
|
||||||
|
format!("Failed to get blob range {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let reader = output.body.into_async_read();
|
||||||
|
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delete_blob(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
// S3 DeleteObject is already idempotent (returns 204 even if not found)
|
||||||
|
self.client
|
||||||
|
.delete_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"S3",
|
||||||
|
format!("Failed to delete blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_exists(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
match self
|
||||||
|
.client
|
||||||
|
.head_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(_) => Ok(true),
|
||||||
|
Err(e) => {
|
||||||
|
// Check if it's a 404 (not found) vs an actual error
|
||||||
|
let service_err = e.into_service_error();
|
||||||
|
if service_err.is_not_found() {
|
||||||
|
Ok(false)
|
||||||
|
} else {
|
||||||
|
Err(DomainError::internal_error(
|
||||||
|
"S3",
|
||||||
|
format!("Failed to check blob {}: {}", hash, service_err),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn blob_size(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||||
|
let hash = hash.to_owned();
|
||||||
|
Box::pin(async move {
|
||||||
|
let key = Self::object_key(&hash);
|
||||||
|
|
||||||
|
let output = self
|
||||||
|
.client
|
||||||
|
.head_object()
|
||||||
|
.bucket(&self.bucket)
|
||||||
|
.key(&key)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::new(
|
||||||
|
ErrorKind::NotFound,
|
||||||
|
"S3",
|
||||||
|
format!("Failed to stat blob {}: {}", hash, e),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
Ok(output.content_length().unwrap_or(0) as u64)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_check(
|
||||||
|
&self,
|
||||||
|
) -> Pin<
|
||||||
|
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
|
||||||
|
> {
|
||||||
|
Box::pin(async move {
|
||||||
|
match self.client.head_bucket().bucket(&self.bucket).send().await {
|
||||||
|
Ok(_) => Ok(StorageHealthStatus {
|
||||||
|
connected: true,
|
||||||
|
backend_type: "s3".to_string(),
|
||||||
|
message: format!("S3 bucket '{}' is accessible", self.bucket),
|
||||||
|
available_bytes: None,
|
||||||
|
}),
|
||||||
|
Err(e) => Ok(StorageHealthStatus {
|
||||||
|
connected: false,
|
||||||
|
backend_type: "s3".to_string(),
|
||||||
|
message: format!("S3 bucket '{}' is not accessible: {}", self.bucket, e),
|
||||||
|
available_bytes: None,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn backend_type(&self) -> &'static str {
|
||||||
|
"s3"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||||
|
None // Remote backend — no local path
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,8 +8,9 @@ use axum::{
|
|||||||
|
|
||||||
use crate::application::dtos::settings_dto::{
|
use crate::application::dtos::settings_dto::{
|
||||||
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, ListUsersQueryDto,
|
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, ListUsersQueryDto,
|
||||||
SaveOidcSettingsDto, TestOidcConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
|
MigrationStateDto, SaveOidcSettingsDto, SaveStorageSettingsDto, StartMigrationDto,
|
||||||
UpdateUserRoleDto,
|
TestOidcConnectionDto, TestStorageConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
|
||||||
|
UpdateUserRoleDto, VerifyMigrationDto,
|
||||||
};
|
};
|
||||||
use crate::application::ports::auth_ports::TokenServicePort;
|
use crate::application::ports::auth_ports::TokenServicePort;
|
||||||
use crate::common::di::AppState;
|
use crate::common::di::AppState;
|
||||||
@@ -24,6 +25,22 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
|
|||||||
.route("/settings/oidc", get(get_oidc_settings))
|
.route("/settings/oidc", get(get_oidc_settings))
|
||||||
.route("/settings/oidc", put(save_oidc_settings))
|
.route("/settings/oidc", put(save_oidc_settings))
|
||||||
.route("/settings/oidc/test", post(test_oidc_connection))
|
.route("/settings/oidc/test", post(test_oidc_connection))
|
||||||
|
// Storage settings
|
||||||
|
.route("/settings/storage", get(get_storage_settings))
|
||||||
|
.route("/settings/storage", put(save_storage_settings))
|
||||||
|
.route("/settings/storage/test", post(test_storage_connection))
|
||||||
|
// Storage migration
|
||||||
|
.route("/storage/migration", get(get_migration_status))
|
||||||
|
.route("/storage/migration/start", post(start_migration))
|
||||||
|
.route("/storage/migration/pause", post(pause_migration))
|
||||||
|
.route("/storage/migration/resume", post(resume_migration))
|
||||||
|
.route("/storage/migration/complete", post(complete_migration))
|
||||||
|
.route("/storage/migration/verify", post(verify_migration))
|
||||||
|
// Encryption key generation
|
||||||
|
.route(
|
||||||
|
"/settings/storage/generate-key",
|
||||||
|
post(generate_encryption_key),
|
||||||
|
)
|
||||||
.route("/settings/general", get(get_general_settings))
|
.route("/settings/general", get(get_general_settings))
|
||||||
// Dashboard / stats
|
// Dashboard / stats
|
||||||
.route("/dashboard", get(get_dashboard_stats))
|
.route("/dashboard", get(get_dashboard_stats))
|
||||||
@@ -148,6 +165,336 @@ async fn test_oidc_connection(
|
|||||||
Ok(Json(result))
|
Ok(Json(result))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
// Storage settings handlers
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// GET /api/admin/settings/storage — get storage backend settings
|
||||||
|
async fn get_storage_settings(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let svc = state
|
||||||
|
.storage_settings_service
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||||
|
|
||||||
|
let settings = svc
|
||||||
|
.get_storage_settings()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to load storage settings: {}", e)))?;
|
||||||
|
|
||||||
|
Ok(Json(settings))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// PUT /api/admin/settings/storage — save storage backend settings
|
||||||
|
async fn save_storage_settings(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
Json(dto): Json<SaveStorageSettingsDto>,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
let (user_id, _) = admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let svc = state
|
||||||
|
.storage_settings_service
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||||
|
|
||||||
|
svc.save_storage_settings(dto, user_id)
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to save storage settings: {}", e)))?;
|
||||||
|
|
||||||
|
Ok((
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(serde_json::json!({
|
||||||
|
"message": "Storage settings saved successfully"
|
||||||
|
})),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/settings/storage/test — test storage backend connection
|
||||||
|
async fn test_storage_connection(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
Json(dto): Json<TestStorageConnectionDto>,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let svc = state
|
||||||
|
.storage_settings_service
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||||
|
|
||||||
|
let result = svc
|
||||||
|
.test_storage_connection(dto)
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Storage connection test failed: {}", e)))?;
|
||||||
|
|
||||||
|
Ok(Json(result))
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
// Storage migration handlers
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// GET /api/admin/storage/migration — current migration progress
|
||||||
|
async fn get_migration_status(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
let s = state.migration_state.read().await;
|
||||||
|
Ok(Json(migration_state_to_dto(&s)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/storage/migration/start — begin background migration
|
||||||
|
async fn start_migration(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
Json(dto): Json<StartMigrationDto>,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||||
|
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
// Check not already running.
|
||||||
|
{
|
||||||
|
let s = state.migration_state.read().await;
|
||||||
|
if s.status == MigrationStatus::Running {
|
||||||
|
return Err(AppError::bad_request("A migration is already running"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let pool = state
|
||||||
|
.db_pool
|
||||||
|
.clone()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Database not available"))?;
|
||||||
|
|
||||||
|
let source = state.core.dedup_service.backend().clone();
|
||||||
|
let svc = state
|
||||||
|
.storage_settings_service
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||||
|
|
||||||
|
// Build target backend from saved settings.
|
||||||
|
let effective = svc
|
||||||
|
.load_effective_storage_config()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
|
||||||
|
|
||||||
|
let target = build_backend_from_config(&effective)
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
|
||||||
|
target
|
||||||
|
.initialize()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
|
||||||
|
|
||||||
|
let concurrency = dto.concurrency.unwrap_or(4).clamp(1, 16);
|
||||||
|
let migration_state = state.migration_state.clone();
|
||||||
|
|
||||||
|
// Spawn the background migration job.
|
||||||
|
tokio::spawn(async move {
|
||||||
|
if let Err(e) = crate::infrastructure::services::migration_job::run_migration(
|
||||||
|
source,
|
||||||
|
target,
|
||||||
|
pool,
|
||||||
|
migration_state,
|
||||||
|
concurrency,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
tracing::error!("Migration job error: {}", e);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
Ok((
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(serde_json::json!({ "message": "Migration started" })),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/storage/migration/pause — pause running migration
|
||||||
|
async fn pause_migration(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let mut s = state.migration_state.write().await;
|
||||||
|
if s.status != MigrationStatus::Running {
|
||||||
|
return Err(AppError::bad_request("No running migration to pause"));
|
||||||
|
}
|
||||||
|
s.status = MigrationStatus::Paused;
|
||||||
|
Ok((
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(serde_json::json!({ "message": "Migration paused" })),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/storage/migration/resume — resume paused migration
|
||||||
|
async fn resume_migration(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
// Set status back to Running — the background task checks on each blob.
|
||||||
|
let mut s = state.migration_state.write().await;
|
||||||
|
if s.status != MigrationStatus::Paused {
|
||||||
|
return Err(AppError::bad_request("No paused migration to resume"));
|
||||||
|
}
|
||||||
|
s.status = MigrationStatus::Running;
|
||||||
|
Ok((
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(serde_json::json!({ "message": "Migration resumed" })),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/storage/migration/complete — finalize migration
|
||||||
|
async fn complete_migration(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let s = state.migration_state.read().await;
|
||||||
|
if s.status != MigrationStatus::Completed {
|
||||||
|
return Err(AppError::bad_request(
|
||||||
|
"Migration must be completed (100%) before finalizing",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
drop(s);
|
||||||
|
|
||||||
|
// Mark as idle — the admin has acknowledged completion.
|
||||||
|
let mut s = state.migration_state.write().await;
|
||||||
|
s.status = MigrationStatus::Idle;
|
||||||
|
|
||||||
|
Ok((
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(
|
||||||
|
serde_json::json!({ "message": "Migration finalized. Restart the server to use the new backend." }),
|
||||||
|
),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/storage/migration/verify — run integrity check
|
||||||
|
async fn verify_migration(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
Json(dto): Json<VerifyMigrationDto>,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let pool = state
|
||||||
|
.db_pool
|
||||||
|
.clone()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Database not available"))?;
|
||||||
|
|
||||||
|
let svc = state
|
||||||
|
.storage_settings_service
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
|
||||||
|
|
||||||
|
let effective = svc
|
||||||
|
.load_effective_storage_config()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
|
||||||
|
|
||||||
|
let target = build_backend_from_config(&effective)
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
|
||||||
|
target
|
||||||
|
.initialize()
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
|
||||||
|
|
||||||
|
let sample_size = dto.sample_size.unwrap_or(100).clamp(1, 1000);
|
||||||
|
|
||||||
|
let result =
|
||||||
|
crate::infrastructure::services::migration_job::verify_migration(target, pool, sample_size)
|
||||||
|
.await
|
||||||
|
.map_err(|e| AppError::internal_error(format!("Verification failed: {}", e)))?;
|
||||||
|
|
||||||
|
Ok(Json(result))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Helper: convert MigrationState to DTO for JSON serialization.
|
||||||
|
fn migration_state_to_dto(
|
||||||
|
s: &crate::infrastructure::services::migration_blob_backend::MigrationState,
|
||||||
|
) -> MigrationStateDto {
|
||||||
|
let throughput = match (s.started_at, s.migrated_bytes) {
|
||||||
|
(Some(start), bytes) if bytes > 0 => {
|
||||||
|
let elapsed = chrono::Utc::now()
|
||||||
|
.signed_duration_since(start)
|
||||||
|
.num_seconds()
|
||||||
|
.max(1) as f64;
|
||||||
|
Some(bytes as f64 / elapsed)
|
||||||
|
}
|
||||||
|
_ => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
MigrationStateDto {
|
||||||
|
status: format!("{:?}", s.status).to_lowercase(),
|
||||||
|
total_blobs: s.total_blobs,
|
||||||
|
migrated_blobs: s.migrated_blobs,
|
||||||
|
migrated_bytes: s.migrated_bytes,
|
||||||
|
failed_blobs: s.failed_blobs.clone(),
|
||||||
|
started_at: s.started_at.map(|d| d.to_rfc3339()),
|
||||||
|
completed_at: s.completed_at.map(|d| d.to_rfc3339()),
|
||||||
|
throughput_bytes_per_sec: throughput,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST /api/admin/settings/storage/generate-key — generate a random AES-256 key.
|
||||||
|
async fn generate_encryption_key(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
) -> Result<impl IntoResponse, AppError> {
|
||||||
|
admin_guard(&state, &headers).await?;
|
||||||
|
|
||||||
|
let key =
|
||||||
|
crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend::generate_key(
|
||||||
|
);
|
||||||
|
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
|
||||||
|
|
||||||
|
Ok(Json(serde_json::json!({
|
||||||
|
"key": key_b64,
|
||||||
|
"warning": "Store this key securely. If lost, encrypted data is IRRECOVERABLY LOST."
|
||||||
|
})))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Helper: build a BlobStorageBackend from StorageConfig.
|
||||||
|
fn build_backend_from_config(
|
||||||
|
config: &crate::common::config::StorageConfig,
|
||||||
|
) -> Result<
|
||||||
|
std::sync::Arc<dyn crate::application::ports::blob_storage_ports::BlobStorageBackend>,
|
||||||
|
String,
|
||||||
|
> {
|
||||||
|
match config.backend {
|
||||||
|
crate::common::config::StorageBackendType::Local => Ok(std::sync::Arc::new(
|
||||||
|
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
|
||||||
|
std::path::Path::new(&config.root_dir),
|
||||||
|
),
|
||||||
|
)),
|
||||||
|
crate::common::config::StorageBackendType::S3 => {
|
||||||
|
let s3 = config.s3.as_ref().ok_or("S3 config missing")?;
|
||||||
|
Ok(std::sync::Arc::new(
|
||||||
|
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
crate::common::config::StorageBackendType::Azure => {
|
||||||
|
let az = config.azure.as_ref().ok_or("Azure config missing")?;
|
||||||
|
Ok(std::sync::Arc::new(
|
||||||
|
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(az),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// GET /api/admin/settings/general — system overview (backward compat)
|
/// GET /api/admin/settings/general — system overview (backward compat)
|
||||||
async fn get_general_settings(
|
async fn get_general_settings(
|
||||||
State(state): State<Arc<AppState>>,
|
State(state): State<Arc<AppState>>,
|
||||||
|
|||||||
@@ -53,6 +53,9 @@
|
|||||||
<button class="admin-tab" id="tab-btn-oidc">
|
<button class="admin-tab" id="tab-btn-oidc">
|
||||||
<i class="fas fa-key"></i> <span data-i18n="admin.tab_oidc">SSO / OIDC</span>
|
<i class="fas fa-key"></i> <span data-i18n="admin.tab_oidc">SSO / OIDC</span>
|
||||||
</button>
|
</button>
|
||||||
|
<button class="admin-tab" id="tab-btn-storage">
|
||||||
|
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
|
||||||
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div id="tab-dashboard" class="tab-content active">
|
<div id="tab-dashboard" class="tab-content active">
|
||||||
@@ -424,6 +427,183 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- ════════ Storage tab ════════ -->
|
||||||
|
<div id="tab-storage" class="tab-content">
|
||||||
|
<div class="admin-card">
|
||||||
|
<h2>
|
||||||
|
<i class="fas fa-database"></i> <span data-i18n="admin.storage_title">Storage Backend</span>
|
||||||
|
</h2>
|
||||||
|
|
||||||
|
<!-- Current status -->
|
||||||
|
<div class="storage-status-grid">
|
||||||
|
<div class="storage-stat">
|
||||||
|
<span class="storage-stat__label" data-i18n="admin.storage_current_backend">Active Backend</span>
|
||||||
|
<span class="storage-stat__value" id="storage-current-backend">—</span>
|
||||||
|
</div>
|
||||||
|
<div class="storage-stat">
|
||||||
|
<span class="storage-stat__label" data-i18n="admin.storage_total_blobs">Total Blobs</span>
|
||||||
|
<span class="storage-stat__value" id="storage-total-blobs">—</span>
|
||||||
|
</div>
|
||||||
|
<div class="storage-stat">
|
||||||
|
<span class="storage-stat__label" data-i18n="admin.storage_total_size">Total Size</span>
|
||||||
|
<span class="storage-stat__value" id="storage-total-size">—</span>
|
||||||
|
</div>
|
||||||
|
<div class="storage-stat">
|
||||||
|
<span class="storage-stat__label" data-i18n="admin.storage_dedup_ratio">Dedup Ratio</span>
|
||||||
|
<span class="storage-stat__value" id="storage-dedup-ratio">—</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Backend selector -->
|
||||||
|
<div class="storage-backend-selector">
|
||||||
|
<label class="storage-backend-option">
|
||||||
|
<input type="radio" name="storage-backend" value="local" checked />
|
||||||
|
<span class="storage-backend-option__card">
|
||||||
|
<i class="fas fa-hdd"></i>
|
||||||
|
<span data-i18n="admin.storage_local">Local Filesystem</span>
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
<label class="storage-backend-option">
|
||||||
|
<input type="radio" name="storage-backend" value="s3" />
|
||||||
|
<span class="storage-backend-option__card">
|
||||||
|
<i class="fas fa-cloud"></i>
|
||||||
|
<span data-i18n="admin.storage_s3">S3-Compatible</span>
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- S3 form (shown when S3 is selected) -->
|
||||||
|
<div id="storage-s3-form" class="storage-form" style="display:none">
|
||||||
|
<!-- Provider preset -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label data-i18n="admin.storage_provider_preset">Provider Preset</label>
|
||||||
|
<select id="storage-preset">
|
||||||
|
<option value="custom" data-i18n="admin.storage_preset_custom">Custom</option>
|
||||||
|
<option value="aws">Amazon S3</option>
|
||||||
|
<option value="backblaze">Backblaze B2</option>
|
||||||
|
<option value="cloudflare-r2">Cloudflare R2</option>
|
||||||
|
<option value="minio">MinIO</option>
|
||||||
|
<option value="digitalocean">DigitalOcean Spaces</option>
|
||||||
|
<option value="wasabi">Wasabi</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Endpoint URL -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label><span data-i18n="admin.storage_endpoint_url">Endpoint URL</span>
|
||||||
|
<span id="badge-s3_endpoint_url"></span></label>
|
||||||
|
<input type="url" id="storage-endpoint-url" placeholder="https://s3.amazonaws.com" />
|
||||||
|
<small data-i18n="admin.storage_endpoint_hint">Leave empty for Amazon S3 default</small>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Bucket -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label><span data-i18n="admin.storage_bucket">Bucket</span>
|
||||||
|
<span id="badge-s3_bucket"></span></label>
|
||||||
|
<input type="text" id="storage-bucket" placeholder="my-oxicloud-bucket" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Region -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label><span data-i18n="admin.storage_region">Region</span>
|
||||||
|
<span id="badge-s3_region"></span></label>
|
||||||
|
<input type="text" id="storage-region" placeholder="us-east-1" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Access Key -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label><span data-i18n="admin.storage_access_key">Access Key ID</span>
|
||||||
|
<span id="badge-s3_access_key"></span></label>
|
||||||
|
<input type="text" id="storage-access-key" placeholder="AKIA..." />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Secret Key -->
|
||||||
|
<div class="form-group">
|
||||||
|
<label><span data-i18n="admin.storage_secret_key">Secret Access Key</span>
|
||||||
|
<span id="badge-s3_secret_key"></span></label>
|
||||||
|
<input type="password" id="storage-secret-key" placeholder="Leave empty to keep current value" />
|
||||||
|
<small id="storage-secret-hint" style="display:none">
|
||||||
|
<i class="fas fa-check-circle secret-icon"></i>
|
||||||
|
<span data-i18n="admin.storage_secret_configured">A secret key is already configured</span>
|
||||||
|
</small>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Force Path Style -->
|
||||||
|
<div class="toggle-row">
|
||||||
|
<label data-i18n="admin.storage_path_style">Force Path Style</label>
|
||||||
|
<label class="switch">
|
||||||
|
<input type="checkbox" id="storage-path-style" /><span class="slider"></span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<small data-i18n="admin.storage_path_style_hint">Required for MinIO and some S3-compatible providers</small>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Action buttons -->
|
||||||
|
<div class="storage-actions">
|
||||||
|
<button class="btn btn-secondary" id="btn-test-storage">
|
||||||
|
<i class="fas fa-vial"></i> <span data-i18n="admin.storage_test_connection">Test Connection</span>
|
||||||
|
</button>
|
||||||
|
<button class="btn btn-primary" id="btn-save-storage">
|
||||||
|
<i class="fas fa-save"></i> <span data-i18n="admin.storage_save">Save</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div id="storage-status" class="alert"></div>
|
||||||
|
|
||||||
|
<!-- Migration section -->
|
||||||
|
<div class="storage-migration-section">
|
||||||
|
<h3><i class="fas fa-exchange-alt"></i> <span data-i18n="admin.storage_migration">Backend Migration</span></h3>
|
||||||
|
|
||||||
|
<!-- Status badge -->
|
||||||
|
<div class="migration-status-row">
|
||||||
|
<span data-i18n="admin.migration_status_label">Status:</span>
|
||||||
|
<span id="migration-status-badge" class="badge badge-migration badge-migration--idle">Idle</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Progress bar (hidden when idle) -->
|
||||||
|
<div id="migration-progress-section" style="display:none">
|
||||||
|
<div class="migration-progress-bar">
|
||||||
|
<div class="migration-progress-bar__fill" id="migration-progress-fill" style="width:0%"></div>
|
||||||
|
</div>
|
||||||
|
<div class="migration-progress-text">
|
||||||
|
<span id="migration-progress-label">0 / 0 blobs (0%)</span>
|
||||||
|
<span id="migration-throughput"></span>
|
||||||
|
</div>
|
||||||
|
<div class="migration-progress-text">
|
||||||
|
<span id="migration-bytes-label"></span>
|
||||||
|
<span id="migration-eta"></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Failed blobs (expandable) -->
|
||||||
|
<details id="migration-failed-section" style="display:none">
|
||||||
|
<summary><i class="fas fa-exclamation-triangle"></i> <span id="migration-failed-count">0</span> <span data-i18n="admin.migration_failed_blobs">failed blobs</span></summary>
|
||||||
|
<pre id="migration-failed-list" class="migration-failed-list"></pre>
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<!-- Action buttons -->
|
||||||
|
<div class="migration-actions">
|
||||||
|
<button class="btn btn-primary" id="btn-start-migration">
|
||||||
|
<i class="fas fa-play"></i> <span data-i18n="admin.migration_start">Start Migration</span>
|
||||||
|
</button>
|
||||||
|
<button class="btn btn-secondary" id="btn-pause-migration" style="display:none">
|
||||||
|
<i class="fas fa-pause"></i> <span data-i18n="admin.migration_pause">Pause</span>
|
||||||
|
</button>
|
||||||
|
<button class="btn btn-primary" id="btn-resume-migration" style="display:none">
|
||||||
|
<i class="fas fa-play"></i> <span data-i18n="admin.migration_resume">Resume</span>
|
||||||
|
</button>
|
||||||
|
<button class="btn btn-secondary" id="btn-verify-migration" style="display:none">
|
||||||
|
<i class="fas fa-check-double"></i> <span data-i18n="admin.migration_verify">Verify Integrity</span>
|
||||||
|
</button>
|
||||||
|
<button class="btn btn-primary" id="btn-complete-migration" style="display:none">
|
||||||
|
<i class="fas fa-flag-checkered"></i> <span data-i18n="admin.migration_complete">Finalize</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div id="migration-verify-result" style="display:none"></div>
|
||||||
|
<div id="migration-status-msg" class="alert" style="display:none"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
@@ -324,6 +324,7 @@ body {
|
|||||||
#oidc-form,
|
#oidc-form,
|
||||||
#secret-hint,
|
#secret-hint,
|
||||||
#password-warning,
|
#password-warning,
|
||||||
|
#storage-secret-hint,
|
||||||
#quota-modal,
|
#quota-modal,
|
||||||
#create-user-modal,
|
#create-user-modal,
|
||||||
#reset-pw-modal {
|
#reset-pw-modal {
|
||||||
@@ -938,3 +939,176 @@ details[open] summary {
|
|||||||
.btn-danger:hover {
|
.btn-danger:hover {
|
||||||
opacity: 0.88;
|
opacity: 0.88;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── Storage Tab ── */
|
||||||
|
|
||||||
|
.storage-status-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
|
||||||
|
gap: 12px;
|
||||||
|
margin-bottom: 24px;
|
||||||
|
}
|
||||||
|
.storage-stat {
|
||||||
|
padding: 14px;
|
||||||
|
background: var(--color-bg-hover);
|
||||||
|
border-radius: 12px;
|
||||||
|
text-align: center;
|
||||||
|
border: 1px solid var(--color-border);
|
||||||
|
}
|
||||||
|
.storage-stat__label {
|
||||||
|
display: block;
|
||||||
|
font-size: 11px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.5px;
|
||||||
|
color: var(--color-text-secondary);
|
||||||
|
margin-bottom: 4px;
|
||||||
|
}
|
||||||
|
.storage-stat__value {
|
||||||
|
display: block;
|
||||||
|
font-size: 1.35rem;
|
||||||
|
font-weight: 700;
|
||||||
|
color: var(--color-text-heading);
|
||||||
|
}
|
||||||
|
|
||||||
|
.storage-backend-selector {
|
||||||
|
display: flex;
|
||||||
|
gap: 12px;
|
||||||
|
margin-bottom: 20px;
|
||||||
|
}
|
||||||
|
.storage-backend-option {
|
||||||
|
flex: 1;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.storage-backend-option input[type="radio"] {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
.storage-backend-option__card {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
padding: 18px 14px;
|
||||||
|
border: 2px solid var(--color-border);
|
||||||
|
border-radius: 12px;
|
||||||
|
transition: border-color 0.2s, background 0.2s;
|
||||||
|
text-align: center;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
.storage-backend-option__card i {
|
||||||
|
font-size: 1.5rem;
|
||||||
|
color: var(--color-text-secondary);
|
||||||
|
transition: color 0.2s;
|
||||||
|
}
|
||||||
|
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card {
|
||||||
|
border-color: var(--color-accent);
|
||||||
|
background: var(--color-accent-subtle);
|
||||||
|
}
|
||||||
|
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card i {
|
||||||
|
color: var(--color-accent);
|
||||||
|
}
|
||||||
|
|
||||||
|
.storage-form {
|
||||||
|
margin-top: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.storage-actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 10px;
|
||||||
|
margin-top: 24px;
|
||||||
|
justify-content: flex-end;
|
||||||
|
}
|
||||||
|
|
||||||
|
.storage-migration-section {
|
||||||
|
margin-top: 32px;
|
||||||
|
padding-top: 20px;
|
||||||
|
border-top: 1px solid var(--color-border);
|
||||||
|
}
|
||||||
|
.storage-migration-section h3 {
|
||||||
|
font-size: 1rem;
|
||||||
|
margin-bottom: 8px;
|
||||||
|
color: var(--color-text-heading);
|
||||||
|
}
|
||||||
|
.storage-migration-section h3 i {
|
||||||
|
color: var(--color-text-secondary);
|
||||||
|
margin-right: 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Migration UI ── */
|
||||||
|
|
||||||
|
.migration-status-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
margin-bottom: 14px;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge-migration {
|
||||||
|
padding: 3px 10px;
|
||||||
|
border-radius: 10px;
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 600;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.4px;
|
||||||
|
}
|
||||||
|
.badge-migration--idle {
|
||||||
|
background: var(--color-bg-hover);
|
||||||
|
color: var(--color-text-secondary);
|
||||||
|
}
|
||||||
|
.badge-migration--running {
|
||||||
|
background: var(--color-accent-subtle);
|
||||||
|
color: var(--color-accent);
|
||||||
|
}
|
||||||
|
.badge-migration--paused {
|
||||||
|
background: var(--color-warning-bg, #fff8e1);
|
||||||
|
color: var(--color-warning-text, #b28704);
|
||||||
|
}
|
||||||
|
.badge-migration--completed {
|
||||||
|
background: var(--color-success-bg, #e8f5e9);
|
||||||
|
color: var(--color-success-text, #2e7d32);
|
||||||
|
}
|
||||||
|
.badge-migration--failed {
|
||||||
|
background: var(--color-danger-bg, #fce4ec);
|
||||||
|
color: var(--color-danger-text, #c62828);
|
||||||
|
}
|
||||||
|
|
||||||
|
.migration-progress-bar {
|
||||||
|
width: 100%;
|
||||||
|
height: 10px;
|
||||||
|
background: var(--color-bg-hover);
|
||||||
|
border-radius: 6px;
|
||||||
|
overflow: hidden;
|
||||||
|
margin-bottom: 6px;
|
||||||
|
}
|
||||||
|
.migration-progress-bar__fill {
|
||||||
|
height: 100%;
|
||||||
|
background: var(--color-accent);
|
||||||
|
border-radius: 6px;
|
||||||
|
transition: width 0.3s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
.migration-progress-text {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
font-size: 12px;
|
||||||
|
color: var(--color-text-secondary);
|
||||||
|
margin-bottom: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.migration-failed-list {
|
||||||
|
max-height: 150px;
|
||||||
|
overflow-y: auto;
|
||||||
|
font-size: 11px;
|
||||||
|
background: var(--color-bg-hover);
|
||||||
|
padding: 8px 12px;
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid var(--color-border);
|
||||||
|
margin-top: 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.migration-actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 10px;
|
||||||
|
margin-top: 16px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|||||||
@@ -138,6 +138,7 @@ function switchTab(name, el) {
|
|||||||
activeTabName = name;
|
activeTabName = name;
|
||||||
if (name === 'users') loadUsers();
|
if (name === 'users') loadUsers();
|
||||||
if (name === 'dashboard') loadDashboard();
|
if (name === 'dashboard') loadDashboard();
|
||||||
|
if (name === 'storage') loadStorage();
|
||||||
}
|
}
|
||||||
|
|
||||||
async function loadDashboard() {
|
async function loadDashboard() {
|
||||||
@@ -693,6 +694,353 @@ async function saveOidcSettings() {
|
|||||||
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.save_btn'))}`;
|
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.save_btn'))}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── Storage tab ── */
|
||||||
|
|
||||||
|
const STORAGE_PRESETS = {
|
||||||
|
'custom': { endpoint: '', region: '', pathStyle: false },
|
||||||
|
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
|
||||||
|
'backblaze': { endpoint: 'https://s3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
|
||||||
|
'cloudflare-r2': { endpoint: 'https://{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
|
||||||
|
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
|
||||||
|
'digitalocean': { endpoint: 'https://{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
|
||||||
|
'wasabi': { endpoint: 'https://s3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
|
||||||
|
};
|
||||||
|
|
||||||
|
function toggleS3Form(visible) {
|
||||||
|
if (visible) showElement('storage-s3-form');
|
||||||
|
else hideElement('storage-s3-form');
|
||||||
|
}
|
||||||
|
|
||||||
|
function onStoragePresetChange() {
|
||||||
|
const preset = document.getElementById('storage-preset').value;
|
||||||
|
const p = STORAGE_PRESETS[preset];
|
||||||
|
if (!p) return;
|
||||||
|
if (p.endpoint) document.getElementById('storage-endpoint-url').value = p.endpoint;
|
||||||
|
if (p.region) document.getElementById('storage-region').value = p.region;
|
||||||
|
document.getElementById('storage-path-style').checked = p.pathStyle;
|
||||||
|
}
|
||||||
|
|
||||||
|
function showStorageStatus(msg, type) {
|
||||||
|
const el = document.getElementById('storage-status');
|
||||||
|
el.textContent = msg;
|
||||||
|
el.className = `alert alert-${type}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadStorage() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/settings/storage`, {
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin'
|
||||||
|
});
|
||||||
|
if (!resp.ok) return;
|
||||||
|
const s = await resp.json();
|
||||||
|
|
||||||
|
// Backend selector
|
||||||
|
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
|
||||||
|
r.checked = r.value === s.backend;
|
||||||
|
});
|
||||||
|
toggleS3Form(s.backend === 's3');
|
||||||
|
|
||||||
|
// S3 fields
|
||||||
|
document.getElementById('storage-endpoint-url').value = s.s3_endpoint_url || '';
|
||||||
|
document.getElementById('storage-bucket').value = s.s3_bucket || '';
|
||||||
|
document.getElementById('storage-region').value = s.s3_region || '';
|
||||||
|
document.getElementById('storage-access-key').value = '';
|
||||||
|
document.getElementById('storage-secret-key').value = '';
|
||||||
|
document.getElementById('storage-path-style').checked = s.s3_force_path_style;
|
||||||
|
|
||||||
|
// Secret hints
|
||||||
|
if (s.s3_access_key_set) {
|
||||||
|
document.getElementById('storage-access-key').placeholder = t('admin.storage_key_placeholder') || 'Leave empty to keep current value';
|
||||||
|
}
|
||||||
|
if (s.s3_secret_key_set) {
|
||||||
|
showElement('storage-secret-hint');
|
||||||
|
} else {
|
||||||
|
hideElement('storage-secret-hint');
|
||||||
|
}
|
||||||
|
|
||||||
|
// ENV badges
|
||||||
|
(s.env_overrides || []).forEach((field) => {
|
||||||
|
const badge = document.getElementById(`badge-${field}`);
|
||||||
|
if (badge) badge.innerHTML = '<span class="badge badge-env">ENV</span>';
|
||||||
|
});
|
||||||
|
|
||||||
|
// Status section
|
||||||
|
document.getElementById('storage-current-backend').textContent = s.current_backend || '—';
|
||||||
|
document.getElementById('storage-total-blobs').textContent = s.total_blobs != null ? s.total_blobs.toLocaleString() : '—';
|
||||||
|
document.getElementById('storage-total-size').textContent = s.total_bytes_stored != null ? formatBytes(s.total_bytes_stored) : '—';
|
||||||
|
document.getElementById('storage-dedup-ratio').textContent = s.dedup_ratio != null ? `${s.dedup_ratio.toFixed(2)}x` : '—';
|
||||||
|
} catch (e) {
|
||||||
|
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also load migration status
|
||||||
|
loadMigrationStatus();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveStorageSettings() {
|
||||||
|
const btn = document.getElementById('btn-save-storage');
|
||||||
|
btn.disabled = true;
|
||||||
|
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.saving'))}`;
|
||||||
|
|
||||||
|
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
|
||||||
|
const body = {
|
||||||
|
backend,
|
||||||
|
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
|
||||||
|
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
|
||||||
|
s3_region: document.getElementById('storage-region').value.trim() || null,
|
||||||
|
s3_access_key: document.getElementById('storage-access-key').value || null,
|
||||||
|
s3_secret_key: document.getElementById('storage-secret-key').value || null,
|
||||||
|
s3_force_path_style: document.getElementById('storage-path-style').checked
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/settings/storage`, {
|
||||||
|
method: 'PUT',
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin',
|
||||||
|
body: JSON.stringify(body)
|
||||||
|
});
|
||||||
|
if (resp.ok) {
|
||||||
|
showStorageStatus(t('admin.storage_saved') || 'Storage settings saved successfully', 'success');
|
||||||
|
loadStorage();
|
||||||
|
} else {
|
||||||
|
const e = await resp.json().catch(() => ({}));
|
||||||
|
showStorageStatus(`Error: ${e.message || resp.statusText}`, 'error');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||||
|
}
|
||||||
|
btn.disabled = false;
|
||||||
|
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.storage_save') || 'Save')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testStorageConnection() {
|
||||||
|
const btn = document.getElementById('btn-test-storage');
|
||||||
|
btn.disabled = true;
|
||||||
|
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.testing') || 'Testing...')}`;
|
||||||
|
|
||||||
|
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
|
||||||
|
const body = {
|
||||||
|
backend,
|
||||||
|
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
|
||||||
|
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
|
||||||
|
s3_region: document.getElementById('storage-region').value.trim() || null,
|
||||||
|
s3_access_key: document.getElementById('storage-access-key').value || null,
|
||||||
|
s3_secret_key: document.getElementById('storage-secret-key').value || null,
|
||||||
|
s3_force_path_style: document.getElementById('storage-path-style').checked
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/settings/storage/test`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin',
|
||||||
|
body: JSON.stringify(body)
|
||||||
|
});
|
||||||
|
const r = await resp.json();
|
||||||
|
if (r.connected) {
|
||||||
|
let msg = `${t('admin.storage_test_success') || 'Connection successful'} (${escapeHtml(r.backend_type)})`;
|
||||||
|
if (r.available_bytes != null) msg += ` — ${formatBytes(r.available_bytes)} available`;
|
||||||
|
showStorageStatus(msg, 'success');
|
||||||
|
} else {
|
||||||
|
showStorageStatus(`${t('admin.storage_test_failure') || 'Connection failed'}: ${escapeHtml(r.message)}`, 'error');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
|
||||||
|
}
|
||||||
|
btn.disabled = false;
|
||||||
|
btn.innerHTML = `<i class="fas fa-vial"></i> ${escapeHtml(t('admin.storage_test_connection') || 'Test Connection')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Migration ── */
|
||||||
|
|
||||||
|
let migrationPollTimer = null;
|
||||||
|
|
||||||
|
function showMigrationMsg(msg, type) {
|
||||||
|
const el = document.getElementById('migration-status-msg');
|
||||||
|
el.textContent = msg;
|
||||||
|
el.className = `alert alert-${type}`;
|
||||||
|
el.style.display = '';
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateMigrationUI(m) {
|
||||||
|
// Status badge
|
||||||
|
const badge = document.getElementById('migration-status-badge');
|
||||||
|
badge.textContent = (m.status || 'idle').charAt(0).toUpperCase() + (m.status || 'idle').slice(1);
|
||||||
|
badge.className = `badge badge-migration badge-migration--${m.status || 'idle'}`;
|
||||||
|
|
||||||
|
const isActive = m.status === 'running' || m.status === 'paused';
|
||||||
|
const isCompleted = m.status === 'completed';
|
||||||
|
|
||||||
|
// Progress section
|
||||||
|
const progressSection = document.getElementById('migration-progress-section');
|
||||||
|
progressSection.style.display = (isActive || isCompleted) ? '' : 'none';
|
||||||
|
|
||||||
|
if (m.total_blobs > 0) {
|
||||||
|
const pct = Math.round((m.migrated_blobs / m.total_blobs) * 100);
|
||||||
|
document.getElementById('migration-progress-fill').style.width = `${pct}%`;
|
||||||
|
document.getElementById('migration-progress-label').textContent =
|
||||||
|
`${m.migrated_blobs.toLocaleString()} / ${m.total_blobs.toLocaleString()} blobs (${pct}%)`;
|
||||||
|
document.getElementById('migration-bytes-label').textContent =
|
||||||
|
`${formatBytes(m.migrated_bytes)} transferred`;
|
||||||
|
|
||||||
|
if (m.throughput_bytes_per_sec && m.status === 'running') {
|
||||||
|
document.getElementById('migration-throughput').textContent =
|
||||||
|
`${formatBytes(Math.round(m.throughput_bytes_per_sec))}/s`;
|
||||||
|
const remaining = m.total_blobs - m.migrated_blobs;
|
||||||
|
if (remaining > 0 && m.throughput_bytes_per_sec > 0) {
|
||||||
|
const avgBlobSize = m.migrated_bytes / Math.max(m.migrated_blobs, 1);
|
||||||
|
const etaSecs = Math.round((remaining * avgBlobSize) / m.throughput_bytes_per_sec);
|
||||||
|
const etaMin = Math.ceil(etaSecs / 60);
|
||||||
|
document.getElementById('migration-eta').textContent =
|
||||||
|
`~${etaMin} min remaining`;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
document.getElementById('migration-throughput').textContent = '';
|
||||||
|
document.getElementById('migration-eta').textContent = '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Failed blobs section
|
||||||
|
const failedSection = document.getElementById('migration-failed-section');
|
||||||
|
if (m.failed_blobs && m.failed_blobs.length > 0) {
|
||||||
|
failedSection.style.display = '';
|
||||||
|
document.getElementById('migration-failed-count').textContent = m.failed_blobs.length;
|
||||||
|
document.getElementById('migration-failed-list').textContent = m.failed_blobs.join('\n');
|
||||||
|
} else {
|
||||||
|
failedSection.style.display = 'none';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Button visibility
|
||||||
|
document.getElementById('btn-start-migration').style.display =
|
||||||
|
(!isActive && !isCompleted) ? '' : 'none';
|
||||||
|
document.getElementById('btn-pause-migration').style.display =
|
||||||
|
m.status === 'running' ? '' : 'none';
|
||||||
|
document.getElementById('btn-resume-migration').style.display =
|
||||||
|
m.status === 'paused' ? '' : 'none';
|
||||||
|
document.getElementById('btn-verify-migration').style.display =
|
||||||
|
isCompleted ? '' : 'none';
|
||||||
|
document.getElementById('btn-complete-migration').style.display =
|
||||||
|
isCompleted ? '' : 'none';
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadMigrationStatus() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration`, {
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin'
|
||||||
|
});
|
||||||
|
if (!resp.ok) return;
|
||||||
|
const m = await resp.json();
|
||||||
|
updateMigrationUI(m);
|
||||||
|
|
||||||
|
// Auto-poll while running
|
||||||
|
if (m.status === 'running') {
|
||||||
|
if (!migrationPollTimer) {
|
||||||
|
migrationPollTimer = setInterval(loadMigrationStatus, 2000);
|
||||||
|
}
|
||||||
|
} else if (migrationPollTimer) {
|
||||||
|
clearInterval(migrationPollTimer);
|
||||||
|
migrationPollTimer = null;
|
||||||
|
}
|
||||||
|
} catch (_e) { /* ignore */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
async function startMigration() {
|
||||||
|
const btn = document.getElementById('btn-start-migration');
|
||||||
|
btn.disabled = true;
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration/start`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin',
|
||||||
|
body: JSON.stringify({ concurrency: 4 })
|
||||||
|
});
|
||||||
|
if (resp.ok) {
|
||||||
|
showMigrationMsg(t('admin.migration_started') || 'Migration started', 'success');
|
||||||
|
loadMigrationStatus();
|
||||||
|
} else {
|
||||||
|
const e = await resp.json().catch(() => ({}));
|
||||||
|
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
|
||||||
|
}
|
||||||
|
btn.disabled = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function pauseMigration() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration/pause`, {
|
||||||
|
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||||
|
});
|
||||||
|
if (resp.ok) {
|
||||||
|
showMigrationMsg(t('admin.migration_paused_msg') || 'Migration paused', 'success');
|
||||||
|
loadMigrationStatus();
|
||||||
|
}
|
||||||
|
} catch (_e) { /* ignore */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
async function resumeMigration() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration/resume`, {
|
||||||
|
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||||
|
});
|
||||||
|
if (resp.ok) {
|
||||||
|
showMigrationMsg(t('admin.migration_resumed_msg') || 'Migration resumed', 'success');
|
||||||
|
loadMigrationStatus();
|
||||||
|
}
|
||||||
|
} catch (_e) { /* ignore */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
async function verifyMigration() {
|
||||||
|
const btn = document.getElementById('btn-verify-migration');
|
||||||
|
btn.disabled = true;
|
||||||
|
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.migration_verifying') || 'Verifying...')}`;
|
||||||
|
const resultDiv = document.getElementById('migration-verify-result');
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration/verify`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: headers(),
|
||||||
|
credentials: 'same-origin',
|
||||||
|
body: JSON.stringify({ sample_size: 100 })
|
||||||
|
});
|
||||||
|
const r = await resp.json();
|
||||||
|
resultDiv.style.display = '';
|
||||||
|
if (r.passed) {
|
||||||
|
resultDiv.innerHTML = `<div class="discovery-result ok"><strong><i class="fas fa-check-circle"></i> ${escapeHtml(t('admin.migration_verify_passed') || 'Verification passed')}</strong><p>${r.sample_checked} blobs checked, ${r.pg_blob_count} total in database</p></div>`;
|
||||||
|
} else {
|
||||||
|
const issues = [];
|
||||||
|
if (r.missing_in_target.length) issues.push(`${r.missing_in_target.length} missing`);
|
||||||
|
if (r.size_mismatches.length) issues.push(`${r.size_mismatches.length} size mismatches`);
|
||||||
|
resultDiv.innerHTML = `<div class="discovery-result fail"><strong><i class="fas fa-times-circle"></i> ${escapeHtml(t('admin.migration_verify_failed') || 'Verification failed')}</strong><p>${issues.join(', ')}</p></div>`;
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
resultDiv.style.display = '';
|
||||||
|
resultDiv.innerHTML = `<div class="discovery-result fail"><i class="fas fa-times-circle"></i> Error: ${escapeHtml(e.message)}</div>`;
|
||||||
|
}
|
||||||
|
btn.disabled = false;
|
||||||
|
btn.innerHTML = `<i class="fas fa-check-double"></i> ${escapeHtml(t('admin.migration_verify') || 'Verify Integrity')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function completeMigration() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch(`${API}/admin/storage/migration/complete`, {
|
||||||
|
method: 'POST', headers: headers(), credentials: 'same-origin'
|
||||||
|
});
|
||||||
|
if (resp.ok) {
|
||||||
|
showMigrationMsg(t('admin.migration_completed_msg') || 'Migration finalized. Restart the server to use the new backend.', 'success');
|
||||||
|
loadMigrationStatus();
|
||||||
|
} else {
|
||||||
|
const e = await resp.json().catch(() => ({}));
|
||||||
|
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function init() {
|
async function init() {
|
||||||
try {
|
try {
|
||||||
const me = await fetch(`${API}/auth/me`, {
|
const me = await fetch(`${API}/auth/me`, {
|
||||||
@@ -775,6 +1123,9 @@ document.getElementById('tab-btn-users').addEventListener('click', function () {
|
|||||||
document.getElementById('tab-btn-oidc').addEventListener('click', function () {
|
document.getElementById('tab-btn-oidc').addEventListener('click', function () {
|
||||||
switchTab('oidc', this);
|
switchTab('oidc', this);
|
||||||
});
|
});
|
||||||
|
document.getElementById('tab-btn-storage').addEventListener('click', function () {
|
||||||
|
switchTab('storage', this);
|
||||||
|
});
|
||||||
|
|
||||||
document.getElementById('ds-registration').addEventListener('change', function () {
|
document.getElementById('ds-registration').addEventListener('change', function () {
|
||||||
toggleRegistration(this.checked);
|
toggleRegistration(this.checked);
|
||||||
@@ -797,3 +1148,20 @@ document.getElementById('cu-submit').addEventListener('click', submitCreateUser)
|
|||||||
|
|
||||||
document.getElementById('btn-close-reset-pw').addEventListener('click', closeResetPasswordModal);
|
document.getElementById('btn-close-reset-pw').addEventListener('click', closeResetPasswordModal);
|
||||||
document.getElementById('rp-submit').addEventListener('click', submitResetPassword);
|
document.getElementById('rp-submit').addEventListener('click', submitResetPassword);
|
||||||
|
|
||||||
|
/* ── Storage event listeners ── */
|
||||||
|
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
|
||||||
|
r.addEventListener('change', function () {
|
||||||
|
toggleS3Form(this.value === 's3');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
document.getElementById('storage-preset').addEventListener('change', onStoragePresetChange);
|
||||||
|
document.getElementById('btn-test-storage').addEventListener('click', testStorageConnection);
|
||||||
|
document.getElementById('btn-save-storage').addEventListener('click', saveStorageSettings);
|
||||||
|
|
||||||
|
/* ── Migration event listeners ── */
|
||||||
|
document.getElementById('btn-start-migration').addEventListener('click', startMigration);
|
||||||
|
document.getElementById('btn-pause-migration').addEventListener('click', pauseMigration);
|
||||||
|
document.getElementById('btn-resume-migration').addEventListener('click', resumeMigration);
|
||||||
|
document.getElementById('btn-verify-migration').addEventListener('click', verifyMigration);
|
||||||
|
document.getElementById('btn-complete-migration').addEventListener('click', completeMigration);
|
||||||
|
|||||||
+44
-1
@@ -566,7 +566,50 @@
|
|||||||
"error_password_short": "Password must be at least 8 characters",
|
"error_password_short": "Password must be at least 8 characters",
|
||||||
"error_generic": "Failed",
|
"error_generic": "Failed",
|
||||||
"error_network": "Network error: {{message}}",
|
"error_network": "Network error: {{message}}",
|
||||||
"error_create_user": "Failed to create user"
|
"error_create_user": "Failed to create user",
|
||||||
|
"tab_storage": "Storage",
|
||||||
|
"storage_title": "Storage Backend",
|
||||||
|
"storage_current_backend": "Active Backend",
|
||||||
|
"storage_total_blobs": "Total Blobs",
|
||||||
|
"storage_total_size": "Total Size",
|
||||||
|
"storage_dedup_ratio": "Dedup Ratio",
|
||||||
|
"storage_backend": "Backend Type",
|
||||||
|
"storage_local": "Local Filesystem",
|
||||||
|
"storage_s3": "S3-Compatible",
|
||||||
|
"storage_provider_preset": "Provider Preset",
|
||||||
|
"storage_preset_custom": "Custom",
|
||||||
|
"storage_endpoint_url": "Endpoint URL",
|
||||||
|
"storage_endpoint_hint": "Leave empty for Amazon S3 default",
|
||||||
|
"storage_bucket": "Bucket",
|
||||||
|
"storage_region": "Region",
|
||||||
|
"storage_access_key": "Access Key ID",
|
||||||
|
"storage_secret_key": "Secret Access Key",
|
||||||
|
"storage_secret_configured": "A secret key is already configured",
|
||||||
|
"storage_key_placeholder": "Leave empty to keep current value",
|
||||||
|
"storage_path_style": "Force Path Style",
|
||||||
|
"storage_path_style_hint": "Required for MinIO and some S3-compatible providers",
|
||||||
|
"storage_test_connection": "Test Connection",
|
||||||
|
"storage_test_success": "Connection successful",
|
||||||
|
"storage_test_failure": "Connection failed",
|
||||||
|
"storage_save": "Save",
|
||||||
|
"storage_saved": "Storage settings saved successfully",
|
||||||
|
"storage_migration": "Backend Migration",
|
||||||
|
"storage_migration_coming_soon": "Backend migration will be available in a future update.",
|
||||||
|
"migration_status_label": "Status:",
|
||||||
|
"migration_start": "Start Migration",
|
||||||
|
"migration_pause": "Pause",
|
||||||
|
"migration_resume": "Resume",
|
||||||
|
"migration_verify": "Verify Integrity",
|
||||||
|
"migration_complete": "Finalize",
|
||||||
|
"migration_started": "Migration started",
|
||||||
|
"migration_paused_msg": "Migration paused",
|
||||||
|
"migration_resumed_msg": "Migration resumed",
|
||||||
|
"migration_completed_msg": "Migration finalized. Restart the server to use the new backend.",
|
||||||
|
"migration_verifying": "Verifying…",
|
||||||
|
"migration_verify_passed": "Verification passed",
|
||||||
|
"migration_verify_failed": "Verification failed",
|
||||||
|
"migration_failed_blobs": "failed blobs",
|
||||||
|
"testing": "Testing…"
|
||||||
},
|
},
|
||||||
"profile": {
|
"profile": {
|
||||||
"page_title": "Profile",
|
"page_title": "Profile",
|
||||||
|
|||||||
+44
-1
@@ -566,7 +566,50 @@
|
|||||||
"error_password_short": "La contraseña debe tener al menos 8 caracteres",
|
"error_password_short": "La contraseña debe tener al menos 8 caracteres",
|
||||||
"error_generic": "Error",
|
"error_generic": "Error",
|
||||||
"error_network": "Error de red: {{message}}",
|
"error_network": "Error de red: {{message}}",
|
||||||
"error_create_user": "Error al crear usuario"
|
"error_create_user": "Error al crear usuario",
|
||||||
|
"tab_storage": "Almacenamiento",
|
||||||
|
"storage_title": "Backend de Almacenamiento",
|
||||||
|
"storage_current_backend": "Backend Activo",
|
||||||
|
"storage_total_blobs": "Total de Blobs",
|
||||||
|
"storage_total_size": "Tamaño Total",
|
||||||
|
"storage_dedup_ratio": "Ratio de Dedup",
|
||||||
|
"storage_backend": "Tipo de Backend",
|
||||||
|
"storage_local": "Sistema de Archivos Local",
|
||||||
|
"storage_s3": "Compatible con S3",
|
||||||
|
"storage_provider_preset": "Proveedor Preconfigurado",
|
||||||
|
"storage_preset_custom": "Personalizado",
|
||||||
|
"storage_endpoint_url": "URL del Endpoint",
|
||||||
|
"storage_endpoint_hint": "Dejar vacío para usar Amazon S3 por defecto",
|
||||||
|
"storage_bucket": "Bucket",
|
||||||
|
"storage_region": "Región",
|
||||||
|
"storage_access_key": "Access Key ID",
|
||||||
|
"storage_secret_key": "Secret Access Key",
|
||||||
|
"storage_secret_configured": "Ya hay una clave secreta configurada",
|
||||||
|
"storage_key_placeholder": "Dejar vacío para mantener el valor actual",
|
||||||
|
"storage_path_style": "Forzar Path Style",
|
||||||
|
"storage_path_style_hint": "Requerido para MinIO y algunos proveedores compatibles con S3",
|
||||||
|
"storage_test_connection": "Probar Conexión",
|
||||||
|
"storage_test_success": "Conexión exitosa",
|
||||||
|
"storage_test_failure": "Conexión fallida",
|
||||||
|
"storage_save": "Guardar",
|
||||||
|
"storage_saved": "Configuración de almacenamiento guardada correctamente",
|
||||||
|
"storage_migration": "Migración de Backend",
|
||||||
|
"storage_migration_coming_soon": "La migración de backend estará disponible en una futura actualización.",
|
||||||
|
"migration_status_label": "Estado:",
|
||||||
|
"migration_start": "Iniciar Migración",
|
||||||
|
"migration_pause": "Pausar",
|
||||||
|
"migration_resume": "Reanudar",
|
||||||
|
"migration_verify": "Verificar Integridad",
|
||||||
|
"migration_complete": "Finalizar",
|
||||||
|
"migration_started": "Migración iniciada",
|
||||||
|
"migration_paused_msg": "Migración pausada",
|
||||||
|
"migration_resumed_msg": "Migración reanudada",
|
||||||
|
"migration_completed_msg": "Migración finalizada. Reinicia el servidor para usar el nuevo backend.",
|
||||||
|
"migration_verifying": "Verificando…",
|
||||||
|
"migration_verify_passed": "Verificación exitosa",
|
||||||
|
"migration_verify_failed": "Verificación fallida",
|
||||||
|
"migration_failed_blobs": "blobs fallidos",
|
||||||
|
"testing": "Probando…"
|
||||||
},
|
},
|
||||||
"profile": {
|
"profile": {
|
||||||
"page_title": "Perfil",
|
"page_title": "Perfil",
|
||||||
|
|||||||
Reference in New Issue
Block a user