feat: pluggable storage backends (S3, Azure, local) with admin UI

Implement 4-phase external storage backends architecture:

Phase 1 - Foundation:
- BlobStorageBackend trait (application/ports/blob_storage_ports.rs)
- LocalBlobBackend: extracted all tokio::fs ops from DedupService
- S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2)
- DedupService refactored to use Arc<dyn BlobStorageBackend>

Phase 2 - Admin Panel:
- StorageSettingsService with DB persistence + env override
- Storage tab in admin panel (backend selector, S3 form, provider presets)
- GET/PUT/POST endpoints for storage settings + connection test
- i18n keys (en/es) and BEM CSS

Phase 3 - Migration:
- MigrationBlobBackend decorator (dual-read: target-first + source fallback)
- Background migration job with parallel transfers + progress tracking
- Migration UI (progress bar, ETA, pause/resume/verify/complete)
- 6 admin API endpoints for migration lifecycle

Phase 4 - Enterprise Extras:
- CachedBlobBackend: LRU disk cache for remote backends
- EncryptedBlobBackend: AES-256-GCM at-rest encryption
- AzureBlobBackend: Azure Blob Storage support
- RetryBlobBackend: exponential backoff for transient errors
- Decorator composition in DI: retry → encryption → cache

All 223 tests passing, clippy clean, fmt verified.
This commit is contained in:
Diocrafts
2026-04-14 21:33:38 +02:00
parent 6fc632af7e
commit cd3733b459
26 changed files with 6870 additions and 308 deletions
@@ -0,0 +1,88 @@
//! Blob Storage Backend Port — abstracts raw byte I/O for content-addressable storage.
//!
//! This trait decouples `DedupService` from any specific storage medium.
//! Implementations include:
//! - `LocalBlobBackend` — local filesystem (default)
//! - `S3BlobBackend` — any S3-compatible service (AWS, Backblaze B2, MinIO, R2…)
//!
//! `DedupService` owns an `Arc<dyn BlobStorageBackend>` and delegates all
//! byte-level I/O through this trait, keeping BLAKE3 hashing, ref-counting
//! and PostgreSQL index logic in `DedupService` itself.
use bytes::Bytes;
use futures::Stream;
use serde::Serialize;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use crate::domain::errors::DomainError;
/// Boxed future alias used by [`BlobStorageBackend`] to keep the trait dyn-compatible.
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
/// Pinned boxed byte stream — the return type for blob reads.
pub type BlobStream = Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>;
/// Health-check result returned by [`BlobStorageBackend::health_check`].
#[derive(Debug, Clone, Serialize)]
pub struct StorageHealthStatus {
/// Whether the backend is reachable and functional.
pub connected: bool,
/// Human-readable backend identifier (e.g. `"local"`, `"s3"`).
pub backend_type: String,
/// Descriptive status message.
pub message: String,
/// Available space in bytes, if the backend can report it.
pub available_bytes: Option<u64>,
}
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
///
/// Every method operates on a *hash key* that uniquely identifies a blob.
/// The backend is responsible for mapping the hash to its own addressing
/// scheme (filesystem path, S3 key, etc.).
///
/// Returns boxed futures so the trait is dyn-compatible (`Arc<dyn BlobStorageBackend>`).
pub trait BlobStorageBackend: Send + Sync + 'static {
/// Perform any one-time setup (create directories, verify bucket, etc.).
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>>;
/// Store a blob from a local temporary file.
///
/// Must be **idempotent**: if the blob already exists the call succeeds
/// without overwriting. Returns the number of bytes stored.
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>>;
/// Stream the full blob content in chunks.
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>>;
/// Stream a byte range of the blob (for HTTP Range requests / video seek).
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> BoxFut<'_, Result<BlobStream, DomainError>>;
/// Delete a blob by hash. Must be **idempotent** (no error if already gone).
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>>;
/// Check if a blob exists in the backend.
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>>;
/// Get blob size in bytes without downloading content.
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>>;
/// Verify connectivity and permissions (used by the admin "Test Connection" button).
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>>;
/// Return the backend type name for display (e.g. `"local"`, `"s3"`).
fn backend_type(&self) -> &'static str;
/// Return the local filesystem path for a blob, if available.
///
/// Only meaningful for local-filesystem backends. Remote backends
/// return `None`; callers that need a local file must stream + spool.
fn local_blob_path(&self, hash: &str) -> Option<PathBuf>;
}
+1
View File
@@ -1,4 +1,5 @@
pub mod auth_ports;
pub mod blob_storage_ports;
pub mod cache_ports;
pub mod calendar_ports;
pub mod carddav_ports;