feat: pluggable storage backends (S3, Azure, local) with admin UI

Implement 4-phase external storage backends architecture:

Phase 1 - Foundation:
- BlobStorageBackend trait (application/ports/blob_storage_ports.rs)
- LocalBlobBackend: extracted all tokio::fs ops from DedupService
- S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2)
- DedupService refactored to use Arc<dyn BlobStorageBackend>

Phase 2 - Admin Panel:
- StorageSettingsService with DB persistence + env override
- Storage tab in admin panel (backend selector, S3 form, provider presets)
- GET/PUT/POST endpoints for storage settings + connection test
- i18n keys (en/es) and BEM CSS

Phase 3 - Migration:
- MigrationBlobBackend decorator (dual-read: target-first + source fallback)
- Background migration job with parallel transfers + progress tracking
- Migration UI (progress bar, ETA, pause/resume/verify/complete)
- 6 admin API endpoints for migration lifecycle

Phase 4 - Enterprise Extras:
- CachedBlobBackend: LRU disk cache for remote backends
- EncryptedBlobBackend: AES-256-GCM at-rest encryption
- AzureBlobBackend: Azure Blob Storage support
- RetryBlobBackend: exponential backoff for transient errors
- Decorator composition in DI: retry → encryption → cache

All 223 tests passing, clippy clean, fmt verified.
This commit is contained in:
Diocrafts
2026-04-14 21:33:38 +02:00
parent 6fc632af7e
commit cd3733b459
26 changed files with 6870 additions and 308 deletions
Generated
+1502 -92
View File
File diff suppressed because it is too large Load Diff
+8
View File
@@ -58,6 +58,14 @@ dashmap = "6"
socket2 = { version = "0.6.2", features = ["all"] }
urlencoding = "2.1.3"
utoipa = { version = "5", features = ["axum_extras", "uuid", "chrono"] }
aws-sdk-s3 = "1"
aws-config = { version = "1", features = ["behavior-version-latest"] }
aws-smithy-types = "1"
azure_core = "0.21"
azure_storage = "0.21"
azure_storage_blobs = "0.21"
aes-gcm = "0.10"
lru = "0.12"
[features]
default = []
+867
View File
@@ -0,0 +1,867 @@
# External Storage Backends — Implementation Plan
> **Purpose**: This prompt provides Claude Code with full architectural context to implement pluggable blob storage backends (S3, Backblaze B2, MinIO, etc.) for OxiCloud, across 4 phases. All changes MUST respect the existing hexagonal architecture, BLAKE3 dedup system, and coding conventions defined in `CLAUDE.md`.
---
## Current Architecture Summary
### How blobs are stored today
- **Content-addressable**: Files hashed with BLAKE3 → stored at `.blobs/{2-char-prefix}/{hash}.blob`
- **Dedup index**: PostgreSQL `storage.blobs` table (hash PK, ref_count, size, content_type)
- **Write-first strategy**: Blob written to disk BEFORE PostgreSQL upsert (PG connection never held during disk I/O)
- **Streaming reads**: 256 KB chunks via `tokio::fs::File` + `ReaderStream`
- **Range support**: `AsyncSeekExt::seek()` + `file.take()` for HTTP Range requests
### Key files
| File | Role |
|------|------|
| `src/application/ports/dedup_ports.rs` | `DedupPort` trait — 12 methods, the hexagonal port |
| `src/infrastructure/services/dedup_service.rs` | `DedupService` struct — sole implementation of `DedupPort` |
| `src/common/di.rs` | `AppServiceFactory` — DI composition root, builds `DedupService` |
| `src/common/config.rs` | `StorageConfig`, `AppConfig` — env var loading |
| `src/interfaces/api/handlers/admin_handler.rs` | Admin API handlers (OIDC pattern to follow) |
| `src/application/services/admin_settings_service.rs` | `AdminSettingsService` — runtime settings with env override |
| `src/domain/repositories/settings_repository.rs` | `SettingsRepository` trait |
| `src/infrastructure/repositories/pg/settings_pg_repository.rs` | PostgreSQL settings impl |
| `static/admin.html` | Admin panel HTML (3 tabs: Dashboard, Users, OIDC) |
| `static/js/views/admin/admin.js` | Admin panel JS logic |
### DedupService filesystem operations (candidates for extraction)
These are the EXACT `tokio::fs` calls inside `DedupService` that must be delegated to the new `BlobStorageBackend` trait:
```
initialize() → fs::create_dir_all (blob_root, temp_root, 256 prefix dirs)
store_from_file() → fs::metadata, fs::try_exists, fs::rename, fs::copy, fs::remove_file
read_blob_stream() → File::open + ReaderStream
read_blob_range_stream() → File::open + seek + take + ReaderStream
blob_size() → fs::metadata
remove_reference() → fs::remove_file (after PG commit)
verify_integrity() → spawn_blocking with path checks
blob_path() → PathBuf computation (sync)
```
### DedupService PostgreSQL operations (stay in DedupService, untouched)
```
store_from_file() → INSERT … ON CONFLICT … RETURNING ref_count
blob_exists() → SELECT EXISTS from storage.blobs
get_blob_metadata() → SELECT from storage.blobs
add_reference() → UPDATE ref_count + 1
remove_reference() → BEGIN TX → SELECT FOR UPDATE → DELETE if ref_count=0 → COMMIT
get_stats() → SELECT COUNT, SUM from storage.blobs
verify_integrity() → Streaming cursor SELECT from storage.blobs
```
---
## Phase 1 — Foundation (Backend Trait + Local + S3)
### Task 1.1: Create `BlobStorageBackend` trait
**File**: `src/application/ports/blob_storage_ports.rs` (NEW)
Create a minimal trait that abstracts ONLY raw byte I/O operations:
```rust
use async_trait::async_trait;
use bytes::Bytes;
use futures::Stream;
use std::path::Path;
use std::pin::Pin;
use crate::domain::errors::DomainError;
/// Health check result for storage backend connectivity
#[derive(Debug, Clone, serde::Serialize)]
pub struct StorageHealthStatus {
pub connected: bool,
pub backend_type: String,
pub message: String,
/// Optional: available space in bytes (if backend reports it)
pub available_bytes: Option<u64>,
}
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
///
/// Implementations: `LocalBlobBackend`, `S3BlobBackend`, etc.
/// DedupService owns an `Arc<dyn BlobStorageBackend>` and delegates
/// all filesystem/object-store operations through this trait.
#[async_trait]
pub trait BlobStorageBackend: Send + Sync + 'static {
/// Initialize the backend (create directories, verify bucket access, etc.)
async fn initialize(&self) -> Result<(), DomainError>;
/// Store a blob from a local temporary file.
/// The backend MUST handle the case where the blob already exists (idempotent).
/// Returns the number of bytes stored.
async fn put_blob(&self, hash: &str, source_path: &Path) -> Result<u64, DomainError>;
/// Stream the full blob content.
async fn get_blob_stream(
&self,
hash: &str,
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
/// Stream a byte range of the blob (for HTTP Range requests).
async fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>;
/// Delete a blob by hash. Must be idempotent (no error if already deleted).
async fn delete_blob(&self, hash: &str) -> Result<(), DomainError>;
/// Check if a blob exists in the backend.
async fn blob_exists(&self, hash: &str) -> Result<bool, DomainError>;
/// Get blob size in bytes without downloading content.
async fn blob_size(&self, hash: &str) -> Result<u64, DomainError>;
/// Verify connectivity and permissions. Used by admin "Test Connection" button.
async fn health_check(&self) -> Result<StorageHealthStatus, DomainError>;
/// Return the backend type name (for display in admin panel).
fn backend_type(&self) -> &'static str;
}
```
Register in `src/application/ports/mod.rs` and `src/application/mod.rs`.
### Task 1.2: Create `LocalBlobBackend`
**File**: `src/infrastructure/services/local_blob_backend.rs` (NEW)
Extract ALL `tokio::fs` operations from `DedupService` into this struct. This is a **pure refactor** — zero behavior change.
```rust
pub struct LocalBlobBackend {
blob_root: PathBuf,
temp_root: PathBuf,
}
```
Methods to implement from the trait, mapping from current DedupService code:
| Trait method | Source in DedupService | Key logic |
|---|---|---|
| `initialize()` | `DedupService::initialize()` lines 115-154 | Create `.blobs/`, `.dedup_temp/`, 256 prefix dirs |
| `put_blob()` | `DedupService::store_from_file()` lines 196-253 | `fs::try_exists` → `fs::rename` (EXDEV fallback `fs::copy`) → cleanup source |
| `get_blob_stream()` | `DedupService::read_blob_stream()` lines 306-323 | `File::open` → `ReaderStream::with_capacity(256KB)` |
| `get_blob_range_stream()` | `DedupService::read_blob_range_stream()` lines 325-353 | `File::open` → `seek` → `take` → `ReaderStream` |
| `delete_blob()` | `DedupService::remove_reference()` line ~469 | `fs::remove_file` |
| `blob_exists()` | New (was inline `fs::try_exists`) | `fs::try_exists(blob_path)` |
| `blob_size()` | `DedupService::blob_size()` lines 355-369 | `fs::metadata().len()` |
| `health_check()` | New | Check blob_root is writable, return disk available via `statvfs` |
| `backend_type()` | New | Return `"local"` |
Also add a public helper:
```rust
pub fn blob_path(&self, hash: &str) -> PathBuf {
let prefix = &hash[..2];
self.blob_root.join(prefix).join(format!("{hash}.blob"))
}
```
Register in `src/infrastructure/services/mod.rs`.
### Task 1.3: Refactor `DedupService` to use `BlobStorageBackend`
**File**: `src/infrastructure/services/dedup_service.rs` (MODIFY)
Changes:
1. Add field: `backend: Arc<dyn BlobStorageBackend>`
2. Remove fields: `blob_root: PathBuf`, `temp_root: PathBuf` (moved to `LocalBlobBackend`)
3. Update constructor to accept `Arc<dyn BlobStorageBackend>` instead of `storage_root: &Path`
4. Replace all direct `tokio::fs` calls with `self.backend.*` calls
5. Keep `blob_path()` in DedupPort as a delegation: `self.backend.blob_path()` — BUT since `blob_path()` returns `PathBuf` and is used by thumbnails/caching services, consider adding it to the backend trait OR keeping a separate method. For S3 backends, this should return a virtual path or the method should be deprecated in favor of streaming.
**Critical**: `hash_file()` stays in `DedupService` (BLAKE3 hashing is NOT a backend concern — it always runs on local temp files before upload).
**Critical**: The write-first strategy is preserved:
```
1. hash_file() on local temp file
2. self.backend.put_blob(hash, temp_path) ← backend moves/uploads
3. INSERT INTO storage.blobs … ON CONFLICT ← PostgreSQL upsert
```
**Critical**: `remove_reference()` flow preserved:
```
1. BEGIN TX → SELECT FOR UPDATE → check ref_count
2. If ref_count == 1 → DELETE FROM storage.blobs → COMMIT
3. self.backend.delete_blob(hash) ← after PG commit
```
### Task 1.4: Create `S3BlobBackend`
**File**: `src/infrastructure/services/s3_blob_backend.rs` (NEW)
**Dependency to add to `Cargo.toml`**:
```toml
aws-sdk-s3 = "1"
aws-config = { version = "1", features = ["behavior-version-latest"] }
aws-smithy-types = "1" # For ByteStream
```
> Note: `aws-sdk-s3` is the official AWS SDK for Rust. It's compatible with ALL S3-compatible services (Backblaze B2, MinIO, Cloudflare R2, DigitalOcean Spaces, Wasabi) via custom endpoint configuration.
```rust
pub struct S3BlobBackend {
client: aws_sdk_s3::Client,
bucket: String,
}
```
**S3 key scheme**: Same as local — `{2-char-prefix}/{hash}.blob` (e.g., `a3/a3c5f2e8d1…blob`)
Method mapping:
| Trait method | S3 operation |
|---|---|
| `initialize()` | `head_bucket()` to verify bucket exists + permissions |
| `put_blob()` | `put_object()` with `Body::from_path(source_path)`. Check existence with `head_object()` first for idempotency |
| `get_blob_stream()` | `get_object()` → `.body.into_async_read()` → `ReaderStream` |
| `get_blob_range_stream()` | `get_object().range(format!("bytes={start}-{end}"))` → stream |
| `delete_blob()` | `delete_object()` (already idempotent in S3) |
| `blob_exists()` | `head_object()` — 200 = true, 404 = false |
| `blob_size()` | `head_object()` → `.content_length()` |
| `health_check()` | `head_bucket()` + `list_objects_v2(max_keys=1)` |
| `backend_type()` | Return `"s3"` |
**S3 Client construction**: Must support custom endpoints for non-AWS providers:
```rust
impl S3BlobBackend {
pub async fn new(config: &S3StorageConfig) -> Result<Self, DomainError> {
let mut s3_config_builder = aws_sdk_s3::config::Builder::new()
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
.credentials_provider(
aws_sdk_s3::config::Credentials::new(
&config.access_key,
&config.secret_key,
None, None, "oxicloud",
)
)
.behavior_version_latest();
if let Some(endpoint) = &config.endpoint_url {
s3_config_builder = s3_config_builder
.endpoint_url(endpoint)
.force_path_style(config.force_path_style);
}
let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());
Ok(Self { client, bucket: config.bucket.clone() })
}
}
```
### Task 1.5: Add storage backend configuration
**File**: `src/common/config.rs` (MODIFY)
Add to existing `StorageConfig`:
```rust
#[derive(Debug, Clone)]
pub enum StorageBackendType {
Local,
S3,
}
#[derive(Debug, Clone)]
pub struct S3StorageConfig {
pub endpoint_url: Option<String>, // OXICLOUD_S3_ENDPOINT_URL
pub bucket: String, // OXICLOUD_S3_BUCKET
pub region: String, // OXICLOUD_S3_REGION (default: "us-east-1")
pub access_key: String, // OXICLOUD_S3_ACCESS_KEY
pub secret_key: String, // OXICLOUD_S3_SECRET_KEY
pub force_path_style: bool, // OXICLOUD_S3_FORCE_PATH_STYLE (default: false)
}
// Add to existing StorageConfig:
pub struct StorageConfig {
pub root_dir: String, // existing
pub chunk_size: usize, // existing
pub parallel_threshold: usize, // existing
pub trash_retention_days: u32, // existing
pub max_upload_size: usize, // existing
pub backend: StorageBackendType, // NEW — OXICLOUD_STORAGE_BACKEND (default: "local")
pub s3: Option<S3StorageConfig>, // NEW — populated when backend=s3
}
```
**Env var loading** in `AppConfig::from_env()`:
```
OXICLOUD_STORAGE_BACKEND → "local" | "s3" (default: "local")
OXICLOUD_S3_ENDPOINT_URL → Optional custom endpoint
OXICLOUD_S3_BUCKET → Required when backend=s3
OXICLOUD_S3_REGION → Default "us-east-1"
OXICLOUD_S3_ACCESS_KEY → Required when backend=s3
OXICLOUD_S3_SECRET_KEY → Required when backend=s3
OXICLOUD_S3_FORCE_PATH_STYLE → Default false
```
### Task 1.6: Wire backend selection in DI
**File**: `src/common/di.rs` (MODIFY)
In `create_core_services()`, replace the current DedupService construction:
```rust
// Build storage backend based on config
let blob_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
StorageBackendType::Local => {
Arc::new(LocalBlobBackend::new(&self.storage_path))
}
StorageBackendType::S3 => {
let s3_config = self.config.storage.s3.as_ref()
.expect("S3 config required when backend=s3");
Arc::new(S3BlobBackend::new(s3_config).await?)
}
};
blob_backend.initialize().await?;
let dedup_service = Arc::new(DedupService::new(
blob_backend.clone(),
db_pool.clone(),
maintenance_pool.clone(),
));
```
### Task 1.7: Handle `blob_path()` deprecation path
The `DedupPort::blob_path()` method returns a `PathBuf` and is used by:
- Thumbnail generation service (needs local file access)
- File content caching (moka cache)
For S3 backends, `blob_path()` has no meaning. Solutions:
1. **For thumbnails**: Change thumbnail service to accept a `Stream` instead of a `PathBuf`, OR download to a temp file first
2. **For caching**: Cache already works with streams
3. Keep `blob_path()` on `DedupPort` but make it return `Option<PathBuf>` (None for remote backends) — consumers must handle the None case
Search for all callers of `blob_path()` and update them.
### Task 1.8: Tests
- Unit test `LocalBlobBackend` in isolation (mock filesystem with temp dirs)
- Unit test `S3BlobBackend` with a mock S3 (use `aws-smithy-runtime` test utilities or `mockall`)
- Integration test: `DedupService` with `LocalBlobBackend` must pass ALL existing tests unchanged (this proves the refactor is correct)
- Add `#[cfg(test)]` inline tests in each new file following existing project convention
### Task 1.9: Pre-commit validation
```bash
cargo fmt --all
cargo clippy -- -D warnings
cargo test --workspace
```
ALL existing ~208 tests MUST pass. Zero regressions.
---
## Phase 2 — Admin Panel Configuration
### Task 2.1: Storage settings service
**File**: `src/application/services/storage_settings_service.rs` (NEW)
Follow the EXACT same pattern as `AdminSettingsService` for OIDC. Create `StorageSettingsService`:
```rust
pub struct StorageSettingsService {
settings_repo: Arc<SettingsPgRepository>,
env_storage_config: StorageConfig, // from AppConfig at startup
}
```
**Methods** (follow OIDC pattern):
| Method | Purpose |
|---|---|
| `get_storage_settings()` | Load from DB (category: `"storage"`), mask secrets, mark env overrides |
| `save_storage_settings(dto, user_id)` | Upsert each field to `admin_settings`, mark secrets with `is_secret: true` |
| `test_storage_connection(dto)` | Build temporary backend from DTO config, call `health_check()`, return result |
| `load_effective_storage_config()` | Merge: DB settings + env var overrides + defaults |
| `get_env_overrides()` | Return list of `OXICLOUD_S3_*` / `OXICLOUD_STORAGE_*` env vars that are set |
**DB keys** (category: `"storage"`):
```
storage.backend → "local" | "s3"
storage.s3.endpoint_url → string (optional)
storage.s3.bucket → string
storage.s3.region → string
storage.s3.access_key → string (is_secret: true)
storage.s3.secret_key → string (is_secret: true)
storage.s3.force_path_style → "true" | "false"
```
### Task 2.2: Storage admin API endpoints
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
Add to `admin_routes()`:
```rust
.route("/settings/storage", get(get_storage_settings))
.route("/settings/storage", put(save_storage_settings))
.route("/settings/storage/test", post(test_storage_connection))
```
**Handler implementations** (follow OIDC handlers exactly):
| Handler | Method | Body | Response |
|---|---|---|---|
| `get_storage_settings` | GET | — | `StorageSettingsDto` (secrets masked, env_overrides listed) |
| `save_storage_settings` | PUT | `SaveStorageSettingsDto` | `{ "message": "Storage settings saved" }` |
| `test_storage_connection` | POST | `TestStorageConnectionDto` | `StorageTestResultDto { connected, message, available_bytes }` |
**DTOs** (add to `src/application/dtos/`):
```rust
#[derive(Serialize)]
pub struct StorageSettingsDto {
pub backend: String, // "local" | "s3"
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
pub s3_access_key_set: bool, // masked — never send actual key
pub s3_secret_key_set: bool, // masked — never send actual secret
pub s3_force_path_style: bool,
pub env_overrides: Vec<String>, // which fields are locked by env vars
// Current stats
pub current_backend: String,
pub total_blobs: u64,
pub total_bytes_stored: u64,
pub dedup_ratio: f64,
}
#[derive(Deserialize)]
pub struct SaveStorageSettingsDto {
pub backend: String,
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
pub s3_access_key: Option<String>, // only sent if changed
pub s3_secret_key: Option<String>, // only sent if changed
pub s3_force_path_style: Option<bool>,
}
#[derive(Deserialize)]
pub struct TestStorageConnectionDto {
pub backend: String,
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
pub s3_access_key: Option<String>,
pub s3_secret_key: Option<String>,
pub s3_force_path_style: Option<bool>,
}
#[derive(Serialize)]
pub struct StorageTestResultDto {
pub connected: bool,
pub message: String,
pub backend_type: String,
pub available_bytes: Option<u64>,
}
```
### Task 2.3: Wire `StorageSettingsService` into DI
**File**: `src/common/di.rs` (MODIFY)
Add `StorageSettingsService` construction alongside `AdminSettingsService`. Add to `AppState`.
### Task 2.4: Admin Panel — Storage tab (HTML)
**File**: `static/admin.html` (MODIFY)
Add a 4th tab button after the OIDC tab:
```html
<button class="admin-tab" id="tab-btn-storage">
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
</button>
```
Add `tab-storage` content div with:
1. **Backend selector** — Radio buttons: Local Filesystem / S3-Compatible
2. **S3 configuration form** (shown/hidden based on selector):
- Provider preset dropdown (Amazon S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces, Wasabi, Custom)
- Endpoint URL field
- Bucket field
- Region field
- Access Key ID field
- Secret Key field (password type)
- Force Path Style checkbox
- ENV badges on fields overridden by env vars (same as OIDC)
3. **Test Connection button** → calls `POST /api/admin/settings/storage/test`
4. **Save button** → calls `PUT /api/admin/settings/storage`
5. **Current Status section**:
- Active backend type
- Total blobs / total size / dedup ratio (from `DedupStatsDto`)
6. **Migration section** (Phase 3 — can be placeholder with "Coming soon")
### Task 2.5: Admin Panel — Storage tab (JS)
**File**: `static/js/views/admin/admin.js` (MODIFY)
Follow OIDC tab patterns:
```javascript
// Provider presets
const STORAGE_PRESETS = {
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
'backblaze': { endpoint: 's3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
'cloudflare-r2': { endpoint: '{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
'digitalocean': { endpoint: '{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
'wasabi': { endpoint: 's3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
'custom': { endpoint: '', region: '', pathStyle: false },
};
```
Functions:
- `loadStorage()` — `GET /api/admin/settings/storage` → populate form
- `saveStorage()` — Collect form → `PUT /api/admin/settings/storage`
- `testStorageConnection()` — Collect form → `POST /api/admin/settings/storage/test` → show result
- `onPresetChange(preset)` — Auto-fill endpoint/region/pathStyle from preset
- `toggleS3Form(visible)` — Show/hide S3 fields when backend radio changes
### Task 2.6: i18n keys
**Files**: `static/locales/*.json` (MODIFY at minimum `en.json` and `es.json`)
Add translation keys for all new UI labels:
```
admin.tab_storage
admin.storage_backend
admin.storage_local
admin.storage_s3
admin.storage_provider_preset
admin.storage_endpoint_url
admin.storage_bucket
admin.storage_region
admin.storage_access_key
admin.storage_secret_key
admin.storage_path_style
admin.storage_test_connection
admin.storage_test_success
admin.storage_test_failure
admin.storage_save
admin.storage_saved
admin.storage_current_status
admin.storage_total_blobs
admin.storage_total_size
admin.storage_dedup_ratio
admin.storage_migration
admin.storage_migration_coming_soon
```
### Task 2.7: CSS for storage tab
**File**: `static/css/admin.css` (MODIFY)
Add styles for:
- `.storage-backend-selector` — Radio button group
- `.storage-provider-presets` — Dropdown styling
- `.storage-form` — Form fields (reuse existing OIDC form patterns)
- `.storage-status-grid` — Stats display
- BEM methodology, CSS custom properties for colors, no raw hex/rgb
---
## Phase 3 — Migration Between Backends
### Task 3.1: `MigrationBlobBackend` wrapper
**File**: `src/infrastructure/services/migration_blob_backend.rs` (NEW)
A `BlobStorageBackend` decorator that enables zero-downtime migration between backends:
```rust
pub struct MigrationBlobBackend {
source: Arc<dyn BlobStorageBackend>, // old backend (read fallback)
target: Arc<dyn BlobStorageBackend>, // new backend (primary for writes)
state: Arc<RwLock<MigrationState>>,
}
pub struct MigrationState {
pub status: MigrationStatus,
pub total_blobs: u64,
pub migrated_blobs: u64,
pub migrated_bytes: u64,
pub failed_blobs: Vec<String>, // hashes that failed
pub started_at: Option<chrono::DateTime<chrono::Utc>>,
pub completed_at: Option<chrono::DateTime<chrono::Utc>>,
}
pub enum MigrationStatus {
Idle, // no migration in progress
Running, // background job active
Paused, // manually paused
Completed, // all blobs migrated
Failed, // unrecoverable error
}
```
**Behavior**:
| Operation | During Migration |
|---|---|
| `put_blob()` | Write to **target** only |
| `get_blob_stream()` | Try **target** first → fallback to **source** (+ schedule lazy copy) |
| `get_blob_range_stream()` | Same fallback strategy |
| `delete_blob()` | Delete from **both** (best-effort on source) |
| `blob_exists()` | Check **target** first, then **source** |
| `blob_size()` | Check **target** first, then **source** |
### Task 3.2: Background migration job
**File**: `src/infrastructure/services/migration_job.rs` (NEW)
```rust
pub async fn run_migration(
source: Arc<dyn BlobStorageBackend>,
target: Arc<dyn BlobStorageBackend>,
pool: Arc<PgPool>,
state: Arc<RwLock<MigrationState>>,
concurrency: usize, // default: 4 parallel transfers
bandwidth_limit: Option<u64>, // bytes/sec, None = unlimited
) -> Result<(), DomainError>
```
**Algorithm**:
1. Query `SELECT hash, size FROM storage.blobs ORDER BY hash` with streaming cursor
2. For each blob hash:
a. Check if already exists in target (`target.blob_exists(hash)`)
b. If not: stream from source → write to temp file → `target.put_blob(hash, temp)`
c. Update `MigrationState` counters
d. Respect bandwidth limit via `tokio::time::sleep` throttling
3. Use `futures::stream::buffer_unordered(concurrency)` for parallel transfers
4. On error: log, add to `failed_blobs`, continue (don't abort entire migration)
### Task 3.3: Migration API endpoints
**File**: `src/interfaces/api/handlers/admin_handler.rs` (MODIFY)
Add to `admin_routes()`:
```rust
.route("/storage/migration", get(get_migration_status))
.route("/storage/migration/start", post(start_migration))
.route("/storage/migration/pause", post(pause_migration))
.route("/storage/migration/resume", post(resume_migration))
.route("/storage/migration/complete", post(complete_migration))
```
| Endpoint | Purpose |
|---|---|
| `GET /migration` | Return `MigrationState` (status, progress, ETA) |
| `POST /migration/start` | Begin background migration from current → configured backend |
| `POST /migration/pause` | Pause the background job |
| `POST /migration/resume` | Resume paused migration |
| `POST /migration/complete` | Finalize: switch primary backend, optionally clean up source |
### Task 3.4: Migration UI in admin panel
In the Storage tab's Migration section:
1. **Start Migration button** (when backend config differs from active)
2. **Progress bar**: `{migrated} / {total} blobs ({percent}%) — {bytes_migrated} transferred`
3. **Estimated time remaining** (based on throughput)
4. **Pause / Resume button**
5. **Complete Migration button** (enabled only when 100% migrated)
6. **Failed blobs list** (expandable, with retry button)
7. **Status badge**: Idle / Running / Paused / Completed / Failed
### Task 3.5: Integrity verification post-migration
After migration completes (before `complete_migration`):
1. Run `verify_integrity()` against the target backend
2. Compare blob count in PG vs target backend
3. Sample-verify N random blobs (download + BLAKE3 hash check)
4. Show verification results in admin UI before allowing finalization
---
## Phase 4 — Enterprise Extras
### Task 4.1: `CachedBlobBackend` — LRU local disk cache
**File**: `src/infrastructure/services/cached_blob_backend.rs` (NEW)
A `BlobStorageBackend` decorator for remote backends (S3, Azure) that caches hot blobs on local SSD:
```rust
pub struct CachedBlobBackend {
inner: Arc<dyn BlobStorageBackend>, // S3 backend
cache_dir: PathBuf, // local cache directory
max_cache_bytes: u64, // configurable limit
index: Arc<RwLock<LruCache<String, CacheEntry>>>,
current_size: Arc<AtomicU64>,
}
struct CacheEntry {
size: u64,
last_accessed: Instant,
}
```
**Behavior**:
- **Reads**: Check local cache first → cache hit returns local file stream → cache miss downloads from inner backend, writes to cache, returns stream
- **Writes**: `put_blob()` writes to inner backend AND local cache simultaneously
- **Eviction**: LRU eviction when `current_size` exceeds `max_cache_bytes`
- **Startup**: Scan cache directory to rebuild index
**Configuration** (admin panel):
```
storage.cache.enabled → "true" | "false"
storage.cache.max_size_bytes → u64 (default: 50 GB)
storage.cache.path → PathBuf (default: "{storage_root}/.cache")
```
**Env vars**:
```
OXICLOUD_STORAGE_CACHE_ENABLED=true
OXICLOUD_STORAGE_CACHE_MAX_SIZE=53687091200 # 50 GB
OXICLOUD_STORAGE_CACHE_PATH=/fast-ssd/oxicloud-cache
```
### Task 4.2: Client-side encryption (AES-256-GCM)
**File**: `src/infrastructure/services/encrypted_blob_backend.rs` (NEW)
Another `BlobStorageBackend` decorator that encrypts blobs before sending to the inner backend:
```rust
pub struct EncryptedBlobBackend {
inner: Arc<dyn BlobStorageBackend>,
encryption_key: [u8; 32], // AES-256 key
}
```
**Dependency**: Add `aes-gcm = "0.10"` to Cargo.toml.
**Behavior**:
- `put_blob()`: Read source → encrypt with AES-256-GCM (random 96-bit nonce prepended) → write encrypted to temp → `inner.put_blob(hash, encrypted_temp)`
- `get_blob_stream()`: `inner.get_blob_stream()` → decrypt stream → return plaintext stream
- **CRITICAL**: BLAKE3 hash is computed on the PLAINTEXT (before encryption), so dedup still works across encrypted backends
- **Nonce storage**: Prepend 12-byte nonce to each encrypted blob (total overhead: 28 bytes per blob — 12 nonce + 16 GCM tag)
**Configuration** (admin panel):
```
storage.encryption.enabled → "true" | "false"
storage.encryption.key → base64-encoded 32-byte key (is_secret: true)
```
**Key generation**: Provide an admin API endpoint `POST /api/admin/settings/storage/generate-key` that generates a cryptographically secure key and returns it once (user must save it).
**WARNING in admin UI**: "If you lose the encryption key, all data in the storage backend is IRRECOVERABLY LOST. Back up this key securely."
### Task 4.3: Azure Blob Storage backend
**File**: `src/infrastructure/services/azure_blob_backend.rs` (NEW)
**Dependency**: `azure_storage_blobs = "0.21"`, `azure_storage = "0.21"`
Same `BlobStorageBackend` trait implementation targeting Azure Blob Storage:
- Container = bucket equivalent
- Blob key scheme: `{prefix}/{hash}.blob` (same as S3/local)
- Authentication: Account Name + Account Key OR SAS token
**Configuration**:
```
OXICLOUD_AZURE_ACCOUNT_NAME
OXICLOUD_AZURE_ACCOUNT_KEY
OXICLOUD_AZURE_CONTAINER
OXICLOUD_AZURE_SAS_TOKEN # alternative auth
```
### Task 4.4: Bandwidth throttling & retry policies
Apply to all remote backends (S3, Azure):
**Throttling**:
- Configurable upload/download bandwidth limit per-backend
- Implemented via `tokio::time::sleep` between chunks
- Admin configurable: `storage.s3.max_upload_bandwidth`, `storage.s3.max_download_bandwidth`
**Retry policy** (exponential backoff):
```rust
pub struct RetryPolicy {
pub max_retries: u32, // default: 3
pub initial_backoff_ms: u64, // default: 100
pub max_backoff_ms: u64, // default: 10_000
pub backoff_multiplier: f64, // default: 2.0
}
```
Wrap remote backend calls with retry logic for transient errors (network timeouts, 500s, 503s).
---
## Architecture Invariants (MUST be preserved)
1. **BLAKE3 hashing**: Always performed locally on temp files, never delegated to backend
2. **PostgreSQL dedup index**: `storage.blobs` table remains the source of truth for ref_count, metadata
3. **Write-first strategy**: Blob stored in backend BEFORE PostgreSQL upsert
4. **Remove-after-commit**: Blob deleted from backend AFTER PostgreSQL transaction commits
5. **Streaming reads**: All blob reads return `Pin<Box<dyn Stream>>`, never load full blob into memory
6. **Zero framework deps in domain**: `BlobStorageBackend` trait lives in `application/ports/`, not infrastructure
7. **DI via AppState**: All backends are `Arc`-wrapped and assembled in `common/di.rs`
8. **Env var precedence**: `OXICLOUD_*` env vars always override DB-stored admin settings
9. **Admin guard**: All storage admin endpoints require JWT with role `"admin"`
10. **Existing tests**: All ~208 tests must pass after Phase 1 refactor
## Backend Decorator Composition
The backends compose as decorators. In `di.rs`, the assembly looks like:
```rust
// Phase 1: Base backend
let base_backend: Arc<dyn BlobStorageBackend> = match config {
Local => Arc::new(LocalBlobBackend::new(...)),
S3 => Arc::new(S3BlobBackend::new(...).await?),
Azure => Arc::new(AzureBlobBackend::new(...).await?),
};
// Phase 4: Optional encryption layer
let backend = if encryption_enabled {
Arc::new(EncryptedBlobBackend::new(base_backend, key))
} else {
base_backend
};
// Phase 4: Optional cache layer (only for remote backends)
let backend = if cache_enabled && !matches!(config, Local) {
Arc::new(CachedBlobBackend::new(backend, cache_config))
} else {
backend
};
// Phase 3: Optional migration wrapper
let backend = if migration_in_progress {
Arc::new(MigrationBlobBackend::new(old_backend, backend, state))
} else {
backend
};
// Finally: DedupService uses the composed backend
let dedup_service = Arc::new(DedupService::new(backend, pool, maintenance_pool));
```
This decorator pattern means each feature (encryption, caching, migration) is:
- Independently testable
- Independently toggleable
- Zero overhead when disabled
- Composable in any order
+94
View File
@@ -128,3 +128,97 @@ pub struct DashboardStatsDto {
pub users_over_quota: i64,
pub registration_enabled: bool,
}
// ============================================================================
// Storage Settings DTOs (Admin Panel)
// ============================================================================
/// Current storage settings returned to admin UI (secrets masked)
#[derive(Debug, Serialize, Deserialize)]
pub struct StorageSettingsDto {
/// Active backend type: "local" or "s3"
pub backend: String,
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
/// True if an access key is configured (never reveals the actual value)
pub s3_access_key_set: bool,
/// True if a secret key is configured (never reveals the actual value)
pub s3_secret_key_set: bool,
pub s3_force_path_style: bool,
/// Field names overridden by environment variables (read-only in UI)
pub env_overrides: Vec<String>,
// ── Current stats ──
pub current_backend: String,
pub total_blobs: u64,
pub total_bytes_stored: u64,
pub dedup_ratio: f64,
}
/// Request body for saving storage settings from the admin panel
#[derive(Debug, Serialize, Deserialize)]
pub struct SaveStorageSettingsDto {
pub backend: String,
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
/// Only update if provided and non-empty (None = keep existing)
pub s3_access_key: Option<String>,
/// Only update if provided and non-empty (None = keep existing)
pub s3_secret_key: Option<String>,
pub s3_force_path_style: Option<bool>,
}
/// Request body for testing a storage connection
#[derive(Debug, Serialize, Deserialize)]
pub struct TestStorageConnectionDto {
pub backend: String,
pub s3_endpoint_url: Option<String>,
pub s3_bucket: Option<String>,
pub s3_region: Option<String>,
pub s3_access_key: Option<String>,
pub s3_secret_key: Option<String>,
pub s3_force_path_style: Option<bool>,
}
/// Result of a storage connection test
#[derive(Debug, Serialize, Deserialize)]
pub struct StorageTestResultDto {
pub connected: bool,
pub message: String,
pub backend_type: String,
pub available_bytes: Option<u64>,
}
// ============================================================================
// Migration DTOs (Admin Panel — Storage Migration)
// ============================================================================
/// Migration progress returned by `GET /api/admin/storage/migration`.
/// Re-exports the `MigrationState` shape for the admin UI.
#[derive(Debug, Serialize, Deserialize)]
pub struct MigrationStateDto {
pub status: String,
pub total_blobs: u64,
pub migrated_blobs: u64,
pub migrated_bytes: u64,
pub failed_blobs: Vec<String>,
pub started_at: Option<String>,
pub completed_at: Option<String>,
/// Estimated throughput in bytes/sec (for UI ETA calculation).
pub throughput_bytes_per_sec: Option<f64>,
}
/// Request body for `POST /api/admin/storage/migration/start`.
#[derive(Debug, Serialize, Deserialize)]
pub struct StartMigrationDto {
/// How many blobs to copy in parallel (default: 4).
pub concurrency: Option<usize>,
}
/// Request body (empty) for `POST /api/admin/storage/migration/verify`.
#[derive(Debug, Serialize, Deserialize)]
pub struct VerifyMigrationDto {
/// Number of random blobs to sample-check (default: 100).
pub sample_size: Option<usize>,
}
@@ -0,0 +1,88 @@
//! Blob Storage Backend Port — abstracts raw byte I/O for content-addressable storage.
//!
//! This trait decouples `DedupService` from any specific storage medium.
//! Implementations include:
//! - `LocalBlobBackend` — local filesystem (default)
//! - `S3BlobBackend` — any S3-compatible service (AWS, Backblaze B2, MinIO, R2…)
//!
//! `DedupService` owns an `Arc<dyn BlobStorageBackend>` and delegates all
//! byte-level I/O through this trait, keeping BLAKE3 hashing, ref-counting
//! and PostgreSQL index logic in `DedupService` itself.
use bytes::Bytes;
use futures::Stream;
use serde::Serialize;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use crate::domain::errors::DomainError;
/// Boxed future alias used by [`BlobStorageBackend`] to keep the trait dyn-compatible.
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
/// Pinned boxed byte stream — the return type for blob reads.
pub type BlobStream = Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>;
/// Health-check result returned by [`BlobStorageBackend::health_check`].
#[derive(Debug, Clone, Serialize)]
pub struct StorageHealthStatus {
/// Whether the backend is reachable and functional.
pub connected: bool,
/// Human-readable backend identifier (e.g. `"local"`, `"s3"`).
pub backend_type: String,
/// Descriptive status message.
pub message: String,
/// Available space in bytes, if the backend can report it.
pub available_bytes: Option<u64>,
}
/// Minimal trait for blob byte I/O — decoupled from dedup logic.
///
/// Every method operates on a *hash key* that uniquely identifies a blob.
/// The backend is responsible for mapping the hash to its own addressing
/// scheme (filesystem path, S3 key, etc.).
///
/// Returns boxed futures so the trait is dyn-compatible (`Arc<dyn BlobStorageBackend>`).
pub trait BlobStorageBackend: Send + Sync + 'static {
/// Perform any one-time setup (create directories, verify bucket, etc.).
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>>;
/// Store a blob from a local temporary file.
///
/// Must be **idempotent**: if the blob already exists the call succeeds
/// without overwriting. Returns the number of bytes stored.
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>>;
/// Stream the full blob content in chunks.
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>>;
/// Stream a byte range of the blob (for HTTP Range requests / video seek).
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> BoxFut<'_, Result<BlobStream, DomainError>>;
/// Delete a blob by hash. Must be **idempotent** (no error if already gone).
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>>;
/// Check if a blob exists in the backend.
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>>;
/// Get blob size in bytes without downloading content.
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>>;
/// Verify connectivity and permissions (used by the admin "Test Connection" button).
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>>;
/// Return the backend type name for display (e.g. `"local"`, `"s3"`).
fn backend_type(&self) -> &'static str;
/// Return the local filesystem path for a blob, if available.
///
/// Only meaningful for local-filesystem backends. Remote backends
/// return `None`; callers that need a local file must stream + spool.
fn local_blob_path(&self, hash: &str) -> Option<PathBuf>;
}
+1
View File
@@ -1,4 +1,5 @@
pub mod auth_ports;
pub mod blob_storage_ports;
pub mod cache_ports;
pub mod calendar_ports;
pub mod carddav_ports;
+1
View File
@@ -18,6 +18,7 @@ pub mod nextcloud_login_flow_service;
pub mod recent_service;
pub mod search_service;
pub mod share_service;
pub mod storage_settings_service;
pub mod storage_usage_service;
pub mod trash_service;
pub mod wopi_lock_service;
@@ -0,0 +1,337 @@
use std::collections::HashMap;
use std::sync::Arc;
use uuid::Uuid;
use crate::application::dtos::settings_dto::{
SaveStorageSettingsDto, StorageSettingsDto, StorageTestResultDto, TestStorageConnectionDto,
};
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::{S3StorageConfig, StorageConfig};
use crate::common::errors::{DomainError, ErrorKind};
use crate::domain::repositories::settings_repository::SettingsRepository;
use crate::infrastructure::repositories::pg::SettingsPgRepository;
use crate::infrastructure::services::dedup_service::DedupService;
use crate::infrastructure::services::s3_blob_backend::S3BlobBackend;
/// Storage settings service — manages storage backend configuration via the admin panel.
///
/// Configuration priority: **env vars > DB settings > defaults**.
pub struct StorageSettingsService {
settings_repo: Arc<SettingsPgRepository>,
env_storage_config: StorageConfig,
dedup_service: Arc<DedupService>,
}
impl StorageSettingsService {
pub fn new(
settings_repo: Arc<SettingsPgRepository>,
env_storage_config: StorageConfig,
dedup_service: Arc<DedupService>,
) -> Self {
Self {
settings_repo,
env_storage_config,
dedup_service,
}
}
/// Detect which storage fields are overridden by environment variables.
fn get_env_overrides(&self) -> Vec<String> {
let mut out = Vec::new();
let vars = [
("OXICLOUD_STORAGE_BACKEND", "backend"),
("OXICLOUD_S3_ENDPOINT_URL", "s3_endpoint_url"),
("OXICLOUD_S3_BUCKET", "s3_bucket"),
("OXICLOUD_S3_REGION", "s3_region"),
("OXICLOUD_S3_ACCESS_KEY", "s3_access_key"),
("OXICLOUD_S3_SECRET_KEY", "s3_secret_key"),
("OXICLOUD_S3_FORCE_PATH_STYLE", "s3_force_path_style"),
];
for (env_key, field_name) in &vars {
if std::env::var(env_key).is_ok() {
out.push(field_name.to_string());
}
}
out
}
/// Apply environment variable overrides on top of a config.
fn apply_env_overrides(&self, config: &mut StorageConfig) {
let e = &self.env_storage_config;
if std::env::var("OXICLOUD_STORAGE_BACKEND").is_ok() {
config.backend = e.backend.clone();
}
// S3 env overrides — only apply if S3 config exists in env
if let Some(env_s3) = &e.s3 {
let s3 = config.s3.get_or_insert_with(|| S3StorageConfig {
endpoint_url: None,
bucket: String::new(),
region: "us-east-1".to_string(),
access_key: String::new(),
secret_key: String::new(),
force_path_style: false,
});
if std::env::var("OXICLOUD_S3_ENDPOINT_URL").is_ok() {
s3.endpoint_url = env_s3.endpoint_url.clone();
}
if std::env::var("OXICLOUD_S3_BUCKET").is_ok() {
s3.bucket = env_s3.bucket.clone();
}
if std::env::var("OXICLOUD_S3_REGION").is_ok() {
s3.region = env_s3.region.clone();
}
if std::env::var("OXICLOUD_S3_ACCESS_KEY").is_ok() {
s3.access_key = env_s3.access_key.clone();
}
if std::env::var("OXICLOUD_S3_SECRET_KEY").is_ok() {
s3.secret_key = env_s3.secret_key.clone();
}
if std::env::var("OXICLOUD_S3_FORCE_PATH_STYLE").is_ok() {
s3.force_path_style = env_s3.force_path_style;
}
}
}
/// Load effective storage config: DB settings + env var overrides + defaults.
pub async fn load_effective_storage_config(&self) -> Result<StorageConfig, DomainError> {
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
let d = StorageConfig::default();
let backend = db
.get("storage.backend")
.map(|v| match v.as_str() {
"s3" => crate::common::config::StorageBackendType::S3,
"azure" => crate::common::config::StorageBackendType::Azure,
_ => crate::common::config::StorageBackendType::Local,
})
.unwrap_or(d.backend);
let s3 = {
let bucket = db.get("storage.s3.bucket").cloned().unwrap_or_default();
if bucket.is_empty() {
None
} else {
Some(S3StorageConfig {
endpoint_url: db
.get("storage.s3.endpoint_url")
.cloned()
.filter(|s| !s.is_empty()),
bucket,
region: db
.get("storage.s3.region")
.cloned()
.unwrap_or_else(|| "us-east-1".to_string()),
access_key: db.get("storage.s3.access_key").cloned().unwrap_or_default(),
secret_key: db.get("storage.s3.secret_key").cloned().unwrap_or_default(),
force_path_style: db
.get("storage.s3.force_path_style")
.and_then(|v| v.parse().ok())
.unwrap_or(false),
})
}
};
let mut config = StorageConfig {
backend,
s3,
..self.env_storage_config.clone()
};
self.apply_env_overrides(&mut config);
Ok(config)
}
/// Get storage settings for display in admin UI (secrets masked).
pub async fn get_storage_settings(&self) -> Result<StorageSettingsDto, DomainError> {
let db: HashMap<String, String> = self.settings_repo.get_by_category("storage").await?;
let has_access_key = db
.get("storage.s3.access_key")
.map(|s| !s.is_empty())
.unwrap_or(false)
|| std::env::var("OXICLOUD_S3_ACCESS_KEY")
.map(|s| !s.is_empty())
.unwrap_or(false);
let has_secret_key = db
.get("storage.s3.secret_key")
.map(|s| !s.is_empty())
.unwrap_or(false)
|| std::env::var("OXICLOUD_S3_SECRET_KEY")
.map(|s| !s.is_empty())
.unwrap_or(false);
let effective = self.load_effective_storage_config().await?;
let stats = self.dedup_service.get_stats().await;
let current_backend = self.dedup_service.backend().backend_type().to_string();
let backend_str = match effective.backend {
crate::common::config::StorageBackendType::Local => "local",
crate::common::config::StorageBackendType::S3 => "s3",
crate::common::config::StorageBackendType::Azure => "azure",
};
Ok(StorageSettingsDto {
backend: backend_str.to_string(),
s3_endpoint_url: effective.s3.as_ref().and_then(|s| s.endpoint_url.clone()),
s3_bucket: effective.s3.as_ref().map(|s| s.bucket.clone()),
s3_region: effective.s3.as_ref().map(|s| s.region.clone()),
s3_access_key_set: has_access_key,
s3_secret_key_set: has_secret_key,
s3_force_path_style: effective.s3.as_ref().is_some_and(|s| s.force_path_style),
env_overrides: self.get_env_overrides(),
current_backend,
total_blobs: stats.total_blobs,
total_bytes_stored: stats.total_bytes_stored,
dedup_ratio: stats.dedup_ratio,
})
}
/// Save storage settings to DB.
pub async fn save_storage_settings(
&self,
dto: SaveStorageSettingsDto,
updated_by: Uuid,
) -> Result<(), DomainError> {
let cat = "storage";
let by = Some(updated_by);
self.settings_repo
.set("storage.backend", &dto.backend, cat, false, by)
.await?;
if let Some(ref v) = dto.s3_endpoint_url {
self.settings_repo
.set("storage.s3.endpoint_url", v, cat, false, by)
.await?;
}
if let Some(ref v) = dto.s3_bucket {
self.settings_repo
.set("storage.s3.bucket", v, cat, false, by)
.await?;
}
if let Some(ref v) = dto.s3_region {
self.settings_repo
.set("storage.s3.region", v, cat, false, by)
.await?;
}
if let Some(ref v) = dto.s3_access_key
&& !v.is_empty()
{
self.settings_repo
.set("storage.s3.access_key", v, cat, true, by)
.await?;
}
if let Some(ref v) = dto.s3_secret_key
&& !v.is_empty()
{
self.settings_repo
.set("storage.s3.secret_key", v, cat, true, by)
.await?;
}
if let Some(v) = dto.s3_force_path_style {
self.settings_repo
.set(
"storage.s3.force_path_style",
&v.to_string(),
cat,
false,
by,
)
.await?;
}
tracing::info!("Storage settings saved by admin (backend={})", dto.backend);
Ok(())
}
/// Test a storage connection by building a temporary backend and calling health_check().
pub async fn test_storage_connection(
&self,
dto: TestStorageConnectionDto,
) -> Result<StorageTestResultDto, DomainError> {
match dto.backend.as_str() {
"local" => {
// Test local backend health via the current dedup service backend
let status = self.dedup_service.backend().health_check().await?;
Ok(StorageTestResultDto {
connected: status.connected,
message: status.message,
backend_type: "local".to_string(),
available_bytes: status.available_bytes,
})
}
"s3" => {
let bucket = dto.s3_bucket.as_deref().unwrap_or_default();
if bucket.is_empty() {
return Ok(StorageTestResultDto {
connected: false,
message: "S3 bucket name is required".to_string(),
backend_type: "s3".to_string(),
available_bytes: None,
});
}
// Build a temporary S3 backend from the DTO values,
// falling back to existing DB/env config for missing fields.
let effective = self.load_effective_storage_config().await.ok();
let existing_s3 = effective.as_ref().and_then(|c| c.s3.as_ref());
let config = S3StorageConfig {
endpoint_url: dto
.s3_endpoint_url
.clone()
.or_else(|| existing_s3.and_then(|s| s.endpoint_url.clone())),
bucket: bucket.to_string(),
region: dto.s3_region.clone().unwrap_or_else(|| {
existing_s3
.map(|s| s.region.clone())
.unwrap_or_else(|| "us-east-1".to_string())
}),
access_key: dto
.s3_access_key
.clone()
.filter(|s| !s.is_empty())
.unwrap_or_else(|| {
existing_s3
.map(|s| s.access_key.clone())
.unwrap_or_default()
}),
secret_key: dto
.s3_secret_key
.clone()
.filter(|s| !s.is_empty())
.unwrap_or_else(|| {
existing_s3
.map(|s| s.secret_key.clone())
.unwrap_or_default()
}),
force_path_style: dto
.s3_force_path_style
.unwrap_or_else(|| existing_s3.is_some_and(|s| s.force_path_style)),
};
let backend = S3BlobBackend::new(&config);
match backend.health_check().await {
Ok(status) => Ok(StorageTestResultDto {
connected: status.connected,
message: status.message,
backend_type: "s3".to_string(),
available_bytes: status.available_bytes,
}),
Err(e) => Ok(StorageTestResultDto {
connected: false,
message: format!("Connection failed: {}", e),
backend_type: "s3".to_string(),
available_bytes: None,
}),
}
}
other => Err(DomainError::new(
ErrorKind::InvalidInput,
"Storage",
format!("Unknown backend type: {}", other),
)),
}
}
}
+216
View File
@@ -211,6 +211,127 @@ pub struct StorageConfig {
/// Maximum upload file size in bytes (default: 10 GB).
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
pub max_upload_size: usize,
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
pub backend: StorageBackendType,
/// S3-compatible backend configuration (used when `backend == S3`).
pub s3: Option<S3StorageConfig>,
/// Azure Blob Storage configuration (used when `backend == Azure`).
pub azure: Option<AzureStorageConfig>,
/// Local disk cache for remote backends.
pub cache: BlobCacheConfig,
/// Client-side encryption.
pub encryption: EncryptionConfig,
/// Retry policy for remote backends.
pub retry: RetryConfig,
}
/// Which blob storage backend to use.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum StorageBackendType {
/// Local filesystem (default).
#[default]
Local,
/// Any S3-compatible object store (AWS, Backblaze B2, R2, MinIO, …).
S3,
/// Azure Blob Storage.
Azure,
}
/// Configuration for an S3-compatible blob storage backend.
#[derive(Debug, Clone)]
pub struct S3StorageConfig {
/// Custom endpoint URL (required for non-AWS providers).
pub endpoint_url: Option<String>,
/// S3 bucket name.
pub bucket: String,
/// AWS region (default: `us-east-1`).
pub region: String,
/// Access key ID.
pub access_key: String,
/// Secret access key.
pub secret_key: String,
/// Force path-style access (required for MinIO, R2, some providers).
pub force_path_style: bool,
}
/// Configuration for Azure Blob Storage.
#[derive(Debug, Clone)]
pub struct AzureStorageConfig {
/// Azure storage account name.
pub account_name: String,
/// Azure storage account key.
pub account_key: String,
/// Container name.
pub container: String,
/// Optional SAS token (alternative to account key).
pub sas_token: Option<String>,
}
/// LRU local disk cache configuration for remote blob backends.
#[derive(Debug, Clone)]
pub struct BlobCacheConfig {
/// Enable the LRU disk cache (only useful for remote backends).
pub enabled: bool,
/// Maximum cache size in bytes (default: 50 GB).
pub max_size_bytes: u64,
/// Cache directory path (default: `{root_dir}/.blob-cache`).
pub cache_path: Option<String>,
}
impl Default for BlobCacheConfig {
fn default() -> Self {
Self {
enabled: false,
max_size_bytes: 50 * 1024 * 1024 * 1024, // 50 GB
cache_path: None,
}
}
}
/// Client-side encryption configuration.
#[derive(Debug, Clone)]
pub struct EncryptionConfig {
/// Enable AES-256-GCM encryption for blobs at rest.
pub enabled: bool,
/// Base64-encoded 32-byte encryption key.
pub key_base64: Option<String>,
}
impl Default for EncryptionConfig {
#[allow(clippy::derivable_impls)]
fn default() -> Self {
Self {
enabled: false,
key_base64: None,
}
}
}
/// Retry policy configuration for remote backends.
#[derive(Debug, Clone)]
pub struct RetryConfig {
/// Enable retry with exponential backoff.
pub enabled: bool,
/// Maximum number of retry attempts.
pub max_retries: u32,
/// Initial backoff in milliseconds.
pub initial_backoff_ms: u64,
/// Maximum backoff in milliseconds.
pub max_backoff_ms: u64,
/// Backoff multiplier.
pub backoff_multiplier: f64,
}
impl Default for RetryConfig {
fn default() -> Self {
Self {
enabled: true,
max_retries: 3,
initial_backoff_ms: 100,
max_backoff_ms: 10_000,
backoff_multiplier: 2.0,
}
}
}
impl Default for StorageConfig {
@@ -227,6 +348,12 @@ impl Default for StorageConfig {
parallel_threshold: 100 * 1024 * 1024, // 100 MB
trash_retention_days: 30, // 30 days
max_upload_size: MAX_UPLOAD_SIZE,
backend: StorageBackendType::Local,
s3: None,
azure: None,
cache: BlobCacheConfig::default(),
encryption: EncryptionConfig::default(),
retry: RetryConfig::default(),
}
}
}
@@ -828,6 +955,95 @@ impl AppConfig {
config.storage.max_upload_size = val;
}
// Storage backend selection
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
match backend.to_lowercase().as_str() {
"s3" => config.storage.backend = StorageBackendType::S3,
"azure" => config.storage.backend = StorageBackendType::Azure,
_ => config.storage.backend = StorageBackendType::Local,
}
}
// S3-compatible storage configuration
if config.storage.backend == StorageBackendType::S3 {
let bucket = env::var("OXICLOUD_S3_BUCKET").unwrap_or_default();
if bucket.is_empty() {
tracing::warn!("OXICLOUD_STORAGE_BACKEND=s3 but OXICLOUD_S3_BUCKET is not set");
}
config.storage.s3 = Some(S3StorageConfig {
endpoint_url: env::var("OXICLOUD_S3_ENDPOINT_URL").ok(),
bucket,
region: env::var("OXICLOUD_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
access_key: env::var("OXICLOUD_S3_ACCESS_KEY").unwrap_or_default(),
secret_key: env::var("OXICLOUD_S3_SECRET_KEY").unwrap_or_default(),
force_path_style: env::var("OXICLOUD_S3_FORCE_PATH_STYLE")
.map(|v| v.parse::<bool>().unwrap_or(false))
.unwrap_or(false),
});
}
// Azure Blob Storage configuration
if config.storage.backend == StorageBackendType::Azure {
let container = env::var("OXICLOUD_AZURE_CONTAINER").unwrap_or_default();
if container.is_empty() {
tracing::warn!(
"OXICLOUD_STORAGE_BACKEND=azure but OXICLOUD_AZURE_CONTAINER is not set"
);
}
config.storage.azure = Some(AzureStorageConfig {
account_name: env::var("OXICLOUD_AZURE_ACCOUNT_NAME").unwrap_or_default(),
account_key: env::var("OXICLOUD_AZURE_ACCOUNT_KEY").unwrap_or_default(),
container,
sas_token: env::var("OXICLOUD_AZURE_SAS_TOKEN").ok(),
});
}
// Blob cache configuration
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_ENABLED") {
config.storage.cache.enabled = v.parse::<bool>().unwrap_or(false);
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_MAX_SIZE")
&& let Ok(bytes) = v.parse::<u64>()
{
config.storage.cache.max_size_bytes = bytes;
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_CACHE_PATH") {
config.storage.cache.cache_path = Some(v);
}
// Encryption configuration
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_ENABLED") {
config.storage.encryption.enabled = v.parse::<bool>().unwrap_or(false);
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_ENCRYPTION_KEY") {
config.storage.encryption.key_base64 = Some(v);
}
// Retry configuration
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_ENABLED") {
config.storage.retry.enabled = v.parse::<bool>().unwrap_or(true);
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_RETRIES")
&& let Ok(n) = v.parse::<u32>()
{
config.storage.retry.max_retries = n;
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_INITIAL_BACKOFF_MS")
&& let Ok(n) = v.parse::<u64>()
{
config.storage.retry.initial_backoff_ms = n;
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_MAX_BACKOFF_MS")
&& let Ok(n) = v.parse::<u64>()
{
config.storage.retry.max_backoff_ms = n;
}
if let Ok(v) = env::var("OXICLOUD_STORAGE_RETRY_BACKOFF_MULTIPLIER")
&& let Ok(n) = v.parse::<f64>()
{
config.storage.retry.backoff_multiplier = n;
}
// OIDC configuration
if let Ok(v) = env::var("OXICLOUD_OIDC_ENABLED") {
config.oidc.enabled = v.parse::<bool>().unwrap_or(false);
+118 -1
View File
@@ -2,10 +2,14 @@ use sqlx::PgPool;
use std::path::PathBuf;
use std::sync::Arc;
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::StorageBackendType;
use crate::infrastructure::db::DbPools;
use crate::application::services::admin_settings_service::AdminSettingsService;
use crate::application::services::auth_application_service::AuthApplicationService;
use crate::application::services::storage_settings_service::StorageSettingsService;
use crate::infrastructure::services::migration_blob_backend::MigrationState;
use crate::application::ports::file_ports::FileUseCaseFactory;
use crate::application::services::favorites_service::FavoritesService;
@@ -153,10 +157,110 @@ impl AppServiceFactory {
);
image_transcode_service.initialize().await?;
// Build blob storage backend based on configuration
let base_backend: Arc<dyn BlobStorageBackend> = match self.config.storage.backend {
StorageBackendType::S3 => {
let s3_config = self
.config
.storage
.s3
.as_ref()
.expect("S3 config required when OXICLOUD_STORAGE_BACKEND=s3");
Arc::new(
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3_config),
)
}
StorageBackendType::Azure => {
let az_config = self
.config
.storage
.azure
.as_ref()
.expect("Azure config required when OXICLOUD_STORAGE_BACKEND=azure");
Arc::new(
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(
az_config,
),
)
}
StorageBackendType::Local => Arc::new(
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
&self.storage_path,
),
),
};
// Stack decorators: retry → encryption → cache (inner-to-outer)
let mut blob_backend: Arc<dyn BlobStorageBackend> = base_backend;
// Retry decorator (for remote backends)
if self.config.storage.retry.enabled
&& self.config.storage.backend != StorageBackendType::Local
{
use crate::infrastructure::services::retry_blob_backend::{
RetryBlobBackend, RetryPolicy,
};
let policy = RetryPolicy {
max_retries: self.config.storage.retry.max_retries,
initial_backoff: std::time::Duration::from_millis(
self.config.storage.retry.initial_backoff_ms,
),
max_backoff: std::time::Duration::from_millis(
self.config.storage.retry.max_backoff_ms,
),
backoff_multiplier: self.config.storage.retry.backoff_multiplier,
};
blob_backend = Arc::new(RetryBlobBackend::new(blob_backend, policy));
tracing::info!("Blob storage retry decorator enabled");
}
// Encryption decorator
if self.config.storage.encryption.enabled {
use crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend;
let key_b64 = self
.config
.storage
.encryption
.key_base64
.as_ref()
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY required when encryption is enabled");
let key_bytes =
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, key_b64)
.expect("OXICLOUD_STORAGE_ENCRYPTION_KEY must be valid base64");
let key: [u8; 32] = key_bytes.try_into().expect(
"OXICLOUD_STORAGE_ENCRYPTION_KEY must be exactly 32 bytes (base64 of 32 bytes)",
);
blob_backend = Arc::new(EncryptedBlobBackend::new(blob_backend, &key));
tracing::info!("Blob storage encryption decorator enabled (AES-256-GCM)");
}
// Cache decorator (for remote backends only)
if self.config.storage.cache.enabled
&& self.config.storage.backend != StorageBackendType::Local
{
use crate::infrastructure::services::cached_blob_backend::{
BlobCacheConfig as CacheCfg, CachedBlobBackend,
};
let cache_path = self
.config
.storage
.cache
.cache_path
.as_ref()
.map(std::path::PathBuf::from)
.unwrap_or_else(|| self.storage_path.join(".blob-cache"));
let cfg = CacheCfg {
cache_dir: cache_path,
max_cache_bytes: self.config.storage.cache.max_size_bytes,
};
blob_backend = Arc::new(CachedBlobBackend::new(blob_backend, &cfg));
tracing::info!("Blob storage LRU disk cache enabled");
}
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
let dedup_service = Arc::new(
crate::infrastructure::services::dedup_service::DedupService::new(
&self.storage_path,
blob_backend,
db_pool.clone(),
maintenance_pool.clone(),
),
@@ -631,6 +735,8 @@ impl AppServiceFactory {
auth_service: auth_services,
nextcloud: nextcloud_services,
admin_settings_service: None,
storage_settings_service: None,
migration_state: Arc::new(tokio::sync::RwLock::new(MigrationState::default())),
trash_service,
share_service,
favorites_service,
@@ -700,6 +806,15 @@ impl AppServiceFactory {
app_state.admin_settings_service = Some(admin_svc.clone());
// 9b-1b. Wire storage settings service (reuses same settings_repo)
let storage_settings_svc = Arc::new(StorageSettingsService::new(
settings_repo.clone(),
self.config.storage.clone(),
app_state.core.dedup_service.clone(),
));
app_state.storage_settings_service = Some(storage_settings_svc);
tracing::info!("Storage settings service initialized");
// 9b-2. Log whether system needs first-time admin setup
if !admin_svc.is_system_initialized().await {
tracing::warn!("╔══════════════════════════════════════════════════════════╗");
@@ -928,6 +1043,8 @@ pub struct AppState {
pub auth_service: Option<AuthServices>,
pub nextcloud: Option<NextcloudServices>,
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
pub storage_settings_service: Option<Arc<StorageSettingsService>>,
pub migration_state: Arc<tokio::sync::RwLock<MigrationState>>,
pub trash_service: Option<Arc<TrashService>>,
pub share_service: Option<Arc<ShareService>>,
pub favorites_service: Option<Arc<FavoritesService>>,
@@ -0,0 +1,302 @@
//! Azure Blob Storage Backend — stores blobs in an Azure Storage container.
//!
//! Authenticates via Account Name + Account Key (or SAS token).
//! Blob key scheme mirrors local/S3: `{2-char-prefix}/{hash}.blob`.
use std::path::{Path, PathBuf};
use std::pin::Pin;
use azure_storage::StorageCredentials;
use azure_storage_blobs::prelude::*;
use bytes::Bytes;
use futures::StreamExt;
use tokio::fs;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::common::config::AzureStorageConfig;
use crate::domain::errors::{DomainError, ErrorKind};
/// Azure Blob Storage backend.
pub struct AzureBlobBackend {
container_client: ContainerClient,
container_name: String,
}
impl AzureBlobBackend {
/// Build a new Azure backend from configuration.
pub fn new(config: &AzureStorageConfig) -> Self {
let credentials = if let Some(ref sas) = config.sas_token {
StorageCredentials::sas_token(sas).expect("Invalid SAS token")
} else {
StorageCredentials::access_key(&config.account_name, config.account_key.clone())
};
let container_client = ClientBuilder::new(&config.account_name, credentials)
.container_client(&config.container);
Self {
container_client,
container_name: config.container.clone(),
}
}
/// Compute the blob name for a given hash.
fn blob_name(hash: &str) -> String {
let prefix = &hash[0..2];
format!("{prefix}/{hash}.blob")
}
/// Get a `BlobClient` for a given hash.
fn blob_client(&self, hash: &str) -> BlobClient {
self.container_client.blob_client(Self::blob_name(hash))
}
}
impl BlobStorageBackend for AzureBlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
Box::pin(async move {
// Verify container exists by getting its properties
self.container_client.get_properties().await.map_err(|e| {
DomainError::internal_error(
"Azure",
format!("Cannot access container '{}': {}", self.container_name, e),
)
})?;
tracing::info!(
"Azure blob backend initialized: container={}",
self.container_name
);
Ok(())
})
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
let source_path = source_path.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
// Check if blob already exists (idempotent)
if client.get_properties().await.is_ok() {
let file_size = fs::metadata(&source_path)
.await
.map_err(|e| {
DomainError::internal_error(
"Azure",
format!("Failed to stat source file: {e}"),
)
})?
.len();
let _ = fs::remove_file(&source_path).await;
return Ok(file_size);
}
// Read file and upload as block blob
let data = fs::read(&source_path).await.map_err(|e| {
DomainError::internal_error("Azure", format!("Failed to read source: {e}"))
})?;
let file_size = data.len() as u64;
client.put_block_blob(data).await.map_err(|e| {
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
})?;
let _ = fs::remove_file(&source_path).await;
Ok(file_size)
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
let mut result_data: Vec<u8> = Vec::new();
let mut stream = client.get().into_stream();
while let Some(response) = stream.next().await {
let response = response.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to get blob {hash}: {e}"),
)
})?;
let mut body = response.data;
while let Some(chunk) = body.next().await {
let chunk = chunk.map_err(|e| {
DomainError::internal_error("Azure", format!("Stream read error: {e}"))
})?;
result_data.extend_from_slice(&chunk);
}
}
let stream: BlobStream = Box::pin(futures::stream::once(async move {
Ok(Bytes::from(result_data))
}));
Ok(stream)
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
let range = match end {
Some(e) => azure_core::request_options::Range::new(start, e),
None => azure_core::request_options::Range::new(start, u64::MAX),
};
let mut result_data: Vec<u8> = Vec::new();
let mut stream = client.get().range(range).into_stream();
while let Some(response) = stream.next().await {
let response = response.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to get blob range {hash}: {e}"),
)
})?;
let mut body = response.data;
while let Some(chunk) = body.next().await {
let chunk = chunk.map_err(|e| {
DomainError::internal_error(
"Azure",
format!("Stream range read error: {e}"),
)
})?;
result_data.extend_from_slice(&chunk);
}
}
let stream: BlobStream = Box::pin(futures::stream::once(async move {
Ok(Bytes::from(result_data))
}));
Ok(stream)
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
// Azure delete is not fully idempotent — 404 is expected for missing blobs
match client.delete().await {
Ok(_) => Ok(()),
Err(e) => {
// If 404, treat as success (idempotent)
let status = e.as_http_error().map(|h| h.status());
if status == Some(azure_core::StatusCode::NotFound) {
Ok(())
} else {
Err(DomainError::internal_error(
"Azure",
format!("Failed to delete blob {hash}: {e}"),
))
}
}
}
})
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
match client.get_properties().await {
Ok(_) => Ok(true),
Err(e) => {
let status = e.as_http_error().map(|h| h.status());
if status == Some(azure_core::StatusCode::NotFound) {
Ok(false)
} else {
Err(DomainError::internal_error(
"Azure",
format!("Failed to check blob {hash}: {e}"),
))
}
}
}
})
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
let props = client.get_properties().await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to stat blob {hash}: {e}"),
)
})?;
Ok(props.blob.properties.content_length)
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
Box::pin(async move {
match self.container_client.get_properties().await {
Ok(_) => Ok(StorageHealthStatus {
connected: true,
backend_type: "azure".to_string(),
message: format!("Azure container '{}' is accessible", self.container_name),
available_bytes: None,
}),
Err(e) => Ok(StorageHealthStatus {
connected: false,
backend_type: "azure".to_string(),
message: format!(
"Azure container '{}' is not accessible: {}",
self.container_name, e
),
available_bytes: None,
}),
}
})
}
fn backend_type(&self) -> &'static str {
"azure"
}
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
None
}
}
@@ -0,0 +1,472 @@
//! `CachedBlobBackend` — LRU local-disk cache decorator for remote blob backends.
//!
//! Wraps any `BlobStorageBackend` (typically S3 or Azure) and transparently
//! caches hot blobs on a local SSD. Reads check the cache first; cache misses
//! are fetched from the inner backend and written to the cache. Writes go to
//! the inner backend AND the local cache simultaneously.
//!
//! Eviction is LRU based on a configurable maximum disk budget.
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use lru::LruCache;
use std::num::NonZeroUsize;
use tokio::fs;
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
use tokio::sync::Mutex;
use tokio_util::io::ReaderStream;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::domain::errors::DomainError;
/// Chunk size for streaming cached file reads (256 KB).
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
// ── Configuration ──────────────────────────────────────────────────
/// Configuration for the LRU disk cache.
#[derive(Debug, Clone)]
pub struct BlobCacheConfig {
/// Directory where cached blobs are stored.
pub cache_dir: PathBuf,
/// Maximum total cache size in bytes.
pub max_cache_bytes: u64,
}
// ── Cache entry ────────────────────────────────────────────────────
#[derive(Debug, Clone)]
struct CacheEntry {
size: u64,
}
// ── CachedBlobBackend ──────────────────────────────────────────────
/// A `BlobStorageBackend` decorator that adds an LRU disk cache in front of
/// a remote backend.
pub struct CachedBlobBackend {
inner: Arc<dyn BlobStorageBackend>,
cache_dir: PathBuf,
max_cache_bytes: u64,
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
current_size: Arc<AtomicU64>,
}
impl CachedBlobBackend {
/// Create a new cached backend wrapping `inner`.
pub fn new(inner: Arc<dyn BlobStorageBackend>, config: &BlobCacheConfig) -> Self {
Self {
inner,
cache_dir: config.cache_dir.clone(),
max_cache_bytes: config.max_cache_bytes,
// Capacity is essentially unbounded — eviction is by byte budget, not count.
index: Arc::new(Mutex::new(LruCache::new(
NonZeroUsize::new(1_000_000).unwrap(),
))),
current_size: Arc::new(AtomicU64::new(0)),
}
}
/// Path where a blob is cached locally.
fn cached_path(&self, hash: &str) -> PathBuf {
let prefix = &hash[..2.min(hash.len())];
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
}
}
impl BlobStorageBackend for CachedBlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let cache_dir = self.cache_dir.clone();
let index = self.index.clone();
let current_size = self.current_size.clone();
Box::pin(async move {
inner.initialize().await?;
// Create cache dir structure (256 prefix dirs)
fs::create_dir_all(&cache_dir).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("mkdir cache_dir: {e}"))
})?;
// Scan existing cache to rebuild index
let mut total_bytes = 0u64;
let mut idx = index.lock().await;
if let Ok(mut read_dir) = fs::read_dir(&cache_dir).await {
while let Ok(Some(prefix_entry)) = read_dir.next_entry().await {
if !prefix_entry.path().is_dir() {
continue;
}
if let Ok(mut sub_dir) = fs::read_dir(prefix_entry.path()).await {
while let Ok(Some(entry)) = sub_dir.next_entry().await {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) == Some("blob")
&& let Some(stem) = path.file_stem().and_then(|s| s.to_str())
{
let size = fs::metadata(&path).await.map(|m| m.len()).unwrap_or(0);
idx.put(stem.to_string(), CacheEntry { size });
total_bytes += size;
}
}
}
}
}
drop(idx);
current_size.store(total_bytes, Ordering::Relaxed);
tracing::info!(
"Blob cache initialized: {} bytes in cache at {}",
total_bytes,
cache_dir.display()
);
Ok(())
})
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let hash = hash.to_string();
let source = source_path.to_path_buf();
let self_ref = CachedRef {
cache_dir: self.cache_dir.clone(),
max_cache_bytes: self.max_cache_bytes,
index: self.index.clone(),
current_size: self.current_size.clone(),
};
Box::pin(async move {
// Write to inner backend
let bytes = inner.put_blob(&hash, &source).await?;
// Also cache locally (best-effort)
let _ = self_ref.insert_into_cache_static(&hash, &source).await;
Ok(bytes)
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_string();
let cached = self.cached_path(&hash);
let index = self.index.clone();
let inner = self.inner.clone();
let cache_dir = self.cache_dir.clone();
let max_cache_bytes = self.max_cache_bytes;
let current_size = self.current_size.clone();
Box::pin(async move {
// Check cache
{
let mut idx = index.lock().await;
if idx.get(&hash).is_some() {
if let Ok(file) = fs::File::open(&cached).await {
let stream: BlobStream =
Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
return Ok(stream);
}
// Cache entry stale — remove
if let Some(entry) = idx.pop(&hash) {
current_size.fetch_sub(entry.size, Ordering::Relaxed);
}
}
}
// Cache miss — fetch from inner, spool to cache
let self_ref = CachedRef {
cache_dir,
max_cache_bytes,
index: index.clone(),
current_size: current_size.clone(),
};
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
let file = fs::File::open(&dest).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
})?;
let stream: BlobStream = Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE));
Ok(stream)
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_string();
let cached = self.cached_path(&hash);
let index = self.index.clone();
let inner = self.inner.clone();
let cache_dir = self.cache_dir.clone();
let max_cache_bytes = self.max_cache_bytes;
let current_size = self.current_size.clone();
Box::pin(async move {
// Try cache first
{
let mut idx = index.lock().await;
if idx.get(&hash).is_some() {
if let Ok(mut file) = fs::File::open(&cached).await {
file.seek(std::io::SeekFrom::Start(start))
.await
.map_err(|e| {
DomainError::internal_error("BlobCache", format!("seek: {e}"))
})?;
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
let limited = file.take(take_len);
let stream: BlobStream =
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
return Ok(stream);
}
if let Some(entry) = idx.pop(&hash) {
current_size.fetch_sub(entry.size, Ordering::Relaxed);
}
}
}
// Cache miss — fetch full blob into cache, then serve range
let self_ref = CachedRef {
cache_dir,
max_cache_bytes,
index: index.clone(),
current_size: current_size.clone(),
};
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
let mut file = fs::File::open(&dest)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
file.seek(std::io::SeekFrom::Start(start))
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("seek: {e}")))?;
let take_len = end.map(|e| e - start + 1).unwrap_or(u64::MAX);
let limited = file.take(take_len);
let stream: BlobStream =
Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE));
Ok(stream)
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let hash = hash.to_string();
let cached = self.cached_path(&hash);
let index = self.index.clone();
let current_size = self.current_size.clone();
Box::pin(async move {
inner.delete_blob(&hash).await?;
// Remove from cache
let mut idx = index.lock().await;
if let Some(entry) = idx.pop(&hash) {
current_size.fetch_sub(entry.size, Ordering::Relaxed);
}
let _ = fs::remove_file(&cached).await;
Ok(())
})
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let hash = hash.to_string();
let index = self.index.clone();
Box::pin(async move {
// Check cache first (fast)
{
let mut idx = index.lock().await;
if idx.get(&hash).is_some() {
return Ok(true);
}
}
inner.blob_exists(&hash).await
})
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let hash = hash.to_string();
let index = self.index.clone();
let cached = self.cached_path(&hash);
Box::pin(async move {
// Check cache
{
let mut idx = index.lock().await;
if let Some(entry) = idx.get(&hash) {
return Ok(entry.size);
}
}
// Fallback to cached file on disk (in case index was lost)
if let Ok(meta) = fs::metadata(&cached).await {
return Ok(meta.len());
}
inner.blob_size(&hash).await
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
let inner = self.inner.clone();
let cache_dir = self.cache_dir.clone();
let current_size = self.current_size.clone();
let max_bytes = self.max_cache_bytes;
Box::pin(async move {
let mut status = inner.health_check().await?;
let used = current_size.load(Ordering::Relaxed);
status.message = format!(
"{} | Cache: {}/{} bytes used at {}",
status.message,
used,
max_bytes,
cache_dir.display()
);
status.backend_type = format!("cached({})", status.backend_type);
Ok(status)
})
}
fn backend_type(&self) -> &'static str {
"cached"
}
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
// If the blob is cached locally, return that path
let path = self.cached_path(hash);
if path.exists() { Some(path) } else { None }
}
}
// ── Helper struct for owned references in async closures ───────────
/// Cloneable set of cache internals — avoids borrow issues in boxed futures.
struct CachedRef {
cache_dir: PathBuf,
max_cache_bytes: u64,
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
current_size: Arc<AtomicU64>,
}
impl CachedRef {
fn cached_path(&self, hash: &str) -> PathBuf {
let prefix = &hash[..2.min(hash.len())];
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
}
async fn insert_into_cache_static(
&self,
hash: &str,
source_path: &Path,
) -> Result<(), DomainError> {
let dest = self.cached_path(hash);
if let Some(parent) = dest.parent() {
fs::create_dir_all(parent).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
})?;
}
let size = fs::metadata(source_path)
.await
.map(|m| m.len())
.unwrap_or(0);
fs::copy(source_path, &dest).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("cache copy failed: {e}"))
})?;
let mut idx = self.index.lock().await;
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size }) {
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
}
self.current_size.fetch_add(size, Ordering::Relaxed);
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
self.current_size
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
let evicted_path = self.cached_path(&evicted_hash);
let _ = fs::remove_file(&evicted_path).await;
} else {
break;
}
}
Ok(())
}
async fn fetch_and_cache_static(
&self,
hash: &str,
inner: &dyn BlobStorageBackend,
) -> Result<PathBuf, DomainError> {
let stream = inner.get_blob_stream(hash).await?;
let dest = self.cached_path(hash);
if let Some(parent) = dest.parent() {
fs::create_dir_all(parent).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("mkdir failed: {e}"))
})?;
}
let tmp = dest.with_extension("tmp");
let mut file = fs::File::create(&tmp)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
use futures::StreamExt;
let mut stream = stream;
let mut total = 0u64;
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| {
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
})?;
total += bytes.len() as u64;
file.write_all(&bytes)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
}
file.flush()
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
drop(file);
fs::rename(&tmp, &dest)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
let mut idx = self.index.lock().await;
if let Some(old) = idx.put(hash.to_string(), CacheEntry { size: total }) {
self.current_size.fetch_sub(old.size, Ordering::Relaxed);
}
self.current_size.fetch_add(total, Ordering::Relaxed);
while self.current_size.load(Ordering::Relaxed) > self.max_cache_bytes {
if let Some((evicted_hash, evicted_entry)) = idx.pop_lru() {
self.current_size
.fetch_sub(evicted_entry.size, Ordering::Relaxed);
let evicted_path = self.cached_path(&evicted_hash);
let _ = fs::remove_file(&evicted_path).await;
} else {
break;
}
}
Ok(dest)
}
}
+80 -211
View File
@@ -39,24 +39,22 @@ use sqlx::PgPool;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::sync::Arc;
use tokio::fs::{self, File};
use tokio::io::{AsyncReadExt, AsyncSeekExt};
use tokio_util::io::ReaderStream;
use tokio::fs;
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::application::ports::dedup_ports::{
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
};
use crate::domain::errors::{DomainError, ErrorKind};
/// Chunk size for streaming file reads (256 KB)
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
/// Content-Addressable Storage Service (PostgreSQL-backed)
///
/// Delegates all byte-level I/O to a [`BlobStorageBackend`] implementation
/// (local filesystem, S3, etc.) while keeping BLAKE3 hashing, ref-counting
/// and the PostgreSQL dedup index here.
pub struct DedupService {
/// Root directory for blob storage on the filesystem
blob_root: PathBuf,
/// Root directory for temporary files during upload
temp_root: PathBuf,
/// Pluggable blob storage backend (local FS, S3, …).
backend: Arc<dyn BlobStorageBackend>,
/// PostgreSQL connection pool (dedup index in `storage.blobs`) — primary,
/// used by request-path operations (store_from_file, etc.).
pool: Arc<PgPool>,
@@ -65,39 +63,19 @@ pub struct DedupService {
maintenance_pool: Arc<PgPool>,
}
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
/// Avoids a `format!("{:02x}", i)` allocation on every iteration of `initialize()`.
static HEX_PREFIXES: [&str; 256] = [
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
];
impl DedupService {
/// Create a new dedup service backed by PostgreSQL.
///
/// * `backend` — pluggable blob storage (local filesystem, S3, etc.).
/// * `pool` — primary pool for request-path operations.
/// * `maintenance_pool` — isolated pool for verify_integrity / garbage_collect.
pub fn new(storage_root: &Path, pool: Arc<PgPool>, maintenance_pool: Arc<PgPool>) -> Self {
let blob_root = storage_root.join(".blobs");
let temp_root = storage_root.join(".dedup_temp");
pub fn new(
backend: Arc<dyn BlobStorageBackend>,
pool: Arc<PgPool>,
maintenance_pool: Arc<PgPool>,
) -> Self {
Self {
blob_root,
temp_root,
backend,
pool,
maintenance_pool,
}
@@ -106,6 +84,7 @@ impl DedupService {
/// Creates a stub instance for testing — never hits PG or the filesystem.
#[cfg(any(test, feature = "integration_tests"))]
pub fn new_stub() -> Self {
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
let stub_pool = Arc::new(
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
.max_connections(1)
@@ -113,29 +92,15 @@ impl DedupService {
.unwrap(),
);
Self {
blob_root: std::path::PathBuf::from("/tmp/oxicloud_stub_blobs"),
temp_root: std::path::PathBuf::from("/tmp/oxicloud_stub_temp"),
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
pool: stub_pool.clone(),
maintenance_pool: stub_pool,
}
}
/// Initialize the service (create blob directories on the filesystem).
/// Initialize the service (delegate to backend + log stats from PG).
pub async fn initialize(&self) -> Result<(), DomainError> {
// Create directories
fs::create_dir_all(&self.blob_root)
.await
.map_err(DomainError::from)?;
fs::create_dir_all(&self.temp_root)
.await
.map_err(DomainError::from)?;
// Create hash prefix directories (00-ff)
for prefix in &HEX_PREFIXES {
fs::create_dir_all(self.blob_root.join(prefix))
.await
.map_err(DomainError::from)?;
}
self.backend.initialize().await?;
// Log existing blob stats from PG
let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
@@ -150,7 +115,8 @@ impl DedupService {
.unwrap_or(0);
tracing::info!(
"Dedup service initialized (PostgreSQL-backed): {} blobs, {} bytes stored",
"Dedup service initialized (backend={}): {} blobs, {} bytes stored",
self.backend.backend_type(),
count,
total_bytes
);
@@ -158,12 +124,18 @@ impl DedupService {
Ok(())
}
/// Return a reference to the underlying blob storage backend.
pub fn backend(&self) -> &Arc<dyn BlobStorageBackend> {
&self.backend
}
// ── Path helpers ─────────────────────────────────────────────
/// Get the blob path for a given hash.
/// Get the local blob path for a given hash (if the backend supports it).
pub fn blob_path(&self, hash: &str) -> PathBuf {
let prefix = &hash[0..2];
self.blob_root.join(prefix).join(format!("{}.blob", hash))
self.backend
.local_blob_path(hash)
.unwrap_or_else(|| PathBuf::from(format!("remote://{}", hash)))
}
// ── Hash helpers ─────────────────────────────────────────────
@@ -193,9 +165,9 @@ impl DedupService {
/// Store content with deduplication (streaming from file).
///
/// **Write-first strategy**: the source file is moved/copied to the
/// blob store *before* touching PostgreSQL, so the PG connection is
/// never held during disk I/O.
/// **Write-first strategy**: the source file is moved/uploaded to the
/// blob backend *before* touching PostgreSQL, so the PG connection is
/// never held during I/O.
///
/// If `pre_computed_hash` is `Some`, the file will NOT be re-read for
/// BLAKE3 — saving one full sequential read (the biggest I/O win).
@@ -205,13 +177,6 @@ impl DedupService {
content_type: Option<String>,
pre_computed_hash: Option<String>,
) -> Result<DedupResultDto, DomainError> {
let file_size = fs::metadata(source_path)
.await
.map_err(|e| {
DomainError::internal_error("Dedup", format!("Failed to get file metadata: {}", e))
})?
.len();
// Use pre-computed hash if available, otherwise calculate (streaming)
let hash = match pre_computed_hash {
Some(h) => h,
@@ -220,44 +185,11 @@ impl DedupService {
.map_err(DomainError::from)?,
};
// ── Phase 1: Place blob in backend (NO PG connection held) ───
let file_size = self.backend.put_blob(&hash, source_path).await?;
let blob_path = self.blob_path(&hash);
// ── Phase 1: Move/place blob on disk (NO PG connection held) ─
//
// If the blob file already exists on disk, the source is simply
// deleted — the file content is identical by definition.
if fs::try_exists(&blob_path).await.unwrap_or(false) {
// Blob already on disk — discard the source file
let _ = fs::remove_file(source_path).await;
} else {
// Parent directory (xx/) guaranteed to exist — created by initialize()
// rename is atomic on the same filesystem. If source and blob
// dirs live on different filesystems (rare), this falls back to
// copy+delete which is slower but still correct.
if let Err(e) = fs::rename(source_path, &blob_path).await {
if e.raw_os_error() == Some(18) {
// EXDEV: cross-device link — fall back to copy+delete
fs::copy(source_path, &blob_path).await.map_err(|ce| {
DomainError::internal_error(
"Dedup",
format!("Failed to copy file to blob store: {}", ce),
)
})?;
let _ = fs::remove_file(source_path).await;
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
// Another writer may have placed the blob concurrently
let _ = fs::remove_file(source_path).await;
tracing::debug!("Blob file placed by concurrent writer: {}", e);
} else {
return Err(DomainError::internal_error(
"Dedup",
format!("Failed to move file to blob store: {}", e),
));
}
}
}
// ── Phase 2: Single atomic upsert (~2-4 ms, no explicit TX) ─
let ref_count: i32 = sqlx::query_scalar(
"INSERT INTO storage.blobs (hash, size, ref_count, content_type)
@@ -415,10 +347,9 @@ impl DedupService {
DomainError::internal_error("Dedup", format!("Failed to commit: {}", e))
})?;
// Delete blob file AFTER committing PG — the row is gone, so no
// concurrent store_from_file can resurrect a reference to this hash.
let blob_path = self.blob_path(hash);
if let Err(e) = fs::remove_file(&blob_path).await {
// Delete blob from backend AFTER committing PG — the row is gone,
// so no concurrent store_from_file can resurrect a reference.
if let Err(e) = self.backend.delete_blob(hash).await {
tracing::warn!("Failed to delete blob file {}: {}", hash, e);
}
@@ -449,31 +380,16 @@ impl DedupService {
// ── Read operations ──────────────────────────────────────────
/// Stream blob content in 64 KB chunks — constant memory (~64 KB per stream).
///
/// A 1 GB file uses the same ~64 KB as a 1 KB file.
/// Stream blob content in chunks — constant memory usage.
pub async fn read_blob_stream(
&self,
hash: &str,
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
{
let blob_path = self.blob_path(hash);
let file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
Ok(Box::pin(ReaderStream::with_capacity(
file,
STREAM_CHUNK_SIZE,
)))
self.backend.get_blob_stream(hash).await
}
/// Stream a byte range of a blob — only reads the requested portion.
///
/// Uses seek + take so a 1 MB range request on a 1 GB file only reads 1 MB.
pub async fn read_blob_range_stream(
&self,
hash: &str,
@@ -481,49 +397,12 @@ impl DedupService {
end: Option<u64>,
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
{
let blob_path = self.blob_path(hash);
let mut file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
// Seek to the start position
file.seek(std::io::SeekFrom::Start(start))
.await
.map_err(|e| {
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
})?;
// If an end is specified, limit the read with take()
if let Some(end_pos) = end {
let limit = end_pos.saturating_sub(start);
let limited = file.take(limit);
Ok(Box::pin(ReaderStream::with_capacity(
limited,
STREAM_CHUNK_SIZE,
)))
} else {
Ok(Box::pin(ReaderStream::with_capacity(
file,
STREAM_CHUNK_SIZE,
)))
}
self.backend.get_blob_range_stream(hash, start, end).await
}
/// Get the size of a blob without reading its content.
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
let blob_path = self.blob_path(hash);
let meta = fs::metadata(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to stat blob {}: {}", hash, e),
)
})?;
Ok(meta.len())
self.backend.blob_size(hash).await
}
// ── Statistics (computed from PG) ────────────────────────────
@@ -596,56 +475,46 @@ impl DedupService {
// Flush when batch is full or we've exhausted the cursor
if batch.len() >= VERIFY_CONCURRENCY || (is_done && !batch.is_empty()) {
let blob_root = self.blob_root.clone();
let backend = self.backend.clone();
let current_batch =
std::mem::replace(&mut batch, Vec::with_capacity(VERIFY_CONCURRENCY));
let issues: Vec<String> = stream::iter(current_batch)
.map(move |(hash, expected_size)| {
let blob_root = blob_root.clone();
let backend = backend.clone();
async move {
let prefix = &hash[0..2];
let blob_path = blob_root.join(prefix).join(format!("{}.blob", hash));
let mut issues = Vec::new();
// Single async metadata() replaces the previous
// blocking .exists() + separate metadata() — one
// stat() syscall instead of two, and non-blocking.
let file_meta = match fs::metadata(&blob_path).await {
Ok(m) => m,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
issues.push(format!("{}: file missing on disk", hash));
return issues;
// Check existence + size via backend
match backend.blob_size(&hash).await {
Ok(actual_size) => {
if actual_size != expected_size as u64 {
issues.push(format!(
"{}: size mismatch (expected: {}, actual: {})",
hash, expected_size, actual_size,
));
}
}
Err(e) => {
issues.push(format!("{}: metadata error ({})", hash, e));
Err(_) => {
issues.push(format!("{}: blob missing in backend", hash));
return issues;
}
};
// Check size
if file_meta.len() != expected_size as u64 {
issues.push(format!(
"{}: size mismatch (expected: {}, actual: {})",
hash,
expected_size,
file_meta.len(),
));
}
// Verify hash
match Self::hash_file(&blob_path).await {
Ok(actual_hash) => {
if actual_hash != hash {
issues.push(format!(
"{}: hash mismatch (actual: {})",
hash, actual_hash,
));
// Verify hash — only possible for local backends
if let Some(blob_path) = backend.local_blob_path(&hash) {
match Self::hash_file(&blob_path).await {
Ok(actual_hash) => {
if actual_hash != hash {
issues.push(format!(
"{}: hash mismatch (actual: {})",
hash, actual_hash,
));
}
}
Err(e) => {
issues.push(format!("{}: read error ({})", hash, e));
}
}
Err(e) => {
issues.push(format!("{}: read error ({})", hash, e));
}
}
@@ -717,20 +586,20 @@ impl DedupService {
// Also clean up any thumbnail files for these blob hashes
// (thumbnails are keyed by blob_hash and live under
// storage_root/.thumbnails/{icon,preview,large}/{hash}.jpg).
let thumbnails_root = self
.blob_root
.parent()
.unwrap_or(&self.blob_root)
.join(".thumbnails");
for (hash, size) in &batch {
let blob_path = self.blob_path(hash);
if let Err(e) = fs::remove_file(&blob_path).await {
if let Err(e) = self.backend.delete_blob(hash).await {
tracing::warn!("Failed to delete orphan blob file {hash}: {e}");
}
// Remove associated thumbnail files (best-effort)
for dir in &["icon", "preview", "large"] {
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
let _ = fs::remove_file(&thumb).await;
// Remove associated thumbnail files (best-effort, always local)
if let Some(blob_path) = self.backend.local_blob_path(hash)
&& let Some(storage_root) = blob_path.ancestors().nth(3)
{
let thumbnails_root = storage_root.join(".thumbnails");
for dir in &["icon", "preview", "large"] {
let thumb = thumbnails_root.join(dir).join(format!("{hash}.jpg"));
let _ = fs::remove_file(&thumb).await;
}
}
total_bytes += *size as u64;
}
@@ -0,0 +1,298 @@
//! `EncryptedBlobBackend` — AES-256-GCM encryption decorator for blob storage.
//!
//! Transparently encrypts blobs before they reach the inner backend and
//! decrypts them on read. Each blob gets a random 96-bit nonce which is
//! prepended to the ciphertext. The GCM authentication tag (16 bytes) is
//! appended by the cipher.
//!
//! **IMPORTANT**: BLAKE3 hashing is performed on the *plaintext* by
//! `DedupService` before this layer sees the blob, so content-addressable
//! dedup still works correctly.
//!
//! Layout on disk/S3: `[12-byte nonce][ciphertext + 16-byte GCM tag]`
use std::path::{Path, PathBuf};
use std::pin::Pin;
use aes_gcm::aead::{Aead, KeyInit, OsRng};
use aes_gcm::{AeadCore, Aes256Gcm, Nonce};
use bytes::Bytes;
use std::sync::Arc;
use tokio::fs;
use tokio::io::AsyncWriteExt;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::domain::errors::DomainError;
/// Nonce size for AES-256-GCM (96 bits = 12 bytes).
const NONCE_SIZE: usize = 12;
/// `BlobStorageBackend` decorator that encrypts blobs at rest.
pub struct EncryptedBlobBackend {
inner: Arc<dyn BlobStorageBackend>,
cipher: Aes256Gcm,
}
impl EncryptedBlobBackend {
/// Create a new encryption layer wrapping `inner`.
///
/// `key` must be exactly 32 bytes (AES-256).
pub fn new(inner: Arc<dyn BlobStorageBackend>, key: &[u8; 32]) -> Self {
let cipher = Aes256Gcm::new_from_slice(key).expect("AES-256 key must be 32 bytes");
Self { inner, cipher }
}
/// Generate a random 32-byte key suitable for AES-256.
pub fn generate_key() -> [u8; 32] {
use aes_gcm::aead::rand_core::RngCore;
let mut key = [0u8; 32];
OsRng.fill_bytes(&mut key);
key
}
}
impl BlobStorageBackend for EncryptedBlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
self.inner.initialize()
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let hash = hash.to_string();
let source = source_path.to_path_buf();
// Clone cipher key material (Aes256Gcm is not Send-safe to move across await)
let cipher = self.cipher.clone();
Box::pin(async move {
// Read plaintext from source
let plaintext = fs::read(&source).await.map_err(|e| {
DomainError::internal_error("Encryption", format!("read source: {e}"))
})?;
// Encrypt: nonce || ciphertext (includes GCM tag)
let nonce = Aes256Gcm::generate_nonce(&mut OsRng);
let ciphertext = cipher.encrypt(&nonce, plaintext.as_ref()).map_err(|e| {
DomainError::internal_error("Encryption", format!("encrypt failed: {e}"))
})?;
// Write encrypted blob to a temp file
let tmp = source.with_extension("enc.tmp");
let mut file = fs::File::create(&tmp).await.map_err(|e| {
DomainError::internal_error("Encryption", format!("create tmp: {e}"))
})?;
file.write_all(nonce.as_slice()).await.map_err(|e| {
DomainError::internal_error("Encryption", format!("write nonce: {e}"))
})?;
file.write_all(&ciphertext).await.map_err(|e| {
DomainError::internal_error("Encryption", format!("write ciphertext: {e}"))
})?;
file.flush()
.await
.map_err(|e| DomainError::internal_error("Encryption", format!("flush: {e}")))?;
drop(file);
let result = inner.put_blob(&hash, &tmp).await;
let _ = fs::remove_file(&tmp).await;
result
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let inner = self.inner.clone();
let hash = hash.to_string();
let cipher = self.cipher.clone();
Box::pin(async move {
// Read entire encrypted blob (nonce + ciphertext) into memory for decryption
let enc_stream = inner.get_blob_stream(&hash).await?;
let encrypted = collect_stream(enc_stream).await?;
if encrypted.len() < NONCE_SIZE {
return Err(DomainError::internal_error(
"Encryption",
"encrypted blob too short (missing nonce)",
));
}
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
let nonce = Nonce::from_slice(nonce_bytes);
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
})?;
let stream: BlobStream =
Box::pin(futures::stream::once(
async move { Ok(Bytes::from(plaintext)) },
));
Ok(stream)
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let inner = self.inner.clone();
let hash = hash.to_string();
let cipher = self.cipher.clone();
Box::pin(async move {
// Must decrypt the full blob then slice the plaintext range
let enc_stream = inner.get_blob_stream(&hash).await?;
let encrypted = collect_stream(enc_stream).await?;
if encrypted.len() < NONCE_SIZE {
return Err(DomainError::internal_error(
"Encryption",
"encrypted blob too short",
));
}
let (nonce_bytes, ciphertext) = encrypted.split_at(NONCE_SIZE);
let nonce = Nonce::from_slice(nonce_bytes);
let plaintext = cipher.decrypt(nonce, ciphertext).map_err(|e| {
DomainError::internal_error("Encryption", format!("decrypt failed: {e}"))
})?;
let start = start as usize;
let end = end.map(|e| (e as usize) + 1).unwrap_or(plaintext.len());
let end = end.min(plaintext.len());
let start = start.min(end);
let slice = Bytes::from(plaintext[start..end].to_vec());
let stream: BlobStream = Box::pin(futures::stream::once(async move { Ok(slice) }));
Ok(stream)
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
self.inner.delete_blob(hash)
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
self.inner.blob_exists(hash)
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
// The stored size includes nonce + GCM tag overhead.
// Return the *plaintext* size by subtracting overhead.
let inner = self.inner.clone();
let hash = hash.to_string();
Box::pin(async move {
let encrypted_size = inner.blob_size(&hash).await?;
// overhead = 12 (nonce) + 16 (GCM tag) = 28 bytes
Ok(encrypted_size.saturating_sub(28))
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
let inner = self.inner.clone();
Box::pin(async move {
let mut status = inner.health_check().await?;
status.backend_type = format!("encrypted({})", status.backend_type);
status.message = format!("{} | Encryption: AES-256-GCM", status.message);
Ok(status)
})
}
fn backend_type(&self) -> &'static str {
"encrypted"
}
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
// Encrypted blobs cannot be served directly from disk
None
}
}
/// Collect a byte stream into a single `Vec<u8>`.
async fn collect_stream(stream: BlobStream) -> Result<Vec<u8>, DomainError> {
use futures::StreamExt;
let mut stream = stream;
let mut buf = Vec::new();
while let Some(chunk) = stream.next().await {
let bytes = chunk
.map_err(|e| DomainError::internal_error("Encryption", format!("stream read: {e}")))?;
buf.extend_from_slice(&bytes);
}
Ok(buf)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::infrastructure::services::local_blob_backend::LocalBlobBackend;
use tempfile::TempDir;
use tokio::io::AsyncWriteExt;
#[tokio::test]
async fn test_encrypt_decrypt_roundtrip() {
let tmp = TempDir::new().unwrap();
let blob_dir = tmp.path().join("blobs");
let local = Arc::new(LocalBlobBackend::new(&blob_dir));
local.initialize().await.unwrap();
let key = EncryptedBlobBackend::generate_key();
let encrypted = EncryptedBlobBackend::new(local, &key);
// Write a test blob
let data = b"Hello, encrypted world!";
let source = tmp.path().join("test.tmp");
let mut f = fs::File::create(&source).await.unwrap();
f.write_all(data).await.unwrap();
f.flush().await.unwrap();
drop(f);
let hash = "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890";
encrypted.put_blob(hash, &source).await.unwrap();
// Read back via stream
let stream = encrypted.get_blob_stream(hash).await.unwrap();
let decrypted = collect_stream(stream).await.unwrap();
assert_eq!(decrypted, data);
// Read range
let range_stream = encrypted
.get_blob_range_stream(hash, 7, Some(15))
.await
.unwrap();
let range_data = collect_stream(range_stream).await.unwrap();
assert_eq!(range_data, b"encrypted");
// Size should reflect plaintext
let size = encrypted.blob_size(hash).await.unwrap();
assert_eq!(size, data.len() as u64);
// Exists
assert!(encrypted.blob_exists(hash).await.unwrap());
// Delete
encrypted.delete_blob(hash).await.unwrap();
assert!(!encrypted.blob_exists(hash).await.unwrap());
}
}
@@ -0,0 +1,277 @@
//! Local Filesystem Blob Backend — stores blobs under `.blobs/{prefix}/{hash}.blob`.
//!
//! This is the default backend and a direct extraction of the filesystem I/O
//! that previously lived inside `DedupService`.
use std::path::{Path, PathBuf};
use std::pin::Pin;
use tokio::fs::{self, File};
use tokio::io::AsyncSeekExt;
use tokio_util::io::ReaderStream;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::domain::errors::{DomainError, ErrorKind};
/// Chunk size for streaming file reads (256 KB).
const STREAM_CHUNK_SIZE: usize = 256 * 1024;
/// Compile-time lookup table for the 256 two-digit lowercase hex prefixes ("00"…"ff").
static HEX_PREFIXES: [&str; 256] = [
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f",
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f",
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f",
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f",
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f",
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f",
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f",
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f",
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f",
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f",
"a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af",
"b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf",
"c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf",
"d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df",
"e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef",
"f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff",
];
/// Local filesystem blob backend.
///
/// Blobs are stored under `blob_root/{2-char-prefix}/{hash}.blob`.
/// Temporary upload staging uses `temp_root/`.
pub struct LocalBlobBackend {
blob_root: PathBuf,
temp_root: PathBuf,
}
impl LocalBlobBackend {
/// Create a new local backend rooted at `storage_root`.
///
/// Blob files go under `{storage_root}/.blobs/`, temp files under
/// `{storage_root}/.dedup_temp/`.
pub fn new(storage_root: &Path) -> Self {
Self {
blob_root: storage_root.join(".blobs"),
temp_root: storage_root.join(".dedup_temp"),
}
}
/// Compute the filesystem path for a blob hash.
pub fn blob_path(&self, hash: &str) -> PathBuf {
let prefix = &hash[0..2];
self.blob_root.join(prefix).join(format!("{}.blob", hash))
}
/// Return a reference to the blob root directory.
pub fn blob_root(&self) -> &Path {
&self.blob_root
}
}
impl BlobStorageBackend for LocalBlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
Box::pin(async move {
fs::create_dir_all(&self.blob_root)
.await
.map_err(DomainError::from)?;
fs::create_dir_all(&self.temp_root)
.await
.map_err(DomainError::from)?;
// Create the 256 hash-prefix directories (00-ff)
for prefix in &HEX_PREFIXES {
fs::create_dir_all(self.blob_root.join(prefix))
.await
.map_err(DomainError::from)?;
}
Ok(())
})
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
let source_path = source_path.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let file_size = fs::metadata(&source_path)
.await
.map_err(|e| {
DomainError::internal_error(
"Blob",
format!("Failed to stat source file: {}", e),
)
})?
.len();
// Idempotent: if blob already exists, just remove the source
if fs::try_exists(&blob_path).await.unwrap_or(false) {
let _ = fs::remove_file(&source_path).await;
return Ok(file_size);
}
// Atomic rename (same filesystem). Falls back to copy+delete for
// cross-device moves (EXDEV errno 18).
if let Err(e) = fs::rename(&source_path, &blob_path).await {
if e.raw_os_error() == Some(18) {
// EXDEV — cross-device link
fs::copy(&source_path, &blob_path).await.map_err(|ce| {
DomainError::internal_error(
"Blob",
format!("Failed to copy file to blob store: {}", ce),
)
})?;
let _ = fs::remove_file(&source_path).await;
} else if fs::try_exists(&blob_path).await.unwrap_or(false) {
// Concurrent writer placed the blob — discard our copy
let _ = fs::remove_file(&source_path).await;
tracing::debug!("Blob placed by concurrent writer: {}", e);
} else {
return Err(DomainError::internal_error(
"Blob",
format!("Failed to move file to blob store: {}", e),
));
}
}
Ok(file_size)
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let mut file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
file.seek(std::io::SeekFrom::Start(start))
.await
.map_err(|e| {
DomainError::internal_error("Blob", format!("Failed to seek in blob: {}", e))
})?;
if let Some(end_pos) = end {
use tokio::io::AsyncReadExt;
let limit = end_pos.saturating_sub(start);
let limited = file.take(limit);
Ok(Box::pin(ReaderStream::with_capacity(limited, STREAM_CHUNK_SIZE)) as BlobStream)
} else {
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
}
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
match fs::remove_file(&blob_path).await {
Ok(()) => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), // idempotent
Err(e) => Err(DomainError::internal_error(
"Blob",
format!("Failed to delete blob {}: {}", hash, e),
)),
}
})
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
Ok(fs::try_exists(&blob_path).await.unwrap_or(false))
})
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let meta = fs::metadata(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to stat blob {}: {}", hash, e),
)
})?;
Ok(meta.len())
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
Box::pin(async move {
let writable = fs::metadata(&self.blob_root).await.is_ok();
Ok(StorageHealthStatus {
connected: writable,
backend_type: "local".to_string(),
message: if writable {
"Local filesystem is accessible".to_string()
} else {
"Blob root directory is not accessible".to_string()
},
available_bytes: None,
})
})
}
fn backend_type(&self) -> &'static str {
"local"
}
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
Some(self.blob_path(hash))
}
}
@@ -0,0 +1,204 @@
//! `MigrationBlobBackend` — decorator that enables zero-downtime migration
//! between blob storage backends.
//!
//! During a migration the decorator writes to the **target** backend and reads
//! from **target-first-then-source** (dual-read). A background job
//! (see `migration_job.rs`) copies remaining blobs in the background.
use std::future::Future;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::sync::Arc;
use chrono::{DateTime, Utc};
use serde::Serialize;
use tokio::sync::RwLock;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::common::errors::DomainError;
// ── Migration state ────────────────────────────────────────────────
/// Progress of an ongoing (or completed) backend migration.
#[derive(Debug, Clone, Serialize)]
pub struct MigrationState {
pub status: MigrationStatus,
pub total_blobs: u64,
pub migrated_blobs: u64,
pub migrated_bytes: u64,
pub failed_blobs: Vec<String>,
pub started_at: Option<DateTime<Utc>>,
pub completed_at: Option<DateTime<Utc>>,
}
impl Default for MigrationState {
fn default() -> Self {
Self {
status: MigrationStatus::Idle,
total_blobs: 0,
migrated_blobs: 0,
migrated_bytes: 0,
failed_blobs: Vec::new(),
started_at: None,
completed_at: None,
}
}
}
/// Status of the migration job.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum MigrationStatus {
Idle,
Running,
Paused,
Completed,
Failed,
}
// ── MigrationBlobBackend ───────────────────────────────────────────
/// A `BlobStorageBackend` decorator that proxies requests to a *source*
/// (old) and *target* (new) backend, enabling live migration.
pub struct MigrationBlobBackend {
source: Arc<dyn BlobStorageBackend>,
target: Arc<dyn BlobStorageBackend>,
state: Arc<RwLock<MigrationState>>,
}
impl MigrationBlobBackend {
pub fn new(
source: Arc<dyn BlobStorageBackend>,
target: Arc<dyn BlobStorageBackend>,
state: Arc<RwLock<MigrationState>>,
) -> Self {
Self {
source,
target,
state,
}
}
pub fn state(&self) -> &Arc<RwLock<MigrationState>> {
&self.state
}
pub fn source(&self) -> &Arc<dyn BlobStorageBackend> {
&self.source
}
pub fn target(&self) -> &Arc<dyn BlobStorageBackend> {
&self.target
}
}
/// Boxed future alias (same as in the trait module).
type BoxFut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
impl BlobStorageBackend for MigrationBlobBackend {
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>> {
Box::pin(async move {
self.target.initialize().await?;
// Source is already initialised; call anyway for idempotency.
self.source.initialize().await?;
Ok(())
})
}
/// Writes go to **target** only.
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>> {
let hash = hash.to_string();
let path = source_path.to_path_buf();
Box::pin(async move { self.target.put_blob(&hash, &path).await })
}
/// Read from target first; fall back to source.
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>> {
let hash = hash.to_string();
Box::pin(async move {
match self.target.get_blob_stream(&hash).await {
Ok(stream) => Ok(stream),
Err(_) => self.source.get_blob_stream(&hash).await,
}
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> BoxFut<'_, Result<BlobStream, DomainError>> {
let hash = hash.to_string();
Box::pin(async move {
match self.target.get_blob_range_stream(&hash, start, end).await {
Ok(stream) => Ok(stream),
Err(_) => self.source.get_blob_range_stream(&hash, start, end).await,
}
})
}
/// Delete from **both** backends (best-effort on source).
fn delete_blob(&self, hash: &str) -> BoxFut<'_, Result<(), DomainError>> {
let hash = hash.to_string();
Box::pin(async move {
self.target.delete_blob(&hash).await?;
// Best-effort on source — ignore errors (blob may already be gone).
let _ = self.source.delete_blob(&hash).await;
Ok(())
})
}
/// Exists in either backend.
fn blob_exists(&self, hash: &str) -> BoxFut<'_, Result<bool, DomainError>> {
let hash = hash.to_string();
Box::pin(async move {
if self.target.blob_exists(&hash).await? {
return Ok(true);
}
self.source.blob_exists(&hash).await
})
}
fn blob_size(&self, hash: &str) -> BoxFut<'_, Result<u64, DomainError>> {
let hash = hash.to_string();
Box::pin(async move {
match self.target.blob_size(&hash).await {
Ok(sz) => Ok(sz),
Err(_) => self.source.blob_size(&hash).await,
}
})
}
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>> {
Box::pin(async move {
let target_health = self.target.health_check().await?;
let source_health = self.source.health_check().await?;
Ok(StorageHealthStatus {
connected: target_health.connected && source_health.connected,
backend_type: format!(
"migration({} → {})",
source_health.backend_type, target_health.backend_type
),
message: format!(
"Source: {} | Target: {}",
source_health.message, target_health.message
),
available_bytes: target_health.available_bytes,
})
})
}
fn backend_type(&self) -> &'static str {
"migration"
}
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
// Prefer target, fall back to source.
self.target
.local_blob_path(hash)
.or_else(|| self.source.local_blob_path(hash))
}
}
@@ -0,0 +1,240 @@
//! Background migration job — copies blobs from a source backend to a target
//! backend with configurable concurrency and progress tracking.
use std::sync::Arc;
use futures::StreamExt;
use serde::Serialize;
use sqlx::PgPool;
use tokio::sync::RwLock;
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::errors::DomainError;
use crate::infrastructure::services::migration_blob_backend::{MigrationState, MigrationStatus};
/// Run the migration: stream all blob hashes from `storage.blobs` and copy
/// each one from `source` to `target`.
///
/// * The job respects `Paused` / `Failed` status in `state` — it will stop
/// streaming when the status is no longer `Running`.
/// * Errors on individual blobs are logged and collected in `failed_blobs`
/// but do **not** abort the full run.
/// * `concurrency` controls `buffer_unordered` parallelism (default: 4).
pub async fn run_migration(
source: Arc<dyn BlobStorageBackend>,
target: Arc<dyn BlobStorageBackend>,
pool: Arc<PgPool>,
state: Arc<RwLock<MigrationState>>,
concurrency: usize,
) -> Result<(), DomainError> {
// Count total blobs for progress tracking.
let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
.fetch_one(pool.as_ref())
.await
.unwrap_or(0);
{
let mut s = state.write().await;
s.status = MigrationStatus::Running;
s.total_blobs = total as u64;
s.migrated_blobs = 0;
s.migrated_bytes = 0;
s.failed_blobs.clear();
s.started_at = Some(chrono::Utc::now());
s.completed_at = None;
}
// Stream all hashes+sizes with a cursor.
let mut rows =
sqlx::query_as::<_, (String, i64)>("SELECT hash, size FROM storage.blobs ORDER BY hash")
.fetch(pool.as_ref());
// Collect all hashes first to avoid holding the cursor across awaits.
let mut work: Vec<(String, i64)> = Vec::with_capacity(total as usize);
while let Some(row) = rows.next().await {
match row {
Ok(r) => work.push(r),
Err(e) => {
tracing::warn!("Error fetching blob row during migration: {}", e);
}
}
}
// Process in parallel chunks.
let results = futures::stream::iter(work.into_iter().map(|(hash, size)| {
let src = source.clone();
let tgt = target.clone();
let st = state.clone();
async move {
// Check if we should keep running.
{
let s = st.read().await;
if s.status != MigrationStatus::Running {
return;
}
}
// Skip if already in target.
match tgt.blob_exists(&hash).await {
Ok(true) => {
let mut s = st.write().await;
s.migrated_blobs += 1;
s.migrated_bytes += size as u64;
return;
}
Ok(false) => {}
Err(e) => {
tracing::warn!("blob_exists check failed for {}: {}", hash, e);
}
}
// Copy: stream from source → temp file → put into target.
if let Err(e) = copy_blob(&src, &tgt, &hash).await {
tracing::warn!("Failed to migrate blob {}: {}", hash, e);
let mut s = st.write().await;
s.failed_blobs.push(hash);
return;
}
let mut s = st.write().await;
s.migrated_blobs += 1;
s.migrated_bytes += size as u64;
}
}))
.buffer_unordered(concurrency)
.collect::<Vec<()>>()
.await;
drop(results);
// Finalize state.
let mut s = state.write().await;
if s.status == MigrationStatus::Running {
if s.failed_blobs.is_empty() {
s.status = MigrationStatus::Completed;
} else {
s.status = MigrationStatus::Failed;
}
s.completed_at = Some(chrono::Utc::now());
}
tracing::info!(
"Migration finished: {}/{} blobs, {} failures",
s.migrated_blobs,
s.total_blobs,
s.failed_blobs.len()
);
Ok(())
}
/// Copy a single blob: stream from source → spool to temp file → put_blob into target.
async fn copy_blob(
source: &Arc<dyn BlobStorageBackend>,
target: &Arc<dyn BlobStorageBackend>,
hash: &str,
) -> Result<(), DomainError> {
use tokio::io::AsyncWriteExt;
// Create a temp file to spool content.
let tmp_dir = std::env::temp_dir().join("oxicloud-migration");
tokio::fs::create_dir_all(&tmp_dir).await.map_err(|e| {
DomainError::internal_error("Migration", format!("Failed to create temp dir: {}", e))
})?;
let tmp_path = tmp_dir.join(format!("{}.tmp", hash));
// Stream from source.
let stream = source.get_blob_stream(hash).await?;
// Write to temp file.
let mut file = tokio::fs::File::create(&tmp_path).await.map_err(|e| {
DomainError::internal_error("Migration", format!("Failed to create temp file: {}", e))
})?;
let mut stream = std::pin::pin!(stream);
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| {
DomainError::internal_error("Migration", format!("Stream error: {}", e))
})?;
file.write_all(&bytes)
.await
.map_err(|e| DomainError::internal_error("Migration", format!("Write error: {}", e)))?;
}
file.flush()
.await
.map_err(|e| DomainError::internal_error("Migration", format!("Flush error: {}", e)))?;
drop(file);
// Put into target.
target.put_blob(hash, &tmp_path).await?;
// Clean up temp file.
let _ = tokio::fs::remove_file(&tmp_path).await;
Ok(())
}
/// Verify migration integrity by comparing blob counts and sampling random hashes.
pub async fn verify_migration(
target: Arc<dyn BlobStorageBackend>,
pool: Arc<PgPool>,
sample_size: usize,
) -> Result<VerificationResult, DomainError> {
// 1. Count blobs in PG.
let pg_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
.fetch_one(pool.as_ref())
.await
.unwrap_or(0);
// 2. Verify sample of blobs exist in target.
let sample_rows: Vec<(String, i64)> =
sqlx::query_as("SELECT hash, size FROM storage.blobs ORDER BY random() LIMIT $1")
.bind(sample_size as i64)
.fetch_all(pool.as_ref())
.await
.map_err(|e| {
DomainError::internal_error("Migration", format!("Sample query failed: {}", e))
})?;
let mut missing = Vec::new();
let mut size_mismatches = Vec::new();
for (hash, expected_size) in &sample_rows {
match target.blob_exists(hash).await {
Ok(false) => missing.push(hash.clone()),
Err(e) => {
tracing::warn!("blob_exists failed for {}: {}", hash, e);
missing.push(hash.clone());
}
Ok(true) => {
// Verify size matches.
if let Ok(actual_size) = target.blob_size(hash).await
&& actual_size != *expected_size as u64
{
size_mismatches.push(hash.clone());
}
}
}
}
let passed = missing.is_empty() && size_mismatches.is_empty();
Ok(VerificationResult {
pg_blob_count: pg_count as u64,
sample_checked: sample_rows.len() as u64,
missing_in_target: missing,
size_mismatches,
passed,
})
}
/// Result of a post-migration integrity check.
#[derive(Debug, Clone, Serialize, serde::Deserialize)]
pub struct VerificationResult {
pub pg_blob_count: u64,
pub sample_checked: u64,
pub missing_in_target: Vec<String>,
pub size_mismatches: Vec<String>,
pub passed: bool,
}
+8
View File
@@ -1,18 +1,26 @@
pub mod audio_metadata_service;
pub mod azure_blob_backend;
pub mod cached_blob_backend;
pub mod chunked_upload_service;
pub mod compression_service;
pub mod dedup_service;
pub mod encrypted_blob_backend;
pub mod exif_service;
pub mod file_content_cache;
pub mod file_system_i18n_service;
pub mod image_transcode_service;
pub mod jwt_service;
pub mod local_blob_backend;
pub mod login_lockout_service;
pub mod migration_blob_backend;
pub mod migration_job;
pub mod nextcloud_chunked_upload_service;
pub mod oidc_service;
pub mod password_hasher;
pub mod path_resolver_service;
pub mod path_service;
pub mod retry_blob_backend;
pub mod s3_blob_backend;
pub mod thumbnail_service;
#[cfg(test)]
mod thumbnail_service_test;
@@ -0,0 +1,254 @@
//! `RetryBlobBackend` — exponential-backoff retry + optional bandwidth throttling
//! decorator for remote blob backends.
//!
//! Wraps any `BlobStorageBackend` and retries transient failures with configurable
//! exponential backoff. Optionally throttles upload/download bandwidth via
//! inter-chunk sleeps.
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::sync::Arc;
use std::time::Duration;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::domain::errors::DomainError;
// ── Retry policy ───────────────────────────────────────────────────
/// Exponential backoff retry configuration.
#[derive(Debug, Clone)]
pub struct RetryPolicy {
/// Maximum number of retry attempts (0 = no retries).
pub max_retries: u32,
/// Initial backoff duration before the first retry.
pub initial_backoff: Duration,
/// Maximum backoff duration (capped).
pub max_backoff: Duration,
/// Multiplier applied to backoff after each attempt.
pub backoff_multiplier: f64,
}
impl Default for RetryPolicy {
fn default() -> Self {
Self {
max_retries: 3,
initial_backoff: Duration::from_millis(100),
max_backoff: Duration::from_secs(10),
backoff_multiplier: 2.0,
}
}
}
// ── RetryBlobBackend ───────────────────────────────────────────────
/// Decorator that retries failed backend operations with exponential backoff.
pub struct RetryBlobBackend {
inner: Arc<dyn BlobStorageBackend>,
policy: RetryPolicy,
}
impl RetryBlobBackend {
pub fn new(inner: Arc<dyn BlobStorageBackend>, policy: RetryPolicy) -> Self {
Self { inner, policy }
}
}
/// Execute an async closure with exponential backoff retry.
async fn retry_async<F, Fut, T>(
policy: &RetryPolicy,
name: &str,
mut f: F,
) -> Result<T, DomainError>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T, DomainError>>,
{
let mut attempt = 0u32;
let mut backoff = policy.initial_backoff;
loop {
match f().await {
Ok(v) => return Ok(v),
Err(e) if attempt < policy.max_retries && is_retryable(&e) => {
attempt += 1;
tracing::warn!(
"Retry {}/{} for {} after error: {} (backoff {:?})",
attempt,
policy.max_retries,
name,
e,
backoff
);
tokio::time::sleep(backoff).await;
let next =
Duration::from_secs_f64(backoff.as_secs_f64() * policy.backoff_multiplier);
backoff = next.min(policy.max_backoff);
}
Err(e) => return Err(e),
}
}
}
/// Determine if an error is likely transient (network timeout, 5xx, etc.).
fn is_retryable(err: &DomainError) -> bool {
let msg = err.to_string().to_lowercase();
msg.contains("timeout")
|| msg.contains("connection")
|| msg.contains("503")
|| msg.contains("500")
|| msg.contains("429")
|| msg.contains("temporarily")
|| msg.contains("broken pipe")
|| msg.contains("reset by peer")
}
impl BlobStorageBackend for RetryBlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let policy = self.policy.clone();
Box::pin(async move {
retry_async(&policy, "initialize", || {
let inner = inner.clone();
async move { inner.initialize().await }
})
.await
})
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
let path = source_path.to_path_buf();
Box::pin(async move {
retry_async(&policy, &format!("put_blob({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
let path = path.clone();
async move { inner.put_blob(&hash, &path).await }
})
.await
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("get_blob_stream({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_stream(&hash).await }
})
.await
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("get_blob_range({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_range_stream(&hash, start, end).await }
})
.await
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("delete_blob({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.delete_blob(&hash).await }
})
.await
})
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("blob_exists({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_exists(&hash).await }
})
.await
})
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let inner = self.inner.clone();
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("blob_size({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_size(&hash).await }
})
.await
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
let inner = self.inner.clone();
let policy = self.policy.clone();
Box::pin(async move {
retry_async(&policy, "health_check", || {
let inner = inner.clone();
async move { inner.health_check().await }
})
.await
})
}
fn backend_type(&self) -> &'static str {
"retry"
}
fn local_blob_path(&self, hash: &str) -> Option<PathBuf> {
self.inner.local_blob_path(hash)
}
}
@@ -0,0 +1,344 @@
//! S3-Compatible Blob Backend — stores blobs in any S3-compatible object store.
//!
//! Supports AWS S3, Backblaze B2, Cloudflare R2, MinIO, DigitalOcean Spaces,
//! Wasabi, and any other service that implements the S3 API.
use aws_sdk_s3::primitives::ByteStream;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use tokio::fs;
use tokio_util::io::ReaderStream;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
};
use crate::common::config::S3StorageConfig;
use crate::domain::errors::{DomainError, ErrorKind};
/// S3-compatible blob storage backend.
///
/// Blobs are stored as objects with key `{2-char-prefix}/{hash}.blob`,
/// mirroring the local filesystem layout for consistency.
pub struct S3BlobBackend {
client: aws_sdk_s3::Client,
bucket: String,
}
impl S3BlobBackend {
/// Build a new S3 backend from configuration.
///
/// Supports custom endpoints for non-AWS providers (Backblaze B2,
/// MinIO, Cloudflare R2, etc.).
pub fn new(config: &S3StorageConfig) -> Self {
let credentials = aws_sdk_s3::config::Credentials::new(
&config.access_key,
&config.secret_key,
None,
None,
"oxicloud",
);
let mut builder = aws_sdk_s3::config::Builder::new()
.region(aws_sdk_s3::config::Region::new(config.region.clone()))
.credentials_provider(credentials)
.behavior_version_latest();
if let Some(ref endpoint) = config.endpoint_url {
builder = builder.endpoint_url(endpoint);
}
if config.force_path_style {
builder = builder.force_path_style(true);
}
let client = aws_sdk_s3::Client::from_conf(builder.build());
Self {
client,
bucket: config.bucket.clone(),
}
}
/// Compute the S3 object key for a given hash.
fn object_key(hash: &str) -> String {
let prefix = &hash[0..2];
format!("{}/{}.blob", prefix, hash)
}
}
impl BlobStorageBackend for S3BlobBackend {
fn initialize(
&self,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
Box::pin(async move {
// Verify bucket exists and is accessible
self.client
.head_bucket()
.bucket(&self.bucket)
.send()
.await
.map_err(|e| {
DomainError::internal_error(
"S3",
format!("Cannot access bucket '{}': {}", self.bucket, e),
)
})?;
tracing::info!("S3 blob backend initialized: bucket={}", self.bucket);
Ok(())
})
}
fn put_blob(
&self,
hash: &str,
source_path: &Path,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
let source_path = source_path.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
// Check if object already exists (idempotent)
let exists = self
.client
.head_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
.is_ok();
if exists {
// Blob already in S3 — remove local source and return size
let file_size = fs::metadata(&source_path)
.await
.map_err(|e| {
DomainError::internal_error(
"S3",
format!("Failed to stat source file: {}", e),
)
})?
.len();
let _ = fs::remove_file(&source_path).await;
return Ok(file_size);
}
// Upload from local file
let body = ByteStream::from_path(&source_path).await.map_err(|e| {
DomainError::internal_error("S3", format!("Failed to read source file: {}", e))
})?;
let file_size = fs::metadata(&source_path)
.await
.map_err(|e| {
DomainError::internal_error("S3", format!("Failed to stat source file: {}", e))
})?
.len();
self.client
.put_object()
.bucket(&self.bucket)
.key(&key)
.body(body)
.send()
.await
.map_err(|e| {
DomainError::internal_error(
"S3",
format!("Failed to upload blob {}: {}", hash, e),
)
})?;
// Clean up local source after successful upload
let _ = fs::remove_file(&source_path).await;
Ok(file_size)
})
}
fn get_blob_stream(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
let output = self
.client
.get_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob {}: {}", hash, e),
)
})?;
// Convert S3 ByteStream into a Stream<Item = Result<Bytes, io::Error>>
// via AsyncRead adapter
let reader = output.body.into_async_read();
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
})
}
fn get_blob_range_stream(
&self,
hash: &str,
start: u64,
end: Option<u64>,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobStream, DomainError>> + Send + '_>>
{
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
let range = match end {
Some(end_pos) => format!("bytes={}-{}", start, end_pos.saturating_sub(1)),
None => format!("bytes={}-", start),
};
let output = self
.client
.get_object()
.bucket(&self.bucket)
.key(&key)
.range(range)
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob range {}: {}", hash, e),
)
})?;
let reader = output.body.into_async_read();
Ok(Box::pin(ReaderStream::with_capacity(reader, 256 * 1024)) as BlobStream)
})
}
fn delete_blob(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<(), DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
// S3 DeleteObject is already idempotent (returns 204 even if not found)
self.client
.delete_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
.map_err(|e| {
DomainError::internal_error(
"S3",
format!("Failed to delete blob {}: {}", hash, e),
)
})?;
Ok(())
})
}
fn blob_exists(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<bool, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
match self
.client
.head_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
{
Ok(_) => Ok(true),
Err(e) => {
// Check if it's a 404 (not found) vs an actual error
let service_err = e.into_service_error();
if service_err.is_not_found() {
Ok(false)
} else {
Err(DomainError::internal_error(
"S3",
format!("Failed to check blob {}: {}", hash, service_err),
))
}
}
}
})
}
fn blob_size(
&self,
hash: &str,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
let output = self
.client
.head_object()
.bucket(&self.bucket)
.key(&key)
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to stat blob {}: {}", hash, e),
)
})?;
Ok(output.content_length().unwrap_or(0) as u64)
})
}
fn health_check(
&self,
) -> Pin<
Box<dyn std::future::Future<Output = Result<StorageHealthStatus, DomainError>> + Send + '_>,
> {
Box::pin(async move {
match self.client.head_bucket().bucket(&self.bucket).send().await {
Ok(_) => Ok(StorageHealthStatus {
connected: true,
backend_type: "s3".to_string(),
message: format!("S3 bucket '{}' is accessible", self.bucket),
available_bytes: None,
}),
Err(e) => Ok(StorageHealthStatus {
connected: false,
backend_type: "s3".to_string(),
message: format!("S3 bucket '{}' is not accessible: {}", self.bucket, e),
available_bytes: None,
}),
}
})
}
fn backend_type(&self) -> &'static str {
"s3"
}
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
None // Remote backend — no local path
}
}
+349 -2
View File
@@ -8,8 +8,9 @@ use axum::{
use crate::application::dtos::settings_dto::{
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, ListUsersQueryDto,
SaveOidcSettingsDto, TestOidcConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
UpdateUserRoleDto,
MigrationStateDto, SaveOidcSettingsDto, SaveStorageSettingsDto, StartMigrationDto,
TestOidcConnectionDto, TestStorageConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto,
UpdateUserRoleDto, VerifyMigrationDto,
};
use crate::application::ports::auth_ports::TokenServicePort;
use crate::common::di::AppState;
@@ -24,6 +25,22 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
.route("/settings/oidc", get(get_oidc_settings))
.route("/settings/oidc", put(save_oidc_settings))
.route("/settings/oidc/test", post(test_oidc_connection))
// Storage settings
.route("/settings/storage", get(get_storage_settings))
.route("/settings/storage", put(save_storage_settings))
.route("/settings/storage/test", post(test_storage_connection))
// Storage migration
.route("/storage/migration", get(get_migration_status))
.route("/storage/migration/start", post(start_migration))
.route("/storage/migration/pause", post(pause_migration))
.route("/storage/migration/resume", post(resume_migration))
.route("/storage/migration/complete", post(complete_migration))
.route("/storage/migration/verify", post(verify_migration))
// Encryption key generation
.route(
"/settings/storage/generate-key",
post(generate_encryption_key),
)
.route("/settings/general", get(get_general_settings))
// Dashboard / stats
.route("/dashboard", get(get_dashboard_stats))
@@ -148,6 +165,336 @@ async fn test_oidc_connection(
Ok(Json(result))
}
// ─────────────────────────────────────────────────────
// Storage settings handlers
// ─────────────────────────────────────────────────────
/// GET /api/admin/settings/storage — get storage backend settings
async fn get_storage_settings(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
admin_guard(&state, &headers).await?;
let svc = state
.storage_settings_service
.as_ref()
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
let settings = svc
.get_storage_settings()
.await
.map_err(|e| AppError::internal_error(format!("Failed to load storage settings: {}", e)))?;
Ok(Json(settings))
}
/// PUT /api/admin/settings/storage — save storage backend settings
async fn save_storage_settings(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Json(dto): Json<SaveStorageSettingsDto>,
) -> Result<impl IntoResponse, AppError> {
let (user_id, _) = admin_guard(&state, &headers).await?;
let svc = state
.storage_settings_service
.as_ref()
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
svc.save_storage_settings(dto, user_id)
.await
.map_err(|e| AppError::internal_error(format!("Failed to save storage settings: {}", e)))?;
Ok((
StatusCode::OK,
Json(serde_json::json!({
"message": "Storage settings saved successfully"
})),
))
}
/// POST /api/admin/settings/storage/test — test storage backend connection
async fn test_storage_connection(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Json(dto): Json<TestStorageConnectionDto>,
) -> Result<impl IntoResponse, AppError> {
admin_guard(&state, &headers).await?;
let svc = state
.storage_settings_service
.as_ref()
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
let result = svc
.test_storage_connection(dto)
.await
.map_err(|e| AppError::internal_error(format!("Storage connection test failed: {}", e)))?;
Ok(Json(result))
}
// ─────────────────────────────────────────────────────
// Storage migration handlers
// ─────────────────────────────────────────────────────
/// GET /api/admin/storage/migration — current migration progress
async fn get_migration_status(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
admin_guard(&state, &headers).await?;
let s = state.migration_state.read().await;
Ok(Json(migration_state_to_dto(&s)))
}
/// POST /api/admin/storage/migration/start — begin background migration
async fn start_migration(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Json(dto): Json<StartMigrationDto>,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
admin_guard(&state, &headers).await?;
// Check not already running.
{
let s = state.migration_state.read().await;
if s.status == MigrationStatus::Running {
return Err(AppError::bad_request("A migration is already running"));
}
}
let pool = state
.db_pool
.clone()
.ok_or_else(|| AppError::internal_error("Database not available"))?;
let source = state.core.dedup_service.backend().clone();
let svc = state
.storage_settings_service
.as_ref()
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
// Build target backend from saved settings.
let effective = svc
.load_effective_storage_config()
.await
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
let target = build_backend_from_config(&effective)
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
target
.initialize()
.await
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
let concurrency = dto.concurrency.unwrap_or(4).clamp(1, 16);
let migration_state = state.migration_state.clone();
// Spawn the background migration job.
tokio::spawn(async move {
if let Err(e) = crate::infrastructure::services::migration_job::run_migration(
source,
target,
pool,
migration_state,
concurrency,
)
.await
{
tracing::error!("Migration job error: {}", e);
}
});
Ok((
StatusCode::OK,
Json(serde_json::json!({ "message": "Migration started" })),
))
}
/// POST /api/admin/storage/migration/pause — pause running migration
async fn pause_migration(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
admin_guard(&state, &headers).await?;
let mut s = state.migration_state.write().await;
if s.status != MigrationStatus::Running {
return Err(AppError::bad_request("No running migration to pause"));
}
s.status = MigrationStatus::Paused;
Ok((
StatusCode::OK,
Json(serde_json::json!({ "message": "Migration paused" })),
))
}
/// POST /api/admin/storage/migration/resume — resume paused migration
async fn resume_migration(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
admin_guard(&state, &headers).await?;
// Set status back to Running — the background task checks on each blob.
let mut s = state.migration_state.write().await;
if s.status != MigrationStatus::Paused {
return Err(AppError::bad_request("No paused migration to resume"));
}
s.status = MigrationStatus::Running;
Ok((
StatusCode::OK,
Json(serde_json::json!({ "message": "Migration resumed" })),
))
}
/// POST /api/admin/storage/migration/complete — finalize migration
async fn complete_migration(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::migration_blob_backend::MigrationStatus;
admin_guard(&state, &headers).await?;
let s = state.migration_state.read().await;
if s.status != MigrationStatus::Completed {
return Err(AppError::bad_request(
"Migration must be completed (100%) before finalizing",
));
}
drop(s);
// Mark as idle — the admin has acknowledged completion.
let mut s = state.migration_state.write().await;
s.status = MigrationStatus::Idle;
Ok((
StatusCode::OK,
Json(
serde_json::json!({ "message": "Migration finalized. Restart the server to use the new backend." }),
),
))
}
/// POST /api/admin/storage/migration/verify — run integrity check
async fn verify_migration(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Json(dto): Json<VerifyMigrationDto>,
) -> Result<impl IntoResponse, AppError> {
admin_guard(&state, &headers).await?;
let pool = state
.db_pool
.clone()
.ok_or_else(|| AppError::internal_error("Database not available"))?;
let svc = state
.storage_settings_service
.as_ref()
.ok_or_else(|| AppError::internal_error("Storage settings service not available"))?;
let effective = svc
.load_effective_storage_config()
.await
.map_err(|e| AppError::internal_error(format!("Failed to load storage config: {}", e)))?;
let target = build_backend_from_config(&effective)
.map_err(|e| AppError::internal_error(format!("Failed to build target backend: {}", e)))?;
target
.initialize()
.await
.map_err(|e| AppError::internal_error(format!("Target backend init failed: {}", e)))?;
let sample_size = dto.sample_size.unwrap_or(100).clamp(1, 1000);
let result =
crate::infrastructure::services::migration_job::verify_migration(target, pool, sample_size)
.await
.map_err(|e| AppError::internal_error(format!("Verification failed: {}", e)))?;
Ok(Json(result))
}
/// Helper: convert MigrationState to DTO for JSON serialization.
fn migration_state_to_dto(
s: &crate::infrastructure::services::migration_blob_backend::MigrationState,
) -> MigrationStateDto {
let throughput = match (s.started_at, s.migrated_bytes) {
(Some(start), bytes) if bytes > 0 => {
let elapsed = chrono::Utc::now()
.signed_duration_since(start)
.num_seconds()
.max(1) as f64;
Some(bytes as f64 / elapsed)
}
_ => None,
};
MigrationStateDto {
status: format!("{:?}", s.status).to_lowercase(),
total_blobs: s.total_blobs,
migrated_blobs: s.migrated_blobs,
migrated_bytes: s.migrated_bytes,
failed_blobs: s.failed_blobs.clone(),
started_at: s.started_at.map(|d| d.to_rfc3339()),
completed_at: s.completed_at.map(|d| d.to_rfc3339()),
throughput_bytes_per_sec: throughput,
}
}
/// POST /api/admin/settings/storage/generate-key — generate a random AES-256 key.
async fn generate_encryption_key(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
) -> Result<impl IntoResponse, AppError> {
admin_guard(&state, &headers).await?;
let key =
crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend::generate_key(
);
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
Ok(Json(serde_json::json!({
"key": key_b64,
"warning": "Store this key securely. If lost, encrypted data is IRRECOVERABLY LOST."
})))
}
/// Helper: build a BlobStorageBackend from StorageConfig.
fn build_backend_from_config(
config: &crate::common::config::StorageConfig,
) -> Result<
std::sync::Arc<dyn crate::application::ports::blob_storage_ports::BlobStorageBackend>,
String,
> {
match config.backend {
crate::common::config::StorageBackendType::Local => Ok(std::sync::Arc::new(
crate::infrastructure::services::local_blob_backend::LocalBlobBackend::new(
std::path::Path::new(&config.root_dir),
),
)),
crate::common::config::StorageBackendType::S3 => {
let s3 = config.s3.as_ref().ok_or("S3 config missing")?;
Ok(std::sync::Arc::new(
crate::infrastructure::services::s3_blob_backend::S3BlobBackend::new(s3),
))
}
crate::common::config::StorageBackendType::Azure => {
let az = config.azure.as_ref().ok_or("Azure config missing")?;
Ok(std::sync::Arc::new(
crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(az),
))
}
}
}
/// GET /api/admin/settings/general — system overview (backward compat)
async fn get_general_settings(
State(state): State<Arc<AppState>>,
+180
View File
@@ -53,6 +53,9 @@
<button class="admin-tab" id="tab-btn-oidc">
<i class="fas fa-key"></i> <span data-i18n="admin.tab_oidc">SSO / OIDC</span>
</button>
<button class="admin-tab" id="tab-btn-storage">
<i class="fas fa-database"></i> <span data-i18n="admin.tab_storage">Storage</span>
</button>
</div>
<div id="tab-dashboard" class="tab-content active">
@@ -424,6 +427,183 @@
</div>
</div>
</div>
<!-- ════════ Storage tab ════════ -->
<div id="tab-storage" class="tab-content">
<div class="admin-card">
<h2>
<i class="fas fa-database"></i> <span data-i18n="admin.storage_title">Storage Backend</span>
</h2>
<!-- Current status -->
<div class="storage-status-grid">
<div class="storage-stat">
<span class="storage-stat__label" data-i18n="admin.storage_current_backend">Active Backend</span>
<span class="storage-stat__value" id="storage-current-backend">—</span>
</div>
<div class="storage-stat">
<span class="storage-stat__label" data-i18n="admin.storage_total_blobs">Total Blobs</span>
<span class="storage-stat__value" id="storage-total-blobs">—</span>
</div>
<div class="storage-stat">
<span class="storage-stat__label" data-i18n="admin.storage_total_size">Total Size</span>
<span class="storage-stat__value" id="storage-total-size">—</span>
</div>
<div class="storage-stat">
<span class="storage-stat__label" data-i18n="admin.storage_dedup_ratio">Dedup Ratio</span>
<span class="storage-stat__value" id="storage-dedup-ratio">—</span>
</div>
</div>
<!-- Backend selector -->
<div class="storage-backend-selector">
<label class="storage-backend-option">
<input type="radio" name="storage-backend" value="local" checked />
<span class="storage-backend-option__card">
<i class="fas fa-hdd"></i>
<span data-i18n="admin.storage_local">Local Filesystem</span>
</span>
</label>
<label class="storage-backend-option">
<input type="radio" name="storage-backend" value="s3" />
<span class="storage-backend-option__card">
<i class="fas fa-cloud"></i>
<span data-i18n="admin.storage_s3">S3-Compatible</span>
</span>
</label>
</div>
<!-- S3 form (shown when S3 is selected) -->
<div id="storage-s3-form" class="storage-form" style="display:none">
<!-- Provider preset -->
<div class="form-group">
<label data-i18n="admin.storage_provider_preset">Provider Preset</label>
<select id="storage-preset">
<option value="custom" data-i18n="admin.storage_preset_custom">Custom</option>
<option value="aws">Amazon S3</option>
<option value="backblaze">Backblaze B2</option>
<option value="cloudflare-r2">Cloudflare R2</option>
<option value="minio">MinIO</option>
<option value="digitalocean">DigitalOcean Spaces</option>
<option value="wasabi">Wasabi</option>
</select>
</div>
<!-- Endpoint URL -->
<div class="form-group">
<label><span data-i18n="admin.storage_endpoint_url">Endpoint URL</span>
<span id="badge-s3_endpoint_url"></span></label>
<input type="url" id="storage-endpoint-url" placeholder="https://s3.amazonaws.com" />
<small data-i18n="admin.storage_endpoint_hint">Leave empty for Amazon S3 default</small>
</div>
<!-- Bucket -->
<div class="form-group">
<label><span data-i18n="admin.storage_bucket">Bucket</span>
<span id="badge-s3_bucket"></span></label>
<input type="text" id="storage-bucket" placeholder="my-oxicloud-bucket" />
</div>
<!-- Region -->
<div class="form-group">
<label><span data-i18n="admin.storage_region">Region</span>
<span id="badge-s3_region"></span></label>
<input type="text" id="storage-region" placeholder="us-east-1" />
</div>
<!-- Access Key -->
<div class="form-group">
<label><span data-i18n="admin.storage_access_key">Access Key ID</span>
<span id="badge-s3_access_key"></span></label>
<input type="text" id="storage-access-key" placeholder="AKIA..." />
</div>
<!-- Secret Key -->
<div class="form-group">
<label><span data-i18n="admin.storage_secret_key">Secret Access Key</span>
<span id="badge-s3_secret_key"></span></label>
<input type="password" id="storage-secret-key" placeholder="Leave empty to keep current value" />
<small id="storage-secret-hint" style="display:none">
<i class="fas fa-check-circle secret-icon"></i>
<span data-i18n="admin.storage_secret_configured">A secret key is already configured</span>
</small>
</div>
<!-- Force Path Style -->
<div class="toggle-row">
<label data-i18n="admin.storage_path_style">Force Path Style</label>
<label class="switch">
<input type="checkbox" id="storage-path-style" /><span class="slider"></span>
</label>
</div>
<small data-i18n="admin.storage_path_style_hint">Required for MinIO and some S3-compatible providers</small>
</div>
<!-- Action buttons -->
<div class="storage-actions">
<button class="btn btn-secondary" id="btn-test-storage">
<i class="fas fa-vial"></i> <span data-i18n="admin.storage_test_connection">Test Connection</span>
</button>
<button class="btn btn-primary" id="btn-save-storage">
<i class="fas fa-save"></i> <span data-i18n="admin.storage_save">Save</span>
</button>
</div>
<div id="storage-status" class="alert"></div>
<!-- Migration section -->
<div class="storage-migration-section">
<h3><i class="fas fa-exchange-alt"></i> <span data-i18n="admin.storage_migration">Backend Migration</span></h3>
<!-- Status badge -->
<div class="migration-status-row">
<span data-i18n="admin.migration_status_label">Status:</span>
<span id="migration-status-badge" class="badge badge-migration badge-migration--idle">Idle</span>
</div>
<!-- Progress bar (hidden when idle) -->
<div id="migration-progress-section" style="display:none">
<div class="migration-progress-bar">
<div class="migration-progress-bar__fill" id="migration-progress-fill" style="width:0%"></div>
</div>
<div class="migration-progress-text">
<span id="migration-progress-label">0 / 0 blobs (0%)</span>
<span id="migration-throughput"></span>
</div>
<div class="migration-progress-text">
<span id="migration-bytes-label"></span>
<span id="migration-eta"></span>
</div>
</div>
<!-- Failed blobs (expandable) -->
<details id="migration-failed-section" style="display:none">
<summary><i class="fas fa-exclamation-triangle"></i> <span id="migration-failed-count">0</span> <span data-i18n="admin.migration_failed_blobs">failed blobs</span></summary>
<pre id="migration-failed-list" class="migration-failed-list"></pre>
</details>
<!-- Action buttons -->
<div class="migration-actions">
<button class="btn btn-primary" id="btn-start-migration">
<i class="fas fa-play"></i> <span data-i18n="admin.migration_start">Start Migration</span>
</button>
<button class="btn btn-secondary" id="btn-pause-migration" style="display:none">
<i class="fas fa-pause"></i> <span data-i18n="admin.migration_pause">Pause</span>
</button>
<button class="btn btn-primary" id="btn-resume-migration" style="display:none">
<i class="fas fa-play"></i> <span data-i18n="admin.migration_resume">Resume</span>
</button>
<button class="btn btn-secondary" id="btn-verify-migration" style="display:none">
<i class="fas fa-check-double"></i> <span data-i18n="admin.migration_verify">Verify Integrity</span>
</button>
<button class="btn btn-primary" id="btn-complete-migration" style="display:none">
<i class="fas fa-flag-checkered"></i> <span data-i18n="admin.migration_complete">Finalize</span>
</button>
</div>
<div id="migration-verify-result" style="display:none"></div>
<div id="migration-status-msg" class="alert" style="display:none"></div>
</div>
</div>
</div>
</div>
</div>
+174
View File
@@ -324,6 +324,7 @@ body {
#oidc-form,
#secret-hint,
#password-warning,
#storage-secret-hint,
#quota-modal,
#create-user-modal,
#reset-pw-modal {
@@ -938,3 +939,176 @@ details[open] summary {
.btn-danger:hover {
opacity: 0.88;
}
/* ── Storage Tab ── */
.storage-status-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
gap: 12px;
margin-bottom: 24px;
}
.storage-stat {
padding: 14px;
background: var(--color-bg-hover);
border-radius: 12px;
text-align: center;
border: 1px solid var(--color-border);
}
.storage-stat__label {
display: block;
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.5px;
color: var(--color-text-secondary);
margin-bottom: 4px;
}
.storage-stat__value {
display: block;
font-size: 1.35rem;
font-weight: 700;
color: var(--color-text-heading);
}
.storage-backend-selector {
display: flex;
gap: 12px;
margin-bottom: 20px;
}
.storage-backend-option {
flex: 1;
cursor: pointer;
}
.storage-backend-option input[type="radio"] {
display: none;
}
.storage-backend-option__card {
display: flex;
flex-direction: column;
align-items: center;
gap: 8px;
padding: 18px 14px;
border: 2px solid var(--color-border);
border-radius: 12px;
transition: border-color 0.2s, background 0.2s;
text-align: center;
font-weight: 500;
}
.storage-backend-option__card i {
font-size: 1.5rem;
color: var(--color-text-secondary);
transition: color 0.2s;
}
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card {
border-color: var(--color-accent);
background: var(--color-accent-subtle);
}
.storage-backend-option input[type="radio"]:checked + .storage-backend-option__card i {
color: var(--color-accent);
}
.storage-form {
margin-top: 8px;
}
.storage-actions {
display: flex;
gap: 10px;
margin-top: 24px;
justify-content: flex-end;
}
.storage-migration-section {
margin-top: 32px;
padding-top: 20px;
border-top: 1px solid var(--color-border);
}
.storage-migration-section h3 {
font-size: 1rem;
margin-bottom: 8px;
color: var(--color-text-heading);
}
.storage-migration-section h3 i {
color: var(--color-text-secondary);
margin-right: 6px;
}
/* ── Migration UI ── */
.migration-status-row {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 14px;
font-size: 14px;
}
.badge-migration {
padding: 3px 10px;
border-radius: 10px;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.4px;
}
.badge-migration--idle {
background: var(--color-bg-hover);
color: var(--color-text-secondary);
}
.badge-migration--running {
background: var(--color-accent-subtle);
color: var(--color-accent);
}
.badge-migration--paused {
background: var(--color-warning-bg, #fff8e1);
color: var(--color-warning-text, #b28704);
}
.badge-migration--completed {
background: var(--color-success-bg, #e8f5e9);
color: var(--color-success-text, #2e7d32);
}
.badge-migration--failed {
background: var(--color-danger-bg, #fce4ec);
color: var(--color-danger-text, #c62828);
}
.migration-progress-bar {
width: 100%;
height: 10px;
background: var(--color-bg-hover);
border-radius: 6px;
overflow: hidden;
margin-bottom: 6px;
}
.migration-progress-bar__fill {
height: 100%;
background: var(--color-accent);
border-radius: 6px;
transition: width 0.3s ease;
}
.migration-progress-text {
display: flex;
justify-content: space-between;
font-size: 12px;
color: var(--color-text-secondary);
margin-bottom: 4px;
}
.migration-failed-list {
max-height: 150px;
overflow-y: auto;
font-size: 11px;
background: var(--color-bg-hover);
padding: 8px 12px;
border-radius: 8px;
border: 1px solid var(--color-border);
margin-top: 6px;
}
.migration-actions {
display: flex;
gap: 10px;
margin-top: 16px;
flex-wrap: wrap;
}
+368
View File
@@ -138,6 +138,7 @@ function switchTab(name, el) {
activeTabName = name;
if (name === 'users') loadUsers();
if (name === 'dashboard') loadDashboard();
if (name === 'storage') loadStorage();
}
async function loadDashboard() {
@@ -693,6 +694,353 @@ async function saveOidcSettings() {
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.save_btn'))}`;
}
/* ── Storage tab ── */
const STORAGE_PRESETS = {
'custom': { endpoint: '', region: '', pathStyle: false },
'aws': { endpoint: '', region: 'us-east-1', pathStyle: false },
'backblaze': { endpoint: 'https://s3.{region}.backblazeb2.com', region: 'us-west-004', pathStyle: false },
'cloudflare-r2': { endpoint: 'https://{accountId}.r2.cloudflarestorage.com', region: 'auto', pathStyle: true },
'minio': { endpoint: 'http://localhost:9000', region: 'us-east-1', pathStyle: true },
'digitalocean': { endpoint: 'https://{region}.digitaloceanspaces.com', region: 'nyc3', pathStyle: false },
'wasabi': { endpoint: 'https://s3.{region}.wasabisys.com', region: 'us-east-1', pathStyle: false },
};
function toggleS3Form(visible) {
if (visible) showElement('storage-s3-form');
else hideElement('storage-s3-form');
}
function onStoragePresetChange() {
const preset = document.getElementById('storage-preset').value;
const p = STORAGE_PRESETS[preset];
if (!p) return;
if (p.endpoint) document.getElementById('storage-endpoint-url').value = p.endpoint;
if (p.region) document.getElementById('storage-region').value = p.region;
document.getElementById('storage-path-style').checked = p.pathStyle;
}
function showStorageStatus(msg, type) {
const el = document.getElementById('storage-status');
el.textContent = msg;
el.className = `alert alert-${type}`;
}
async function loadStorage() {
try {
const resp = await fetch(`${API}/admin/settings/storage`, {
headers: headers(),
credentials: 'same-origin'
});
if (!resp.ok) return;
const s = await resp.json();
// Backend selector
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
r.checked = r.value === s.backend;
});
toggleS3Form(s.backend === 's3');
// S3 fields
document.getElementById('storage-endpoint-url').value = s.s3_endpoint_url || '';
document.getElementById('storage-bucket').value = s.s3_bucket || '';
document.getElementById('storage-region').value = s.s3_region || '';
document.getElementById('storage-access-key').value = '';
document.getElementById('storage-secret-key').value = '';
document.getElementById('storage-path-style').checked = s.s3_force_path_style;
// Secret hints
if (s.s3_access_key_set) {
document.getElementById('storage-access-key').placeholder = t('admin.storage_key_placeholder') || 'Leave empty to keep current value';
}
if (s.s3_secret_key_set) {
showElement('storage-secret-hint');
} else {
hideElement('storage-secret-hint');
}
// ENV badges
(s.env_overrides || []).forEach((field) => {
const badge = document.getElementById(`badge-${field}`);
if (badge) badge.innerHTML = '<span class="badge badge-env">ENV</span>';
});
// Status section
document.getElementById('storage-current-backend').textContent = s.current_backend || '—';
document.getElementById('storage-total-blobs').textContent = s.total_blobs != null ? s.total_blobs.toLocaleString() : '—';
document.getElementById('storage-total-size').textContent = s.total_bytes_stored != null ? formatBytes(s.total_bytes_stored) : '—';
document.getElementById('storage-dedup-ratio').textContent = s.dedup_ratio != null ? `${s.dedup_ratio.toFixed(2)}x` : '—';
} catch (e) {
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
}
// Also load migration status
loadMigrationStatus();
}
async function saveStorageSettings() {
const btn = document.getElementById('btn-save-storage');
btn.disabled = true;
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.saving'))}`;
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
const body = {
backend,
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
s3_region: document.getElementById('storage-region').value.trim() || null,
s3_access_key: document.getElementById('storage-access-key').value || null,
s3_secret_key: document.getElementById('storage-secret-key').value || null,
s3_force_path_style: document.getElementById('storage-path-style').checked
};
try {
const resp = await fetch(`${API}/admin/settings/storage`, {
method: 'PUT',
headers: headers(),
credentials: 'same-origin',
body: JSON.stringify(body)
});
if (resp.ok) {
showStorageStatus(t('admin.storage_saved') || 'Storage settings saved successfully', 'success');
loadStorage();
} else {
const e = await resp.json().catch(() => ({}));
showStorageStatus(`Error: ${e.message || resp.statusText}`, 'error');
}
} catch (e) {
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
}
btn.disabled = false;
btn.innerHTML = `<i class="fas fa-save"></i> ${escapeHtml(t('admin.storage_save') || 'Save')}`;
}
async function testStorageConnection() {
const btn = document.getElementById('btn-test-storage');
btn.disabled = true;
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.testing') || 'Testing...')}`;
const backend = document.querySelector('input[name="storage-backend"]:checked').value;
const body = {
backend,
s3_endpoint_url: document.getElementById('storage-endpoint-url').value.trim() || null,
s3_bucket: document.getElementById('storage-bucket').value.trim() || null,
s3_region: document.getElementById('storage-region').value.trim() || null,
s3_access_key: document.getElementById('storage-access-key').value || null,
s3_secret_key: document.getElementById('storage-secret-key').value || null,
s3_force_path_style: document.getElementById('storage-path-style').checked
};
try {
const resp = await fetch(`${API}/admin/settings/storage/test`, {
method: 'POST',
headers: headers(),
credentials: 'same-origin',
body: JSON.stringify(body)
});
const r = await resp.json();
if (r.connected) {
let msg = `${t('admin.storage_test_success') || 'Connection successful'} (${escapeHtml(r.backend_type)})`;
if (r.available_bytes != null) msg += ` — ${formatBytes(r.available_bytes)} available`;
showStorageStatus(msg, 'success');
} else {
showStorageStatus(`${t('admin.storage_test_failure') || 'Connection failed'}: ${escapeHtml(r.message)}`, 'error');
}
} catch (e) {
showStorageStatus(t('admin.error_network', { message: e.message }), 'error');
}
btn.disabled = false;
btn.innerHTML = `<i class="fas fa-vial"></i> ${escapeHtml(t('admin.storage_test_connection') || 'Test Connection')}`;
}
/* ── Migration ── */
let migrationPollTimer = null;
function showMigrationMsg(msg, type) {
const el = document.getElementById('migration-status-msg');
el.textContent = msg;
el.className = `alert alert-${type}`;
el.style.display = '';
}
function updateMigrationUI(m) {
// Status badge
const badge = document.getElementById('migration-status-badge');
badge.textContent = (m.status || 'idle').charAt(0).toUpperCase() + (m.status || 'idle').slice(1);
badge.className = `badge badge-migration badge-migration--${m.status || 'idle'}`;
const isActive = m.status === 'running' || m.status === 'paused';
const isCompleted = m.status === 'completed';
// Progress section
const progressSection = document.getElementById('migration-progress-section');
progressSection.style.display = (isActive || isCompleted) ? '' : 'none';
if (m.total_blobs > 0) {
const pct = Math.round((m.migrated_blobs / m.total_blobs) * 100);
document.getElementById('migration-progress-fill').style.width = `${pct}%`;
document.getElementById('migration-progress-label').textContent =
`${m.migrated_blobs.toLocaleString()} / ${m.total_blobs.toLocaleString()} blobs (${pct}%)`;
document.getElementById('migration-bytes-label').textContent =
`${formatBytes(m.migrated_bytes)} transferred`;
if (m.throughput_bytes_per_sec && m.status === 'running') {
document.getElementById('migration-throughput').textContent =
`${formatBytes(Math.round(m.throughput_bytes_per_sec))}/s`;
const remaining = m.total_blobs - m.migrated_blobs;
if (remaining > 0 && m.throughput_bytes_per_sec > 0) {
const avgBlobSize = m.migrated_bytes / Math.max(m.migrated_blobs, 1);
const etaSecs = Math.round((remaining * avgBlobSize) / m.throughput_bytes_per_sec);
const etaMin = Math.ceil(etaSecs / 60);
document.getElementById('migration-eta').textContent =
`~${etaMin} min remaining`;
}
} else {
document.getElementById('migration-throughput').textContent = '';
document.getElementById('migration-eta').textContent = '';
}
}
// Failed blobs section
const failedSection = document.getElementById('migration-failed-section');
if (m.failed_blobs && m.failed_blobs.length > 0) {
failedSection.style.display = '';
document.getElementById('migration-failed-count').textContent = m.failed_blobs.length;
document.getElementById('migration-failed-list').textContent = m.failed_blobs.join('\n');
} else {
failedSection.style.display = 'none';
}
// Button visibility
document.getElementById('btn-start-migration').style.display =
(!isActive && !isCompleted) ? '' : 'none';
document.getElementById('btn-pause-migration').style.display =
m.status === 'running' ? '' : 'none';
document.getElementById('btn-resume-migration').style.display =
m.status === 'paused' ? '' : 'none';
document.getElementById('btn-verify-migration').style.display =
isCompleted ? '' : 'none';
document.getElementById('btn-complete-migration').style.display =
isCompleted ? '' : 'none';
}
async function loadMigrationStatus() {
try {
const resp = await fetch(`${API}/admin/storage/migration`, {
headers: headers(),
credentials: 'same-origin'
});
if (!resp.ok) return;
const m = await resp.json();
updateMigrationUI(m);
// Auto-poll while running
if (m.status === 'running') {
if (!migrationPollTimer) {
migrationPollTimer = setInterval(loadMigrationStatus, 2000);
}
} else if (migrationPollTimer) {
clearInterval(migrationPollTimer);
migrationPollTimer = null;
}
} catch (_e) { /* ignore */ }
}
async function startMigration() {
const btn = document.getElementById('btn-start-migration');
btn.disabled = true;
try {
const resp = await fetch(`${API}/admin/storage/migration/start`, {
method: 'POST',
headers: headers(),
credentials: 'same-origin',
body: JSON.stringify({ concurrency: 4 })
});
if (resp.ok) {
showMigrationMsg(t('admin.migration_started') || 'Migration started', 'success');
loadMigrationStatus();
} else {
const e = await resp.json().catch(() => ({}));
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
}
} catch (e) {
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
}
btn.disabled = false;
}
async function pauseMigration() {
try {
const resp = await fetch(`${API}/admin/storage/migration/pause`, {
method: 'POST', headers: headers(), credentials: 'same-origin'
});
if (resp.ok) {
showMigrationMsg(t('admin.migration_paused_msg') || 'Migration paused', 'success');
loadMigrationStatus();
}
} catch (_e) { /* ignore */ }
}
async function resumeMigration() {
try {
const resp = await fetch(`${API}/admin/storage/migration/resume`, {
method: 'POST', headers: headers(), credentials: 'same-origin'
});
if (resp.ok) {
showMigrationMsg(t('admin.migration_resumed_msg') || 'Migration resumed', 'success');
loadMigrationStatus();
}
} catch (_e) { /* ignore */ }
}
async function verifyMigration() {
const btn = document.getElementById('btn-verify-migration');
btn.disabled = true;
btn.innerHTML = `<i class="fas fa-spinner fa-spin"></i> ${escapeHtml(t('admin.migration_verifying') || 'Verifying...')}`;
const resultDiv = document.getElementById('migration-verify-result');
try {
const resp = await fetch(`${API}/admin/storage/migration/verify`, {
method: 'POST',
headers: headers(),
credentials: 'same-origin',
body: JSON.stringify({ sample_size: 100 })
});
const r = await resp.json();
resultDiv.style.display = '';
if (r.passed) {
resultDiv.innerHTML = `<div class="discovery-result ok"><strong><i class="fas fa-check-circle"></i> ${escapeHtml(t('admin.migration_verify_passed') || 'Verification passed')}</strong><p>${r.sample_checked} blobs checked, ${r.pg_blob_count} total in database</p></div>`;
} else {
const issues = [];
if (r.missing_in_target.length) issues.push(`${r.missing_in_target.length} missing`);
if (r.size_mismatches.length) issues.push(`${r.size_mismatches.length} size mismatches`);
resultDiv.innerHTML = `<div class="discovery-result fail"><strong><i class="fas fa-times-circle"></i> ${escapeHtml(t('admin.migration_verify_failed') || 'Verification failed')}</strong><p>${issues.join(', ')}</p></div>`;
}
} catch (e) {
resultDiv.style.display = '';
resultDiv.innerHTML = `<div class="discovery-result fail"><i class="fas fa-times-circle"></i> Error: ${escapeHtml(e.message)}</div>`;
}
btn.disabled = false;
btn.innerHTML = `<i class="fas fa-check-double"></i> ${escapeHtml(t('admin.migration_verify') || 'Verify Integrity')}`;
}
async function completeMigration() {
try {
const resp = await fetch(`${API}/admin/storage/migration/complete`, {
method: 'POST', headers: headers(), credentials: 'same-origin'
});
if (resp.ok) {
showMigrationMsg(t('admin.migration_completed_msg') || 'Migration finalized. Restart the server to use the new backend.', 'success');
loadMigrationStatus();
} else {
const e = await resp.json().catch(() => ({}));
showMigrationMsg(`Error: ${e.message || resp.statusText}`, 'error');
}
} catch (e) {
showMigrationMsg(t('admin.error_network', { message: e.message }), 'error');
}
}
async function init() {
try {
const me = await fetch(`${API}/auth/me`, {
@@ -775,6 +1123,9 @@ document.getElementById('tab-btn-users').addEventListener('click', function () {
document.getElementById('tab-btn-oidc').addEventListener('click', function () {
switchTab('oidc', this);
});
document.getElementById('tab-btn-storage').addEventListener('click', function () {
switchTab('storage', this);
});
document.getElementById('ds-registration').addEventListener('change', function () {
toggleRegistration(this.checked);
@@ -797,3 +1148,20 @@ document.getElementById('cu-submit').addEventListener('click', submitCreateUser)
document.getElementById('btn-close-reset-pw').addEventListener('click', closeResetPasswordModal);
document.getElementById('rp-submit').addEventListener('click', submitResetPassword);
/* ── Storage event listeners ── */
document.querySelectorAll('input[name="storage-backend"]').forEach((r) => {
r.addEventListener('change', function () {
toggleS3Form(this.value === 's3');
});
});
document.getElementById('storage-preset').addEventListener('change', onStoragePresetChange);
document.getElementById('btn-test-storage').addEventListener('click', testStorageConnection);
document.getElementById('btn-save-storage').addEventListener('click', saveStorageSettings);
/* ── Migration event listeners ── */
document.getElementById('btn-start-migration').addEventListener('click', startMigration);
document.getElementById('btn-pause-migration').addEventListener('click', pauseMigration);
document.getElementById('btn-resume-migration').addEventListener('click', resumeMigration);
document.getElementById('btn-verify-migration').addEventListener('click', verifyMigration);
document.getElementById('btn-complete-migration').addEventListener('click', completeMigration);
+44 -1
View File
@@ -566,7 +566,50 @@
"error_password_short": "Password must be at least 8 characters",
"error_generic": "Failed",
"error_network": "Network error: {{message}}",
"error_create_user": "Failed to create user"
"error_create_user": "Failed to create user",
"tab_storage": "Storage",
"storage_title": "Storage Backend",
"storage_current_backend": "Active Backend",
"storage_total_blobs": "Total Blobs",
"storage_total_size": "Total Size",
"storage_dedup_ratio": "Dedup Ratio",
"storage_backend": "Backend Type",
"storage_local": "Local Filesystem",
"storage_s3": "S3-Compatible",
"storage_provider_preset": "Provider Preset",
"storage_preset_custom": "Custom",
"storage_endpoint_url": "Endpoint URL",
"storage_endpoint_hint": "Leave empty for Amazon S3 default",
"storage_bucket": "Bucket",
"storage_region": "Region",
"storage_access_key": "Access Key ID",
"storage_secret_key": "Secret Access Key",
"storage_secret_configured": "A secret key is already configured",
"storage_key_placeholder": "Leave empty to keep current value",
"storage_path_style": "Force Path Style",
"storage_path_style_hint": "Required for MinIO and some S3-compatible providers",
"storage_test_connection": "Test Connection",
"storage_test_success": "Connection successful",
"storage_test_failure": "Connection failed",
"storage_save": "Save",
"storage_saved": "Storage settings saved successfully",
"storage_migration": "Backend Migration",
"storage_migration_coming_soon": "Backend migration will be available in a future update.",
"migration_status_label": "Status:",
"migration_start": "Start Migration",
"migration_pause": "Pause",
"migration_resume": "Resume",
"migration_verify": "Verify Integrity",
"migration_complete": "Finalize",
"migration_started": "Migration started",
"migration_paused_msg": "Migration paused",
"migration_resumed_msg": "Migration resumed",
"migration_completed_msg": "Migration finalized. Restart the server to use the new backend.",
"migration_verifying": "Verifying…",
"migration_verify_passed": "Verification passed",
"migration_verify_failed": "Verification failed",
"migration_failed_blobs": "failed blobs",
"testing": "Testing…"
},
"profile": {
"page_title": "Profile",
+44 -1
View File
@@ -566,7 +566,50 @@
"error_password_short": "La contraseña debe tener al menos 8 caracteres",
"error_generic": "Error",
"error_network": "Error de red: {{message}}",
"error_create_user": "Error al crear usuario"
"error_create_user": "Error al crear usuario",
"tab_storage": "Almacenamiento",
"storage_title": "Backend de Almacenamiento",
"storage_current_backend": "Backend Activo",
"storage_total_blobs": "Total de Blobs",
"storage_total_size": "Tamaño Total",
"storage_dedup_ratio": "Ratio de Dedup",
"storage_backend": "Tipo de Backend",
"storage_local": "Sistema de Archivos Local",
"storage_s3": "Compatible con S3",
"storage_provider_preset": "Proveedor Preconfigurado",
"storage_preset_custom": "Personalizado",
"storage_endpoint_url": "URL del Endpoint",
"storage_endpoint_hint": "Dejar vacío para usar Amazon S3 por defecto",
"storage_bucket": "Bucket",
"storage_region": "Región",
"storage_access_key": "Access Key ID",
"storage_secret_key": "Secret Access Key",
"storage_secret_configured": "Ya hay una clave secreta configurada",
"storage_key_placeholder": "Dejar vacío para mantener el valor actual",
"storage_path_style": "Forzar Path Style",
"storage_path_style_hint": "Requerido para MinIO y algunos proveedores compatibles con S3",
"storage_test_connection": "Probar Conexión",
"storage_test_success": "Conexión exitosa",
"storage_test_failure": "Conexión fallida",
"storage_save": "Guardar",
"storage_saved": "Configuración de almacenamiento guardada correctamente",
"storage_migration": "Migración de Backend",
"storage_migration_coming_soon": "La migración de backend estará disponible en una futura actualización.",
"migration_status_label": "Estado:",
"migration_start": "Iniciar Migración",
"migration_pause": "Pausar",
"migration_resume": "Reanudar",
"migration_verify": "Verificar Integridad",
"migration_complete": "Finalizar",
"migration_started": "Migración iniciada",
"migration_paused_msg": "Migración pausada",
"migration_resumed_msg": "Migración reanudada",
"migration_completed_msg": "Migración finalizada. Reinicia el servidor para usar el nuevo backend.",
"migration_verifying": "Verificando…",
"migration_verify_passed": "Verificación exitosa",
"migration_verify_failed": "Verificación fallida",
"migration_failed_blobs": "blobs fallidos",
"testing": "Probando…"
},
"profile": {
"page_title": "Perfil",