docs: update documentation to reflect 100% blob storage model
Rewrite documentation to match the new architecture where all file metadata lives in PostgreSQL and content is stored as content-addressed blobs via DedupService. Updated files: - internal-architecture.md: complete rewrite — new DB schema, blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated DI container, service groups, architecture diagram, data flows - file-system-safety.md: repurposed as storage-safety.md — covers PostgreSQL ACID guarantees + DedupService atomic writes - caching-architecture.md: updated repo references to blob repos, removed write-behind cache section, updated upload/download flows - trash-feature-summary.md: rewritten for soft-delete model (is_trashed flag, trash_items VIEW, TrashDbRepository) - share-integration.md: clarified ShareFsRepository scope, updated DI snippet, added blob storage context note - deduplication.md: updated DI snippet (dedup injected into repos) - deployment.md: updated feature matrix (file storage requires DB) - important-delta-sync-implementation.md: updated DI references Removed legacy references: IdMappingPort, StorageMediator, WriteBehindCache, FsFileRepository, FsFolderRepository, TrashFsRepository, folder_ids.json, file_ids.json.
This commit is contained in:
Dionisio
+274
-370
@@ -1,370 +1,274 @@
|
||||
# 04 - Caching Architecture
|
||||
|
||||
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
||||
|
||||
## Cache Layers Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 2: MMAP (memmap2, 10-100MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Streaming (FramedRead, ≥100MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 4: File Metadata Cache (adaptive TTL) │ All file ops
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 5: Write-Behind Cache (<256KB uploads) │ Uploads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 6: Buffer Pool (reusable I/O buffers) │ Compression
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Layer 0: HTTP Cache Middleware
|
||||
|
||||
**File**: `src/interfaces/middleware/cache.rs`
|
||||
|
||||
Generic HTTP caching layer applied to API endpoints.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max entries | 1,000 |
|
||||
| Default max-age | 60 seconds |
|
||||
| Eviction | LRU (oldest 10% when full) |
|
||||
| Cleanup | Background task every 5 minutes |
|
||||
|
||||
Features:
|
||||
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
||||
- `Cache-Control` header injection
|
||||
- Implements Tower `Layer` + `Service` traits for Axum integration
|
||||
- Per-request key: method + URI
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: File Content Cache (Download Tier 1)
|
||||
|
||||
**File**: `src/infrastructure/services/file_content_cache.rs`
|
||||
|
||||
In-memory LRU cache for small files, served directly from RAM.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 10 MB per file |
|
||||
| Max total cache size | 512 MB |
|
||||
| Max entries | 10,000 |
|
||||
| Structure | `lru::LruCache<String, CacheEntry>` |
|
||||
| Latency | ~0.1ms |
|
||||
|
||||
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
||||
|
||||
Methods:
|
||||
- `should_cache(size)` -- checks if file fits in cache
|
||||
- `get(file_id)` → `Option<(Bytes, String, String)>` -- returns (data, etag, content_type)
|
||||
- `put(file_id, content, etag, content_type)` -- inserts with LRU eviction
|
||||
- `invalidate(file_id)`, `clear()`
|
||||
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
||||
|
||||
Port: implements **ContentCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: MMAP (Download Tier 2)
|
||||
|
||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
Memory-mapped I/O for medium files using `memmap2`.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | 10 MB - 100 MB |
|
||||
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
||||
| Latency | ~1-5ms |
|
||||
|
||||
Current implementation copies mmap'd data to `Bytes` (`Bytes::copy_from_slice(&mmap[..])`). Not true zero-copy, but still benefits from kernel page cache.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Streaming (Download Tier 3)
|
||||
|
||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
Chunked streaming for large files using tokio-util codecs.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | ≥100 MB |
|
||||
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
||||
| Implementation | `FramedRead` + `BytesCodec` |
|
||||
| RAM usage | Near zero (one chunk at a time) |
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: File Metadata Cache
|
||||
|
||||
**File**: `src/infrastructure/services/file_metadata_cache.rs`
|
||||
|
||||
Caches filesystem metadata (existence, size, MIME type, timestamps) to avoid repeated `stat()` calls.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Default file TTL | 60 seconds |
|
||||
| Default directory TTL | 120 seconds |
|
||||
| Max entries | 10,000 |
|
||||
| Adaptive TTL multiplier | 5x for popular entries (≥10 accesses) |
|
||||
| LRU eviction | Frees 10% capacity when full |
|
||||
| Cleanup | Background task runs periodically |
|
||||
|
||||
**CachedMetadata:**
|
||||
```rust
|
||||
pub struct FileMetadata {
|
||||
pub path: PathBuf,
|
||||
pub exists: bool,
|
||||
pub entry_type: CacheEntryType, // File | Directory | Unknown
|
||||
pub size: Option<u64>,
|
||||
pub mime_type: Option<String>,
|
||||
pub created_at: Option<u64>,
|
||||
pub modified_at: Option<u64>,
|
||||
pub last_access: Instant,
|
||||
pub expires_at: Instant,
|
||||
pub access_count: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**Adaptive TTL**: entries accessed ≥10 times get 5x the configured TTL, keeping frequently accessed file metadata in cache longer.
|
||||
|
||||
Port: implements **MetadataCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 5: Write-Behind Cache
|
||||
|
||||
**File**: `src/infrastructure/services/write_behind_cache.rs`
|
||||
|
||||
Buffers small uploads in RAM and confirms immediately. Flushes to disk asynchronously.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 1 MB per file |
|
||||
| Max total cache | 100 MB |
|
||||
| Max pending duration | 30 seconds |
|
||||
| Flush interval | 100 ms |
|
||||
| Write strategy | Atomic (temp file + rename) |
|
||||
|
||||
Architecture:
|
||||
- `put_pending(file_id, content, target_path)` stores bytes in `HashMap<String, PendingWrite>`
|
||||
- Background `flush_worker` processes **FlushCommands** via `mpsc` channel
|
||||
- Periodic checker force-flushes entries older than 30 seconds
|
||||
- `get_pending(file_id)` serves reads while data is still in RAM (before flush)
|
||||
|
||||
Port: implements **WriteBehindCachePort** trait.
|
||||
|
||||
Statistics:
|
||||
```rust
|
||||
pub struct WriteBehindStatsDto {
|
||||
pub pending_count: usize,
|
||||
pub pending_bytes: usize,
|
||||
pub total_writes: u64,
|
||||
pub total_bytes_written: u64,
|
||||
pub cache_hits: u64,
|
||||
pub avg_flush_time_us: u64,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Layer 6: Buffer Pool
|
||||
|
||||
**File**: `src/infrastructure/services/buffer_pool.rs`
|
||||
|
||||
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Buffer size | 64 KB |
|
||||
| Max buffers | 100 |
|
||||
| Buffer TTL | 60 seconds |
|
||||
| Concurrency control | `tokio::sync::Semaphore` |
|
||||
|
||||
Features:
|
||||
- `get_buffer()` -- borrows a buffer (blocks if pool exhausted)
|
||||
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
||||
- Expired buffers are cleaned periodically via `start_cleaner()`
|
||||
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All cache-related config in `src/common/config.rs`:
|
||||
|
||||
```rust
|
||||
pub struct CacheConfig {
|
||||
pub file_ttl_ms: u64, // default: 60,000 (1 min)
|
||||
pub directory_ttl_ms: u64, // default: 120,000 (2 min)
|
||||
pub max_entries: usize, // default: 10,000
|
||||
}
|
||||
|
||||
pub struct ResourceConfig {
|
||||
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
||||
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
||||
pub max_in_memory_file_size_mb: u64, // 50 MB
|
||||
}
|
||||
```
|
||||
|
||||
## Download Flow
|
||||
|
||||
```
|
||||
Request → ETag check (304?) → Range request (206?)
|
||||
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
||||
→ file size < 100MB? → Tier 2: MMAP (kernel page cache)
|
||||
→ file size ≥ 100MB → Tier 3: Streaming (chunked)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Range Requests (HTTP 206 Partial Content)
|
||||
|
||||
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
**Crate**: `http-range-header = "0.4"` for parsing.
|
||||
|
||||
### Request Processing Flow
|
||||
|
||||
```
|
||||
Range header present?
|
||||
├─ parse_range_header(range_str)
|
||||
│ ├─ Parse OK → ranges.validate(file_size)
|
||||
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
||||
│ │ │ ├─ Stream OK → 206 Partial Content
|
||||
│ │ │ └─ Stream Err → fall through to normal download (200)
|
||||
│ │ └─ Invalid → 416 Range Not Satisfiable
|
||||
│ └─ Parse Err → fall through to normal download (200)
|
||||
└─ No Range header → normal 3-tier download
|
||||
```
|
||||
|
||||
### Response Headers (206)
|
||||
|
||||
| Header | Value |
|
||||
|---|---|
|
||||
| `Content-Type` | File MIME type |
|
||||
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
||||
| `Content-Length` | Range length (end - start + 1) |
|
||||
| `Accept-Ranges` | `bytes` |
|
||||
| `ETag` | `"{file_id}-{modified_at}"` |
|
||||
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
||||
|
||||
### 416 Range Not Satisfiable
|
||||
|
||||
Returned when `ranges.validate(file_size)` fails:
|
||||
```
|
||||
HTTP/1.1 416 Range Not Satisfiable
|
||||
Content-Range: bytes */12345
|
||||
```
|
||||
|
||||
### File Seek Implementation
|
||||
|
||||
`get_file_range_stream()` at the repository level:
|
||||
|
||||
```rust
|
||||
async fn get_file_range_stream(
|
||||
&self, id: &str, start: u64, end: Option<u64>,
|
||||
) -> Result<Box<dyn Stream<...> + Send>, DomainError>
|
||||
```
|
||||
|
||||
1. Opens the file with `TokioFile::open()`
|
||||
2. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
||||
3. Limits read to `range_length` via `fh.take(range_length)`
|
||||
4. Wraps in `FramedRead` + `BytesCodec`
|
||||
|
||||
Adaptive chunk size:
|
||||
|
||||
| Range size | Chunk size |
|
||||
|---|---|
|
||||
| ≤ 1 MB | 8 KB |
|
||||
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
||||
|
||||
### Tier Interaction
|
||||
|
||||
Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They always use direct file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Multipart ranges not supported**: only the first range in a multi-range request is served. Additional ranges are ignored.
|
||||
- **`If-Range` not handled**: no conditional range support.
|
||||
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
||||
|
||||
---
|
||||
|
||||
## Upload Flow
|
||||
|
||||
```
|
||||
Request → file size < 256KB? → Write-behind cache (instant 201, async flush)
|
||||
→ file size < 1MB? → Buffered write (sync)
|
||||
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to temp + rename)
|
||||
```
|
||||
|
||||
### Upload Strategy Selection
|
||||
|
||||
**File**: `src/application/services/file_upload_service.rs`
|
||||
|
||||
```rust
|
||||
pub enum UploadStrategy {
|
||||
WriteBehind, // < 256 KB — instant response, async disk write
|
||||
Buffered, // 256 KB – 1 MB — sync write to final path
|
||||
Streaming, // ≥ 1 MB — chunk-by-chunk write to temp file + rename
|
||||
}
|
||||
```
|
||||
|
||||
| Constant | Value |
|
||||
|---|---|
|
||||
| `WRITE_BEHIND_THRESHOLD` | 256 KB |
|
||||
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
||||
|
||||
### Handler-Level Buffering
|
||||
|
||||
Both upload handlers (`upload_file` and `upload_file_with_cache`) buffer the **entire multipart body in RAM** as `Vec<Bytes>` before calling the service layer:
|
||||
|
||||
```rust
|
||||
let mut chunks: Vec<Bytes> = Vec::new();
|
||||
while let Some(chunk) = field.chunk().await {
|
||||
chunks.push(chunk);
|
||||
}
|
||||
// All bytes are now in RAM
|
||||
upload_service.smart_upload(..., chunks, total_size).await
|
||||
```
|
||||
|
||||
The "streaming" in `UploadStrategy::Streaming` refers to the **service→repository** path, not the HTTP-body→disk path. By the time `save_file_from_stream()` is called, data is already in memory.
|
||||
|
||||
### Streaming Path (≥ 1 MB): Service → Repository
|
||||
|
||||
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
||||
|
||||
```rust
|
||||
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
||||
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
||||
```
|
||||
|
||||
**`save_file_from_stream()` implementation** (`file_fs_write_repository.rs`):
|
||||
|
||||
1. Resolves target path + generates unique name if collision
|
||||
2. Creates temp file: `{target_path}.tmp.upload`
|
||||
3. Iterates stream, writing each chunk with `fh.write_all(&chunk)`
|
||||
4. Calls `fh.flush()` + `fh.sync_all()` for durability
|
||||
5. Atomic rename: `fs::rename(temp_path, final_path)`
|
||||
6. Post-write: ID mapping, cache invalidation, metadata update
|
||||
|
||||
### Buffered Path (256 KB - 1 MB)
|
||||
|
||||
Uses `save_file()` -- writes all bytes directly to the **final path** (no temp file). For larger content, writes in chunks of **ResourceConfig.chunk_size_bytes** (1 MB).
|
||||
|
||||
### Write-Behind Path (< 256 KB)
|
||||
|
||||
See **Layer 5** above. Instant `201`, background flush within 30 seconds.
|
||||
|
||||
### Dedup Pre-Check
|
||||
|
||||
Runs for **all upload strategies** before writing. Re-combines all chunks into a single `Vec<u8>` for hash computation, which means data is temporarily duplicated in RAM during dedup processing.
|
||||
# 04 - Caching Architecture
|
||||
|
||||
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
||||
|
||||
## Cache Layers Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 2: MMAP (memmap2, 10-100MB blobs) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Streaming (FramedRead, ≥100MB blobs) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 4: Buffer Pool (reusable I/O buffers) │ Compression
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
> **Note:** File metadata (name, size, MIME type, folder) is served from PostgreSQL — no separate filesystem metadata cache is needed.
|
||||
|
||||
---
|
||||
|
||||
## Layer 0: HTTP Cache Middleware
|
||||
|
||||
**File**: `src/interfaces/middleware/cache.rs`
|
||||
|
||||
Generic HTTP caching layer applied to API endpoints.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max entries | 1,000 |
|
||||
| Default max-age | 60 seconds |
|
||||
| Eviction | LRU (oldest 10% when full) |
|
||||
| Cleanup | Background task every 5 minutes |
|
||||
|
||||
Features:
|
||||
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
||||
- `Cache-Control` header injection
|
||||
- Implements Tower `Layer` + `Service` traits for Axum integration
|
||||
- Per-request key: method + URI
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: File Content Cache (Download Tier 1)
|
||||
|
||||
**File**: `src/infrastructure/services/file_content_cache.rs`
|
||||
|
||||
In-memory LRU cache for small files, served directly from RAM.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 10 MB per file |
|
||||
| Max total cache size | 512 MB |
|
||||
| Max entries | 10,000 |
|
||||
| Structure | `lru::LruCache<String, CacheEntry>` |
|
||||
| Latency | ~0.1ms |
|
||||
|
||||
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
||||
|
||||
Methods:
|
||||
- `should_cache(size)` — checks if file fits in cache
|
||||
- `get(file_id)` → `Option<(Bytes, String, String)>` — returns (data, etag, content_type)
|
||||
- `put(file_id, content, etag, content_type)` — inserts with LRU eviction
|
||||
- `invalidate(file_id)`, `clear()`
|
||||
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
||||
|
||||
Port: implements **ContentCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: MMAP (Download Tier 2)
|
||||
|
||||
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
Memory-mapped I/O for medium blobs using `memmap2`.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | 10 MB - 100 MB |
|
||||
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
||||
| Latency | ~1-5ms |
|
||||
|
||||
The blob file (`.blobs/{prefix}/{hash}.blob`) is memory-mapped and its contents copied to `Bytes`. Benefits from kernel page cache for frequently accessed blobs.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Streaming (Download Tier 3)
|
||||
|
||||
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
Chunked streaming for large blobs using tokio-util codecs.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | ≥100 MB |
|
||||
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
||||
| Implementation | `FramedRead` + `BytesCodec` |
|
||||
| RAM usage | Near zero (one chunk at a time) |
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: Buffer Pool
|
||||
|
||||
**File**: `src/infrastructure/services/buffer_pool.rs`
|
||||
|
||||
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Buffer size | 64 KB |
|
||||
| Max buffers | 100 |
|
||||
| Buffer TTL | 60 seconds |
|
||||
| Concurrency control | `tokio::sync::Semaphore` |
|
||||
|
||||
Features:
|
||||
- `get_buffer()` — borrows a buffer (blocks if pool exhausted)
|
||||
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
||||
- Expired buffers are cleaned periodically via `start_cleaner()`
|
||||
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All cache-related config in `src/common/config.rs`:
|
||||
|
||||
```rust
|
||||
pub struct ResourceConfig {
|
||||
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
||||
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
||||
pub max_in_memory_file_size_mb: u64, // 50 MB
|
||||
}
|
||||
```
|
||||
|
||||
## Download Flow
|
||||
|
||||
```
|
||||
Request → ETag check (304?) → Range request (206?)
|
||||
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
||||
→ file size < 100MB? → Tier 2: MMAP (kernel page cache on blob file)
|
||||
→ file size ≥ 100MB → Tier 3: Streaming (chunked from blob file)
|
||||
```
|
||||
|
||||
In all tiers, metadata (file name, size, MIME type) comes from a PostgreSQL `SELECT` on `storage.files`. Content is read from the DedupService blob at `.blobs/{prefix}/{hash}.blob`.
|
||||
|
||||
---
|
||||
|
||||
## Range Requests (HTTP 206 Partial Content)
|
||||
|
||||
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
**Crate**: `http-range-header = "0.4"` for parsing.
|
||||
|
||||
### Request Processing Flow
|
||||
|
||||
```
|
||||
Range header present?
|
||||
├─ parse_range_header(range_str)
|
||||
│ ├─ Parse OK → ranges.validate(file_size)
|
||||
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
||||
│ │ │ ├─ Stream OK → 206 Partial Content
|
||||
│ │ │ └─ Stream Err → fall through to normal download (200)
|
||||
│ │ └─ Invalid → 416 Range Not Satisfiable
|
||||
│ └─ Parse Err → fall through to normal download (200)
|
||||
└─ No Range header → normal 3-tier download
|
||||
```
|
||||
|
||||
### Response Headers (206)
|
||||
|
||||
| Header | Value |
|
||||
|---|---|
|
||||
| `Content-Type` | File MIME type |
|
||||
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
||||
| `Content-Length` | Range length (end - start + 1) |
|
||||
| `Accept-Ranges` | `bytes` |
|
||||
| `ETag` | `"{file_id}-{modified_at}"` |
|
||||
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
||||
|
||||
### 416 Range Not Satisfiable
|
||||
|
||||
Returned when `ranges.validate(file_size)` fails:
|
||||
```
|
||||
HTTP/1.1 416 Range Not Satisfiable
|
||||
Content-Range: bytes */12345
|
||||
```
|
||||
|
||||
### Blob File Seek Implementation
|
||||
|
||||
`get_file_range_stream()` at the repository level:
|
||||
|
||||
1. Resolves blob path from `blob_hash` via DedupService
|
||||
2. Opens the blob file with `TokioFile::open()`
|
||||
3. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
||||
4. Limits read to `range_length` via `fh.take(range_length)`
|
||||
5. Wraps in `FramedRead` + `BytesCodec`
|
||||
|
||||
Adaptive chunk size:
|
||||
|
||||
| Range size | Chunk size |
|
||||
|---|---|
|
||||
| ≤ 1 MB | 8 KB |
|
||||
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
||||
|
||||
### Tier Interaction
|
||||
|
||||
Range requests **bypass all download tiers** (LRU, MMAP). They always use direct blob file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Multipart ranges not supported**: only the first range in a multi-range request is served.
|
||||
- **`If-Range` not handled**: no conditional range support.
|
||||
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
||||
|
||||
---
|
||||
|
||||
## Upload Flow
|
||||
|
||||
```
|
||||
Request → file size < 1MB? → Buffered write (sync to blob store)
|
||||
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to blob store)
|
||||
```
|
||||
|
||||
### Upload Strategy Selection
|
||||
|
||||
**File**: `src/application/services/file_upload_service.rs`
|
||||
|
||||
```rust
|
||||
pub enum UploadStrategy {
|
||||
Buffered, // < 1 MB — collect bytes, write to blob store
|
||||
Streaming, // ≥ 1 MB — chunk-by-chunk write via save_file_from_stream
|
||||
}
|
||||
```
|
||||
|
||||
| Constant | Value |
|
||||
|---|---|
|
||||
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
||||
|
||||
### Handler-Level Buffering
|
||||
|
||||
Upload handlers buffer the multipart body in RAM as `Vec<Bytes>` before calling the service layer:
|
||||
|
||||
```rust
|
||||
let mut chunks: Vec<Bytes> = Vec::new();
|
||||
while let Some(chunk) = field.chunk().await {
|
||||
chunks.push(chunk);
|
||||
}
|
||||
upload_service.smart_upload(..., chunks, total_size).await
|
||||
```
|
||||
|
||||
### Buffered Path (< 1 MB)
|
||||
|
||||
Uses `save_file()` — all bytes are passed to `FileBlobWriteRepository`, which calls `DedupService.store_bytes()` to compute hash and store the blob, then INSERTs metadata into `storage.files`.
|
||||
|
||||
### Streaming Path (≥ 1 MB)
|
||||
|
||||
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
||||
|
||||
```rust
|
||||
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
||||
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
||||
```
|
||||
|
||||
`FileBlobWriteRepository.save_file_from_stream()` collects the stream, stores via DedupService, and INSERTs metadata.
|
||||
|
||||
### Dedup Integration
|
||||
|
||||
Deduplication is handled at the **repository layer** (not the service layer) for all upload strategies. `FileBlobWriteRepository` always calls `DedupService.store_bytes()` which:
|
||||
|
||||
1. Computes SHA-256 hash of content
|
||||
2. Checks if blob already exists (dedup hit → increment ref count, skip write)
|
||||
3. If new → atomic write to `.blobs/{prefix}/{hash}.blob`
|
||||
4. Returns the hash for storage in `storage.files.blob_hash`
|
||||
|
||||
@@ -285,9 +285,12 @@ pub struct CoreServices {
|
||||
// ...
|
||||
}
|
||||
|
||||
// Injected into application services:
|
||||
FileUploadService::new_full(... core.dedup_service.clone())
|
||||
FileManagementService::new_full(... core.dedup_service.clone())
|
||||
// Injected into blob repositories (which handle dedup internally):
|
||||
FileBlobReadRepository::new(pool, core.dedup_service.clone(), folder_repo)
|
||||
FileBlobWriteRepository::new(pool, core.dedup_service.clone(), folder_repo)
|
||||
|
||||
// Also injected into FileManagementService for ref cleanup on delete:
|
||||
FileManagementService::new_full(write, read, trash, core.dedup_service.clone())
|
||||
```
|
||||
|
||||
## Persistence
|
||||
|
||||
+4
-4
@@ -174,17 +174,17 @@ Hardcoded defaults in `src/common/config.rs`:
|
||||
|
||||
| Feature | Requires DB | Requires Auth | Feature Flag |
|
||||
|---|---|---|---|
|
||||
| File storage | No | No | Always on |
|
||||
| File storage | Yes | No | Always on |
|
||||
| Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` |
|
||||
| OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` |
|
||||
| File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` |
|
||||
| Trash | No | No | `OXICLOUD_ENABLE_TRASH` |
|
||||
| Search | No | No | `OXICLOUD_ENABLE_SEARCH` |
|
||||
| Trash | Yes | No | `OXICLOUD_ENABLE_TRASH` |
|
||||
| Search | Yes | No | `OXICLOUD_ENABLE_SEARCH` |
|
||||
| Favorites | Yes | Yes | Always on (when DB available) |
|
||||
| Recent items | Yes | Yes | Always on (when DB available) |
|
||||
| Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` |
|
||||
| Admin panel | Yes | Yes | Always on (when auth enabled) |
|
||||
| WebDAV | No | Optional | Always on |
|
||||
| WebDAV | Yes | Optional | Always on |
|
||||
| CalDAV | Yes | Yes | Always on (when DB available) |
|
||||
| CardDAV | Yes | Yes | Always on (when DB available) |
|
||||
| Deduplication | No | No | Always on |
|
||||
|
||||
+125
-156
@@ -1,156 +1,125 @@
|
||||
# 03 - File System Safety
|
||||
|
||||
OxiCloud ensures data integrity and durability during file operations through atomic writes, fsync, and directory synchronization. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss.
|
||||
|
||||
---
|
||||
|
||||
## The Problem: Buffered I/O
|
||||
|
||||
Standard filesystem operations use buffered I/O by default:
|
||||
|
||||
```rust
|
||||
// This operation may not immediately persist to disk
|
||||
fs::write(path, content)
|
||||
```
|
||||
|
||||
When an application writes data, the OS typically:
|
||||
|
||||
1. Accepts the write into memory buffers
|
||||
2. Acknowledges completion to the application
|
||||
3. Schedules the actual disk write for later
|
||||
|
||||
A crash during that window means data loss -- the data exists only in memory buffers that haven't been flushed.
|
||||
|
||||
---
|
||||
|
||||
## OxiCloud's Approach
|
||||
|
||||
All safety mechanisms live in the **FileSystemUtils** service.
|
||||
|
||||
### Atomic Write Pattern
|
||||
|
||||
Files are written using write-then-rename:
|
||||
|
||||
```rust
|
||||
/// Writes data to a file with fsync to ensure durability
|
||||
/// Uses a safe atomic write pattern: write to temp file, fsync, rename
|
||||
pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError>
|
||||
```
|
||||
|
||||
Steps:
|
||||
1. Write to a temporary file in the same directory
|
||||
2. Call `fsync` to ensure data is on disk
|
||||
3. Atomically rename the temp file to the target file
|
||||
4. Sync the parent directory to ensure the rename is persisted
|
||||
|
||||
### Directory Synchronization
|
||||
|
||||
```rust
|
||||
/// Creates directories with fsync
|
||||
pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||
```
|
||||
|
||||
Directories are created, their entries persisted to disk, and parent directories synchronized too.
|
||||
|
||||
### Rename and Delete Operations
|
||||
|
||||
```rust
|
||||
/// Renames a file or directory with proper syncing
|
||||
pub async fn rename_with_sync<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> Result<(), IoError>
|
||||
|
||||
/// Removes a file with directory syncing
|
||||
pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||
```
|
||||
|
||||
Both complete the operation itself, then update and sync the parent directory entry.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### fsync on Files
|
||||
|
||||
```rust
|
||||
// Write file content
|
||||
file.write_all(contents).await?;
|
||||
|
||||
// Ensure data is synced to disk
|
||||
file.flush().await?;
|
||||
file.sync_all().await?;
|
||||
```
|
||||
|
||||
`sync_all()` instructs the OS to flush data and metadata to the physical storage device.
|
||||
|
||||
### fsync on Directories
|
||||
|
||||
```rust
|
||||
// Sync a directory to ensure its contents (entries) are durable
|
||||
async fn sync_directory<P: AsRef<Path>>(path: P) -> Result<(), IoError> {
|
||||
let dir_file = OpenOptions::new().read(true).open(path).await?;
|
||||
dir_file.sync_all().await
|
||||
}
|
||||
```
|
||||
|
||||
Required after any operation that modifies directory entries (create, rename, delete).
|
||||
|
||||
---
|
||||
|
||||
## Usage in the Codebase
|
||||
|
||||
### File Write Repository
|
||||
|
||||
```rust
|
||||
// Write the file to disk using atomic write with fsync
|
||||
tokio::time::timeout(
|
||||
self.config.timeouts.file_write_timeout(),
|
||||
FileSystemUtils::atomic_write(&abs_path, &content)
|
||||
).await
|
||||
```
|
||||
|
||||
### File Move Operations
|
||||
|
||||
```rust
|
||||
// Move the file physically with fsync
|
||||
time::timeout(
|
||||
self.config.timeouts.file_timeout(),
|
||||
FileSystemUtils::rename_with_sync(&old_abs_path, &new_abs_path)
|
||||
).await
|
||||
```
|
||||
|
||||
### Directory Creation
|
||||
|
||||
```rust
|
||||
// Ensure the parent directory exists with proper syncing
|
||||
self.ensure_parent_directory(&abs_path).await?;
|
||||
|
||||
// Implementation uses FileSystemUtils
|
||||
async fn ensure_parent_directory(&self, abs_path: &PathBuf) -> FileRepositoryResult<()> {
|
||||
if let Some(parent) = abs_path.parent() {
|
||||
time::timeout(
|
||||
self.config.timeouts.dir_timeout(),
|
||||
FileSystemUtils::create_dir_with_sync(parent)
|
||||
).await
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Data durability** -- critical data is synced to persistent storage
|
||||
2. **Crash resilience** -- recovery from unexpected failures without data loss
|
||||
3. **Consistency** -- file operations maintain a consistent filesystem state
|
||||
4. **Atomic operations** -- file writes appear as all-or-nothing
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
Syncing to disk costs more than buffered writes. OxiCloud mitigates this by:
|
||||
|
||||
1. Applying these measures only to critical operations
|
||||
2. Using timeouts to prevent indefinite blocking
|
||||
3. Implementing parallel processing for large files
|
||||
|
||||
The tradeoff favors safety for critical data while maintaining good performance for most operations.
|
||||
# 03 - Storage Safety
|
||||
|
||||
OxiCloud ensures data integrity and durability through a combination of PostgreSQL transactional guarantees and atomic blob writes. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss.
|
||||
|
||||
---
|
||||
|
||||
## Storage Model
|
||||
|
||||
OxiCloud uses a **100% blob storage model**:
|
||||
|
||||
- **Metadata** (file names, folder hierarchy, sizes, MIME types, trash status) lives in **PostgreSQL** — protected by ACID transactions.
|
||||
- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob` — protected by atomic writes and fsync.
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL Safety (Metadata)
|
||||
|
||||
All file and folder metadata operations use PostgreSQL transactions:
|
||||
|
||||
- **Single-row operations** (INSERT, UPDATE, DELETE) are inherently atomic.
|
||||
- **Multi-step operations** (e.g., move file: UPDATE folder_id + UPDATE path) use explicit transactions via `sqlx`.
|
||||
- **Foreign key constraints** prevent orphaned records (e.g., files referencing non-existent folders).
|
||||
- **Unique constraints** prevent duplicate names within the same parent folder.
|
||||
- **Soft-delete** for trash (`is_trashed = TRUE`) preserves data until explicit permanent deletion.
|
||||
|
||||
The `storage.trash_items` VIEW provides a unified read interface over trashed files and folders without duplicating data.
|
||||
|
||||
---
|
||||
|
||||
## Blob Storage Safety (Content)
|
||||
|
||||
### DedupService Atomic Writes
|
||||
|
||||
**File:** `src/infrastructure/services/dedup_service.rs`
|
||||
|
||||
When storing file content, DedupService uses the following pattern:
|
||||
|
||||
1. **Hash computation** — SHA-256 hash of content determines the blob path
|
||||
2. **Deduplication check** — if a blob with the same hash exists, only increment the reference counter (no write needed)
|
||||
3. **Atomic write** — if new content:
|
||||
- Write to a temporary file (`.blob.tmp`)
|
||||
- Call `fsync` to ensure data reaches persistent storage
|
||||
- Atomically rename temp file to final path (`.blobs/{prefix}/{hash}.blob`)
|
||||
4. **Reference counting** — track how many files reference each blob
|
||||
|
||||
This ensures that a blob either fully exists or doesn't — no partial writes.
|
||||
|
||||
### FileSystemUtils
|
||||
|
||||
**File:** `src/infrastructure/services/file_system_utils.rs`
|
||||
|
||||
Low-level utilities used internally by DedupService and other infrastructure services:
|
||||
|
||||
```rust
|
||||
/// Atomic write: temp file → fsync → rename
|
||||
pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError>
|
||||
|
||||
/// Directory creation with fsync
|
||||
pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||
|
||||
/// Rename with directory sync
|
||||
pub async fn rename_with_sync<P, Q>(from: P, to: Q) -> Result<(), IoError>
|
||||
|
||||
/// Delete with directory sync
|
||||
pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||
```
|
||||
|
||||
### fsync Guarantees
|
||||
|
||||
- `sync_all()` on written files ensures data and metadata reach the physical storage device
|
||||
- Directory entries are synced after create/rename/delete operations
|
||||
- Prevents data loss during crashes or power failures between OS buffer flush and disk write
|
||||
|
||||
---
|
||||
|
||||
## Transaction Flow: File Upload
|
||||
|
||||
```
|
||||
1. DedupService.store_bytes(content)
|
||||
→ Compute SHA-256 hash
|
||||
→ Check if blob exists (dedup hit → increment ref, return hash)
|
||||
→ Write to .blobs/{prefix}/{hash}.blob.tmp
|
||||
→ fsync + rename → .blobs/{prefix}/{hash}.blob
|
||||
|
||||
2. FileBlobWriteRepository.save_file()
|
||||
→ BEGIN TRANSACTION
|
||||
→ INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
|
||||
→ COMMIT
|
||||
```
|
||||
|
||||
If step 1 fails, no metadata is written. If step 2 fails, the blob exists but is unreferenced (cleaned up by garbage collection). Data is never in an inconsistent state.
|
||||
|
||||
## Transaction Flow: File Deletion
|
||||
|
||||
```
|
||||
1. FileBlobWriteRepository.delete_file_permanently()
|
||||
→ BEGIN TRANSACTION
|
||||
→ DELETE FROM storage.files WHERE id = $1 (captures blob_hash first)
|
||||
→ COMMIT
|
||||
|
||||
2. DedupService.decrement_ref(blob_hash)
|
||||
→ Decrement reference counter
|
||||
→ If counter reaches 0, delete the blob file
|
||||
```
|
||||
|
||||
If step 2 fails, an unreferenced blob may remain on disk (occupies space but is not a correctness issue). Future garbage collection can clean these up.
|
||||
|
||||
---
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **ACID transactions** — metadata operations are atomic, consistent, isolated, and durable
|
||||
2. **Content-addressable storage** — identical content is stored once, referenced by hash
|
||||
3. **Crash resilience** — atomic blob writes + PostgreSQL WAL ensure recovery
|
||||
4. **No partial writes** — temp file + rename pattern guarantees all-or-nothing
|
||||
5. **Referential integrity** — foreign keys prevent orphaned metadata
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- PostgreSQL connection pooling (`sqlx::PgPool`) amortizes connection overhead
|
||||
- Dedup hash computation is CPU-bound but avoids unnecessary disk writes for duplicate content
|
||||
- Blob fsync adds latency vs. buffered writes, but ensures durability for critical user data
|
||||
- Content cache (in-memory LRU) serves repeat reads without disk or DB access
|
||||
|
||||
@@ -462,7 +462,7 @@ src/
|
||||
│ └── delta_sync_handler.rs # NUEVO: Endpoints API
|
||||
│
|
||||
└── common/
|
||||
└── di.rs # Añadir: delta_sync_service a CoreServices
|
||||
└── di.rs # Añadir: delta_sync_service a AppState
|
||||
```
|
||||
|
||||
### Main service (delta_sync_service.rs)
|
||||
@@ -1133,7 +1133,7 @@ thiserror = "1.0" # Para errores tipados (probablemente ya existe)
|
||||
- [ ] Implement **generate_delta()**
|
||||
- [ ] Implement **apply_delta()**
|
||||
- [ ] Create handler and API endpoints
|
||||
- [ ] Integrate into DI (**CoreServices**)
|
||||
- [ ] Integrate into DI (**AppState**)
|
||||
- [ ] Add routes in `routes.rs`
|
||||
- [ ] Integrate with upload (automatic indexing)
|
||||
- [ ] Integrate with delete (signature cleanup)
|
||||
|
||||
+474
-477
@@ -1,477 +1,474 @@
|
||||
# 01 - Internal Architecture
|
||||
|
||||
OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers:
|
||||
|
||||
```
|
||||
Domain → Application → Infrastructure → Interfaces
|
||||
```
|
||||
|
||||
All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Injection Container
|
||||
|
||||
### AppServiceFactory
|
||||
|
||||
**File:** `src/common/di.rs`
|
||||
|
||||
```rust
|
||||
pub struct AppServiceFactory {
|
||||
storage_path: PathBuf,
|
||||
locales_path: PathBuf,
|
||||
config: AppConfig,
|
||||
}
|
||||
```
|
||||
|
||||
Initialization order in `build_app_state()`:
|
||||
|
||||
1. **Core services** -- path, caches, ID mapping, thumbnail, write-behind, chunked upload, transcode, dedup, compression
|
||||
2. **Repository services** -- folder repo (stub mediator first), then **FileSystemStorageMediator** (real), file repos, metadata cache, buffer pool
|
||||
3. **Trash service** (if **enable_trash** enabled)
|
||||
4. **Application services** -- folder, file upload/retrieval/management, search, i18n
|
||||
5. **Share service** (if **enable_file_sharing** enabled)
|
||||
6. **DB-dependent services** -- favorites, recent, storage usage, auth (via **auth_factory**)
|
||||
7. **Preload** translations + metadata cache
|
||||
8. **ZIP service** (needs file retrieval + folder service, wired last)
|
||||
9. **Assemble AppState** + admin settings + CalDAV/CardDAV
|
||||
|
||||
### AppState (Global State)
|
||||
|
||||
```rust
|
||||
pub struct AppState {
|
||||
pub core: CoreServices,
|
||||
pub repositories: RepositoryServices,
|
||||
pub applications: ApplicationServices,
|
||||
pub db_pool: Option<Arc<PgPool>>,
|
||||
pub auth_service: Option<AuthServices>,
|
||||
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||
pub storage_usage_service: Option<Arc<dyn StorageUsagePort>>,
|
||||
pub calendar_service: Option<Arc<dyn StorageUseCase>>,
|
||||
pub contact_service: Option<Arc<dyn StorageUseCase>>,
|
||||
pub calendar_use_case: Option<Arc<dyn CalendarUseCase>>,
|
||||
pub addressbook_use_case: Option<Arc<dyn AddressBookUseCase>>,
|
||||
pub contact_use_case: Option<Arc<dyn ContactUseCase>>,
|
||||
}
|
||||
```
|
||||
|
||||
Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`.
|
||||
|
||||
### Service Groups
|
||||
|
||||
```rust
|
||||
pub struct CoreServices {
|
||||
pub path_service: Arc<PathService>,
|
||||
pub file_content_cache: Arc<dyn ContentCachePort>,
|
||||
pub id_mapping_service: Arc<dyn IdMappingPort>, // folder IDs
|
||||
pub file_id_mapping_service: Arc<IdMappingService>, // file IDs (concrete)
|
||||
pub id_mapping_optimizer: Arc<IdMappingOptimizer>,
|
||||
pub thumbnail_service: Arc<dyn ThumbnailPort>,
|
||||
pub write_behind_cache: Arc<dyn WriteBehindCachePort>,
|
||||
pub chunked_upload_service: Arc<dyn ChunkedUploadPort>,
|
||||
pub image_transcode_service: Arc<dyn ImageTranscodePort>,
|
||||
pub dedup_service: Arc<dyn DedupPort>,
|
||||
pub compression_service: Arc<dyn CompressionPort>,
|
||||
pub zip_service: Arc<dyn ZipPort>,
|
||||
pub config: AppConfig,
|
||||
}
|
||||
|
||||
pub struct RepositoryServices {
|
||||
pub folder_repository: Arc<dyn FolderStoragePort>,
|
||||
pub file_read_repository: Arc<dyn FileReadPort>,
|
||||
pub file_write_repository: Arc<dyn FileWritePort>,
|
||||
pub i18n_repository: Arc<dyn I18nService>,
|
||||
pub storage_mediator: Arc<dyn StorageMediator>,
|
||||
pub metadata_cache: Arc<FileMetadataCache>,
|
||||
pub trash_repository: Option<Arc<dyn TrashRepository>>,
|
||||
}
|
||||
|
||||
pub struct ApplicationServices {
|
||||
pub folder_service_concrete: Arc<FolderService>,
|
||||
pub folder_service: Arc<dyn FolderUseCase>,
|
||||
pub file_upload_service: Arc<dyn FileUploadUseCase>,
|
||||
pub file_retrieval_service: Arc<dyn FileRetrievalUseCase>,
|
||||
pub file_management_service: Arc<dyn FileManagementUseCase>,
|
||||
pub file_use_case_factory: Arc<dyn FileUseCaseFactory>,
|
||||
pub i18n_service: Arc<I18nApplicationService>,
|
||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||
pub search_service: Option<Arc<dyn SearchUseCase>>,
|
||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||
}
|
||||
|
||||
pub struct AuthServices {
|
||||
pub token_service: Arc<dyn TokenServicePort>,
|
||||
pub auth_application_service: Arc<AuthApplicationService>,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ID Mapping System
|
||||
|
||||
Maps bidirectionally between **filesystem StoragePaths** and **UUID identifiers**. Two separate instances exist: one for folders (`folder_ids.json`), one for files (`file_ids.json`).
|
||||
|
||||
### StoragePath (Domain Value Object)
|
||||
|
||||
**File:** `src/domain/services/path_service.rs`
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Default)]
|
||||
pub struct StoragePath {
|
||||
segments: Vec<String>, // e.g., ["Mi Carpeta - admin", "file.txt"]
|
||||
}
|
||||
```
|
||||
|
||||
| Method | Description |
|
||||
|---|---|
|
||||
| `root()` | Empty path (storage root) |
|
||||
| `from_string(path)` | Parse from `/`-delimited string |
|
||||
| `join(segment)` | Append a segment |
|
||||
| `file_name()` | Last segment |
|
||||
| `parent()` | All segments except last |
|
||||
| `to_string()` | Join segments with `/` |
|
||||
|
||||
### IdMappingPort (Application Port)
|
||||
|
||||
**File:** `src/application/ports/outbound.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait IdMappingPort: Send + Sync + 'static {
|
||||
async fn get_or_create_id(&self, path: &StoragePath) -> Result<String, DomainError>;
|
||||
async fn get_path_by_id(&self, id: &str) -> Result<StoragePath, DomainError>;
|
||||
async fn update_path(&self, id: &str, new_path: &StoragePath) -> Result<(), DomainError>;
|
||||
async fn remove_id(&self, id: &str) -> Result<(), DomainError>;
|
||||
async fn save_changes(&self) -> Result<(), DomainError>;
|
||||
// Default impls for PathBuf variants:
|
||||
async fn get_file_path(&self, file_id: &str) -> Result<PathBuf, DomainError>;
|
||||
async fn update_file_path(&self, file_id: &str, new_path: &PathBuf) -> Result<(), DomainError>;
|
||||
}
|
||||
```
|
||||
|
||||
### IdMappingService (Base Implementation)
|
||||
|
||||
**File:** `src/infrastructure/services/id_mapping_service.rs`
|
||||
|
||||
```rust
|
||||
pub struct IdMappingService {
|
||||
map_path: PathBuf, // e.g., storage/file_ids.json
|
||||
id_map: RwLock<IdMap>,
|
||||
save_mutex: Mutex<()>,
|
||||
timeouts: TimeoutConfig,
|
||||
pending_save: RwLock<bool>,
|
||||
}
|
||||
|
||||
struct IdMap {
|
||||
path_to_id: HashMap<String, String>,
|
||||
id_to_path: HashMap<String, String>,
|
||||
version: u32,
|
||||
}
|
||||
```
|
||||
|
||||
Operations:
|
||||
- `get_or_create_id(path)` -- Read-lock first (cache hit). Write-lock on miss, generates `Uuid::new_v4()`.
|
||||
- `save_pending_changes()` -- Atomic write: serialize → write `.tmp` file → rename over original (with retry).
|
||||
- `new(map_path)` -- Loads from JSON, rebuilds inverse map if inconsistent.
|
||||
|
||||
Persistence format (`storage/file_ids.json`):
|
||||
```json
|
||||
{
|
||||
"path_to_id": { "/Mi Carpeta - admin/doc.pdf": "a1b2c3d4-..." },
|
||||
"id_to_path": { "a1b2c3d4-...": "/Mi Carpeta - admin/doc.pdf" },
|
||||
"version": 42
|
||||
}
|
||||
```
|
||||
|
||||
### IdMappingOptimizer (Cache Layer)
|
||||
|
||||
**File:** `src/infrastructure/services/id_mapping_optimizer.rs`
|
||||
|
||||
Wraps **IdMappingService** with an in-memory TTL cache:
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max cache entries | 10,000 |
|
||||
| TTL | 300 s (5 min) |
|
||||
| Cleanup interval | 150 s (2.5 min) |
|
||||
| Batch threshold | ≥ 20 queued items |
|
||||
| Max concurrent batches | 2 (semaphore) |
|
||||
|
||||
```rust
|
||||
pub struct IdMappingOptimizer {
|
||||
base_service: Arc<IdMappingService>,
|
||||
path_to_id_cache: RwLock<HashMap<String, (String, Instant)>>,
|
||||
id_to_path_cache: RwLock<HashMap<String, (String, Instant)>>,
|
||||
stats: RwLock<OptimizerStats>,
|
||||
batch_limiter: Semaphore,
|
||||
pending_batch: Mutex<BatchQueue>,
|
||||
}
|
||||
```
|
||||
|
||||
Lookup flow: check cache → if miss, queue request → trigger batch if ≥ 20 pending → fallback to **base_service** → update cache.
|
||||
|
||||
On `update_path` / `remove_id`, cache entries are invalidated first, then delegated.
|
||||
|
||||
Used only for folder ID mapping. File ID mapping uses the base **IdMappingService** directly.
|
||||
|
||||
---
|
||||
|
||||
## Path Service
|
||||
|
||||
**File:** `src/infrastructure/services/path_service.rs`
|
||||
|
||||
```rust
|
||||
pub struct PathService {
|
||||
root_path: PathBuf, // e.g., ./storage
|
||||
}
|
||||
```
|
||||
|
||||
### Path Resolution
|
||||
|
||||
| Method | Description |
|
||||
|---|---|
|
||||
| `resolve_path(storage_path)` | Appends **StoragePath** segments to **root_path** → absolute `PathBuf` |
|
||||
| `to_storage_path(physical_path)` | Strips **root_path** prefix → **StoragePath** (returns `None` if outside root) |
|
||||
| `create_file_path(folder, name)` | Combines folder path + filename |
|
||||
| `is_direct_child(parent, child)` | Check parent-child relationship |
|
||||
| `is_in_root(path)` | Verify path is within storage root |
|
||||
|
||||
### Path Validation
|
||||
|
||||
`validate_path(path)` rejects:
|
||||
- Empty path segments
|
||||
- Segments containing dangerous characters: `\`, `:`, `*`, `?`, `"`, `<`, `>`, `|`
|
||||
- Segments starting with `.` (exception: `.well-known` for WebDAV/CalDAV/CardDAV)
|
||||
|
||||
### Trait Implementations
|
||||
|
||||
- **StoragePort** -- `resolve_path()`, `ensure_directory()` (validates first, then `fs::create_dir_all`), `file_exists()`, `directory_exists()`
|
||||
- **StorageMediator** -- Simplified stub variant. Folder lookup methods return `NotFound`.
|
||||
|
||||
---
|
||||
|
||||
## Storage Mediator
|
||||
|
||||
Bridges folder IDs to filesystem paths by combining folder repository, path service, and ID mapping.
|
||||
|
||||
### StorageMediator Trait
|
||||
|
||||
**File:** `src/application/services/storage_mediator.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait StorageMediator: Send + Sync + 'static {
|
||||
async fn get_folder_path(&self, folder_id: &str) -> StorageMediatorResult<PathBuf>;
|
||||
async fn get_folder_storage_path(&self, folder_id: &str) -> StorageMediatorResult<StoragePath>;
|
||||
async fn get_folder(&self, folder_id: &str) -> StorageMediatorResult<Folder>;
|
||||
async fn file_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
|
||||
async fn file_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
|
||||
async fn folder_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
|
||||
async fn folder_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
|
||||
fn resolve_path(&self, relative_path: &Path) -> PathBuf;
|
||||
fn resolve_storage_path(&self, storage_path: &StoragePath) -> PathBuf;
|
||||
async fn ensure_directory(&self, path: &Path) -> StorageMediatorResult<()>;
|
||||
async fn ensure_storage_directory(&self, storage_path: &StoragePath) -> StorageMediatorResult<()>;
|
||||
}
|
||||
```
|
||||
|
||||
### FileSystemStorageMediator
|
||||
|
||||
```rust
|
||||
pub struct FileSystemStorageMediator {
|
||||
pub folder_storage_port: Arc<dyn FolderStoragePort>,
|
||||
pub path_service: Arc<dyn StoragePort>,
|
||||
pub id_mapping: Arc<dyn IdMappingPort>,
|
||||
}
|
||||
```
|
||||
|
||||
Folder ID → filesystem path resolution:
|
||||
|
||||
```
|
||||
folder_id ──► FolderStoragePort.get_folder(id)
|
||||
──► Folder entity
|
||||
──► IdMappingPort.get_path_by_id(folder.id())
|
||||
──► StoragePath
|
||||
──► StoragePort.resolve_path(storage_path)
|
||||
──► PathBuf (absolute)
|
||||
```
|
||||
|
||||
A **StubStorageMediator** also exists (returns `/tmp` paths) for DI bootstrap before the real mediator is available.
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
### Session Entity
|
||||
|
||||
**File:** `src/domain/entities/session.rs`
|
||||
|
||||
```rust
|
||||
pub struct Session {
|
||||
id: String, // UUID v4
|
||||
user_id: String,
|
||||
refresh_token: String,
|
||||
expires_at: DateTime<Utc>,
|
||||
ip_address: Option<String>,
|
||||
user_agent: Option<String>,
|
||||
created_at: DateTime<Utc>,
|
||||
revoked: bool,
|
||||
}
|
||||
```
|
||||
|
||||
Constructors:
|
||||
- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` -- generates UUID, panics if **user_id** or **refresh_token** empty
|
||||
- `Session::from_raw(...)` -- for DB reconstruction
|
||||
|
||||
### SessionRepository (Domain Port)
|
||||
|
||||
**File:** `src/domain/repositories/session_repository.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SessionRepository: Send + Sync + 'static {
|
||||
async fn create_session(&self, session: Session) -> SessionRepositoryResult<Session>;
|
||||
async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult<Session>;
|
||||
async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult<Session>;
|
||||
async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult<Vec<Session>>;
|
||||
async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>;
|
||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult<u64>;
|
||||
async fn delete_expired_sessions(&self) -> SessionRepositoryResult<u64>;
|
||||
}
|
||||
```
|
||||
|
||||
### SessionStoragePort (Application Port)
|
||||
|
||||
**File:** `src/application/ports/auth_ports.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SessionStoragePort: Send + Sync + 'static {
|
||||
async fn create_session(&self, session: Session) -> Result<Session, DomainError>;
|
||||
async fn get_session_by_refresh_token(&self, token: &str) -> Result<Session, DomainError>;
|
||||
async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>;
|
||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> Result<u64, DomainError>;
|
||||
}
|
||||
```
|
||||
|
||||
### SessionPgRepository (Infrastructure)
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct SessionPgRepository {
|
||||
pool: Arc<PgPool>,
|
||||
}
|
||||
```
|
||||
|
||||
Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction.
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS auth.sessions (
|
||||
id VARCHAR(36) PRIMARY KEY,
|
||||
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
||||
refresh_token TEXT NOT NULL UNIQUE,
|
||||
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||
ip_address TEXT,
|
||||
user_agent TEXT,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
revoked BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id);
|
||||
CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token);
|
||||
CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at);
|
||||
CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked)
|
||||
WHERE NOT revoked AND is_session_active(expires_at);
|
||||
```
|
||||
|
||||
### Auth Service
|
||||
|
||||
**File:** `src/application/services/auth_application_service.rs`
|
||||
|
||||
**AuthApplicationService** orchestrates authentication using:
|
||||
- **UserStoragePort** -- user CRUD
|
||||
- **SessionStoragePort** -- session lifecycle
|
||||
- **PasswordHasherPort** -- Argon2id hashing
|
||||
- **TokenServicePort** -- JWT generation/validation
|
||||
- `RwLock<OidcState>` -- hot-reloadable OIDC configuration
|
||||
- `Mutex<HashMap<String, PendingOidcFlow>>` -- in-flight OIDC login states
|
||||
|
||||
Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**.
|
||||
|
||||
---
|
||||
|
||||
## File Use Case Factory
|
||||
|
||||
**File:** `src/application/services/file_use_case_factory.rs`
|
||||
|
||||
```rust
|
||||
pub trait FileUseCaseFactory: Send + Sync + 'static {
|
||||
fn create_file_upload_use_case(&self) -> Arc<dyn FileUploadUseCase>;
|
||||
fn create_file_retrieval_use_case(&self) -> Arc<dyn FileRetrievalUseCase>;
|
||||
fn create_file_management_use_case(&self) -> Arc<dyn FileManagementUseCase>;
|
||||
}
|
||||
```
|
||||
|
||||
**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**. The main DI-wired services use `*_full()` constructors that inject write-behind cache, dedup, content cache, and transcode ports for full optimization.
|
||||
|
||||
### File Operation Port Hierarchy
|
||||
|
||||
| Port | Key Methods |
|
||||
|---|---|
|
||||
| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `WriteBehind` <256KB, `Buffered` 256KB-1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` |
|
||||
| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (write-behind → content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` |
|
||||
| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Interfaces Layer │
|
||||
│ Axum Router → API Routes + Middleware (Auth, Compress) │
|
||||
└─────────────────────┬───────────────────────────────────────┘
|
||||
│ Arc<AppState>
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │
|
||||
│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │
|
||||
│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │
|
||||
│ │ FileMgmt │ │ I18nService │ │ TrashService │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │
|
||||
│ │ Ports (traits) │ │ │
|
||||
└─────────┼────────────────┼─────────────────────┼────────────┘
|
||||
│ │ │
|
||||
┌─────────▼────────────────▼─────────────────────▼────────────┐
|
||||
│ Infrastructure Layer │
|
||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ FileFsRead/ │ │ IdMapping │ │ SessionPg │ │
|
||||
│ │ FileFsWrite │ │ + Optimizer │ │ UserPg │ │
|
||||
│ │ FolderFs │ │ PathService │ │ JwtTokenService │ │
|
||||
│ │ TrashFs │ │ StorageMed. │ │ Argon2Hasher │ │
|
||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ ContentCache │ │ Thumbnail │ │ WriteBehind │ │
|
||||
│ │ MetadataCache │ │ Transcode │ │ BufferPool │ │
|
||||
│ │ BufferPool │ │ Dedup │ │ Compression │ │
|
||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Domain Layer │
|
||||
│ Entities: File, Folder, Session, User, Calendar, Contact │
|
||||
│ Value Objects: StoragePath │
|
||||
│ Repository Traits: SessionRepository, ... │
|
||||
│ Domain Errors │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
# 01 - Internal Architecture
|
||||
|
||||
OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers:
|
||||
|
||||
```
|
||||
Domain → Application → Infrastructure → Interfaces
|
||||
```
|
||||
|
||||
All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup.
|
||||
|
||||
---
|
||||
|
||||
## Storage Model: 100% Blob Storage
|
||||
|
||||
OxiCloud uses a **100% blob storage model** where:
|
||||
|
||||
- **File metadata** (name, folder, size, user, timestamps, trash status) is stored in **PostgreSQL** (`storage.files` table).
|
||||
- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob`.
|
||||
- **Folder structure** is purely virtual — represented as rows in `storage.folders` (no filesystem directories per user).
|
||||
- **Trash** is a soft-delete flag (`is_trashed`, `trashed_at`) on files and folders, exposed via `storage.trash_items` VIEW.
|
||||
|
||||
There are no filesystem-based ID mappings, no `folder_ids.json`/`file_ids.json`, and no storage mediator.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Injection Container
|
||||
|
||||
### AppServiceFactory
|
||||
|
||||
**File:** `src/common/di.rs`
|
||||
|
||||
```rust
|
||||
pub struct AppServiceFactory {
|
||||
storage_path: PathBuf,
|
||||
locales_path: PathBuf,
|
||||
config: AppConfig,
|
||||
}
|
||||
```
|
||||
|
||||
Initialization order in `build_app_state()`:
|
||||
|
||||
1. **Core services** — path, content cache, thumbnail, chunked upload, transcode, dedup, compression
|
||||
2. **Repository services** — `FolderDbRepository`, `FileBlobReadRepository`, `FileBlobWriteRepository`, `TrashDbRepository` (all PgPool-backed)
|
||||
3. **Trash service** (if **enable_trash** enabled)
|
||||
4. **Application services** — folder, file upload/retrieval/management, search, i18n
|
||||
5. **Share service** (if **enable_file_sharing** enabled)
|
||||
6. **DB-dependent services** — favorites, recent, storage usage, auth (via **auth_factory**)
|
||||
7. **Preload** translations
|
||||
8. **ZIP service** (needs file retrieval + folder service, wired last)
|
||||
9. **Assemble AppState** + admin settings + CalDAV/CardDAV
|
||||
|
||||
### AppState (Global State)
|
||||
|
||||
```rust
|
||||
pub struct AppState {
|
||||
pub core: CoreServices,
|
||||
pub repositories: RepositoryServices,
|
||||
pub applications: ApplicationServices,
|
||||
pub db_pool: Option<Arc<PgPool>>,
|
||||
pub auth_service: Option<AuthServices>,
|
||||
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||
pub storage_usage_service: Option<Arc<dyn StorageUsagePort>>,
|
||||
pub calendar_service: Option<Arc<dyn StorageUseCase>>,
|
||||
pub contact_service: Option<Arc<dyn StorageUseCase>>,
|
||||
pub calendar_use_case: Option<Arc<dyn CalendarUseCase>>,
|
||||
pub addressbook_use_case: Option<Arc<dyn AddressBookUseCase>>,
|
||||
pub contact_use_case: Option<Arc<dyn ContactUseCase>>,
|
||||
}
|
||||
```
|
||||
|
||||
Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`.
|
||||
|
||||
### Service Groups
|
||||
|
||||
```rust
|
||||
pub struct CoreServices {
|
||||
pub path_service: Arc<PathService>,
|
||||
pub file_content_cache: Arc<dyn ContentCachePort>,
|
||||
pub thumbnail_service: Arc<dyn ThumbnailPort>,
|
||||
pub chunked_upload_service: Arc<dyn ChunkedUploadPort>,
|
||||
pub image_transcode_service: Arc<dyn ImageTranscodePort>,
|
||||
pub dedup_service: Arc<dyn DedupPort>,
|
||||
pub compression_service: Arc<dyn CompressionPort>,
|
||||
pub zip_service: Arc<dyn ZipPort>,
|
||||
pub config: AppConfig,
|
||||
}
|
||||
|
||||
pub struct RepositoryServices {
|
||||
pub folder_repository: Arc<dyn FolderStoragePort>,
|
||||
pub folder_repo_concrete: Arc<FolderDbRepository>,
|
||||
pub file_read_repository: Arc<dyn FileReadPort>,
|
||||
pub file_write_repository: Arc<dyn FileWritePort>,
|
||||
pub i18n_repository: Arc<dyn I18nService>,
|
||||
pub trash_repository: Option<Arc<dyn TrashRepository>>,
|
||||
}
|
||||
|
||||
pub struct ApplicationServices {
|
||||
pub folder_service_concrete: Arc<FolderService>,
|
||||
pub folder_service: Arc<dyn FolderUseCase>,
|
||||
pub file_upload_service: Arc<dyn FileUploadUseCase>,
|
||||
pub file_retrieval_service: Arc<dyn FileRetrievalUseCase>,
|
||||
pub file_management_service: Arc<dyn FileManagementUseCase>,
|
||||
pub file_use_case_factory: Arc<dyn FileUseCaseFactory>,
|
||||
pub i18n_service: Arc<I18nApplicationService>,
|
||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||
pub search_service: Option<Arc<dyn SearchUseCase>>,
|
||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||
}
|
||||
|
||||
pub struct AuthServices {
|
||||
pub token_service: Arc<dyn TokenServicePort>,
|
||||
pub auth_application_service: Arc<AuthApplicationService>,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema (Storage)
|
||||
|
||||
All file and folder metadata lives in the `storage` PostgreSQL schema:
|
||||
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS storage;
|
||||
|
||||
CREATE TABLE storage.folders (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name TEXT NOT NULL,
|
||||
parent_id UUID REFERENCES storage.folders(id) ON DELETE CASCADE,
|
||||
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
|
||||
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
trashed_at TIMESTAMPTZ,
|
||||
original_parent_id UUID,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE storage.files (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name TEXT NOT NULL,
|
||||
folder_id UUID NOT NULL REFERENCES storage.folders(id) ON DELETE CASCADE,
|
||||
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
|
||||
blob_hash TEXT NOT NULL,
|
||||
size BIGINT NOT NULL DEFAULT 0,
|
||||
mime_type TEXT NOT NULL DEFAULT 'application/octet-stream',
|
||||
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
trashed_at TIMESTAMPTZ,
|
||||
original_folder_id UUID,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE OR REPLACE VIEW storage.trash_items AS
|
||||
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
|
||||
user_id, size, mime_type, trashed_at, created_at
|
||||
FROM storage.files WHERE is_trashed = TRUE
|
||||
UNION ALL
|
||||
SELECT id, name, 'folder' AS item_type, parent_id,
|
||||
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
|
||||
FROM storage.folders WHERE is_trashed = TRUE;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repository Layer (Infrastructure)
|
||||
|
||||
All repositories use **PgPool** for metadata and **DedupService** for blob content.
|
||||
|
||||
### FolderDbRepository
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/folder_db_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct FolderDbRepository {
|
||||
pool: Option<Arc<PgPool>>,
|
||||
}
|
||||
```
|
||||
|
||||
Implements `FolderRepository`. Uses recursive CTEs for path building, unique constraints for name dedup within parent, and soft-delete flags for trash operations.
|
||||
|
||||
Key methods: `create_folder`, `get_folder`, `get_folder_by_path`, `list_folders`, `rename_folder`, `move_folder`, `delete_folder`, `move_to_trash`, `restore_from_trash`, `create_home_folder`, `get_folder_user_id`.
|
||||
|
||||
`new_stub()` creates a pool-less instance for `AppState::default()`.
|
||||
|
||||
### FileBlobReadRepository
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct FileBlobReadRepository {
|
||||
pool: Arc<PgPool>,
|
||||
dedup: Arc<dyn DedupPort>,
|
||||
folder_repo: Arc<FolderDbRepository>,
|
||||
}
|
||||
```
|
||||
|
||||
Implements `FileReadPort`. Reads metadata from `storage.files` and content from blob store via `dedup.read_blob()` / `read_blob_bytes()`.
|
||||
|
||||
Key methods: `get_file`, `list_files`, `get_file_content`, `get_file_stream`, `get_file_range_stream`, `get_file_mmap`, `get_file_path`, `get_parent_folder_id`.
|
||||
|
||||
### FileBlobWriteRepository
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/file_blob_write_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct FileBlobWriteRepository {
|
||||
pool: Arc<PgPool>,
|
||||
dedup: Arc<dyn DedupPort>,
|
||||
folder_repo: Arc<FolderDbRepository>,
|
||||
}
|
||||
```
|
||||
|
||||
Implements `FileWritePort`. Stores content via `dedup.store_bytes()` (returns hash), then INSERTs metadata into `storage.files`.
|
||||
|
||||
Key methods: `save_file`, `save_file_from_stream`, `move_file`, `rename_file`, `delete_file`, `update_file_content`, `move_to_trash`, `restore_from_trash`, `delete_file_permanently`.
|
||||
|
||||
### TrashDbRepository
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct TrashDbRepository {
|
||||
pool: Arc<PgPool>,
|
||||
retention_days: u32,
|
||||
}
|
||||
```
|
||||
|
||||
Implements `TrashRepository`. Reads from `storage.trash_items` VIEW. `clear_trash` DELETEs rows where `is_trashed = TRUE`. `get_expired_items` checks `trashed_at` against the configured retention period.
|
||||
|
||||
---
|
||||
|
||||
## Path Service
|
||||
|
||||
**File:** `src/infrastructure/services/path_service.rs`
|
||||
|
||||
```rust
|
||||
pub struct PathService {
|
||||
root_path: PathBuf, // e.g., ./storage
|
||||
}
|
||||
```
|
||||
|
||||
Used for resolving storage root paths (blob storage directory, thumbnail paths, etc.). Not used for per-user folder resolution — that is handled by `FolderDbRepository` via PostgreSQL.
|
||||
|
||||
### StoragePath (Domain Value Object)
|
||||
|
||||
**File:** `src/domain/services/path_service.rs`
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Default)]
|
||||
pub struct StoragePath {
|
||||
segments: Vec<String>,
|
||||
}
|
||||
```
|
||||
|
||||
| Method | Description |
|
||||
|---|---|
|
||||
| `root()` | Empty path (storage root) |
|
||||
| `from_string(path)` | Parse from `/`-delimited string |
|
||||
| `join(segment)` | Append a segment |
|
||||
| `file_name()` | Last segment |
|
||||
| `parent()` | All segments except last |
|
||||
| `to_string()` | Join segments with `/` |
|
||||
|
||||
### Trait Implementations
|
||||
|
||||
- **StoragePort** — `resolve_path()`, `ensure_directory()`, `file_exists()`, `directory_exists()`
|
||||
|
||||
---
|
||||
|
||||
## Session Management
|
||||
|
||||
### Session Entity
|
||||
|
||||
**File:** `src/domain/entities/session.rs`
|
||||
|
||||
```rust
|
||||
pub struct Session {
|
||||
id: String, // UUID v4
|
||||
user_id: String,
|
||||
refresh_token: String,
|
||||
expires_at: DateTime<Utc>,
|
||||
ip_address: Option<String>,
|
||||
user_agent: Option<String>,
|
||||
created_at: DateTime<Utc>,
|
||||
revoked: bool,
|
||||
}
|
||||
```
|
||||
|
||||
Constructors:
|
||||
- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` — generates UUID, panics if **user_id** or **refresh_token** empty
|
||||
- `Session::from_raw(...)` — for DB reconstruction
|
||||
|
||||
### SessionRepository (Domain Port)
|
||||
|
||||
**File:** `src/domain/repositories/session_repository.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SessionRepository: Send + Sync + 'static {
|
||||
async fn create_session(&self, session: Session) -> SessionRepositoryResult<Session>;
|
||||
async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult<Session>;
|
||||
async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult<Session>;
|
||||
async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult<Vec<Session>>;
|
||||
async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>;
|
||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult<u64>;
|
||||
async fn delete_expired_sessions(&self) -> SessionRepositoryResult<u64>;
|
||||
}
|
||||
```
|
||||
|
||||
### SessionStoragePort (Application Port)
|
||||
|
||||
**File:** `src/application/ports/auth_ports.rs`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SessionStoragePort: Send + Sync + 'static {
|
||||
async fn create_session(&self, session: Session) -> Result<Session, DomainError>;
|
||||
async fn get_session_by_refresh_token(&self, token: &str) -> Result<Session, DomainError>;
|
||||
async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>;
|
||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> Result<u64, DomainError>;
|
||||
}
|
||||
```
|
||||
|
||||
### SessionPgRepository (Infrastructure)
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct SessionPgRepository {
|
||||
pool: Arc<PgPool>,
|
||||
}
|
||||
```
|
||||
|
||||
Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction.
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS auth.sessions (
|
||||
id VARCHAR(36) PRIMARY KEY,
|
||||
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
||||
refresh_token TEXT NOT NULL UNIQUE,
|
||||
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||
ip_address TEXT,
|
||||
user_agent TEXT,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
revoked BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id);
|
||||
CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token);
|
||||
CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at);
|
||||
CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked)
|
||||
WHERE NOT revoked AND is_session_active(expires_at);
|
||||
```
|
||||
|
||||
### Auth Service
|
||||
|
||||
**File:** `src/application/services/auth_application_service.rs`
|
||||
|
||||
**AuthApplicationService** orchestrates authentication using:
|
||||
- **UserStoragePort** — user CRUD
|
||||
- **SessionStoragePort** — session lifecycle
|
||||
- **PasswordHasherPort** — Argon2id hashing
|
||||
- **TokenServicePort** — JWT generation/validation
|
||||
- `RwLock<OidcState>` — hot-reloadable OIDC configuration
|
||||
- `Mutex<HashMap<String, PendingOidcFlow>>` — in-flight OIDC login states
|
||||
|
||||
Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**.
|
||||
|
||||
---
|
||||
|
||||
## File Use Case Factory
|
||||
|
||||
**File:** `src/application/services/file_use_case_factory.rs`
|
||||
|
||||
```rust
|
||||
pub trait FileUseCaseFactory: Send + Sync + 'static {
|
||||
fn create_file_upload_use_case(&self) -> Arc<dyn FileUploadUseCase>;
|
||||
fn create_file_retrieval_use_case(&self) -> Arc<dyn FileRetrievalUseCase>;
|
||||
fn create_file_management_use_case(&self) -> Arc<dyn FileManagementUseCase>;
|
||||
}
|
||||
```
|
||||
|
||||
**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**.
|
||||
|
||||
### File Operation Port Hierarchy
|
||||
|
||||
| Port | Key Methods |
|
||||
|---|---|
|
||||
| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `Buffered` <1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` |
|
||||
| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` |
|
||||
| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Interfaces Layer │
|
||||
│ Axum Router → API Routes + Middleware (Auth, Compress) │
|
||||
└─────────────────────┬───────────────────────────────────────┘
|
||||
│ Arc<AppState>
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │
|
||||
│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │
|
||||
│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │
|
||||
│ │ FileMgmt │ │ I18nService │ │ TrashService │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │
|
||||
│ │ Ports (traits) │ │ │
|
||||
└─────────┼────────────────┼─────────────────────┼────────────┘
|
||||
│ │ │
|
||||
┌─────────▼────────────────▼─────────────────────▼────────────┐
|
||||
│ Infrastructure Layer │
|
||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ FileBlobRead │ │ PathService │ │ SessionPg │ │
|
||||
│ │ FileBlobWrite │ │ DedupService │ │ UserPg │ │
|
||||
│ │ FolderDb │ │ Thumbnail │ │ JwtTokenService │ │
|
||||
│ │ TrashDb │ │ Transcode │ │ Argon2Hasher │ │
|
||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ ContentCache │ │ Compression │ │ ChunkedUpload │ │
|
||||
│ │ BufferPool │ │ ZipService │ │ ShareFsRepo │ │
|
||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────▼───────────────────────────────────────┐
|
||||
│ Domain Layer │
|
||||
│ Entities: File, Folder, Session, User, Calendar, Contact │
|
||||
│ Value Objects: StoragePath │
|
||||
│ Repository Traits: FolderRepository, TrashRepository, ... │
|
||||
│ Domain Errors │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Data Flow: File Upload
|
||||
|
||||
```
|
||||
HTTP Request (multipart)
|
||||
→ FileUploadService.smart_upload()
|
||||
→ FileBlobWriteRepository.save_file() / save_file_from_stream()
|
||||
→ DedupService.store_bytes() → .blobs/{prefix}/{hash}.blob
|
||||
→ INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
|
||||
→ 201 Created (FileDto)
|
||||
```
|
||||
|
||||
### Data Flow: File Download
|
||||
|
||||
```
|
||||
HTTP Request (GET /api/files/{id}/download)
|
||||
→ FileRetrievalService.get_file_optimized()
|
||||
→ ContentCache hit? → serve from RAM
|
||||
→ FileBlobReadRepository.get_file_content() / get_file_stream()
|
||||
→ SELECT blob_hash FROM storage.files WHERE id = $1
|
||||
→ DedupService.read_blob(hash) → bytes from .blobs/
|
||||
→ Optional WebP transcode → response
|
||||
```
|
||||
|
||||
### Data Flow: Folder Operations
|
||||
|
||||
```
|
||||
HTTP Request (POST /api/folders)
|
||||
→ FolderService.create_folder()
|
||||
→ FolderDbRepository.create_folder()
|
||||
→ INSERT INTO storage.folders (name, parent_id, user_id, ...)
|
||||
→ 201 Created (FolderDto)
|
||||
```
|
||||
|
||||
+31
-24
@@ -134,7 +134,7 @@ Handles: shared element validation, permission management, unique link/token gen
|
||||
|
||||
## Infrastructure
|
||||
|
||||
**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share links to the filesystem:
|
||||
**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share link metadata to a local JSON file:
|
||||
|
||||
```rust
|
||||
pub struct ShareFsRepository {
|
||||
@@ -157,7 +157,9 @@ struct ShareRecord {
|
||||
}
|
||||
```
|
||||
|
||||
Stores shared links in a JSON file. Supports queries and updates, search by ID/token/user, and pagination.
|
||||
Stores share link records in a JSON file. Supports queries and updates, search by ID/token/user, and pagination.
|
||||
|
||||
> **Note:** This repository stores *share link metadata* only (tokens, permissions, expiration). The actual file/folder content is accessed via `FileReadPort` / `FolderStoragePort` which use the blob storage model (PostgreSQL metadata + DedupService blobs).
|
||||
|
||||
## API Handlers and Routes
|
||||
|
||||
@@ -256,26 +258,30 @@ The service is instantiated via **AppServiceFactory** in `src/common/di.rs` and
|
||||
|
||||
```rust
|
||||
// In AppServiceFactory::create_share_service()
|
||||
let share_service: Option<Arc<dyn ShareUseCase>> = if config.features.enable_file_sharing {
|
||||
let share_repository = Arc::new(ShareFsRepository::new(Arc::new(config.clone())));
|
||||
let share_service = Arc::new(ShareService::new(
|
||||
Arc::new(config.clone()),
|
||||
share_repository,
|
||||
file_read_repository.clone(),
|
||||
folder_repository.clone(),
|
||||
password_hasher.clone(),
|
||||
));
|
||||
Some(share_service)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
pub fn create_share_service(&self, repos: &RepositoryServices)
|
||||
-> Option<Arc<dyn ShareUseCase>>
|
||||
{
|
||||
if !self.config.features.enable_file_sharing {
|
||||
return None;
|
||||
}
|
||||
|
||||
// Add to AppState
|
||||
let app_state = AppState {
|
||||
// ...
|
||||
share_service: share_service.clone(),
|
||||
// ...
|
||||
};
|
||||
let share_repository = Arc::new(ShareFsRepository::new(
|
||||
Arc::new(self.config.clone())
|
||||
));
|
||||
|
||||
let password_hasher: Arc<dyn PasswordHasherPort> =
|
||||
Arc::new(Argon2PasswordHasher::new());
|
||||
|
||||
let service = Arc::new(ShareService::new(
|
||||
Arc::new(self.config.clone()),
|
||||
share_repository,
|
||||
repos.file_read_repository.clone(), // FileBlobReadRepository
|
||||
repos.folder_repository.clone(), // FolderDbRepository
|
||||
password_hasher,
|
||||
));
|
||||
|
||||
Some(service)
|
||||
}
|
||||
```
|
||||
|
||||
## Workflows
|
||||
@@ -353,8 +359,9 @@ HTTP status code mapping:
|
||||
|
||||
## Technical Notes
|
||||
|
||||
- **Performance**: JSON file-based storage works for moderate volumes. For higher load, migrate to a database.
|
||||
- **Scalability**: the design supports horizontal scaling via distributed or cloud-based repositories.
|
||||
- **Share metadata** is stored in a local JSON file via `ShareFsRepository`. This is separate from the 100% blob storage model used for file content.
|
||||
- **File/folder lookups** during share access go through `FileReadPort` / `FolderStoragePort`, which read metadata from PostgreSQL and content from the DedupService blob store.
|
||||
- **Scalability**: for higher load, share metadata could be migrated to PostgreSQL using the same hexagonal architecture (implement `ShareRepository` with PgPool).
|
||||
- **Maintenance**: clear separation of concerns makes testing and maintenance straightforward.
|
||||
|
||||
The sharing feature is enabled by default in the current configuration.
|
||||
The sharing feature is enabled via `OXICLOUD_ENABLE_FILE_SHARING` configuration flag.
|
||||
|
||||
+117
-79
@@ -1,79 +1,117 @@
|
||||
# 14 - Trash Feature
|
||||
|
||||
Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup.
|
||||
|
||||
## Architecture
|
||||
|
||||
Follows the hexagonal architecture:
|
||||
|
||||
1. **Domain Layer** (`/src/domain/`):
|
||||
- Entities: **TrashedItem** representing files and folders in the trash
|
||||
- Repository interfaces: **TrashRepository** defining trash management operations
|
||||
|
||||
2. **Application Layer** (`/src/application/`):
|
||||
- DTOs: **TrashedItemDto** for data transfer between layers
|
||||
- Ports: **TrashUseCase** defining available operations
|
||||
- Services: **TrashService** implementing the trash use cases
|
||||
|
||||
3. **Infrastructure Layer** (`/src/infrastructure/`):
|
||||
- Repositories: **TrashFsRepository** for filesystem-based trash storage
|
||||
- Trash-related methods in existing repositories: `FileWriteRepository::move_to_trash()`, `FolderRepository::move_to_trash()`, etc.
|
||||
- Services: **TrashCleanupService** for automatic cleanup of expired items
|
||||
|
||||
4. **Interface Layer** (`/src/interfaces/`):
|
||||
- API handlers: `trash_handler.rs` with HTTP endpoints for trash operations
|
||||
- Routes: updated `routes.rs` to include trash endpoints
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Soft Deletion** -- files and folders move to trash, not immediately deleted
|
||||
2. **Per-User Trash** -- each user has an isolated trash bin
|
||||
3. **Retention Policy** -- items auto-delete after a configurable period
|
||||
4. **Restoration** -- items can be restored to their original location
|
||||
5. **Permanent Deletion** -- items can be permanently deleted before retention expires
|
||||
6. **Empty Trash** -- wipe everything in the trash at once
|
||||
|
||||
## API Endpoints
|
||||
|
||||
- `GET /api/trash` or `GET /api/trash/` -- list all items in the user's trash
|
||||
- `DELETE /api/trash/files/:id` -- move a file to trash
|
||||
- `DELETE /api/trash/folders/:id` -- move a folder to trash
|
||||
- `POST /api/trash/:id/restore` -- restore an item to its original location
|
||||
- `DELETE /api/trash/:id` -- permanently delete an item from trash
|
||||
- `DELETE /api/trash/empty` -- empty the entire trash bin
|
||||
|
||||
## Testing
|
||||
|
||||
1. **Unit Tests** -- testing **TrashService**:
|
||||
- Move files and folders to trash
|
||||
- Restore items from trash
|
||||
- Permanent deletion
|
||||
- Empty trash operation
|
||||
|
||||
2. **Integration Tests** -- Python script hitting the API endpoints:
|
||||
- End-to-end testing of all trash operations
|
||||
- Verification of move, list, restore, and delete behavior
|
||||
|
||||
3. **Shell Script** -- for manual testing and demonstration
|
||||
|
||||
## Configuration
|
||||
|
||||
- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true)
|
||||
- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)
|
||||
|
||||
## Implementation Details
|
||||
|
||||
1. **Physical File Storage** -- when items are trashed, they physically move to a `.trash` directory
|
||||
2. **Metadata Storage** -- trashed item info stored in a separate database table or file
|
||||
3. **User Isolation** -- trash items are isolated by user ID
|
||||
4. **Automatic Cleanup** -- a background job runs periodically to clean up expired items
|
||||
5. **Transaction Safety** -- operations are atomic with proper error handling
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Trash Quotas** -- limit trash storage per user
|
||||
2. **Batch Operations** -- trash, restore, or delete multiple items at once
|
||||
3. **Storage Optimization** -- deduplication for trashed items
|
||||
4. **Version Control** -- track file versions when moving to trash
|
||||
5. **Scheduled Cleanup** -- let users configure custom retention periods
|
||||
6. **Trash Monitoring** -- metrics and alerts for trash usage and cleanup
|
||||
# 14 - Trash Feature
|
||||
|
||||
Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup.
|
||||
|
||||
## Architecture
|
||||
|
||||
Follows the hexagonal architecture:
|
||||
|
||||
1. **Domain Layer** (`/src/domain/`):
|
||||
- Entities: **TrashedItem** representing files and folders in the trash
|
||||
- Repository interfaces: **TrashRepository** defining trash management operations
|
||||
|
||||
2. **Application Layer** (`/src/application/`):
|
||||
- DTOs: **TrashedItemDto** for data transfer between layers
|
||||
- Ports: **TrashUseCase** defining available operations
|
||||
- Services: **TrashService** implementing the trash use cases
|
||||
|
||||
3. **Infrastructure Layer** (`/src/infrastructure/`):
|
||||
- Repositories: **TrashDbRepository** (PostgreSQL) — reads from `storage.trash_items` VIEW, manages soft-delete flags
|
||||
- Trash-related methods in file/folder repositories: `FileBlobWriteRepository::move_to_trash()`, `FolderDbRepository::move_to_trash()`, etc.
|
||||
- Services: **TrashCleanupService** for automatic cleanup of expired items
|
||||
|
||||
4. **Interface Layer** (`/src/interfaces/`):
|
||||
- API handlers: `trash_handler.rs` with HTTP endpoints for trash operations
|
||||
- Routes: updated `routes.rs` to include trash endpoints
|
||||
|
||||
## Storage Model
|
||||
|
||||
Trash uses a **soft-delete** model in PostgreSQL:
|
||||
|
||||
- Files and folders have `is_trashed` (BOOLEAN) and `trashed_at` (TIMESTAMPTZ) columns
|
||||
- When an item is trashed, `is_trashed` is set to `TRUE` and `trashed_at` records the timestamp
|
||||
- `original_parent_id` / `original_folder_id` stores the original location for restore
|
||||
- The `storage.trash_items` VIEW provides a unified list of all trashed items (files + folders)
|
||||
- No physical file movement occurs — blob content stays at `.blobs/{prefix}/{hash}.blob`
|
||||
- Permanent deletion removes the DB row and decrements the blob reference counter
|
||||
|
||||
```sql
|
||||
CREATE OR REPLACE VIEW storage.trash_items AS
|
||||
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
|
||||
user_id, size, mime_type, trashed_at, created_at
|
||||
FROM storage.files WHERE is_trashed = TRUE
|
||||
UNION ALL
|
||||
SELECT id, name, 'folder' AS item_type, parent_id,
|
||||
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
|
||||
FROM storage.folders WHERE is_trashed = TRUE;
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Soft Deletion** — files and folders are flagged as trashed, not immediately deleted
|
||||
2. **Per-User Trash** — each user has an isolated trash bin (filtered by `user_id`)
|
||||
3. **Retention Policy** — items auto-delete after a configurable period
|
||||
4. **Restoration** — items can be restored to their original location via `original_parent_id`/`original_folder_id`
|
||||
5. **Permanent Deletion** — items can be permanently deleted before retention expires (removes DB row + decrements blob ref)
|
||||
6. **Empty Trash** — wipe everything in the trash at once
|
||||
|
||||
## API Endpoints
|
||||
|
||||
- `GET /api/trash` or `GET /api/trash/` — list all items in the user's trash
|
||||
- `DELETE /api/trash/files/:id` — move a file to trash
|
||||
- `DELETE /api/trash/folders/:id` — move a folder to trash
|
||||
- `POST /api/trash/:id/restore` — restore an item to its original location
|
||||
- `DELETE /api/trash/:id` — permanently delete an item from trash
|
||||
- `DELETE /api/trash/empty` — empty the entire trash bin
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### TrashDbRepository
|
||||
|
||||
**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
|
||||
|
||||
```rust
|
||||
pub struct TrashDbRepository {
|
||||
pool: Arc<PgPool>,
|
||||
retention_days: u32,
|
||||
}
|
||||
```
|
||||
|
||||
Key methods:
|
||||
- `get_trash_items(user_id)` — SELECT from `storage.trash_items` WHERE `user_id = $1`
|
||||
- `clear_trash(user_id)` — DELETE from `storage.files` and `storage.folders` WHERE `is_trashed = TRUE AND user_id = $1`
|
||||
- `get_expired_items()` — finds items where `trashed_at + retention_days < NOW()`
|
||||
|
||||
### TrashService
|
||||
|
||||
**File:** `src/application/services/trash_service.rs`
|
||||
|
||||
Constructor: `TrashService::new(trash_repo, file_read, file_write, folder_repo, retention_days)`
|
||||
|
||||
Orchestrates trash operations by delegating to the appropriate repository:
|
||||
- Moving a file to trash → `FileBlobWriteRepository::move_to_trash()`
|
||||
- Moving a folder to trash → `FolderDbRepository::move_to_trash()`
|
||||
- Permanent deletion → removes DB row + calls `DedupService::decrement_ref()` to clean blob if unreferenced
|
||||
|
||||
### TrashCleanupService
|
||||
|
||||
**File:** `src/infrastructure/services/trash_cleanup_service.rs`
|
||||
|
||||
Background job that runs every 24 hours to permanently delete items past the retention period.
|
||||
|
||||
## Testing
|
||||
|
||||
1. **Unit Tests** — testing **TrashService**:
|
||||
- Move files and folders to trash
|
||||
- Restore items from trash
|
||||
- Permanent deletion
|
||||
- Empty trash operation
|
||||
|
||||
2. **Integration Tests** — Python script hitting the API endpoints:
|
||||
- End-to-end testing of all trash operations
|
||||
- Verification of move, list, restore, and delete behavior
|
||||
|
||||
## Configuration
|
||||
|
||||
- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true)
|
||||
- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)
|
||||
|
||||
Reference in New Issue
Block a user