cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

docs: update documentation to reflect 100% blob storage model

Browse Source 
Rewrite documentation to match the new architecture where all
file metadata lives in PostgreSQL and content is stored as
content-addressed blobs via DedupService.

Updated files:
- internal-architecture.md: complete rewrite — new DB schema,
  blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated
  DI container, service groups, architecture diagram, data flows
- file-system-safety.md: repurposed as storage-safety.md —
  covers PostgreSQL ACID guarantees + DedupService atomic writes
- caching-architecture.md: updated repo references to blob repos,
  removed write-behind cache section, updated upload/download flows
- trash-feature-summary.md: rewritten for soft-delete model
  (is_trashed flag, trash_items VIEW, TrashDbRepository)
- share-integration.md: clarified ShareFsRepository scope,
  updated DI snippet, added blob storage context note
- deduplication.md: updated DI snippet (dedup injected into repos)
- deployment.md: updated feature matrix (file storage requires DB)
- important-delta-sync-implementation.md: updated DI references

Removed legacy references: IdMappingPort, StorageMediator,
WriteBehindCache, FsFileRepository, FsFolderRepository,
TrashFsRepository, folder_ids.json, file_ids.json.
This commit is contained in:
Dionisio
2026-02-14 18:27:30 +01:00
parent 5d2bc36d74
commit e071841ec2
8 changed files with 1033 additions and 1115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/caching-architecture.md
+44 -140
View File
@@ -10,18 +10,16 @@ OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to k
├─────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────┤
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads │ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
├─────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────┤
│ Layer 2: MMAP (memmap2, 10-100MB files) │ Downloads │ Layer 2: MMAP (memmap2, 10-100MB blobs) │ Downloads
├─────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────┤
│ Layer 3: Streaming (FramedRead, ≥100MB files) │ Downloads │ Layer 3: Streaming (FramedRead, ≥100MB blobs) │ Downloads
├─────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────┤
│ Layer 4: File Metadata Cache (adaptive TTL) │ All file ops │ Layer 4: Buffer Pool (reusable I/O buffers) │ Compression
├─────────────────────────────────────────────────────┤
│ Layer 5: Write-Behind Cache (<256KB uploads) │ Uploads
├─────────────────────────────────────────────────────┤
│ Layer 6: Buffer Pool (reusable I/O buffers) │ Compression
└─────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────┘
``` ```
> **Note:** File metadata (name, size, MIME type, folder) is served from PostgreSQL — no separate filesystem metadata cache is needed.
--- ---
## Layer 0: HTTP Cache Middleware ## Layer 0: HTTP Cache Middleware
@@ -62,9 +60,9 @@ In-memory LRU cache for small files, served directly from RAM.
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }` **CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
Methods: Methods:
- `should_cache(size)` -- checks if file fits in cache - `should_cache(size)` — checks if file fits in cache
- `get(file_id)` → `Option<(Bytes, String, String)>` -- returns (data, etag, content_type) - `get(file_id)` → `Option<(Bytes, String, String)>` — returns (data, etag, content_type)
- `put(file_id, content, etag, content_type)` -- inserts with LRU eviction - `put(file_id, content, etag, content_type)` — inserts with LRU eviction
- `invalidate(file_id)`, `clear()` - `invalidate(file_id)`, `clear()`
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }` - `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
@@ -74,9 +72,9 @@ Port: implements **ContentCachePort** trait.
## Layer 2: MMAP (Download Tier 2) ## Layer 2: MMAP (Download Tier 2)
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs` **File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
Memory-mapped I/O for medium files using `memmap2`. Memory-mapped I/O for medium blobs using `memmap2`.
| Parameter | Value | | Parameter | Value |
|---|---| |---|---|
@@ -84,15 +82,15 @@ Memory-mapped I/O for medium files using `memmap2`.
| Implementation | `memmap2::Mmap` via `spawn_blocking` | | Implementation | `memmap2::Mmap` via `spawn_blocking` |
| Latency | ~1-5ms | | Latency | ~1-5ms |
Current implementation copies mmap'd data to `Bytes` (`Bytes::copy_from_slice(&mmap[..])`). Not true zero-copy, but still benefits from kernel page cache. The blob file (`.blobs/{prefix}/{hash}.blob`) is memory-mapped and its contents copied to `Bytes`. Benefits from kernel page cache for frequently accessed blobs.
--- ---
## Layer 3: Streaming (Download Tier 3) ## Layer 3: Streaming (Download Tier 3)
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs` **File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
Chunked streaming for large files using tokio-util codecs. Chunked streaming for large blobs using tokio-util codecs.
| Parameter | Value | | Parameter | Value |
|---|---| |---|---|
@@ -103,80 +101,7 @@ Chunked streaming for large files using tokio-util codecs.
--- ---
## Layer 4: File Metadata Cache ## Layer 4: Buffer Pool
**File**: `src/infrastructure/services/file_metadata_cache.rs`
Caches filesystem metadata (existence, size, MIME type, timestamps) to avoid repeated `stat()` calls.
| Parameter | Value |
|---|---|
| Default file TTL | 60 seconds |
| Default directory TTL | 120 seconds |
| Max entries | 10,000 |
| Adaptive TTL multiplier | 5x for popular entries (≥10 accesses) |
| LRU eviction | Frees 10% capacity when full |
| Cleanup | Background task runs periodically |
**CachedMetadata:**
```rust
pub struct FileMetadata {
pub path: PathBuf,
pub exists: bool,
pub entry_type: CacheEntryType, // File | Directory | Unknown
pub size: Option<u64>,
pub mime_type: Option<String>,
pub created_at: Option<u64>,
pub modified_at: Option<u64>,
pub last_access: Instant,
pub expires_at: Instant,
pub access_count: usize,
}
```
**Adaptive TTL**: entries accessed ≥10 times get 5x the configured TTL, keeping frequently accessed file metadata in cache longer.
Port: implements **MetadataCachePort** trait.
---
## Layer 5: Write-Behind Cache
**File**: `src/infrastructure/services/write_behind_cache.rs`
Buffers small uploads in RAM and confirms immediately. Flushes to disk asynchronously.
| Parameter | Value |
|---|---|
| Max file size | 1 MB per file |
| Max total cache | 100 MB |
| Max pending duration | 30 seconds |
| Flush interval | 100 ms |
| Write strategy | Atomic (temp file + rename) |
Architecture:
- `put_pending(file_id, content, target_path)` stores bytes in `HashMap<String, PendingWrite>`
- Background `flush_worker` processes **FlushCommands** via `mpsc` channel
- Periodic checker force-flushes entries older than 30 seconds
- `get_pending(file_id)` serves reads while data is still in RAM (before flush)
Port: implements **WriteBehindCachePort** trait.
Statistics:
```rust
pub struct WriteBehindStatsDto {
pub pending_count: usize,
pub pending_bytes: usize,
pub total_writes: u64,
pub total_bytes_written: u64,
pub cache_hits: u64,
pub avg_flush_time_us: u64,
}
```
---
## Layer 6: Buffer Pool
**File**: `src/infrastructure/services/buffer_pool.rs` **File**: `src/infrastructure/services/buffer_pool.rs`
@@ -190,7 +115,7 @@ Reusable byte buffer pool to reduce allocation pressure during compression opera
| Concurrency control | `tokio::sync::Semaphore` | | Concurrency control | `tokio::sync::Semaphore` |
Features: Features:
- `get_buffer()` -- borrows a buffer (blocks if pool exhausted) - `get_buffer()` — borrows a buffer (blocks if pool exhausted)
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn` - **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
- Expired buffers are cleaned periodically via `start_cleaner()` - Expired buffers are cleaned periodically via `start_cleaner()`
- Tracks stats: gets, hits, misses, returns, evictions, waits - Tracks stats: gets, hits, misses, returns, evictions, waits
@@ -202,12 +127,6 @@ Features:
All cache-related config in `src/common/config.rs`: All cache-related config in `src/common/config.rs`:
```rust ```rust
pub struct CacheConfig {
pub file_ttl_ms: u64, // default: 60,000 (1 min)
pub directory_ttl_ms: u64, // default: 120,000 (2 min)
pub max_entries: usize, // default: 10,000
}
pub struct ResourceConfig { pub struct ResourceConfig {
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary) pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size) pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
@@ -220,15 +139,17 @@ pub struct ResourceConfig {
``` ```
Request → ETag check (304?) → Range request (206?) Request → ETag check (304?) → Range request (206?)
→ file size < 10MB? → Tier 1: LRU cache (RAM) → file size < 10MB? → Tier 1: LRU cache (RAM)
→ file size < 100MB? → Tier 2: MMAP (kernel page cache) → file size < 100MB? → Tier 2: MMAP (kernel page cache on blob file)
→ file size ≥ 100MB → Tier 3: Streaming (chunked) → file size ≥ 100MB → Tier 3: Streaming (chunked from blob file)
``` ```
In all tiers, metadata (file name, size, MIME type) comes from a PostgreSQL `SELECT` on `storage.files`. Content is read from the DedupService blob at `.blobs/{prefix}/{hash}.blob`.
--- ---
## Range Requests (HTTP 206 Partial Content) ## Range Requests (HTTP 206 Partial Content)
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/file_fs_read_repository.rs` **Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
**Crate**: `http-range-header = "0.4"` for parsing. **Crate**: `http-range-header = "0.4"` for parsing.
@@ -265,20 +186,15 @@ HTTP/1.1 416 Range Not Satisfiable
Content-Range: bytes */12345 Content-Range: bytes */12345
``` ```
### File Seek Implementation ### Blob File Seek Implementation
`get_file_range_stream()` at the repository level: `get_file_range_stream()` at the repository level:
```rust 1. Resolves blob path from `blob_hash` via DedupService
async fn get_file_range_stream( 2. Opens the blob file with `TokioFile::open()`
&self, id: &str, start: u64, end: Option<u64>, 3. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
) -> Result<Box<dyn Stream<...> + Send>, DomainError> 4. Limits read to `range_length` via `fh.take(range_length)`
``` 5. Wraps in `FramedRead` + `BytesCodec`
1. Opens the file with `TokioFile::open()`
2. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
3. Limits read to `range_length` via `fh.take(range_length)`
4. Wraps in `FramedRead` + `BytesCodec`
Adaptive chunk size: Adaptive chunk size:
@@ -289,11 +205,11 @@ Adaptive chunk size:
### Tier Interaction ### Tier Interaction
Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They always use direct file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path. Range requests **bypass all download tiers** (LRU, MMAP). They always use direct blob file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
### Limitations ### Limitations
- **Multipart ranges not supported**: only the first range in a multi-range request is served. Additional ranges are ignored. - **Multipart ranges not supported**: only the first range in a multi-range request is served.
- **`If-Range` not handled**: no conditional range support. - **`If-Range` not handled**: no conditional range support.
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked. - **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
@@ -302,9 +218,8 @@ Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They alw
## Upload Flow ## Upload Flow
``` ```
Request → file size < 256KB? → Write-behind cache (instant 201, async flush) Request → file size < 1MB? → Buffered write (sync to blob store)
→ file size < 1MB? → Buffered write (sync) → file size ≥ 1MB → Streaming write (chunk-by-chunk to blob store)
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to temp + rename)
``` ```
### Upload Strategy Selection ### Upload Strategy Selection
@@ -313,33 +228,32 @@ Request → file size < 256KB? → Write-behind cache (instant 201, async flush)
```rust ```rust
pub enum UploadStrategy { pub enum UploadStrategy {
WriteBehind, // < 256 KB — instant response, async disk write Buffered, // < 1 MB — collect bytes, write to blob store
Buffered, // 256 KB – 1 MB — sync write to final path Streaming, // ≥ 1 MB — chunk-by-chunk write via save_file_from_stream
Streaming, // ≥ 1 MB — chunk-by-chunk write to temp file + rename
} }
``` ```
| Constant | Value | | Constant | Value |
|---|---| |---|---|
| `WRITE_BEHIND_THRESHOLD` | 256 KB |
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB | | `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
### Handler-Level Buffering ### Handler-Level Buffering
Both upload handlers (`upload_file` and `upload_file_with_cache`) buffer the **entire multipart body in RAM** as `Vec<Bytes>` before calling the service layer: Upload handlers buffer the multipart body in RAM as `Vec<Bytes>` before calling the service layer:
```rust ```rust
let mut chunks: Vec<Bytes> = Vec::new(); let mut chunks: Vec<Bytes> = Vec::new();
while let Some(chunk) = field.chunk().await { while let Some(chunk) = field.chunk().await {
chunks.push(chunk); chunks.push(chunk);
} }
// All bytes are now in RAM
upload_service.smart_upload(..., chunks, total_size).await upload_service.smart_upload(..., chunks, total_size).await
``` ```
The "streaming" in `UploadStrategy::Streaming` refers to the **service→repository** path, not the HTTP-body→disk path. By the time `save_file_from_stream()` is called, data is already in memory. ### Buffered Path (< 1 MB)
### Streaming Path (≥ 1 MB): Service → Repository Uses `save_file()` — all bytes are passed to `FileBlobWriteRepository`, which calls `DedupService.store_bytes()` to compute hash and store the blob, then INSERTs metadata into `storage.files`.
### Streaming Path (≥ 1 MB)
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`: `smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
@@ -348,23 +262,13 @@ let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
``` ```
**`save_file_from_stream()` implementation** (`file_fs_write_repository.rs`): `FileBlobWriteRepository.save_file_from_stream()` collects the stream, stores via DedupService, and INSERTs metadata.
1. Resolves target path + generates unique name if collision ### Dedup Integration
2. Creates temp file: `{target_path}.tmp.upload`
3. Iterates stream, writing each chunk with `fh.write_all(&chunk)`
4. Calls `fh.flush()` + `fh.sync_all()` for durability
5. Atomic rename: `fs::rename(temp_path, final_path)`
6. Post-write: ID mapping, cache invalidation, metadata update
### Buffered Path (256 KB - 1 MB) Deduplication is handled at the **repository layer** (not the service layer) for all upload strategies. `FileBlobWriteRepository` always calls `DedupService.store_bytes()` which:
Uses `save_file()` -- writes all bytes directly to the **final path** (no temp file). For larger content, writes in chunks of **ResourceConfig.chunk_size_bytes** (1 MB). 1. Computes SHA-256 hash of content
2. Checks if blob already exists (dedup hit → increment ref count, skip write)
### Write-Behind Path (< 256 KB) 3. If new → atomic write to `.blobs/{prefix}/{hash}.blob`
4. Returns the hash for storage in `storage.files.blob_hash`
See **Layer 5** above. Instant `201`, background flush within 30 seconds.
### Dedup Pre-Check
Runs for **all upload strategies** before writing. Re-combines all chunks into a single `Vec<u8>` for hash computation, which means data is temporarily duplicated in RAM during dedup processing.
doc/deduplication.md
+6 -3
View File
@@ -285,9 +285,12 @@ pub struct CoreServices {
// ... // ...
} }
// Injected into application services: // Injected into blob repositories (which handle dedup internally):
FileUploadService::new_full(... core.dedup_service.clone()) FileBlobReadRepository::new(pool, core.dedup_service.clone(), folder_repo)
FileManagementService::new_full(... core.dedup_service.clone()) FileBlobWriteRepository::new(pool, core.dedup_service.clone(), folder_repo)
// Also injected into FileManagementService for ref cleanup on delete:
FileManagementService::new_full(write, read, trash, core.dedup_service.clone())
``` ```
## Persistence ## Persistence
doc/deployment.md
+4 -4
View File
@@ -174,17 +174,17 @@ Hardcoded defaults in `src/common/config.rs`:
| Feature | Requires DB | Requires Auth | Feature Flag | | Feature | Requires DB | Requires Auth | Feature Flag |
|---|---|---|---| |---|---|---|---|
| File storage | No | No | Always on | | File storage | Yes | No | Always on |
| Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` | | Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` |
| OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` | | OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` |
| File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` | | File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` |
| Trash | No | No | `OXICLOUD_ENABLE_TRASH` | | Trash | Yes | No | `OXICLOUD_ENABLE_TRASH` |
| Search | No | No | `OXICLOUD_ENABLE_SEARCH` | | Search | Yes | No | `OXICLOUD_ENABLE_SEARCH` |
| Favorites | Yes | Yes | Always on (when DB available) | | Favorites | Yes | Yes | Always on (when DB available) |
| Recent items | Yes | Yes | Always on (when DB available) | | Recent items | Yes | Yes | Always on (when DB available) |
| Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` | | Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` |
| Admin panel | Yes | Yes | Always on (when auth enabled) | | Admin panel | Yes | Yes | Always on (when auth enabled) |
| WebDAV | No | Optional | Always on | | WebDAV | Yes | Optional | Always on |
| CalDAV | Yes | Yes | Always on (when DB available) | | CalDAV | Yes | Yes | Always on (when DB available) |
| CardDAV | Yes | Yes | Always on (when DB available) | | CardDAV | Yes | Yes | Always on (when DB available) |
| Deduplication | No | No | Always on | | Deduplication | No | No | Always on |
doc/file-system-safety.md
+82 -113
View File
@@ -1,156 +1,125 @@
# 03 - File System Safety # 03 - Storage Safety
OxiCloud ensures data integrity and durability during file operations through atomic writes, fsync, and directory synchronization. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss. OxiCloud ensures data integrity and durability through a combination of PostgreSQL transactional guarantees and atomic blob writes. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss.
--- ---
## The Problem: Buffered I/O ## Storage Model
Standard filesystem operations use buffered I/O by default: OxiCloud uses a **100% blob storage model**:
```rust - **Metadata** (file names, folder hierarchy, sizes, MIME types, trash status) lives in **PostgreSQL** — protected by ACID transactions.
// This operation may not immediately persist to disk - **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob` — protected by atomic writes and fsync.
fs::write(path, content)
```
When an application writes data, the OS typically:
1. Accepts the write into memory buffers
2. Acknowledges completion to the application
3. Schedules the actual disk write for later
A crash during that window means data loss -- the data exists only in memory buffers that haven't been flushed.
--- ---
## OxiCloud's Approach ## PostgreSQL Safety (Metadata)
All safety mechanisms live in the **FileSystemUtils** service. All file and folder metadata operations use PostgreSQL transactions:
### Atomic Write Pattern - **Single-row operations** (INSERT, UPDATE, DELETE) are inherently atomic.
- **Multi-step operations** (e.g., move file: UPDATE folder_id + UPDATE path) use explicit transactions via `sqlx`.
- **Foreign key constraints** prevent orphaned records (e.g., files referencing non-existent folders).
- **Unique constraints** prevent duplicate names within the same parent folder.
- **Soft-delete** for trash (`is_trashed = TRUE`) preserves data until explicit permanent deletion.
Files are written using write-then-rename: The `storage.trash_items` VIEW provides a unified read interface over trashed files and folders without duplicating data.
---
## Blob Storage Safety (Content)
### DedupService Atomic Writes
**File:** `src/infrastructure/services/dedup_service.rs`
When storing file content, DedupService uses the following pattern:
1. **Hash computation** — SHA-256 hash of content determines the blob path
2. **Deduplication check** — if a blob with the same hash exists, only increment the reference counter (no write needed)
3. **Atomic write** — if new content:
- Write to a temporary file (`.blob.tmp`)
- Call `fsync` to ensure data reaches persistent storage
- Atomically rename temp file to final path (`.blobs/{prefix}/{hash}.blob`)
4. **Reference counting** — track how many files reference each blob
This ensures that a blob either fully exists or doesn't — no partial writes.
### FileSystemUtils
**File:** `src/infrastructure/services/file_system_utils.rs`
Low-level utilities used internally by DedupService and other infrastructure services:
```rust ```rust
/// Writes data to a file with fsync to ensure durability /// Atomic write: temp file → fsync → rename
/// Uses a safe atomic write pattern: write to temp file, fsync, rename
pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError> pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError>
```
Steps: /// Directory creation with fsync
1. Write to a temporary file in the same directory
2. Call `fsync` to ensure data is on disk
3. Atomically rename the temp file to the target file
4. Sync the parent directory to ensure the rename is persisted
### Directory Synchronization
```rust
/// Creates directories with fsync
pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError> pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
```
Directories are created, their entries persisted to disk, and parent directories synchronized too. /// Rename with directory sync
pub async fn rename_with_sync<P, Q>(from: P, to: Q) -> Result<(), IoError>
### Rename and Delete Operations /// Delete with directory sync
```rust
/// Renames a file or directory with proper syncing
pub async fn rename_with_sync<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> Result<(), IoError>
/// Removes a file with directory syncing
pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError> pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
``` ```
Both complete the operation itself, then update and sync the parent directory entry. ### fsync Guarantees
- `sync_all()` on written files ensures data and metadata reach the physical storage device
- Directory entries are synced after create/rename/delete operations
- Prevents data loss during crashes or power failures between OS buffer flush and disk write
--- ---
## Implementation Details ## Transaction Flow: File Upload
### fsync on Files ```
1. DedupService.store_bytes(content)
→ Compute SHA-256 hash
→ Check if blob exists (dedup hit → increment ref, return hash)
→ Write to .blobs/{prefix}/{hash}.blob.tmp
→ fsync + rename → .blobs/{prefix}/{hash}.blob
```rust 2. FileBlobWriteRepository.save_file()
// Write file content → BEGIN TRANSACTION
file.write_all(contents).await?; → INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
→ COMMIT
// Ensure data is synced to disk
file.flush().await?;
file.sync_all().await?;
``` ```
`sync_all()` instructs the OS to flush data and metadata to the physical storage device. If step 1 fails, no metadata is written. If step 2 fails, the blob exists but is unreferenced (cleaned up by garbage collection). Data is never in an inconsistent state.
### fsync on Directories ## Transaction Flow: File Deletion
```rust ```
// Sync a directory to ensure its contents (entries) are durable 1. FileBlobWriteRepository.delete_file_permanently()
async fn sync_directory<P: AsRef<Path>>(path: P) -> Result<(), IoError> { → BEGIN TRANSACTION
let dir_file = OpenOptions::new().read(true).open(path).await?; → DELETE FROM storage.files WHERE id = $1 (captures blob_hash first)
dir_file.sync_all().await → COMMIT
}
2. DedupService.decrement_ref(blob_hash)
→ Decrement reference counter
→ If counter reaches 0, delete the blob file
``` ```
Required after any operation that modifies directory entries (create, rename, delete). If step 2 fails, an unreferenced blob may remain on disk (occupies space but is not a correctness issue). Future garbage collection can clean these up.
---
## Usage in the Codebase
### File Write Repository
```rust
// Write the file to disk using atomic write with fsync
tokio::time::timeout(
self.config.timeouts.file_write_timeout(),
FileSystemUtils::atomic_write(&abs_path, &content)
).await
```
### File Move Operations
```rust
// Move the file physically with fsync
time::timeout(
self.config.timeouts.file_timeout(),
FileSystemUtils::rename_with_sync(&old_abs_path, &new_abs_path)
).await
```
### Directory Creation
```rust
// Ensure the parent directory exists with proper syncing
self.ensure_parent_directory(&abs_path).await?;
// Implementation uses FileSystemUtils
async fn ensure_parent_directory(&self, abs_path: &PathBuf) -> FileRepositoryResult<()> {
if let Some(parent) = abs_path.parent() {
time::timeout(
self.config.timeouts.dir_timeout(),
FileSystemUtils::create_dir_with_sync(parent)
).await
}
}
```
--- ---
## Benefits ## Benefits
1. **Data durability** -- critical data is synced to persistent storage 1. **ACID transactions** — metadata operations are atomic, consistent, isolated, and durable
2. **Crash resilience** -- recovery from unexpected failures without data loss 2. **Content-addressable storage** — identical content is stored once, referenced by hash
3. **Consistency** -- file operations maintain a consistent filesystem state 3. **Crash resilience** — atomic blob writes + PostgreSQL WAL ensure recovery
4. **Atomic operations** -- file writes appear as all-or-nothing 4. **No partial writes** — temp file + rename pattern guarantees all-or-nothing
5. **Referential integrity** — foreign keys prevent orphaned metadata
--- ---
## Performance Considerations ## Performance Considerations
Syncing to disk costs more than buffered writes. OxiCloud mitigates this by: - PostgreSQL connection pooling (`sqlx::PgPool`) amortizes connection overhead
- Dedup hash computation is CPU-bound but avoids unnecessary disk writes for duplicate content
1. Applying these measures only to critical operations - Blob fsync adds latency vs. buffered writes, but ensures durability for critical user data
2. Using timeouts to prevent indefinite blocking - Content cache (in-memory LRU) serves repeat reads without disk or DB access
3. Implementing parallel processing for large files
The tradeoff favors safety for critical data while maintaining good performance for most operations.
doc/important-delta-sync-implementation.md
+2 -2
View File
@@ -462,7 +462,7 @@ src/
│ └── delta_sync_handler.rs # NUEVO: Endpoints API │ └── delta_sync_handler.rs # NUEVO: Endpoints API
│ │
└── common/ └── common/
└── di.rs # Añadir: delta_sync_service a CoreServices └── di.rs # Añadir: delta_sync_service a AppState
``` ```
### Main service (delta_sync_service.rs) ### Main service (delta_sync_service.rs)
@@ -1133,7 +1133,7 @@ thiserror = "1.0" # Para errores tipados (probablemente ya existe)
- [ ] Implement **generate_delta()** - [ ] Implement **generate_delta()**
- [ ] Implement **apply_delta()** - [ ] Implement **apply_delta()**
- [ ] Create handler and API endpoints - [ ] Create handler and API endpoints
- [ ] Integrate into DI (**CoreServices**) - [ ] Integrate into DI (**AppState**)
- [ ] Add routes in `routes.rs` - [ ] Add routes in `routes.rs`
- [ ] Integrate with upload (automatic indexing) - [ ] Integrate with upload (automatic indexing)
- [ ] Integrate with delete (signature cleanup) - [ ] Integrate with delete (signature cleanup)
doc/internal-architecture.md
+170 -173
View File
@@ -10,6 +10,19 @@ All cross-layer dependencies point inward via trait-based ports. The DI containe
--- ---
## Storage Model: 100% Blob Storage
OxiCloud uses a **100% blob storage model** where:
- **File metadata** (name, folder, size, user, timestamps, trash status) is stored in **PostgreSQL** (`storage.files` table).
- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob`.
- **Folder structure** is purely virtual — represented as rows in `storage.folders` (no filesystem directories per user).
- **Trash** is a soft-delete flag (`is_trashed`, `trashed_at`) on files and folders, exposed via `storage.trash_items` VIEW.
There are no filesystem-based ID mappings, no `folder_ids.json`/`file_ids.json`, and no storage mediator.
---
## Dependency Injection Container ## Dependency Injection Container
### AppServiceFactory ### AppServiceFactory
@@ -26,13 +39,13 @@ pub struct AppServiceFactory {
Initialization order in `build_app_state()`: Initialization order in `build_app_state()`:
1. **Core services** -- path, caches, ID mapping, thumbnail, write-behind, chunked upload, transcode, dedup, compression 1. **Core services** — path, content cache, thumbnail, chunked upload, transcode, dedup, compression
2. **Repository services** -- folder repo (stub mediator first), then **FileSystemStorageMediator** (real), file repos, metadata cache, buffer pool 2. **Repository services** — `FolderDbRepository`, `FileBlobReadRepository`, `FileBlobWriteRepository`, `TrashDbRepository` (all PgPool-backed)
3. **Trash service** (if **enable_trash** enabled) 3. **Trash service** (if **enable_trash** enabled)
4. **Application services** -- folder, file upload/retrieval/management, search, i18n 4. **Application services** — folder, file upload/retrieval/management, search, i18n
5. **Share service** (if **enable_file_sharing** enabled) 5. **Share service** (if **enable_file_sharing** enabled)
6. **DB-dependent services** -- favorites, recent, storage usage, auth (via **auth_factory**) 6. **DB-dependent services** — favorites, recent, storage usage, auth (via **auth_factory**)
7. **Preload** translations + metadata cache 7. **Preload** translations
8. **ZIP service** (needs file retrieval + folder service, wired last) 8. **ZIP service** (needs file retrieval + folder service, wired last)
9. **Assemble AppState** + admin settings + CalDAV/CardDAV 9. **Assemble AppState** + admin settings + CalDAV/CardDAV
@@ -67,11 +80,7 @@ Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `w
pub struct CoreServices { pub struct CoreServices {
pub path_service: Arc<PathService>, pub path_service: Arc<PathService>,
pub file_content_cache: Arc<dyn ContentCachePort>, pub file_content_cache: Arc<dyn ContentCachePort>,
pub id_mapping_service: Arc<dyn IdMappingPort>, // folder IDs
pub file_id_mapping_service: Arc<IdMappingService>, // file IDs (concrete)
pub id_mapping_optimizer: Arc<IdMappingOptimizer>,
pub thumbnail_service: Arc<dyn ThumbnailPort>, pub thumbnail_service: Arc<dyn ThumbnailPort>,
pub write_behind_cache: Arc<dyn WriteBehindCachePort>,
pub chunked_upload_service: Arc<dyn ChunkedUploadPort>, pub chunked_upload_service: Arc<dyn ChunkedUploadPort>,
pub image_transcode_service: Arc<dyn ImageTranscodePort>, pub image_transcode_service: Arc<dyn ImageTranscodePort>,
pub dedup_service: Arc<dyn DedupPort>, pub dedup_service: Arc<dyn DedupPort>,
@@ -82,11 +91,10 @@ pub struct CoreServices {
pub struct RepositoryServices { pub struct RepositoryServices {
pub folder_repository: Arc<dyn FolderStoragePort>, pub folder_repository: Arc<dyn FolderStoragePort>,
pub folder_repo_concrete: Arc<FolderDbRepository>,
pub file_read_repository: Arc<dyn FileReadPort>, pub file_read_repository: Arc<dyn FileReadPort>,
pub file_write_repository: Arc<dyn FileWritePort>, pub file_write_repository: Arc<dyn FileWritePort>,
pub i18n_repository: Arc<dyn I18nService>, pub i18n_repository: Arc<dyn I18nService>,
pub storage_mediator: Arc<dyn StorageMediator>,
pub metadata_cache: Arc<FileMetadataCache>,
pub trash_repository: Option<Arc<dyn TrashRepository>>, pub trash_repository: Option<Arc<dyn TrashRepository>>,
} }
@@ -113,112 +121,116 @@ pub struct AuthServices {
--- ---
## ID Mapping System ## Database Schema (Storage)
Maps bidirectionally between **filesystem StoragePaths** and **UUID identifiers**. Two separate instances exist: one for folders (`folder_ids.json`), one for files (`file_ids.json`). All file and folder metadata lives in the `storage` PostgreSQL schema:
### StoragePath (Domain Value Object) ```sql
CREATE SCHEMA IF NOT EXISTS storage;
**File:** `src/domain/services/path_service.rs` CREATE TABLE storage.folders (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
parent_id UUID REFERENCES storage.folders(id) ON DELETE CASCADE,
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
trashed_at TIMESTAMPTZ,
original_parent_id UUID,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE storage.files (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
folder_id UUID NOT NULL REFERENCES storage.folders(id) ON DELETE CASCADE,
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
blob_hash TEXT NOT NULL,
size BIGINT NOT NULL DEFAULT 0,
mime_type TEXT NOT NULL DEFAULT 'application/octet-stream',
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
trashed_at TIMESTAMPTZ,
original_folder_id UUID,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE OR REPLACE VIEW storage.trash_items AS
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
user_id, size, mime_type, trashed_at, created_at
FROM storage.files WHERE is_trashed = TRUE
UNION ALL
SELECT id, name, 'folder' AS item_type, parent_id,
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
FROM storage.folders WHERE is_trashed = TRUE;
```
---
## Repository Layer (Infrastructure)
All repositories use **PgPool** for metadata and **DedupService** for blob content.
### FolderDbRepository
**File:** `src/infrastructure/repositories/pg/folder_db_repository.rs`
```rust ```rust
#[derive(Debug, Clone, PartialEq, Eq, Default)] pub struct FolderDbRepository {
pub struct StoragePath { pool: Option<Arc<PgPool>>,
segments: Vec<String>, // e.g., ["Mi Carpeta - admin", "file.txt"]
} }
``` ```
| Method | Description | Implements `FolderRepository`. Uses recursive CTEs for path building, unique constraints for name dedup within parent, and soft-delete flags for trash operations.
|---|---|
| `root()` | Empty path (storage root) |
| `from_string(path)` | Parse from `/`-delimited string |
| `join(segment)` | Append a segment |
| `file_name()` | Last segment |
| `parent()` | All segments except last |
| `to_string()` | Join segments with `/` |
### IdMappingPort (Application Port) Key methods: `create_folder`, `get_folder`, `get_folder_by_path`, `list_folders`, `rename_folder`, `move_folder`, `delete_folder`, `move_to_trash`, `restore_from_trash`, `create_home_folder`, `get_folder_user_id`.
**File:** `src/application/ports/outbound.rs` `new_stub()` creates a pool-less instance for `AppState::default()`.
### FileBlobReadRepository
**File:** `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
```rust ```rust
#[async_trait] pub struct FileBlobReadRepository {
pub trait IdMappingPort: Send + Sync + 'static { pool: Arc<PgPool>,
async fn get_or_create_id(&self, path: &StoragePath) -> Result<String, DomainError>; dedup: Arc<dyn DedupPort>,
async fn get_path_by_id(&self, id: &str) -> Result<StoragePath, DomainError>; folder_repo: Arc<FolderDbRepository>,
async fn update_path(&self, id: &str, new_path: &StoragePath) -> Result<(), DomainError>;
async fn remove_id(&self, id: &str) -> Result<(), DomainError>;
async fn save_changes(&self) -> Result<(), DomainError>;
// Default impls for PathBuf variants:
async fn get_file_path(&self, file_id: &str) -> Result<PathBuf, DomainError>;
async fn update_file_path(&self, file_id: &str, new_path: &PathBuf) -> Result<(), DomainError>;
} }
``` ```
### IdMappingService (Base Implementation) Implements `FileReadPort`. Reads metadata from `storage.files` and content from blob store via `dedup.read_blob()` / `read_blob_bytes()`.
**File:** `src/infrastructure/services/id_mapping_service.rs` Key methods: `get_file`, `list_files`, `get_file_content`, `get_file_stream`, `get_file_range_stream`, `get_file_mmap`, `get_file_path`, `get_parent_folder_id`.
### FileBlobWriteRepository
**File:** `src/infrastructure/repositories/pg/file_blob_write_repository.rs`
```rust ```rust
pub struct IdMappingService { pub struct FileBlobWriteRepository {
map_path: PathBuf, // e.g., storage/file_ids.json pool: Arc<PgPool>,
id_map: RwLock<IdMap>, dedup: Arc<dyn DedupPort>,
save_mutex: Mutex<()>, folder_repo: Arc<FolderDbRepository>,
timeouts: TimeoutConfig,
pending_save: RwLock<bool>,
}
struct IdMap {
path_to_id: HashMap<String, String>,
id_to_path: HashMap<String, String>,
version: u32,
} }
``` ```
Operations: Implements `FileWritePort`. Stores content via `dedup.store_bytes()` (returns hash), then INSERTs metadata into `storage.files`.
- `get_or_create_id(path)` -- Read-lock first (cache hit). Write-lock on miss, generates `Uuid::new_v4()`.
- `save_pending_changes()` -- Atomic write: serialize → write `.tmp` file → rename over original (with retry).
- `new(map_path)` -- Loads from JSON, rebuilds inverse map if inconsistent.
Persistence format (`storage/file_ids.json`): Key methods: `save_file`, `save_file_from_stream`, `move_file`, `rename_file`, `delete_file`, `update_file_content`, `move_to_trash`, `restore_from_trash`, `delete_file_permanently`.
```json
{
"path_to_id": { "/Mi Carpeta - admin/doc.pdf": "a1b2c3d4-..." },
"id_to_path": { "a1b2c3d4-...": "/Mi Carpeta - admin/doc.pdf" },
"version": 42
}
```
### IdMappingOptimizer (Cache Layer) ### TrashDbRepository
**File:** `src/infrastructure/services/id_mapping_optimizer.rs` **File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
Wraps **IdMappingService** with an in-memory TTL cache:
| Parameter | Value |
|---|---|
| Max cache entries | 10,000 |
| TTL | 300 s (5 min) |
| Cleanup interval | 150 s (2.5 min) |
| Batch threshold | ≥ 20 queued items |
| Max concurrent batches | 2 (semaphore) |
```rust ```rust
pub struct IdMappingOptimizer { pub struct TrashDbRepository {
base_service: Arc<IdMappingService>, pool: Arc<PgPool>,
path_to_id_cache: RwLock<HashMap<String, (String, Instant)>>, retention_days: u32,
id_to_path_cache: RwLock<HashMap<String, (String, Instant)>>,
stats: RwLock<OptimizerStats>,
batch_limiter: Semaphore,
pending_batch: Mutex<BatchQueue>,
} }
``` ```
Lookup flow: check cache → if miss, queue request → trigger batch if ≥ 20 pending → fallback to **base_service** → update cache. Implements `TrashRepository`. Reads from `storage.trash_items` VIEW. `clear_trash` DELETEs rows where `is_trashed = TRUE`. `get_expired_items` checks `trashed_at` against the configured retention period.
On `update_path` / `remove_id`, cache entries are invalidated first, then delegated.
Used only for folder ID mapping. File ID mapping uses the base **IdMappingService** directly.
--- ---
@@ -232,77 +244,31 @@ pub struct PathService {
} }
``` ```
### Path Resolution Used for resolving storage root paths (blob storage directory, thumbnail paths, etc.). Not used for per-user folder resolution — that is handled by `FolderDbRepository` via PostgreSQL.
### StoragePath (Domain Value Object)
**File:** `src/domain/services/path_service.rs`
```rust
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct StoragePath {
segments: Vec<String>,
}
```
| Method | Description | | Method | Description |
|---|---| |---|---|
| `resolve_path(storage_path)` | Appends **StoragePath** segments to **root_path** → absolute `PathBuf` | | `root()` | Empty path (storage root) |
| `to_storage_path(physical_path)` | Strips **root_path** prefix → **StoragePath** (returns `None` if outside root) | | `from_string(path)` | Parse from `/`-delimited string |
| `create_file_path(folder, name)` | Combines folder path + filename | | `join(segment)` | Append a segment |
| `is_direct_child(parent, child)` | Check parent-child relationship | | `file_name()` | Last segment |
| `is_in_root(path)` | Verify path is within storage root | | `parent()` | All segments except last |
| `to_string()` | Join segments with `/` |
### Path Validation
`validate_path(path)` rejects:
- Empty path segments
- Segments containing dangerous characters: `\`, `:`, `*`, `?`, `"`, `<`, `>`, `|`
- Segments starting with `.` (exception: `.well-known` for WebDAV/CalDAV/CardDAV)
### Trait Implementations ### Trait Implementations
- **StoragePort** -- `resolve_path()`, `ensure_directory()` (validates first, then `fs::create_dir_all`), `file_exists()`, `directory_exists()` - **StoragePort** — `resolve_path()`, `ensure_directory()`, `file_exists()`, `directory_exists()`
- **StorageMediator** -- Simplified stub variant. Folder lookup methods return `NotFound`.
---
## Storage Mediator
Bridges folder IDs to filesystem paths by combining folder repository, path service, and ID mapping.
### StorageMediator Trait
**File:** `src/application/services/storage_mediator.rs`
```rust
#[async_trait]
pub trait StorageMediator: Send + Sync + 'static {
async fn get_folder_path(&self, folder_id: &str) -> StorageMediatorResult<PathBuf>;
async fn get_folder_storage_path(&self, folder_id: &str) -> StorageMediatorResult<StoragePath>;
async fn get_folder(&self, folder_id: &str) -> StorageMediatorResult<Folder>;
async fn file_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
async fn file_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
async fn folder_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
async fn folder_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
fn resolve_path(&self, relative_path: &Path) -> PathBuf;
fn resolve_storage_path(&self, storage_path: &StoragePath) -> PathBuf;
async fn ensure_directory(&self, path: &Path) -> StorageMediatorResult<()>;
async fn ensure_storage_directory(&self, storage_path: &StoragePath) -> StorageMediatorResult<()>;
}
```
### FileSystemStorageMediator
```rust
pub struct FileSystemStorageMediator {
pub folder_storage_port: Arc<dyn FolderStoragePort>,
pub path_service: Arc<dyn StoragePort>,
pub id_mapping: Arc<dyn IdMappingPort>,
}
```
Folder ID → filesystem path resolution:
```
folder_id ──► FolderStoragePort.get_folder(id)
──► Folder entity
──► IdMappingPort.get_path_by_id(folder.id())
──► StoragePath
──► StoragePort.resolve_path(storage_path)
──► PathBuf (absolute)
```
A **StubStorageMediator** also exists (returns `/tmp` paths) for DI bootstrap before the real mediator is available.
--- ---
@@ -326,8 +292,8 @@ pub struct Session {
``` ```
Constructors: Constructors:
- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` -- generates UUID, panics if **user_id** or **refresh_token** empty - `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` — generates UUID, panics if **user_id** or **refresh_token** empty
- `Session::from_raw(...)` -- for DB reconstruction - `Session::from_raw(...)` — for DB reconstruction
### SessionRepository (Domain Port) ### SessionRepository (Domain Port)
@@ -386,7 +352,6 @@ CREATE TABLE IF NOT EXISTS auth.sessions (
revoked BOOLEAN NOT NULL DEFAULT FALSE revoked BOOLEAN NOT NULL DEFAULT FALSE
); );
-- Indexes
CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id); CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id);
CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token); CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token);
CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at); CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at);
@@ -399,12 +364,12 @@ CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked)
**File:** `src/application/services/auth_application_service.rs` **File:** `src/application/services/auth_application_service.rs`
**AuthApplicationService** orchestrates authentication using: **AuthApplicationService** orchestrates authentication using:
- **UserStoragePort** -- user CRUD - **UserStoragePort** — user CRUD
- **SessionStoragePort** -- session lifecycle - **SessionStoragePort** — session lifecycle
- **PasswordHasherPort** -- Argon2id hashing - **PasswordHasherPort** — Argon2id hashing
- **TokenServicePort** -- JWT generation/validation - **TokenServicePort** — JWT generation/validation
- `RwLock<OidcState>` -- hot-reloadable OIDC configuration - `RwLock<OidcState>` — hot-reloadable OIDC configuration
- `Mutex<HashMap<String, PendingOidcFlow>>` -- in-flight OIDC login states - `Mutex<HashMap<String, PendingOidcFlow>>` — in-flight OIDC login states
Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**. Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**.
@@ -422,14 +387,14 @@ pub trait FileUseCaseFactory: Send + Sync + 'static {
} }
``` ```
**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**. The main DI-wired services use `*_full()` constructors that inject write-behind cache, dedup, content cache, and transcode ports for full optimization. **AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**.
### File Operation Port Hierarchy ### File Operation Port Hierarchy
| Port | Key Methods | | Port | Key Methods |
|---|---| |---|---|
| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `WriteBehind` <256KB, `Buffered` 256KB-1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` | | **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `Buffered` <1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` |
| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (write-behind → content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` | | **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` |
| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) | | **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) |
--- ---
@@ -455,15 +420,14 @@ pub trait FileUseCaseFactory: Send + Sync + 'static {
┌─────────▼────────────────▼─────────────────────▼────────────┐ ┌─────────▼────────────────▼─────────────────────▼────────────┐
│ Infrastructure Layer │ │ Infrastructure Layer │
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ FileFsRead/ │ │ IdMapping │ │ SessionPg │ │ │ │ FileBlobRead │ │ PathService │ │ SessionPg │ │
│ │ FileFsWrite │ │ + Optimizer │ │ UserPg │ │ │ │ FileBlobWrite │ │ DedupService │ │ UserPg │ │
│ │ FolderFs │ │ PathService │ │ JwtTokenService │ │ │ │ FolderDb │ │ Thumbnail │ │ JwtTokenService │ │
│ │ TrashFs │ │ StorageMed. │ │ Argon2Hasher │ │ │ │ TrashDb │ │ Transcode │ │ Argon2Hasher │ │
│ └────────────────┘ └──────────────┘ └──────────────────┘ │ │ └────────────────┘ └──────────────┘ └──────────────────┘ │
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ ContentCache │ │ Thumbnail │ │ WriteBehind │ │ │ │ ContentCache │ │ Compression │ │ ChunkedUpload │ │
│ │ MetadataCache │ │ Transcode │ │ BufferPool │ │ │ │ BufferPool │ │ ZipService │ │ ShareFsRepo │ │
│ │ BufferPool │ │ Dedup │ │ Compression │ │
│ └────────────────┘ └──────────────┘ └──────────────────┘ │ │ └────────────────┘ └──────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
│ │
@@ -471,7 +435,40 @@ pub trait FileUseCaseFactory: Send + Sync + 'static {
│ Domain Layer │ │ Domain Layer │
│ Entities: File, Folder, Session, User, Calendar, Contact │ │ Entities: File, Folder, Session, User, Calendar, Contact │
│ Value Objects: StoragePath │ │ Value Objects: StoragePath │
│ Repository Traits: SessionRepository, ... │ │ Repository Traits: FolderRepository, TrashRepository, ... │
│ Domain Errors │ │ Domain Errors │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
``` ```
### Data Flow: File Upload
```
HTTP Request (multipart)
→ FileUploadService.smart_upload()
→ FileBlobWriteRepository.save_file() / save_file_from_stream()
→ DedupService.store_bytes() → .blobs/{prefix}/{hash}.blob
→ INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
→ 201 Created (FileDto)
```
### Data Flow: File Download
```
HTTP Request (GET /api/files/{id}/download)
→ FileRetrievalService.get_file_optimized()
→ ContentCache hit? → serve from RAM
→ FileBlobReadRepository.get_file_content() / get_file_stream()
→ SELECT blob_hash FROM storage.files WHERE id = $1
→ DedupService.read_blob(hash) → bytes from .blobs/
→ Optional WebP transcode → response
```
### Data Flow: Folder Operations
```
HTTP Request (POST /api/folders)
→ FolderService.create_folder()
→ FolderDbRepository.create_folder()
→ INSERT INTO storage.folders (name, parent_id, user_id, ...)
→ 201 Created (FolderDto)
```
doc/share-integration.md
+31 -24
View File
@@ -134,7 +134,7 @@ Handles: shared element validation, permission management, unique link/token gen
## Infrastructure ## Infrastructure
**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share links to the filesystem: **ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share link metadata to a local JSON file:
```rust ```rust
pub struct ShareFsRepository { pub struct ShareFsRepository {
@@ -157,7 +157,9 @@ struct ShareRecord {
} }
``` ```
Stores shared links in a JSON file. Supports queries and updates, search by ID/token/user, and pagination. Stores share link records in a JSON file. Supports queries and updates, search by ID/token/user, and pagination.
> **Note:** This repository stores *share link metadata* only (tokens, permissions, expiration). The actual file/folder content is accessed via `FileReadPort` / `FolderStoragePort` which use the blob storage model (PostgreSQL metadata + DedupService blobs).
## API Handlers and Routes ## API Handlers and Routes
@@ -256,26 +258,30 @@ The service is instantiated via **AppServiceFactory** in `src/common/di.rs` and
```rust ```rust
// In AppServiceFactory::create_share_service() // In AppServiceFactory::create_share_service()
let share_service: Option<Arc<dyn ShareUseCase>> = if config.features.enable_file_sharing { pub fn create_share_service(&self, repos: &RepositoryServices)
let share_repository = Arc::new(ShareFsRepository::new(Arc::new(config.clone()))); -> Option<Arc<dyn ShareUseCase>>
let share_service = Arc::new(ShareService::new( {
Arc::new(config.clone()), if !self.config.features.enable_file_sharing {
share_repository, return None;
file_read_repository.clone(), }
folder_repository.clone(),
password_hasher.clone(),
));
Some(share_service)
} else {
None
};
// Add to AppState let share_repository = Arc::new(ShareFsRepository::new(
let app_state = AppState { Arc::new(self.config.clone())
// ... ));
share_service: share_service.clone(),
// ... let password_hasher: Arc<dyn PasswordHasherPort> =
}; Arc::new(Argon2PasswordHasher::new());
let service = Arc::new(ShareService::new(
Arc::new(self.config.clone()),
share_repository,
repos.file_read_repository.clone(), // FileBlobReadRepository
repos.folder_repository.clone(), // FolderDbRepository
password_hasher,
));
Some(service)
}
``` ```
## Workflows ## Workflows
@@ -353,8 +359,9 @@ HTTP status code mapping:
## Technical Notes ## Technical Notes
- **Performance**: JSON file-based storage works for moderate volumes. For higher load, migrate to a database. - **Share metadata** is stored in a local JSON file via `ShareFsRepository`. This is separate from the 100% blob storage model used for file content.
- **Scalability**: the design supports horizontal scaling via distributed or cloud-based repositories. - **File/folder lookups** during share access go through `FileReadPort` / `FolderStoragePort`, which read metadata from PostgreSQL and content from the DedupService blob store.
- **Scalability**: for higher load, share metadata could be migrated to PostgreSQL using the same hexagonal architecture (implement `ShareRepository` with PgPool).
- **Maintenance**: clear separation of concerns makes testing and maintenance straightforward. - **Maintenance**: clear separation of concerns makes testing and maintenance straightforward.
The sharing feature is enabled by default in the current configuration. The sharing feature is enabled via `OXICLOUD_ENABLE_FILE_SHARING` configuration flag.
doc/trash-feature-summary.md
+73 -35
View File
@@ -16,64 +16,102 @@ Follows the hexagonal architecture:
- Services: **TrashService** implementing the trash use cases - Services: **TrashService** implementing the trash use cases
3. **Infrastructure Layer** (`/src/infrastructure/`): 3. **Infrastructure Layer** (`/src/infrastructure/`):
- Repositories: **TrashFsRepository** for filesystem-based trash storage - Repositories: **TrashDbRepository** (PostgreSQL) — reads from `storage.trash_items` VIEW, manages soft-delete flags
- Trash-related methods in existing repositories: `FileWriteRepository::move_to_trash()`, `FolderRepository::move_to_trash()`, etc. - Trash-related methods in file/folder repositories: `FileBlobWriteRepository::move_to_trash()`, `FolderDbRepository::move_to_trash()`, etc.
- Services: **TrashCleanupService** for automatic cleanup of expired items - Services: **TrashCleanupService** for automatic cleanup of expired items
4. **Interface Layer** (`/src/interfaces/`): 4. **Interface Layer** (`/src/interfaces/`):
- API handlers: `trash_handler.rs` with HTTP endpoints for trash operations - API handlers: `trash_handler.rs` with HTTP endpoints for trash operations
- Routes: updated `routes.rs` to include trash endpoints - Routes: updated `routes.rs` to include trash endpoints
## Storage Model
Trash uses a **soft-delete** model in PostgreSQL:
- Files and folders have `is_trashed` (BOOLEAN) and `trashed_at` (TIMESTAMPTZ) columns
- When an item is trashed, `is_trashed` is set to `TRUE` and `trashed_at` records the timestamp
- `original_parent_id` / `original_folder_id` stores the original location for restore
- The `storage.trash_items` VIEW provides a unified list of all trashed items (files + folders)
- No physical file movement occurs — blob content stays at `.blobs/{prefix}/{hash}.blob`
- Permanent deletion removes the DB row and decrements the blob reference counter
```sql
CREATE OR REPLACE VIEW storage.trash_items AS
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
user_id, size, mime_type, trashed_at, created_at
FROM storage.files WHERE is_trashed = TRUE
UNION ALL
SELECT id, name, 'folder' AS item_type, parent_id,
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
FROM storage.folders WHERE is_trashed = TRUE;
```
## Key Features ## Key Features
1. **Soft Deletion** -- files and folders move to trash, not immediately deleted 1. **Soft Deletion** — files and folders are flagged as trashed, not immediately deleted
2. **Per-User Trash** -- each user has an isolated trash bin 2. **Per-User Trash** — each user has an isolated trash bin (filtered by `user_id`)
3. **Retention Policy** -- items auto-delete after a configurable period 3. **Retention Policy** — items auto-delete after a configurable period
4. **Restoration** -- items can be restored to their original location 4. **Restoration** — items can be restored to their original location via `original_parent_id`/`original_folder_id`
5. **Permanent Deletion** -- items can be permanently deleted before retention expires 5. **Permanent Deletion** — items can be permanently deleted before retention expires (removes DB row + decrements blob ref)
6. **Empty Trash** -- wipe everything in the trash at once 6. **Empty Trash** — wipe everything in the trash at once
## API Endpoints ## API Endpoints
- `GET /api/trash` or `GET /api/trash/` -- list all items in the user's trash - `GET /api/trash` or `GET /api/trash/` — list all items in the user's trash
- `DELETE /api/trash/files/:id` -- move a file to trash - `DELETE /api/trash/files/:id` — move a file to trash
- `DELETE /api/trash/folders/:id` -- move a folder to trash - `DELETE /api/trash/folders/:id` — move a folder to trash
- `POST /api/trash/:id/restore` -- restore an item to its original location - `POST /api/trash/:id/restore` — restore an item to its original location
- `DELETE /api/trash/:id` -- permanently delete an item from trash - `DELETE /api/trash/:id` — permanently delete an item from trash
- `DELETE /api/trash/empty` -- empty the entire trash bin - `DELETE /api/trash/empty` — empty the entire trash bin
## Implementation Details
### TrashDbRepository
**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
```rust
pub struct TrashDbRepository {
pool: Arc<PgPool>,
retention_days: u32,
}
```
Key methods:
- `get_trash_items(user_id)` — SELECT from `storage.trash_items` WHERE `user_id = $1`
- `clear_trash(user_id)` — DELETE from `storage.files` and `storage.folders` WHERE `is_trashed = TRUE AND user_id = $1`
- `get_expired_items()` — finds items where `trashed_at + retention_days < NOW()`
### TrashService
**File:** `src/application/services/trash_service.rs`
Constructor: `TrashService::new(trash_repo, file_read, file_write, folder_repo, retention_days)`
Orchestrates trash operations by delegating to the appropriate repository:
- Moving a file to trash → `FileBlobWriteRepository::move_to_trash()`
- Moving a folder to trash → `FolderDbRepository::move_to_trash()`
- Permanent deletion → removes DB row + calls `DedupService::decrement_ref()` to clean blob if unreferenced
### TrashCleanupService
**File:** `src/infrastructure/services/trash_cleanup_service.rs`
Background job that runs every 24 hours to permanently delete items past the retention period.
## Testing ## Testing
1. **Unit Tests** -- testing **TrashService**: 1. **Unit Tests** — testing **TrashService**:
- Move files and folders to trash - Move files and folders to trash
- Restore items from trash - Restore items from trash
- Permanent deletion - Permanent deletion
- Empty trash operation - Empty trash operation
2. **Integration Tests** -- Python script hitting the API endpoints: 2. **Integration Tests** — Python script hitting the API endpoints:
- End-to-end testing of all trash operations - End-to-end testing of all trash operations
- Verification of move, list, restore, and delete behavior - Verification of move, list, restore, and delete behavior
3. **Shell Script** -- for manual testing and demonstration
## Configuration ## Configuration
- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true) - **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true)
- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**) - **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)
## Implementation Details
1. **Physical File Storage** -- when items are trashed, they physically move to a `.trash` directory
2. **Metadata Storage** -- trashed item info stored in a separate database table or file
3. **User Isolation** -- trash items are isolated by user ID
4. **Automatic Cleanup** -- a background job runs periodically to clean up expired items
5. **Transaction Safety** -- operations are atomic with proper error handling
## Future Enhancements
1. **Trash Quotas** -- limit trash storage per user
2. **Batch Operations** -- trash, restore, or delete multiple items at once
3. **Storage Optimization** -- deduplication for trashed items
4. **Version Control** -- track file versions when moving to trash
5. **Scheduled Cleanup** -- let users configure custom retention periods
6. **Trash Monitoring** -- metrics and alerts for trash usage and cleanup