docs: update documentation to reflect 100% blob storage model
Rewrite documentation to match the new architecture where all file metadata lives in PostgreSQL and content is stored as content-addressed blobs via DedupService. Updated files: - internal-architecture.md: complete rewrite — new DB schema, blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated DI container, service groups, architecture diagram, data flows - file-system-safety.md: repurposed as storage-safety.md — covers PostgreSQL ACID guarantees + DedupService atomic writes - caching-architecture.md: updated repo references to blob repos, removed write-behind cache section, updated upload/download flows - trash-feature-summary.md: rewritten for soft-delete model (is_trashed flag, trash_items VIEW, TrashDbRepository) - share-integration.md: clarified ShareFsRepository scope, updated DI snippet, added blob storage context note - deduplication.md: updated DI snippet (dedup injected into repos) - deployment.md: updated feature matrix (file storage requires DB) - important-delta-sync-implementation.md: updated DI references Removed legacy references: IdMappingPort, StorageMediator, WriteBehindCache, FsFileRepository, FsFolderRepository, TrashFsRepository, folder_ids.json, file_ids.json.
This commit is contained in:
Dionisio
+274
-370
@@ -1,370 +1,274 @@
|
||||
# 04 - Caching Architecture
|
||||
|
||||
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
||||
|
||||
## Cache Layers Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 2: MMAP (memmap2, 10-100MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Streaming (FramedRead, ≥100MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 4: File Metadata Cache (adaptive TTL) │ All file ops
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 5: Write-Behind Cache (<256KB uploads) │ Uploads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 6: Buffer Pool (reusable I/O buffers) │ Compression
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Layer 0: HTTP Cache Middleware
|
||||
|
||||
**File**: `src/interfaces/middleware/cache.rs`
|
||||
|
||||
Generic HTTP caching layer applied to API endpoints.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max entries | 1,000 |
|
||||
| Default max-age | 60 seconds |
|
||||
| Eviction | LRU (oldest 10% when full) |
|
||||
| Cleanup | Background task every 5 minutes |
|
||||
|
||||
Features:
|
||||
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
||||
- `Cache-Control` header injection
|
||||
- Implements Tower `Layer` + `Service` traits for Axum integration
|
||||
- Per-request key: method + URI
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: File Content Cache (Download Tier 1)
|
||||
|
||||
**File**: `src/infrastructure/services/file_content_cache.rs`
|
||||
|
||||
In-memory LRU cache for small files, served directly from RAM.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 10 MB per file |
|
||||
| Max total cache size | 512 MB |
|
||||
| Max entries | 10,000 |
|
||||
| Structure | `lru::LruCache<String, CacheEntry>` |
|
||||
| Latency | ~0.1ms |
|
||||
|
||||
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
||||
|
||||
Methods:
|
||||
- `should_cache(size)` -- checks if file fits in cache
|
||||
- `get(file_id)` → `Option<(Bytes, String, String)>` -- returns (data, etag, content_type)
|
||||
- `put(file_id, content, etag, content_type)` -- inserts with LRU eviction
|
||||
- `invalidate(file_id)`, `clear()`
|
||||
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
||||
|
||||
Port: implements **ContentCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: MMAP (Download Tier 2)
|
||||
|
||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
Memory-mapped I/O for medium files using `memmap2`.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | 10 MB - 100 MB |
|
||||
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
||||
| Latency | ~1-5ms |
|
||||
|
||||
Current implementation copies mmap'd data to `Bytes` (`Bytes::copy_from_slice(&mmap[..])`). Not true zero-copy, but still benefits from kernel page cache.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Streaming (Download Tier 3)
|
||||
|
||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
Chunked streaming for large files using tokio-util codecs.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | ≥100 MB |
|
||||
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
||||
| Implementation | `FramedRead` + `BytesCodec` |
|
||||
| RAM usage | Near zero (one chunk at a time) |
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: File Metadata Cache
|
||||
|
||||
**File**: `src/infrastructure/services/file_metadata_cache.rs`
|
||||
|
||||
Caches filesystem metadata (existence, size, MIME type, timestamps) to avoid repeated `stat()` calls.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Default file TTL | 60 seconds |
|
||||
| Default directory TTL | 120 seconds |
|
||||
| Max entries | 10,000 |
|
||||
| Adaptive TTL multiplier | 5x for popular entries (≥10 accesses) |
|
||||
| LRU eviction | Frees 10% capacity when full |
|
||||
| Cleanup | Background task runs periodically |
|
||||
|
||||
**CachedMetadata:**
|
||||
```rust
|
||||
pub struct FileMetadata {
|
||||
pub path: PathBuf,
|
||||
pub exists: bool,
|
||||
pub entry_type: CacheEntryType, // File | Directory | Unknown
|
||||
pub size: Option<u64>,
|
||||
pub mime_type: Option<String>,
|
||||
pub created_at: Option<u64>,
|
||||
pub modified_at: Option<u64>,
|
||||
pub last_access: Instant,
|
||||
pub expires_at: Instant,
|
||||
pub access_count: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**Adaptive TTL**: entries accessed ≥10 times get 5x the configured TTL, keeping frequently accessed file metadata in cache longer.
|
||||
|
||||
Port: implements **MetadataCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 5: Write-Behind Cache
|
||||
|
||||
**File**: `src/infrastructure/services/write_behind_cache.rs`
|
||||
|
||||
Buffers small uploads in RAM and confirms immediately. Flushes to disk asynchronously.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 1 MB per file |
|
||||
| Max total cache | 100 MB |
|
||||
| Max pending duration | 30 seconds |
|
||||
| Flush interval | 100 ms |
|
||||
| Write strategy | Atomic (temp file + rename) |
|
||||
|
||||
Architecture:
|
||||
- `put_pending(file_id, content, target_path)` stores bytes in `HashMap<String, PendingWrite>`
|
||||
- Background `flush_worker` processes **FlushCommands** via `mpsc` channel
|
||||
- Periodic checker force-flushes entries older than 30 seconds
|
||||
- `get_pending(file_id)` serves reads while data is still in RAM (before flush)
|
||||
|
||||
Port: implements **WriteBehindCachePort** trait.
|
||||
|
||||
Statistics:
|
||||
```rust
|
||||
pub struct WriteBehindStatsDto {
|
||||
pub pending_count: usize,
|
||||
pub pending_bytes: usize,
|
||||
pub total_writes: u64,
|
||||
pub total_bytes_written: u64,
|
||||
pub cache_hits: u64,
|
||||
pub avg_flush_time_us: u64,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Layer 6: Buffer Pool
|
||||
|
||||
**File**: `src/infrastructure/services/buffer_pool.rs`
|
||||
|
||||
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Buffer size | 64 KB |
|
||||
| Max buffers | 100 |
|
||||
| Buffer TTL | 60 seconds |
|
||||
| Concurrency control | `tokio::sync::Semaphore` |
|
||||
|
||||
Features:
|
||||
- `get_buffer()` -- borrows a buffer (blocks if pool exhausted)
|
||||
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
||||
- Expired buffers are cleaned periodically via `start_cleaner()`
|
||||
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All cache-related config in `src/common/config.rs`:
|
||||
|
||||
```rust
|
||||
pub struct CacheConfig {
|
||||
pub file_ttl_ms: u64, // default: 60,000 (1 min)
|
||||
pub directory_ttl_ms: u64, // default: 120,000 (2 min)
|
||||
pub max_entries: usize, // default: 10,000
|
||||
}
|
||||
|
||||
pub struct ResourceConfig {
|
||||
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
||||
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
||||
pub max_in_memory_file_size_mb: u64, // 50 MB
|
||||
}
|
||||
```
|
||||
|
||||
## Download Flow
|
||||
|
||||
```
|
||||
Request → ETag check (304?) → Range request (206?)
|
||||
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
||||
→ file size < 100MB? → Tier 2: MMAP (kernel page cache)
|
||||
→ file size ≥ 100MB → Tier 3: Streaming (chunked)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Range Requests (HTTP 206 Partial Content)
|
||||
|
||||
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/file_fs_read_repository.rs`
|
||||
|
||||
**Crate**: `http-range-header = "0.4"` for parsing.
|
||||
|
||||
### Request Processing Flow
|
||||
|
||||
```
|
||||
Range header present?
|
||||
├─ parse_range_header(range_str)
|
||||
│ ├─ Parse OK → ranges.validate(file_size)
|
||||
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
||||
│ │ │ ├─ Stream OK → 206 Partial Content
|
||||
│ │ │ └─ Stream Err → fall through to normal download (200)
|
||||
│ │ └─ Invalid → 416 Range Not Satisfiable
|
||||
│ └─ Parse Err → fall through to normal download (200)
|
||||
└─ No Range header → normal 3-tier download
|
||||
```
|
||||
|
||||
### Response Headers (206)
|
||||
|
||||
| Header | Value |
|
||||
|---|---|
|
||||
| `Content-Type` | File MIME type |
|
||||
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
||||
| `Content-Length` | Range length (end - start + 1) |
|
||||
| `Accept-Ranges` | `bytes` |
|
||||
| `ETag` | `"{file_id}-{modified_at}"` |
|
||||
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
||||
|
||||
### 416 Range Not Satisfiable
|
||||
|
||||
Returned when `ranges.validate(file_size)` fails:
|
||||
```
|
||||
HTTP/1.1 416 Range Not Satisfiable
|
||||
Content-Range: bytes */12345
|
||||
```
|
||||
|
||||
### File Seek Implementation
|
||||
|
||||
`get_file_range_stream()` at the repository level:
|
||||
|
||||
```rust
|
||||
async fn get_file_range_stream(
|
||||
&self, id: &str, start: u64, end: Option<u64>,
|
||||
) -> Result<Box<dyn Stream<...> + Send>, DomainError>
|
||||
```
|
||||
|
||||
1. Opens the file with `TokioFile::open()`
|
||||
2. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
||||
3. Limits read to `range_length` via `fh.take(range_length)`
|
||||
4. Wraps in `FramedRead` + `BytesCodec`
|
||||
|
||||
Adaptive chunk size:
|
||||
|
||||
| Range size | Chunk size |
|
||||
|---|---|
|
||||
| ≤ 1 MB | 8 KB |
|
||||
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
||||
|
||||
### Tier Interaction
|
||||
|
||||
Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They always use direct file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Multipart ranges not supported**: only the first range in a multi-range request is served. Additional ranges are ignored.
|
||||
- **`If-Range` not handled**: no conditional range support.
|
||||
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
||||
|
||||
---
|
||||
|
||||
## Upload Flow
|
||||
|
||||
```
|
||||
Request → file size < 256KB? → Write-behind cache (instant 201, async flush)
|
||||
→ file size < 1MB? → Buffered write (sync)
|
||||
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to temp + rename)
|
||||
```
|
||||
|
||||
### Upload Strategy Selection
|
||||
|
||||
**File**: `src/application/services/file_upload_service.rs`
|
||||
|
||||
```rust
|
||||
pub enum UploadStrategy {
|
||||
WriteBehind, // < 256 KB — instant response, async disk write
|
||||
Buffered, // 256 KB – 1 MB — sync write to final path
|
||||
Streaming, // ≥ 1 MB — chunk-by-chunk write to temp file + rename
|
||||
}
|
||||
```
|
||||
|
||||
| Constant | Value |
|
||||
|---|---|
|
||||
| `WRITE_BEHIND_THRESHOLD` | 256 KB |
|
||||
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
||||
|
||||
### Handler-Level Buffering
|
||||
|
||||
Both upload handlers (`upload_file` and `upload_file_with_cache`) buffer the **entire multipart body in RAM** as `Vec<Bytes>` before calling the service layer:
|
||||
|
||||
```rust
|
||||
let mut chunks: Vec<Bytes> = Vec::new();
|
||||
while let Some(chunk) = field.chunk().await {
|
||||
chunks.push(chunk);
|
||||
}
|
||||
// All bytes are now in RAM
|
||||
upload_service.smart_upload(..., chunks, total_size).await
|
||||
```
|
||||
|
||||
The "streaming" in `UploadStrategy::Streaming` refers to the **service→repository** path, not the HTTP-body→disk path. By the time `save_file_from_stream()` is called, data is already in memory.
|
||||
|
||||
### Streaming Path (≥ 1 MB): Service → Repository
|
||||
|
||||
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
||||
|
||||
```rust
|
||||
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
||||
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
||||
```
|
||||
|
||||
**`save_file_from_stream()` implementation** (`file_fs_write_repository.rs`):
|
||||
|
||||
1. Resolves target path + generates unique name if collision
|
||||
2. Creates temp file: `{target_path}.tmp.upload`
|
||||
3. Iterates stream, writing each chunk with `fh.write_all(&chunk)`
|
||||
4. Calls `fh.flush()` + `fh.sync_all()` for durability
|
||||
5. Atomic rename: `fs::rename(temp_path, final_path)`
|
||||
6. Post-write: ID mapping, cache invalidation, metadata update
|
||||
|
||||
### Buffered Path (256 KB - 1 MB)
|
||||
|
||||
Uses `save_file()` -- writes all bytes directly to the **final path** (no temp file). For larger content, writes in chunks of **ResourceConfig.chunk_size_bytes** (1 MB).
|
||||
|
||||
### Write-Behind Path (< 256 KB)
|
||||
|
||||
See **Layer 5** above. Instant `201`, background flush within 30 seconds.
|
||||
|
||||
### Dedup Pre-Check
|
||||
|
||||
Runs for **all upload strategies** before writing. Re-combines all chunks into a single `Vec<u8>` for hash computation, which means data is temporarily duplicated in RAM during dedup processing.
|
||||
# 04 - Caching Architecture
|
||||
|
||||
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
||||
|
||||
## Cache Layers Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 2: MMAP (memmap2, 10-100MB blobs) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Streaming (FramedRead, ≥100MB blobs) │ Downloads
|
||||
├─────────────────────────────────────────────────────┤
|
||||
│ Layer 4: Buffer Pool (reusable I/O buffers) │ Compression
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
> **Note:** File metadata (name, size, MIME type, folder) is served from PostgreSQL — no separate filesystem metadata cache is needed.
|
||||
|
||||
---
|
||||
|
||||
## Layer 0: HTTP Cache Middleware
|
||||
|
||||
**File**: `src/interfaces/middleware/cache.rs`
|
||||
|
||||
Generic HTTP caching layer applied to API endpoints.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max entries | 1,000 |
|
||||
| Default max-age | 60 seconds |
|
||||
| Eviction | LRU (oldest 10% when full) |
|
||||
| Cleanup | Background task every 5 minutes |
|
||||
|
||||
Features:
|
||||
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
||||
- `Cache-Control` header injection
|
||||
- Implements Tower `Layer` + `Service` traits for Axum integration
|
||||
- Per-request key: method + URI
|
||||
|
||||
---
|
||||
|
||||
## Layer 1: File Content Cache (Download Tier 1)
|
||||
|
||||
**File**: `src/infrastructure/services/file_content_cache.rs`
|
||||
|
||||
In-memory LRU cache for small files, served directly from RAM.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Max file size | 10 MB per file |
|
||||
| Max total cache size | 512 MB |
|
||||
| Max entries | 10,000 |
|
||||
| Structure | `lru::LruCache<String, CacheEntry>` |
|
||||
| Latency | ~0.1ms |
|
||||
|
||||
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
||||
|
||||
Methods:
|
||||
- `should_cache(size)` — checks if file fits in cache
|
||||
- `get(file_id)` → `Option<(Bytes, String, String)>` — returns (data, etag, content_type)
|
||||
- `put(file_id, content, etag, content_type)` — inserts with LRU eviction
|
||||
- `invalidate(file_id)`, `clear()`
|
||||
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
||||
|
||||
Port: implements **ContentCachePort** trait.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2: MMAP (Download Tier 2)
|
||||
|
||||
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
Memory-mapped I/O for medium blobs using `memmap2`.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | 10 MB - 100 MB |
|
||||
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
||||
| Latency | ~1-5ms |
|
||||
|
||||
The blob file (`.blobs/{prefix}/{hash}.blob`) is memory-mapped and its contents copied to `Bytes`. Benefits from kernel page cache for frequently accessed blobs.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3: Streaming (Download Tier 3)
|
||||
|
||||
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
Chunked streaming for large blobs using tokio-util codecs.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| File range | ≥100 MB |
|
||||
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
||||
| Implementation | `FramedRead` + `BytesCodec` |
|
||||
| RAM usage | Near zero (one chunk at a time) |
|
||||
|
||||
---
|
||||
|
||||
## Layer 4: Buffer Pool
|
||||
|
||||
**File**: `src/infrastructure/services/buffer_pool.rs`
|
||||
|
||||
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Buffer size | 64 KB |
|
||||
| Max buffers | 100 |
|
||||
| Buffer TTL | 60 seconds |
|
||||
| Concurrency control | `tokio::sync::Semaphore` |
|
||||
|
||||
Features:
|
||||
- `get_buffer()` — borrows a buffer (blocks if pool exhausted)
|
||||
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
||||
- Expired buffers are cleaned periodically via `start_cleaner()`
|
||||
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All cache-related config in `src/common/config.rs`:
|
||||
|
||||
```rust
|
||||
pub struct ResourceConfig {
|
||||
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
||||
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
||||
pub max_in_memory_file_size_mb: u64, // 50 MB
|
||||
}
|
||||
```
|
||||
|
||||
## Download Flow
|
||||
|
||||
```
|
||||
Request → ETag check (304?) → Range request (206?)
|
||||
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
||||
→ file size < 100MB? → Tier 2: MMAP (kernel page cache on blob file)
|
||||
→ file size ≥ 100MB → Tier 3: Streaming (chunked from blob file)
|
||||
```
|
||||
|
||||
In all tiers, metadata (file name, size, MIME type) comes from a PostgreSQL `SELECT` on `storage.files`. Content is read from the DedupService blob at `.blobs/{prefix}/{hash}.blob`.
|
||||
|
||||
---
|
||||
|
||||
## Range Requests (HTTP 206 Partial Content)
|
||||
|
||||
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||
|
||||
**Crate**: `http-range-header = "0.4"` for parsing.
|
||||
|
||||
### Request Processing Flow
|
||||
|
||||
```
|
||||
Range header present?
|
||||
├─ parse_range_header(range_str)
|
||||
│ ├─ Parse OK → ranges.validate(file_size)
|
||||
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
||||
│ │ │ ├─ Stream OK → 206 Partial Content
|
||||
│ │ │ └─ Stream Err → fall through to normal download (200)
|
||||
│ │ └─ Invalid → 416 Range Not Satisfiable
|
||||
│ └─ Parse Err → fall through to normal download (200)
|
||||
└─ No Range header → normal 3-tier download
|
||||
```
|
||||
|
||||
### Response Headers (206)
|
||||
|
||||
| Header | Value |
|
||||
|---|---|
|
||||
| `Content-Type` | File MIME type |
|
||||
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
||||
| `Content-Length` | Range length (end - start + 1) |
|
||||
| `Accept-Ranges` | `bytes` |
|
||||
| `ETag` | `"{file_id}-{modified_at}"` |
|
||||
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
||||
|
||||
### 416 Range Not Satisfiable
|
||||
|
||||
Returned when `ranges.validate(file_size)` fails:
|
||||
```
|
||||
HTTP/1.1 416 Range Not Satisfiable
|
||||
Content-Range: bytes */12345
|
||||
```
|
||||
|
||||
### Blob File Seek Implementation
|
||||
|
||||
`get_file_range_stream()` at the repository level:
|
||||
|
||||
1. Resolves blob path from `blob_hash` via DedupService
|
||||
2. Opens the blob file with `TokioFile::open()`
|
||||
3. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
||||
4. Limits read to `range_length` via `fh.take(range_length)`
|
||||
5. Wraps in `FramedRead` + `BytesCodec`
|
||||
|
||||
Adaptive chunk size:
|
||||
|
||||
| Range size | Chunk size |
|
||||
|---|---|
|
||||
| ≤ 1 MB | 8 KB |
|
||||
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
||||
|
||||
### Tier Interaction
|
||||
|
||||
Range requests **bypass all download tiers** (LRU, MMAP). They always use direct blob file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Multipart ranges not supported**: only the first range in a multi-range request is served.
|
||||
- **`If-Range` not handled**: no conditional range support.
|
||||
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
||||
|
||||
---
|
||||
|
||||
## Upload Flow
|
||||
|
||||
```
|
||||
Request → file size < 1MB? → Buffered write (sync to blob store)
|
||||
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to blob store)
|
||||
```
|
||||
|
||||
### Upload Strategy Selection
|
||||
|
||||
**File**: `src/application/services/file_upload_service.rs`
|
||||
|
||||
```rust
|
||||
pub enum UploadStrategy {
|
||||
Buffered, // < 1 MB — collect bytes, write to blob store
|
||||
Streaming, // ≥ 1 MB — chunk-by-chunk write via save_file_from_stream
|
||||
}
|
||||
```
|
||||
|
||||
| Constant | Value |
|
||||
|---|---|
|
||||
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
||||
|
||||
### Handler-Level Buffering
|
||||
|
||||
Upload handlers buffer the multipart body in RAM as `Vec<Bytes>` before calling the service layer:
|
||||
|
||||
```rust
|
||||
let mut chunks: Vec<Bytes> = Vec::new();
|
||||
while let Some(chunk) = field.chunk().await {
|
||||
chunks.push(chunk);
|
||||
}
|
||||
upload_service.smart_upload(..., chunks, total_size).await
|
||||
```
|
||||
|
||||
### Buffered Path (< 1 MB)
|
||||
|
||||
Uses `save_file()` — all bytes are passed to `FileBlobWriteRepository`, which calls `DedupService.store_bytes()` to compute hash and store the blob, then INSERTs metadata into `storage.files`.
|
||||
|
||||
### Streaming Path (≥ 1 MB)
|
||||
|
||||
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
||||
|
||||
```rust
|
||||
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
||||
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
||||
```
|
||||
|
||||
`FileBlobWriteRepository.save_file_from_stream()` collects the stream, stores via DedupService, and INSERTs metadata.
|
||||
|
||||
### Dedup Integration
|
||||
|
||||
Deduplication is handled at the **repository layer** (not the service layer) for all upload strategies. `FileBlobWriteRepository` always calls `DedupService.store_bytes()` which:
|
||||
|
||||
1. Computes SHA-256 hash of content
|
||||
2. Checks if blob already exists (dedup hit → increment ref count, skip write)
|
||||
3. If new → atomic write to `.blobs/{prefix}/{hash}.blob`
|
||||
4. Returns the hash for storage in `storage.files.blob_hash`
|
||||
|
||||
Reference in New Issue
Block a user