e3f04d58aa
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
313 lines
12 KiB
Rust
313 lines
12 KiB
Rust
use bytes::Bytes;
|
||
use futures::Stream;
|
||
use std::pin::Pin;
|
||
use std::sync::Arc;
|
||
use uuid::Uuid;
|
||
|
||
use crate::application::dtos::file_dto::FileDto;
|
||
use crate::application::ports::storage_ports::CopyFolderTreeResult;
|
||
use crate::application::services::file_management_service::FileManagementService;
|
||
use crate::application::services::file_retrieval_service::FileRetrievalService;
|
||
use crate::application::services::file_upload_service::FileUploadService;
|
||
use crate::common::errors::DomainError;
|
||
use crate::domain::services::authorization::Permission;
|
||
|
||
// ─────────────────────────────────────────────────────
|
||
// Upload port
|
||
// ─────────────────────────────────────────────────────
|
||
|
||
/// A blob already stored in the content-addressable chunk store, carrying
|
||
/// exactly ONE reference that the receiving method takes ownership of.
|
||
///
|
||
/// Produced by the upload-ingest layer (`interfaces::upload_ingest`), which
|
||
/// streams the request body straight into the CDC dedup store. Methods that
|
||
/// accept a `StoredBlob` either attach the reference to a file row or
|
||
/// release it on failure — callers never need to compensate themselves.
|
||
#[derive(Debug, Clone)]
|
||
pub struct StoredBlob {
|
||
/// BLAKE3 of the full content (manifest / blob key).
|
||
pub hash: String,
|
||
/// Content size in bytes.
|
||
pub size: u64,
|
||
/// `false` when the content already existed (dedup hit) — forwarded to
|
||
/// lifecycle hooks so e.g. thumbnails aren't regenerated for known blobs.
|
||
pub is_new_blob: bool,
|
||
}
|
||
|
||
/// Primary port for file upload operations.
|
||
///
|
||
/// **All upload paths converge on streaming-into-the-chunk-store** — content
|
||
/// never passes through this port; it is ingested by the interface layer
|
||
/// (CDC chunking + hashing while the body arrives, no spool file) and only
|
||
/// the resulting [`StoredBlob`] reference travels through here.
|
||
pub trait FileUploadUseCase: Send + Sync + 'static {
|
||
/// Register a new file row pointing at an already-ingested blob.
|
||
///
|
||
/// Takes ownership of the blob's reference (released on failure).
|
||
async fn upload_file_streaming(
|
||
&self,
|
||
name: String,
|
||
folder_id: Option<String>,
|
||
content_type: String,
|
||
blob: StoredBlob,
|
||
) -> Result<FileDto, DomainError>;
|
||
|
||
/// Replace the content of the file at `path` with an already-ingested
|
||
/// blob, or create the file when it doesn't exist (WebDAV/WOPI PUT).
|
||
///
|
||
/// Takes ownership of the blob's reference (released on failure).
|
||
async fn update_file_streaming(
|
||
&self,
|
||
path: &str,
|
||
blob: StoredBlob,
|
||
content_type: &str,
|
||
modified_at: Option<i64>,
|
||
) -> Result<FileDto, DomainError>;
|
||
}
|
||
|
||
// ─────────────────────────────────────────────────────
|
||
// Retrieval / download port
|
||
// ─────────────────────────────────────────────────────
|
||
|
||
/// Optimized file content returned by the retrieval service.
|
||
///
|
||
/// The handler only needs to map each variant to the appropriate HTTP
|
||
/// response; all caching / transcoding / mmap decisions happen in the
|
||
/// application layer.
|
||
pub enum OptimizedFileContent {
|
||
/// Small-file content (possibly transcoded / compressed) already in RAM.
|
||
Bytes {
|
||
data: Bytes,
|
||
mime_type: Arc<str>,
|
||
was_transcoded: bool,
|
||
},
|
||
/// Memory-mapped file (10–100 MB).
|
||
Mmap(Bytes),
|
||
/// Streaming download for very large files (≥100 MB).
|
||
Stream(Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>),
|
||
}
|
||
|
||
/// Primary port for file retrieval operations
|
||
pub trait FileRetrievalUseCase: Send + Sync + 'static {
|
||
/// Gets a file by its ID (system/internal — no ownership check).
|
||
async fn get_file(&self, id: &str) -> Result<FileDto, DomainError>;
|
||
|
||
/// Gets a file by its ID, enforcing that `caller_id` is the owner.
|
||
///
|
||
/// Returns `NotFound` if the file does not exist **or** belongs to
|
||
/// another user. All user-facing handlers should use this method.
|
||
async fn get_file_with_perms(&self, id: &str, caller_id: Uuid) -> Result<FileDto, DomainError>;
|
||
|
||
async fn get_file_or_trashed_with_perms(
|
||
&self,
|
||
id: &str,
|
||
caller_id: Uuid,
|
||
) -> Result<FileDto, DomainError>;
|
||
|
||
/// Gets a file by its path (for WebDAV)
|
||
async fn get_file_by_path(&self, path: &str) -> Result<FileDto, DomainError>;
|
||
|
||
/// Lists files in a folder
|
||
async fn list_files(&self, folder_id: Option<&str>) -> Result<Vec<FileDto>, DomainError>;
|
||
|
||
/// Lists files in a folder, scoped to the authenticated user.
|
||
///
|
||
/// Uses SQL-level `AND user_id` filtering — no in-memory post-filter.
|
||
/// All user-facing list handlers should use this method.
|
||
async fn list_files_with_perms(
|
||
&self,
|
||
folder_id: Option<&str>,
|
||
owner_id: Uuid,
|
||
) -> Result<Vec<FileDto>, DomainError>;
|
||
|
||
/// Gets file content as a stream (for large files)
|
||
async fn get_file_stream(
|
||
&self,
|
||
id: &str,
|
||
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
|
||
|
||
/// Gets file content as a stream, enforcing that `caller_id` is the owner.
|
||
async fn get_file_stream_with_perms(
|
||
&self,
|
||
id: &str,
|
||
caller_id: Uuid,
|
||
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
|
||
|
||
/// Optimized multi-tier download.
|
||
///
|
||
/// Internalises: write-behind lookup → content-cache → WebP transcode →
|
||
/// mmap → streaming, returning an `OptimizedFileContent` variant so the
|
||
/// handler only builds the HTTP response.
|
||
async fn get_file_optimized(
|
||
&self,
|
||
id: &str,
|
||
accept_webp: bool,
|
||
prefer_original: bool,
|
||
) -> Result<(FileDto, OptimizedFileContent), DomainError>;
|
||
|
||
/// Ownership-scoped optimized download.
|
||
///
|
||
/// Verifies `caller_id` owns the file before returning content.
|
||
/// All user-facing download handlers should use this.
|
||
async fn get_file_optimized_with_perms(
|
||
&self,
|
||
id: &str,
|
||
caller_id: Uuid,
|
||
accept_webp: bool,
|
||
prefer_original: bool,
|
||
) -> Result<(FileDto, OptimizedFileContent), DomainError>;
|
||
|
||
/// Like `get_file_optimized` but accepts an already-fetched `FileDto`,
|
||
/// avoiding a redundant metadata query when the handler already has it.
|
||
async fn get_file_optimized_preloaded(
|
||
&self,
|
||
id: &str,
|
||
file_dto: FileDto,
|
||
accept_webp: bool,
|
||
prefer_original: bool,
|
||
) -> Result<(FileDto, OptimizedFileContent), DomainError> {
|
||
// Default: ignore pre-fetched meta, re-fetch everything.
|
||
let _ = file_dto;
|
||
self.get_file_optimized(id, accept_webp, prefer_original)
|
||
.await
|
||
}
|
||
|
||
/// Range-based streaming for HTTP Range Requests (video seek, resumable DL).
|
||
async fn get_file_range_stream(
|
||
&self,
|
||
id: &str,
|
||
start: u64,
|
||
end: Option<u64>,
|
||
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
|
||
|
||
/// Ownership-scoped range stream — verifies caller owns the file first.
|
||
async fn get_file_range_stream_with_perms(
|
||
&self,
|
||
id: &str,
|
||
caller_id: Uuid,
|
||
start: u64,
|
||
end: Option<u64>,
|
||
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
|
||
|
||
/// Streams every file in the subtree rooted at `folder_id`.
|
||
///
|
||
/// Returns a streaming cursor — RAM stays O(1) per row. Callers
|
||
/// consume incrementally (e.g. group into a HashMap by folder_id)
|
||
/// without materializing the full result set.
|
||
async fn stream_files_in_subtree(
|
||
&self,
|
||
folder_id: &str,
|
||
) -> Result<Pin<Box<dyn Stream<Item = Result<FileDto, DomainError>> + Send>>, DomainError>;
|
||
|
||
/// Lists files in a folder with LIMIT/OFFSET pagination.
|
||
///
|
||
/// Used by streaming WebDAV PROPFIND to avoid loading all files at once.
|
||
/// Default: falls back to `list_files` (loads all, then slices in memory).
|
||
async fn list_files_batch(
|
||
&self,
|
||
folder_id: Option<&str>,
|
||
offset: i64,
|
||
limit: i64,
|
||
) -> Result<Vec<FileDto>, DomainError> {
|
||
let all = self.list_files(folder_id).await?;
|
||
Ok(all
|
||
.into_iter()
|
||
.skip(offset as usize)
|
||
.take(limit as usize)
|
||
.collect())
|
||
}
|
||
|
||
/// Like [`list_files_batch`], but scoped to a specific owner.
|
||
///
|
||
/// Used by streaming WebDAV PROPFIND so that each user only sees their
|
||
/// own files, even in shared folder_id namespaces.
|
||
async fn list_files_batch_with_perms(
|
||
&self,
|
||
folder_id: Option<&str>,
|
||
owner_id: Uuid,
|
||
offset: i64,
|
||
limit: i64,
|
||
) -> Result<Vec<FileDto>, DomainError> {
|
||
let all = self.list_files_batch(folder_id, offset, limit).await?;
|
||
let owner_str = owner_id.to_string();
|
||
Ok(all
|
||
.into_iter()
|
||
.filter(|f| f.owner_id.as_deref().is_some_and(|o| o == owner_str))
|
||
.collect())
|
||
}
|
||
}
|
||
|
||
/// Primary port for file management operations
|
||
pub trait FileManagementUseCase: Send + Sync + 'static {
|
||
async fn require_permission(
|
||
&self,
|
||
caller_id: Uuid,
|
||
permission: Permission,
|
||
file_id: &str,
|
||
) -> Result<(), DomainError>;
|
||
|
||
/// Moves a file, enforcing that `caller_id` is the owner.
|
||
async fn move_file_with_perms(
|
||
&self,
|
||
file_id: &str,
|
||
caller_id: Uuid,
|
||
folder_id: Option<String>,
|
||
) -> Result<FileDto, DomainError>;
|
||
|
||
/// Copies a file, enforcing that `caller_id` is the owner.
|
||
async fn copy_file_with_perms(
|
||
&self,
|
||
file_id: &str,
|
||
caller_id: Uuid,
|
||
target_folder_id: Option<String>,
|
||
) -> Result<FileDto, DomainError>;
|
||
|
||
/// Renames a file, enforcing that `caller_id` is the owner.
|
||
async fn rename_file_with_perms(
|
||
&self,
|
||
file_id: &str,
|
||
caller_id: Uuid,
|
||
new_name: &str,
|
||
) -> Result<FileDto, DomainError>;
|
||
|
||
/// Deletes a file, enforcing that `caller_id` is the owner.
|
||
async fn delete_file_with_perms(&self, id: &str, caller_id: Uuid) -> Result<(), DomainError>;
|
||
|
||
/// Smart delete: trash-first with dedup reference cleanup.
|
||
///
|
||
/// 1. Tries to move to trash (soft delete).
|
||
/// 2. Falls back to permanent delete if trash unavailable/failed.
|
||
/// 3. Decrements the dedup reference count for the content hash.
|
||
///
|
||
/// Returns `Ok(true)` when trashed, `Ok(false)` when permanently deleted.
|
||
async fn delete_and_cleanup_with_perms(
|
||
&self,
|
||
id: &str,
|
||
user_id: Uuid,
|
||
) -> Result<bool, DomainError>;
|
||
|
||
/// Copies an entire folder subtree atomically (WebDAV COPY Depth: infinity).
|
||
/// enforcing that `caller_id` owns both the source folder
|
||
/// and the target parent folder.
|
||
///
|
||
/// Creates a copy of `source_folder_id` (with optional name override) under
|
||
/// `target_parent_id`, including ALL sub-folders and files. Files are
|
||
/// zero-copy (blob ref_counts incremented in batch).
|
||
///
|
||
/// Default: returns error (only available with PostgreSQL backend).
|
||
async fn copy_folder_tree_with_perms(
|
||
&self,
|
||
source_folder_id: &str,
|
||
caller_id: Uuid,
|
||
target_parent_id: Option<String>,
|
||
dest_name: Option<String>,
|
||
) -> Result<CopyFolderTreeResult, DomainError>;
|
||
}
|
||
|
||
/// Factory for creating file use case implementations
|
||
pub trait FileUseCaseFactory: Send + Sync + 'static {
|
||
fn create_file_upload_use_case(&self) -> Arc<FileUploadService>;
|
||
fn create_file_retrieval_use_case(&self) -> Arc<FileRetrievalService>;
|
||
fn create_file_management_use_case(&self) -> Arc<FileManagementService>;
|
||
}
|