Files
Oxicloud/src/application/ports/file_ports.rs
T
Claude e3f04d58aa Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 13:06:33 +00:00

313 lines
12 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
use bytes::Bytes;
use futures::Stream;
use std::pin::Pin;
use std::sync::Arc;
use uuid::Uuid;
use crate::application::dtos::file_dto::FileDto;
use crate::application::ports::storage_ports::CopyFolderTreeResult;
use crate::application::services::file_management_service::FileManagementService;
use crate::application::services::file_retrieval_service::FileRetrievalService;
use crate::application::services::file_upload_service::FileUploadService;
use crate::common::errors::DomainError;
use crate::domain::services::authorization::Permission;
// ─────────────────────────────────────────────────────
// Upload port
// ─────────────────────────────────────────────────────
/// A blob already stored in the content-addressable chunk store, carrying
/// exactly ONE reference that the receiving method takes ownership of.
///
/// Produced by the upload-ingest layer (`interfaces::upload_ingest`), which
/// streams the request body straight into the CDC dedup store. Methods that
/// accept a `StoredBlob` either attach the reference to a file row or
/// release it on failure — callers never need to compensate themselves.
#[derive(Debug, Clone)]
pub struct StoredBlob {
/// BLAKE3 of the full content (manifest / blob key).
pub hash: String,
/// Content size in bytes.
pub size: u64,
/// `false` when the content already existed (dedup hit) — forwarded to
/// lifecycle hooks so e.g. thumbnails aren't regenerated for known blobs.
pub is_new_blob: bool,
}
/// Primary port for file upload operations.
///
/// **All upload paths converge on streaming-into-the-chunk-store** — content
/// never passes through this port; it is ingested by the interface layer
/// (CDC chunking + hashing while the body arrives, no spool file) and only
/// the resulting [`StoredBlob`] reference travels through here.
pub trait FileUploadUseCase: Send + Sync + 'static {
/// Register a new file row pointing at an already-ingested blob.
///
/// Takes ownership of the blob's reference (released on failure).
async fn upload_file_streaming(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
blob: StoredBlob,
) -> Result<FileDto, DomainError>;
/// Replace the content of the file at `path` with an already-ingested
/// blob, or create the file when it doesn't exist (WebDAV/WOPI PUT).
///
/// Takes ownership of the blob's reference (released on failure).
async fn update_file_streaming(
&self,
path: &str,
blob: StoredBlob,
content_type: &str,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError>;
}
// ─────────────────────────────────────────────────────
// Retrieval / download port
// ─────────────────────────────────────────────────────
/// Optimized file content returned by the retrieval service.
///
/// The handler only needs to map each variant to the appropriate HTTP
/// response; all caching / transcoding / mmap decisions happen in the
/// application layer.
pub enum OptimizedFileContent {
/// Small-file content (possibly transcoded / compressed) already in RAM.
Bytes {
data: Bytes,
mime_type: Arc<str>,
was_transcoded: bool,
},
/// Memory-mapped file (10–100 MB).
Mmap(Bytes),
/// Streaming download for very large files (≥100 MB).
Stream(Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>),
}
/// Primary port for file retrieval operations
pub trait FileRetrievalUseCase: Send + Sync + 'static {
/// Gets a file by its ID (system/internal — no ownership check).
async fn get_file(&self, id: &str) -> Result<FileDto, DomainError>;
/// Gets a file by its ID, enforcing that `caller_id` is the owner.
///
/// Returns `NotFound` if the file does not exist **or** belongs to
/// another user. All user-facing handlers should use this method.
async fn get_file_with_perms(&self, id: &str, caller_id: Uuid) -> Result<FileDto, DomainError>;
async fn get_file_or_trashed_with_perms(
&self,
id: &str,
caller_id: Uuid,
) -> Result<FileDto, DomainError>;
/// Gets a file by its path (for WebDAV)
async fn get_file_by_path(&self, path: &str) -> Result<FileDto, DomainError>;
/// Lists files in a folder
async fn list_files(&self, folder_id: Option<&str>) -> Result<Vec<FileDto>, DomainError>;
/// Lists files in a folder, scoped to the authenticated user.
///
/// Uses SQL-level `AND user_id` filtering — no in-memory post-filter.
/// All user-facing list handlers should use this method.
async fn list_files_with_perms(
&self,
folder_id: Option<&str>,
owner_id: Uuid,
) -> Result<Vec<FileDto>, DomainError>;
/// Gets file content as a stream (for large files)
async fn get_file_stream(
&self,
id: &str,
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
/// Gets file content as a stream, enforcing that `caller_id` is the owner.
async fn get_file_stream_with_perms(
&self,
id: &str,
caller_id: Uuid,
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
/// Optimized multi-tier download.
///
/// Internalises: write-behind lookup → content-cache → WebP transcode →
/// mmap → streaming, returning an `OptimizedFileContent` variant so the
/// handler only builds the HTTP response.
async fn get_file_optimized(
&self,
id: &str,
accept_webp: bool,
prefer_original: bool,
) -> Result<(FileDto, OptimizedFileContent), DomainError>;
/// Ownership-scoped optimized download.
///
/// Verifies `caller_id` owns the file before returning content.
/// All user-facing download handlers should use this.
async fn get_file_optimized_with_perms(
&self,
id: &str,
caller_id: Uuid,
accept_webp: bool,
prefer_original: bool,
) -> Result<(FileDto, OptimizedFileContent), DomainError>;
/// Like `get_file_optimized` but accepts an already-fetched `FileDto`,
/// avoiding a redundant metadata query when the handler already has it.
async fn get_file_optimized_preloaded(
&self,
id: &str,
file_dto: FileDto,
accept_webp: bool,
prefer_original: bool,
) -> Result<(FileDto, OptimizedFileContent), DomainError> {
// Default: ignore pre-fetched meta, re-fetch everything.
let _ = file_dto;
self.get_file_optimized(id, accept_webp, prefer_original)
.await
}
/// Range-based streaming for HTTP Range Requests (video seek, resumable DL).
async fn get_file_range_stream(
&self,
id: &str,
start: u64,
end: Option<u64>,
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
/// Ownership-scoped range stream — verifies caller owns the file first.
async fn get_file_range_stream_with_perms(
&self,
id: &str,
caller_id: Uuid,
start: u64,
end: Option<u64>,
) -> Result<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>, DomainError>;
/// Streams every file in the subtree rooted at `folder_id`.
///
/// Returns a streaming cursor — RAM stays O(1) per row. Callers
/// consume incrementally (e.g. group into a HashMap by folder_id)
/// without materializing the full result set.
async fn stream_files_in_subtree(
&self,
folder_id: &str,
) -> Result<Pin<Box<dyn Stream<Item = Result<FileDto, DomainError>> + Send>>, DomainError>;
/// Lists files in a folder with LIMIT/OFFSET pagination.
///
/// Used by streaming WebDAV PROPFIND to avoid loading all files at once.
/// Default: falls back to `list_files` (loads all, then slices in memory).
async fn list_files_batch(
&self,
folder_id: Option<&str>,
offset: i64,
limit: i64,
) -> Result<Vec<FileDto>, DomainError> {
let all = self.list_files(folder_id).await?;
Ok(all
.into_iter()
.skip(offset as usize)
.take(limit as usize)
.collect())
}
/// Like [`list_files_batch`], but scoped to a specific owner.
///
/// Used by streaming WebDAV PROPFIND so that each user only sees their
/// own files, even in shared folder_id namespaces.
async fn list_files_batch_with_perms(
&self,
folder_id: Option<&str>,
owner_id: Uuid,
offset: i64,
limit: i64,
) -> Result<Vec<FileDto>, DomainError> {
let all = self.list_files_batch(folder_id, offset, limit).await?;
let owner_str = owner_id.to_string();
Ok(all
.into_iter()
.filter(|f| f.owner_id.as_deref().is_some_and(|o| o == owner_str))
.collect())
}
}
/// Primary port for file management operations
pub trait FileManagementUseCase: Send + Sync + 'static {
async fn require_permission(
&self,
caller_id: Uuid,
permission: Permission,
file_id: &str,
) -> Result<(), DomainError>;
/// Moves a file, enforcing that `caller_id` is the owner.
async fn move_file_with_perms(
&self,
file_id: &str,
caller_id: Uuid,
folder_id: Option<String>,
) -> Result<FileDto, DomainError>;
/// Copies a file, enforcing that `caller_id` is the owner.
async fn copy_file_with_perms(
&self,
file_id: &str,
caller_id: Uuid,
target_folder_id: Option<String>,
) -> Result<FileDto, DomainError>;
/// Renames a file, enforcing that `caller_id` is the owner.
async fn rename_file_with_perms(
&self,
file_id: &str,
caller_id: Uuid,
new_name: &str,
) -> Result<FileDto, DomainError>;
/// Deletes a file, enforcing that `caller_id` is the owner.
async fn delete_file_with_perms(&self, id: &str, caller_id: Uuid) -> Result<(), DomainError>;
/// Smart delete: trash-first with dedup reference cleanup.
///
/// 1. Tries to move to trash (soft delete).
/// 2. Falls back to permanent delete if trash unavailable/failed.
/// 3. Decrements the dedup reference count for the content hash.
///
/// Returns `Ok(true)` when trashed, `Ok(false)` when permanently deleted.
async fn delete_and_cleanup_with_perms(
&self,
id: &str,
user_id: Uuid,
) -> Result<bool, DomainError>;
/// Copies an entire folder subtree atomically (WebDAV COPY Depth: infinity).
/// enforcing that `caller_id` owns both the source folder
/// and the target parent folder.
///
/// Creates a copy of `source_folder_id` (with optional name override) under
/// `target_parent_id`, including ALL sub-folders and files. Files are
/// zero-copy (blob ref_counts incremented in batch).
///
/// Default: returns error (only available with PostgreSQL backend).
async fn copy_folder_tree_with_perms(
&self,
source_folder_id: &str,
caller_id: Uuid,
target_parent_id: Option<String>,
dest_name: Option<String>,
) -> Result<CopyFolderTreeResult, DomainError>;
}
/// Factory for creating file use case implementations
pub trait FileUseCaseFactory: Send + Sync + 'static {
fn create_file_upload_use_case(&self) -> Arc<FileUploadService>;
fn create_file_retrieval_use_case(&self) -> Arc<FileRetrievalService>;
fn create_file_management_use_case(&self) -> Arc<FileManagementService>;
}