cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Stream uploads directly into the CDC chunk store (no spool, single write)

Browse Source 
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/application/ports/chunked_upload_ports.rs
+19 -5
View File
@@ -98,6 +98,21 @@ pub struct UploadStatusResponseDto {
pub is_complete: bool,
}
/// A completed upload session, ready to be streamed into the blob store.
///
/// `chunk_paths` lists every chunk file in assembly order — the caller
/// concatenates them as one byte stream (CDC chunking + hashing happen in
/// that single pass). The files stay on disk until `finalize_upload`, so a
/// failed completion (e.g. checksum mismatch) remains retryable.
#[derive(Debug, Clone)]
pub struct CompletedUploadParts {
pub chunk_paths: Vec<PathBuf>,
pub filename: String,
pub folder_id: Option<String>,
pub content_type: String,
pub total_size: u64,
}
/// Port for chunked/resumable upload operations.
///
/// Implementations manage upload sessions, chunk storage, reassembly,
@@ -137,16 +152,15 @@ pub trait ChunkedUploadPort: Send + Sync + 'static {
user_id: Uuid,
) -> Result<UploadStatusResponseDto, DomainError>;
/// Assemble all chunks into the final file.
/// Validate completion and hand back the ordered chunk parts.
///
/// Returns `(assembled_file_path, filename, folder_id, content_type, total_size, blake3_hash)`.
/// The hash is computed during assembly (hash-on-write), eliminating a
/// second sequential read of the assembled file.
/// No assembled file is written — the caller streams the parts straight
/// into the content-addressable store.
async fn complete_upload(
&self,
upload_id: &str,
user_id: Uuid,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), DomainError>;
) -> Result<CompletedUploadParts, DomainError>;
/// Finalize upload: clean up the session and temporary files.
async fn finalize_upload(&self, upload_id: &str, user_id: Uuid) -> Result<(), DomainError>;
src/application/ports/dedup_ports.rs
-12
View File
@@ -80,18 +80,6 @@ pub struct DedupStatsDto {
/// duplicate storage automatically. Multiple file references can
/// point to the same physical blob.
pub trait DedupPort: Send + Sync + 'static {
/// Store content with deduplication (streaming from file).
///
/// If `pre_computed_hash` is provided (e.g. hash-on-write from the handler),
/// the file will NOT be re-read to calculate the hash — saving one full
/// sequential read of the file.
async fn store_from_file(
&self,
source_path: &Path,
content_type: Option<String>,
pre_computed_hash: Option<String>,
) -> Result<DedupResultDto, DomainError>;
/// Check if a blob with the given hash exists.
async fn blob_exists(&self, hash: &str) -> bool;
src/application/ports/file_ports.rs
+29 -53
View File
@@ -1,6 +1,5 @@
use bytes::Bytes;
use futures::Stream;
use std::path::Path;
use std::pin::Pin;
use std::sync::Arc;
use uuid::Uuid;
@@ -17,74 +16,51 @@ use crate::domain::services::authorization::Permission;
// Upload port
// ─────────────────────────────────────────────────────
/// A blob already stored in the content-addressable chunk store, carrying
/// exactly ONE reference that the receiving method takes ownership of.
///
/// Produced by the upload-ingest layer (`interfaces::upload_ingest`), which
/// streams the request body straight into the CDC dedup store. Methods that
/// accept a `StoredBlob` either attach the reference to a file row or
/// release it on failure — callers never need to compensate themselves.
#[derive(Debug, Clone)]
pub struct StoredBlob {
/// BLAKE3 of the full content (manifest / blob key).
pub hash: String,
/// Content size in bytes.
pub size: u64,
/// `false` when the content already existed (dedup hit) — forwarded to
/// lifecycle hooks so e.g. thumbnails aren't regenerated for known blobs.
pub is_new_blob: bool,
}
/// Primary port for file upload operations.
///
/// **All upload paths converge on streaming-to-disk** — no method accepts
/// `Vec<u8>` for content. Even `create_file` / `update_file` (WebDAV
/// helpers that receive `&[u8]`) spool to a temp file internally so that
/// peak RAM stays at ~256 KB regardless of file size.
///
/// - Normal uploads: handler spools multipart to temp file → `upload_file_streaming`
/// - Chunked uploads: chunks already on disk → `upload_file_from_path`
/// - WebDAV PUT (new): handler streams to temp file → `update_file_streaming`
/// - WebDAV PUT (small/compat): `create_file` / `update_file` spool internally
/// **All upload paths converge on streaming-into-the-chunk-store** — content
/// never passes through this port; it is ingested by the interface layer
/// (CDC chunking + hashing while the body arrives, no spool file) and only
/// the resulting [`StoredBlob`] reference travels through here.
pub trait FileUploadUseCase: Send + Sync + 'static {
/// Upload from a temp file already on disk (true streaming, ~256 KB RAM).
/// Register a new file row pointing at an already-ingested blob.
///
/// When `pre_computed_hash` is `Some`, the blob store skips the hash
/// re-read — the handler already computed it during the multipart spool.
/// Takes ownership of the blob's reference (released on failure).
async fn upload_file_streaming(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
temp_path: &Path,
size: u64,
pre_computed_hash: Option<String>,
blob: StoredBlob,
) -> Result<FileDto, DomainError>;
/// Upload from a file already assembled on disk (chunked uploads).
/// Replace the content of the file at `path` with an already-ingested
/// blob, or create the file when it doesn't exist (WebDAV/WOPI PUT).
///
/// Same as `upload_file_streaming` but with a separate name for clarity.
async fn upload_file_from_path(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
file_path: &Path,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError>;
/// Creates a new file at the specified path (for WebDAV)
async fn create_file(
&self,
parent_path: &str,
filename: &str,
content: &[u8],
content_type: &str,
) -> Result<FileDto, DomainError>;
/// Updates the content of an existing file (for WebDAV)
async fn update_file(
&self,
path: &str,
content: &[u8],
content_type: &str,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError>;
/// Streaming update — spools body to a temp file with incremental hash,
/// then atomically replaces the file content via dedup store.
///
/// Peak RAM: ~256 KB regardless of file size.
/// Used by WebDAV PUT for large files.
/// Takes ownership of the blob's reference (released on failure).
async fn update_file_streaming(
&self,
path: &str,
temp_path: &Path,
size: u64,
blob: StoredBlob,
content_type: &str,
pre_computed_hash: Option<String>,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError>;
}
src/application/ports/storage_ports.rs
+14 -16
View File
@@ -251,21 +251,21 @@ pub struct CopyFolderTreeResult {
/// Secondary port for file **writing**.
///
/// Covers: upload (buffered + streaming), move, delete, update,
/// and deferred registration for the write-behind cache.
/// Covers: upload registration, move, delete, update, and deferred
/// registration for the write-behind cache.
pub trait FileWritePort: Send + Sync + 'static {
/// Streaming upload — saves a file from a temp file already on disk.
/// Register a file row pointing at a blob already stored in the
/// content-addressable chunk store.
///
/// When `pre_computed_hash` is provided, the dedup service skips the
/// hash re-read — zero extra I/O beyond the initial spool.
async fn save_file_from_temp(
/// Takes ownership of one blob reference: on any failure the reference
/// is released before the error is returned.
async fn save_file_with_blob(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
temp_path: &std::path::Path,
blob_hash: &str,
size: u64,
pre_computed_hash: Option<String>,
) -> Result<File, DomainError>;
/// Moves a file to another folder.
@@ -281,22 +281,20 @@ pub trait FileWritePort: Send + Sync + 'static {
/// Deletes a file.
async fn delete_file(&self, id: &str) -> Result<(), DomainError>;
/// Streaming update — replaces file content from a temp file on disk.
/// Atomically swap a file's content to a blob already stored in the
/// content-addressable chunk store.
///
/// When `pre_computed_hash` is provided, the dedup service skips the
/// hash re-read — zero extra I/O beyond the initial spool.
/// Peak RAM: ~256 KB regardless of file size.
/// Takes ownership of one blob reference (released on failure); the
/// previous content's reference is dropped after the swap.
///
/// Returns `(new_blob_hash, updated_at_epoch)` — everything a caller
/// needs to rebuild the fresh entity/ETag from a `File` it already
/// holds, without re-reading the row it just updated.
async fn update_file_content_from_temp(
async fn update_file_content_with_blob(
&self,
file_id: &str,
temp_path: &std::path::Path,
blob_hash: &str,
size: u64,
content_type: Option<String>,
pre_computed_hash: Option<String>,
modified_at: Option<i64>,
) -> Result<(String, i64), DomainError>;
src/application/services/file_upload_service.rs
+25 -189
View File
@@ -1,15 +1,13 @@
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::application::dtos::file_dto::FileDto;
use crate::application::ports::file_lifecycle::FileLifecycleHook;
use crate::application::ports::file_ports::FileUploadUseCase;
use crate::application::ports::file_ports::{FileUploadUseCase, StoredBlob};
use crate::application::ports::storage_ports::{FileReadPort, FileWritePort};
use crate::application::services::storage_usage_service::StorageUsageService;
use crate::common::errors::DomainError;
use crate::infrastructure::repositories::pg::FileBlobReadRepository;
use crate::infrastructure::repositories::pg::FileBlobWriteRepository;
use crate::infrastructure::services::dedup_service::DedupService;
use crate::infrastructure::services::file_content_cache::FileContentCache;
use tracing::{debug, info, warn};
@@ -34,20 +32,16 @@ fn extract_username_from_path(path: &str) -> Option<String> {
/// Service for file upload operations.
///
/// **Every upload path converges on streaming-to-disk** — there is no
/// `Vec<u8>` buffer path.
///
/// - **Normal uploads**: handler spools multipart to temp file → `upload_file_streaming`
/// - **Chunked uploads**: chunks already on disk → `upload_file_from_path`
/// - **WebDAV PUT (large)**: handler streams body to temp file → `update_file_streaming`
/// - **WebDAV PUT (small / compat)**: `create_file` / `update_file` spool `&[u8]`
/// to a temp file internally, then call the streaming path.
///
/// Peak RAM usage during any upload: ~256 KB (streaming hash) regardless of file size.
/// Content never passes through this service: the interface layer streams
/// the request body straight into the CDC chunk store (no spool file, no
/// full-body buffering) and hands over a [`StoredBlob`] reference. This
/// service registers the metadata row, keeps caches coherent and fires
/// lifecycle hooks. Blob-reference ownership is consumed by the write
/// port, which releases it on failure — callers never compensate.
pub struct FileUploadService {
/// Write port — handles save, streaming, deferred registration
/// Write port — registers file rows against ingested blobs.
file_write: Arc<FileBlobWriteRepository>,
/// Read port — needed for WebDAV create_file / update_file
/// Read port — needed for WebDAV/WOPI update-by-path.
file_read: Option<Arc<FileBlobReadRepository>>,
/// Optional storage usage tracking
storage_usage_service: Option<Arc<StorageUsageService>>,
@@ -55,9 +49,6 @@ pub struct FileUploadService {
content_cache: Option<Arc<FileContentCache>>,
/// Single lifecycle dispatcher — fires on_file_created / on_file_updated.
file_lifecycle_hook: Option<Arc<dyn FileLifecycleHook>>,
/// Directory for spool temp files (`&[u8]` upload variants). When `Some`,
/// keeps spools off tmpfs/RAM so they don't count against the cgroup limit.
upload_temp_dir: Option<PathBuf>,
}
impl FileUploadService {
@@ -69,7 +60,6 @@ impl FileUploadService {
storage_usage_service: None,
content_cache: None,
file_lifecycle_hook: None,
upload_temp_dir: None,
}
}
@@ -84,16 +74,9 @@ impl FileUploadService {
storage_usage_service: None,
content_cache: None,
file_lifecycle_hook: None,
upload_temp_dir: None,
}
}
/// Configures the spool directory for the `&[u8]` upload variants.
pub fn with_upload_temp_dir(mut self, dir: Option<PathBuf>) -> Self {
self.upload_temp_dir = dir;
self
}
/// Configures the content cache for invalidation on file updates.
pub fn with_content_cache(mut self, cache: Arc<FileContentCache>) -> Self {
self.content_cache = Some(cache);
@@ -117,11 +100,6 @@ impl FileUploadService {
// ── private helpers ──────────────────────────────────────────
/// Create a spool temp file, honoring the configured upload temp dir.
fn new_temp(&self) -> std::io::Result<tempfile::NamedTempFile> {
crate::common::temp::new_spool_temp_file(self.upload_temp_dir.as_deref())
}
/// Optionally update storage usage after a successful upload.
fn maybe_update_storage_usage(&self, file: &FileDto) {
if let Some(storage_service) = &self.storage_usage_service {
@@ -146,172 +124,37 @@ impl FileUploadService {
}
impl FileUploadUseCase for FileUploadService {
/// Streaming upload from a temp file on disk.
///
/// Peak RAM: ~256 KB (hash calculation) regardless of file size.
/// The temp file is consumed (moved/deleted) by the blob store.
/// Register a new file row pointing at an already-ingested blob.
async fn upload_file_streaming(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
temp_path: &Path,
size: u64,
pre_computed_hash: Option<String>,
blob: StoredBlob,
) -> Result<FileDto, DomainError> {
let (file, is_new_blob) = self
let file = self
.file_write
.save_file_from_temp_with_dedup(
name.clone(),
folder_id,
content_type,
temp_path,
size,
pre_computed_hash,
)
.save_file_with_blob(name.clone(), folder_id, content_type, &blob.hash, blob.size)
.await?;
let dto = FileDto::from(file);
info!(
"📡 STREAMING UPLOAD: {} ({} bytes, ID: {})",
name, size, dto.id
name, blob.size, dto.id
);
self.maybe_update_storage_usage(&dto);
if let Some(hook) = &self.file_lifecycle_hook {
hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, is_new_blob);
hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, blob.is_new_blob);
}
Ok(dto)
}
/// Upload from a file already on disk (chunked uploads).
async fn upload_file_from_path(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
file_path: &Path,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError> {
let size = tokio::fs::metadata(file_path)
.await
.map_err(|e| {
DomainError::internal_error(
"FileUpload",
format!("Failed to read file metadata: {}", e),
)
})?
.len();
self.upload_file_streaming(
name,
folder_id,
content_type,
file_path,
size,
pre_computed_hash,
)
.await
}
/// Creates a file at a specific path (for WebDAV PUT on new resource).
///
/// Spools the in-memory `&[u8]` to a temp file with hash-on-write,
/// then delegates to the streaming path. Peak RAM: the caller's
/// buffer + ~256 KB for the hasher.
async fn create_file(
&self,
parent_path: &str,
filename: &str,
content: &[u8],
content_type: &str,
) -> Result<FileDto, DomainError> {
// Look up the folder ID by folder path
let parent_id = if !parent_path.is_empty() {
if let Some(file_read) = &self.file_read {
file_read.get_folder_id_by_path(parent_path).await.ok()
} else {
None
}
} else {
None
};
// Spool to temp file + hash
let temp = self
.new_temp()
.map_err(|e| DomainError::internal_error("FileUpload", format!("temp file: {e}")))?;
tokio::fs::write(temp.path(), content)
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("write temp: {e}")))?;
let hash = DedupService::hash_file(temp.path())
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("hash: {e}")))?;
let (file, is_new_blob) = self
.file_write
.save_file_from_temp_with_dedup(
filename.to_string(),
parent_id,
content_type.to_string(),
temp.path(),
content.len() as u64,
Some(hash),
)
.await?;
let dto = FileDto::from(file);
self.maybe_update_storage_usage(&dto);
if let Some(hook) = &self.file_lifecycle_hook {
hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, is_new_blob);
}
Ok(dto)
}
/// Updates an existing file's content, or creates it if not found (for WebDAV PUT).
///
/// Spools the in-memory `&[u8]` to a temp file with hash-on-write,
/// then delegates to the streaming update/create path.
async fn update_file(
&self,
path: &str,
content: &[u8],
content_type: &str,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError> {
// Spool to temp file + hash
let temp = self
.new_temp()
.map_err(|e| DomainError::internal_error("FileUpload", format!("temp file: {e}")))?;
tokio::fs::write(temp.path(), content)
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("write temp: {e}")))?;
let hash = DedupService::hash_file(temp.path())
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("hash: {e}")))?;
self.update_file_streaming(
path,
temp.path(),
content.len() as u64,
content_type,
Some(hash),
modified_at,
)
.await
}
/// Streaming update — replaces file content from a temp file on disk.
///
/// Uses `update_file_content_from_temp` which passes the pre-computed hash
/// to dedup, avoiding a second full read of the file.
/// For new files (not found at `path`), falls back to `upload_file_streaming`.
///
/// Peak RAM: ~256 KB regardless of file size.
/// Swap the content of the file at `path` to an already-ingested blob,
/// creating the file when it doesn't exist (WebDAV/NextCloud/WOPI PUT).
async fn update_file_streaming(
&self,
path: &str,
temp_path: &Path,
size: u64,
blob: StoredBlob,
content_type: &str,
pre_computed_hash: Option<String>,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError> {
// Try to find the existing file first
@@ -321,14 +164,7 @@ impl FileUploadUseCase for FileUploadService {
let file_id = file.id().to_string();
let (new_hash, updated_at) = self
.file_write
.update_file_content_from_temp(
&file_id,
temp_path,
size,
Some(content_type.to_string()),
pre_computed_hash,
modified_at,
)
.update_file_content_with_blob(&file_id, &blob.hash, blob.size, modified_at)
.await?;
// Invalidate content cache — file content has changed.
if let Some(cc) = &self.content_cache {
@@ -343,7 +179,7 @@ impl FileUploadUseCase for FileUploadService {
parts.id,
parts.name,
parts.storage_path,
size,
blob.size,
parts.mime_type,
parts.folder_id,
parts.created_at,
@@ -381,15 +217,15 @@ impl FileUploadUseCase for FileUploadService {
None
};
let (created, is_new_blob) = self
let is_new_blob = blob.is_new_blob;
let created = self
.file_write
.save_file_from_temp_with_dedup(
.save_file_with_blob(
filename.to_string(),
parent_id,
content_type.to_string(),
temp_path,
size,
pre_computed_hash,
&blob.hash,
blob.size,
)
.await?;
let dto = FileDto::from(created);
src/application/services/idor_protection_test.rs
+5 -8
View File
@@ -6,7 +6,7 @@
use bytes::Bytes;
use futures::Stream;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::path::PathBuf;
use std::pin::Pin;
use std::sync::Mutex;
use uuid::Uuid;
@@ -169,14 +169,13 @@ impl MockFileWritePort {
}
impl FileWritePort for MockFileWritePort {
async fn save_file_from_temp(
async fn save_file_with_blob(
&self,
_name: String,
_folder_id: Option<String>,
_content_type: String,
_temp_path: &Path,
_blob_hash: &str,
_size: u64,
_pre_computed_hash: Option<String>,
) -> Result<File, DomainError> {
unimplemented!()
}
@@ -205,13 +204,11 @@ impl FileWritePort for MockFileWritePort {
Ok(())
}
async fn update_file_content_from_temp(
async fn update_file_content_with_blob(
&self,
_file_id: &str,
_temp_path: &Path,
_blob_hash: &str,
_size: u64,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>,
) -> Result<(String, i64), DomainError> {
Ok((String::new(), 0))
src/application/services/trash_service_test.rs
+4 -7
View File
@@ -554,14 +554,13 @@ impl FileReadPort for MockFileRepository {
}
impl FileWritePort for MockFileRepository {
async fn save_file_from_temp(
async fn save_file_with_blob(
&self,
_name: String,
_folder_id: Option<String>,
_content_type: String,
_temp_path: &std::path::Path,
_blob_hash: &str,
_size: u64,
_pre_computed_hash: Option<String>,
) -> std::result::Result<File, DomainError> {
unimplemented!()
}
@@ -586,13 +585,11 @@ impl FileWritePort for MockFileRepository {
Ok(())
}
async fn update_file_content_from_temp(
async fn update_file_content_with_blob(
&self,
_file_id: &str,
_temp_path: &std::path::Path,
_blob_hash: &str,
_size: u64,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>,
) -> std::result::Result<(String, i64), DomainError> {
Ok((String::new(), 0))