cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

Stream uploads directly into the CDC chunk store (no spool, single write)

Browse Source 
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/infrastructure/services/chunked_upload_service.rs
+41 -127
View File
@@ -6,14 +6,14 @@
//! (updated atomically on each chunk) are stored alongside the chunk files.
//! On boot the service scans `temp_base_dir` and recovers any active sessions.
//! - Parallel chunk transfers (up to 6 concurrent)
//! - Automatic reassembly with hash-on-write (BLAKE3)
//! - Completion streams the ordered parts straight into the CDC blob store
//! - Expiration cleanup (24 h)
//!
//! Protocol:
//! 1. POST /api/uploads → Create upload session, get upload_id
//! 2. PATCH /api/uploads/:id → Upload chunks (parallel OK)
//! 3. HEAD /api/uploads/:id → Check progress
//! 4. POST /api/uploads/:id/complete → Finalize and assemble
//! 4. POST /api/uploads/:id/complete → Stream parts into the blob store
use chrono::{DateTime, Utc};
use dashmap::DashMap;
@@ -28,7 +28,8 @@ use tokio::io::AsyncWriteExt;
use uuid::Uuid;
use crate::application::ports::chunked_upload_ports::{
ChunkUploadResponseDto, ChunkedUploadPort, CreateUploadResponseDto, UploadStatusResponseDto,
ChunkUploadResponseDto, ChunkedUploadPort, CompletedUploadParts, CreateUploadResponseDto,
UploadStatusResponseDto,
};
use crate::domain::errors::{DomainError, ErrorKind};
@@ -603,7 +604,7 @@ impl ChunkedUploadService {
///
/// Used by the streaming REST PUT path: the handler calls
/// `prepare_chunk` → streams body to disk via
/// `interfaces::upload_spool::stream_body_to_path` → calls
/// `interfaces::upload_ingest::stream_body_to_path` → calls
/// `commit_chunk` to finalise. This lets the body bypass the
/// in-memory `Bytes` allocation entirely (peak heap ~one HTTP
/// frame instead of "chunk size").
@@ -936,24 +937,23 @@ impl ChunkedUploadService {
})
}
/// Assemble chunks into final file and return the path + pre-computed BLAKE3 hash.
/// Validate completion and return the chunk parts in assembly order.
///
/// **Hash-on-Write**: BLAKE3 is computed while copying chunks into the
/// assembled file, eliminating the second sequential read that dedup_service
/// would otherwise need.
///
/// Returns `(assembled_file_path, filename, folder_id, content_type, total_size, blake3_hash)`.
/// No assembled file is written and nothing is hashed here — the caller
/// streams the parts straight into the CDC chunk store, which computes
/// BLAKE3 and dedup-checks each CDC chunk in that single read pass. The
/// part files stay on disk until `finalize_upload`, so a completion
/// that fails downstream (e.g. client checksum mismatch) is retryable.
async fn complete_upload_inner(
&self,
upload_id: &str,
user_id: &str,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), String> {
// Verify ownership before assembly
) -> Result<CompletedUploadParts, String> {
// Verify ownership before completion
self.verify_session_owner(upload_id, user_id)?;
// Get session and validate completion.
// Clone the session data and drop the DashMap ref immediately
// so the shard is not held during the expensive assembly step.
// Clone the session data and drop the DashMap ref immediately.
let session = {
let entry = self
.sessions
@@ -971,118 +971,29 @@ impl ChunkedUploadService {
entry.clone()
};
// Assemble file with hash-on-write.
//
// The entire loop is offloaded to spawn_blocking because BLAKE3
// hashing is CPU-bound and would otherwise block a Tokio worker,
// starving all other connections.
// Synchronous I/O is used inside the blocking thread — it avoids
// the async reactor overhead and is actually faster for this
// sequential workload.
let assembled_path = session.temp_dir.join("assembled");
let chunks_meta: Vec<(usize, PathBuf)> = session
.chunks
// Chunk indices are unique (upload_chunk rejects duplicates) but may
// have arrived out of order — sort to recover the assembly order.
let mut indices: Vec<usize> = session.chunks.iter().map(|c| c.index).collect();
indices.sort_unstable();
let chunk_paths: Vec<PathBuf> = indices
.iter()
.map(|c| {
(
c.index,
session.temp_dir.join(format!("chunk_{:06}", c.index)),
)
})
.map(|index| session.temp_dir.join(format!("chunk_{:06}", index)))
.collect();
let total_size = session.total_size;
let hash = tokio::task::spawn_blocking(move || -> Result<String, String> {
use std::io::{BufWriter as StdBufWriter, Read, Write};
let raw_output = std::fs::OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(&assembled_path)
.map_err(|e| format!("Failed to create assembled file: {e}"))?;
// Pre-allocate assembled file to reduce fragmentation
let _ = raw_output.set_len(total_size);
// 512 KB I/O buffers — 8× fewer syscalls than 64 KB
let mut output = StdBufWriter::with_capacity(524_288, raw_output);
let mut hasher = blake3::Hasher::new();
// For files >10 MB, use multithreaded BLAKE3 hashing (all cores)
const RAYON_THRESHOLD: u64 = 10 * 1024 * 1024;
let use_rayon = total_size > RAYON_THRESHOLD;
// Single 512 KB read buffer reused across all chunks (avoids N allocations)
let mut buf = vec![0u8; 524_288];
for (index, chunk_path) in &chunks_meta {
let mut chunk_file = std::fs::File::open(chunk_path)
.map_err(|e| format!("Failed to open chunk {index}: {e}"))?;
loop {
let n = chunk_file
.read(&mut buf)
.map_err(|e| format!("Failed to read chunk {index}: {e}"))?;
if n == 0 {
break;
}
if use_rayon {
hasher.update_rayon(&buf[..n]);
} else {
hasher.update(&buf[..n]);
}
output.write_all(&buf[..n]).map_err(|e| {
format!("Failed to write chunk {index} to assembled file: {e}")
})?;
}
}
output
.flush()
.map_err(|e| format!("Failed to flush assembled file: {e}"))?;
// ── Durability boundary ────────────────────────────────────
// `flush` drains BufWriter's userspace buffer but leaves the
// bytes in the kernel page cache. Without `sync_all`, a
// power loss between this `complete_upload` returning 2xx
// and the OS writeback timer firing (~5 s default) loses
// the merged blob — and PG's metadata row references a hash
// that no longer exists on disk. Reclaim the BufWriter's
// inner File via `into_inner` so we can `sync_all` it; the
// BufWriter would otherwise drop without flushing on the
// inner handle.
let raw_output = output
.into_inner()
.map_err(|e| format!("into_inner on BufWriter failed: {e}"))?;
raw_output
.sync_all()
.map_err(|e| format!("Failed to fsync assembled file: {e}"))?;
// Clean up chunk files (keep assembled) — already on a blocking thread
for (_index, chunk_path) in &chunks_meta {
let _ = std::fs::remove_file(chunk_path);
}
Ok(hasher.finalize().to_hex().to_string())
})
.await
.map_err(|e| format!("Assembly task panicked: {e}"))??;
let assembled_path = session.temp_dir.join("assembled");
tracing::info!(
"✅ Assembled chunked upload: {} ({} bytes from {} chunks)",
"✅ Chunked upload complete: {} ({} bytes in {} chunks, streamed to CAS)",
session.filename,
session.total_size,
session.chunks.len()
chunk_paths.len()
);
Ok((
assembled_path,
session.filename.clone(),
session.folder_id.clone(),
session.content_type.clone(),
session.total_size,
hash,
))
Ok(CompletedUploadParts {
chunk_paths,
filename: session.filename.clone(),
folder_id: session.folder_id.clone(),
content_type: session.content_type.clone(),
total_size: session.total_size,
})
}
/// Finalize upload: remove session from RAM, then clean disk OUTSIDE lock.
@@ -1187,7 +1098,7 @@ impl ChunkedUploadPort for ChunkedUploadService {
&self,
upload_id: &str,
user_id: Uuid,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), DomainError> {
) -> Result<CompletedUploadParts, DomainError> {
self.complete_upload_inner(upload_id, &user_id.to_string())
.await
.map_err(|e| DomainError::new(ErrorKind::InternalError, "ChunkedUpload", e))
@@ -1458,18 +1369,21 @@ mod tests {
assert_eq!(status.completed_chunks, 2);
assert!(status.pending_chunks.is_empty());
// 4. Complete (assemble)
let (path, filename, _folder, _ct, size, hash) = service
// 4. Complete — returns the ordered chunk parts, no assembled file
let parts = service
.complete_upload_inner(&id, "test-user")
.await
.expect("complete");
assert_eq!(filename, "test.txt");
assert_eq!(size, 1024);
assert!(!hash.is_empty());
assert!(path.exists());
assert_eq!(parts.filename, "test.txt");
assert_eq!(parts.total_size, 1024);
assert_eq!(parts.chunk_paths.len(), 2);
// 5. Verify assembled content
let content = fs::read(&path).await.expect("read assembled");
// 5. Verify the parts concatenate to the original content in order
let mut content = Vec::new();
for path in &parts.chunk_paths {
assert!(path.exists(), "chunk part must remain until finalize");
content.extend_from_slice(&fs::read(path).await.expect("read part"));
}
assert_eq!(&content[..512], &[b'A'; 512]);
assert_eq!(&content[512..], &[b'B'; 512]);
src/infrastructure/services/dedup_service.rs
+647 -629
View File
File diff suppressed because it is too large Load Diff
src/infrastructure/services/local_blob_backend.rs
+7 -6
View File
@@ -244,12 +244,13 @@ impl BlobStorageBackend for LocalBlobBackend {
// Atomic rename (same filesystem). Falls back to copy+delete for
// cross-device moves (EXDEV errno 18).
//
// Durability boundary: the caller is responsible for
// having sync_all'd the source file before invoking this
// function — both `interfaces::upload_spool::spool_body_to_temp`
// and `chunked_upload_service::complete_upload_inner`
// (the two production producers of `source_path`) now do
// so. We fsync the parent of `blob_path` AFTER the rename
// Durability boundary: the caller is responsible for having
// sync_all'd the source file before invoking this function.
// (The streaming upload path writes chunks via
// `put_blob_from_bytes_unsynced` + a batched `sync_blobs`
// sweep instead; this move-based entry point remains for
// whole-file producers such as migration tooling and tests.)
// We fsync the parent of `blob_path` AFTER the rename
// so the dirent change itself becomes durable; without
// that, a power loss can resurrect the old (unrenamed)
// name even when the file contents survive.
src/infrastructure/services/nextcloud_chunked_upload_service.rs
+55 -111
View File
@@ -73,7 +73,7 @@ impl NextcloudChunkedUploadService {
/// Store a chunk in the session directory. Buffers `data` in memory —
/// use [`safe_chunk_path`](Self::safe_chunk_path) + the
/// `interfaces/upload_spool::stream_body_to_path` helper to stream the
/// `interfaces/upload_ingest::stream_body_to_path` helper to stream the
/// HTTP body directly to disk and avoid materialising the whole chunk
/// in RAM.
pub async fn store_chunk(
@@ -93,24 +93,14 @@ impl NextcloudChunkedUploadService {
Ok(())
}
/// Assemble all chunks in numeric order into a temp file, computing
/// the BLAKE3 of the concatenated stream **during** the same read/
/// write pass (hash-on-write).
/// List the session's chunk files in assembly (numeric) order.
///
/// Returns `(temp_path, total_size, blake3_hex)`. The caller passes
/// the hash to the upload service as `pre_computed_hash` so the
/// downstream dedup layer never has to re-read the assembled file
/// to compute it — saving one full file-sized read pass per upload.
///
/// The read/hash/write loop runs inside `spawn_blocking` because
/// BLAKE3 is CPU-bound and would otherwise starve the Tokio worker
/// running other connections; synchronous I/O is used inside the
/// blocking thread because the workload is sequential and the
/// async reactor overhead would only slow it down. For files larger
/// than ~10 MB BLAKE3's Rayon mode parallelises across cores —
/// mirrors what `ChunkedUploadService::complete_upload_inner` does
/// for the REST chunked path.
pub async fn assemble(&self, user: &str, upload_id: &str) -> Result<(PathBuf, u64, String)> {
/// The caller streams these directly into the CDC chunk store
/// (`interfaces::upload_ingest::stream_from_files`) — chunking, BLAKE3
/// hashing and dedup checks happen in that single read pass, so no
/// assembled temp file is ever written. The chunk parts stay on disk
/// until [`cleanup`](Self::cleanup), keeping completion retryable.
pub async fn ordered_chunk_paths(&self, user: &str, upload_id: &str) -> Result<Vec<PathBuf>> {
let session_dir = self.safe_session_dir(user, upload_id)?;
let mut entries: Vec<String> = Vec::new();
@@ -124,8 +114,11 @@ impl NextcloudChunkedUploadService {
.map_err(|e| DomainError::internal_error("ChunkedUpload", e.to_string()))?
{
let name = entry.file_name().to_string_lossy().to_string();
if name == ".file" {
continue; // Skip the assembly marker.
// `.file` is the NC-protocol assembly marker; `.assembled` is
// the staging file older releases wrote — sessions in flight
// across an upgrade may still contain one.
if name == ".file" || name == ".assembled" {
continue;
}
entries.push(name);
}
@@ -133,74 +126,7 @@ impl NextcloudChunkedUploadService {
// Sort chunks numerically (Nextcloud sends them as "00001", "00002", ...).
entries.sort();
let temp_path = session_dir.join(".assembled");
let chunk_paths: Vec<PathBuf> = entries.iter().map(|n| session_dir.join(n)).collect();
let assembled_for_blocking = temp_path.clone();
// Read/hash/write loop runs synchronously on the blocking pool.
// BLAKE3 is computed in the same pass that copies bytes from chunk
// files into the assembled file — no second read after the fact.
let (total_size, hash) =
tokio::task::spawn_blocking(move || -> std::io::Result<(u64, String)> {
use std::io::{BufWriter as StdBufWriter, Read, Write};
let raw_output = std::fs::OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(&assembled_for_blocking)?;
// 512 KB write buffer — 8× fewer syscalls than 64 KB.
let mut output = StdBufWriter::with_capacity(524_288, raw_output);
let mut hasher = blake3::Hasher::new();
let mut buf = vec![0u8; 524_288];
let mut total: u64 = 0;
// Files >10 MB benefit from BLAKE3's multi-threaded mode.
// The threshold matches the REST chunked path's heuristic.
const RAYON_THRESHOLD_PER_FRAME: usize = 128 * 1024;
for chunk_path in &chunk_paths {
let mut chunk_file = std::fs::File::open(chunk_path)?;
loop {
let n = chunk_file.read(&mut buf)?;
if n == 0 {
break;
}
if n >= RAYON_THRESHOLD_PER_FRAME {
hasher.update_rayon(&buf[..n]);
} else {
hasher.update(&buf[..n]);
}
output.write_all(&buf[..n])?;
total += n as u64;
}
}
output.flush()?;
// ── Durability boundary ─────────────────────────────────
// sync_all is the actual fsync; without it, a power loss
// before the kernel writeback timer (~5 s) loses
// acknowledged data. Pull the inner File out of the
// BufWriter so we can sync the underlying handle —
// dropping the BufWriter wouldn't trigger fsync. macOS
// caveat: fsync there flushes to the disk controller
// only; true durability needs F_FULLFSYNC, not exposed
// by std.
let raw_output = output
.into_inner()
.map_err(|e| std::io::Error::other(format!("into_inner: {e}")))?;
raw_output.sync_all()?;
Ok((total, hasher.finalize().to_hex().to_string()))
})
.await
.map_err(|e| {
DomainError::internal_error("ChunkedUpload", format!("assemble task: {e}"))
})?
.map_err(|e| DomainError::internal_error("ChunkedUpload", e.to_string()))?;
Ok((temp_path, total_size, hash))
Ok(entries.iter().map(|n| session_dir.join(n)).collect())
}
/// Delete the upload session directory.
@@ -260,9 +186,10 @@ impl NextcloudChunkedUploadService {
let name = entry.file_name().to_string_lossy().to_string();
// Filter internal markers — `.file` is the NC-protocol
// assembly trigger target (it never reaches the disk
// because MOVE redirects it), `.assembled` is our own
// staging file from `assemble()`. Surfacing either to
// the client would confuse its chunk-count check.
// because MOVE redirects it), `.assembled` is the staging
// file pre-streaming releases wrote (kept for sessions in
// flight across an upgrade). Surfacing either to the
// client would confuse its chunk-count check.
if name == ".file" || name == ".assembled" {
continue;
}
@@ -331,8 +258,18 @@ mod tests {
assert!(!svc.session_exists("alice", "upload-999").await);
}
/// Concatenate the session's chunk files in assembly order — mirrors
/// what `upload_ingest::stream_from_files` feeds the CDC store.
async fn concat_chunks(svc: &NextcloudChunkedUploadService, user: &str, id: &str) -> Vec<u8> {
let mut out = Vec::new();
for path in svc.ordered_chunk_paths(user, id).await.unwrap() {
out.extend_from_slice(&fs::read(&path).await.unwrap());
}
out
}
#[tokio::test]
async fn test_store_and_assemble_chunks() {
async fn test_store_and_order_chunks() {
let (svc, _dir) = test_service();
svc.create_session("alice", "upload-002").await.unwrap();
@@ -343,20 +280,14 @@ mod tests {
.await
.unwrap();
let (temp_path, size, hash) = svc.assemble("alice", "upload-002").await.unwrap();
let assembled = fs::read(&temp_path).await.unwrap();
assert_eq!(assembled, b"Hello, World!");
assert_eq!(size, 13);
// BLAKE3("Hello, World!") — proves hash-on-write happens during
// the assemble pass, not via a re-read.
assert_eq!(
hash,
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8"
concat_chunks(&svc, "alice", "upload-002").await,
b"Hello, World!"
);
}
#[tokio::test]
async fn test_assemble_chunks_in_sorted_order() {
async fn test_chunk_paths_sorted_regardless_of_upload_order() {
let (svc, _dir) = test_service();
svc.create_session("alice", "upload-003").await.unwrap();
@@ -371,16 +302,29 @@ mod tests {
.await
.unwrap();
let (temp_path, size, hash) = svc.assemble("alice", "upload-003").await.unwrap();
let assembled = fs::read(&temp_path).await.unwrap();
assert_eq!(assembled, b"ABC");
assert_eq!(size, 3);
// BLAKE3("ABC") — confirms sort happened (chunks were stored in
// order 3,1,2 but the hash matches "ABC", not "CAB" or "BAC").
assert_eq!(
hash,
"d1717274597cf0289694f75d96d444b992a096f1afd8e7bbfa6ebb1d360fedfc"
);
// Chunks were stored in order 3,1,2 but must concatenate as "ABC".
assert_eq!(concat_chunks(&svc, "alice", "upload-003").await, b"ABC");
}
#[tokio::test]
async fn test_internal_markers_excluded_from_chunk_paths() {
let (svc, _dir) = test_service();
svc.create_session("alice", "upload-005").await.unwrap();
svc.store_chunk("alice", "upload-005", "00001", b"data")
.await
.unwrap();
// Stale staging file from a pre-streaming release.
svc.store_chunk("alice", "upload-005", ".assembled", b"old")
.await
.unwrap();
let paths = svc
.ordered_chunk_paths("alice", "upload-005")
.await
.unwrap();
assert_eq!(paths.len(), 1, "markers must be filtered out");
assert!(paths[0].ends_with("00001"));
}
#[tokio::test]