Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
@@ -898,10 +898,11 @@ async fn handle_head(
|
||||
/**
|
||||
* Handles PUT requests to create or update files.
|
||||
*
|
||||
* **Streaming implementation**: the request body is spooled to a temp file
|
||||
* with incremental BLAKE3 hashing. Peak RAM usage is ~256 KB regardless
|
||||
* of file size. The temp file is then atomically moved into blob storage
|
||||
* via `update_file_streaming`.
|
||||
* **Streaming implementation**: the request body is streamed straight into
|
||||
* the CDC chunk store (FastCDC + BLAKE3 while the bytes arrive — no spool
|
||||
* file, no re-read; peak RAM is bounded regardless of file size), then the
|
||||
* file row is atomically swapped onto the ingested blob via
|
||||
* `update_file_streaming`.
|
||||
*
|
||||
* @param state The application state containing service dependencies
|
||||
* @param path The requested resource path
|
||||
@@ -913,7 +914,7 @@ async fn handle_put(
|
||||
req: Request<Body>,
|
||||
path: String,
|
||||
) -> Result<Response<Body>, AppError> {
|
||||
use crate::interfaces::upload_spool::spool_body_to_temp;
|
||||
use crate::interfaces::upload_ingest;
|
||||
|
||||
let user = extract_user(&req)?;
|
||||
|
||||
@@ -972,32 +973,31 @@ async fn handle_put(
|
||||
.unwrap_or("application/octet-stream")
|
||||
.to_string();
|
||||
|
||||
// ── Streaming spool: body → temp file + incremental hash ──
|
||||
// Shared with the NextCloud-compat PUT handler; peak heap ~one frame
|
||||
// regardless of file size. Honors `upload_temp_dir` to keep the spool
|
||||
// off tmpfs/RAM.
|
||||
let spooled = spool_body_to_temp(
|
||||
// ── Streaming ingest: body → CDC chunk store ──────────────
|
||||
// Shared with the NextCloud-compat PUT handler; chunking + hashing +
|
||||
// dedup checks run while the body arrives — no spool file, no re-read.
|
||||
let filename = crate::common::mime_detect::filename_from_path(&path).to_string();
|
||||
let ingested = upload_ingest::ingest_body_to_cas(
|
||||
req.into_body(),
|
||||
&state.core.dedup_service,
|
||||
&filename,
|
||||
&content_type,
|
||||
max_upload,
|
||||
state.core.config.storage.upload_temp_dir.clone(),
|
||||
)
|
||||
.await?;
|
||||
let temp_path = spooled.temp.path().to_path_buf();
|
||||
let total_bytes = spooled.size as usize;
|
||||
let hash = spooled.hash;
|
||||
|
||||
// ── Quota enforcement ────────────────────────────────────
|
||||
if let Some(storage_svc) = state.storage_usage_service.as_ref()
|
||||
&& let Err(err) = storage_svc
|
||||
.check_storage_quota(user.id, total_bytes as u64)
|
||||
.check_storage_quota(user.id, ingested.size)
|
||||
.await
|
||||
{
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
upload_ingest::discard_ingested(&state.core.dedup_service, &ingested).await;
|
||||
tracing::warn!(
|
||||
"⛔ WEBDAV PUT REJECTED (quota): user={}, file={}, size={}",
|
||||
user.id,
|
||||
path,
|
||||
total_bytes
|
||||
ingested.size
|
||||
);
|
||||
return Err(AppError::new(
|
||||
StatusCode::INSUFFICIENT_STORAGE,
|
||||
@@ -1006,21 +1006,12 @@ async fn handle_put(
|
||||
));
|
||||
}
|
||||
|
||||
// ── Atomic store: temp file → dedup blob + DB metadata update ──
|
||||
// ── Atomic store: swap the file row onto the ingested blob ──
|
||||
let content_type = ingested.content_type.clone();
|
||||
let result = file_upload_service
|
||||
.update_file_streaming(
|
||||
&path,
|
||||
&temp_path,
|
||||
total_bytes as u64,
|
||||
&content_type,
|
||||
Some(hash),
|
||||
None,
|
||||
)
|
||||
.update_file_streaming(&path, ingested.stored(), &content_type, None)
|
||||
.await;
|
||||
|
||||
// Clean up temp file (may already be moved by dedup, ignore error)
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
|
||||
match result {
|
||||
Ok(_file_dto) => Ok(Response::builder()
|
||||
.status(StatusCode::NO_CONTENT)
|
||||
|
||||
Reference in New Issue
Block a user