cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

Stream uploads directly into the CDC chunk store (no spool, single write)

Browse Source 
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/interfaces/api/handlers/chunked_upload_handler.rs
+85 -133
View File
@@ -4,7 +4,7 @@
//! - POST /api/uploads → Create upload session
//! - PATCH /api/uploads/:id → Upload a chunk
//! - HEAD /api/uploads/:id → Get upload status
//! - POST /api/uploads/:id/complete → Assemble and finalize
//! - POST /api/uploads/:id/complete → Stream parts into the blob store
//! - DELETE /api/uploads/:id → Cancel upload
use axum::{
@@ -27,7 +27,7 @@ use crate::common::di::AppState;
use crate::domain::services::authorization::Permission;
use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::upload_spool::stream_body_to_path;
use crate::interfaces::upload_ingest::{self, stream_body_to_path};
/// Request body for creating an upload session
#[derive(Debug, Deserialize, ToSchema)]
@@ -64,35 +64,32 @@ pub struct CompleteUploadResponse {
/// Optional body for `POST /api/uploads/{id}/complete`.
///
/// When the client supplies `checksum`, the server compares it against
/// the assembled file's hash BEFORE promoting the blob to storage —
/// failure aborts the upload atomically (no orphaned blob, no DB row).
/// This is the end-to-end integrity check: per-chunk MD5 proves each
/// chunk arrived intact, but only the final hash catches assembly /
/// promotion bugs and mis-ordered chunks.
/// the streamed content's hash BEFORE the file row is created — failure
/// releases the blob reference and returns 400, with the chunk parts
/// kept on disk for a retry. This is the end-to-end integrity check:
/// per-chunk MD5 proves each chunk arrived intact, but only the final
/// hash catches mis-ordered or corrupted assemblies.
///
/// **`blake3` is highly recommended** — it's the algorithm the server
/// already runs over the assembled file during hash-on-write
/// assembly, so verification is a string comparison with zero extra
/// I/O and zero extra CPU. It's also the same algorithm the server
/// uses for blob-storage addressing, so the value the client sends
/// equals the `content_hash` they'd later read back from
/// `GET /api/files/{id}`. `md5` and `sha256` are accepted for
/// compatibility with legacy client tooling but each triggers a
/// second hash pass over the assembled file (~30–100 ms depending
/// on size).
/// **`blake3` is highly recommended** — it's the content-addressing
/// algorithm of the blob store itself, so verification is a string
/// comparison against the hash the store already computed, and the
/// value the client sends equals the `content_hash` they'd later read
/// back from `GET /api/files/{id}`. `md5` and `sha256` are accepted
/// for legacy client tooling; they are computed by an in-flight tee
/// during the same streaming pass — no extra disk read either way.
///
/// `Default` keeps the existing wire shape: clients that POST with no
/// body get today's behavior (no verification, server just returns
/// what it computed).
#[derive(Debug, Default, Deserialize, ToSchema)]
pub struct CompleteUploadRequest {
/// Lowercase hex digest the client expects the assembled file to
/// Lowercase hex digest the client expects the streamed content to
/// hash to. Compared case-insensitively. Omit to skip verification.
pub checksum: Option<String>,
/// Algorithm name. `blake3` is the recommended choice (default —
/// matches the server's hash-on-write algorithm, zero extra cost).
/// `md5`, `sha256` / `sha-256` are accepted but trigger an extra
/// hash pass. Unknown values return 400.
/// matches the blob store's content-addressing algorithm). `md5`,
/// `sha256` / `sha-256` are accepted and computed in-flight.
/// Unknown values return 400.
pub checksumalg: Option<String>,
}
@@ -307,70 +304,15 @@ impl ChunkedUploadHandler {
}
}
/// Compute the requested checksum of the assembled file.
///
/// For `Blake3` the server already has the hash from hash-on-write
/// assembly — we just return it (zero I/O, zero CPU). For `Md5` and
/// `Sha256` we re-read the assembled file on the blocking pool and
/// hash it; the cost (~30–100 ms for typical files) is the trade-off
/// for accepting non-default algorithms.
async fn compute_assembled_hash(
assembled_path: &std::path::Path,
alg: ChecksumAlg,
blake3_already_computed: &str,
) -> Result<String, std::io::Error> {
match alg {
ChecksumAlg::Blake3 => Ok(blake3_already_computed.to_string()),
ChecksumAlg::Md5 | ChecksumAlg::Sha256 => {
let path = assembled_path.to_path_buf();
tokio::task::spawn_blocking(move || -> Result<String, std::io::Error> {
use std::io::Read;
let mut file = std::fs::File::open(&path)?;
let mut buf = vec![0u8; 524_288];
match alg {
ChecksumAlg::Md5 => {
use md5::Digest as _;
let mut h = md5::Md5::new();
loop {
let n = file.read(&mut buf)?;
if n == 0 {
break;
}
h.update(&buf[..n]);
}
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
}
ChecksumAlg::Sha256 => {
use sha2::Digest as _;
let mut h = sha2::Sha256::new();
loop {
let n = file.read(&mut buf)?;
if n == 0 {
break;
}
h.update(&buf[..n]);
}
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
}
// Blake3 handled above — this branch is unreachable but
// keeps the match exhaustive without an else-clause.
ChecksumAlg::Blake3 => unreachable!(),
}
})
.await
.map_err(|e| std::io::Error::other(format!("hash task join failed: {e}")))?
}
}
}
/// POST /api/uploads/:upload_id/complete - Finalize upload
///
/// Assembles all chunks into the final file and creates the file record.
/// When `body.checksum` is supplied, the assembled file's hash is
/// verified before the blob is promoted to storage — mismatch
/// returns 400 and the assembled temp is removed (the session
/// itself is kept so the client can re-issue complete after
/// diagnosing).
/// Streams the uploaded chunk parts, in order, straight into the CDC
/// chunk store and creates the file record — no assembled temp file is
/// ever written. When `body.checksum` is supplied it is verified from
/// the same streaming pass (BLAKE3 comes from the store itself;
/// MD5/SHA-256 are computed by an in-flight tee) — mismatch returns 400
/// with the blob reference released, and the chunk parts stay on disk
/// so the client can re-issue complete after diagnosing.
pub(super) async fn complete_upload_impl(
State(state): State<Arc<AppState>>,
auth_user: AuthUser,
@@ -379,10 +321,11 @@ impl ChunkedUploadHandler {
) -> impl IntoResponse {
let chunked_service = &state.core.chunked_upload_service;
let upload_service = &state.applications.file_upload_service;
let dedup = &state.core.dedup_service;
// ── Parse the optional algorithm BEFORE assembly so a bad
// `checksumalg` doesn't waste the (potentially expensive)
// hash work on a request we'll reject anyway.
// ── Parse the optional algorithm BEFORE completion so a bad
// `checksumalg` doesn't waste any work on a request we'll
// reject anyway.
let alg = match body.checksumalg.as_deref() {
Some(name) => match ChecksumAlg::parse(name) {
Some(a) => Some(a),
@@ -397,37 +340,54 @@ impl ChunkedUploadHandler {
};
let expected_checksum = body.checksum.as_deref();
// Assemble chunks (hash-on-write: BLAKE3 computed during assembly)
let (assembled_path, filename, folder_id, content_type, total_size, hash) =
match chunked_service
.complete_upload(&upload_id, auth_user.id)
.await
{
Ok(result) => result,
Err(e) => {
return AppError::from(e).into_response();
}
};
// Validate completion and get the chunk parts in assembly order.
let parts = match chunked_service
.complete_upload(&upload_id, auth_user.id)
.await
{
Ok(result) => result,
Err(e) => {
return AppError::from(e).into_response();
}
};
// MD5/SHA-256 verification taps the stream while it is ingested;
// BLAKE3 needs no tee — the store's own content hash IS BLAKE3.
let alg = expected_checksum.map(|_| alg.unwrap_or(ChecksumAlg::Blake3));
let tee = match alg {
Some(ChecksumAlg::Md5) | Some(ChecksumAlg::Sha256) => {
Some(upload_ingest::checksum_tee(alg.unwrap()))
}
_ => None,
};
// ── Stream the parts into the CDC chunk store ───────────────
let ingested = match upload_ingest::ingest_stream_to_cas(
upload_ingest::stream_from_files(parts.chunk_paths),
dedup,
&parts.filename,
&parts.content_type,
usize::MAX,
tee.clone(),
)
.await
{
Ok(ingested) => ingested,
Err(e) => return e.into_response(),
};
// ── End-to-end integrity verification ───────────────────────
// Only fires when the client supplied an `expected` checksum.
// For BLAKE3 (the documented preferred choice) this is a string
// comparison against the hash assembly already produced. For
// MD5/SHA-256 we re-hash the assembled file on the blocking pool.
if let Some(expected) = expected_checksum {
let alg = alg.unwrap_or(ChecksumAlg::Blake3);
let computed = match Self::compute_assembled_hash(&assembled_path, alg, &hash).await {
Ok(c) => c,
Err(e) => {
let _ = tokio::fs::remove_file(&assembled_path).await;
return AppError::internal_error(format!(
"Failed to compute assembled checksum: {e}"
))
.into_response();
}
if let (Some(expected), Some(alg)) = (expected_checksum, alg) {
let computed = match alg {
ChecksumAlg::Blake3 => Some(ingested.hash.clone()),
_ => tee.as_ref().and_then(upload_ingest::finalize_checksum_tee),
};
let Some(computed) = computed else {
upload_ingest::discard_ingested(dedup, &ingested).await;
return AppError::internal_error("Checksum tee produced no digest").into_response();
};
if !computed.eq_ignore_ascii_case(expected) {
let _ = tokio::fs::remove_file(&assembled_path).await;
upload_ingest::discard_ingested(dedup, &ingested).await;
tracing::warn!(
target: "audit",
event = "chunked_upload.checksum_mismatch",
@@ -449,36 +409,28 @@ impl ChunkedUploadHandler {
}
}
// ── MIME detection (magic bytes + extension fallback) ─────
let content_type = crate::common::mime_detect::refine_content_type_from_file(
&assembled_path,
&filename,
&content_type,
)
.await;
// Upload from assembled file on disk — zero extra RAM copies, hash pre-computed
// Register the file row against the ingested blob.
let size = ingested.size;
match upload_service
.upload_file_from_path(
filename.clone(),
folder_id.clone(),
content_type,
&assembled_path,
Some(hash),
.upload_file_streaming(
parts.filename.clone(),
parts.folder_id.clone(),
ingested.content_type.clone(),
ingested.stored(),
)
.await
{
Ok(file) => {
// Cleanup session
// Cleanup session (removes the chunk part files)
let _ = chunked_service
.finalize_upload(&upload_id, auth_user.id)
.await;
tracing::info!(
"✅ CHUNKED UPLOAD COMPLETE: {} (ID: {}, {} bytes)",
filename,
parts.filename,
file.id,
total_size
size
);
(
@@ -486,14 +438,14 @@ impl ChunkedUploadHandler {
Json(CompleteUploadResponse {
file_id: file.id,
filename: file.name,
size: total_size,
size,
path: file.path,
}),
)
.into_response()
}
Err(e) => {
tracing::error!("Failed to create file from assembled upload: {:?}", e);
tracing::error!("Failed to create file from chunked upload: {:?}", e);
AppError::internal_error(format!("Failed to create file: {}", e)).into_response()
}
}
src/interfaces/api/handlers/dedup_handler.rs
+49 -112
View File
@@ -5,12 +5,11 @@ use axum::{
response::IntoResponse,
};
use serde::Serialize;
use tokio::io::AsyncWriteExt;
use utoipa::ToSchema;
use crate::application::ports::dedup_ports::DedupResultDto;
use crate::common::di::AppState;
use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::upload_ingest;
use std::sync::Arc;
/// Global application state for dependency injection
@@ -155,10 +154,10 @@ impl DedupHandler {
/// Upload content with automatic deduplication (streaming).
///
/// Spools the upload to a temp file while computing the BLAKE3 hash
/// incrementally (hash-on-write). Memory usage is constant (~512 KB)
/// regardless of file size. Then delegates to `store_from_file` with
/// the pre-computed hash so the file is never re-read for hashing.
/// Streams the multipart field straight into the CDC chunk store —
/// chunking, BLAKE3 hashing and dedup checks happen while the bytes
/// arrive (no temp file, no re-read; peak RAM is bounded regardless
/// of file size).
///
/// POST /api/dedup/upload
pub(super) async fn upload_with_dedup_impl(
@@ -177,64 +176,29 @@ impl DedupHandler {
.content_type()
.unwrap_or("application/octet-stream")
.to_string();
let filename = field.file_name().unwrap_or("unnamed").to_string();
// ── Spool to temp file + BLAKE3 hash-on-write ────────
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp");
let temp_path = temp_dir.join(format!("dedup-{}", uuid::Uuid::new_v4()));
let mut total_size: u64 = 0;
let mut hasher = blake3::Hasher::new();
let mut field = field;
let spool_result: Result<(), String> = async {
let file = tokio::fs::File::create(&temp_path)
.await
.map_err(|e| format!("Failed to create temp file: {}", e))?;
// 512 KB buffer — reduces write syscalls
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
loop {
match field.chunk().await {
Ok(Some(chunk)) => {
total_size += chunk.len() as u64;
hasher.update(&chunk);
writer
.write_all(&chunk)
.await
.map_err(|e| format!("Failed to write chunk: {}", e))?;
}
Ok(None) => break,
Err(e) => {
return Err(format!(
"Connection lost during upload (received {} bytes): {}",
total_size, e
));
}
}
// ── Stream into the CDC chunk store ──────────────────
let source = upload_ingest::multipart_field_stream(field);
let ingested = match upload_ingest::ingest_stream_to_cas(
source,
dedup,
&filename,
&content_type,
usize::MAX,
None,
)
.await
{
Ok(ingested) => ingested,
Err(e) => {
tracing::warn!("Dedup upload ingest failed: {}", e.message);
return e.into_response();
}
};
writer
.flush()
.await
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
Ok(())
}
.await;
if let Err(msg) = spool_result {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::warn!("Dedup upload spool failed: {}", msg);
return Response::builder()
.status(StatusCode::BAD_REQUEST)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(format!(r#"{{"error": "{}"}}"#, msg)))
.unwrap()
.into_response();
}
if total_size == 0 {
let _ = tokio::fs::remove_file(&temp_path).await;
if ingested.size == 0 {
upload_ingest::discard_ingested(dedup, &ingested).await;
return Response::builder()
.status(StatusCode::BAD_REQUEST)
.header(header::CONTENT_TYPE, "application/json")
@@ -243,60 +207,33 @@ impl DedupHandler {
.into_response();
}
let hash = hasher.finalize().to_hex().to_string();
let metadata = dedup.get_blob_metadata(&ingested.hash).await;
// ── Store with deduplication (pre-computed hash) ──────
match dedup
.store_from_file(&temp_path, Some(content_type), Some(hash))
.await
{
Ok(result) => {
let (is_new, bytes_saved) = match &result {
DedupResultDto::NewBlob { .. } => (true, 0),
DedupResultDto::ExistingBlob { saved_bytes, .. } => {
(false, *saved_bytes)
}
};
let response = DedupUploadResponse {
is_new: ingested.is_new_blob,
hash: ingested.hash.clone(),
size: ingested.size,
bytes_saved: ingested.bytes_saved,
ref_count: metadata.map(|m| m.ref_count).unwrap_or(1),
};
let metadata = dedup.get_blob_metadata(result.hash()).await;
tracing::info!(
"🔗 Dedup upload: hash={}, new={}, saved={}",
ingested.hash,
ingested.is_new_blob,
ingested.bytes_saved
);
let response = DedupUploadResponse {
is_new,
hash: result.hash().to_string(),
size: result.size(),
bytes_saved,
ref_count: metadata.map(|m| m.ref_count).unwrap_or(1),
};
tracing::info!(
"🔗 Dedup upload: hash={}, new={}, saved={}",
result.hash(),
is_new,
bytes_saved
);
return Response::builder()
.status(if is_new {
StatusCode::CREATED
} else {
StatusCode::OK
})
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(serde_json::to_string(&response).unwrap()))
.unwrap()
.into_response();
}
Err(e) => {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("Dedup upload failed: {}", e);
return Response::builder()
.status(StatusCode::INTERNAL_SERVER_ERROR)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(r#"{"error": "Upload failed"}"#))
.unwrap()
.into_response();
}
}
return Response::builder()
.status(if ingested.is_new_blob {
StatusCode::CREATED
} else {
StatusCode::OK
})
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(serde_json::to_string(&response).unwrap()))
.unwrap()
.into_response();
}
}
src/interfaces/api/handlers/file_handler.rs
+37 -101
View File
@@ -21,6 +21,7 @@ use crate::common::di::AppState;
use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::range_requests::not_modified_response;
use crate::interfaces::upload_ingest;
use crate::{application::dtos::file_dto::FileDto, domain::services::authorization::Permission};
use std::sync::Arc;
@@ -51,11 +52,12 @@ impl FileHandler {
// UPLOAD
// ═══════════════════════════════════════════════════════════════════════
/// Streaming file upload — constant ~64 KB RAM regardless of file size.
/// Streaming file upload — bounded RAM regardless of file size.
///
/// **Hash-on-Write**: BLAKE3 is computed while spooling the multipart
/// body to the temp file. This eliminates the second sequential read
/// that dedup_service would otherwise need, cutting total I/O in half.
/// The multipart body is streamed straight into the CDC chunk store:
/// chunking, hashing and dedup checks happen while the bytes arrive.
/// No spool file, no re-read — chunks the store already has are never
/// written to disk at all.
pub async fn upload_file(
State(state): State<GlobalState>,
auth_user: AuthUser,
@@ -71,7 +73,7 @@ impl FileHandler {
/// [`Self::upload_file_with_thumbnails`].
///
/// Returns `(FileDto, blob_hash)` on success. The blob hash is the
/// BLAKE3 digest computed during the hash-on-write spool and is
/// BLAKE3 digest computed during the streaming ingest and is
/// propagated without an extra database round-trip so that callers
/// (e.g. thumbnail generation) can resolve the physical blob path
/// immediately.
@@ -158,120 +160,55 @@ impl FileHandler {
}
}
// ── Spool multipart field to temp file + hash-on-write ──
// .dedup_temp is created once by DedupService::initialize() at startup
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp");
let temp_path = temp_dir.join(format!("upload-{}", uuid::Uuid::new_v4()));
let mut total_size: u64 = 0;
let mut hasher = blake3::Hasher::new();
let spool_result: Result<(), String> = async {
let file = tokio::fs::File::create(&temp_path)
.await
.map_err(|e| format!("Failed to create temp file: {}", e))?;
// Pre-allocate if Content-Length is known (reduces fragmentation)
let hint = field
.headers()
.get(axum::http::header::CONTENT_LENGTH)
.and_then(|v| v.to_str().ok())
.and_then(|s| s.parse::<u64>().ok());
if let Some(len) = hint {
let _ = file.set_len(len).await; // best-effort
}
// 512 KB buffer — 8× fewer write syscalls than 64 KB
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
let mut field = field;
// IMPORTANT: use explicit match instead of `while let Ok(Some(..))`.
// The old pattern silently swallowed Err (client disconnect)
// and accepted partially received data as a complete upload.
loop {
match field.chunk().await {
Ok(Some(chunk)) => {
total_size += chunk.len() as u64;
hasher.update(&chunk);
tokio::io::AsyncWriteExt::write_all(&mut writer, &chunk)
.await
.map_err(|e| format!("Failed to write chunk: {}", e))?;
}
Ok(None) => break, // End of field — upload complete
Err(e) => {
return Err(format!(
"Connection lost during upload (received {} bytes): {}",
total_size, e
));
}
}
}
tokio::io::AsyncWriteExt::flush(&mut writer)
.await
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
Ok(())
}
.await;
if let Err(e) = spool_result {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("❌ UPLOAD SPOOL FAILED: {} - {}", filename, e);
return Err(Self::domain_error_response(
crate::common::errors::DomainError::internal_error("FileUpload", e),
));
}
// Empty file — use streaming path with the (empty) temp file
if total_size == 0 {
let hash = hasher.finalize().to_hex().to_string();
let dto = upload_service
.upload_file_streaming(
filename,
folder_id,
content_type,
&temp_path,
0,
Some(hash.clone()),
)
.await
.map_err(Self::domain_error_response)?;
return Ok((dto, hash));
}
// Finalize hash
let hash = hasher.finalize().to_hex().to_string();
// ── MIME detection (magic bytes + extension fallback) ─
let content_type = crate::common::mime_detect::refine_content_type_from_file(
&temp_path,
// ── Stream the field into the CDC chunk store ────────
// Chunking (FastCDC) + hashing (BLAKE3) + dedup checks +
// MIME sniffing all happen while the bytes arrive; chunks
// the store already has never touch the disk. Size is
// capped globally by DefaultBodyLimit.
let dedup = &state.core.dedup_service;
let source = upload_ingest::multipart_field_stream(field);
let ingested = match upload_ingest::ingest_stream_to_cas(
source,
dedup,
&filename,
&content_type,
usize::MAX,
None,
)
.await;
.await
{
Ok(ingested) => ingested,
Err(e) => {
tracing::error!("❌ UPLOAD INGEST FAILED: {} - {}", filename, e.message);
return Err(e.into_response());
}
};
// ── Quota enforcement ────────────────────────────────
// ── Quota enforcement (exact size now known) ─────────
if let Some(storage_svc) = state.storage_usage_service.as_ref()
&& let Err(err) = storage_svc
.check_storage_quota(auth_user.id, total_size)
.check_storage_quota(auth_user.id, ingested.size)
.await
{
let _ = tokio::fs::remove_file(&temp_path).await;
upload_ingest::discard_ingested(dedup, &ingested).await;
tracing::warn!(
"⛔ UPLOAD REJECTED (quota): user={}, file={}, size={}",
auth_user.username,
filename,
total_size
ingested.size
);
return Err(Self::quota_error_response(err));
}
// ── Streaming upload (temp file → blob store, hash pre-computed) ─
// ── Register the file row against the ingested blob ──
let hash = ingested.hash.clone();
let size = ingested.size;
match upload_service
.upload_file_streaming(
filename.clone(),
folder_id,
content_type,
&temp_path,
total_size,
Some(hash.clone()),
ingested.content_type.clone(),
ingested.stored(),
)
.await
{
@@ -279,13 +216,12 @@ impl FileHandler {
tracing::info!(
"✅ STREAMING UPLOAD: {} ({} bytes, ID: {})",
filename,
total_size,
size,
file.id
);
return Ok((file, hash));
}
Err(err) => {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("❌ UPLOAD FAILED: {} - {}", filename, err);
return Err(Self::domain_error_response(err));
}
src/interfaces/api/handlers/webdav_handler.rs
+20 -29
View File
@@ -898,10 +898,11 @@ async fn handle_head(
/**
* Handles PUT requests to create or update files.
*
* **Streaming implementation**: the request body is spooled to a temp file
* with incremental BLAKE3 hashing. Peak RAM usage is ~256 KB regardless
* of file size. The temp file is then atomically moved into blob storage
* via `update_file_streaming`.
* **Streaming implementation**: the request body is streamed straight into
* the CDC chunk store (FastCDC + BLAKE3 while the bytes arrive — no spool
* file, no re-read; peak RAM is bounded regardless of file size), then the
* file row is atomically swapped onto the ingested blob via
* `update_file_streaming`.
*
* @param state The application state containing service dependencies
* @param path The requested resource path
@@ -913,7 +914,7 @@ async fn handle_put(
req: Request<Body>,
path: String,
) -> Result<Response<Body>, AppError> {
use crate::interfaces::upload_spool::spool_body_to_temp;
use crate::interfaces::upload_ingest;
let user = extract_user(&req)?;
@@ -972,32 +973,31 @@ async fn handle_put(
.unwrap_or("application/octet-stream")
.to_string();
// ── Streaming spool: body → temp file + incremental hash ──
// Shared with the NextCloud-compat PUT handler; peak heap ~one frame
// regardless of file size. Honors `upload_temp_dir` to keep the spool
// off tmpfs/RAM.
let spooled = spool_body_to_temp(
// ── Streaming ingest: body → CDC chunk store ──────────────
// Shared with the NextCloud-compat PUT handler; chunking + hashing +
// dedup checks run while the body arrives — no spool file, no re-read.
let filename = crate::common::mime_detect::filename_from_path(&path).to_string();
let ingested = upload_ingest::ingest_body_to_cas(
req.into_body(),
&state.core.dedup_service,
&filename,
&content_type,
max_upload,
state.core.config.storage.upload_temp_dir.clone(),
)
.await?;
let temp_path = spooled.temp.path().to_path_buf();
let total_bytes = spooled.size as usize;
let hash = spooled.hash;
// ── Quota enforcement ────────────────────────────────────
if let Some(storage_svc) = state.storage_usage_service.as_ref()
&& let Err(err) = storage_svc
.check_storage_quota(user.id, total_bytes as u64)
.check_storage_quota(user.id, ingested.size)
.await
{
let _ = tokio::fs::remove_file(&temp_path).await;
upload_ingest::discard_ingested(&state.core.dedup_service, &ingested).await;
tracing::warn!(
"⛔ WEBDAV PUT REJECTED (quota): user={}, file={}, size={}",
user.id,
path,
total_bytes
ingested.size
);
return Err(AppError::new(
StatusCode::INSUFFICIENT_STORAGE,
@@ -1006,21 +1006,12 @@ async fn handle_put(
));
}
// ── Atomic store: temp file → dedup blob + DB metadata update ──
// ── Atomic store: swap the file row onto the ingested blob ──
let content_type = ingested.content_type.clone();
let result = file_upload_service
.update_file_streaming(
&path,
&temp_path,
total_bytes as u64,
&content_type,
Some(hash),
None,
)
.update_file_streaming(&path, ingested.stored(), &content_type, None)
.await;
// Clean up temp file (may already be moved by dedup, ignore error)
let _ = tokio::fs::remove_file(&temp_path).await;
match result {
Ok(_file_dto) => Ok(Response::builder()
.status(StatusCode::NO_CONTENT)
src/interfaces/api/handlers/wopi_handler.rs
+17 -64
View File
@@ -165,10 +165,6 @@ async fn put_file(
State(state): State<WopiState>,
req: Request<Body>,
) -> Response {
use http_body_util::BodyStream;
use tokio::io::AsyncWriteExt;
use tokio_stream::StreamExt;
let claims = match state
.token_service
.validate_token(&token_query.access_token)
@@ -218,75 +214,32 @@ async fn put_file(
Err(_) => return StatusCode::NOT_FOUND.into_response(),
};
// ── Streaming spool: body → temp file + incremental BLAKE3 ──
let temp_file = match tempfile::NamedTempFile::new() {
Ok(f) => f,
Err(e) => {
tracing::error!("WOPI PutFile: failed to create temp file: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
let temp_path = temp_file.path().to_path_buf();
let mut file_out = match tokio::fs::File::create(&temp_path).await {
Ok(f) => f,
Err(e) => {
tracing::error!("WOPI PutFile: failed to open temp file: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
// ── Streaming ingest: body → CDC chunk store (no temp file) ──
let content_type = file.mime_type.clone();
let mut hasher = blake3::Hasher::new();
let mut total_bytes: u64 = 0;
let mut stream = BodyStream::new(req.into_body());
while let Some(frame_result) = stream.next().await {
let frame = match frame_result {
Ok(f) => f,
Err(e) => {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("WOPI PutFile: body read error: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len() as u64;
hasher.update(chunk);
if let Err(e) = file_out.write_all(chunk).await {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("WOPI PutFile: temp write error: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
let ingested = match crate::interfaces::upload_ingest::ingest_body_to_cas(
req.into_body(),
&state.app_state.core.dedup_service,
&file.name,
&content_type,
usize::MAX,
)
.await
{
Ok(ingested) => ingested,
Err(e) => {
tracing::error!("WOPI PutFile: ingest failed: {}", e.message);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
}
if let Err(e) = file_out.flush().await {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("WOPI PutFile: flush error: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
drop(file_out);
};
let hash = hasher.finalize().to_hex().to_string();
// ── Atomic store: temp file → dedup blob + DB metadata update ──
// ── Atomic store: swap the file row onto the ingested blob ──
let result = state
.app_state
.applications
.file_upload_service
.update_file_streaming(
&file.path,
&temp_path,
total_bytes,
&content_type,
Some(hash),
None,
)
.update_file_streaming(&file.path, ingested.stored(), &content_type, None)
.await;
// Clean up temp file (may already be moved by dedup, ignore error)
let _ = tokio::fs::remove_file(&temp_path).await;
match result {
Ok(_file_dto) => StatusCode::OK.into_response(),
Err(e) => {
src/interfaces/mod.rs
+1 -1
View File
@@ -3,7 +3,7 @@ pub mod errors;
pub mod middleware;
pub mod nextcloud;
pub mod range_requests;
pub mod upload_spool;
pub mod upload_ingest;
pub mod web;
pub use api::create_api_routes;
src/interfaces/nextcloud/uploads_handler.rs
+38 -39
View File
@@ -7,10 +7,12 @@ use std::sync::Arc;
use crate::application::ports::file_ports::{FileRetrievalUseCase, FileUploadUseCase};
use crate::common::di::AppState;
use crate::common::mime_detect::{filename_from_path, refine_content_type_from_file};
use crate::common::mime_detect::filename_from_path;
use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
use crate::interfaces::upload_spool::stream_body_to_path;
use crate::interfaces::upload_ingest::{
discard_ingested, ingest_stream_to_cas, stream_body_to_path, stream_from_files,
};
/// Dispatch Nextcloud chunked upload WebDAV requests.
///
@@ -239,18 +241,17 @@ async fn handle_assemble(
let dest_subpath = extract_files_subpath(&destination, &user.username)
.ok_or_else(|| AppError::bad_request("Invalid Destination URL"))?;
// Assemble chunks into a temp file with hash-on-write (BLAKE3 computed
// during the same read/write loop that copies chunks into the
// assembled file). The hash is passed downstream as `pre_computed_hash`
// so the dedup layer never re-reads the assembled file just to compute
// it — saves one full file-sized read pass per upload.
let (temp_path, size, blake3_hash) = nc
// Stream the chunk parts, in order, straight into the CDC chunk store —
// no assembled temp file is ever written. Chunking (FastCDC), BLAKE3
// hashing, dedup checks and MIME sniffing (magic bytes off the first
// part) all happen in that single read pass. The parts stay on disk
// until the session cleanup below, so a failed completion is retryable.
let chunk_paths = nc
.chunked_uploads
.assemble(&user.username, upload_id)
.ordered_chunk_paths(&user.username, upload_id)
.await
.map_err(|e| AppError::internal_error(format!("Failed to assemble chunks: {}", e)))?;
.map_err(|e| AppError::internal_error(format!("Failed to list chunks: {}", e)))?;
// Write assembled file to storage via the upload service.
let upload_service = &state.applications.file_upload_service;
let file_service = &state.applications.file_retrieval_service;
let folder_service = &state.applications.folder_service;
@@ -261,35 +262,31 @@ async fn handle_assemble(
dest_subpath.trim_matches('/')
);
// Detect content type via magic bytes + extension fallback.
let filename = filename_from_path(&dest_subpath);
let content_type =
refine_content_type_from_file(&temp_path, filename, "application/octet-stream").await;
let filename = filename_from_path(&dest_subpath).to_string();
let ingested = ingest_stream_to_cas(
stream_from_files(chunk_paths),
&state.core.dedup_service,
&filename,
"application/octet-stream",
usize::MAX,
None,
)
.await?;
let content_type = ingested.content_type.clone();
// Check if file exists (update vs create).
let existing = file_service.get_file_by_path(&internal_path).await;
let etag: Option<String> = if existing.is_ok() {
let dto = upload_service
.update_file_streaming(
&internal_path,
&temp_path,
size,
&content_type,
Some(blake3_hash.clone()),
oc_mtime,
)
.update_file_streaming(&internal_path, ingested.stored(), &content_type, oc_mtime)
.await
.map_err(|e| AppError::internal_error(format!("Failed to update file: {}", e)))?;
Some(dto.etag)
} else {
// New-file branch: resolve the parent folder by path and pass the
// assembled file's path directly to `upload_file_from_path` so the
// bytes never get read back into RAM. Previously this branch did
// `tokio::fs::read(&temp_path)` — an extra full file-sized read
// pass AND a peak-RAM allocation equal to the upload size, which
// defeated the streaming model on large NC uploads.
// New-file branch: resolve the parent folder by path and register
// the file row against the already-ingested blob.
let (parent_sub, filename) = match dest_subpath.rsplit_once('/') {
Some((p, n)) => (p, n),
None => ("", dest_subpath.as_str()),
@@ -302,18 +299,23 @@ async fn handle_assemble(
let parent_internal = parent_internal.trim_end_matches('/');
use crate::application::ports::folder_ports::FolderUseCase;
let parent_folder = folder_service
.get_folder_by_path(parent_internal)
.await
.map_err(|e| AppError::internal_error(format!("Parent folder lookup failed: {}", e)))?;
let parent_folder = match folder_service.get_folder_by_path(parent_internal).await {
Ok(folder) => folder,
Err(e) => {
discard_ingested(&state.core.dedup_service, &ingested).await;
return Err(AppError::internal_error(format!(
"Parent folder lookup failed: {}",
e
)));
}
};
let dto = upload_service
.upload_file_from_path(
.upload_file_streaming(
filename.to_string(),
Some(parent_folder.id),
content_type.to_string(),
&temp_path,
Some(blake3_hash),
ingested.stored(),
)
.await
.map_err(|e| AppError::internal_error(format!("Failed to create file: {}", e)))?;
@@ -321,9 +323,6 @@ async fn handle_assemble(
Some(dto.etag)
};
// Clean up temp file (session cleanup below removes the directory anyway).
let _ = tokio::fs::remove_file(&temp_path).await;
// Cleanup session.
let _ = nc.chunked_uploads.cleanup(&user.username, upload_id).await;
src/interfaces/nextcloud/webdav_handler.rs
+19 -30
View File
@@ -22,12 +22,12 @@ use crate::application::ports::file_ports::{
use crate::application::ports::folder_ports::FolderUseCase;
use crate::application::ports::trash_ports::TrashUseCase;
use crate::common::di::AppState;
use crate::common::mime_detect::{filename_from_path, refine_content_type_from_file};
use crate::common::mime_detect::filename_from_path;
use crate::interfaces::api::handlers::webdav_handler::PROPFIND_BATCH_SIZE;
use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
use crate::interfaces::range_requests::{not_modified_response, range_response};
use crate::interfaces::upload_spool::spool_body_to_temp;
use crate::interfaces::upload_ingest::ingest_body_to_cas;
/// Extension trait to map XML write errors to `String` concisely.
trait XmlResultExt<T> {
@@ -575,46 +575,35 @@ async fn handle_put(
// at 95 % loses everything.
let max_upload = state.core.config.storage.direct_put_max_bytes;
// Stream the body to a temp file + incremental hash — never buffer the
// full upload in RAM. The old `body::to_bytes` path loaded the entire
// file (e.g. an 800 MB ISO) into anonymous memory before any dedup logic,
// OOMKilling the process even on dedup hits. Shared with the native
// WebDAV PUT handler; peak heap ~one frame regardless of file size.
let spooled = spool_body_to_temp(
req.into_body(),
max_upload,
state.core.config.storage.upload_temp_dir.clone(),
)
.await?;
// Detect real MIME type from the first bytes on disk (no full read).
// Stream the body straight into the CDC chunk store — never buffer the
// full upload in RAM and never spool it to disk. Chunking, hashing,
// dedup checks and MIME sniffing (magic bytes off the first frames)
// all run while the body arrives; chunks the store already has are
// never written at all. Shared with the native WebDAV PUT handler.
//
// `filename` is owned so we don't hold a borrow of the `subpath` param
// across the await (which would make the handler future non-Send).
let filename = filename_from_path(subpath).to_string();
let content_type =
refine_content_type_from_file(spooled.temp.path(), &filename, &claimed_type).await;
let ingested = ingest_body_to_cas(
req.into_body(),
&state.core.dedup_service,
&filename,
&claimed_type,
max_upload,
)
.await?;
let content_type = ingested.content_type.clone();
// Distinguish create (201) vs update (204) for the response status.
let existed = file_service.get_file_by_path(&internal_path).await.is_ok();
// Single streaming path — handles both update and create internally,
// passing the precomputed hash so the dedup fast path can short-circuit
// without re-reading the file.
// swapping the file row onto the already-ingested blob.
let stored = upload_service
.update_file_streaming(
&internal_path,
spooled.temp.path(),
spooled.size,
&content_type,
Some(spooled.hash),
oc_mtime,
)
.update_file_streaming(&internal_path, ingested.stored(), &content_type, oc_mtime)
.await
.map_err(|e| AppError::internal_error(format!("Failed to store file: {}", e)))?;
// dedup may have already moved the temp on a new-blob store; ignore error.
let _ = tokio::fs::remove_file(spooled.temp.path()).await;
let status = if existed {
StatusCode::NO_CONTENT
} else {
src/interfaces/upload_ingest.rs
+588
View File
@@ -0,0 +1,588 @@
//! Shared streaming upload ingestion: request body → CDC chunk store.
//!
//! Used by every upload surface (REST multipart, native WebDAV PUT,
//! NextCloud PUT, chunked-upload assembly, WOPI PutFile) so none of them
//! buffers the full body in RAM **or spools it to a temp file**. The bytes
//! flow straight into [`DedupService::store_from_stream`], which chunks
//! (FastCDC), hashes (BLAKE3) and dedup-checks them while they arrive —
//! each uploaded byte touches the disk at most once, and not at all when
//! the store already has its chunk.
//!
//! MIME refinement happens in-flight: when the claimed Content-Type is
//! generic, the first bytes are peeked off the stream for magic-byte
//! detection before being forwarded unchanged.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::Mutex as StdMutex;
use std::sync::atomic::{AtomicBool, Ordering};
use axum::body::Body;
use bytes::Bytes;
use futures::stream::{self, Stream, StreamExt, TryStreamExt};
use http_body_util::BodyStream;
// The `Digest` trait (re-exported by both `md5` and `sha2` from the
// `digest` crate) gives `Md5` and `Sha256` their `new` / `update` /
// `finalize` methods. Importing once via `sha2` covers both —
// otherwise every call site would need fully-qualified
// `<md5::Md5 as md5::Digest>::…` syntax.
use sha2::Digest as _;
use tokio::io::AsyncWriteExt;
use tokio_util::io::ReaderStream;
use crate::application::ports::chunked_upload_ports::ChecksumAlg;
use crate::application::ports::file_ports::StoredBlob;
use crate::common::mime_detect::{MAGIC_BYTES_LEN, is_generic_mime, refine_content_type};
use crate::infrastructure::services::dedup_service::DedupService;
use crate::interfaces::errors::AppError;
/// Content stored in the chunk store by one upload ingest.
///
/// The ingest holds ONE blob reference; pass [`IngestedBlob::stored`] to the
/// upload service (which takes ownership of the reference) or hand it back
/// via [`discard_ingested`] when the upload is rejected after the fact.
pub struct IngestedBlob {
/// BLAKE3 of the full content (the blob/manifest key).
pub hash: String,
/// Total bytes ingested.
pub size: u64,
/// Refined content type (claimed type or magic-byte detection).
pub content_type: String,
/// `false` when the exact content already existed (dedup hit).
pub is_new_blob: bool,
/// Bytes that did not need to be transferred to storage (dedup hit).
pub bytes_saved: u64,
}
impl IngestedBlob {
/// The blob reference to hand to the upload service.
pub fn stored(&self) -> StoredBlob {
StoredBlob {
hash: self.hash.clone(),
size: self.size,
is_new_blob: self.is_new_blob,
}
}
}
/// Hand back the blob reference taken by a successful ingest when the upload
/// is rejected after the fact (quota exceeded, checksum mismatch, …).
pub async fn discard_ingested(dedup: &DedupService, blob: &IngestedBlob) {
if let Err(e) = dedup.remove_reference(&blob.hash).await {
tracing::warn!(
"Failed to release blob reference of rejected upload {}: {e}",
&blob.hash[..blob.hash.len().min(12)]
);
}
}
/// Shared mutable tee for computing a client-requested checksum during the
/// ingest pass (REST chunked uploads) — no post-store re-read needed.
pub type ChecksumTee = Arc<StdMutex<Option<IncrementalHasher>>>;
/// Create a checksum tee for [`ingest_stream_to_cas`].
pub fn checksum_tee(alg: ChecksumAlg) -> ChecksumTee {
Arc::new(StdMutex::new(Some(IncrementalHasher::new(alg))))
}
/// Finalize a checksum tee into its lowercase hex digest.
pub fn finalize_checksum_tee(tee: &ChecksumTee) -> Option<String> {
tee.lock()
.ok()
.and_then(|mut h| h.take())
.map(IncrementalHasher::finalize_hex)
}
/// Out-of-band state observed by the stream adapters while the dedup engine
/// consumes the stream — lets the caller map an opaque engine error back to
/// the precise HTTP failure (413 vs 400).
struct IngestFlags {
too_large: AtomicBool,
source_error: StdMutex<Option<String>>,
}
/// Stream a request body (or any byte stream) into the CDC chunk store.
///
/// Single pass: size-cap enforcement, optional checksum tee, MIME sniffing
/// (first [`MAGIC_BYTES_LEN`] bytes, only when `claimed_type` is generic)
/// and the CDC chunk/hash/store pipeline all run while the bytes arrive.
/// Peak heap is bounded by the dedup engine (~9 MiB) regardless of size.
///
/// On error nothing stays referenced — the engine compensates internally.
pub async fn ingest_stream_to_cas<S, E>(
source: S,
dedup: &Arc<DedupService>,
filename: &str,
claimed_type: &str,
max_bytes: usize,
checksum: Option<ChecksumTee>,
) -> Result<IngestedBlob, AppError>
where
S: Stream<Item = Result<Bytes, E>> + Send,
E: std::fmt::Display,
{
let flags = Arc::new(IngestFlags {
too_large: AtomicBool::new(false),
source_error: StdMutex::new(None),
});
// ── Adapter: cap + checksum tee + error capture ──────────────
let adapter_flags = flags.clone();
let mut total: usize = 0;
let counted = source.map(move |item| match item {
Ok(bytes) => {
total += bytes.len();
if total > max_bytes {
adapter_flags.too_large.store(true, Ordering::Relaxed);
return Err(std::io::Error::other("upload exceeds size cap"));
}
if let Some(tee) = &checksum
&& let Ok(mut hasher) = tee.lock()
&& let Some(hasher) = hasher.as_mut()
{
hasher.update(&bytes);
}
Ok(bytes)
}
Err(e) => {
let message = e.to_string();
if let Ok(mut slot) = adapter_flags.source_error.lock() {
*slot = Some(message.clone());
}
Err(std::io::Error::other(message))
}
});
// `fuse` is load-bearing: when the source is shorter than the MIME peek
// (< MAGIC_BYTES_LEN), the peek loop drains it to None and the `chain`
// below polls it once more — non-fused sources (e.g. `stream::unfold`,
// as used for multipart fields) panic on a post-None poll.
let mut counted = Box::pin(counted.fuse());
// ── In-flight MIME sniff (only when the claimed type is generic) ──
let mut head: Vec<Result<Bytes, std::io::Error>> = Vec::new();
let content_type = if is_generic_mime(claimed_type) {
let mut head_len = 0usize;
while head_len < MAGIC_BYTES_LEN {
match counted.next().await {
Some(Ok(bytes)) => {
head_len += bytes.len();
head.push(Ok(bytes));
}
Some(Err(e)) => {
head.push(Err(e));
break;
}
None => break,
}
}
let mut magic = Vec::with_capacity(head_len.min(MAGIC_BYTES_LEN));
for item in head.iter().flatten() {
let take = (MAGIC_BYTES_LEN - magic.len()).min(item.len());
magic.extend_from_slice(&item[..take]);
if magic.len() >= MAGIC_BYTES_LEN {
break;
}
}
refine_content_type(&magic, filename, claimed_type)
} else {
claimed_type.to_string()
};
// ── Store: peeked head + remainder, one continuous stream ────
let full_stream = stream::iter(head).chain(counted);
let result = dedup
.store_from_stream(full_stream, Some(content_type.clone()))
.await;
match result {
Ok(stored) => {
let is_new_blob = !stored.was_deduplicated();
let bytes_saved = match &stored {
crate::application::ports::dedup_ports::DedupResultDto::ExistingBlob {
saved_bytes,
..
} => *saved_bytes,
_ => 0,
};
Ok(IngestedBlob {
hash: stored.hash().to_string(),
size: stored.size(),
content_type,
is_new_blob,
bytes_saved,
})
}
Err(e) => {
if flags.too_large.load(Ordering::Relaxed) {
return Err(AppError::payload_too_large(format!(
"Upload body exceeds the direct-PUT cap ({max_bytes} bytes). \
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
Chunked uploads are resumable on transient failure."
)));
}
let source_error = flags.source_error.lock().ok().and_then(|s| s.clone());
if let Some(message) = source_error {
return Err(AppError::bad_request(format!(
"Failed to read request body: {message}"
)));
}
Err(AppError::from(e))
}
}
}
/// [`ingest_stream_to_cas`] for an HTTP request body.
pub async fn ingest_body_to_cas(
body: Body,
dedup: &Arc<DedupService>,
filename: &str,
claimed_type: &str,
max_bytes: usize,
) -> Result<IngestedBlob, AppError> {
let source = BodyStream::new(body).filter_map(|item| async move {
match item {
Ok(frame) => frame.into_data().ok().map(Ok),
Err(e) => Some(Err(e)),
}
});
ingest_stream_to_cas(source, dedup, filename, claimed_type, max_bytes, None).await
}
/// Adapt a multipart field into a byte stream for [`ingest_stream_to_cas`].
///
/// Terminates after the first error — multipart fields are not resumable.
pub fn multipart_field_stream(
field: axum::extract::multipart::Field<'_>,
) -> impl Stream<Item = Result<Bytes, axum::extract::multipart::MultipartError>> + Send + '_ {
stream::unfold((field, false), |(mut field, done)| async move {
if done {
return None;
}
match field.chunk().await {
Ok(Some(bytes)) => Some((Ok(bytes), (field, false))),
Ok(None) => None,
Err(e) => Some((Err(e), (field, true))),
}
})
}
/// Concatenate already-uploaded chunk part files into one byte stream, in
/// the given order — feeds chunked-upload assembly into the CDC store
/// without ever materializing an assembled file on disk.
pub fn stream_from_files(
paths: Vec<PathBuf>,
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
.and_then(|path| async move {
tokio::fs::File::open(path)
.await
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
})
.try_flatten()
}
/// Result of a streamed write to a caller-supplied path.
pub struct StreamedToPath {
/// Total bytes written.
pub bytes_written: u64,
/// Lowercase hex digest, populated only when `checksum_alg=Some(_)`
/// was passed. The algorithm is identified by [`StreamedToPath::alg`].
pub checksum_hex: Option<String>,
/// Algorithm used to compute `checksum_hex`. Echoed back so the
/// caller can include it in audit logs or response headers.
pub alg: Option<ChecksumAlg>,
}
/// Stream an HTTP request body directly to a known destination file,
/// enforcing `max_bytes` as a hard size limit.
///
/// Used by the chunked-upload PUT handlers — each chunk has a
/// deterministic on-disk path (`NextcloudChunkedUploadService::safe_chunk_path`
/// for the NC surface, `ChunkedUploadService::prepare_chunk` for the
/// REST surface), so there's no spool/move dance. Peak heap is ~one
/// HTTP frame regardless of chunk size or `max_bytes`.
///
/// `checksum_alg` is the optional client-requested integrity check
/// (default `md5` per the legacy `Content-MD5` contract; `blake3`
/// available for forward-compat). When `Some`, the hash is computed
/// incrementally during streaming — no extra disk read for verification.
///
/// On size overflow the partial file is removed before the function
/// returns, so a client retry against the same chunk name starts from
/// a clean slate. On any other I/O error the partial file is also
/// removed and the error surfaces — callers can assume the path is
/// either fully written or absent.
pub async fn stream_body_to_path(
body: Body,
path: &Path,
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
let mut hasher = checksum_alg.map(IncrementalHasher::new);
while let Some(frame_result) = stream.next().await {
let frame = match frame_result {
Ok(f) => f,
Err(e) => {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::bad_request(format!(
"Failed to read request body: {e}"
)));
}
};
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_bytes {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::payload_too_large(format!(
"Chunk exceeds maximum size of {max_bytes} bytes"
)));
}
if let Some(h) = hasher.as_mut() {
h.update(chunk);
}
if let Err(e) = file.write_all(chunk).await {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::internal_error(format!(
"Failed to write chunk: {e}"
)));
}
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush chunk file: {e}")))?;
drop(file);
Ok(StreamedToPath {
bytes_written: total_bytes as u64,
checksum_hex: hasher.map(IncrementalHasher::finalize_hex),
alg: checksum_alg,
})
}
/// Algorithm-agnostic incremental hasher used by [`stream_body_to_path`]
/// and the [`ChecksumTee`] of chunked-upload completion.
/// Per-frame `update` is sub-millisecond for all three algorithms at the
/// 64 KB frame sizes axum's body stream produces, so we don't need
/// `spawn_blocking` (which the old buffered path used because it hashed
/// the full multi-MB chunk in one shot).
pub enum IncrementalHasher {
Md5(md5::Md5),
Sha256(sha2::Sha256),
// Boxing — blake3::Hasher is ~1.7 KB on the stack while md5::Md5
// (~100 bytes) and sha2::Sha256 (~100 bytes) are tiny; boxing the
// outlier keeps the enum size proportional to the common case
// rather than the worst case.
Blake3(Box<blake3::Hasher>),
}
impl IncrementalHasher {
fn new(alg: ChecksumAlg) -> Self {
match alg {
ChecksumAlg::Md5 => Self::Md5(md5::Md5::new()),
ChecksumAlg::Sha256 => Self::Sha256(sha2::Sha256::new()),
ChecksumAlg::Blake3 => Self::Blake3(Box::new(blake3::Hasher::new())),
}
}
fn update(&mut self, bytes: &[u8]) {
match self {
Self::Md5(h) => h.update(bytes),
Self::Sha256(h) => h.update(bytes),
Self::Blake3(h) => {
h.update(bytes);
}
}
}
fn finalize_hex(self) -> String {
match self {
Self::Md5(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Sha256(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Blake3(h) => h.finalize().to_hex().to_string(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use bytes::Bytes;
#[tokio::test]
async fn stream_body_to_path_caps_oversized() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
// 5 MiB body, 4 MiB cap → must reject.
let body = Body::from(Bytes::from(vec![0u8; 5 * 1024 * 1024]));
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
assert!(
result.is_err(),
"expected PayloadTooLarge, got Ok(bytes_written={})",
result.ok().map(|r| r.bytes_written).unwrap_or(0)
);
// Partial file must be removed on rejection.
assert!(
!path.exists(),
"rejected chunk file should be removed, but {} still exists",
path.display()
);
}
#[tokio::test]
async fn stream_body_to_path_accepts_under_cap() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 1024 * 1024])); // 1 MiB
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
let outcome = result.expect("should succeed");
assert_eq!(outcome.bytes_written, 1024 * 1024);
assert!(outcome.checksum_hex.is_none(), "no alg requested → no hash");
assert!(path.exists());
}
#[tokio::test]
async fn stream_body_to_path_caps_at_exact_boundary() {
// Edge case: body exactly equal to cap should succeed; cap+1 must fail.
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 100]));
let outcome = stream_body_to_path(body, &path, 100, None)
.await
.expect("100 bytes at 100-byte cap should succeed");
assert_eq!(outcome.bytes_written, 100);
let path2 = temp_dir.path().join("chunk2");
let body = Body::from(Bytes::from(vec![1u8; 101]));
assert!(
stream_body_to_path(body, &path2, 100, None).await.is_err(),
"101 bytes at 100-byte cap must reject"
);
assert!(!path2.exists());
}
#[tokio::test]
async fn ingest_rejects_oversized_before_touching_storage() {
// 1 KiB body against a 100-byte cap: the adapter must abort the
// stream before any flush, so the stub dedup service (which cannot
// reach PG) is never asked to settle a batch.
let dedup = Arc::new(DedupService::new_stub());
let source = stream::iter(vec![Ok::<_, std::io::Error>(Bytes::from(vec![0u8; 1024]))]);
let result = ingest_stream_to_cas(
source,
&dedup,
"file.bin",
"application/octet-stream",
100,
None,
)
.await;
let err = result.err().expect("oversized body must be rejected");
assert_eq!(err.status_code, axum::http::StatusCode::PAYLOAD_TOO_LARGE);
}
#[tokio::test]
async fn ingest_surfaces_source_errors_as_bad_request() {
let dedup = Arc::new(DedupService::new_stub());
let source = stream::iter(vec![
Ok::<_, std::io::Error>(Bytes::from_static(b"partial")),
Err(std::io::Error::other("connection reset by peer")),
]);
let result =
ingest_stream_to_cas(source, &dedup, "file.bin", "text/plain", usize::MAX, None).await;
let err = result.err().expect("source error must surface");
assert_eq!(err.status_code, axum::http::StatusCode::BAD_REQUEST);
assert!(
err.message.contains("connection reset by peer"),
"original cause must be preserved: {}",
err.message
);
}
/// Regression: a source shorter than the MIME peek (< MAGIC_BYTES_LEN)
/// is drained to None during sniffing and then polled once more by the
/// `chain` that re-attaches the peeked head. Non-fused sources — like
/// the `stream::unfold` used for multipart fields — panic on that
/// post-None poll ("Unfold must not be polled after it returned
/// `Poll::Ready(None)`") unless the ingest fuses the stream first.
/// The stub dedup service can't reach PG, so an orderly `Err` (not a
/// panic) proves the stream layer survived.
#[tokio::test]
async fn ingest_short_stream_with_generic_mime_does_not_repoll_source() {
let dedup = Arc::new(DedupService::new_stub());
let source = stream::unfold(false, |done| async move {
if done {
None
} else {
Some((Ok::<_, std::io::Error>(Bytes::from_static(b"tiny")), true))
}
});
let result = ingest_stream_to_cas(
source,
&dedup,
"tiny.bin",
"application/octet-stream",
usize::MAX,
None,
)
.await;
assert!(
result.is_err(),
"stub DB must reject the store — but only AFTER the stream \
layer survived the post-peek poll"
);
}
#[tokio::test]
async fn stream_from_files_concatenates_in_order() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let a = temp_dir.path().join("a");
let b = temp_dir.path().join("b");
tokio::fs::write(&a, b"Hello, ").await.unwrap();
tokio::fs::write(&b, b"World!").await.unwrap();
let mut out = Vec::new();
let s = stream_from_files(vec![a, b]);
futures::pin_mut!(s);
while let Some(chunk) = s.next().await {
out.extend_from_slice(&chunk.expect("read"));
}
assert_eq!(out, b"Hello, World!");
}
#[tokio::test]
async fn checksum_tee_roundtrip() {
let tee = checksum_tee(ChecksumAlg::Md5);
if let Ok(mut h) = tee.lock()
&& let Some(h) = h.as_mut()
{
h.update(b"hello world");
}
let hex = finalize_checksum_tee(&tee).expect("digest");
assert_eq!(hex, "5eb63bbbe01eeed093cb22bb8f5acdc3");
assert!(
finalize_checksum_tee(&tee).is_none(),
"second finalize returns None"
);
}
}
src/interfaces/upload_spool.rs
-289
View File
@@ -1,289 +0,0 @@
//! Shared streaming upload spool: request body → temp file + incremental hash.
//!
//! Used by both the native WebDAV PUT handler and the NextCloud-compat PUT
//! handler so neither buffers the full request body in memory. Peak heap is
//! ~one HTTP frame regardless of file size; the body is written to a temp
//! file (off tmpfs when [`StorageConfig::upload_temp_dir`] is configured) and
//! BLAKE3-hashed on the fly so the dedup layer can short-circuit on a hit.
use std::path::{Path, PathBuf};
use axum::body::Body;
use http_body_util::BodyStream;
// The `Digest` trait (re-exported by both `md5` and `sha2` from the
// `digest` crate) gives `Md5` and `Sha256` their `new` / `update` /
// `finalize` methods. Importing once via `sha2` covers both —
// otherwise every call site would need fully-qualified
// `<md5::Md5 as md5::Digest>::…` syntax.
use sha2::Digest as _;
use tempfile::NamedTempFile;
use tokio::io::AsyncWriteExt;
use tokio_stream::StreamExt;
use crate::application::ports::chunked_upload_ports::ChecksumAlg;
use crate::common::temp::new_spool_temp_file;
use crate::interfaces::errors::AppError;
/// Outcome of spooling a request body to disk.
pub struct SpooledBody {
/// The temp file holding the body. Kept alive by the caller (dropping it
/// removes the file unless the dedup layer already consumed/moved it).
pub temp: NamedTempFile,
/// Hex-encoded BLAKE3 of the full body — matches `DedupService::hash_file`,
/// so passing it as `pre_computed_hash` enables the dedup fast path.
pub hash: String,
/// Total bytes written.
pub size: u64,
}
/// Stream an HTTP request body to a temp file, computing its BLAKE3 hash
/// incrementally and enforcing `max_upload` as a hard size limit.
///
/// Peak heap is ~one frame — the body is never fully buffered in RAM.
///
/// `temp_dir` is taken by value (not `&Path`) so the returned future captures
/// no borrowed lifetime — required for the handler future to stay `Send`.
pub async fn spool_body_to_temp(
body: Body,
max_upload: usize,
temp_dir: Option<PathBuf>,
) -> Result<SpooledBody, AppError> {
let temp = new_spool_temp_file(temp_dir.as_deref())
.map_err(|e| AppError::internal_error(format!("Failed to create temp file: {e}")))?;
let temp_path = temp.path().to_path_buf();
let mut file = tokio::fs::File::create(&temp_path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open temp file: {e}")))?;
let mut hasher = blake3::Hasher::new();
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
while let Some(frame_result) = stream.next().await {
let frame = frame_result
.map_err(|e| AppError::bad_request(format!("Failed to read request body: {e}")))?;
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_upload {
// Abort early — stop reading, delete temp file.
drop(file);
let _ = tokio::fs::remove_file(&temp_path).await;
return Err(AppError::payload_too_large(format!(
"Upload body exceeds the direct-PUT cap ({max_upload} bytes). \
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
Chunked uploads are resumable on transient failure."
)));
}
hasher.update(chunk);
file.write_all(chunk).await.map_err(|e| {
AppError::internal_error(format!("Failed to write to temp file: {e}"))
})?;
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush temp file: {e}")))?;
drop(file);
let hash = hasher.finalize().to_hex().to_string();
Ok(SpooledBody {
temp,
hash,
size: total_bytes as u64,
})
}
/// Result of a streamed write to a caller-supplied path.
pub struct StreamedToPath {
/// Total bytes written.
pub bytes_written: u64,
/// Lowercase hex digest, populated only when `checksum_alg=Some(_)`
/// was passed. The algorithm is identified by [`StreamedToPath::alg`].
pub checksum_hex: Option<String>,
/// Algorithm used to compute `checksum_hex`. Echoed back so the
/// caller can include it in audit logs or response headers.
pub alg: Option<ChecksumAlg>,
}
/// Stream an HTTP request body directly to a known destination file,
/// enforcing `max_bytes` as a hard size limit.
///
/// Used by the chunked-upload PUT handlers — each chunk has a
/// deterministic on-disk path (`NextcloudChunkedUploadService::safe_chunk_path`
/// for the NC surface, `ChunkedUploadService::prepare_chunk` for the
/// REST surface), so there's no spool/move dance. Peak heap is ~one
/// HTTP frame regardless of chunk size or `max_bytes`.
///
/// `checksum_alg` is the optional client-requested integrity check
/// (default `md5` per the legacy `Content-MD5` contract; `blake3`
/// available for forward-compat). When `Some`, the hash is computed
/// incrementally during streaming — no extra disk read for verification.
///
/// On size overflow the partial file is removed before the function
/// returns, so a client retry against the same chunk name starts from
/// a clean slate. On any other I/O error the partial file is also
/// removed and the error surfaces — callers can assume the path is
/// either fully written or absent.
pub async fn stream_body_to_path(
body: Body,
path: &Path,
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
let mut hasher = checksum_alg.map(IncrementalHasher::new);
while let Some(frame_result) = stream.next().await {
let frame = match frame_result {
Ok(f) => f,
Err(e) => {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::bad_request(format!(
"Failed to read request body: {e}"
)));
}
};
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_bytes {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::payload_too_large(format!(
"Chunk exceeds maximum size of {max_bytes} bytes"
)));
}
if let Some(h) = hasher.as_mut() {
h.update(chunk);
}
if let Err(e) = file.write_all(chunk).await {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::internal_error(format!(
"Failed to write chunk: {e}"
)));
}
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush chunk file: {e}")))?;
drop(file);
Ok(StreamedToPath {
bytes_written: total_bytes as u64,
checksum_hex: hasher.map(IncrementalHasher::finalize_hex),
alg: checksum_alg,
})
}
/// Algorithm-agnostic incremental hasher used by [`stream_body_to_path`].
/// Per-frame `update` is sub-millisecond for all three algorithms at the
/// 64 KB frame sizes axum's body stream produces, so we don't need
/// `spawn_blocking` (which the old buffered path used because it hashed
/// the full multi-MB chunk in one shot).
enum IncrementalHasher {
Md5(md5::Md5),
Sha256(sha2::Sha256),
// Boxing — blake3::Hasher is ~1.7 KB on the stack while md5::Md5
// (~100 bytes) and sha2::Sha256 (~100 bytes) are tiny; boxing the
// outlier keeps the enum size proportional to the common case
// rather than the worst case.
Blake3(Box<blake3::Hasher>),
}
impl IncrementalHasher {
fn new(alg: ChecksumAlg) -> Self {
match alg {
ChecksumAlg::Md5 => Self::Md5(md5::Md5::new()),
ChecksumAlg::Sha256 => Self::Sha256(sha2::Sha256::new()),
ChecksumAlg::Blake3 => Self::Blake3(Box::new(blake3::Hasher::new())),
}
}
fn update(&mut self, bytes: &[u8]) {
match self {
Self::Md5(h) => h.update(bytes),
Self::Sha256(h) => h.update(bytes),
Self::Blake3(h) => {
h.update(bytes);
}
}
}
fn finalize_hex(self) -> String {
match self {
Self::Md5(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Sha256(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Blake3(h) => h.finalize().to_hex().to_string(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use bytes::Bytes;
#[tokio::test]
async fn stream_body_to_path_caps_oversized() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
// 5 MiB body, 4 MiB cap → must reject.
let body = Body::from(Bytes::from(vec![0u8; 5 * 1024 * 1024]));
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
assert!(
result.is_err(),
"expected PayloadTooLarge, got Ok(bytes_written={})",
result.ok().map(|r| r.bytes_written).unwrap_or(0)
);
// Partial file must be removed on rejection.
assert!(
!path.exists(),
"rejected chunk file should be removed, but {} still exists",
path.display()
);
}
#[tokio::test]
async fn stream_body_to_path_accepts_under_cap() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 1024 * 1024])); // 1 MiB
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
let outcome = result.expect("should succeed");
assert_eq!(outcome.bytes_written, 1024 * 1024);
assert!(outcome.checksum_hex.is_none(), "no alg requested → no hash");
assert!(path.exists());
}
#[tokio::test]
async fn stream_body_to_path_caps_at_exact_boundary() {
// Edge case: body exactly equal to cap should succeed; cap+1 must fail.
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 100]));
let outcome = stream_body_to_path(body, &path, 100, None)
.await
.expect("100 bytes at 100-byte cap should succeed");
assert_eq!(outcome.bytes_written, 100);
let path2 = temp_dir.path().join("chunk2");
let body = Body::from(Bytes::from(vec![1u8; 101]));
assert!(
stream_body_to_path(body, &path2, 100, None).await.is_err(),
"101 bytes at 100-byte cap must reject"
);
assert!(!path2.exists());
}
}