Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
@@ -4,7 +4,7 @@
|
||||
//! - POST /api/uploads → Create upload session
|
||||
//! - PATCH /api/uploads/:id → Upload a chunk
|
||||
//! - HEAD /api/uploads/:id → Get upload status
|
||||
//! - POST /api/uploads/:id/complete → Assemble and finalize
|
||||
//! - POST /api/uploads/:id/complete → Stream parts into the blob store
|
||||
//! - DELETE /api/uploads/:id → Cancel upload
|
||||
|
||||
use axum::{
|
||||
@@ -27,7 +27,7 @@ use crate::common::di::AppState;
|
||||
use crate::domain::services::authorization::Permission;
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::middleware::auth::AuthUser;
|
||||
use crate::interfaces::upload_spool::stream_body_to_path;
|
||||
use crate::interfaces::upload_ingest::{self, stream_body_to_path};
|
||||
|
||||
/// Request body for creating an upload session
|
||||
#[derive(Debug, Deserialize, ToSchema)]
|
||||
@@ -64,35 +64,32 @@ pub struct CompleteUploadResponse {
|
||||
/// Optional body for `POST /api/uploads/{id}/complete`.
|
||||
///
|
||||
/// When the client supplies `checksum`, the server compares it against
|
||||
/// the assembled file's hash BEFORE promoting the blob to storage —
|
||||
/// failure aborts the upload atomically (no orphaned blob, no DB row).
|
||||
/// This is the end-to-end integrity check: per-chunk MD5 proves each
|
||||
/// chunk arrived intact, but only the final hash catches assembly /
|
||||
/// promotion bugs and mis-ordered chunks.
|
||||
/// the streamed content's hash BEFORE the file row is created — failure
|
||||
/// releases the blob reference and returns 400, with the chunk parts
|
||||
/// kept on disk for a retry. This is the end-to-end integrity check:
|
||||
/// per-chunk MD5 proves each chunk arrived intact, but only the final
|
||||
/// hash catches mis-ordered or corrupted assemblies.
|
||||
///
|
||||
/// **`blake3` is highly recommended** — it's the algorithm the server
|
||||
/// already runs over the assembled file during hash-on-write
|
||||
/// assembly, so verification is a string comparison with zero extra
|
||||
/// I/O and zero extra CPU. It's also the same algorithm the server
|
||||
/// uses for blob-storage addressing, so the value the client sends
|
||||
/// equals the `content_hash` they'd later read back from
|
||||
/// `GET /api/files/{id}`. `md5` and `sha256` are accepted for
|
||||
/// compatibility with legacy client tooling but each triggers a
|
||||
/// second hash pass over the assembled file (~30–100 ms depending
|
||||
/// on size).
|
||||
/// **`blake3` is highly recommended** — it's the content-addressing
|
||||
/// algorithm of the blob store itself, so verification is a string
|
||||
/// comparison against the hash the store already computed, and the
|
||||
/// value the client sends equals the `content_hash` they'd later read
|
||||
/// back from `GET /api/files/{id}`. `md5` and `sha256` are accepted
|
||||
/// for legacy client tooling; they are computed by an in-flight tee
|
||||
/// during the same streaming pass — no extra disk read either way.
|
||||
///
|
||||
/// `Default` keeps the existing wire shape: clients that POST with no
|
||||
/// body get today's behavior (no verification, server just returns
|
||||
/// what it computed).
|
||||
#[derive(Debug, Default, Deserialize, ToSchema)]
|
||||
pub struct CompleteUploadRequest {
|
||||
/// Lowercase hex digest the client expects the assembled file to
|
||||
/// Lowercase hex digest the client expects the streamed content to
|
||||
/// hash to. Compared case-insensitively. Omit to skip verification.
|
||||
pub checksum: Option<String>,
|
||||
/// Algorithm name. `blake3` is the recommended choice (default —
|
||||
/// matches the server's hash-on-write algorithm, zero extra cost).
|
||||
/// `md5`, `sha256` / `sha-256` are accepted but trigger an extra
|
||||
/// hash pass. Unknown values return 400.
|
||||
/// matches the blob store's content-addressing algorithm). `md5`,
|
||||
/// `sha256` / `sha-256` are accepted and computed in-flight.
|
||||
/// Unknown values return 400.
|
||||
pub checksumalg: Option<String>,
|
||||
}
|
||||
|
||||
@@ -307,70 +304,15 @@ impl ChunkedUploadHandler {
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the requested checksum of the assembled file.
|
||||
///
|
||||
/// For `Blake3` the server already has the hash from hash-on-write
|
||||
/// assembly — we just return it (zero I/O, zero CPU). For `Md5` and
|
||||
/// `Sha256` we re-read the assembled file on the blocking pool and
|
||||
/// hash it; the cost (~30–100 ms for typical files) is the trade-off
|
||||
/// for accepting non-default algorithms.
|
||||
async fn compute_assembled_hash(
|
||||
assembled_path: &std::path::Path,
|
||||
alg: ChecksumAlg,
|
||||
blake3_already_computed: &str,
|
||||
) -> Result<String, std::io::Error> {
|
||||
match alg {
|
||||
ChecksumAlg::Blake3 => Ok(blake3_already_computed.to_string()),
|
||||
ChecksumAlg::Md5 | ChecksumAlg::Sha256 => {
|
||||
let path = assembled_path.to_path_buf();
|
||||
tokio::task::spawn_blocking(move || -> Result<String, std::io::Error> {
|
||||
use std::io::Read;
|
||||
let mut file = std::fs::File::open(&path)?;
|
||||
let mut buf = vec![0u8; 524_288];
|
||||
match alg {
|
||||
ChecksumAlg::Md5 => {
|
||||
use md5::Digest as _;
|
||||
let mut h = md5::Md5::new();
|
||||
loop {
|
||||
let n = file.read(&mut buf)?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
h.update(&buf[..n]);
|
||||
}
|
||||
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
|
||||
}
|
||||
ChecksumAlg::Sha256 => {
|
||||
use sha2::Digest as _;
|
||||
let mut h = sha2::Sha256::new();
|
||||
loop {
|
||||
let n = file.read(&mut buf)?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
h.update(&buf[..n]);
|
||||
}
|
||||
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
|
||||
}
|
||||
// Blake3 handled above — this branch is unreachable but
|
||||
// keeps the match exhaustive without an else-clause.
|
||||
ChecksumAlg::Blake3 => unreachable!(),
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(format!("hash task join failed: {e}")))?
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// POST /api/uploads/:upload_id/complete - Finalize upload
|
||||
///
|
||||
/// Assembles all chunks into the final file and creates the file record.
|
||||
/// When `body.checksum` is supplied, the assembled file's hash is
|
||||
/// verified before the blob is promoted to storage — mismatch
|
||||
/// returns 400 and the assembled temp is removed (the session
|
||||
/// itself is kept so the client can re-issue complete after
|
||||
/// diagnosing).
|
||||
/// Streams the uploaded chunk parts, in order, straight into the CDC
|
||||
/// chunk store and creates the file record — no assembled temp file is
|
||||
/// ever written. When `body.checksum` is supplied it is verified from
|
||||
/// the same streaming pass (BLAKE3 comes from the store itself;
|
||||
/// MD5/SHA-256 are computed by an in-flight tee) — mismatch returns 400
|
||||
/// with the blob reference released, and the chunk parts stay on disk
|
||||
/// so the client can re-issue complete after diagnosing.
|
||||
pub(super) async fn complete_upload_impl(
|
||||
State(state): State<Arc<AppState>>,
|
||||
auth_user: AuthUser,
|
||||
@@ -379,10 +321,11 @@ impl ChunkedUploadHandler {
|
||||
) -> impl IntoResponse {
|
||||
let chunked_service = &state.core.chunked_upload_service;
|
||||
let upload_service = &state.applications.file_upload_service;
|
||||
let dedup = &state.core.dedup_service;
|
||||
|
||||
// ── Parse the optional algorithm BEFORE assembly so a bad
|
||||
// `checksumalg` doesn't waste the (potentially expensive)
|
||||
// hash work on a request we'll reject anyway.
|
||||
// ── Parse the optional algorithm BEFORE completion so a bad
|
||||
// `checksumalg` doesn't waste any work on a request we'll
|
||||
// reject anyway.
|
||||
let alg = match body.checksumalg.as_deref() {
|
||||
Some(name) => match ChecksumAlg::parse(name) {
|
||||
Some(a) => Some(a),
|
||||
@@ -397,37 +340,54 @@ impl ChunkedUploadHandler {
|
||||
};
|
||||
let expected_checksum = body.checksum.as_deref();
|
||||
|
||||
// Assemble chunks (hash-on-write: BLAKE3 computed during assembly)
|
||||
let (assembled_path, filename, folder_id, content_type, total_size, hash) =
|
||||
match chunked_service
|
||||
.complete_upload(&upload_id, auth_user.id)
|
||||
.await
|
||||
{
|
||||
Ok(result) => result,
|
||||
Err(e) => {
|
||||
return AppError::from(e).into_response();
|
||||
}
|
||||
};
|
||||
// Validate completion and get the chunk parts in assembly order.
|
||||
let parts = match chunked_service
|
||||
.complete_upload(&upload_id, auth_user.id)
|
||||
.await
|
||||
{
|
||||
Ok(result) => result,
|
||||
Err(e) => {
|
||||
return AppError::from(e).into_response();
|
||||
}
|
||||
};
|
||||
|
||||
// MD5/SHA-256 verification taps the stream while it is ingested;
|
||||
// BLAKE3 needs no tee — the store's own content hash IS BLAKE3.
|
||||
let alg = expected_checksum.map(|_| alg.unwrap_or(ChecksumAlg::Blake3));
|
||||
let tee = match alg {
|
||||
Some(ChecksumAlg::Md5) | Some(ChecksumAlg::Sha256) => {
|
||||
Some(upload_ingest::checksum_tee(alg.unwrap()))
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
|
||||
// ── Stream the parts into the CDC chunk store ───────────────
|
||||
let ingested = match upload_ingest::ingest_stream_to_cas(
|
||||
upload_ingest::stream_from_files(parts.chunk_paths),
|
||||
dedup,
|
||||
&parts.filename,
|
||||
&parts.content_type,
|
||||
usize::MAX,
|
||||
tee.clone(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(ingested) => ingested,
|
||||
Err(e) => return e.into_response(),
|
||||
};
|
||||
|
||||
// ── End-to-end integrity verification ───────────────────────
|
||||
// Only fires when the client supplied an `expected` checksum.
|
||||
// For BLAKE3 (the documented preferred choice) this is a string
|
||||
// comparison against the hash assembly already produced. For
|
||||
// MD5/SHA-256 we re-hash the assembled file on the blocking pool.
|
||||
if let Some(expected) = expected_checksum {
|
||||
let alg = alg.unwrap_or(ChecksumAlg::Blake3);
|
||||
let computed = match Self::compute_assembled_hash(&assembled_path, alg, &hash).await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
let _ = tokio::fs::remove_file(&assembled_path).await;
|
||||
return AppError::internal_error(format!(
|
||||
"Failed to compute assembled checksum: {e}"
|
||||
))
|
||||
.into_response();
|
||||
}
|
||||
if let (Some(expected), Some(alg)) = (expected_checksum, alg) {
|
||||
let computed = match alg {
|
||||
ChecksumAlg::Blake3 => Some(ingested.hash.clone()),
|
||||
_ => tee.as_ref().and_then(upload_ingest::finalize_checksum_tee),
|
||||
};
|
||||
let Some(computed) = computed else {
|
||||
upload_ingest::discard_ingested(dedup, &ingested).await;
|
||||
return AppError::internal_error("Checksum tee produced no digest").into_response();
|
||||
};
|
||||
if !computed.eq_ignore_ascii_case(expected) {
|
||||
let _ = tokio::fs::remove_file(&assembled_path).await;
|
||||
upload_ingest::discard_ingested(dedup, &ingested).await;
|
||||
tracing::warn!(
|
||||
target: "audit",
|
||||
event = "chunked_upload.checksum_mismatch",
|
||||
@@ -449,36 +409,28 @@ impl ChunkedUploadHandler {
|
||||
}
|
||||
}
|
||||
|
||||
// ── MIME detection (magic bytes + extension fallback) ─────
|
||||
let content_type = crate::common::mime_detect::refine_content_type_from_file(
|
||||
&assembled_path,
|
||||
&filename,
|
||||
&content_type,
|
||||
)
|
||||
.await;
|
||||
|
||||
// Upload from assembled file on disk — zero extra RAM copies, hash pre-computed
|
||||
// Register the file row against the ingested blob.
|
||||
let size = ingested.size;
|
||||
match upload_service
|
||||
.upload_file_from_path(
|
||||
filename.clone(),
|
||||
folder_id.clone(),
|
||||
content_type,
|
||||
&assembled_path,
|
||||
Some(hash),
|
||||
.upload_file_streaming(
|
||||
parts.filename.clone(),
|
||||
parts.folder_id.clone(),
|
||||
ingested.content_type.clone(),
|
||||
ingested.stored(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(file) => {
|
||||
// Cleanup session
|
||||
// Cleanup session (removes the chunk part files)
|
||||
let _ = chunked_service
|
||||
.finalize_upload(&upload_id, auth_user.id)
|
||||
.await;
|
||||
|
||||
tracing::info!(
|
||||
"✅ CHUNKED UPLOAD COMPLETE: {} (ID: {}, {} bytes)",
|
||||
filename,
|
||||
parts.filename,
|
||||
file.id,
|
||||
total_size
|
||||
size
|
||||
);
|
||||
|
||||
(
|
||||
@@ -486,14 +438,14 @@ impl ChunkedUploadHandler {
|
||||
Json(CompleteUploadResponse {
|
||||
file_id: file.id,
|
||||
filename: file.name,
|
||||
size: total_size,
|
||||
size,
|
||||
path: file.path,
|
||||
}),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to create file from assembled upload: {:?}", e);
|
||||
tracing::error!("Failed to create file from chunked upload: {:?}", e);
|
||||
AppError::internal_error(format!("Failed to create file: {}", e)).into_response()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,12 +5,11 @@ use axum::{
|
||||
response::IntoResponse,
|
||||
};
|
||||
use serde::Serialize;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use utoipa::ToSchema;
|
||||
|
||||
use crate::application::ports::dedup_ports::DedupResultDto;
|
||||
use crate::common::di::AppState;
|
||||
use crate::interfaces::middleware::auth::AuthUser;
|
||||
use crate::interfaces::upload_ingest;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Global application state for dependency injection
|
||||
@@ -155,10 +154,10 @@ impl DedupHandler {
|
||||
|
||||
/// Upload content with automatic deduplication (streaming).
|
||||
///
|
||||
/// Spools the upload to a temp file while computing the BLAKE3 hash
|
||||
/// incrementally (hash-on-write). Memory usage is constant (~512 KB)
|
||||
/// regardless of file size. Then delegates to `store_from_file` with
|
||||
/// the pre-computed hash so the file is never re-read for hashing.
|
||||
/// Streams the multipart field straight into the CDC chunk store —
|
||||
/// chunking, BLAKE3 hashing and dedup checks happen while the bytes
|
||||
/// arrive (no temp file, no re-read; peak RAM is bounded regardless
|
||||
/// of file size).
|
||||
///
|
||||
/// POST /api/dedup/upload
|
||||
pub(super) async fn upload_with_dedup_impl(
|
||||
@@ -177,64 +176,29 @@ impl DedupHandler {
|
||||
.content_type()
|
||||
.unwrap_or("application/octet-stream")
|
||||
.to_string();
|
||||
let filename = field.file_name().unwrap_or("unnamed").to_string();
|
||||
|
||||
// ── Spool to temp file + BLAKE3 hash-on-write ────────
|
||||
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp");
|
||||
let temp_path = temp_dir.join(format!("dedup-{}", uuid::Uuid::new_v4()));
|
||||
|
||||
let mut total_size: u64 = 0;
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut field = field;
|
||||
|
||||
let spool_result: Result<(), String> = async {
|
||||
let file = tokio::fs::File::create(&temp_path)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to create temp file: {}", e))?;
|
||||
|
||||
// 512 KB buffer — reduces write syscalls
|
||||
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
|
||||
|
||||
loop {
|
||||
match field.chunk().await {
|
||||
Ok(Some(chunk)) => {
|
||||
total_size += chunk.len() as u64;
|
||||
hasher.update(&chunk);
|
||||
writer
|
||||
.write_all(&chunk)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to write chunk: {}", e))?;
|
||||
}
|
||||
Ok(None) => break,
|
||||
Err(e) => {
|
||||
return Err(format!(
|
||||
"Connection lost during upload (received {} bytes): {}",
|
||||
total_size, e
|
||||
));
|
||||
}
|
||||
}
|
||||
// ── Stream into the CDC chunk store ──────────────────
|
||||
let source = upload_ingest::multipart_field_stream(field);
|
||||
let ingested = match upload_ingest::ingest_stream_to_cas(
|
||||
source,
|
||||
dedup,
|
||||
&filename,
|
||||
&content_type,
|
||||
usize::MAX,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(ingested) => ingested,
|
||||
Err(e) => {
|
||||
tracing::warn!("Dedup upload ingest failed: {}", e.message);
|
||||
return e.into_response();
|
||||
}
|
||||
};
|
||||
|
||||
writer
|
||||
.flush()
|
||||
.await
|
||||
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
|
||||
if let Err(msg) = spool_result {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::warn!("Dedup upload spool failed: {}", msg);
|
||||
return Response::builder()
|
||||
.status(StatusCode::BAD_REQUEST)
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(format!(r#"{{"error": "{}"}}"#, msg)))
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
|
||||
if total_size == 0 {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
if ingested.size == 0 {
|
||||
upload_ingest::discard_ingested(dedup, &ingested).await;
|
||||
return Response::builder()
|
||||
.status(StatusCode::BAD_REQUEST)
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
@@ -243,60 +207,33 @@ impl DedupHandler {
|
||||
.into_response();
|
||||
}
|
||||
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
let metadata = dedup.get_blob_metadata(&ingested.hash).await;
|
||||
|
||||
// ── Store with deduplication (pre-computed hash) ──────
|
||||
match dedup
|
||||
.store_from_file(&temp_path, Some(content_type), Some(hash))
|
||||
.await
|
||||
{
|
||||
Ok(result) => {
|
||||
let (is_new, bytes_saved) = match &result {
|
||||
DedupResultDto::NewBlob { .. } => (true, 0),
|
||||
DedupResultDto::ExistingBlob { saved_bytes, .. } => {
|
||||
(false, *saved_bytes)
|
||||
}
|
||||
};
|
||||
let response = DedupUploadResponse {
|
||||
is_new: ingested.is_new_blob,
|
||||
hash: ingested.hash.clone(),
|
||||
size: ingested.size,
|
||||
bytes_saved: ingested.bytes_saved,
|
||||
ref_count: metadata.map(|m| m.ref_count).unwrap_or(1),
|
||||
};
|
||||
|
||||
let metadata = dedup.get_blob_metadata(result.hash()).await;
|
||||
tracing::info!(
|
||||
"🔗 Dedup upload: hash={}, new={}, saved={}",
|
||||
ingested.hash,
|
||||
ingested.is_new_blob,
|
||||
ingested.bytes_saved
|
||||
);
|
||||
|
||||
let response = DedupUploadResponse {
|
||||
is_new,
|
||||
hash: result.hash().to_string(),
|
||||
size: result.size(),
|
||||
bytes_saved,
|
||||
ref_count: metadata.map(|m| m.ref_count).unwrap_or(1),
|
||||
};
|
||||
|
||||
tracing::info!(
|
||||
"🔗 Dedup upload: hash={}, new={}, saved={}",
|
||||
result.hash(),
|
||||
is_new,
|
||||
bytes_saved
|
||||
);
|
||||
|
||||
return Response::builder()
|
||||
.status(if is_new {
|
||||
StatusCode::CREATED
|
||||
} else {
|
||||
StatusCode::OK
|
||||
})
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(serde_json::to_string(&response).unwrap()))
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
Err(e) => {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("Dedup upload failed: {}", e);
|
||||
return Response::builder()
|
||||
.status(StatusCode::INTERNAL_SERVER_ERROR)
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(r#"{"error": "Upload failed"}"#))
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
}
|
||||
return Response::builder()
|
||||
.status(if ingested.is_new_blob {
|
||||
StatusCode::CREATED
|
||||
} else {
|
||||
StatusCode::OK
|
||||
})
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(serde_json::to_string(&response).unwrap()))
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ use crate::common::di::AppState;
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::middleware::auth::AuthUser;
|
||||
use crate::interfaces::range_requests::not_modified_response;
|
||||
use crate::interfaces::upload_ingest;
|
||||
use crate::{application::dtos::file_dto::FileDto, domain::services::authorization::Permission};
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -51,11 +52,12 @@ impl FileHandler {
|
||||
// UPLOAD
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
/// Streaming file upload — constant ~64 KB RAM regardless of file size.
|
||||
/// Streaming file upload — bounded RAM regardless of file size.
|
||||
///
|
||||
/// **Hash-on-Write**: BLAKE3 is computed while spooling the multipart
|
||||
/// body to the temp file. This eliminates the second sequential read
|
||||
/// that dedup_service would otherwise need, cutting total I/O in half.
|
||||
/// The multipart body is streamed straight into the CDC chunk store:
|
||||
/// chunking, hashing and dedup checks happen while the bytes arrive.
|
||||
/// No spool file, no re-read — chunks the store already has are never
|
||||
/// written to disk at all.
|
||||
pub async fn upload_file(
|
||||
State(state): State<GlobalState>,
|
||||
auth_user: AuthUser,
|
||||
@@ -71,7 +73,7 @@ impl FileHandler {
|
||||
/// [`Self::upload_file_with_thumbnails`].
|
||||
///
|
||||
/// Returns `(FileDto, blob_hash)` on success. The blob hash is the
|
||||
/// BLAKE3 digest computed during the hash-on-write spool and is
|
||||
/// BLAKE3 digest computed during the streaming ingest and is
|
||||
/// propagated without an extra database round-trip so that callers
|
||||
/// (e.g. thumbnail generation) can resolve the physical blob path
|
||||
/// immediately.
|
||||
@@ -158,120 +160,55 @@ impl FileHandler {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Spool multipart field to temp file + hash-on-write ──
|
||||
// .dedup_temp is created once by DedupService::initialize() at startup
|
||||
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp");
|
||||
let temp_path = temp_dir.join(format!("upload-{}", uuid::Uuid::new_v4()));
|
||||
|
||||
let mut total_size: u64 = 0;
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let spool_result: Result<(), String> = async {
|
||||
let file = tokio::fs::File::create(&temp_path)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to create temp file: {}", e))?;
|
||||
|
||||
// Pre-allocate if Content-Length is known (reduces fragmentation)
|
||||
let hint = field
|
||||
.headers()
|
||||
.get(axum::http::header::CONTENT_LENGTH)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|s| s.parse::<u64>().ok());
|
||||
if let Some(len) = hint {
|
||||
let _ = file.set_len(len).await; // best-effort
|
||||
}
|
||||
|
||||
// 512 KB buffer — 8× fewer write syscalls than 64 KB
|
||||
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
|
||||
let mut field = field;
|
||||
// IMPORTANT: use explicit match instead of `while let Ok(Some(..))`.
|
||||
// The old pattern silently swallowed Err (client disconnect)
|
||||
// and accepted partially received data as a complete upload.
|
||||
loop {
|
||||
match field.chunk().await {
|
||||
Ok(Some(chunk)) => {
|
||||
total_size += chunk.len() as u64;
|
||||
hasher.update(&chunk);
|
||||
tokio::io::AsyncWriteExt::write_all(&mut writer, &chunk)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to write chunk: {}", e))?;
|
||||
}
|
||||
Ok(None) => break, // End of field — upload complete
|
||||
Err(e) => {
|
||||
return Err(format!(
|
||||
"Connection lost during upload (received {} bytes): {}",
|
||||
total_size, e
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
tokio::io::AsyncWriteExt::flush(&mut writer)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
|
||||
if let Err(e) = spool_result {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("❌ UPLOAD SPOOL FAILED: {} - {}", filename, e);
|
||||
return Err(Self::domain_error_response(
|
||||
crate::common::errors::DomainError::internal_error("FileUpload", e),
|
||||
));
|
||||
}
|
||||
|
||||
// Empty file — use streaming path with the (empty) temp file
|
||||
if total_size == 0 {
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
let dto = upload_service
|
||||
.upload_file_streaming(
|
||||
filename,
|
||||
folder_id,
|
||||
content_type,
|
||||
&temp_path,
|
||||
0,
|
||||
Some(hash.clone()),
|
||||
)
|
||||
.await
|
||||
.map_err(Self::domain_error_response)?;
|
||||
return Ok((dto, hash));
|
||||
}
|
||||
|
||||
// Finalize hash
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
|
||||
// ── MIME detection (magic bytes + extension fallback) ─
|
||||
let content_type = crate::common::mime_detect::refine_content_type_from_file(
|
||||
&temp_path,
|
||||
// ── Stream the field into the CDC chunk store ────────
|
||||
// Chunking (FastCDC) + hashing (BLAKE3) + dedup checks +
|
||||
// MIME sniffing all happen while the bytes arrive; chunks
|
||||
// the store already has never touch the disk. Size is
|
||||
// capped globally by DefaultBodyLimit.
|
||||
let dedup = &state.core.dedup_service;
|
||||
let source = upload_ingest::multipart_field_stream(field);
|
||||
let ingested = match upload_ingest::ingest_stream_to_cas(
|
||||
source,
|
||||
dedup,
|
||||
&filename,
|
||||
&content_type,
|
||||
usize::MAX,
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
.await
|
||||
{
|
||||
Ok(ingested) => ingested,
|
||||
Err(e) => {
|
||||
tracing::error!("❌ UPLOAD INGEST FAILED: {} - {}", filename, e.message);
|
||||
return Err(e.into_response());
|
||||
}
|
||||
};
|
||||
|
||||
// ── Quota enforcement ────────────────────────────────
|
||||
// ── Quota enforcement (exact size now known) ─────────
|
||||
if let Some(storage_svc) = state.storage_usage_service.as_ref()
|
||||
&& let Err(err) = storage_svc
|
||||
.check_storage_quota(auth_user.id, total_size)
|
||||
.check_storage_quota(auth_user.id, ingested.size)
|
||||
.await
|
||||
{
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
upload_ingest::discard_ingested(dedup, &ingested).await;
|
||||
tracing::warn!(
|
||||
"⛔ UPLOAD REJECTED (quota): user={}, file={}, size={}",
|
||||
auth_user.username,
|
||||
filename,
|
||||
total_size
|
||||
ingested.size
|
||||
);
|
||||
return Err(Self::quota_error_response(err));
|
||||
}
|
||||
|
||||
// ── Streaming upload (temp file → blob store, hash pre-computed) ─
|
||||
// ── Register the file row against the ingested blob ──
|
||||
let hash = ingested.hash.clone();
|
||||
let size = ingested.size;
|
||||
match upload_service
|
||||
.upload_file_streaming(
|
||||
filename.clone(),
|
||||
folder_id,
|
||||
content_type,
|
||||
&temp_path,
|
||||
total_size,
|
||||
Some(hash.clone()),
|
||||
ingested.content_type.clone(),
|
||||
ingested.stored(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -279,13 +216,12 @@ impl FileHandler {
|
||||
tracing::info!(
|
||||
"✅ STREAMING UPLOAD: {} ({} bytes, ID: {})",
|
||||
filename,
|
||||
total_size,
|
||||
size,
|
||||
file.id
|
||||
);
|
||||
return Ok((file, hash));
|
||||
}
|
||||
Err(err) => {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("❌ UPLOAD FAILED: {} - {}", filename, err);
|
||||
return Err(Self::domain_error_response(err));
|
||||
}
|
||||
|
||||
@@ -898,10 +898,11 @@ async fn handle_head(
|
||||
/**
|
||||
* Handles PUT requests to create or update files.
|
||||
*
|
||||
* **Streaming implementation**: the request body is spooled to a temp file
|
||||
* with incremental BLAKE3 hashing. Peak RAM usage is ~256 KB regardless
|
||||
* of file size. The temp file is then atomically moved into blob storage
|
||||
* via `update_file_streaming`.
|
||||
* **Streaming implementation**: the request body is streamed straight into
|
||||
* the CDC chunk store (FastCDC + BLAKE3 while the bytes arrive — no spool
|
||||
* file, no re-read; peak RAM is bounded regardless of file size), then the
|
||||
* file row is atomically swapped onto the ingested blob via
|
||||
* `update_file_streaming`.
|
||||
*
|
||||
* @param state The application state containing service dependencies
|
||||
* @param path The requested resource path
|
||||
@@ -913,7 +914,7 @@ async fn handle_put(
|
||||
req: Request<Body>,
|
||||
path: String,
|
||||
) -> Result<Response<Body>, AppError> {
|
||||
use crate::interfaces::upload_spool::spool_body_to_temp;
|
||||
use crate::interfaces::upload_ingest;
|
||||
|
||||
let user = extract_user(&req)?;
|
||||
|
||||
@@ -972,32 +973,31 @@ async fn handle_put(
|
||||
.unwrap_or("application/octet-stream")
|
||||
.to_string();
|
||||
|
||||
// ── Streaming spool: body → temp file + incremental hash ──
|
||||
// Shared with the NextCloud-compat PUT handler; peak heap ~one frame
|
||||
// regardless of file size. Honors `upload_temp_dir` to keep the spool
|
||||
// off tmpfs/RAM.
|
||||
let spooled = spool_body_to_temp(
|
||||
// ── Streaming ingest: body → CDC chunk store ──────────────
|
||||
// Shared with the NextCloud-compat PUT handler; chunking + hashing +
|
||||
// dedup checks run while the body arrives — no spool file, no re-read.
|
||||
let filename = crate::common::mime_detect::filename_from_path(&path).to_string();
|
||||
let ingested = upload_ingest::ingest_body_to_cas(
|
||||
req.into_body(),
|
||||
&state.core.dedup_service,
|
||||
&filename,
|
||||
&content_type,
|
||||
max_upload,
|
||||
state.core.config.storage.upload_temp_dir.clone(),
|
||||
)
|
||||
.await?;
|
||||
let temp_path = spooled.temp.path().to_path_buf();
|
||||
let total_bytes = spooled.size as usize;
|
||||
let hash = spooled.hash;
|
||||
|
||||
// ── Quota enforcement ────────────────────────────────────
|
||||
if let Some(storage_svc) = state.storage_usage_service.as_ref()
|
||||
&& let Err(err) = storage_svc
|
||||
.check_storage_quota(user.id, total_bytes as u64)
|
||||
.check_storage_quota(user.id, ingested.size)
|
||||
.await
|
||||
{
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
upload_ingest::discard_ingested(&state.core.dedup_service, &ingested).await;
|
||||
tracing::warn!(
|
||||
"⛔ WEBDAV PUT REJECTED (quota): user={}, file={}, size={}",
|
||||
user.id,
|
||||
path,
|
||||
total_bytes
|
||||
ingested.size
|
||||
);
|
||||
return Err(AppError::new(
|
||||
StatusCode::INSUFFICIENT_STORAGE,
|
||||
@@ -1006,21 +1006,12 @@ async fn handle_put(
|
||||
));
|
||||
}
|
||||
|
||||
// ── Atomic store: temp file → dedup blob + DB metadata update ──
|
||||
// ── Atomic store: swap the file row onto the ingested blob ──
|
||||
let content_type = ingested.content_type.clone();
|
||||
let result = file_upload_service
|
||||
.update_file_streaming(
|
||||
&path,
|
||||
&temp_path,
|
||||
total_bytes as u64,
|
||||
&content_type,
|
||||
Some(hash),
|
||||
None,
|
||||
)
|
||||
.update_file_streaming(&path, ingested.stored(), &content_type, None)
|
||||
.await;
|
||||
|
||||
// Clean up temp file (may already be moved by dedup, ignore error)
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
|
||||
match result {
|
||||
Ok(_file_dto) => Ok(Response::builder()
|
||||
.status(StatusCode::NO_CONTENT)
|
||||
|
||||
@@ -165,10 +165,6 @@ async fn put_file(
|
||||
State(state): State<WopiState>,
|
||||
req: Request<Body>,
|
||||
) -> Response {
|
||||
use http_body_util::BodyStream;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use tokio_stream::StreamExt;
|
||||
|
||||
let claims = match state
|
||||
.token_service
|
||||
.validate_token(&token_query.access_token)
|
||||
@@ -218,75 +214,32 @@ async fn put_file(
|
||||
Err(_) => return StatusCode::NOT_FOUND.into_response(),
|
||||
};
|
||||
|
||||
// ── Streaming spool: body → temp file + incremental BLAKE3 ──
|
||||
let temp_file = match tempfile::NamedTempFile::new() {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
tracing::error!("WOPI PutFile: failed to create temp file: {}", e);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
};
|
||||
let temp_path = temp_file.path().to_path_buf();
|
||||
|
||||
let mut file_out = match tokio::fs::File::create(&temp_path).await {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
tracing::error!("WOPI PutFile: failed to open temp file: {}", e);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
};
|
||||
|
||||
// ── Streaming ingest: body → CDC chunk store (no temp file) ──
|
||||
let content_type = file.mime_type.clone();
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut total_bytes: u64 = 0;
|
||||
let mut stream = BodyStream::new(req.into_body());
|
||||
|
||||
while let Some(frame_result) = stream.next().await {
|
||||
let frame = match frame_result {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("WOPI PutFile: body read error: {}", e);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
};
|
||||
if let Some(chunk) = frame.data_ref() {
|
||||
total_bytes += chunk.len() as u64;
|
||||
hasher.update(chunk);
|
||||
if let Err(e) = file_out.write_all(chunk).await {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("WOPI PutFile: temp write error: {}", e);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
let ingested = match crate::interfaces::upload_ingest::ingest_body_to_cas(
|
||||
req.into_body(),
|
||||
&state.app_state.core.dedup_service,
|
||||
&file.name,
|
||||
&content_type,
|
||||
usize::MAX,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(ingested) => ingested,
|
||||
Err(e) => {
|
||||
tracing::error!("WOPI PutFile: ingest failed: {}", e.message);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
}
|
||||
if let Err(e) = file_out.flush().await {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("WOPI PutFile: flush error: {}", e);
|
||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
||||
}
|
||||
drop(file_out);
|
||||
};
|
||||
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
|
||||
// ── Atomic store: temp file → dedup blob + DB metadata update ──
|
||||
// ── Atomic store: swap the file row onto the ingested blob ──
|
||||
let result = state
|
||||
.app_state
|
||||
.applications
|
||||
.file_upload_service
|
||||
.update_file_streaming(
|
||||
&file.path,
|
||||
&temp_path,
|
||||
total_bytes,
|
||||
&content_type,
|
||||
Some(hash),
|
||||
None,
|
||||
)
|
||||
.update_file_streaming(&file.path, ingested.stored(), &content_type, None)
|
||||
.await;
|
||||
|
||||
// Clean up temp file (may already be moved by dedup, ignore error)
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
|
||||
match result {
|
||||
Ok(_file_dto) => StatusCode::OK.into_response(),
|
||||
Err(e) => {
|
||||
|
||||
Reference in New Issue
Block a user