Stream uploads directly into the CDC chunk store (no spool, single write)

Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
+4 -19
View File
@@ -228,12 +228,6 @@ pub struct StorageConfig {
/// returns 413 with a "use chunked upload" hint when a direct PUT
/// exceeds this cap. Env: `OXICLOUD_DIRECT_PUT_MAX_BYTES`.
pub direct_put_max_bytes: usize,
/// Directory for upload spool temp files. When `Some`, large uploads are
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
/// containers, where the spool's page-cache counts against the cgroup
/// memory limit and can trigger OOMKill on large files). Env:
/// `OXICLOUD_UPLOAD_TMPDIR`.
pub upload_temp_dir: Option<PathBuf>,
/// Root directory for chunked-upload sessions. When `Some`, chunks land
/// under `{chunk_dir}/{upload_id}/` (REST) and
/// `{chunk_dir}/nextcloud/{user}/{upload_id}/` (NC). When `None`, falls
@@ -401,7 +395,6 @@ impl Default for StorageConfig {
max_upload_size: MAX_UPLOAD_SIZE,
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
direct_put_max_bytes: 1024 * 1024 * 1024, // 1 GiB — pushes larger uploads onto the chunked protocol
upload_temp_dir: None,
chunk_dir: None,
usage_reconcile_secs: 600, // 10 minutes
tree_etag_flush_ms: 500,
@@ -1281,18 +1274,10 @@ impl AppConfig {
config.storage.direct_put_max_bytes = val;
}
// Upload spool directory — keep large upload temp files off tmpfs/RAM
// (otherwise their page-cache counts against the cgroup memory limit).
if let Ok(dir) = env::var("OXICLOUD_UPLOAD_TMPDIR")
&& !dir.trim().is_empty()
{
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
}
// Chunked-upload session root — separate from the PUT spool because
// chunked sessions accumulate disk on long uploads (multi-chunk
// resumable transfers) while PUT spool is short-lived. Sysadmins
// commonly want one of them on fast/local storage (NVMe) and the
// other on bulk storage; this knob lets that be expressed.
// Chunked-upload session root — chunked sessions accumulate disk on
// long uploads (multi-chunk resumable transfers); sysadmins commonly
// want them on fast/local storage (NVMe). This knob lets that be
// expressed.
if let Ok(dir) = env::var("OXICLOUD_CHUNK_DIR")
&& !dir.trim().is_empty()
{