Delta-upload client: FastCDC in WASM + overlapped worker pipeline

Phase 2 — the client side of "upload only what changed", closing the
delta-sync plan.

WASM (wasm/oxicloud-hash): DeltaChunker adds incremental FastCDC with
the server's exact crate and parameters (64K/256K/1M) next to the BLAKE3
hasher. The incremental split is provably identical to a single pass:
every chunk except the last ends on a content/max-size condition whose
decision window was fully buffered, so only the tail is provisional and
re-examined as slices arrive. A mirror test — the client twin of the
server's stream≡slice test — chunks 4 MiB of xorshift noise with
adversarial slice sizes (7 B … 8 MiB) and requires boundary-for-boundary
equality with one FastCDC pass. Vendored artifacts rebuilt (55 KB wasm).

Worker (static/js/workers/deltaWorker.js): the full protocol off the
main thread with OVERLAPPED stages — 8 MiB file slices feed the chunker
while earlier batches (256 hashes) negotiate and their missing chunks
upload through a 2-deep PUT pool (≤8 MiB framed bodies, bytes re-sliced
from the File at send time, never hoarded). Commit handles 409
still_missing by uploading exactly the named hashes and retrying.

Orchestrator (features/files/deltaUpload.js): threshold (8 MiB),
worker lifecycle + size-scaled timeout, progress relay to the upload
bell, conclusive-outcome mapping (201/200, 507 quota, 409 name
conflict) and silent fallback to the byte upload for everything else.
Wired into uploadFiles and uploadFolderEntries, which now surface one
batch summary of the bytes dedup saved. This subsumes the whole-file
instant-upload module — a fully-known file negotiates to nothing
missing and the commit short-circuits on possession — so
instantUpload.js and hashWorker.js are removed (the /api/dedup/check
and /api/files/by-hash endpoints remain for API clients).

Verified end-to-end against PostgreSQL 16 — the cross-boundary proof
the whole design hangs on, in both directions: a 24 MB file byte-
uploaded (server-side CDC) then edited and delta-negotiated with
WASM-computed chunks reported missing 1/74 (boundaries bit-identical),
synced with 344 KB on the wire vs 24 MB (98.6% saved) and downloaded
byte-identical; inversely, a file created via delta then byte-uploaded
as identical content produced a server-side manifest DEDUP HIT with the
same content_hash. Insertion at the head of the file (the adversarial
CDC case) still negotiated missing 1/74. Chunk+hash throughput ≈275 MB/s
in V8 with SIMD128.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 15:44:44 +00:00
parent 44967da7f1
commit 5d034b0d09
11 changed files with 885 additions and 306 deletions
+12 -7
View File
@@ -12,15 +12,20 @@ Editing a few bytes of a 500 MB file re-uploads ~1 MiB instead of
## Who can use it
Any authenticated API client. The OxiCloud web frontend adopts it in a
later phase; generic WebDAV/NextCloud clients cannot (their protocols
have no delta concept) — they keep uploading full bytes, and the server
keeps deduplicating those on write.
Any authenticated API client. The OxiCloud web frontend uses it
automatically for files ≥ 8 MiB (`features/files/deltaUpload.js` +
`workers/deltaWorker.js`, chunking with the vendored WASM build of the
server's own FastCDC+BLAKE3 crates, falling back to a plain byte upload
on any failure). Generic WebDAV/NextCloud clients cannot (their
protocols have no delta concept) — they keep uploading full bytes, and
the server keeps deduplicating those on write.
Chunk boundaries are the **client's choice**: matching the server's
FastCDC parameters maximizes cross-version sharing, but any split with
chunks of 1 byte … 1 MiB is valid — correctness is guaranteed by
server-side verification, not by the chunking scheme.
FastCDC parameters (64 KB / 256 KB / 1 MiB, as the bundled WASM module
does) maximizes cross-version sharing — including against versions that
entered through plain byte uploads — but any split with chunks of
1 byte … 1 MiB is valid; correctness is guaranteed by server-side
verification, not by the chunking scheme.
## The three steps