Delta download: file manifest + user-scoped chunk fetch for sync clients
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
@@ -541,3 +541,54 @@
|
||||
* @property {string} hash BLAKE3 of the owned content (64 hex chars)
|
||||
*/
|
||||
|
||||
// ------------------- Delta sync (chunk negotiation)
|
||||
|
||||
/**
|
||||
* One chunk reference on the delta wire: terse on purpose (a 10 GB file
|
||||
* is ~40 000 of these). Mirrors `ChunkRef` on the server
|
||||
* (`delta_upload_service.rs`).
|
||||
* @typedef {Object} DeltaChunkRef
|
||||
* @property {string} h BLAKE3 of the chunk (64 hex chars)
|
||||
* @property {number} s chunk size in bytes (1 ..= 1 MiB)
|
||||
*/
|
||||
|
||||
/**
|
||||
* Response of `POST /api/files/delta/negotiate` — the distinct chunk
|
||||
* hashes the caller must upload (user-scoped, advisory).
|
||||
* @typedef {Object} DeltaNegotiateAnswer
|
||||
* @property {string[]} missing
|
||||
*/
|
||||
|
||||
/**
|
||||
* Request body of `POST /api/files/delta/commit`. Exactly one of
|
||||
* (`name` + `folder_id`) or `file_id` selects create vs update mode.
|
||||
* 201/200 responses carry a {@link FileItem}; 409 carries
|
||||
* `{still_missing: string[]}` (upload those chunks and retry).
|
||||
* @typedef {Object} DeltaCommitRequest
|
||||
* @property {string} file_hash BLAKE3 of the whole file (verified server-side)
|
||||
* @property {DeltaChunkRef[]} chunks full sequence, in file order
|
||||
* @property {string} [name] create mode: file name
|
||||
* @property {string} [folder_id] create mode: target folder
|
||||
* @property {string} [file_id] update mode: file whose content is replaced
|
||||
*/
|
||||
|
||||
/**
|
||||
* Response of `GET /api/files/{id}/manifest` — the recipe to rebuild a
|
||||
* file from chunks (delta download, step 1). Immutable per `file_hash`;
|
||||
* the endpoint serves it with `ETag: file_hash` so polling sync clients
|
||||
* revalidate with a 304 for free.
|
||||
* @typedef {Object} DeltaManifestAnswer
|
||||
* @property {string} file_hash BLAKE3 of the whole file
|
||||
* @property {number} total_size bytes
|
||||
* @property {DeltaChunkRef[]} chunks full sequence, in file order
|
||||
*/
|
||||
|
||||
/**
|
||||
* Request body of `POST /api/files/delta/download` (delta download,
|
||||
* step 2). Responds with `[u32 BE length][bytes]` frames in request
|
||||
* order, or 404 `{not_available: string[]}` for chunks outside the
|
||||
* caller's files.
|
||||
* @typedef {Object} DeltaDownloadRequest
|
||||
* @property {string[]} hashes distinct chunk hashes to fetch
|
||||
*/
|
||||
|
||||
|
||||
Reference in New Issue
Block a user