Delta download: file manifest + user-scoped chunk fetch for sync clients
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
@@ -108,6 +108,33 @@ If the caller already owns the exact `file_hash`, the commit
|
||||
short-circuits to a pure reference bump — chunks aren't even looked at
|
||||
(same as `POST /api/files/by-hash`).
|
||||
|
||||
## Delta download (sync clients)
|
||||
|
||||
The inverse direction, for a client app that already holds an older
|
||||
version locally and wants the server's current one:
|
||||
|
||||
1. `GET /api/files/{id}/manifest` → `{ file_hash, total_size, chunks }`
|
||||
— the file's chunk recipe. **Owner-scoped** like the rest of the
|
||||
delta surface (shared files use the regular download endpoints).
|
||||
Served with `ETag: "<file_hash>"`; a manifest is immutable for a
|
||||
given hash, so `If-None-Match` revalidation answers 304 — polling
|
||||
sync clients pay one header round-trip per unchanged file.
|
||||
2. Diff the manifest against the local chunk inventory (chunk the local
|
||||
copy with the same WASM module the upload direction ships).
|
||||
3. `POST /api/files/delta/download` with `{ "hashes": […] }` → the
|
||||
requested chunks as `[u32 BE length][bytes]` frames in request order
|
||||
(the same wire format as the upload direction). Entitlement is the
|
||||
same possession rule as negotiate/commit: chunks must be reachable
|
||||
through the caller's own files; anything else → 404
|
||||
`{ "not_available": […] }` — deliberately indistinguishable from
|
||||
"never existed". Batches are bounded by `OXICLOUD_CHUNK_MAX_BYTES`;
|
||||
split large deltas across requests.
|
||||
4. Reassemble locally per the manifest order and verify the whole-file
|
||||
BLAKE3 against `file_hash`.
|
||||
|
||||
Editing 3 bytes of a 24 MB file on one device costs a second device one
|
||||
manifest GET plus ~1 chunk (~256 KB) instead of 24 MB.
|
||||
|
||||
## Security model
|
||||
|
||||
- **No content oracle.** Possession is proven per chunk: without bytes
|
||||
@@ -122,8 +149,9 @@ short-circuits to a pure reference bump — chunks aren't even looked at
|
||||
commit. Orphan chunks are GC-swept.
|
||||
- **Audit.** Rejections emit `delta_upload.rejected` with stable
|
||||
`reason` keys: `rate_limited`, `chunk_verification_failed`,
|
||||
`file_hash_mismatch`. AuthZ denials surface as the engine's standard
|
||||
`authz.denied`.
|
||||
`file_hash_mismatch` — and `delta_download.rejected` with
|
||||
`manifest_not_owner` / `chunks_not_owned`. AuthZ denials surface as
|
||||
the engine's standard `authz.denied`.
|
||||
|
||||
## Error summary
|
||||
|
||||
|
||||
Reference in New Issue
Block a user