Delta-upload protocol: negotiate chunks by hash, upload only what changed
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
@@ -81,6 +81,9 @@ use crate::interfaces::api::handlers::file_handler::MoveFilePayload;
|
||||
handlers::file_handler::list_files_query,
|
||||
handlers::file_handler::upload_file_with_thumbnails,
|
||||
handlers::file_handler::create_file_by_hash,
|
||||
handlers::delta_upload_handler::delta_negotiate,
|
||||
handlers::delta_upload_handler::delta_upload_chunks,
|
||||
handlers::delta_upload_handler::delta_commit,
|
||||
handlers::file_handler::download_file,
|
||||
handlers::file_handler::get_thumbnail,
|
||||
handlers::file_handler::upload_thumbnail,
|
||||
@@ -262,6 +265,13 @@ use crate::interfaces::api::handlers::file_handler::MoveFilePayload;
|
||||
ResourceContentDto,
|
||||
// File schemas
|
||||
FileDto,
|
||||
// Delta-upload schemas
|
||||
crate::application::services::delta_upload_service::ChunkRef,
|
||||
crate::application::services::delta_upload_service::DeltaNegotiateRequest,
|
||||
crate::application::services::delta_upload_service::DeltaNegotiateResponse,
|
||||
crate::application::services::delta_upload_service::DeltaChunksResponse,
|
||||
crate::application::services::delta_upload_service::DeltaCommitRequest,
|
||||
handlers::delta_upload_handler::DeltaStillMissingResponse,
|
||||
MoveFilePayload,
|
||||
PaginationDto,
|
||||
PaginationRequestDto,
|
||||
|
||||
Reference in New Issue
Block a user