Step 7 of docs/plan/derived-blobs.md, write path first — the plan is
explicit that fixing it before the import means transcode_import only
has to handle history, not a moving target.
ImageTranscodeService now reads and writes storage.content_derived_blobs
under kind='transcode', keyed by the BLAKE3 of the SOURCE content. The
hash is threaded in from file_retrieval_service, which already holds it
as dto.content_hash; hashing here would be a BLAKE3 over the whole file
on every request. Callers without one (external mounts) keep the local
cache untouched, which is what the service did before this tier existed.
Negative verdicts become rows rather than zero-byte .skip files. A
transcode that came out larger is deterministic in the content, so it is
worth remembering; the row survives moka eviction, a restart, and the
deletion of .transcoded/, none of which the marker does. Only that
verdict is persisted — a timeout or a read error returns Err and is
recorded nowhere, because a momentary failure written here would mark a
perfectly transcodable image hopeless with nothing to retry it.
Representation is a NULL blob_hash, per the plan: a sentinel hash would
stop blob_hash naming a real Blob and every consumer would need to learn
the exception. A CHECK keeps blob_hash and content_type NULL together —
a type without bytes describes nothing, bytes without a type cannot be
served.
Two consumers had to be corrected for NULLs first, both of which would
have broken on the first negative row ever written:
* satellites_consistency reported them as derived_dangling_blob at
data_loss severity. SQL comparison against NULL is NULL, so EXISTS was
false and a row correctly pointing at nothing read as an artifact that
had gone missing.
* blob_reference_sources::list_referenced_blobs decodes blob_hash into
String, so the first NULL would have failed the decode and taken the
whole enumeration down. It would also have been wrong if it decoded —
a negative row holds no reference, which is why the counting forms
(WHERE blob_hash = <hash>) already exclude it for free.
lookup_derived returns a three-way answer because Option collapses the
two cases a caller deciding whether to spend a decode most needs apart:
never attempted, versus attempted and known not worth it.
DedupService is attached after construction via a OnceLock. DI builds
the transcode service ~240 lines before DedupService exists, and the
retrieval path that needs it is wired earlier still, so a constructor
argument would mean reordering more than this is worth.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Permanent blob leak, three rows per image. Confirmed green after this.
The leftovers named their source, and it had no manifest, no blob row
and no files. Nothing will ever reap that hash again, so
purge_derived_blobs can never fire for it — meaning the rows were
written AFTER the source died, not left behind by a reap that skipped
them. Two earlier attempts assumed the latter and fixed the wrong thing.
Background thumbnail generation is spawned and unawaited, so an upload
deleted promptly — constant in a test suite, occasional for real users —
has its render finish after GC reaped the blob and then record three
mappings to a corpse. Each pins its own thumbnail blob at ref_count 1,
which GC is thereafter CORRECT to refuse: that is why three passes with
force=true reclaimed nothing and why the leak was invisible, a healthy
system declining to delete referenced data.
store_derived_blob now inserts only WHERE the source still exists,
checking both tables since source_hash names a Blob — a manifest for CDC
content, a bare blob row for legacy whole-file content. A refused insert
falls into the existing `inserted == 0` branch and releases the
reference, so the thumbnail blob becomes collectible rather than
stranded.
Closed in both directions: if the source dies before the statement's
snapshot the row is refused; if after, that reap's purge finds the row.
034f1050 stays — the bulk manifest reap genuinely lacked the purge that
reap_blob had, and two manifest reap paths with only one purging is its
own defect. It just was not this one.
Still missing, and now clearly worth building: the orphan-mapping check
the plan's coverage matrix already lists (content_derived_blobs.
source_hash with no Blob behind it). This stops new ones; nothing yet
finds the ones already on disk.
Real leak, found by storage_cleanup_check.sh: three blobs surviving a
full teardown, all `derived=1`, all naming one `src` whose manifest,
blob row and files were already gone. The source had been reaped without
its derived rows being purged.
`reap_blob` purges correctly for the single-blob path. The BULK manifest
reap did not — it iterated the deleted batch only to invalidate the
manifest cache, so every manifest reaped that way left its
content_derived_blobs rows behind.
The predicate is not at fault. It protects a manifest that IS a derived
artifact (content_derived_blobs.blob_hash) and deliberately not one that
is the SOURCE of them, because counting source_hash as a reference would
pin every original for as long as a thumbnail existed. The source is
therefore reaped correctly and the purge simply has to follow it.
The consequence is permanent, not cosmetic: the orphaned row holds
chunk_manifests.ref_count at 1 on the thumbnail's own blob, so GC is
thereafter CORRECT to refuse it — which is why three passes with
force=true reclaimed nothing. Every deleted image left three behind, one
per size, growing forever.
Fixed at the reap rather than in any deletion path, which is where all
of them converge: folder cascade, drive deletion, user deletion and
single-file delete all reach it through the decrement trigger, so one
call covers every route.
1. Per-file overrides must beat the content tier in RAM as well as on
disk. The content-keyed lookup ran first, so a thumbnail already
rendered from the file's content sat in RAM under content(blob_hash)
and shadowed a preview uploaded afterwards — permanently. Invisible
before the moka rekey, because both lived under one file-id key and
the upload simply overwrote the render. Order is now uniform:
per-file RAM, per-file disk (ext-), per-file DB, then content RAM,
blob-hash disk, derived blob.
2. store_attached_blob never wrote a row. Its RETURNING clause compared
the stored hash against EXCLUDED, and PostgreSQL only permits
EXCLUDED in the SET and WHERE of DO UPDATE — a runtime syntax error
on every call. The superseded hash now comes from a SELECT taken
before the upsert; losing that race leaves one stale reference, which
the manifest recompute reports, rather than anything being lost.
The second hid behind the first for a whole cycle, and behind
`ext-{file_id}.jpg`: the ORIGINAL kept serving its uploaded preview from
local disk, so the feature looked healthy. Only a copy, which has a
different file_id and therefore no ext- file, depends on the row — and
the row was never there. The handler's best-effort warn! completed the
disguise, so it is now error!: a failure there means copies silently
lose the preview, and nothing else signals it.
Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else —
keyed by file id, on local disk. No copy path duplicates it and no other
instance can see it, so a copied file lost the preview its owner
uploaded. Silently: the server falls back to rendering one from the
source, or to 204 for a PDF, which has no render path at all. A
user-supplied preview is not derivable from the content, so once lost it
is gone.
The PUT now also records a storage.file_attached_blobs row, which
copy_file_satellites already duplicates, so both copy paths carry it.
Best-effort: the sidecar has already succeeded by then and the user can
see their thumbnail, so failing the request would report an error for an
operation that visibly worked.
Read path consults attachments ahead of every content-derived tier: an
uploaded preview is an explicit choice about THIS file and must beat
anything rendered from its content. Cached under the per-file key — a
content key would leak those bytes to every other file sharing the
content, which is the poisoning the file-keyed table exists to prevent.
store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin:
re-uploading a preview is a deliberate replacement, where a re-derived
thumbnail is the same bytes again. The superseded blob's reference is
released, or it would be pinned forever with nothing pointing at it.
Deletion goes through a trigger, not a hook. file_id is ON DELETE
CASCADE, and on_file_deleted fires AFTER delete_file — by then the
cascade has run and there is nothing left to enumerate. This matters
most for folder deletion, where PG cascades folders to files to
attachments and Rust never sees the rows at all. storage.decrement_blob_ref
keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused
verbatim rather than transcribed into a second trigger that can drift.
DELETE only: a replacement updates in place and is handled in Rust, so
adding UPDATE would double-decrement.
Extracted read_blob_to_bytes, shared by the attached and derived tiers —
the only difference between them is which table produced the hash.
tests/api/attached_thumbnail_copy.hurl guards it. The file is red and
the uploaded thumbnail is green, so a render could never produce the
uploaded bytes; the pre-upload render is captured first and required to
change, which stops three identical renders from satisfying the
byte-equality. Then both copy paths must serve the upload, and after the
original is purged and GC runs, both copies must still serve it — each
holds its own reference, because the rows are duplicated rather than
shared.
Step 9 of docs/plan/derived-blobs.md. content_derived_blobs holds bytes
that are a pure function of a file's content, so they are keyed by that
content and shared by every file holding it. This table holds the
opposite: bytes a user supplied or chose, which must never be shared
across files. The key is what enforces it.
That difference is a security boundary, not a modelling preference. A
content-keyed client preview would let user A upload a file plus a
preview that misrepresents it; when user B later uploads the same bytes,
dedup matches and B is served A's preview. Content-keying is only safe
when the server can derive the bytes — there is nothing to poison,
because the same input yields the same output for everyone.
Required now rather than deferred: the SPA already generates and PUTs
previews for PDFs, and there is no server-side regeneration path for
them, so the sidecar migration has nowhere else to put those bytes.
uploaded_by is NOT NULL with no foreign key, per the provenance
convention rather than the plan's sketch. A FK with ON DELETE SET NULL
discards the audit trail exactly when it matters, and without an
ON DELETE clause it would block deleting a user outright. Deleting the
uploader must not rewrite history.
FileAttachedReferenceSource is registered in built_in_registry before
anything writes to the table, so dedup_gc's reap predicate already knows
it exists — otherwise the first sweep after the first attachment would
delete it. Manifest level only, like the derived source: these blobs are
almost always single-chunk, so contributing at chunk level would
double-count against the aliased hash.
copy_file_satellites gains one arm: attachments are DUPLICATED, since
the key is file_id and the copy is a different file, with uploaded_by
carried over — the person who supplied the bytes did not change because
someone copied the file. Each duplicate takes its own reference, so the
bytes stay deduplicated while the mapping does not.
Both golden SQL tests updated: the new fragment lands inside the reap
predicate's NOT(...) group and as a summed term in the manifest
recompute. Verified on a scratch PG with every migration applied — the
attachment duplicates to 2 rows holding 2 references with provenance
intact, while the content-keyed thumbnail stays 1 row reachable from
both files.
Caught by the api-test storage check: 15 blob files left on disk after a
full cleanup. Since 3736b577 thumbnails are stored as derived blobs, and
each content_derived_blobs row holds a manifest reference — but nothing
ever deleted those rows, so the reference outlived the source and GC
could never reclaim the bytes. The plan specifies this cascade; I
implemented the write and read paths and missed it.
Adds `purge_derived_blobs`, the delete counterpart of
`store_derived_blob`: deletes every row derived from a source hash and
releases the reference each held. It lives on DedupService alongside its
store/find siblings because ThumbnailService cannot hold a DedupService —
it implements BlobLifecycleHook, and holding one would close the
DedupService -> BlobLifecycleService -> hook -> DedupService cycle the
existing comment warns about.
All five reap sites now go through `reap_blob`, which purges then fires
the lifecycle hooks, so no path can drop a blob without first releasing
what was derived from it. Previously each site called fire_blob_hooks
directly, which only cleaned the sidecar files ThumbnailService owns.
`reap_blob` is boxed because it is mutually recursive with
`remove_reference`: releasing a thumbnail's reference can reap the
thumbnail's own blob, which re-enters here. It terminates after one
level — nothing is derived from a thumbnail, so the inner purge finds no
rows. That bound is a property of the data, not an invariant the code
enforces, so it is stated at the definition.
fmt, clippy --all-features --all-targets, unit tests clean. The api-test
storage check is the real verdict — it is what found this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, read path — Option 2 of the two shapes discussed: the derived
blob is consulted LAST, after the sidecar, not first.
Read order is now
moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob
For every thumbnail already on disk the new branch is never reached, so
the database stays off the hot path and a fault in it cannot break a
working gallery. It answers only what disk cannot: a thumbnail rendered
by another instance, or a box whose sidecar was never populated. Legacy
content keeps serving from disk until `derived_import` migrates it.
That inverts the plan's stated order deliberately. Derived-blob-first is
right for the END state, because it is what lets the sidecar be deleted;
sidecar-first is right transitionally, because the risky reordering
should happen after the table has been seen serving real reads. The flip
belongs in the release that removes the sidecar, and the comment at the
branch says so.
The existing precedence is preserved and now documented: the file-keyed
client upload (ext-) is checked BEFORE the content-keyed server render.
That ordering is a security property, not a preference — content-keyed
artifacts are shared across every file with that content, so checking
the file-keyed one first is what keeps one user's uploaded preview from
ever being served for another user's identical file.
Shape notes:
* `find_derived_blob` lands on DedupPort/DedupService as the read
counterpart of `store_derived_blob`, so ThumbnailService needs no pool
field — and therefore ThumbnailService::new, DI and three tests are
untouched.
* It carries `content_type`, which is what will retire the byte-sniffing
in the handlers once reads are table-primary.
* The parameter is `Option<&DedupService>`, concrete rather than
`&dyn DedupPort`: DedupPort uses native `async fn` and so is not
dyn-compatible, and ThumbnailPort is never used as a trait object
(checked) — both handlers hold the concrete Arc. `None` means
sidecar-only, which is exactly today's behaviour and what the abstract
port impl passes.
fmt, clippy --all-features --all-targets, 35 unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.
That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.
DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:
* writes the bytes through the normal CDC path, so derived blobs
inherit the backend, encryption, migration and key rotation that
source content already gets;
* records (source_hash, kind, variant) -> blob_hash;
* releases the reference store_from_stream took IF the mapping
already existed. Two instances racing to render the same thumbnail
must leave ref_count at 1, not 2 — otherwise every re-render
inflates it and pins the blob forever.
ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.
The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.
Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.
fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reap statement is assembled from the registered reference sources,
so it cannot be grepped out of the source tree — and it DELETES
manifests. Hiding it behind a debug filter an operator has to know to
enable was the wrong default: if what GC considers "referenced" ever
changes, that has to be visible on the next boot without anyone going
looking for it.
Reported in testing: `RUST_LOG=info,oxicloud::dedup=debug` did not
surface it, while a global `RUST_LOG=debug` did — at the cost of an
unusably noisy boot. Rather than have operators carry a special filter
for a line describing a destructive statement, promote it.
The statement is whitespace-collapsed into a single `statement` field
so a multi-line query does not sprawl across the boot log, and the
registered `sources` are logged alongside it — that list is what
actually determines the predicate, so a change to it is the thing worth
noticing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes step 1 of docs/plan/derived-blobs.md. The chunk-level
`actual_ref_count` recompute was two correlated subqueries written
inline; it now sums the registered reference sources instead, so
`blobs_consistency` and `dedup_gc` answer "what references this hash"
from one place. If they ever diverged the sweep would bless counts the
collector disagrees with — and the collector wins, destructively.
No behaviour change: the generated expression is the same legacy-files
term (guarded by NOT EXISTS) plus the same manifests-citing-this-chunk
term, and a golden test pins the whole statement byte-for-byte.
Built once at construction, like the reap statement, so the sweep runs
a fixed query per page rather than assembling SQL inside the loop. The
builder refuses an empty registry rather than emitting a query where
every blob looks unreferenced and the entire table reports
refcount_mismatch; there is a test.
DI now constructs one registry and hands the same instance to both
consumers — `DedupService::reference_registry()` is what
`BlobsConsistencyCheck` receives, so agreement is structural rather
than a convention someone has to maintain.
The long comment explaining the single-chunk double-count trap moved
from the query site to the builder's doc comment, where the NOT EXISTS
guard it describes actually lives.
fmt, clippy --all-features --all-targets and the 17 affected unit tests
all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prerequisite 0 of docs/plan/derived-blobs.md. The zero-ref manifest
sweep read:
WHERE m.ref_count <= 0
OR NOT EXISTS (SELECT 1 FROM storage.files f
WHERE f.blob_hash = m.file_hash)
That OR hardcodes "storage.files is the only thing that can reference a
manifest". A thumbnail manifest held by storage.content_derived_blobs
has ref_count = 1, so the first clause is false — but no files row names
a thumbnail's Blob hash, so NOT EXISTS is true, the OR fires, and the
manifest is deleted, its chunks dereferenced and the bytes reaped on the
next sweep. Landing content_derived_blobs before this fix would destroy
the derived tier on the first GC run.
The second clause is not merely defensive: it is the ONLY reap path for
bulk deletes (user cascade, empty_trash), where the PG trigger touches
storage.blobs but never decrements the manifest and the per-file
cleanup_if_orphaned call is skipped. So the fix has to preserve that
role, not just add tables to the NOT EXISTS. It is now the union of
every registered manifest-level source.
Assembled once, not per sweep. An earlier cut of this change put a
format! inside the DELETE, which made the most dangerous statement in
the file unreadable, un-pasteable into psql, and injection-shaped even
though every input is &'static str. The statement is now built at
construction and stored on DedupService, so:
* the reap loop runs a fixed statement with no string work,
* the SQL string is stable, so prepared-statement cache keys are too,
* a golden test pins it byte-for-byte — a reviewer reads the SQL in
the test rather than mentally evaluating the registry,
* initialize() logs it at debug with the contributing source names,
recovering the "paste it into psql" property the literal had.
The registry is mandatory rather than Option. An empty registry makes
"nothing references it" vacuously true for every row, so the builder
panics instead of emitting a statement that would delete every manifest
in the database; DedupService::new always registers the two built-in
sources, so that panic is unreachable by construction. There is a test
for it.
Adds ref_exists_sql to the port, defaulting to (count) > 0 and
overridden by FilesReferenceSource with a real EXISTS. Without it the
reap predicate would have traded today's short-circuiting NOT EXISTS
for a COUNT(*) = 0 that scans every referrer — a regression precisely
on heavily-deduplicated blobs, which is what GC walks most.
fmt, clippy --all-features --all-targets, and the 11 affected unit
tests all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prevent services accessing directly to localstorage and prefer using an astraction layer
to expose full blob. The abstraction layer (dedup services) will cover backend storage
election (local, s3, ...), encryption, etc
This change permit audio_metadata_service, media_metadaa_service, face_indexing_service to handle
blobs without worring of the backend.
note: prefered way to handle blob is the streamed way. Some services may not have this possibility
Benchmark-gated, same rule as ROUND2-22: BEFORE/AFTER with a value-equivalence
gate and rollback-on-regression. Two harnesses — bench_round23_micro (no
Postgres; deterministic allocation gate) and bench_round23_queries (live
Postgres; p50 latency + strict equivalence gate against seeded fixtures). See
benches/ROUND23.md.
- J1: contact_pg_repository::row_to_contact (+ the inlined contact_group sibling)
decode the 3 JSONB columns via sqlx::types::Json<T> (one from_slice pass)
instead of row.get::<serde_json::Value> + from_value (a throwaway Value DOM
per column, walked a second time). Per contact row of every list / multiget /
CardDAV sync. Micro 84 -> 33 allocs/op (2.15x); PG 3794 -> 2360 ns/contact
(1.61x) on 500 real rows.
- J2: DrivePolicies::from_value deserializes straight from the borrow
(T::deserialize(&Value)) instead of from_value(value.clone()) — dropping the
full-DOM clone on every drive-policy read (move/copy, share, grant); one-line
body change, all 7 callers unchanged. Micro 5 -> 0 allocs/op (11.51x).
- P1: get_user_profile overlaps the two independent caller+target reads with
tokio::join! (self-case still a single fetch; caller-error precedence
preserved via caller_res? first) instead of two serial round-trips. PG
577 -> 312 us/call (1.85x).
- G1: subject_group remove_member computes the child's transitive-user recursive
CTE once and reuses it for both the would-empty pre-check and the cache
invalidation, instead of running the identical CTE twice (the edge delete is
above the child, so its descendants can't change). PG 829 -> 412 us/removal
(2.01x).
- U1: dedup_service (store_loose_chunks final registration + the ingest
run_rollback) reshapes the owned, dead-after Vec<(String,i64)> via
into_iter().unzip() instead of cloning every 64-byte hash for the
sync_blobs(&[String]) + UNNEST bind. Micro 256 -> 0 hash clones.
Verified: cargo clippy --features bench --all-targets -D warnings clean, cargo
fmt --all --check clean, cargo test --lib --features bench = 529 passed / 0
failed. The PG benches run against a local PostgreSQL 16 (schema applied from
migrations/); every equivalence gate passes.
The download_zip per-item N+1 (the audit's highest raw-latency candidate) is
deferred to a dedicated pass: its fix moves the sole authorization inside the
stream call, so it needs an AuthZ-ordering + anti-enumeration proof, not a perf
banner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKyQ4AnYtgp1JtjzweyMeo
Round 21 of the benchmark-gated perf sweep. Six behaviour-preserving,
allocation-reducing changes, each with a BEFORE/AFTER counting-allocator
section in examples/bench_round21_micro.rs and a byte/-value equivalence
gate; all six pass their deterministic alloc gate (a non-winning AFTER
exits 1 = rollback).
- R1: pre-size the 16 CalDAV/CardDAV row-mapper Vecs (+1 HashMap) with
Vec::with_capacity(rows.len()) — the ROUND20 §I1 file-side pattern
extended to the calendar/contact repos it deferred. 7 → 1 allocs/op.
- R2: settle_batch binds a borrowed Vec<&str> instead of cloning every
chunk hash into a Vec<String> (sqlx encodes &[&str] as text[]
identically; favorites_pg_repository.rs:271 precedent). 33 → 1 allocs/op,
39x wall.
- R3: store_loose_chunks keys its intra-request dedup set on the raw
[u8;32] BLAKE3 digest and moves the hex on a duplicate (the ROUND17 §D2
pattern applied to the delta-upload sibling). 401 → 209 allocs/op.
- R4: CardDAV getetag emits borrowed pre-escaped " quotes via a shared
write_quoted_etag helper (ROUND20 §C1 pattern, all 4 CardDAV etag sites).
3 → 0 allocs/op.
- R5: BDAY stamped via the new fmt::compact_date stack renderer instead of
chrono's strftime interpreter (chrono fallback out of the 4-digit-year
range; byte-identical, unit-tested vs chrono). 2 → 0 allocs/op, 10.5x wall.
- R6: NC trashbin folder content-type via Cow::Borrowed instead of
.to_string() on the constant (ROUND16 §M1 pattern). 1 → 0 allocs/op.
See benches/ROUND21.md for the full write-up and the deferred-items list
(HeaderMap-clone hot handlers, Query→typed-struct, WebDAV dead-props
HashSet, and others surfaced by the audit that want their own validated
pass). Validated: cargo fmt, cargo clippy --features bench --all-targets
-D warnings, cargo test --lib --features test_utils (529 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015gHVq5Wy2TzdWeSqtEmK6m
Targets the content-addressable dedup write path (the ROUND15-deferred
"dedup_service hash-String re-allocations") from both ends — the streaming
ingest loop and the delta-commit verification read — plus a CardDAV vCard
micro-cut. Every change is benchmark-gated with a hard rollback rule; no
PostgreSQL needed for any arm (benches/ROUND17.md).
Backend (counting-allocator, examples/bench_round17_micro.rs):
- D2 chunk-ingest (store_from_stream, the hottest write path — every chunk of
every upload): the 64-char hex hash String was allocated 3x per chunk
(to_hex + chunk_hashes clone + session_seen insert-clone, the last dropped on
a duplicate). The intra-upload dedup set now keys on the raw 32-byte BLAKE3
digest ([u8;32], Copy, no heap) and the manifest push is branch-split so a
duplicate moves the hex in: 3 -> 2 allocs/new chunk, 3 -> 1/duplicate.
Measured 214 -> 149 allocs/op (1.14x wall) on a 64-chunk 1-in-2-dup batch;
smaller/faster set too (32B inline keys vs 64B heap Strings).
- D1 hash_chunk_sequence (delta-commit verification): took chunks by
&[(String,u64)] and fed the backend stream with iter().cloned(), re-cloning
every chunk hash a second time on top of the owned Vec the caller already
built. Take the Vec by value + into_iter(): 65 -> 0 internal allocs/op, ~2.5us
of clone work removed per verify.
- V1 vCard TYPE tokens (contact_to_vcard + generate_vcard, 5 sites): each
EMAIL/TEL/ADR TYPE= param used ty.to_uppercase() — a throw-away String per
token per contact. New shared fmt::push_upper writes the upper-cased chars
straight into the buffer (byte-identical to str::to_uppercase, unit-tested):
13 -> 5 allocs/op, 1.19x wall.
Gates: each section asserts byte/-value equivalence (D2 the ordered manifest +
sizes + write-set; D1 the removed clone is a pure copy; V1 the full vCard) and
exits non-zero if an AFTER arm fails to reduce allocations. push_upper is
unit-tested byte-equal to str::to_uppercase (fmt::tests). Verified end-to-end:
cargo fmt clean, clippy --release --all-targets --features bench -D warnings
clean, and the harness prints GATE PASS against the built release lib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMmt7vNETYUbEG3Hc17LDx
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):
- Range requests (REST/DAV/shares) answered from the moka content cache
for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
per-session counter (lazy rebuild on cold start). 1,000-chunk upload
gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
the source stream keeps flowing during PG pin + backend writes;
rollback ledger shared + lock-serialized so compensation stays exact
on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
a bounded Web Worker pool (File handles by reference); vitest gate
asserts the pool beats sequential (first gate draft posting buffers
was 2.6x slower and was rewritten — copies dominated).
Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
plan:
```
What step 5 is actually doing
Goal: stop writing to storage.files.user_id / storage.folders.user_id on new INSERTs. This is a prerequisite for step 6 (dropping the column entirely). The column has existed since the pre-D0 single-owner era; every read that used to key on it has already been migrated to drive-membership grants over the last N days.
What has to change together for this to be safe:
┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Piece │ What changes │ Why │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Swap storage.files uniqueness indexes from (folder_id, name, user_id) to (drive_id, folder_id, name) │ Otherwise, two new rows with user_id = NULL would both be allowed (PG treats NULLs as distinct) — │
│ (a) │ │ uniqueness silently breaks │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ ALTER user_id DROP NOT NULL on both tables │ Otherwise, dropping the INSERT bind violates NOT NULL and every write 500s │
│ (b) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Drop dead user_id-leading indexes │ Cheap cleanup — nothing scans them anymore │
│ (c) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ PL/pgSQL (d) │ Rewrite storage.copy_folder_tree without user_id in the INSERT column list │ Cross-drive copy runs entirely in SQL, needs the same treatment │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (e) │ ~12 INSERT column-list drops across folder_db_repository, file_blob_write_repository, drive_pg_repository, dedup_service, │ Actual write path │
│ │ folder_service, load-seed.rs │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (f) │ Simplify folder_db_repository::create_folder's parent lookup to only fetch drive_id (was fetching (user_id, drive_id)) │ It's fetching a value it no longer needs │
└──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Stamp `orphaned_at = now()` on the ref-0 row so it sits inside the
GC grace window for the duration of this test. Without it,
`orphaned_at IS NULL` is treated by `garbage_collect` as
"pre-migration, immediately reapable" — and any sibling test in
the shared pool that calls `garbage_collect()` (e.g.
`garbage_collect_respects_grace_and_cross_checks`) would race
with the pin below and delete the row first.
call fire_blob_hooks to respect lifecycle
dedup_service::garbage_collect_with_grac must call fire_blob_hooks() once blob are dropped
so other services like thumbnail can proceed to their cleanup
renable thumbnail test, ensure that blob lifecycle correctly
trigger thumbnail cleanup on blob deletion
need to call `/api/admin/internal/trigger-gc?force=true`
read_blob_bytes read the same storage.chunk_manifests PK row twice per
full-blob read — blob_size (SELECT total_size) then read_blob_stream
(SELECT chunk_hashes) — even though both columns live in one row. Fold
them into a single `SELECT chunk_hashes, total_size` and share the chunk
stream builder via a new stream_chunks helper. The legacy (no-manifest)
path is unchanged. Output is identical; the read just costs one fewer DB
round-trip.
Benchmark (examples/bench_blob_manifest.rs, isolates the manifest lookup
against the real Postgres): ~1.9x throughput and p50/p99 roughly halved on
that sub-step; the win is the removed round-trip under pool pressure during
upload bursts. Note this is the manifest sub-step only — end-to-end
read_blob_bytes is dominated by the actual chunk reads, and it is a
background path (thumbnail generation / EXIF / indexing), not normal gallery
serving. Methodology + honest framing in benches/BLOB-MANIFEST.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
remove_manifest_reference unlinked a chunk's backing file right after the
row-delete committed — the same TOCTOU the GC grace window was added to
close: a concurrent upload of identical content can re-reference (pin) the
chunk in the gap between commit and unlink, after which the deferred
unlink strands a referenced chunk with no bytes.
Route physical chunk reclamation through the single grace-protected path:
on last reference, delete the manifest and decrement its chunks (stamping
orphaned_at on the ones that reach 0), but leave the chunk rows and files
for garbage_collect() to reclaim once orphaned past the grace window. The
manifest deletion and its blob-keyed thumbnail hook stay eager.
remove_legacy_reference and cleanup_if_orphaned's legacy path are left as
eager deletes on purpose: a legacy whole-file hash can never be re-created
by an ingest (uploads are always CDC now), so there is no writer to race —
the existing "row gone ⇒ no resurrection" reasoning holds for them.
Adds an integration test asserting a CDC manifest dereference leaves
chunks orphaned-but-present, then reclaimed by a post-grace GC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
Deduplication GC (garbage_collect, Phase 2):
- Add an orphan grace period before a ref_count=0 blob's backing file is
physically deleted, mirroring git's gc.pruneExpire. New
storage.blobs.orphaned_at records when a blob last reached ref_count 0;
the delete trigger and every decrement / 0-ref insert path stamp it,
every re-reference clears it.
- Cross-check that no manifest lists the chunk and no file points at the
blob before deleting it (mirrors Phase 1's file check), so a stale
ref_count can only delay collection, never delete live content.
- Unlink the backing files with bounded parallel fan-out.
Together these close a TOCTOU where a concurrent upload of identical
content could re-reference a chunk in the window between the GC row
delete committing and the backing file being unlinked. Individual file
deletes still reclaim eagerly; only bulk empty-trash and the periodic
sweep observe the grace window.
Trash: match ErrorKind::NotFound instead of substring-matching the error
message when treating an already-deleted item as success.
Content-index worker: supervise the drain loop and restart it with
backoff after a panic, instead of letting a panic silently freeze the
search index while the dirty queue grows unbounded.
Adds migration 20260802000000_blob_gc_grace.sql and an integration test
covering the grace window and reference cross-checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
- permit reuse of blob where file is trashed (by-hash and chunked)
- add hurl test on /api/files/by-hash
- add anti enumeration of blob (404 is always blob_not_owned_by_caller)
read_blob_stream / read_blob_range_stream reassembled a CDC file by fetching
its chunks with `buffered(1)` — strictly sequential, so the next chunk's
backend fetch (a file `open` locally; a full request round-trip on S3/Azure)
only started after the current chunk was fully drained.
A benchmark of the exact pipeline (stream::iter(chunks).map(get).buffered(K)
.try_flatten()) showed a blind `buffered(4)` is the WRONG fix: on a local
disk it is neutral on a warm page cache and ~37% SLOWER cold, because
concurrent opens turn one sequential read into several competing random-I/O
streams over content-addressed (scattered) chunk files. The win is entirely
on remote backends, where per-chunk request latency dominates and overlapping
fetches hide it (≈ linear in K).
So the read-ahead depth is now a backend hint, not a constant:
- BlobStorageBackend::read_prefetch() default 1 (sequential; safe for local).
- S3 / Azure override to 8 (overlap GETs to hide TTFB).
- cached / encrypted / retry / migration delegate to the backend that serves
the bytes.
- Both CDC read paths use `self.backend.read_prefetch().max(1)`.
Net: local backend unchanged (no regression); remote reassembly ~4-8x faster.
Ordered `buffered` (not buffer_unordered) keeps chunks in sequence.
Bench (per-chunk fetch-latency model): buffered(1)->(4)/(8) = x3.9 / x7.8
@1ms, x4.0 / x8.1 @5ms, x4.0 / x8.0 @20ms. Local warm: 230ms@1 vs 227ms@4
(noise); local cold: 425ms@1 vs 585ms@4 (why local stays at 1).
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Edouard Vanbelle