The disk checks above prove nothing leaked. These prove the bookkeeping
behind them is honest: every refcount matches what the reference sources
hold, and no row points at bytes that are gone.
End of suite is the only place this is cheap. One database serves every
hurl file, so by here the counters have absorbed every upload, copy,
move, share, trash and purge the suite performed — across both copy
paths, the derived tier and the attached tier. Drift that no individual
test would notice, because each only inspects its own file, surfaces as a
mismatch.
Runs after the GC drain deliberately: mid-sweep state is legitimately
inconsistent — a manifest can sit at zero waiting for the next pass — so
checking earlier would report normal in-flight state as drift.
Zero findings is the assertion. These four tenants are read-only, so
anything they report is a real invariant violation rather than a repair
opportunity. A job missing from the build is skipped with a warning
instead of failing, so this does not break on a feature-gated build.
Unknown job names and unwrapped-vs-wrapped response shapes both degrade
to a visible warning rather than a silent pass: list_job_runs currently
returns a bare array, and the .runs/.items fallbacks exist so a future
wrapping does not quietly turn the whole check into a no-op.
Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else —
keyed by file id, on local disk. No copy path duplicates it and no other
instance can see it, so a copied file lost the preview its owner
uploaded. Silently: the server falls back to rendering one from the
source, or to 204 for a PDF, which has no render path at all. A
user-supplied preview is not derivable from the content, so once lost it
is gone.
The PUT now also records a storage.file_attached_blobs row, which
copy_file_satellites already duplicates, so both copy paths carry it.
Best-effort: the sidecar has already succeeded by then and the user can
see their thumbnail, so failing the request would report an error for an
operation that visibly worked.
Read path consults attachments ahead of every content-derived tier: an
uploaded preview is an explicit choice about THIS file and must beat
anything rendered from its content. Cached under the per-file key — a
content key would leak those bytes to every other file sharing the
content, which is the poisoning the file-keyed table exists to prevent.
store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin:
re-uploading a preview is a deliberate replacement, where a re-derived
thumbnail is the same bytes again. The superseded blob's reference is
released, or it would be pinned forever with nothing pointing at it.
Deletion goes through a trigger, not a hook. file_id is ON DELETE
CASCADE, and on_file_deleted fires AFTER delete_file — by then the
cascade has run and there is nothing left to enumerate. This matters
most for folder deletion, where PG cascades folders to files to
attachments and Rust never sees the rows at all. storage.decrement_blob_ref
keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused
verbatim rather than transcribed into a second trigger that can drift.
DELETE only: a replacement updates in place and is handled in Rust, so
adding UPDATE would double-decrement.
Extracted read_blob_to_bytes, shared by the attached and derived tiers —
the only difference between them is which table produced the hash.
tests/api/attached_thumbnail_copy.hurl guards it. The file is red and
the uploaded thumbnail is green, so a render could never produce the
uploaded bytes; the pre-upload render is captured first and required to
change, which stops three identical renders from satisfying the
byte-equality. Then both copy paths must serve the upload, and after the
original is purged and GC runs, both copies must still serve it — each
holds its own reference, because the rows are duplicated rather than
shared.
Step 9 of docs/plan/derived-blobs.md. content_derived_blobs holds bytes
that are a pure function of a file's content, so they are keyed by that
content and shared by every file holding it. This table holds the
opposite: bytes a user supplied or chose, which must never be shared
across files. The key is what enforces it.
That difference is a security boundary, not a modelling preference. A
content-keyed client preview would let user A upload a file plus a
preview that misrepresents it; when user B later uploads the same bytes,
dedup matches and B is served A's preview. Content-keying is only safe
when the server can derive the bytes — there is nothing to poison,
because the same input yields the same output for everyone.
Required now rather than deferred: the SPA already generates and PUTs
previews for PDFs, and there is no server-side regeneration path for
them, so the sidecar migration has nowhere else to put those bytes.
uploaded_by is NOT NULL with no foreign key, per the provenance
convention rather than the plan's sketch. A FK with ON DELETE SET NULL
discards the audit trail exactly when it matters, and without an
ON DELETE clause it would block deleting a user outright. Deleting the
uploader must not rewrite history.
FileAttachedReferenceSource is registered in built_in_registry before
anything writes to the table, so dedup_gc's reap predicate already knows
it exists — otherwise the first sweep after the first attachment would
delete it. Manifest level only, like the derived source: these blobs are
almost always single-chunk, so contributing at chunk level would
double-count against the aliased hash.
copy_file_satellites gains one arm: attachments are DUPLICATED, since
the key is file_id and the copy is a different file, with uploaded_by
carried over — the person who supplied the bytes did not change because
someone copied the file. Each duplicate takes its own reference, so the
bytes stay deduplicated while the mapping does not.
Both golden SQL tests updated: the new fragment lands inside the reap
predicate's NOT(...) group and as a summed term in the manifest
recompute. Verified on a scratch PG with every migration applied — the
attachment duplicates to 2 rows holding 2 references with provenance
intact, while the content-keyed thumbnail stays 1 row reachable from
both files.
fe9c4f49 made the ETag content-keyed, but the RAM tier was still keyed on
file_id, so the two disagreed about what identifies a thumbnail. Replacing
a file's content preserves its id, so the moka entry stayed reachable while
the ETag had already changed — and invalidation runs from the spawned task
in on_file_updated. A request landing in that window got the NEW ETag over
the OLD bytes, and because the response is immutable with a one-year
max-age, the client cached those stale bytes permanently. The bug fe9c4f49
set out to fix, arriving through a different door.
Keying on the hash removes the window rather than narrowing it: new content
is a different key, so the old entry cannot be hit. Correctness no longer
depends on the invalidation task winning a race against the next request.
This also aligns the RAM tier with what disk already did — sidecars have
always been written to get_thumbnail_path(blob_hash, ...). The tier that
had the bug was the one keyed differently from every other. Two further
consequences: N copies of one photo now share a single entry instead of
occupying N for identical bytes, and delete_thumbnails shrinks to the
external entries, which are the only genuinely per-file artifacts.
Video frames stay file-keyed under an `ext-{file_id}` id — they are
per-file by nature. The namespaces cannot collide: hashes are 64 hex
characters.
get_cached_thumbnail takes blob_hash as an Option, and a caller without one
now skips the RAM tier and falls through to disk rather than consulting a
file-id key. That is correct, not merely tolerable — a file-id key is the
stale entry this change exists to prevent. Both HTTP handlers resolve the
hash to build the ETag, so only internal callers that never had one are
affected.
GET /api/files/{id} is the download route — it returned the PNG bytes,
so the jsonpath capture failed on a UTF-8 decode. /{id}/metadata is the
EXIF endpoint and carries no FileDto either. Listing the folder gives
the DTO, and since the folder holds exactly this one file, count == 1
also proves the WebDAV PUT overwrote in place rather than creating a
second file beside it.
Also drops an unused bytes capture and records why the body is not
asserted after the overwrite: the moka tier is keyed on file_id and
invalidated from the spawned task in on_file_updated, so a request
landing first sees the previous bytes under the new ETag. Asserting on
bytes would be a race.
The upload omitted folder_id, which the handler needs to resolve the
file's owner — it answers 500, not a root upload. Every other upload in
the suite passes it; this was the only one that did not, which is why
nothing caught it earlier.
The folder also gives the WebDAV overwrite a deterministic path
(/webdav/hurl-etag-src/<name>) instead of depending on where a
folder-less upload would have landed. Teardown now removes it and
purges it from trash, keeping the shared database clean for the files
that run after.
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format),
which is one term short: a thumbnail is a function of those PLUS the
renderer. Change the encoder or a quality setting and identical inputs
produce different bytes under an unchanged ETag — the same staleness
class the commit fixed, one level down. It bites when an already-cached
thumbnail is re-rendered after a renderer change.
Keying on the derived blob's own hash removes the term entirely: the
ETag IS the hash of the bytes, so any output change invalidates by
construction. It is self-consistent for free, because store_derived_blob
is ON CONFLICT DO NOTHING — a re-render never displaces the stored row,
so the ETag always equals what the derived tier will serve. No renderer
version constant to remember to bump.
Records why it cannot land yet. The derived tier is read LAST by design,
so an ETag naming the derived hash would describe a tier the response
probably did not come from; sidecar and derived agree at creation but
diverge if a sidecar is re-rendered while the derived row stays pinned
by DO NOTHING. An ETag that lies about the body is worse than one that
is merely coarse. Also the tier is WebP-only (variant is the size, with
no format term) and empty for anything predating this work until
derived_import backfills.
So it lands at step 10 with the flip, keeping today's form as the
fallback for ungenerated variants and formats the tier does not hold.
The LEFT JOIN already planned for the read path returns the derived
hash in the same query, so it costs no extra round-trip.
The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with
Cache-Control: public, max-age=31536000, immutable. Replacing a file's
content preserves its id — file_upload_service rebuilds the entity with
parts.id and a new hash, then fires on_file_updated, which deletes and
regenerates the thumbnails — so the server produced a new thumbnail while
still advertising the old ETag. Because `immutable` tells a conforming
browser not to revalidate at all inside the freshness window, clients kept
rendering the previous image for up to a year, unfixably.
Keyed on the content hash the directive becomes honest: a thumbnail is a
pure function of (source bytes, size, format), so that triple identifies
the response. New content yields a new ETag.
The same change fixes the opposite direction. A copy, or any dedup twin,
had a different id and therefore a different ETag, so clients refetched
bytes they already held even though both are served from the same derived
blob. Now identical content agrees on an ETag and revalidates to 304
across files, users and copies.
Both thumbnail endpoints were affected: the REST handler and the
NextCloud preview handler.
Cost is one PK lookup ahead of the 304 decision, where the id-keyed
version needed none — paid for by no longer serving stale images. It is
partly recovered: both handlers already resolved the same hash further
down for the render path, and that second lookup is now gone, so the
cache-miss path is unchanged and only the 304 path pays. The resolved
hash is also handed to get_cached_thumbnail instead of None, saving the
service its own lookup.
No new disclosure: content_hash is already on FileDto and returned by
GET /api/files/{id}.
Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite
in place via WebDAV PUT, assert the ETag changed, assert a client holding
the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the
sharing direction: a copy answers with the SAME ETag and revalidates to
304, which is the one externally observable consequence of content-keying
and was not previously testable.
The byte-identity assertions were documented as proving that a copy
shares the original's content_derived_blobs row. They prove no such
thing: rendering is deterministic in the source bytes and the variant,
so a copy that re-rendered from scratch returns identical bytes. The
copy is in fact a moka hit — that cache is keyed on
(source_hash, size, format), which the copy shares — so it never
reaches the derived tier here at all.
Nor is there an assertion that would fix it. Duplication is impossible
by construction: the PK is (source_hash, kind, variant), a copy carries
the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING.
The schema enforces the property, so no runtime behaviour can violate
it and there is nothing to catch.
Same limitation narrows step 11: it proves the SOURCE content survived
GC, not the derived blob — a reaped derived blob is re-rendered
transparently from the live source.
What the file does prove is unchanged and is the part that was broken:
both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and
purging the original does not destroy the copies. No assertions changed.
Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was
written twice — the copy_file CTE and storage.copy_folder_tree — and had
already drifted: the tree path bumped storage.blobs only, missing
manifests, which was silent data loss on any multi-chunk file. Fixing it
meant writing the same logic a second time. Step 9 adds a file-keyed
satellite table, which would mean a third and fourth.
Two SQL functions:
storage.add_blob_references(TEXT[]) — the manifest-first reference
contract for SQL callers, returning hashes that matched no registry
row. Set-based so the tree path keeps its single-statement cost; a
per-row helper would have made a 10k-file copy 10k calls.
storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus
the blob reference. The body is the copy-semantics declaration: what
is absent (comments, favorites, content-keyed derived rows) is listed
with its reason, so the taxonomy is executable rather than documented
elsewhere and drifting.
Both copy paths now call it. The single-file path becomes a real
transaction, which also fixes the reference being best-effort: a failed
add_reference used to log a warning and leave a copy holding no
reference at all — the exact shape that gets its content reaped. It
cannot be a CTE arm, because data-modifying CTEs share one snapshot and
the function must read the row the INSERT just wrote.
Verified against a scratch PG with all migrations applied: multi-chunk
manifest 1→2, single-chunk alias bumped at manifest level only (the
NOT EXISTS guard), chunks behind a manifest untouched, dead properties
duplicated, length mismatch rejected, repeats counted.
tests/api/derived_blob_copy.hurl covers it end-to-end and answers the
question the copy raises: content_derived_blobs is NOT copied. A copy
carries the same blob_hash, so it resolves the same derived row — the
test asserts byte-identical thumbnails from both copy paths, then
deletes the original, runs GC, and requires both copies to still serve.
That last step only passes if the references are real.
5343fdda switched S3 blob enumeration from an opaque continuation token
to a hash cursor (StartAfter), per the port contract. Its fallback for a
page containing no canonical blob was wrong: it stored the full key
(`0a/junk.tmp`), stripped it to a basename (`junk.tmp`), and the next
call fed that to `object_key()` — producing `ju/junk.tmp.blob`. Wrong
shard and a doubled extension, so the resume jumped to an arbitrary
position: skipped objects, or backwards into a loop.
The cursor can only ever be a real hash, because `object_key()` is
applied to it. So instead of synthesising one, keep listing internally
until the page holds at least one blob or the bucket is exhausted. The
continuation token is used only inside the call and never escapes.
Two pathological cases cannot produce a cursor at all — `is_truncated`
with no token (protocol violation), and a run of foreign keys long
enough to buffer the bucket. Both now fail loudly. A visible job failure
beats a sweep reporting "no missing blobs" having read a fraction of
them.
Extract `hash_from_object_key` as the paired inverse of `object_key`,
with the round-trip and the rejection set under test. It also now
requires the shard to match the hash's own prefix, which the inline
filter did not check.
Precondition for the merge-join in backend_consistency (step 6 /
option A of docs/plan/derived-blobs.md), landed separately because it
is independently useful and carries the risk.
Two contract changes on BlobStorageBackend::list_blob_hashes:
1. Entries MUST be in ascending hash order. Every shipped backend
already did this — local sorts within each shard and walks 00..ff,
and since the shard IS the hash prefix that is globally sorted; S3
and Azure list lexicographically by key and blobs/<xx>/<hash> sorts
identically to <hash>. It was accidental, and a future backend
enumerating in any other order would have silently made the
merge-join emit bogus blob_missing_from_backend findings at
data_loss severity.
2. The cursor is the last hash returned, not an opaque backend token.
This is what lets a caller resume from a checkpoint it already
holds — the merge-join keeps one cursor for both the DB walk and
the backend walk instead of a compound one, which in turn means
blobs_consistency's existing cursor format survives and no paused
run is stranded.
Local already derived its position from a hash; it now emits the bare
hash instead of "<shard>/<hash>", and still accepts both legacy forms
so a run paused across this deploy resumes. The bare-shard form works
through the same path unchanged, since "3f" sorts before every 64-char
hash beginning "3f".
S3 moves from continuation_token to StartAfter, which supports this
natively. One non-obvious case handled: a page can contain only
non-canonical keys (.tmp spool files, .corrupt sidecars), which are
filtered into `unknowns`, leaving `blobs` empty — a naive
blobs.last() would return no cursor and silently end enumeration while
is_truncated said otherwise, making an audit job under-report. It now
falls back to the last key seen; StartAfter is a string comparison, so
a non-hash resume point is fine. "Cursor is a hash" constrains what
callers may synthesise, not what backends may return.
Azure is unaffected — it does not implement list_blob_hashes (TODO,
inherits the NotSupported default).
Adds the first test for enumeration at all: ordering across shards with
deliberately out-of-order inserts, complete paged traversal, and
resume from a caller-synthesised cursor.
NOT verified against real S3 — no bucket available here. The local path
is covered by the new test; the StartAfter change is reasoned from the
API contract and needs exercising against a real bucket before it is
relied on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One dedup_gc pass cannot fully drain now that thumbnails are derived
blobs. Reaping a source releases the references its derived artifacts
hold (each content_derived_blobs row pins a manifest), and those
releases happen mid-sweep — the derived chunks are stamped orphaned as
the pass is already walking past them, because
remove_manifest_reference deliberately does not unlink, to avoid racing
a concurrent upload re-referencing the same chunk. They are collectible
only on the NEXT sweep, which is why the check saw 15 leftover blobs.
Loops until a pass reclaims nothing rather than hardcoding two. Two is
correct only while the derivation graph is one level deep — a thumbnail
is derived from a file, nothing is derived from a thumbnail. That is a
property of the data, not an invariant the code enforces, so a fixed
count would silently under-drain the day transcodes-of-thumbnails or
E2E-wrapped derivatives exist, and the failure would surface as a
confusing leftover-file assertion rather than the design change it is.
Bounded at 3 with a warning if it does not settle.
Sleeps between passes. The JobRegistry serialises runs of the same job,
so a back-to-back trigger risks rejection as already-running — which
returns 0 reaped and would exit the loop early, declaring success with
blobs still on disk. A false pass is worse than a slow one. It also
gives the previous pass's detached unlink tasks (spawned by
on_blob_deleted, awaited by nothing) time to land.
Deliberately NOT fixed in production code: derived chunks land inside
the 1-hour orphan grace, so a second immediate sweep would collect
nothing there and the next scheduled run picks them up. A fixpoint loop
in garbage_collect would be dead code outside force=true, which is only
this test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught by the api-test storage check: 15 blob files left on disk after a
full cleanup. Since 3736b577 thumbnails are stored as derived blobs, and
each content_derived_blobs row holds a manifest reference — but nothing
ever deleted those rows, so the reference outlived the source and GC
could never reclaim the bytes. The plan specifies this cascade; I
implemented the write and read paths and missed it.
Adds `purge_derived_blobs`, the delete counterpart of
`store_derived_blob`: deletes every row derived from a source hash and
releases the reference each held. It lives on DedupService alongside its
store/find siblings because ThumbnailService cannot hold a DedupService —
it implements BlobLifecycleHook, and holding one would close the
DedupService -> BlobLifecycleService -> hook -> DedupService cycle the
existing comment warns about.
All five reap sites now go through `reap_blob`, which purges then fires
the lifecycle hooks, so no path can drop a blob without first releasing
what was derived from it. Previously each site called fire_blob_hooks
directly, which only cleaned the sidecar files ThumbnailService owns.
`reap_blob` is boxed because it is mutually recursive with
`remove_reference`: releasing a thumbnail's reference can reap the
thumbnail's own blob, which re-enters here. It terminates after one
level — nothing is derived from a thumbnail, so the inner purge finds no
rows. That bound is a property of the data, not an invariant the code
enforces, so it is stated at the definition.
fmt, clippy --all-features --all-targets, unit tests clean. The api-test
storage check is the real verdict — it is what found this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, read path — Option 2 of the two shapes discussed: the derived
blob is consulted LAST, after the sidecar, not first.
Read order is now
moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob
For every thumbnail already on disk the new branch is never reached, so
the database stays off the hot path and a fault in it cannot break a
working gallery. It answers only what disk cannot: a thumbnail rendered
by another instance, or a box whose sidecar was never populated. Legacy
content keeps serving from disk until `derived_import` migrates it.
That inverts the plan's stated order deliberately. Derived-blob-first is
right for the END state, because it is what lets the sidecar be deleted;
sidecar-first is right transitionally, because the risky reordering
should happen after the table has been seen serving real reads. The flip
belongs in the release that removes the sidecar, and the comment at the
branch says so.
The existing precedence is preserved and now documented: the file-keyed
client upload (ext-) is checked BEFORE the content-keyed server render.
That ordering is a security property, not a preference — content-keyed
artifacts are shared across every file with that content, so checking
the file-keyed one first is what keeps one user's uploaded preview from
ever being served for another user's identical file.
Shape notes:
* `find_derived_blob` lands on DedupPort/DedupService as the read
counterpart of `store_derived_blob`, so ThumbnailService needs no pool
field — and therefore ThumbnailService::new, DI and three tests are
untouched.
* It carries `content_type`, which is what will retire the byte-sniffing
in the handlers once reads are table-primary.
* The parameter is `Option<&DedupService>`, concrete rather than
`&dyn DedupPort`: DedupPort uses native `async fn` and so is not
dyn-compatible, and ThumbnailPort is never used as a trait object
(checked) — both handlers hold the concrete Arc. `None` means
sidecar-only, which is exactly today's behaviour and what the abstract
port impl passes.
fmt, clippy --all-features --all-targets, 35 unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.
That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.
DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:
* writes the bytes through the normal CDC path, so derived blobs
inherit the backend, encryption, migration and key rotation that
source content already gets;
* records (source_hash, kind, variant) -> blob_hash;
* releases the reference store_from_stream took IF the mapping
already existed. Two instances racing to render the same thumbnail
must leave ref_count at 1, not 2 — otherwise every re-render
inflates it and pins the blob forever.
ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.
The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.
Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.
fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-13 runner tier is being phased out by GitHub — queues persistently
exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to
'cargo install --features bundled-assets' from source, Docker
--platform linux/amd64, or a Linux VM.
JS-based GH Actions (checkout, artifact steps, setup-node) can't run
inside Alpine on ARM64 — Node.js binary shipped by the actions
requires glibc, and the x64-Alpine workaround doesn't extend to
arm64. Cross-compile natively via 'rustup target add' + musl-tools
instead.
The 'broken pipe' error under set -euo pipefail was masking the real
answer. Capture status + content-type + first-char in one curl, dump
first 200 bytes of body on failure so we can see what the server
actually served.
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in
example.env and docs/config/env.md — closes the discoverability gap
where the env var was only visible in Rust docstrings.
Also lands docs/plan/bundled-binary.md — the design record referenced
from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs,
and the Dockerfile.
Adds the tag-triggered workflow that builds 4 musl-linux + macOS
tarballs and attaches them to the tag's GitHub Release. Ships a
matching install guide (docs/install/binary.md) with SHA256SUMS
verify, systemd unit, upgrade flow, and hardware notes. Adds
[package.metadata.binstall] so 'cargo binstall oxicloud' works
automatically once the first release lands.
Also re-enables incremental compilation in the dev profile — the
'modest single-crate savings' rationale from when the crate was small
has been outgrown; full rebuild ~10 min is now the dev-loop bottleneck.
execute once via SQL migration to fix old entries
next: will be catch by manifests_consistency job, if new are found it means new bug discovered
no auto repair to prevent hidding bugs, operator can still invoke repair while waiting a fix
Silent data loss on an ordinary UI folder copy.
The function bumped only storage.blobs:
UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
FROM (...) hc WHERE b.hash = hc.blob_hash;
but a CDC file's blob_hash names a MANIFEST, not a chunk. For any
multi-chunk file that predicate matches zero rows, so the copy took no
reference at all. Delete the original afterwards and remove_reference
walks the manifest to 0, dedup_gc reaps the manifest and every chunk
behind it, and the copy is unreadable.
Single-chunk files escaped by accident: their whole-file hash equals
their lone chunk's hash, so the UPDATE did match — bumping the wrong
counter, which surfaces as a manifest under-count plus a blob
over-count rather than as loss. That asymmetry is why the bug survived:
small files, which dominate most test corpora, look fine.
Reproduced through the UI on a 5 MiB / 18-chunk file:
chunk_manifests.ref_count stayed at 1 while two storage.files rows
referenced it, and manifests_consistency reported
manifest_refcount_mismatch with delta 1, reap_risk true.
The fix mirrors DedupService::add_reference — manifest first, blobs only
as fallback, with a NOT EXISTS guard so a single-chunk file is not
counted at both levels (which would turn the under-count into an
over-count). orphaned_at is cleared on the blobs branch, as
add_reference does when resurrecting a blob inside its GC grace window.
Only the reference-counting block changes; the rest of the function is
20260902000001 verbatim.
Existing drift is deliberately NOT repaired here — a schema migration
cannot know which counter is authoritative. manifests_consistency
reports it; repair belongs with the recovery framework.
NOT executed against a database: the test instance was down and the dev
instance is read-only by convention. A parse error would fail at boot,
before any data is touched. Verify by re-running the reproduction — a
folder copy of a >1 MiB file should now leave ref_count at 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reap statement is assembled from the registered reference sources,
so it cannot be grepped out of the source tree — and it DELETES
manifests. Hiding it behind a debug filter an operator has to know to
enable was the wrong default: if what GC considers "referenced" ever
changes, that has to be visible on the next boot without anyone going
looking for it.
Reported in testing: `RUST_LOG=info,oxicloud::dedup=debug` did not
surface it, while a global `RUST_LOG=debug` did — at the cost of an
unusably noisy boot. Rather than have operators carry a special filter
for a line describing a destructive statement, promote it.
The statement is whitespace-collapsed into a single `statement` field
so a multi-line query does not sprawl across the boot log, and the
registered `sources` are logged alongside it — that list is what
actually determines the predicate, so a change to it is the thing worth
noticing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 3 (prerequisite 2) of docs/plan/derived-blobs.md, and the last one
before the thumbnail slice.
There are two reference counters and only one was ever verified.
add_reference bumps chunk_manifests.ref_count first and only falls back
to storage.blobs.ref_count, so a reference lands on whichever counter
its hash names: chunk references feed storage.blobs and are reconciled
by blobs_consistency::refcount_mismatch, while Blob references — every
CDC file, and every derived artifact once those exist — feed
chunk_manifests.ref_count, which nothing reconciled.
That gap was survivable only because dedup_gc's reap predicate carried a
second clause ("no storage.files row references this manifest") that
quietly compensated for drift on the bulk-delete paths where ref_count
is never decremented. Generalising that clause to the reference registry
in 1c8ead49 — so thumbnails stop being reaped — removed the
compensation, which is precisely why the counter now needs checking
directly. The two changes have to ship together.
Adds manifests_consistency, a recoverable job reporting
manifest_refcount_mismatch (severity inconsistent). The finding carries
reap_risk so an operator can triage: an under-count means GC reaps a
manifest whose content is still reachable, taking its chunks with it,
while an over-count merely pins storage.
A separate job rather than a second phase of blobs_consistency: one
subject per job, as the other five consistency tenants do, and it avoids
changing the cursor format of an existing recoverable job — which would
strand any run paused across the deploy.
The page query is assembled from the same registry dedup_gc reaps from
(via DedupService::reference_registry), built once at construction, and
pinned by a golden test. Two invariants the test guards: the files term
carries no NOT EXISTS guard — that guard keeps CDC rows out of the
*chunk* level and here would count nothing — and chunk_hashes appears
nowhere, since a manifest citing its own chunks is not a referrer of
itself.
fmt, clippy --all-features --all-targets, and 3 new unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes step 1 of docs/plan/derived-blobs.md. The chunk-level
`actual_ref_count` recompute was two correlated subqueries written
inline; it now sums the registered reference sources instead, so
`blobs_consistency` and `dedup_gc` answer "what references this hash"
from one place. If they ever diverged the sweep would bless counts the
collector disagrees with — and the collector wins, destructively.
No behaviour change: the generated expression is the same legacy-files
term (guarded by NOT EXISTS) plus the same manifests-citing-this-chunk
term, and a golden test pins the whole statement byte-for-byte.
Built once at construction, like the reap statement, so the sweep runs
a fixed query per page rather than assembling SQL inside the loop. The
builder refuses an empty registry rather than emitting a query where
every blob looks unreferenced and the entire table reports
refcount_mismatch; there is a test.
DI now constructs one registry and hands the same instance to both
consumers — `DedupService::reference_registry()` is what
`BlobsConsistencyCheck` receives, so agreement is structural rather
than a convention someone has to maintain.
The long comment explaining the single-chunk double-count trap moved
from the query site to the builder's doc comment, where the NOT EXISTS
guard it describes actually lives.
fmt, clippy --all-features --all-targets and the 17 affected unit tests
all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prerequisite 0 of docs/plan/derived-blobs.md. The zero-ref manifest
sweep read:
WHERE m.ref_count <= 0
OR NOT EXISTS (SELECT 1 FROM storage.files f
WHERE f.blob_hash = m.file_hash)
That OR hardcodes "storage.files is the only thing that can reference a
manifest". A thumbnail manifest held by storage.content_derived_blobs
has ref_count = 1, so the first clause is false — but no files row names
a thumbnail's Blob hash, so NOT EXISTS is true, the OR fires, and the
manifest is deleted, its chunks dereferenced and the bytes reaped on the
next sweep. Landing content_derived_blobs before this fix would destroy
the derived tier on the first GC run.
The second clause is not merely defensive: it is the ONLY reap path for
bulk deletes (user cascade, empty_trash), where the PG trigger touches
storage.blobs but never decrements the manifest and the per-file
cleanup_if_orphaned call is skipped. So the fix has to preserve that
role, not just add tables to the NOT EXISTS. It is now the union of
every registered manifest-level source.
Assembled once, not per sweep. An earlier cut of this change put a
format! inside the DELETE, which made the most dangerous statement in
the file unreadable, un-pasteable into psql, and injection-shaped even
though every input is &'static str. The statement is now built at
construction and stored on DedupService, so:
* the reap loop runs a fixed statement with no string work,
* the SQL string is stable, so prepared-statement cache keys are too,
* a golden test pins it byte-for-byte — a reviewer reads the SQL in
the test rather than mentally evaluating the registry,
* initialize() logs it at debug with the contributing source names,
recovering the "paste it into psql" property the literal had.
The registry is mandatory rather than Option. An empty registry makes
"nothing references it" vacuously true for every row, so the builder
panics instead of emitting a statement that would delete every manifest
in the database; DedupService::new always registers the two built-in
sources, so that panic is unreachable by construction. There is a test
for it.
Adds ref_exists_sql to the port, defaulting to (count) > 0 and
overridden by FilesReferenceSource with a real EXISTS. Without it the
reap predicate would have traded today's short-circuiting NOT EXISTS
for a COUNT(*) = 0 that scans every referrer — a regression precisely
on heavily-deduplicated blobs, which is what GC walks most.
fmt, clippy --all-features --all-targets, and the 11 affected unit
tests all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 of docs/plan/derived-blobs.md. Makes "who references this blob
hash" an extension point instead of SQL hardcoded in two places
(dedup_gc's reap predicate and blobs_consistency's refcount recompute,
both naming storage.files and storage.chunk_manifests directly). Adding
a blob-owning table without teaching those two risks silent orphaning:
GC sees ref_count = 0 and reaps live content.
Behaviour is unchanged — this commit only introduces the port and the
two sources that reproduce today's SQL. Wiring follows.
Two levels, not one. add_reference bumps chunk_manifests.ref_count first
and only falls back to storage.blobs.ref_count, so a reference lands on
whichever counter its hash names and the two must be recomputed
separately. RefLevel is a parameter rather than a property of a source,
because storage.files legitimately contributes at both: a manifest-less
legacy row references a chunk, a CDC row references a Blob. The
NOT EXISTS guard on the chunk-level files term is load-bearing — for a
single-chunk file the whole-file hash equals its lone chunk's hash, so
without it the row is counted at both levels.
SQL fragments rather than a per-hash count. blobs_consistency recomputes
with one query per page, the expected count inlined as correlated
subqueries; asking each source for a count per hash would turn that into
sources x rows round-trips. So sources emit a fragment the registry sums
into the existing page query, and count_references exists only for the
on-demand path where the candidate set is already filtered to
ref_count = 0.
Fragments use their own aliases (cnt_f, cnt_m) rather than the sweeps'
outer-row aliases (b, m). A fragment reusing `m` would shadow the outer
alias in the manifest sweep and silently correlate against itself;
there is a test for it.
The SQL builders are free functions so the shape can be asserted without
constructing a pool — sqlx's connect_lazy still needs a Tokio context,
and the fragments are pure string assembly anyway.
9 unit tests. fmt, clippy --all-features --all-targets, build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
derived-blobs.md — consolidates several review rounds.
BLOCKER found while building the coverage matrix: the zero-ref manifest
sweep in dedup_gc (dedup_service.rs:2574) deletes a manifest when
`ref_count <= 0 OR NOT EXISTS (SELECT 1 FROM storage.files ...)`. That
OR hardcodes "storage.files is the only thing that can reference a
manifest", so a thumbnail manifest held only by content_derived_blobs
is deleted on the next GC run, its chunks dereferenced and the bytes
reaped. Promoted to prerequisite 0 and delivery step 2. It is also the
missing half of the unreconciled chunk_manifests.ref_count: that OR is
the hack that made the drift survivable.
Adds the 13-edge consistency coverage matrix (rows 1-6 and 13 covered,
7-11 not), and records that backend_consistency needs NO change — the
backend holds chunks, which neither new table references.
BlobReferenceSource correction: `ref_level()` was wrong because
FilesReferenceSource spans both levels (chunk for legacy manifest-less
rows, Blob for CDC rows). Replaced with
`ref_count_sql(level, alias) -> Option<String>`.
Also: migration of existing sidecar content; schema trim to the columns
nothing else owns (no size/format/codec/renderer; content_type kept as
non-key since it removes today's byte-sniffing); ext-{file_id}.jpg
corrected — the client generator ships and covers PDF, which has no
server-side rasteriser, so file_attached_blobs is required rather than
deferred; uploaded_by on the shares NOT NULL/no-FK convention; the
mermaid relation map; copy and version semantics with the
copy_file_satellites consolidation; the storage.files vs file_metadata
table-identity fix; DedupService -> BlobHandler recorded as decided.
NEW hidden-system.md — retires auth.users.image TEXT (inline base64
avatar, up to 512 KiB, already worked around with a narrow projection
after it was measured detoasting M avatars per group fan-out) in favour
of *_file_id pointers at ordinary storage.files rows in one shared
hidden system drive. Because storage.files is already a
BlobReferenceSource, a file pointer costs zero new reference sources
and zero new consistency edges. Records why the alternatives lose,
the drive's required properties (hidden at enumeration, trash off,
quota exempt, boot fail-fast), per-kind visibility in code, the
secrets exclusion rule, the avatar migration, and the future object
catalogue.
Docs only; no code or schema changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enrich OxiCloud to maximise the use of `dedup` Engine
2 cases will be covered:
- blobs issues from other blobs (thumbnail automatic generation from blob)
- by filename (ex: thumbnail uploaded from users)
A local cache will be added when blobs are remote (S3 or similar)
Edouard Vanbelle