Two revisions from working through step 10.
**Deletion moves into the import jobs, not a release.** Sidecars are
local disk, so a release cannot know whether every instance has drained
— gating on "an empty tail" asks an operator to coordinate a fact
nothing reports, and there is no telling when or whether they trigger
the jobs at all. Each job unlinking what it has imported makes every
instance drain itself. Constrained three ways: verify the derived blob
reads back before unlinking (a store that reported success but landed
unreadable would otherwise take the last copy), only after the
read-order flip (or the derived tier takes its first production traffic
by accident), and opt-in, since a migration that deletes by default is
surprising. Scheduled tick rather than boot trigger — idempotent and
resumable, so periodic is safe, while walking .thumbnails/ at startup
delays readiness for nothing.
**Found while checking the dual-write assumption: it does not hold.**
store_derived_blob has ONE call site; fs::write(&thumb_path, …) has
five. get_thumbnail, generate_and_persist and
generate_all_sizes_background all persist sidecar-only. That breaks the
migration's premise rather than being untidy — on-demand renders keep
producing un-migrated state after the import runs, so the tail never
empties and the deletion gate never opens. One persist_thumbnail owning
sidecar + derived + moka is therefore a prerequisite, and it makes "stop
writing sidecars" a later one-line change instead of four edits. Noted
that ThumbnailService holds no DedupService, so it must be threaded
through.
Also corrects a claim I put in thumb_derived_import's own docs:
transcoding is NOT a later step. ImageTranscodeService exists and caches
.transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it
must re-key file→content — legitimate only because a transcode is
derivable. Its .skip markers remain an open question.
Nothing exercised these jobs. Their unit tests cover the directory walk
— which files each claims — but neither had ever executed a run.
The test environment always starts fresh, so there is no pre-migration
data to import. This creates it, and the reconstruction is EXACT rather
than an imitation: the on-disk layout did not change in this work. A
rendered thumbnail has always been written to {size}/{hash}.webp and an
uploaded preview to {size}/ext-{id}.jpg; the only new thing is the row.
So upload through the real API, then delete the row, and what remains on
disk is byte-for-byte what a pre-migration install has.
Deleting the row must also release the reference it held, or the
manufactured state would carry a reference no legacy install ever had
and storage_cleanup_check.sh would report a leak this script caused.
file_attached_blobs has an ON DELETE trigger for that;
content_derived_blobs does not — its Rust purge path releases explicitly
— so the strip decrements it directly.
Three assertions, in increasing order of what they catch:
1. Both rows come back, and the imported attached row carries the nil
uploader sentinel rather than a fabricated one.
2. A COPY inherits the imported preview. This is the user-visible
point and was impossible before the row existed: the ext- sidecar
is keyed by file_id, no copy path duplicates it, so the copy
silently fell back to a render.
3. Re-running imports nothing and changes no refcount. The likeliest
silent defect — store_attached_blob is ON CONFLICT DO UPDATE, so an
import that skipped its existence check would release and retake a
reference every run, invisible except as drift.
psql runs inside the compose container rather than depending on a host
binary, matching how spawn-db.sh probes readiness. Ordered before
storage_cleanup_check.sh, which deletes everything it needs.
Both imports run over ONE directory, where the two legacy shapes sit
side by side, so the property worth asserting spans them: together they
must claim every real sidecar exactly once, and neither may take the
other's. A job that drifted into the other's shape would content-key
user-supplied bytes — sharing one user's uploaded preview onto every
file with identical content — and no per-job test in isolation would
notice.
So the fixture is shared. `legacy_tree` builds a directory holding a
content-keyed .webp pair, an ext- upload, and a stray README, and both
test modules walk it: derived claims exactly the two hashes in sorted
order, attached claims exactly the ext- file, the two sets are disjoint,
and between them they account for all three real sidecars.
`sidecar_names` became an associated function taking the root instead of
reading `self`, which is what makes this testable at all — the walk is
the half that decides which files a job claims, and it needed no pool to
verify. Sorting is asserted rather than assumed, since the cursor
resumes by skipping everything at or before it and a stable order is the
only thing that makes that correct.
A missing size directory is covered too: normal on a fresh install, and
it must yield no work rather than abort the walk.
Not covered here, and it needs a pooled fixture that does not exist: the
round trip itself — store the blob, write the row, and confirm a COPY
inherits the preview. That belongs in the API-level harness, where the
legacy state can be manufactured through the real write path and then
stripped.
Twin of thumb_derived_import, for the other sidecar shape:
{thumbnails_root}/{size}/ext-{file_id}.jpg, the previews a user
uploaded — notably the SPA's client-side PDF generator, which has no
server-side render path at all.
Until a row exists, a copy of the file LOSES the preview: the sidecar is
keyed by file_id, no copy path duplicates it, and the server silently
falls back to rendering from the source, or to nothing for a PDF. That
is the bug file_attached_blobs closed for new uploads; this closes it
for everything already on disk.
Separate job rather than an arm of the derived import, because the
keying differs and that difference is the security boundary. These bytes
are not derivable from the file's content, so content-keying them would
share one user's uploaded preview onto every file with identical
content. Each job's name filter rejects the other's shape, and both
directions are under test.
Idempotence needs more care here than in the derived twin.
store_attached_blob is ON CONFLICT DO UPDATE, so calling it for an
existing row releases the previous reference and takes a new one —
harmless once, but a job doing it every run would churn refcounts. The
row is therefore checked first and the store reached only on a genuine
insert.
uploaded_by is the nil sentinel: disk records no uploader, and inventing
one — the file's created_by, say — would fabricate provenance that could
later read as evidence an Editor replaced someone's preview. The column
is NOT NULL with no FK precisely so provenance survives, and a sentinel
says "unknown" honestly.
Orphaned sidecars (no storage.files row) are counted and reported, not
deleted. This job imports; it does not reclaim. Existence is checked
explicitly rather than letting the foreign key reject the insert, so an
orphan is counted as one instead of surfacing as an opaque constraint
error.
First half of step 10. Every server-rendered thumbnail written before
content_derived_blobs existed lives only as
{thumbnails_root}/{size}/{hash}.webp — local-disk state that another
instance cannot see, a backend migration does not carry, and no
consistency job covers. This walks those files into the blob store and
records the mapping, so the derived tier can become authoritative and
the sidecar can be deleted.
A registered JobRegistry tenant rather than a script: the volume is
unbounded, so it needs a cursor, resume, cooperative cancel and run
history, and an operator needs somewhere to watch it. Cursor is
{size_dir}/{filename} over a sorted walk, which totally orders the
traversal.
Idempotent by construction — each file is skipped when a row already
exists, and store_derived_blob is ON CONFLICT DO NOTHING with
release-on-conflict beneath it, so re-runs cannot inflate refcounts.
Re-running is the expected operator behaviour, since Phase 3 (deleting
the sidecars) is gated on a run reporting zero imported.
hash_from_sidecar_name deliberately rejects ext-{file_id}.jpg. Those
bytes are user-supplied and file-keyed; importing them here would
content-key them and share one user's uploaded preview onto every file
with identical content. They belong to thumb_attached_import. Both the
accept and the reject set are under test.
Unreadable files and store failures record a finding and continue: a
sidecar removed by a concurrent GC unlink between listing and read is
expected, not fatal, and the file is left in place for the next run.
Registered unconditionally rather than behind a flag — a migration
nobody can find is a migration nobody runs.
Thumbnails were served `public, max-age=31536000, immutable`. Two
problems, and the first is a security one.
`public` on a Permission::Read gated resource lets any shared cache — a
corporate proxy, a CDN — store one user's thumbnail and serve it to
another. `Vary: Accept` was no defence: it does not vary on
Authorization. Now `private`.
`immutable` was a promise this URL cannot keep. It is keyed by file id,
and its bytes change when a preview is uploaded, when content is
replaced, or when an attachment is removed. `immutable` tells a client
not to revalidate at all during the freshness lifetime, so with a
one-year max-age a browser that fetched once would never see a new
preview — which also made the content-keyed ETag unobservable in
practice. A correct validator is worthless if nothing asks. Now
`no-cache`, which still stores the body and only requires revalidation,
answered by the ETag with a body-less 304.
The hurl tests could not have caught this: hurl always sends the
request, so If-None-Match was exercised and passed while a browser
obeying `immutable` never got that far. Same "correct on the wire, wrong
in practice" shape as the bugs before it, so the test now asserts the
directives themselves rather than only the 304 behaviour.
One definition, shared by the REST and NextCloud endpoints, which are
gated identically and must not drift. /_app/immutable is untouched:
those are hash-named static assets, genuinely content-addressed and
public, where the directive is honest.
Cost is a conditional request per thumbnail per page load. Recovering it
needs a content-addressed URL — where `immutable` would be true — but
that puts the hash in the URL of an authorized resource, so it stays
`private` regardless, and it touches the SPA and the file DTO. Separate
change.
fe9c4f49 keyed the ETag on the SOURCE file's content hash. That is wrong
whenever the response comes from a satellite table, and for attachments
it is wrong in two ways.
Uploading a preview does not change the file's content, so a
source-keyed ETag does not change either — and with `immutable` set,
clients never revalidate and keep the previous render for up to a year.
The exact staleness fe9c4f49 set out to fix, re-entering through the
attachment path.
Worse: a copy inherits the source hash, so an original and a copy have
identical ETags. Give either one a different uploaded preview and they
serve different bytes under one validator, which a shared cache may hand
to either request. That is a collision, not just staleness.
thumbnail_content_id resolves the identity through the same tier
precedence the read path uses: an attached blob's own hash, else a
derived blob's own hash, else the source-keyed form. An ETag naming a
different tier than the one answering is worse than a coarse one, so the
two orders must not drift.
Derived-hash keying is strictly better than source-keying and never
worse. The sidecar and the derived row are written from the same bytes;
where they can diverge — a sidecar re-rendered while the derived row
stays pinned by ON CONFLICT DO NOTHING — source-keying is wrong too,
because the renderer is not part of that key. This is the step 10 change
arriving early, forced by the attachment case; the plan note stands for
the read-order flip itself.
Known gap: a legacy ext-{file_id}.jpg with no file_attached_blobs row
yet falls through to the source-keyed form. No worse than today, and it
resolves when the import backfills.
attached_thumbnail_copy.hurl now asserts ETags, which is why this went
unnoticed: it compared bytes only, and thumbnail_etag_content_keyed
covers content replacement rather than preview upload. A fresh GET
returned the right bytes throughout — the same "healthy locally, broken
for anyone caching" shape as the two bugs before it.
1. Per-file overrides must beat the content tier in RAM as well as on
disk. The content-keyed lookup ran first, so a thumbnail already
rendered from the file's content sat in RAM under content(blob_hash)
and shadowed a preview uploaded afterwards — permanently. Invisible
before the moka rekey, because both lived under one file-id key and
the upload simply overwrote the render. Order is now uniform:
per-file RAM, per-file disk (ext-), per-file DB, then content RAM,
blob-hash disk, derived blob.
2. store_attached_blob never wrote a row. Its RETURNING clause compared
the stored hash against EXCLUDED, and PostgreSQL only permits
EXCLUDED in the SET and WHERE of DO UPDATE — a runtime syntax error
on every call. The superseded hash now comes from a SELECT taken
before the upsert; losing that race leaves one stale reference, which
the manifest recompute reports, rather than anything being lost.
The second hid behind the first for a whole cycle, and behind
`ext-{file_id}.jpg`: the ORIGINAL kept serving its uploaded preview from
local disk, so the feature looked healthy. Only a copy, which has a
different file_id and therefore no ext- file, depends on the row — and
the row was never there. The handler's best-effort warn! completed the
disguise, so it is now error!: a failure there means copies silently
lose the preview, and nothing else signals it.
The disk check proves no BYTES are left. This proves no ROWS are, which
fails differently and worse: a stale storage.blobs row with nothing
behind it means a reference was never released, and dedup_gc will skip
it forever because its count never reaches zero. Silent, permanent, and
invisible to a check that only looks at the filesystem.
Zero is the right assertion, not "fewer than before". By this point the
suite has deleted its users, their drives and everything cascading
beneath, and the disk check has already insisted the blob store is
empty. A non-zero registry beside an empty disk is exactly the
divergence the consistency jobs report — caught here first because one
number is easier to read than a findings list.
Degrades to a warning if the endpoint is unavailable rather than
failing, so a build without the admin dedup surface still runs the rest.
The disk checks above prove nothing leaked. These prove the bookkeeping
behind them is honest: every refcount matches what the reference sources
hold, and no row points at bytes that are gone.
End of suite is the only place this is cheap. One database serves every
hurl file, so by here the counters have absorbed every upload, copy,
move, share, trash and purge the suite performed — across both copy
paths, the derived tier and the attached tier. Drift that no individual
test would notice, because each only inspects its own file, surfaces as a
mismatch.
Runs after the GC drain deliberately: mid-sweep state is legitimately
inconsistent — a manifest can sit at zero waiting for the next pass — so
checking earlier would report normal in-flight state as drift.
Zero findings is the assertion. These four tenants are read-only, so
anything they report is a real invariant violation rather than a repair
opportunity. A job missing from the build is skipped with a warning
instead of failing, so this does not break on a feature-gated build.
Unknown job names and unwrapped-vs-wrapped response shapes both degrade
to a visible warning rather than a silent pass: list_job_runs currently
returns a bare array, and the .runs/.items fallbacks exist so a future
wrapping does not quietly turn the whole check into a no-op.
Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else —
keyed by file id, on local disk. No copy path duplicates it and no other
instance can see it, so a copied file lost the preview its owner
uploaded. Silently: the server falls back to rendering one from the
source, or to 204 for a PDF, which has no render path at all. A
user-supplied preview is not derivable from the content, so once lost it
is gone.
The PUT now also records a storage.file_attached_blobs row, which
copy_file_satellites already duplicates, so both copy paths carry it.
Best-effort: the sidecar has already succeeded by then and the user can
see their thumbnail, so failing the request would report an error for an
operation that visibly worked.
Read path consults attachments ahead of every content-derived tier: an
uploaded preview is an explicit choice about THIS file and must beat
anything rendered from its content. Cached under the per-file key — a
content key would leak those bytes to every other file sharing the
content, which is the poisoning the file-keyed table exists to prevent.
store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin:
re-uploading a preview is a deliberate replacement, where a re-derived
thumbnail is the same bytes again. The superseded blob's reference is
released, or it would be pinned forever with nothing pointing at it.
Deletion goes through a trigger, not a hook. file_id is ON DELETE
CASCADE, and on_file_deleted fires AFTER delete_file — by then the
cascade has run and there is nothing left to enumerate. This matters
most for folder deletion, where PG cascades folders to files to
attachments and Rust never sees the rows at all. storage.decrement_blob_ref
keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused
verbatim rather than transcribed into a second trigger that can drift.
DELETE only: a replacement updates in place and is handled in Rust, so
adding UPDATE would double-decrement.
Extracted read_blob_to_bytes, shared by the attached and derived tiers —
the only difference between them is which table produced the hash.
tests/api/attached_thumbnail_copy.hurl guards it. The file is red and
the uploaded thumbnail is green, so a render could never produce the
uploaded bytes; the pre-upload render is captured first and required to
change, which stops three identical renders from satisfying the
byte-equality. Then both copy paths must serve the upload, and after the
original is purged and GC runs, both copies must still serve it — each
holds its own reference, because the rows are duplicated rather than
shared.
Step 9 of docs/plan/derived-blobs.md. content_derived_blobs holds bytes
that are a pure function of a file's content, so they are keyed by that
content and shared by every file holding it. This table holds the
opposite: bytes a user supplied or chose, which must never be shared
across files. The key is what enforces it.
That difference is a security boundary, not a modelling preference. A
content-keyed client preview would let user A upload a file plus a
preview that misrepresents it; when user B later uploads the same bytes,
dedup matches and B is served A's preview. Content-keying is only safe
when the server can derive the bytes — there is nothing to poison,
because the same input yields the same output for everyone.
Required now rather than deferred: the SPA already generates and PUTs
previews for PDFs, and there is no server-side regeneration path for
them, so the sidecar migration has nowhere else to put those bytes.
uploaded_by is NOT NULL with no foreign key, per the provenance
convention rather than the plan's sketch. A FK with ON DELETE SET NULL
discards the audit trail exactly when it matters, and without an
ON DELETE clause it would block deleting a user outright. Deleting the
uploader must not rewrite history.
FileAttachedReferenceSource is registered in built_in_registry before
anything writes to the table, so dedup_gc's reap predicate already knows
it exists — otherwise the first sweep after the first attachment would
delete it. Manifest level only, like the derived source: these blobs are
almost always single-chunk, so contributing at chunk level would
double-count against the aliased hash.
copy_file_satellites gains one arm: attachments are DUPLICATED, since
the key is file_id and the copy is a different file, with uploaded_by
carried over — the person who supplied the bytes did not change because
someone copied the file. Each duplicate takes its own reference, so the
bytes stay deduplicated while the mapping does not.
Both golden SQL tests updated: the new fragment lands inside the reap
predicate's NOT(...) group and as a summed term in the manifest
recompute. Verified on a scratch PG with every migration applied — the
attachment duplicates to 2 rows holding 2 references with provenance
intact, while the content-keyed thumbnail stays 1 row reachable from
both files.
fe9c4f49 made the ETag content-keyed, but the RAM tier was still keyed on
file_id, so the two disagreed about what identifies a thumbnail. Replacing
a file's content preserves its id, so the moka entry stayed reachable while
the ETag had already changed — and invalidation runs from the spawned task
in on_file_updated. A request landing in that window got the NEW ETag over
the OLD bytes, and because the response is immutable with a one-year
max-age, the client cached those stale bytes permanently. The bug fe9c4f49
set out to fix, arriving through a different door.
Keying on the hash removes the window rather than narrowing it: new content
is a different key, so the old entry cannot be hit. Correctness no longer
depends on the invalidation task winning a race against the next request.
This also aligns the RAM tier with what disk already did — sidecars have
always been written to get_thumbnail_path(blob_hash, ...). The tier that
had the bug was the one keyed differently from every other. Two further
consequences: N copies of one photo now share a single entry instead of
occupying N for identical bytes, and delete_thumbnails shrinks to the
external entries, which are the only genuinely per-file artifacts.
Video frames stay file-keyed under an `ext-{file_id}` id — they are
per-file by nature. The namespaces cannot collide: hashes are 64 hex
characters.
get_cached_thumbnail takes blob_hash as an Option, and a caller without one
now skips the RAM tier and falls through to disk rather than consulting a
file-id key. That is correct, not merely tolerable — a file-id key is the
stale entry this change exists to prevent. Both HTTP handlers resolve the
hash to build the ETag, so only internal callers that never had one are
affected.
GET /api/files/{id} is the download route — it returned the PNG bytes,
so the jsonpath capture failed on a UTF-8 decode. /{id}/metadata is the
EXIF endpoint and carries no FileDto either. Listing the folder gives
the DTO, and since the folder holds exactly this one file, count == 1
also proves the WebDAV PUT overwrote in place rather than creating a
second file beside it.
Also drops an unused bytes capture and records why the body is not
asserted after the overwrite: the moka tier is keyed on file_id and
invalidated from the spawned task in on_file_updated, so a request
landing first sees the previous bytes under the new ETag. Asserting on
bytes would be a race.
The upload omitted folder_id, which the handler needs to resolve the
file's owner — it answers 500, not a root upload. Every other upload in
the suite passes it; this was the only one that did not, which is why
nothing caught it earlier.
The folder also gives the WebDAV overwrite a deterministic path
(/webdav/hurl-etag-src/<name>) instead of depending on where a
folder-less upload would have landed. Teardown now removes it and
purges it from trash, keeping the shared database clean for the files
that run after.
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format),
which is one term short: a thumbnail is a function of those PLUS the
renderer. Change the encoder or a quality setting and identical inputs
produce different bytes under an unchanged ETag — the same staleness
class the commit fixed, one level down. It bites when an already-cached
thumbnail is re-rendered after a renderer change.
Keying on the derived blob's own hash removes the term entirely: the
ETag IS the hash of the bytes, so any output change invalidates by
construction. It is self-consistent for free, because store_derived_blob
is ON CONFLICT DO NOTHING — a re-render never displaces the stored row,
so the ETag always equals what the derived tier will serve. No renderer
version constant to remember to bump.
Records why it cannot land yet. The derived tier is read LAST by design,
so an ETag naming the derived hash would describe a tier the response
probably did not come from; sidecar and derived agree at creation but
diverge if a sidecar is re-rendered while the derived row stays pinned
by DO NOTHING. An ETag that lies about the body is worse than one that
is merely coarse. Also the tier is WebP-only (variant is the size, with
no format term) and empty for anything predating this work until
derived_import backfills.
So it lands at step 10 with the flip, keeping today's form as the
fallback for ungenerated variants and formats the tier does not hold.
The LEFT JOIN already planned for the read path returns the derived
hash in the same query, so it costs no extra round-trip.
The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with
Cache-Control: public, max-age=31536000, immutable. Replacing a file's
content preserves its id — file_upload_service rebuilds the entity with
parts.id and a new hash, then fires on_file_updated, which deletes and
regenerates the thumbnails — so the server produced a new thumbnail while
still advertising the old ETag. Because `immutable` tells a conforming
browser not to revalidate at all inside the freshness window, clients kept
rendering the previous image for up to a year, unfixably.
Keyed on the content hash the directive becomes honest: a thumbnail is a
pure function of (source bytes, size, format), so that triple identifies
the response. New content yields a new ETag.
The same change fixes the opposite direction. A copy, or any dedup twin,
had a different id and therefore a different ETag, so clients refetched
bytes they already held even though both are served from the same derived
blob. Now identical content agrees on an ETag and revalidates to 304
across files, users and copies.
Both thumbnail endpoints were affected: the REST handler and the
NextCloud preview handler.
Cost is one PK lookup ahead of the 304 decision, where the id-keyed
version needed none — paid for by no longer serving stale images. It is
partly recovered: both handlers already resolved the same hash further
down for the render path, and that second lookup is now gone, so the
cache-miss path is unchanged and only the 304 path pays. The resolved
hash is also handed to get_cached_thumbnail instead of None, saving the
service its own lookup.
No new disclosure: content_hash is already on FileDto and returned by
GET /api/files/{id}.
Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite
in place via WebDAV PUT, assert the ETag changed, assert a client holding
the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the
sharing direction: a copy answers with the SAME ETag and revalidates to
304, which is the one externally observable consequence of content-keying
and was not previously testable.
The byte-identity assertions were documented as proving that a copy
shares the original's content_derived_blobs row. They prove no such
thing: rendering is deterministic in the source bytes and the variant,
so a copy that re-rendered from scratch returns identical bytes. The
copy is in fact a moka hit — that cache is keyed on
(source_hash, size, format), which the copy shares — so it never
reaches the derived tier here at all.
Nor is there an assertion that would fix it. Duplication is impossible
by construction: the PK is (source_hash, kind, variant), a copy carries
the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING.
The schema enforces the property, so no runtime behaviour can violate
it and there is nothing to catch.
Same limitation narrows step 11: it proves the SOURCE content survived
GC, not the derived blob — a reaped derived blob is re-rendered
transparently from the live source.
What the file does prove is unchanged and is the part that was broken:
both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and
purging the original does not destroy the copies. No assertions changed.
Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was
written twice — the copy_file CTE and storage.copy_folder_tree — and had
already drifted: the tree path bumped storage.blobs only, missing
manifests, which was silent data loss on any multi-chunk file. Fixing it
meant writing the same logic a second time. Step 9 adds a file-keyed
satellite table, which would mean a third and fourth.
Two SQL functions:
storage.add_blob_references(TEXT[]) — the manifest-first reference
contract for SQL callers, returning hashes that matched no registry
row. Set-based so the tree path keeps its single-statement cost; a
per-row helper would have made a 10k-file copy 10k calls.
storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus
the blob reference. The body is the copy-semantics declaration: what
is absent (comments, favorites, content-keyed derived rows) is listed
with its reason, so the taxonomy is executable rather than documented
elsewhere and drifting.
Both copy paths now call it. The single-file path becomes a real
transaction, which also fixes the reference being best-effort: a failed
add_reference used to log a warning and leave a copy holding no
reference at all — the exact shape that gets its content reaped. It
cannot be a CTE arm, because data-modifying CTEs share one snapshot and
the function must read the row the INSERT just wrote.
Verified against a scratch PG with all migrations applied: multi-chunk
manifest 1→2, single-chunk alias bumped at manifest level only (the
NOT EXISTS guard), chunks behind a manifest untouched, dead properties
duplicated, length mismatch rejected, repeats counted.
tests/api/derived_blob_copy.hurl covers it end-to-end and answers the
question the copy raises: content_derived_blobs is NOT copied. A copy
carries the same blob_hash, so it resolves the same derived row — the
test asserts byte-identical thumbnails from both copy paths, then
deletes the original, runs GC, and requires both copies to still serve.
That last step only passes if the references are real.
5343fdda switched S3 blob enumeration from an opaque continuation token
to a hash cursor (StartAfter), per the port contract. Its fallback for a
page containing no canonical blob was wrong: it stored the full key
(`0a/junk.tmp`), stripped it to a basename (`junk.tmp`), and the next
call fed that to `object_key()` — producing `ju/junk.tmp.blob`. Wrong
shard and a doubled extension, so the resume jumped to an arbitrary
position: skipped objects, or backwards into a loop.
The cursor can only ever be a real hash, because `object_key()` is
applied to it. So instead of synthesising one, keep listing internally
until the page holds at least one blob or the bucket is exhausted. The
continuation token is used only inside the call and never escapes.
Two pathological cases cannot produce a cursor at all — `is_truncated`
with no token (protocol violation), and a run of foreign keys long
enough to buffer the bucket. Both now fail loudly. A visible job failure
beats a sweep reporting "no missing blobs" having read a fraction of
them.
Extract `hash_from_object_key` as the paired inverse of `object_key`,
with the round-trip and the rejection set under test. It also now
requires the shard to match the hash's own prefix, which the inline
filter did not check.
Precondition for the merge-join in backend_consistency (step 6 /
option A of docs/plan/derived-blobs.md), landed separately because it
is independently useful and carries the risk.
Two contract changes on BlobStorageBackend::list_blob_hashes:
1. Entries MUST be in ascending hash order. Every shipped backend
already did this — local sorts within each shard and walks 00..ff,
and since the shard IS the hash prefix that is globally sorted; S3
and Azure list lexicographically by key and blobs/<xx>/<hash> sorts
identically to <hash>. It was accidental, and a future backend
enumerating in any other order would have silently made the
merge-join emit bogus blob_missing_from_backend findings at
data_loss severity.
2. The cursor is the last hash returned, not an opaque backend token.
This is what lets a caller resume from a checkpoint it already
holds — the merge-join keeps one cursor for both the DB walk and
the backend walk instead of a compound one, which in turn means
blobs_consistency's existing cursor format survives and no paused
run is stranded.
Local already derived its position from a hash; it now emits the bare
hash instead of "<shard>/<hash>", and still accepts both legacy forms
so a run paused across this deploy resumes. The bare-shard form works
through the same path unchanged, since "3f" sorts before every 64-char
hash beginning "3f".
S3 moves from continuation_token to StartAfter, which supports this
natively. One non-obvious case handled: a page can contain only
non-canonical keys (.tmp spool files, .corrupt sidecars), which are
filtered into `unknowns`, leaving `blobs` empty — a naive
blobs.last() would return no cursor and silently end enumeration while
is_truncated said otherwise, making an audit job under-report. It now
falls back to the last key seen; StartAfter is a string comparison, so
a non-hash resume point is fine. "Cursor is a hash" constrains what
callers may synthesise, not what backends may return.
Azure is unaffected — it does not implement list_blob_hashes (TODO,
inherits the NotSupported default).
Adds the first test for enumeration at all: ordering across shards with
deliberately out-of-order inserts, complete paged traversal, and
resume from a caller-synthesised cursor.
NOT verified against real S3 — no bucket available here. The local path
is covered by the new test; the StartAfter change is reasoned from the
API contract and needs exercising against a real bucket before it is
relied on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One dedup_gc pass cannot fully drain now that thumbnails are derived
blobs. Reaping a source releases the references its derived artifacts
hold (each content_derived_blobs row pins a manifest), and those
releases happen mid-sweep — the derived chunks are stamped orphaned as
the pass is already walking past them, because
remove_manifest_reference deliberately does not unlink, to avoid racing
a concurrent upload re-referencing the same chunk. They are collectible
only on the NEXT sweep, which is why the check saw 15 leftover blobs.
Loops until a pass reclaims nothing rather than hardcoding two. Two is
correct only while the derivation graph is one level deep — a thumbnail
is derived from a file, nothing is derived from a thumbnail. That is a
property of the data, not an invariant the code enforces, so a fixed
count would silently under-drain the day transcodes-of-thumbnails or
E2E-wrapped derivatives exist, and the failure would surface as a
confusing leftover-file assertion rather than the design change it is.
Bounded at 3 with a warning if it does not settle.
Sleeps between passes. The JobRegistry serialises runs of the same job,
so a back-to-back trigger risks rejection as already-running — which
returns 0 reaped and would exit the loop early, declaring success with
blobs still on disk. A false pass is worse than a slow one. It also
gives the previous pass's detached unlink tasks (spawned by
on_blob_deleted, awaited by nothing) time to land.
Deliberately NOT fixed in production code: derived chunks land inside
the 1-hour orphan grace, so a second immediate sweep would collect
nothing there and the next scheduled run picks them up. A fixpoint loop
in garbage_collect would be dead code outside force=true, which is only
this test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught by the api-test storage check: 15 blob files left on disk after a
full cleanup. Since 3736b577 thumbnails are stored as derived blobs, and
each content_derived_blobs row holds a manifest reference — but nothing
ever deleted those rows, so the reference outlived the source and GC
could never reclaim the bytes. The plan specifies this cascade; I
implemented the write and read paths and missed it.
Adds `purge_derived_blobs`, the delete counterpart of
`store_derived_blob`: deletes every row derived from a source hash and
releases the reference each held. It lives on DedupService alongside its
store/find siblings because ThumbnailService cannot hold a DedupService —
it implements BlobLifecycleHook, and holding one would close the
DedupService -> BlobLifecycleService -> hook -> DedupService cycle the
existing comment warns about.
All five reap sites now go through `reap_blob`, which purges then fires
the lifecycle hooks, so no path can drop a blob without first releasing
what was derived from it. Previously each site called fire_blob_hooks
directly, which only cleaned the sidecar files ThumbnailService owns.
`reap_blob` is boxed because it is mutually recursive with
`remove_reference`: releasing a thumbnail's reference can reap the
thumbnail's own blob, which re-enters here. It terminates after one
level — nothing is derived from a thumbnail, so the inner purge finds no
rows. That bound is a property of the data, not an invariant the code
enforces, so it is stated at the definition.
fmt, clippy --all-features --all-targets, unit tests clean. The api-test
storage check is the real verdict — it is what found this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, read path — Option 2 of the two shapes discussed: the derived
blob is consulted LAST, after the sidecar, not first.
Read order is now
moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob
For every thumbnail already on disk the new branch is never reached, so
the database stays off the hot path and a fault in it cannot break a
working gallery. It answers only what disk cannot: a thumbnail rendered
by another instance, or a box whose sidecar was never populated. Legacy
content keeps serving from disk until `derived_import` migrates it.
That inverts the plan's stated order deliberately. Derived-blob-first is
right for the END state, because it is what lets the sidecar be deleted;
sidecar-first is right transitionally, because the risky reordering
should happen after the table has been seen serving real reads. The flip
belongs in the release that removes the sidecar, and the comment at the
branch says so.
The existing precedence is preserved and now documented: the file-keyed
client upload (ext-) is checked BEFORE the content-keyed server render.
That ordering is a security property, not a preference — content-keyed
artifacts are shared across every file with that content, so checking
the file-keyed one first is what keeps one user's uploaded preview from
ever being served for another user's identical file.
Shape notes:
* `find_derived_blob` lands on DedupPort/DedupService as the read
counterpart of `store_derived_blob`, so ThumbnailService needs no pool
field — and therefore ThumbnailService::new, DI and three tests are
untouched.
* It carries `content_type`, which is what will retire the byte-sniffing
in the handlers once reads are table-primary.
* The parameter is `Option<&DedupService>`, concrete rather than
`&dyn DedupPort`: DedupPort uses native `async fn` and so is not
dyn-compatible, and ThumbnailPort is never used as a trait object
(checked) — both handlers hold the concrete Arc. `None` means
sidecar-only, which is exactly today's behaviour and what the abstract
port impl passes.
fmt, clippy --all-features --all-targets, 35 unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.
That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.
DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:
* writes the bytes through the normal CDC path, so derived blobs
inherit the backend, encryption, migration and key rotation that
source content already gets;
* records (source_hash, kind, variant) -> blob_hash;
* releases the reference store_from_stream took IF the mapping
already existed. Two instances racing to render the same thumbnail
must leave ref_count at 1, not 2 — otherwise every re-render
inflates it and pins the blob forever.
ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.
The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.
Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.
fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-13 runner tier is being phased out by GitHub — queues persistently
exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to
'cargo install --features bundled-assets' from source, Docker
--platform linux/amd64, or a Linux VM.
JS-based GH Actions (checkout, artifact steps, setup-node) can't run
inside Alpine on ARM64 — Node.js binary shipped by the actions
requires glibc, and the x64-Alpine workaround doesn't extend to
arm64. Cross-compile natively via 'rustup target add' + musl-tools
instead.
The 'broken pipe' error under set -euo pipefail was masking the real
answer. Capture status + content-type + first-char in one curl, dump
first 200 bytes of body on failure so we can see what the server
actually served.
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in
example.env and docs/config/env.md — closes the discoverability gap
where the env var was only visible in Rust docstrings.
Also lands docs/plan/bundled-binary.md — the design record referenced
from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs,
and the Dockerfile.
Adds the tag-triggered workflow that builds 4 musl-linux + macOS
tarballs and attaches them to the tag's GitHub Release. Ships a
matching install guide (docs/install/binary.md) with SHA256SUMS
verify, systemd unit, upgrade flow, and hardware notes. Adds
[package.metadata.binstall] so 'cargo binstall oxicloud' works
automatically once the first release lands.
Also re-enables incremental compilation in the dev profile — the
'modest single-crate savings' rationale from when the crate was small
has been outgrown; full rebuild ~10 min is now the dev-loop bottleneck.
execute once via SQL migration to fix old entries
next: will be catch by manifests_consistency job, if new are found it means new bug discovered
no auto repair to prevent hidding bugs, operator can still invoke repair while waiting a fix
Silent data loss on an ordinary UI folder copy.
The function bumped only storage.blobs:
UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
FROM (...) hc WHERE b.hash = hc.blob_hash;
but a CDC file's blob_hash names a MANIFEST, not a chunk. For any
multi-chunk file that predicate matches zero rows, so the copy took no
reference at all. Delete the original afterwards and remove_reference
walks the manifest to 0, dedup_gc reaps the manifest and every chunk
behind it, and the copy is unreadable.
Single-chunk files escaped by accident: their whole-file hash equals
their lone chunk's hash, so the UPDATE did match — bumping the wrong
counter, which surfaces as a manifest under-count plus a blob
over-count rather than as loss. That asymmetry is why the bug survived:
small files, which dominate most test corpora, look fine.
Reproduced through the UI on a 5 MiB / 18-chunk file:
chunk_manifests.ref_count stayed at 1 while two storage.files rows
referenced it, and manifests_consistency reported
manifest_refcount_mismatch with delta 1, reap_risk true.
The fix mirrors DedupService::add_reference — manifest first, blobs only
as fallback, with a NOT EXISTS guard so a single-chunk file is not
counted at both levels (which would turn the under-count into an
over-count). orphaned_at is cleared on the blobs branch, as
add_reference does when resurrecting a blob inside its GC grace window.
Only the reference-counting block changes; the rest of the function is
20260902000001 verbatim.
Existing drift is deliberately NOT repaired here — a schema migration
cannot know which counter is authoritative. manifests_consistency
reports it; repair belongs with the recovery framework.
NOT executed against a database: the test instance was down and the dev
instance is read-only by convention. A parse error would fail at boot,
before any data is touched. Verify by re-running the reproduction — a
folder copy of a >1 MiB file should now leave ref_count at 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reap statement is assembled from the registered reference sources,
so it cannot be grepped out of the source tree — and it DELETES
manifests. Hiding it behind a debug filter an operator has to know to
enable was the wrong default: if what GC considers "referenced" ever
changes, that has to be visible on the next boot without anyone going
looking for it.
Reported in testing: `RUST_LOG=info,oxicloud::dedup=debug` did not
surface it, while a global `RUST_LOG=debug` did — at the cost of an
unusably noisy boot. Rather than have operators carry a special filter
for a line describing a destructive statement, promote it.
The statement is whitespace-collapsed into a single `statement` field
so a multi-line query does not sprawl across the boot log, and the
registered `sources` are logged alongside it — that list is what
actually determines the predicate, so a change to it is the thing worth
noticing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 3 (prerequisite 2) of docs/plan/derived-blobs.md, and the last one
before the thumbnail slice.
There are two reference counters and only one was ever verified.
add_reference bumps chunk_manifests.ref_count first and only falls back
to storage.blobs.ref_count, so a reference lands on whichever counter
its hash names: chunk references feed storage.blobs and are reconciled
by blobs_consistency::refcount_mismatch, while Blob references — every
CDC file, and every derived artifact once those exist — feed
chunk_manifests.ref_count, which nothing reconciled.
That gap was survivable only because dedup_gc's reap predicate carried a
second clause ("no storage.files row references this manifest") that
quietly compensated for drift on the bulk-delete paths where ref_count
is never decremented. Generalising that clause to the reference registry
in 1c8ead49 — so thumbnails stop being reaped — removed the
compensation, which is precisely why the counter now needs checking
directly. The two changes have to ship together.
Adds manifests_consistency, a recoverable job reporting
manifest_refcount_mismatch (severity inconsistent). The finding carries
reap_risk so an operator can triage: an under-count means GC reaps a
manifest whose content is still reachable, taking its chunks with it,
while an over-count merely pins storage.
A separate job rather than a second phase of blobs_consistency: one
subject per job, as the other five consistency tenants do, and it avoids
changing the cursor format of an existing recoverable job — which would
strand any run paused across the deploy.
The page query is assembled from the same registry dedup_gc reaps from
(via DedupService::reference_registry), built once at construction, and
pinned by a golden test. Two invariants the test guards: the files term
carries no NOT EXISTS guard — that guard keeps CDC rows out of the
*chunk* level and here would count nothing — and chunk_hashes appears
nowhere, since a manifest citing its own chunks is not a referrer of
itself.
fmt, clippy --all-features --all-targets, and 3 new unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes step 1 of docs/plan/derived-blobs.md. The chunk-level
`actual_ref_count` recompute was two correlated subqueries written
inline; it now sums the registered reference sources instead, so
`blobs_consistency` and `dedup_gc` answer "what references this hash"
from one place. If they ever diverged the sweep would bless counts the
collector disagrees with — and the collector wins, destructively.
No behaviour change: the generated expression is the same legacy-files
term (guarded by NOT EXISTS) plus the same manifests-citing-this-chunk
term, and a golden test pins the whole statement byte-for-byte.
Built once at construction, like the reap statement, so the sweep runs
a fixed query per page rather than assembling SQL inside the loop. The
builder refuses an empty registry rather than emitting a query where
every blob looks unreferenced and the entire table reports
refcount_mismatch; there is a test.
DI now constructs one registry and hands the same instance to both
consumers — `DedupService::reference_registry()` is what
`BlobsConsistencyCheck` receives, so agreement is structural rather
than a convention someone has to maintain.
The long comment explaining the single-chunk double-count trap moved
from the query site to the builder's doc comment, where the NOT EXISTS
guard it describes actually lives.
fmt, clippy --all-features --all-targets and the 17 affected unit tests
all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edouard Vanbelle