The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with
Cache-Control: public, max-age=31536000, immutable. Replacing a file's
content preserves its id — file_upload_service rebuilds the entity with
parts.id and a new hash, then fires on_file_updated, which deletes and
regenerates the thumbnails — so the server produced a new thumbnail while
still advertising the old ETag. Because `immutable` tells a conforming
browser not to revalidate at all inside the freshness window, clients kept
rendering the previous image for up to a year, unfixably.
Keyed on the content hash the directive becomes honest: a thumbnail is a
pure function of (source bytes, size, format), so that triple identifies
the response. New content yields a new ETag.
The same change fixes the opposite direction. A copy, or any dedup twin,
had a different id and therefore a different ETag, so clients refetched
bytes they already held even though both are served from the same derived
blob. Now identical content agrees on an ETag and revalidates to 304
across files, users and copies.
Both thumbnail endpoints were affected: the REST handler and the
NextCloud preview handler.
Cost is one PK lookup ahead of the 304 decision, where the id-keyed
version needed none — paid for by no longer serving stale images. It is
partly recovered: both handlers already resolved the same hash further
down for the render path, and that second lookup is now gone, so the
cache-miss path is unchanged and only the 304 path pays. The resolved
hash is also handed to get_cached_thumbnail instead of None, saving the
service its own lookup.
No new disclosure: content_hash is already on FileDto and returned by
GET /api/files/{id}.
Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite
in place via WebDAV PUT, assert the ETag changed, assert a client holding
the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the
sharing direction: a copy answers with the SAME ETag and revalidates to
304, which is the one externally observable consequence of content-keying
and was not previously testable.
The byte-identity assertions were documented as proving that a copy
shares the original's content_derived_blobs row. They prove no such
thing: rendering is deterministic in the source bytes and the variant,
so a copy that re-rendered from scratch returns identical bytes. The
copy is in fact a moka hit — that cache is keyed on
(source_hash, size, format), which the copy shares — so it never
reaches the derived tier here at all.
Nor is there an assertion that would fix it. Duplication is impossible
by construction: the PK is (source_hash, kind, variant), a copy carries
the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING.
The schema enforces the property, so no runtime behaviour can violate
it and there is nothing to catch.
Same limitation narrows step 11: it proves the SOURCE content survived
GC, not the derived blob — a reaped derived blob is re-rendered
transparently from the live source.
What the file does prove is unchanged and is the part that was broken:
both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and
purging the original does not destroy the copies. No assertions changed.
Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was
written twice — the copy_file CTE and storage.copy_folder_tree — and had
already drifted: the tree path bumped storage.blobs only, missing
manifests, which was silent data loss on any multi-chunk file. Fixing it
meant writing the same logic a second time. Step 9 adds a file-keyed
satellite table, which would mean a third and fourth.
Two SQL functions:
storage.add_blob_references(TEXT[]) — the manifest-first reference
contract for SQL callers, returning hashes that matched no registry
row. Set-based so the tree path keeps its single-statement cost; a
per-row helper would have made a 10k-file copy 10k calls.
storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus
the blob reference. The body is the copy-semantics declaration: what
is absent (comments, favorites, content-keyed derived rows) is listed
with its reason, so the taxonomy is executable rather than documented
elsewhere and drifting.
Both copy paths now call it. The single-file path becomes a real
transaction, which also fixes the reference being best-effort: a failed
add_reference used to log a warning and leave a copy holding no
reference at all — the exact shape that gets its content reaped. It
cannot be a CTE arm, because data-modifying CTEs share one snapshot and
the function must read the row the INSERT just wrote.
Verified against a scratch PG with all migrations applied: multi-chunk
manifest 1→2, single-chunk alias bumped at manifest level only (the
NOT EXISTS guard), chunks behind a manifest untouched, dead properties
duplicated, length mismatch rejected, repeats counted.
tests/api/derived_blob_copy.hurl covers it end-to-end and answers the
question the copy raises: content_derived_blobs is NOT copied. A copy
carries the same blob_hash, so it resolves the same derived row — the
test asserts byte-identical thumbnails from both copy paths, then
deletes the original, runs GC, and requires both copies to still serve.
That last step only passes if the references are real.
Edouard Vanbelle