Edouard Vanbelle
a7938344dd
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
3.6 KiB
SQL
70 lines
3.6 KiB
SQL
-- Derived content as blobs — tier-2 refactor, step 5.
|
|
-- See `docs/plan/derived-blobs.md`.
|
|
--
|
|
-- Maps a source Blob to the artifacts derived FROM it: thumbnails today,
|
|
-- transcodes next. Both the mapping key and the value are BLAKE3 hashes,
|
|
-- but they mean different things:
|
|
--
|
|
-- * `source_hash` — the Blob the artifact was derived from. A
|
|
-- *dependent* reference: it keeps nothing alive (the file does), and
|
|
-- when that Blob dies these rows are deleted with it.
|
|
-- * `blob_hash` — the derived Blob itself. A reference *holder*: it
|
|
-- bumps `chunk_manifests.ref_count`, which is why
|
|
-- `ContentDerivedReferenceSource` must be registered before the first
|
|
-- row is written, or `dedup_gc` reaps the content on its next sweep.
|
|
--
|
|
-- KEYING — the rule this table exists to enforce:
|
|
--
|
|
-- Bytes that are a pure deterministic function of the source content
|
|
-- belong here, content-keyed, and dedupe across every file holding
|
|
-- that content. Bytes that are user-supplied or user-chosen do NOT:
|
|
-- they must be file-keyed, because content-keying them lets one user's
|
|
-- upload be served for another user's identical file. Client-uploaded
|
|
-- previews (PDF page 1, video poster frames) are the live example and
|
|
-- belong in a separate file-keyed table.
|
|
--
|
|
-- `variant` is opaque text. New axes go INSIDE it, never into new
|
|
-- columns: 'preview-avif' beside 'preview', '720p-av1' beside '720p'.
|
|
-- That is what keeps this table from growing a column per rendering
|
|
-- parameter.
|
|
--
|
|
-- No FK on either hash column, for the reason
|
|
-- `20260701000000_content_search_index.sql` already documents: a hash
|
|
-- resolves to either `storage.blobs` (legacy whole blob) or
|
|
-- `storage.chunk_manifests` (CDC file hash), so the reference cannot be
|
|
-- expressed as a single FK. Orphans are reclaimed by GC and reported by
|
|
-- the consistency jobs instead.
|
|
--
|
|
-- No `size` column: the bytes are content-addressed, so their length is
|
|
-- an immutable fact the blob layer already owns via `blob_hash`.
|
|
-- `content_type` IS stored — the thumbnail handler byte-sniffs every
|
|
-- response today, and this retires that.
|
|
|
|
CREATE TABLE IF NOT EXISTS storage.content_derived_blobs (
|
|
source_hash VARCHAR(64) NOT NULL,
|
|
kind TEXT NOT NULL CHECK (kind IN ('thumbnail', 'transcode')),
|
|
variant TEXT NOT NULL,
|
|
blob_hash VARCHAR(64) NOT NULL,
|
|
content_type TEXT NOT NULL,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
PRIMARY KEY (source_hash, kind, variant)
|
|
);
|
|
|
|
-- Reverse lookup: "what still references this derived Blob?" — used by
|
|
-- the manifest-level refcount recompute in `manifests_consistency` and by
|
|
-- `dedup_gc`'s reap predicate.
|
|
CREATE INDEX IF NOT EXISTS idx_content_derived_blobs_blob_hash
|
|
ON storage.content_derived_blobs (blob_hash);
|
|
|
|
COMMENT ON TABLE storage.content_derived_blobs IS
|
|
'Server-derived artifacts (thumbnails, transcodes) keyed by the BLAKE3 of their SOURCE content. Content-keyed on purpose: identical content shares one derivation. User-supplied bytes must NOT be stored here — see docs/plan/derived-blobs.md.';
|
|
|
|
COMMENT ON COLUMN storage.content_derived_blobs.source_hash IS
|
|
'The Blob this was derived from. Dependent reference — holds no ref_count; rows are deleted when the source Blob is reaped.';
|
|
|
|
COMMENT ON COLUMN storage.content_derived_blobs.blob_hash IS
|
|
'The derived Blob. Reference HOLDER — bumps chunk_manifests.ref_count via DedupService::add_reference.';
|
|
|
|
COMMENT ON COLUMN storage.content_derived_blobs.variant IS
|
|
'Opaque rendering discriminator (icon | preview | large | 720p...). New axes go inside this string, never into new columns.';
|