From 620800ba32bbc3e728f526dd6d2451f5a8e087e6 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sun, 30 Aug 2026 22:42:34 +0200 Subject: [PATCH] =?UTF-8?q?docs(plan):=20add=20step=2012=20=E2=80=94=20doc?= =?UTF-8?q?ument=20the=20two=20satellite=20tables?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tables exist and carry COMMENT ON text, but nothing explains the pair together, and the relationship is the part that matters: they hold the same kind of artifact under two different keys, and the keying difference IS the security boundary. Content keying shares one derivation across identical content, which is exactly what must not happen for user-supplied bytes — one user's uploaded preview would be served for every file whose content matches. A single table with a kind column would not prevent that; the split does. Today that reasoning lives only in scattered prose across this plan and two job doc-comments, so someone adding a third artifact type has nothing to read. Scopes the page: column-by-column structure including the parts that mislead (nullable blob_hash meaning a negative verdict, uploaded_by NOT NULL with no FK), worked examples that make the rule checkable, lifecycle and which consistency job covers which failure, and the NULL-handling trap that has already caused two bugs — comparison against NULL silently excludes negative rows, which is correct for refcounts and wrong for dangling checks. Lands in docs/architecture/ beside backend-storage.md, which documents the blob layer underneath. Co-Authored-By: Claude Opus 5 (1M context) --- docs/plan/derived-blobs.md | 59 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 705be5d3..3b9e63e3 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1567,6 +1567,65 @@ hardcoded SQL). New sources bolt on independently. land at any point; last is easiest, since every earlier slice would otherwise rebase across it. +12. **Document the two satellite tables** — `storage.content_derived_blobs` + and `storage.file_attached_blobs`, with worked examples. **Not + started.** + + The tables exist and each carries `COMMENT ON` text, but nothing + explains the pair *together*, and the thing that matters about them + is the relationship: they hold the same kind of artifact under two + different keys, and **the keying difference is the security + boundary**. Someone adding a third artifact type has to get that + right, and today the only place it is written down is prose + scattered through this plan and two job doc-comments. + + What the page needs: + + * **Column-by-column structure** of both tables, including the + parts that are easy to get wrong: `blob_hash` is nullable on + `content_derived_blobs` (a NULL is a NEGATIVE row — the + derivation was attempted and is known not to be worth storing), + paired with `content_type` by a CHECK; `uploaded_by` on + `file_attached_blobs` is `NOT NULL` with no FK, so provenance + survives a user deletion, and imported rows carry the nil-UUID + sentinel rather than a fabricated owner. + * **Why two tables and not one with a discriminator.** + `content_derived_blobs` is keyed by the BLAKE3 of the SOURCE, so + identical content shares one derivation — which is exactly what + must NOT happen for user-supplied bytes, or one user's uploaded + preview would be served for every file with matching content. + That is the poisoning vector; the split is what prevents it, and + a `kind` column on a single table would not. + * **Worked examples**, which is what turns the rule into something + checkable: + - the same image uploaded twice → one `content_derived_blobs` row, + two files, one thumbnail blob; + - a PDF with a client-uploaded preview → one `file_attached_blobs` + row per file, duplicated on copy, never shared; + - a screenshot WebP cannot shrink → a negative row with NULL + `blob_hash`, and what a reader is expected to conclude from it; + - the same content transcoded and thumbnailed → two rows, same + `source_hash`, different `kind`. + * **Lifecycle**: which rows hold a reference (positive + `blob_hash` bumps `chunk_manifests.ref_count`; a negative row + holds none), what reaps them (`purge_derived_blobs` on source + reap; the decrement trigger on `file_attached_blobs`), and which + consistency job covers which failure — the point being that a + satellite row whose SOURCE is gone breaks no refcount invariant, + which is why `satellites_consistency` had to exist at all. + * **The NULL-handling trap**, worth a paragraph because it has + already bitten twice: SQL comparison against NULL is NULL, so + `EXISTS (… WHERE blob_hash = h)` silently excludes negative rows. + That is correct for refcounts (a negative row holds none) and + WRONG for dangling checks (it reported every negative row as + `data_loss`). Anything joining on `blob_hash` must decide which + of the two it wants. + + Home: `docs/architecture/`, alongside `backend-storage.md`, which + already documents the blob layer these sit on top of. Operator- + facing rather than end-user, so schema and SQL are appropriate + here in a way they are not in `docs/guide/`. + Tracked separately, **not** part of this plan: the `storage.copy_folder_tree` refcount bug (see the copy section). It is a production data-loss bug on a path this plan doesn't otherwise