Edouard Vanbelle
9c63f9969a
Step 7 of docs/plan/derived-blobs.md, write path first — the plan is explicit that fixing it before the import means transcode_import only has to handle history, not a moving target. ImageTranscodeService now reads and writes storage.content_derived_blobs under kind='transcode', keyed by the BLAKE3 of the SOURCE content. The hash is threaded in from file_retrieval_service, which already holds it as dto.content_hash; hashing here would be a BLAKE3 over the whole file on every request. Callers without one (external mounts) keep the local cache untouched, which is what the service did before this tier existed. Negative verdicts become rows rather than zero-byte .skip files. A transcode that came out larger is deterministic in the content, so it is worth remembering; the row survives moka eviction, a restart, and the deletion of .transcoded/, none of which the marker does. Only that verdict is persisted — a timeout or a read error returns Err and is recorded nowhere, because a momentary failure written here would mark a perfectly transcodable image hopeless with nothing to retry it. Representation is a NULL blob_hash, per the plan: a sentinel hash would stop blob_hash naming a real Blob and every consumer would need to learn the exception. A CHECK keeps blob_hash and content_type NULL together — a type without bytes describes nothing, bytes without a type cannot be served. Two consumers had to be corrected for NULLs first, both of which would have broken on the first negative row ever written: * satellites_consistency reported them as derived_dangling_blob at data_loss severity. SQL comparison against NULL is NULL, so EXISTS was false and a row correctly pointing at nothing read as an artifact that had gone missing. * blob_reference_sources::list_referenced_blobs decodes blob_hash into String, so the first NULL would have failed the decode and taken the whole enumeration down. It would also have been wrong if it decoded — a negative row holds no reference, which is why the counting forms (WHERE blob_hash = <hash>) already exclude it for free. lookup_derived returns a three-way answer because Option collapses the two cases a caller deciding whether to spend a decode most needs apart: never attempted, versus attempted and known not worth it. DedupService is attached after construction via a OnceLock. DI builds the transcode service ~240 lines before DedupService exists, and the retrieval path that needs it is wired earlier still, so a constructor argument would mean reordering more than this is worth. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
64 lines
3.5 KiB
SQL
64 lines
3.5 KiB
SQL
-- Negative rows in `storage.content_derived_blobs`.
|
|
--
|
|
-- Some derivations can only be known to be useless by doing the whole
|
|
-- expensive job. `ImageTranscodeService` learns that WebP is not smaller
|
|
-- than the original by decoding and re-encoding the whole image; a
|
|
-- thumbnail renderer learns a source is undecodable, or over the
|
|
-- 50-megapixel ceiling, only by attempting it. Recomputing that verdict
|
|
-- on every request is the same cost as computing it the first time.
|
|
--
|
|
-- Today those verdicts live in RAM (moka's zero-weight empty-Bytes
|
|
-- convention) and, for transcodes, as zero-byte `.skip` files on local
|
|
-- disk. Both vanish: moka evicts, and the local disk is exactly what
|
|
-- this plan is deleting. So the verdict is stored here, next to the
|
|
-- positive derivations, as a row whose derived Blob is NULL.
|
|
--
|
|
-- ## Why NULL rather than a sentinel hash
|
|
--
|
|
-- A reserved hash was considered and rejected. It would stop
|
|
-- `blob_hash` naming a real Blob, and every consumer — the refcount
|
|
-- recompute in `manifests_consistency`, `dedup_gc`'s reap predicate,
|
|
-- `satellites_consistency`'s dangling check — would need to learn the
|
|
-- exception or silently mis-handle it. NULL is already the SQL way to
|
|
-- say "no Blob", and those consumers all join on `blob_hash`, so a NULL
|
|
-- drops out of the join instead of matching something fictional.
|
|
--
|
|
-- ## The CHECK matters
|
|
--
|
|
-- A row with a `blob_hash` but no `content_type` is unserveable; a row
|
|
-- with a `content_type` but no `blob_hash` claims a type for bytes that
|
|
-- do not exist. Both are bugs that would surface far from their cause,
|
|
-- so the pair moves together or not at all.
|
|
--
|
|
-- ## What must NOT become a negative row
|
|
--
|
|
-- Only failures that are DETERMINISTIC IN THE CONTENT. A transcode that
|
|
-- was not smaller, or a source that cannot be decoded, will fail the
|
|
-- same way forever — those are worth remembering. A generation timeout,
|
|
-- a closed semaphore, an I/O error reading the source Blob are
|
|
-- properties of the moment, not the content; persisting one marks a
|
|
-- perfectly good image as underivable permanently, and nothing ever
|
|
-- retries it. The asymmetry sets the default: a wrongly-cached
|
|
-- transient is silent and forever, a missing negative merely costs
|
|
-- repeated work. When in doubt, do not write the row.
|
|
|
|
ALTER TABLE storage.content_derived_blobs
|
|
ALTER COLUMN blob_hash DROP NOT NULL,
|
|
ALTER COLUMN content_type DROP NOT NULL;
|
|
|
|
ALTER TABLE storage.content_derived_blobs
|
|
DROP CONSTRAINT IF EXISTS content_derived_blobs_positive_or_negative;
|
|
|
|
ALTER TABLE storage.content_derived_blobs
|
|
ADD CONSTRAINT content_derived_blobs_positive_or_negative
|
|
CHECK (
|
|
(blob_hash IS NOT NULL AND content_type IS NOT NULL)
|
|
OR (blob_hash IS NULL AND content_type IS NULL)
|
|
);
|
|
|
|
COMMENT ON COLUMN storage.content_derived_blobs.blob_hash IS
|
|
'The derived Blob, or NULL for a NEGATIVE row: the derivation was attempted and is known not to be worth storing (transcode came out larger, source undecodable, source over the decode ceiling). Reference HOLDER when present — bumps chunk_manifests.ref_count via DedupService::add_reference. Only content-deterministic failures may be recorded as negatives; transient ones (timeout, semaphore, I/O) must not, or a momentary failure becomes permanent.';
|
|
|
|
COMMENT ON COLUMN storage.content_derived_blobs.content_type IS
|
|
'MIME type of the derived Blob. NULL exactly when blob_hash is NULL — the CHECK keeps the pair together, since a type without bytes describes nothing and bytes without a type cannot be served.';
|