Resolves the .skip-marker question. The table can say "here is the
artifact" but not "there is deliberately no artifact", and absence of a
row is ambiguous — it means both never-attempted and attempted-and-not-
worth-it, which destroys the only thing a negative cache holds.
Two live cases, not one. Transcodes drop a .skip marker when WebP is not
smaller. Thumbnails return empty Bytes for undecodable sources and ones
over MAX_DECODE_PIXELS — RAM-only today, so after moka evicts, a
60-megapixel upload has its full decode attempted again, forever. Same
gap, same table, so do both together.
Nullable rather than a sentinel hash: a sentinel stops blob_hash naming
a real blob and every future reader has to know the lie. Costs are one
`AND blob_hash IS NOT NULL` in ContentDerivedReferenceSource, the same
guard on the dangling-derived check, and dropping NOT NULL.
Permanent vs transient is the load-bearing split, and today's code
cannot tell them apart: generate_and_persist collapses every error into
empty Bytes, timeouts and semaphore closures included. Survivable while
the sentinel lives in moka, which evicts. Persist that same signal and a
thumbnail that timed out once under load is unrenderable forever. So the
renderer must return a typed outcome first, and only
permanently-unrenderable earns a row. The asymmetry sets the default —
a wrongly-cached transient is silent and permanent, a not-cached
permanent only costs repeated work — so unclassified errors are treated
as transient.
No TTL and no renderer-version term. Time is the wrong axis: the verdict
is deterministic in (content, encoder) and does not decay, so a TTL
re-attempts an OOMing decode on a schedule while still leaving staleness
for most of the window after a deploy. What removes the need for a
mechanism is that negative rows are disposable — they hold no data, so
a library upgrade invalidates them with a DELETE ... WHERE blob_hash IS
NULL in the same migration as the dependency bump. Recorded explicitly
so nobody later builds the expiry logic this replaces.
Investigated but deliberately not started — the remaining pieces span a
service, its DI wiring and an unresolved design question, and a
half-wired service is worse than none.
ImageTranscodeService does not write the derived tier at all today, the
same gap persist_rendered closed for thumbnails, and it must close
before transcode_import can converge or the import chases a growing
cache.
Three pieces, and the first is smaller than it looks. get_transcoded
takes file_id while the table is content-keyed, but the hash is already
in scope one frame up — file_retrieval_service::try_transcode is called
where dto.content_hash is live — so it is a parameter to thread, not a
lookup to invent. Explicitly NOT by hashing original_content on the
fly, which would be a BLAKE3 over the whole file per request.
Second, the service has no BlobHandler field, so this touches the
constructor and DI; ThumbnailService hit the same ordering problem and
solved it with a per-call parameter, which is the cheaper precedent.
Third, the .skip markers stay unresolved: a negative verdict has no
bytes, so it does not fit a table whose row points at a blob. Either
leave them local and recompute per instance, or model a sentinel. It is
the only genuinely open design question left in step 10.
Answering "when is transcode_import planned": it is the remaining third
of 10(b), but it must not be next, and both reasons were learned on the
thumbnail side rather than predicted.
Format has to move into `variant` first. Transcodes are inherently
multi-format and `variant` is keyed on size alone — the same gap found
in 10c that keeps JPEG thumbnails on the sidecar. Importing before that
means migrating into a schema that cannot hold the data without
collisions across formats. One migration unblocks both, which is the
argument for doing it before either.
And step 7 must precede the import. ImageTranscodeService writes only
its file-keyed disk cache today, so the import would run against a cache
that is still growing and never reach an empty tail — exactly the trap
persist_rendered had to close for thumbnails, where one of four render
paths recorded a row and the tail could never empty.
Order: format-in-variant → step 7 → transcode_import.
Also records why the .skip markers are still open: a cached negative
verdict has no bytes, so it does not fit a table whose point is pointing
at a blob.
Step 10c. The sidecar is local disk — invisible to other instances,
uncarried by a backend migration, uncovered by any consistency job.
Reading the derived tier first is what makes that state deletable. Not
the cost it appears to be: CachedBlobBackend gives the blob read a local
disk cache and moka absorbs the repeats above it.
Not a two-line swap, for two reasons.
A derived MISS must fall through to the sidecar; the old code
terminated the lookup with `?` because it was last. While the imports
drain, most content has a sidecar and no row — terminating there would
report "no thumbnail" for nearly all of it.
And the derived tier is WebP-only. store_derived_blob writes image/webp
and keys `variant` on the size alone, with no format term, so a JPEG
request matches the WebP row and would be served the wrong codec. The
old ordering hid this because the .jpg sidecar won first. So the lookup
is gated to WebP, JPEG clients stay on the sidecar — and the sidecar
cannot be deleted for them until `variant` encodes format. That is a new
prerequisite for step 10e, recorded in the plan rather than discovered
later.
Also corrects the plan: I had written that this flip removes the
derived-hash ETag hazard. It does not. A first render still creates the
row as a side effect of producing the body, whatever the read order, so
two consecutive reads still straddle its appearance. The real fix is
resolving the ETag after generation on the 200 path — a 304 only fires
when the client already holds a validator, which implies the row exists.
That is a handler restructure, not an ordering change.
Closing out step 10(a). The remaining `None` call sites in
persist_rendered looked like an open gap; they are not reachable in
production. Both `get_thumbnail` and the path variant of
`generate_all_sizes_background` are called only from the `ThumbnailPort`
impl, and nothing holds a `dyn ThumbnailPort` — which the existing note
in get_cached_thumbnail already recorded and a grep confirms. Live
renders go through get_thumbnail_from_blob and
generate_all_sizes_background_from_blob, both of which carry a
DedupService and dual-write.
So threading a DedupService through them would be work with no runtime
effect. Recorded at each call site instead, with the condition that
matters: gaining a real caller means taking a DedupService first, or the
gap persist_rendered exists to close reopens — sidecar-only output the
import can never see, so the tail never empties and the deletion gate
never opens.
Marks 10(a) done in the plan with that caveat stated rather than
implied.
The ETag is computed BEFORE the body. On a cache miss no
content_derived_blobs row exists, so thumbnail_content_id returned the
source-keyed form — then rendering created that row, and the next
request resolved to the derived hash instead. The validator changed as a
side effect of producing the body, making every first render
immediately stale.
Latent until aaf08532: before the consolidation, the on-demand render
path never wrote a derived row, so the flip had nothing to trigger it.
Fixing one gap exposed the other.
Caught by thumbnail_etag_content_keyed.hurl — two consecutive GETs of an
unchanged file stopped revalidating to 304.
The plan already said derived-hash keying must land WITH the read-order
flip and not before; I brought it forward anyway when the attachment
case forced the attached half. This is the evidence for the constraint,
so the plan now records the attempt and why it failed rather than
leaving the note as untested caution.
The attached lookup stays — it has no such window, since an upload
writes its row synchronously before any read can observe it, and it
fixes a real collision: a copy inherits the source hash, so an original
and a copy carrying different uploaded previews would otherwise share
one validator while serving different bytes.
The flip removes the hazard for the derived half too: once that tier is
authoritative it is populated before it is consulted, so no row can
appear between two reads.
Two revisions from working through step 10.
**Deletion moves into the import jobs, not a release.** Sidecars are
local disk, so a release cannot know whether every instance has drained
— gating on "an empty tail" asks an operator to coordinate a fact
nothing reports, and there is no telling when or whether they trigger
the jobs at all. Each job unlinking what it has imported makes every
instance drain itself. Constrained three ways: verify the derived blob
reads back before unlinking (a store that reported success but landed
unreadable would otherwise take the last copy), only after the
read-order flip (or the derived tier takes its first production traffic
by accident), and opt-in, since a migration that deletes by default is
surprising. Scheduled tick rather than boot trigger — idempotent and
resumable, so periodic is safe, while walking .thumbnails/ at startup
delays readiness for nothing.
**Found while checking the dual-write assumption: it does not hold.**
store_derived_blob has ONE call site; fs::write(&thumb_path, …) has
five. get_thumbnail, generate_and_persist and
generate_all_sizes_background all persist sidecar-only. That breaks the
migration's premise rather than being untidy — on-demand renders keep
producing un-migrated state after the import runs, so the tail never
empties and the deletion gate never opens. One persist_thumbnail owning
sidecar + derived + moka is therefore a prerequisite, and it makes "stop
writing sidecars" a later one-line change instead of four edits. Noted
that ThumbnailService holds no DedupService, so it must be threaded
through.
Also corrects a claim I put in thumb_derived_import's own docs:
transcoding is NOT a later step. ImageTranscodeService exists and caches
.transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it
must re-key file→content — legitimate only because a transcode is
derivable. Its .skip markers remain an open question.
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format),
which is one term short: a thumbnail is a function of those PLUS the
renderer. Change the encoder or a quality setting and identical inputs
produce different bytes under an unchanged ETag — the same staleness
class the commit fixed, one level down. It bites when an already-cached
thumbnail is re-rendered after a renderer change.
Keying on the derived blob's own hash removes the term entirely: the
ETag IS the hash of the bytes, so any output change invalidates by
construction. It is self-consistent for free, because store_derived_blob
is ON CONFLICT DO NOTHING — a re-render never displaces the stored row,
so the ETag always equals what the derived tier will serve. No renderer
version constant to remember to bump.
Records why it cannot land yet. The derived tier is read LAST by design,
so an ETag naming the derived hash would describe a tier the response
probably did not come from; sidecar and derived agree at creation but
diverge if a sidecar is re-rendered while the derived row stays pinned
by DO NOTHING. An ETag that lies about the body is worse than one that
is merely coarse. Also the tier is WebP-only (variant is the size, with
no format term) and empty for anything predating this work until
derived_import backfills.
So it lands at step 10 with the flip, keeping today's form as the
fallback for ungenerated variants and formats the tier does not hold.
The LEFT JOIN already planned for the read path returns the derived
hash in the same query, so it costs no extra round-trip.
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enrich OxiCloud to maximise the use of `dedup` Engine
2 cases will be covered:
- blobs issues from other blobs (thumbnail automatic generation from blob)
- by filename (ex: thumbnail uploaded from users)
A local cache will be added when blobs are remote (S3 or similar)
Prevent services accessing directly to localstorage and prefer using an astraction layer
to expose full blob. The abstraction layer (dedup services) will cover backend storage
election (local, s3, ...), encryption, etc
This change permit audio_metadata_service, media_metadaa_service, face_indexing_service to handle
blobs without worring of the backend.
note: prefered way to handle blob is the streamed way. Some services may not have this possibility
Edouard Vanbelle