Oxicloud/docs/plan/derived-blobs.md

# Plan — Derived content as blobs (tier-2 refactor)

**Status:** design captured 2026-08-02, revised 2026-08-16 — keying
rule, CDC reuse, backend-dispatch rule, the
`content_derived_blobs` / `file_attached_blobs` pair, copy/version
semantics, a consistency coverage matrix with **three** hard
prerequisites (one of them a `dedup_gc` predicate that would delete
the entire derived tier), migration of the existing sidecar content,
and a schema trim down to the columns that carry information nothing
else owns. Not implemented.

Follow-up to `fix/services-use-blob-abstraction` — that
PR normalised the **read-side** (services consume blobs through
`BlobStorageBackend` uniformly). This plan tackles the **write-side**:
services that today write derived artifacts (thumbnails, transcodes)
to a local sidecar directory and would benefit from writing them
through the backend abstraction instead.

## Context — the three-tier storage taxonomy

Today the codebase runs three implicit tiers with no explicit
separation:

| Tier | Purpose | Loss on reboot? | Where today |
|---|---|---|---|
| **1 — Temp** | Pure scratch, deletable at reboot | ✅ fine | `std::env::temp_dir()` (ad-hoc callers). Now unified under `OXICLOUD_TEMP_DIR` (`AppConfig::temp_dir`). |
| **2 — Persistent spool** | Caches; expensive but rebuildable | ⚠️ possible but painful | `<storage_path>/.thumbnails/`, `<storage_path>/.transcoded/`, `<storage_path>/.blob-cache/`, `<storage_path>/.search-index/`, `<storage_path>/.plugin-logs/` — all mixed into tier-3 storage today. |
| **3 — Persistent data** | Source of truth | ❌ never | `<storage_path>/.blobs/` (Local) OR S3/Azure bucket, via `BlobStorageBackend`. Already correctly configured via `OXICLOUD_STORAGE_ENTRIES`. |

**Today's misclassification**: tier-2 sidecars live under
`<storage_path>` — the same directory as tier-3 source-of-truth
data. Ops resizing / moving / backing up tier-3 accidentally moves
tier-2 caches with it. Loss of tier 2 is expensive (regenerate
thumbnails for every photo) but not data loss; conflating them
means backup policies can't distinguish "must preserve" from "can
rebuild".

## The relation map (after this refactor)

Solid arrows **hold a reference** (bump a `ref_count`); dashed arrows
are **dependents** — they must be cleaned up when their target dies but
they keep nothing alive.

```mermaid
flowchart TB
  subgraph RES["RESOURCE LAYER · keyed by UUID"]
    FILES["<b>storage.files</b><br/>id UUID PK<br/>blob_hash VARCHAR(64)<br/>name · folder_id · mime_type"]
    FAB["<b>storage.file_attached_blobs</b><br/>(file_id, kind, variant) PK<br/>blob_hash · uploaded_by<br/><i>user-supplied · never shared</i>"]
    FMD["<b>storage.file_metadata</b> (EXIF)<br/>file_id PK<br/><i>⚠ content-derived, file-keyed</i>"]
  end

  subgraph CON["CONTENT LAYER · keyed by BLAKE3 of source bytes"]
    CDB["<b>storage.content_derived_blobs</b><br/>(source_hash, kind, variant) PK<br/>blob_hash<br/><i>pure f(content) · dedupes</i>"]
    BET["<b>storage.blob_extracted_text</b><br/>blob_hash PK"]
    FACES["<b>faces.faces</b><br/>blob_hash"]
  end

  BLOB["<b>BLOB</b> — the content of a file<br/>BLAKE3 of plaintext<br/><b>storage.chunk_manifests</b><br/>file_hash PK · chunk_hashes[]<br/><b>ref_count</b>"]
  CHUNK["<b>CHUNK</b> — physical payload<br/>BLAKE3 of the fragment<br/><b>storage.blobs</b><br/>hash PK · <b>ref_count</b> · orphaned_at"]
  BACKEND[("<b>BlobStorageBackend</b><br/>Local .blobs/ · S3 · Azure<br/>+encryption +retry +cache")]

  FILES -->|"FK file_id · CASCADE"| FAB
  FILES -->|"FK file_id · CASCADE"| FMD
  FILES -->|"blob_hash"| BLOB
  FILES -.->|"legacy pre-CDC · no manifest"| CHUNK
  CDB -.->|"source_hash · dependent"| BLOB
  CDB -->|"blob_hash"| BLOB
  FAB -->|"blob_hash"| BLOB
  BET -.->|"dependent cache"| BLOB
  FACES -.->|"dependent cache"| BLOB
  BLOB -->|"chunk_hashes[] · 1..N ordered"| CHUNK
  CHUNK -->|bytes| BACKEND
```

Three things to read off it:

1. **`content_derived_blobs` touches the Blob layer twice with
   opposite meanings** — `source_hash` is a dependent (it keeps
   nothing alive; the file does), `blob_hash` is a reference holder.
   Conflating them is how you get either a leak or a premature reap.
2. **Every new solid arrow into the Blob layer feeds
   `chunk_manifests.ref_count`** — the counter nothing reconciles
   today. See the prerequisites below.
3. **The two new tables meet the rest of the graph only at the Blob
   layer.** `content_derived_blobs` has no edge to `storage.files` at
   all: it reaches a file only by sharing that file's `blob_hash`.
   That is exactly what makes it dedupe across files — and exactly why
   it must never hold user-chosen bytes.

## Multi-instance driver

Single-instance: tier-2-as-local-cache works fine. Rebuild after
reboot is annoying but bounded.

Multi-instance (2+ app servers behind a load balancer):

- Request for thumbnail `abc123.jpg` lands on instance A → generates
  it → stores locally at `.thumbnails/abc123.jpg`.
- Same-URL retry lands on instance B → cache miss → regenerates
  from source.
- Every derived asset gets recomputed N times (N = instance count)
  at worst.

Wasteful compute, wasteful storage, inconsistent latency. The
long-term fix is to put derived content on tier 3 (shared) with a
local read-through cache in front. Multi-instance isn't the near-
term target, but the design should leave the door open.

## Design decision — derived content IS a blob

The blob storage abstraction is already:

- Backend-agnostic (Local / S3 / Azure)
- Encrypted uniformly (`EncryptedBlobBackend` wrapper)
- Consistency-checked (`blobs_consistency`)
- Migratable (`backend_migration`)
- Rotatable (`backend_rotate`)
- Multi-instance-ready (S3/Azure natively; Local via network mount)

Reusing it for derived artifacts means no second abstraction to
build and maintain, and all the operational surface (audit,
migration, key rotation) applies to derived content by default.

### Keying — content-address only pure functions of the content

**The rule:** an artifact may be keyed by its source's content hash
**iff** it is a deterministic pure function of the source bytes.
Anything influenced by user choice must be keyed by the resource it
was attached to, never by content.

| Artifact | Function of | Content-keyable? |
|---|---|---|
| server thumbnail | `f(blob bytes, variant)` | ✅ any user uploading identical bytes derives identical output — nothing to poison |
| transcode | `f(blob bytes, target)` | ✅ |
| extracted text | `f(blob bytes)` | ✅ — `storage.blob_extracted_text` |
| face vectors | `f(blob bytes)` | ✅ — `faces.faces` |
| client-uploaded preview | `f(user's choice)` | ❌ **must be file-keyed** — `storage.file_attached_blobs`, see below |

This isn't a new pattern: `storage.blob_extracted_text` already
chose content-keying for the same reason, and the migration says so
(`migrations/20260701000000_content_search_index.sql:22-28`) —
"extraction is keyed by `blob_hash`, not by file: N copies of the
same PDF cost ONE extraction, and rename/move/copy never
re-extract." `faces.faces` is keyed on `blob_hash` too. Thumbnails
are the same class of artifact, and file-keying them would make
them the odd one out among three sibling features while costing:

- **the dedup fast path** — `ThumbnailRefreshHook::on_file_created`
  returns early when `!is_new_blob`, so 100 users uploading the same
  photo cost one render. File-keying means either N renders or a
  join back through `files.blob_hash` (content-keying
  through the back door, slower and with more code).
- **free copies and free versions** — `on_file_copied` is a no-op
  today precisely because the key is content, and future versioning
  inherits the same property. See the copy/version axes below.

For the derived side the hash is over the **produced** bytes, so:

- Two files with identical thumbnails (same variant of the same
  source → identical bytes → identical hash) share the physical
  blob. Dedup wins for free.
- Two variants of one source (256px vs 512px) produce different
  blobs. Also correct.

The variant spec lives in the referring DB row, not in the storage
key. Storage stays one keyspace; ownership stays per-service.

### Corollary — point at a file, never at a blob

Both tables in this plan exist because their content is *not* a file:
a thumbnail has no name, no folder and no place in a user's tree. When
a binary **can** be a file, make it one and point at it with a
`*_file_id` FK — `storage.files` is already a `BlobReferenceSource`,
already covered by every consistency edge, already GC-integrated, so a
file pointer costs **zero** new reference sources and zero new
consistency checks.

That is the rule that stops the next person adding a fourth
blob-referencing table. It is what `docs/plan/hidden-system.md`
applies to user avatars, backgrounds and signatures, and it extends to
owners that are not users at all —
`carddav.contacts.photo_file_id` would retire the inlined
`photo_url TEXT` on the same terms.

### Schema

```sql
CREATE TABLE storage.content_derived_blobs (
    source_hash  VARCHAR(64) NOT NULL,  -- source Blob (no FK — see below)
    kind         TEXT NOT NULL,         -- 'thumbnail' | 'transcode'
    variant      TEXT NOT NULL,         -- 'icon' | 'preview' | 'large' | '720p'
    blob_hash    VARCHAR(64) NOT NULL,  -- the DERIVED Blob
    content_type TEXT NOT NULL,         -- served directly; no byte-sniffing
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (source_hash, kind, variant)
);
CREATE INDEX ON storage.content_derived_blobs(blob_hash);
```

**`variant` is opaque text. New axes go inside it, never into new
columns.** This is the rule that keeps the table from growing, and it
disposes of three columns earlier drafts proposed:

- **No `format` column.** WebP vs JPEG looks like a second axis, but
  only the canonical rendering is persisted (below), so there is one
  row per variant. If a format migration ever happens — AVIF is the
  plausible one — it is `variant = 'preview-avif'` beside
  `'preview'`. Data change, not a PK migration.
- **No `codec` column.** Transcoding here is a *playability
  fallback*, not bandwidth optimisation: one widely-compatible
  rendition (H.264/AAC in MP4), no negotiation, nothing to
  distinguish. `<video>` probes the codec from the container, so the
  header only owes `video/mp4`. A second codec would be
  `variant = '720p-av1'`. (HLS/DASH manifests do need declared
  codecs, but that is a segmented architecture this table doesn't
  model and would need its own structure regardless.)
- **No `renderer` / spec-version column.** It would exist to
  invalidate rows when render parameters change (400 px → 512 px,
  q82 → q85) — but such a change is global, so invalidation is
  `DELETE … WHERE kind = 'thumbnail'` with or without it. The
  `blob_extracted_text.extractor` precedent does **not** transfer:
  that column exists because extraction caches *terminal negative
  results* per blob (`'failed'`, `'unsupported'`) that must be
  retried on an extractor bump. Thumbnails never persist a failure —
  the failure path stores a zero-weight moka sentinel that evicts
  immediately. No cached negatives, nothing version-dependent.

**Only the canonical rendering is persisted to tier 3.** WebP covers
~97% of clients; persisting the JPEG fallback too would add one
object per size per source — a 50% increase in derived object count
and in the `blobs_consistency` probe cost, to serve 3% of requests.
Re-encoding JPEG from an already-decoded 400 px WebP is
sub-millisecond, so the fallback is generated on demand and held in
the moka RAM tier only. Same principle as the "persist only `large`,
derive icon/preview" option below: **tier 3 stores the canonical
rendering; everything else is derived on demand.**

**No `size` column.** The bytes are content-addressed, so their
length is an immutable fact the blob layer already owns
(`chunk_manifests.total_size` / `storage.blobs.size`, reachable via
`blob_hash`). Copying it here would create a second source of truth
for something this table has no authority to assert. Every consumer
is covered without it: `Content-Length` comes from the bytes in hand,
storage reporting joins on `blob_hash`, thumbnails are never
range-served, and `read_blob_bytes` already takes its buffer hint
from the manifest row.

`content_type` is the one thing worth storing rather than
recomputing: the handler byte-sniffs every response today
(`mime_detect::thumbnail_content_type`), and this deletes that. It is
a per-row fact the producer knows for certain. It is *not* part of
the key — identity is (source, kind, variant).

One table with a `kind` discriminator rather than separate
`storage.thumbnails` / `storage.transcodes`: `count_references`,
`list_referenced_blobs`, the GC cascade and the
`backend_consistency` walk are byte-identical between the two, so
two tables means maintaining a duplicate of that SQL — which
`AGENTS.md § Code duplication` forbids. `kind` costs one column.

**No FK on `source_hash` or `blob_hash`**, for the reason the
search-index migration already documents: a file hash resolves to
either `storage.blobs` (legacy whole blob) or
`storage.chunk_manifests` (CDC file hash), so the reference can't be
expressed as a single FK. Orphans are reclaimed by GC instead.

**No `origin` column.** The 2026-08-02 draft had
`origin = 'server_derived' | 'client_provided'` so consistency-check
severity could differ. With client previews excluded from this table
(below), every row is server-derived and the severity is uniformly
"warning, regenerable" — the column carries no information. Re-add
it only if that changes.

**Deletion is app-layer.** `on_blob_deleted(source_hash)` does
`DELETE FROM storage.content_derived_blobs WHERE source_hash = $1 RETURNING blob_hash`
then `remove_reference()` per row — a one-for-one replacement of
today's `delete_blob_thumbnails` unlink loop, no new mechanism. A raw
SQL `ON DELETE CASCADE` would drop the mapping row without
decrementing the refcount, but *not* silently: once the reference
registry exists the refcount is derivable, so the drift surfaces as a
`refcount_mismatch` finding, and the interim state is an **over**-count
(blob retained longer than needed) never an under-count (live data
reaped). So a cascade is tolerable where it's ergonomic — see
`file_attached_blobs` below — provided the registry is in place to
reconcile it.

### The pair — `content_derived_blobs` and `file_attached_blobs`

Two tables, one keying difference, and that difference *is* the
security boundary (see the client-preview section):

```
storage.content_derived_blobs  (source_hash, kind, variant)
storage.file_attached_blobs    (file_id,     kind, variant)
```

**The name states the key**, because the key is the only thing that
differs and choosing the wrong one is a silent poisoning bug rather
than a compile error. An earlier draft called these
`derived_blobs` / `attached_blobs`; that pairing was rejected because
`derived` vs `attached` is the wrong axis of symmetry — a thumbnail is,
in plain English, "attached to" a file, so both words plausibly
describe either table and the names carry no signal about keying. The
`content_` / `file_` prefixes are non-overlapping and answer the only
question an implementor needs to ask.

```sql
CREATE TABLE storage.file_attached_blobs (
    file_id      UUID NOT NULL REFERENCES storage.files(id) ON DELETE CASCADE,
    kind         TEXT NOT NULL CHECK (kind IN ('preview', 'subtitle', 'cover_art')),
    variant      TEXT NOT NULL,         -- 'preview' | 'en' | 'cover'
    blob_hash    VARCHAR(64) NOT NULL,  -- content-addressed bytes (dedup preserved)
    content_type TEXT NOT NULL,
    uploaded_by  UUID NOT NULL REFERENCES auth.users(id),
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (file_id, kind, variant)
);
CREATE INDEX ON storage.file_attached_blobs(blob_hash);
```

Same trims as `content_derived_blobs` — no `size`, no `format`, no
`codec`. Two deliberate differences:

- **No `renderer`-style column even in principle.** These bytes
  aren't rendered, they're supplied; there is no spec to version.
  That falls straight out of the purity rule.
- **`uploaded_by` is kept.** It is the one column here that is
  genuinely new information rather than a copy of something the blob
  layer owns, and it is the only trace that an Editor on a shared
  file replaced the owner's preview — untraceable today.

**Routing rule — put this as a comment on both tables:**

> Bytes are a pure deterministic function of the file's content →
> `content_derived_blobs` (content-keyed, dedupes across files, regenerable).
> Bytes are user-supplied or user-chosen → `file_attached_blobs`
> (file-keyed, never shared across files, not regenerable).

Generic naming rather than `file_previews` because the family is
real, and each member would otherwise be a new table plus a new
`BlobReferenceSource` plus a new term in the consistency recompute.

### Negative verdicts — `blob_hash` must be nullable

*(Resolved 2026-08-27. Supersedes the "`.skip` markers are an open
question" note.)*

The table says "here is the artifact". It cannot say **"there is
deliberately no artifact"**, and absence of a row is ambiguous — it
means both *never attempted* and *attempted, not worth it*. Collapsing
those destroys the only information a negative cache exists to hold.

Two live cases, not one:

* **Transcode not beneficial.** `ImageTranscodeService` can only learn
  whether WebP is smaller by doing the full decode + encode. When it is
  not, it records a zero-byte `{file_id}.{ext}.skip` marker so the next
  GET does not repeat the work.
* **Thumbnail not renderable.** `generate_and_persist` returns empty
  `Bytes` — "moka's zero-weight negative-entry convention" — for sources
  over `MAX_DECODE_PIXELS` (50 MP) or that fail to decode. This one is
  **RAM-only**: after moka evicts, a 60-megapixel upload has its full
  decode attempted again, forever.

So this is not a transcode quirk. The rule generalises to every `kind`:

> **Any derivation whose failure is deterministic in the source content
> is worth memoising negatively.**

**The discriminator, or the table fills with noise.** Persist a negative
only when it is *both* expensive to compute *and* deterministic in the
content. `can_transcode(mime)` and "not an image" are cheap metadata
checks — recomputing is free and a row would be pure overhead. It is the
ones that cost a decode that earn a row.

**Permanent vs transient is the load-bearing split.** "Deterministic"
above means *a property of the content*, not merely *a failure that
happened*:

| Permanent — cache it | Transient — never cache it |
|---|---|
| decode failed (corrupt / unsupported) | generation timeout |
| over `MAX_DECODE_PIXELS` | decode semaphore closed |
| transcode result not smaller | blob read I/O error, OOM under load |

**Today's code cannot tell them apart, and that must be fixed before any
of this is persisted.** `generate_and_persist` collapses *every* error
into empty `Bytes` — timeouts and semaphore closures included. That is
survivable while the sentinel lives only in moka, which evicts; write
the same signal to the database and a thumbnail that timed out once
under load is marked unrenderable **forever**. So the renderer must
return a typed outcome — rendered / permanently-unrenderable / transient
failure — and only the middle one earns a row.

The asymmetry sets the default. A wrongly-cached transient is silent and
permanent; a not-cached permanent merely costs repeated work. **When in
doubt, do not cache** — treat unclassified errors as transient.

**Representation: nullable `blob_hash`.** A sentinel hash was considered
and rejected — it stops `blob_hash` naming a real blob, and every future
reader has to know the lie. NULL says what is true. Costs:

* `ContentDerivedReferenceSource` needs `AND blob_hash IS NOT NULL`; a
  row holding no blob holds no reference.
* The dangling-derived check (row 9 of the coverage matrix) needs the
  same guard, or every negative verdict reports as a broken row.
* The `NOT NULL` constraint is dropped.

**No TTL, and no renderer-version term either.** Both were considered.
Time is the wrong axis: the verdict is deterministic in
`(content, encoder)` and does not decay, so a TTL re-attempts an OOMing
decode on a schedule — reintroducing the exact waste the negative
exists to prevent — while still leaving staleness for most of the window
after a deploy.

What makes the mechanism unnecessary is that **negative rows are
disposable**: they hold no data, so discarding one costs only a
re-derivation. Upgrading the image library invalidates them with a line
in the same migration as the dependency bump —

```sql
DELETE FROM storage.content_derived_blobs WHERE blob_hash IS NULL;
```

— which beats a version term that must be remembered and leaves dead
rows behind when bumped. Write this down, or someone later builds the
expiry logic this paragraph exists to prevent.

Same shape, outside this table and not solved here:
`blob_extracted_text` (no extractable text) and `faces.faces` (no faces
detected) are both deterministic negatives currently indistinguishable
from "never processed".
With `kind` it's a one-line `ALTER … CHECK`:

| Kind | Why it lands here |
|---|---|
| `preview` | client-uploaded thumbnail (today's case) |
| `e2e_thumbnail` | strongest future case — in an E2E drive the server *cannot* derive thumbnails, so the client must upload them. "Vault" is already reserved as a future E2E drive kind (`project_drive_naming_and_vault_reservation`) |
| `subtitle` | user-supplied caption tracks, one per language (`variant = 'en'`) |
| `cover_art` | user override of embedded/derived art; user-chosen video poster frame |
| `metadata_sidecar` | XMP / `.nfo` uploaded alongside a photo |
| `signature` | detached signature over the file, per signer |

Avoid `file_sidecars` as a name despite it being the natural
media-world term: this codebase already uses "sidecar" for the
tier-2 local directories (see the sidecar section below), and
overloading it would undo that vocabulary.

Note the `(…, kind, variant)` shape is identical across both tables,
so one generic reference-source implementation parameterised by
table + column covers both — the same one-implementation property
the rest of this plan is built on.

**Naming and comments are advisory; the boundary needs a test.** The
one guard that actually stops a future implementor is an integration
test asserting both halves at once: two users uploading byte-identical
content resolve to the **same** `content_derived_blobs` rows (dedup
works), and a client preview uploaded by one of them produces **no**
row reachable by the other (no cross-file sharing). That test fails
loudly the moment someone rekeys either table. Ship it with step 4,
not after.

Structural help, in descending order of reliability: the test; the
column types (`file_id UUID` vs `source_hash VARCHAR(64)`, so a row
cannot be copied between tables); the absence of any `file_id` column
on `content_derived_blobs`; the table comments.

### Write path — reuse `store_from_stream`, don't special-case CDC

Derived blobs go through `DedupService::store_from_stream()`
**unchanged**. An earlier draft of this plan proposed a dedicated
single-chunk write path to avoid the manifest row; that was
optimising the wrong thing. What it costs to reuse the standard
path:

- **+1 `chunk_manifests` row per derived blob.** `CDC_MIN_CHUNK` is
  65_536, and WebP q82 thumbnails land at roughly 3–8 KB (icon),
  15–30 KB (preview), 40–90 KB (large) — so icon and preview are
  always below the minimum chunk size and emit exactly one chunk;
  `large` occasionally splits into two.
- **One manifest lookup per read**, served from RAM by
  `manifest_cached` — not a per-read query.
- **A CDC pass over ~12 KB** — below min-chunk, so a single pass
  with no boundary search. Negligible.

What it buys: zero new write path, zero new read path, and
ref-counting, GC, `add_reference` / `remove_reference` and both
consistency jobs all work on derived blobs with no changes, because
they are already manifest-aware. One `store_from_stream` call == one
reference == one `content_derived_blobs` row, symmetric on delete.

Two gotchas that come with the reuse:

1. `store_from_stream` fires `fire_blob_creation_hooks`. The only
   non-dispatcher `BlobLifecycleHook` implementor today is
   `ThumbnailService`, whose `on_blob_created` is a no-op — so no
   spurious work now. But creating a thumbnail now fires
   blob-creation hooks, so any future hook (search indexing, face
   detection) must not treat every new blob as user content. Add a
   `kind`/content-type guard to the hook contract before the second
   implementor lands.
2. GC of a *derived* blob fires `on_blob_deleted(derived_hash)`,
   which looks for derived-of-derived rows, finds none and stops.
   One level deep, terminates — but that's incidental rather than
   designed. Comment it at the recursion point.

### No backend-type branching in the service

`src/AGENTS.md` already forbids the shape this refactor must avoid:

> - **Never hand-craft blob paths.** No `blob_root: PathBuf` fields […]
> - **Persistent state = backend**, not `<storage_path>/*` sidecars.

`thumbnail_service` is cited there as a read-side reference impl,
and the read side is compliant. The write side is the violation:
`thumbnails_root: PathBuf` is exactly the banned field, and
`get_thumbnail_path()` is the hand-crafted path.

There must be **no `if backend is local { .thumbnails/… } else { blob }`
anywhere in the service.** `ThumbnailService` holds
`Arc<DedupService>`, reads and writes through it, and never learns
which backend it is sitting on. The local-vs-remote difference is
expressed once, as decorator composition in `common/di.rs`:

```rust
if self.config.storage.cache.enabled && active_backend_kind != StorageBackendType::Local {
    blob_backend = Arc::new(CachedBlobBackend::new(blob_backend, &cfg));
}
```

Local deployments write derived blobs into `<storage>/.blobs/` via
`LocalBlobBackend` with no cache decorator (a cache would be a
byte-identical second copy on the same disk). Remote deployments get
the cache. Same service code both ways — and it is the same branch
that already governs source blobs, not a new one.

What this deletes from `thumbnail_service.rs`:

- `thumbnails_root` field, `get_thumbnail_path()`,
  `ThumbnailSize::dir_name()` (becomes `variant()`, feeding the DB
  column)
- `initialize()`'s `create_dir_all` loop
- every `fs::read` / `fs::write` / `fs::metadata` / `remove_file`
- the three `all_exist` stat loops → one indexed query each
- `delete_blob_thumbnails`'s unlink loop and the duplicate of it
  inside `on_blob_deleted`

Net deletion, which is the main argument for this shape.

Two adjacent cleanups in the same file: `stream_blob_to_temp` uses
`self.thumbnails_root` as its temp directory and must move to
`AppConfig::temp_dir` per the `OXICLOUD_TEMP_DIR` rule; and
`store_external_thumbnail`'s `ext-{file_id}.jpg` write is the last
hand-crafted path once the rest is converted.

### Client-uploaded thumbnails — file-keyed, and NOT in `content_derived_blobs`

Some clients (NC desktop, mobile apps) upload their own encoded
previews alongside the file. These are **not derivable** — losing
them means asking the client to regenerate, which may not be
possible.

They are also **not a function of the content**, and that makes
content-keying a cross-user poisoning vector:

1. User A uploads file X plus a preview that does not depict X.
   There is no validation that can catch this — verifying a preview
   faithfully represents its source means re-deriving and comparing,
   at which point accepting the client's upload is pointless.
2. User B uploads the same file X. Dedup matches on `source_hash`.
3. B is served A's preview.

So client previews **stay file-keyed, in `storage.file_attached_blobs`,
and out of `storage.content_derived_blobs`.** This is a precondition for
`content_derived_blobs` having no `source_file_id` column, not an independent
choice — the two decisions must land together.

Worked example. User A uploads `image.png` (file id `7f3e…9c`,
content hash `a1b2c3…`), then `PUT`s their own preview:

```
storage.content_derived_blobs      -- server-derived, content-keyed
 source_hash | kind      | variant | blob_hash | content_type
 a1b2c3…     | thumbnail | icon    | 9a8b…     | image/webp
 a1b2c3…     | thumbnail | preview | d4e5f6…   | image/webp
 a1b2c3…     | thumbnail | large   | c7d8…     | image/webp

storage.file_attached_blobs        -- client-supplied, file-keyed
 file_id  | kind    | variant | blob_hash | content_type | uploaded_by
 7f3e…9c  | preview | preview | e1f2…     | image/jpeg   | A
```

Both sets of bytes travel the same dispatch
(`store_from_stream` → blob → backend → encryption), so the *bytes*
stay content-addressed and dedupe: two users uploading
byte-identical previews converge on one object at `ref_count = 2`.
Only the **mapping** is per-file — and the mapping is the part that
carries the trust problem. When user B uploads the same
`image.png`, B matches `a1b2c3…` in `content_derived_blobs` and gets the
server-derived thumbnails; B has no `file_attached_blobs` row, so A's
preview is unreachable.

Read precedence is unchanged from today (the client's preview wins):
`file_attached_blobs` for `(file_id, 'preview', …)` first, else
`content_derived_blobs` for `(source_hash, 'thumbnail', 'preview')`.
Both fold into the query the handler already issues.

`uploaded_by` is new. Today there is no provenance at all on a
client preview, and an Editor on a shared file can overwrite the
owner's — same family as the known Editor-can-rename gap
(`bug_drive_rename_editor_can_do_it`), and worth the same decision.

Today's code is already safe, implicitly, via its choice of
filename; the risk is losing that in the migration:

- write is file-keyed — `store_external_thumbnail` writes only
  `ext-{file_id}.jpg`, never into the `{blob_hash}.{ext}` space
- read checks the file-keyed path *before* the content-keyed one in
  `get_cached_thumbnail`, so a preview only surfaces for its own
  file
- `PUT …/thumbnail/{size}` requires `Permission::Update` on the
  target file

Two things keep the boundary after the refactor:

- **The schema is self-guarding.** With no `source_file_id` column
  there is nowhere to put a client preview, so making the mistake
  requires writing a migration — which gets reviewed. Keeping a
  nullable column would be an attractive nuisance; omitting it *is*
  the enforcement.
- **State the invariant in the migration**, in the style
  `content_search_index.sql` already uses: *keyed by `source_hash`
  because derived content is a pure function of the source bytes;
  client-uploaded previews are NOT derived, are user-chosen, and
  must never be keyed here or one user's preview would be served for
  another user's identical file.*

Note the pressure this is under: a "unify the two write paths" pass
would produce exactly the vulnerability. The two axes are separate —
client previews share the **dispatch** (they become blobs via
`store_from_stream` like everything else, satisfying the
no-backend-branching rule) while keeping **file keying** in their own
small mapping table. Storage dedup is preserved either way, because
the derived bytes are still content-addressed: two byte-identical
previews converge on one object at `ref_count = 2`. Only the mapping
is per-file, and the mapping is the part that carries the trust
problem.

**This is a live, shipping feature — not a deferred one.**
`frontend/src/lib/utils/thumbnail.ts` generates all three canonical
sizes client-side and `PUT`s them back, for
`SUPPORTED_MIME_TYPE = [image/*, application/pdf, video/*]`, using
vendored pdf.js to render page 1 of a PDF. It is the fallback for
mime types the server cannot handle.

PDF is the case that matters: **the backend has no PDF rasteriser**
(no pdfium, no poppler), so a client-supplied PDF thumbnail is the
only one that will ever exist. Unlike video frames, which
`generate_video_thumbnails_background` can rebuild with ffmpeg, a
dropped PDF thumbnail is gone until some user happens to reopen that
document in the SPA. So `storage.file_attached_blobs` is **required
by the migration** and ships with the thumbnail slice, not after it.

PDF also gives the keying rule its sharpest example. A client-rendered
page 1 *feels* like a pure function of the content — far more so than
a video poster frame — which is exactly why the rule is about **who
controls the bytes**, not about whether an honest implementation would
be deterministic. Nothing stops a client `PUT`ting an arbitrary JPEG
as "page 1". Content-keyed, every user holding that PDF sees the
forgery; file-keyed, it stays with the uploader's own file.

`uploaded_by` follows the `storage.shares.created_by` convention —
`NOT NULL`, **no FK** — so the audit fact survives the uploader's
account being deleted (users are hard-deleted here). A FK would force
either `ON DELETE CASCADE`, destroying other people's thumbnails when
an account closes, or `ON DELETE SET NULL`, destroying the audit trail
this column exists for. Rows imported from the sidecar, which records
no uploader, get an all-zeros sentinel; the UI's "removed user"
fallback renders both cases.

Unrelated bug spotted in the same function: the `PUT` uses a raw
`fetch` with `getCsrfHeaders()` rather than `apiFetch`, so no DPoP
proof is attached — the raw-fetch bypass class that lands on Gate C.
It also `Promise.all`s the responses without checking status, so a
rejection would be silent. Worth verifying against current DPoP
enforcement on `/api/files/*`.

## Copy and version semantics — the second and third axes

Keying is not the only property a satellite table has. Two more must
be declared explicitly, because neither is derivable from the key and
getting either wrong is silent data loss:

| | keyed by | on copy | on new version |
|---|---|---|---|
| thumbnails, transcodes | content | free — shares hash | free — new hash gets its own rows; old rows stay valid for the old version |
| previews, subtitles, cover art | file (→ *version*, see below) | **duplicate** + take a reference | **invalidate** / attach to the new version |
| comments | file | **do NOT duplicate** — a copy is a new artifact; the discussion belongs to the original | unaffected — a conversation spans versions |
| WebDAV dead properties | file | duplicate (RFC 4918 §8.8) | unaffected — properties describe the resource |
| EXIF (`storage.file_metadata`) | content ⚠ *file-keyed today* | would be free | would be free |

**`on_file_copied` stops being a no-op.** It is one today only because
every derived artifact is content-keyed, so a copy shares them for
free. The moment `file_attached_blobs` exists, copy must duplicate its
rows and take a blob reference for each.

### The copy fan-out is exactly two sites — and they have already drifted

| | single-file copy | folder-tree copy (cascading) |
|---|---|---|
| impl | `copy_file` CTE, `file_blob_write_repository.rs:592` | `storage.copy_folder_tree` (latest def: migration `20260902000001`) |
| `storage.files` row | ✅ in CTE | ✅ |
| dead properties | ✅ `dead_prop_copy` CTE arm | ✅ folder + file props |
| blob reference | ✅ `dedup.add_reference()` — manifest-aware | ❌ hand-rolled `UPDATE storage.blobs` — **misses manifests** |

That last cell is a live data-loss bug, not a plan concern — see the
note at the end of this section. It is the same logic written twice,
once correctly and once not, which is exactly what
`AGENTS.md § Code duplication` forbids. Adding file-keyed tables
without consolidating first means writing it a third and fourth time.

So single-source the fan-out:

```sql
-- ONE place that knows what follows a file on copy.
CREATE FUNCTION storage.copy_file_satellites(old_file_id UUID, new_file_id UUID)
RETURNS void AS $$
BEGIN
    INSERT INTO storage.webdav_dead_properties (file_id, namespace, local_name, value)
    SELECT new_file_id, namespace, local_name, value
      FROM storage.webdav_dead_properties WHERE file_id = old_file_id;

    INSERT INTO storage.file_attached_blobs
        (file_id, kind, variant, blob_hash, size, uploaded_by)
    SELECT new_file_id, kind, variant, blob_hash, size, uploaded_by
      FROM storage.file_attached_blobs WHERE file_id = old_file_id;
    -- then take a MANIFEST-AWARE reference per inserted blob_hash

    -- storage.comments: deliberately NOT copied. See the table above.
END;
$$ LANGUAGE plpgsql;
```

Both paths call it — the tree function per row in `_copy_file_map`,
the single-file path as one statement. Three properties worth having:

1. **The function body *is* the copy-semantics declaration.** Comments
   are absent, and that absence is the documented decision rather than
   an omission someone has to notice. The taxonomy becomes executable.
2. Adding a file-keyed table is one edit, so the two paths cannot
   drift again.
3. The refcount bump lives there once, manifest-aware, which fixes the
   tree-copy bug as a consequence of consolidating rather than as a
   separate patch.

References inside it must go through the same manifest-first logic as
`add_reference`, never `UPDATE storage.blobs`. That argues for a
`storage.add_blob_reference(hash)` SQL helper so the contract exists
exactly once in the database too.

### Versioning (future) — free on the content side

When file versioning lands, each version has its own content hash.
The content-keyed half needs **nothing**: version N's thumbnails are
the `content_derived_blobs` rows with `source_hash = H_N`, already
present, already shared with any other file holding that content.
Rolling back to an old version has its thumbnails instantly, with no
re-derivation. Versioning and dedup turn out to be the same mechanism
— which is the strongest retroactive argument for content-keying:
file-keyed thumbnails would need a re-derived set per version with no
sharing between a version and a copy of it.

What propagates is a *reference*, not a copy: each `file_versions` row
holds a reference on its Blob, which transitively keeps that version's
derived rows alive, since `on_blob_deleted(source_hash)` only fires
once the source Blob genuinely dies. So `file_versions` is simply
another `BlobReferenceSource` — same shape, no new machinery.

The file-keyed half is what versioning complicates: a client-uploaded
preview belongs to a *version*, not a file, because version 2's
content makes version 1's preview wrong. Expect
`file_attached_blobs`'s key to migrate from `file_id` to `version_id`
at that point. Today's `on_file_updated` (delete, then regenerate) is
the degenerate case — versioning with history depth 1. Comments stay
keyed on `file_id`.

### ⚠ Pre-existing bug found while mapping this

`storage.copy_folder_tree` bumps refcounts with
`UPDATE storage.blobs … WHERE b.hash = hc.blob_hash`
(migration `20260902000001`, lines 141-153). A CDC file's `blob_hash`
names a **manifest**, so for a multi-chunk file that predicate matches
zero rows and the copy takes **no reference at all**. Deleting the
original then drops the manifest 1 → 0, deletes it and dereferences
every chunk, and GC reaps them after the grace window — leaving the
copy pointing at nothing.

Single-chunk files (under `CDC_MIN_CHUNK`) escape by accident: their
whole-file hash equals their lone chunk's hash, so the UPDATE matches,
and `read_blob_bytes`'s no-manifest fallback still finds the bytes. So
the symptom is size-dependent — copy a folder of files ≥ 64 KB, delete
the original, lose the copies.

The Rust caller (`file_blob_write_repository.rs:1038-1067`) only
invokes the function, so nothing compensates at the application layer.
Derived from reading the SQL against the refcount contract, **not from
a reproduction** — needs a test before anyone acts on it. Same root
cause as prerequisite 2 below: the manifest counter is written by
`dedup_service` and reconciled by nothing, so this has been invisible.

Related, much narrower: the single-file path calls `add_reference`
*after* its CTE, best-effort with a warning on failure
(`:708-715`), so a failure there leaves a file row holding no
reference. Belongs in the same transaction.

## `BlobReferenceSource` — reference tracking abstraction

Adding new blob-owning services without teaching the ref-count +
consistency machinery about them causes silent orphaning risk:
`dedup_gc` sees `ref_count = 0` and reaps live content.

The extension point:

```rust
#[async_trait]
pub trait BlobReferenceSource: Send + Sync {
    /// Short stable identifier for logs / consistency finding
    /// `source` fields. Suggested: `"files"`, `"chunks"`,
    /// `"derived"`.
    fn source_name(&self) -> &'static str;

    /// Count of references this source holds on `blob_hash`.
    /// **On-demand path only** (`dedup_gc` checking one reap
    /// candidate). MUST NOT be used by the consistency sweep — see
    /// `ref_count_sql` below.
    async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;

    /// A correlated-subquery fragment counting this source's
    /// references **at `level`** to the outer row's hash, e.g.
    /// `"(SELECT COUNT(*) FROM storage.content_derived_blobs d WHERE d.blob_hash = m.file_hash)"`.
    /// `None` when this source holds no references at that level.
    ///
    /// The registry sums the fragments per level into that level's
    /// existing per-page SELECT, so each sweep stays ONE query per
    /// page instead of degrading to (sources × rows) round-trips.
    /// Identifiers only — never interpolate caller input.
    ///
    /// A source may contribute at BOTH levels, so this is **not** a
    /// per-source constant: `FilesReferenceSource` references a chunk
    /// for manifest-less legacy rows and a Blob for CDC rows. An
    /// earlier draft modelled this as `fn ref_level(&self) -> RefLevel`
    /// and was wrong for exactly that source.
    fn ref_count_sql(&self, level: RefLevel, outer_alias: &str) -> Option<String>;

    /// Iterate the source's referenced blobs, paged by the
    /// implementation's natural cursor (typically a DB PK). Used
    /// by `backend_consistency` to walk the backend against the
    /// union of all sources.
    async fn list_referenced_blobs(
        &self,
        cursor: Option<Vec<u8>>,
        limit: usize,
    ) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;

    /// Optional notify hook: `dedup_gc` reaped this blob. Sources
    /// that maintain their own denormalised refcount table can
    /// clean up here. Most sources leave this as the trait default
    /// (noop).
    fn on_blob_reaped(&self, _blob_hash: &str) {}
}
```

Wired via a `BlobReferenceRegistry`:

```rust
pub struct BlobReferenceRegistry {
    sources: Vec<Arc<dyn BlobReferenceSource>>,
}

impl BlobReferenceRegistry {
    pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>);
    pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError>;
    // ... etc.
}
```

Current implicit sources become the first two explicit
registrations:

- `FilesReferenceSource` — wraps `storage.files.blob_hash`
- `ChunksReferenceSource` — wraps `storage.chunk_manifests.chunk_hashes[]`

Tier-2 migration adds two more — one per table, not one per `kind`,
and both satisfied by the same generic implementation parameterised
by table + column:

- `ContentDerivedReferenceSource` — `storage.content_derived_blobs.blob_hash`
- `FileAttachedReferenceSource` — `storage.file_attached_blobs.blob_hash`

`ChunksReferenceSource` stays bespoke (array containment,
`b.hash = ANY(m.chunk_hashes)`).

### Two counters — get the level right

`add_reference` bumps `chunk_manifests.ref_count` first and only
falls back to `storage.blobs.ref_count`. So references land at
whichever level the hash names:

| Reference holder | References a… | Feeds |
|---|---|---|
| `chunk_manifests.chunk_hashes[]` | chunk | `storage.blobs.ref_count` |
| `files.blob_hash` (legacy, no manifest) | whole-file blob | `storage.blobs.ref_count` |
| `files.blob_hash` (CDC) | Blob via manifest | `chunk_manifests.ref_count` |
| `content_derived_blobs.blob_hash` | Blob via manifest | `chunk_manifests.ref_count` |
| `file_attached_blobs.blob_hash` | Blob via manifest | `chunk_manifests.ref_count` |

Both new tables reference *manifests*, never chunks — hence the
`level` parameter on `ref_count_sql`. They return `None` for the chunk
level. Adding their fragments to the chunk-level recompute would
double-count systematically.

**The aliasing trap is the norm here, not an edge case.** Today's
recompute carries a `NOT EXISTS` clause because a single-chunk file's
`file_hash` equals its lone chunk's hash (~40% of uploads per the
comment at `blobs_consistency_service.rs:388`). Derived blobs sit
below `CDC_MIN_CHUNK`, so ~100% of them are single-chunk and hit that
aliasing case. Level-correctness is not optional.

Then:

- **`dedup_gc`** — orphan iff `registry.total_references(hash) == 0`
  (with the existing grace window). Per-hash `count_references` is
  fine here: candidates are already filtered to `ref_count = 0` past
  the grace window, so the set is small.
- **`blobs_consistency`** — `refcount_mismatch` sums the sources'
  `ref_count_sql` fragments into its existing per-page SELECT
  (`blobs_consistency_service.rs:395-411`), which is already
  set-based. It must NOT be rewritten to call
  `registry.total_references` per row — that would turn one query per
  page into (sources × blobs) round-trips.
- **`backend_consistency`** — walks the backend and unions all
  `list_referenced_blobs` streams for the "did we lose bytes"
  check.

### Consistency coverage — what is and isn't tracked

Findings each job reports today, and where the new tables land:

| # | Edge | Direction | Mechanism | Status |
|---|---|---|---|---|
| 1 | backend → `storage.blobs` | orphan bytes | `orphan_blob` (backend_consistency) | ✓ |
| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` (backend_consistency) | ✓ |
| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` (backend_consistency, `?deep=true`) | ✓ |
| 4 | manifest → chunks | chunk reaped | `chunk_missing` (files_consistency) | ✓ |
| 5 | `files` → Blob | dangling | `missing_blob` (files_consistency) | ✓ |
| 6 | `storage.blobs.ref_count` | recompute | `refcount_mismatch` | ✓ chunk level only |
| 7 | `chunk_manifests.ref_count` | recompute | `refcount_mismatch` (manifests_consistency) | ✓ manifest level |
| 8 | manifest orphan reaping | GC predicate | registry `NOT EXISTS` union, no `ref_count` | ✓ |
| 9 | derived/attached → Blob | dangling | `derived_dangling_blob`, `attached_dangling_blob` (satellites_consistency) | ✓ |
| 10 | `content_derived_blobs.source_hash` → Blob | orphan mapping | `derived_orphan_mapping` (satellites_consistency) | ✓ |
| 11 | chunk at `ref_count = 0` past grace, still present | GC lag | — | ✗ a stalled GC is silent |
| 12 | `blob_extracted_text`, `faces.faces` orphans | dependents | search worker self-janitors; `faces` unverified | ~ verify |
| 13 | `file_attached_blobs.file_id` → `files` | dangling | FK `ON DELETE CASCADE` | ✓ DB-enforced |

**`backend_consistency` needs no change.** Its probe asks "does this
backend object have a `storage.blobs` row?" — and the backend holds
**chunks**, which neither new table ever references. A derived blob's
bytes are registered there by `store_from_stream` like any other
chunk. Adding the new tables to that probe would look up Blob hashes
in a table of chunk hashes and, for multi-chunk blobs, match nothing.

Rows 9-11 are new work; row 12 wants a check. Row 7 is the
pre-existing hole. Row 8 is the blocker:

### ⚠ Blocker — `dedup_gc` will delete every derived blob

> **RESOLVED.** Rows 7 and 8 are both closed, and this section is kept
> because the reasoning explains why the predicate looks the way it does.
>
> * **Row 8** — the predicate is registry-driven and, since the
>   `ref_count` arm was removed, contains no counter at all:
>   `WHERE <no registered source references it>`. The counter could
>   otherwise delete on its own, which made a reference that was never
>   taken into data loss rather than a wrong number. GC phase 2 got the
>   same treatment, with the registry predicate ANDed onto its existing
>   guards rather than replacing them (the counting fragments are too
>   narrow to be a reap guard — see `blob_reap_sql`).
> * **Row 7** — `ManifestsConsistencyCheck`
>   (`manifests_consistency_service.rs`) reconciles
>   `chunk_manifests.ref_count` against the same registry, so the
>   counter is corrected rather than trusted.
>
> The two were indeed coupled, as predicted below — but the resolution
> inverted the dependency. Rather than the recompute making the counter
> safe to trust, the reap predicate stopped trusting it, which demotes
> counter drift from data loss to a space leak that the recompute then
> reports.

The zero-ref manifest sweep (`dedup_service.rs:2574`) is:

```sql
DELETE FROM storage.chunk_manifests
 WHERE m.ref_count <= 0
    OR NOT EXISTS (SELECT 1 FROM storage.files f WHERE f.blob_hash = m.file_hash)
```

That `OR` hardcodes **"`storage.files` is the only thing that can
reference a manifest."** A thumbnail's manifest has `ref_count = 1`,
held by its `content_derived_blobs` row, so the first condition is
false — but no `storage.files` row points at a thumbnail's Blob hash,
so `NOT EXISTS` is true, the `OR` fires, and the manifest is deleted,
its chunks dereferenced and the bytes reaped.

Ship `content_derived_blobs` without changing this and **the first
`dedup_gc` run destroys the entire derived tier.** Not drift —
immediate, total loss.

It is also the missing half of row 7: that `OR` is the self-healing
hack that makes an unreconciled `chunk_manifests.ref_count`
survivable. Adding referrers breaks the assumption the hack rests on,
which is why 7 and 8 have to be fixed together. The predicate becomes
registry-driven — the union of every source's manifest-level
`ref_count_sql`, not a hardcoded table name.

### Three hard prerequisites, not sequencing preferences

0. **Fix the `dedup_gc` manifest predicate** (row 8) **before a single
   derived blob is written.** Everything else in this plan is
   recoverable; this one silently deletes data.
1. **Registry before the tables.** `blobs_consistency` derives
   expected refcounts from `storage.files` + manifests only. Ship
   `storage.content_derived_blobs` first and *every* derived blob becomes a
   `refcount_mismatch` finding — a flood, and one an operator might
   "repair".
2. **`chunk_manifests.ref_count` must become verified.** It is
   currently maintained by `dedup_service` and reconciled by
   *nothing*: `blobs_consistency` only recomputes
   `storage.blobs.ref_count`, and the manifest-level integrity it
   defers to `files_consistency::chunk_missing` is a different check
   (manifests pointing at reaped chunks). Since both new tables feed
   the manifest counter, registering a source would give the
   *illusion* of coverage, not coverage. Failure mode: a manifest
   stuck at `ref_count > 0` forever, so its chunks are never
   reclaimed. A manifest-level recompute is part of this work.

## Read path and caching

Request carries `(file_id, size, format)`; the derived hash is
BLAKE3 of bytes that don't exist yet, so it is not computable from
the request. Read order:

1. **moka RAM tier** — keyed by `(source_hash, size, format)`.
   (Today it is keyed by `file_id`; rekeying to `source_hash` costs
   nothing — the handler already has the hash from the row it
   loaded — and stops N copies of one photo occupying N entries for
   identical bytes.)
2. **DB** — `content_derived_blobs` lookup for the variant. This is free:
   the miss path already queries `get_blob_hash(&id)`, so a
   `LEFT JOIN` on `content_derived_blobs` returns the source hash and the
   derived hash in one query. Net DB cost unchanged from today.
3. **`dedup.read_blob_bytes(derived_hash)`** — through the normal
   backend stack, which is where the disk cache lives.
4. Generate only if step 2 found no row.

### HTTP ETag — move to the derived hash when the order flips

Shipped ahead of this plan (2026-08-24): the thumbnail ETag is
`"thumb-{source_hash}-{size}-{format}"` on both the REST and the
NextCloud preview endpoint. It replaced a `file_id`-keyed ETag that,
combined with `Cache-Control: immutable`, meant replacing a file's
content never invalidated the client's copy — `file_id` survives the
replacement, so the ETag did too, for a year.

**That key is still one term short.** A thumbnail is a function of
`(source bytes, size, format, RENDERER)`. Change the encoder, a quality
setting, or EXIF-rotation handling, and identical inputs produce
different bytes under an unchanged ETag — the same staleness class, one
level down. It bites when an already-cached thumbnail is re-rendered
after a renderer change (sidecar evicted, regenerated on miss).

**The fix is to key on the derived blob's own hash** — the ETag then
*is* the hash of the bytes served, so any change in output invalidates
by construction, with no version constant to remember to bump. It is
self-consistent for free: `store_derived_blob` is
`ON CONFLICT DO NOTHING`, so a re-render never displaces the stored
row, and the ETag therefore always equals what the derived tier will
serve.

**It must land with the read-order flip, not before.** Today the
derived tier is deliberately read LAST, so an ETag naming the derived
hash would describe a tier the response probably did not come from.
The two agree at creation — `render_and_persist_all_webp` writes both
from the same bytes — but diverge if a sidecar is re-rendered while the
derived row stays pinned by `DO NOTHING`. Sidecar is served, ETag
describes the other one: an ETag that lies about the body is worse than
one that is merely coarse. Two further reasons it has to wait: the
derived tier is WebP-only (`variant = size.dir_name()`, no format in
the key), so non-WebP clients have no row to key on; and nothing
predating this work has a row until `derived_import` backfills.

So at step 10, alongside the flip: ETag becomes the derived hash, with
the current `source_hash` form kept as the fallback for a variant not
yet generated and for formats the derived tier does not hold. The
`LEFT JOIN` in step 2 above already returns the derived hash in the
same query, so the ETag costs no extra round-trip.

**Attempted early (2026-08-26) and reverted — the constraint above is
load-bearing.** The attached half shipped and is correct, because an
upload writes its row synchronously before any read can observe it. The
*derived* half was brought forward at the same time and had to be backed
out: the ETag is computed **before** the body, so on a cache miss no row
exists and the handler emits the source-keyed form — then rendering
creates the row, and the very next request resolves to the derived hash.
The validator changed as a side effect of producing the body, so every
first render was immediately stale. Caught by
`thumbnail_etag_content_keyed.hurl`, where two consecutive GETs of an
unchanged file stopped revalidating to 304.

**The flip alone does NOT remove the hazard** *(corrected 2026-08-26 —
an earlier revision of this paragraph claimed it did)*. A first render
still creates the row as a side effect of producing the body, whatever
the read order, so two consecutive reads would still straddle its
appearance.

What actually removes it is resolving the ETag **after** generation on
the 200 path. A 304 can only fire when the client already holds a
validator, which means it has been served before, which means the row
exists — so the *conditional* path can safely consult the derived hash
up front, while the *generating* path computes it from bytes it now
holds. That is a handler restructure, not an ordering change, and it is
the actual prerequisite for the derived-hash ETag.

**The disk cache is `CachedBlobBackend`, reused unchanged.** No
thumbnail-specific cache, no second root path. Routing derived
blobs through the same stack gets, for free:

- **single-flight per hash** — a gallery cold-load where 50 clients
  race one thumbnail collapses to one S3 GET
- **write-through on put** — the instance that generated the
  thumbnail already has it locally, so upload→view-gallery never
  round-trips to S3
- byte-budget eviction with unlink, and a restart-survivable index

**One thing to test rather than assume.** Thumbnails and source
blobs have opposite cache profiles: small / hot / expensive to
regenerate versus large / cold / cheap to re-fetch. Sharing one LRU
budget (`OXICLOUD_STORAGE_CACHE_MAX_SIZE`, default 50 GB) means a
sequential multi-GB video read is exactly the scan pattern that
flushes a working set — and flushing thumbnails costs a re-render,
not a re-download. moka 0.12's TinyLFU admission *should* resist
this (a one-shot large entry denied rather than evicting
frequently-hit small ones), and the eviction listener unlinks on
`RemovalCause::Size` so a denied entry shouldn't leak its file. Both
deserve a test, because the failure mode is silent: unexplained CPU
on the thumbnail path, not a cache metric.

If it does interfere, the fix that preserves the one-implementation
rule is **two instances of `CachedBlobBackend` with separate
budgets** — same type, same factory, different config — not a second
cache type. Honest cost: `DedupService` would need a second backend
handle plus a content-class selector, since derived blobs have
manifests and must still be reassembled through `DedupService`. Ship
the shared cache, measure, split only if the test says so. The knob
would be `OXICLOUD_STORAGE_DERIVED_CACHE_MAX_SIZE` alongside the
existing `OXICLOUD_STORAGE_CACHE_MAX_SIZE`.

## Cost consequences to budget for

1M photos × 3 sizes ≈ **3M additional backend objects and 3M
additional `storage.blobs` + `chunk_manifests` rows** (~100 KB of
derived content per photo, so ~100 GB total) — 3M and not 6M because
only the canonical WebP is persisted; the JPEG fallback never reaches
tier 3. Two costs land:

- **PUT requests at upload** — one-off, modest (~$15 per 3M on AWS
  pricing). Use `put_blob_from_bytes_unsynced` + a batched
  `sync_blobs`, not `put_blob_from_bytes`: the latter does a
  `head_object` before every PUT on S3, doubling the request count
  for an idempotency check content-addressing already guarantees.
- **`blobs_consistency` request amplification** — it does one
  `blob_exists` HEAD per blob row, so 4× the rows is 4× the S3
  requests, forever. This is the dominant recurring cost. Fix by
  diffing against `list_blob_hashes` pages in bulk (one LIST per
  1000 keys instead of 1000 HEADs) rather than per-row probes.
  Alternative: exempt derived rows from the byte-level check, since
  they are regenerable — but the bulk-LIST fix is better and helps
  source blobs too.

Note for object-store deployments: IA/Glacier tiers bill a 128 KB
minimum per object, so an 8 KB icon is billed at 128 KB. **Open
option** (config, not a code branch): persist only the `large`
variant to tier 3 and derive icon/preview from it on demand — the
render path already decodes once for all sizes, and resampling an
800px WebP is sub-millisecond. That is 1M objects instead of 3M. It
changes only *how many variants get a `content_derived_blobs` row*, so it
stays a single code path.

## Sidecar directories after this refactor

| Sidecar today | After |
|---|---|
| `.thumbnails/` | **Gone.** Derived blobs live in tier 3; caching is `CachedBlobBackend` in `.blob-cache/`, keyed by hash like every other blob. |
| `.transcoded/` | Gone, same shape. |
| `.blob-cache/` | Stays. Owned by `CachedBlobBackend`, path via `OXICLOUD_STORAGE_CACHE_PATH`. Now serves derived blobs too. |
| `.search-index/` | Open question — see non-goals. |
| `.plugin-logs/` | Ops-local; stays. |
| `.uploads/` | Tier 1 already; migrates to `OXICLOUD_TEMP_DIR`. |

**`OXICLOUD_SPOOL_DIR` is probably no longer worth adding.** The
2026-08-02 draft reserved it as the home for `.thumbnails/`,
`.transcoded/` and `.blob-cache/`. The first two now disappear
entirely rather than becoming local caches, and `.blob-cache/`
already has its own `OXICLOUD_STORAGE_CACHE_PATH`. That leaves
`.search-index/` (a non-goal) and `.plugin-logs/` (ops-local) — not
enough to justify a new config surface. Either drop step 2 below or
reduce it to documenting the existing `OXICLOUD_STORAGE_CACHE_PATH`
as the tier-2 relocation knob.

## Migrating the existing sidecar content

### Inventory → destination

| On disk today | Goes to |
|---|---|
| `.thumbnails/{icon\|preview\|large}/{blob_hash}.webp` | `content_derived_blobs(source_hash = stem, kind='thumbnail', variant = dir, blob_hash = BLAKE3(bytes), content_type='image/webp')` |
| `.thumbnails/{…}/{blob_hash}.jpg` | **Deleted, not imported** — tier 3 is canonical-only |
| `.thumbnails/{…}/ext-{file_id}.jpg` | `file_attached_blobs(file_id = stem, kind='preview', variant = dir, content_type='image/jpeg', uploaded_by = sentinel)` |
| `.transcoded/webp/{file_id}…` | Deferred to the transcode slice — and it is a **re-keying**, since transcodes are file-keyed today (`get_cache_path(file_id, …)`) and must join `storage.files` to resolve `blob_hash` |
| `.transcoded/` skip markers | Not imported — a negative verdict costs one transcode attempt to recompute |

The server-derived thumbnail case is lossless: the path already
encodes `(source_hash, variant)`, and the derived hash is just BLAKE3
of the file's own bytes. **No source read, no decode.**

### Why import rather than regenerate

Regenerating 1M photos × 3 sizes means 1M GETs of **full-size
originals** — multi-MB each, from S3 on remote backends — plus 1M
decodes and 3M encodes. Importing reads ~20 KB local files, hashes,
and PUTs. It is the difference between a background job and a
maintenance window, and on object storage it is a real egress and
request bill.

### The `ext-` case, unfolded

This is the hard one, and it triages by the *file's* mime type:

- **`application/pdf` — must import.** Irreplaceable: no server-side
  rasteriser exists. Dropping it loses the thumbnail until a user
  reopens the document in the SPA.
- **`video/*` — should import.** `generate_video_thumbnails_background`
  could rebuild it, but an ffmpeg run per video costs far more than
  copying a 20 KB JPEG.
- **`image/*` — safe to drop.** The server regenerates from source and
  produces a *better* result (WebP rather than the client's JPEG).
  Importing is still cheaper; either is defensible.

Three complications specific to this path:

1. **Three files per file**, not one — the client uploads icon,
   preview and large. On a document-heavy install
   `file_attached_blobs` can rival the derived table in row count.
2. **`file_id` may be stale.** The file may since have been deleted.
   Verify against `storage.files.id`; skip and log otherwise.
3. **No uploader is recorded.** Use the all-zeros sentinel (possible
   only because `uploaded_by` has no FK).

### Strategy — read-through, then a batch tail

**Phase 1 (release N): dual read.** The read path checks the new
tables first, falls back to the legacy sidecar on a miss, and imports
that entry inline before serving. Hot content migrates itself under
real traffic. Writes go only to the new tables. The legacy path lives
in **one clearly-marked module** — it temporarily reintroduces the
`thumbnails_root` construction this plan exists to delete, which is
acceptable only with a named removal release.

**Phase 2: `derived_import`**, a registered `JobRegistry` job (*not* a
sqlx migration — migrations are SQL-only and run at boot; this walks a
filesystem and a remote backend for hours). It sweeps the cold tail:

- **Resumable** — checkpoint a `(dir, filename)` cursor, same shape as
  the consistency jobs.
- **Idempotent** — `INSERT … ON CONFLICT DO NOTHING`, and on conflict
  *release* the reference the blob write just took, or re-runs inflate
  refcounts.
- **Verifies the source exists** before inserting. `.thumbnails/` can
  hold orphans if `on_blob_deleted` ever failed, and `source_hash` has
  no FK — importing one creates a row pointing at nothing that holds a
  reference forever.
- Reports imported / skipped-orphan / skipped-jpeg / failed counts.

**Phase 3: the job deletes, not a release.** *(revised 2026-08-26 —
supersedes "delete in release N+1, gated on an empty tail")*

Sidecars are **local disk**. A release cannot know whether every
instance has drained, so gating deletion on "the tail is empty" asks an
operator to coordinate a fact nothing reports — and there is no way to
know when, or whether, they will trigger the jobs at all. Instead the
import unlinks each sidecar it has successfully imported, so **each
instance drains itself** and the directory becomes removable once
genuinely empty.

Three constraints on that:

- **Verify readback before unlinking.** Import → read the derived blob
  back through the normal stack → *then* delete. A store that reported
  success but landed unreadable would otherwise take the last copy with
  it. Cheap next to the decode already performed, and it is the
  difference between a migration and a data-loss bug.
- **Only after the read-order flip.** Deleting while the sidecar is
  still read *first* sends reads to the derived tier as a side effect of
  the migration — its first production traffic arriving by accident
  rather than by decision.
- **Opt-in** (`?delete_imported=true`). A migration that deletes on its
  default setting is surprising, and it is the same instinct as
  no-silent-auto-repair: early runs import only, so an operator can
  inspect before committing.

Register it as a **scheduled tick**, not a boot-time trigger: it is
idempotent and resumable, so periodic is safe, whereas walking a large
`.thumbnails/` during startup delays readiness for nothing.

**Phase 4: the fallback disables itself.** *(revised 2026-08-29 —
supersedes "the only remaining release is removing the fallback read
path")*

Removing the fallback in a release has the same flaw as gating deletion
on an empty tail, one level up: sidecars are local disk, so no release
can know that every instance has drained. Ed's framing is the answer —
the only removal you can actually write is `if the tier is gone, return`.

So `ThumbnailService::initialize` probes the size directories once at
boot and stores the result. When absent, every fallback read
short-circuits on a relaxed atomic load, no syscall. The code stays,
costs nothing, and can be deleted whenever — or never. No coordination,
no named release.

Two things had to change for absence to be reachable at all:

- **`initialize` no longer creates the directories.** It
  `create_dir_all`-ed all three at every boot, so the job removed them
  and the next restart put them back; the absence this gates on was
  unreachable by construction. Nothing has written a sidecar since step
  10d2, so there was nothing to create them for.
- **The probe tests the size directories, not the root.** On macOS
  Finder leaves a `.DS_Store` in the root, which blocks `remove_dir`
  there permanently. Gating on the root would keep the fallback alive on
  every developer machine for a reason unrelated to thumbnails. No size
  directory means no sidecar.

The root removal now reports its outcome instead of discarding it —
it is the one result an operator is waiting for, and `.DS_Store` is a
failure worth naming rather than a silent no-op.

### Prerequisite: one persist function (found 2026-08-26)

**Four render paths write a sidecar; only one also writes the derived
row.** `store_derived_blob` has a single call site — in
`render_and_persist_all_webp` — while `fs::write(&thumb_path, …)` has
five. `get_thumbnail`, `generate_and_persist` and
`generate_all_sizes_background` all persist sidecar-only.

That breaks the migration's premise rather than merely being untidy: an
on-demand render (cache miss, a size never generated, an evicted
sidecar) keeps producing un-migrated state *after* the import runs, so
the tail never empties and the deletion gate never opens.

So before the imports can converge, all render paths must go through
**one** `persist_thumbnail` that writes the sidecar, the derived blob
and the moka entry together — the same single-source move as
`storage.copy_file_satellites`. What it writes then becomes a policy in
one place, so "stop writing sidecars" is later a one-line change rather
than four edits.

Interim setting is **dual-write**, for two reasons: it is what makes the
backlog finite, and it leaves reads untouched while the derived tier is
still unproven. Cost is one extra local `fs::write` per render,
negligible beside the decode. Note the sidecar *read* path must survive
until the directories are empty regardless, so stopping the write early
buys nothing.

Cost to be aware of: `ThumbnailService` holds no `DedupService` — it is
a per-call parameter (`dedup: Option<&DedupService>`) — so the
consolidation threads it through those paths, and
`generate_and_persist` takes a `thumb_path` where it will need the
`blob_hash` instead.

### The "just delete it" opt-out is no longer universally safe

For a small install with no PDFs, `rm -rf .thumbnails/` and lazy
regeneration is still fine — the content is regenerable by
definition. **PDFs break that**, since nothing server-side can rebuild
them. So the opt-out is safe only where `application/pdf` thumbnails
don't exist, and the importer should refuse to run in delete-only mode
if it finds any.

## Delivery order

Coarse — the trait + registry ship first (empty-impl for
`FilesReferenceSource` + `ChunksReferenceSource` mirroring today's
hardcoded SQL). New sources bolt on independently.

1. **`BlobReferenceSource` trait + registry** in
   `application/ports/`. `FilesReferenceSource` and
   `ChunksReferenceSource` implementations mirroring current SQL,
   with per-level `ref_count_sql` fragments summed into the existing
   per-page SELECT; wire into `dedup_gc` + `blobs_consistency` behind
   an integration test that proves the union equals the pre-refactor
   count on a real DB. **Blocks step 5** (prerequisite 1).
2. **`dedup_gc` manifest predicate becomes registry-driven** —
   replaces the hardcoded `OR NOT EXISTS (… storage.files …)` with the
   union of every source's manifest-level fragment. **Hard blocker**
   (prerequisite 0): until this lands, writing a derived blob means
   the next GC run deletes it. Ship with an integration test that
   creates a manifest referenced *only* from `content_derived_blobs`
   and asserts GC leaves it alone.
3. **Manifest-level refcount verification** — recompute
   `chunk_manifests.ref_count` against its actual referrers, the
   counter nothing reconciles today. **Also blocks step 5**
   (prerequisite 2): without it, derived-blob refcount drift is
   invisible, and step 2 removes the `OR` that used to mask it.
4. ~~`OXICLOUD_SPOOL_DIR`~~ — reduced to a docs change, or dropped;
   see the sidecar section.
5. **`ThumbnailService` writes go through `DedupService`**. New
   `storage.content_derived_blobs` table + `ContentDerivedReferenceSource`.
   Deletes `thumbnails_root`, `get_thumbnail_path`, and every
   filesystem call in the service. Fold the `content_derived_blobs` lookup
   into the handler's existing `get_blob_hash` query.
6. **`blobs_consistency` bulk-LIST diff** — before, or immediately
   after, step 5 lands at scale; per-row HEADs do not survive a 4×
   row count.
7. **`ImageTranscodeService`** — same shape, `kind = 'transcode'`,
   no new table. **Done 2026-08-30**, validated on a snapshot restore
   against S3: 13 cached transcodes imported, 5 `.skip` markers
   collapsing to 3 negative rows (three of them named the same content),
   `file_gone` exercised with a synthetic id, and every deletion
   preceded by a byte-for-byte readback.

   Two things landed beyond the three pieces below. The memory cache is
   now keyed by content as well, not just the durable tier — it was
   still `{file_id}:{ext}`, so identical content held two RAM entries
   and the second file missed. And `run_or_resume` binds a run to the
   flags it started with, so a paused `?repair=true` import can no
   longer resume as import-only.

   Still local-disk, deliberately: hash-less callers (external mounts)
   have no content identity, so they keep `.transcoded/` and it will
   not disappear on installs that have them. Absence is only expected
   elsewhere.

   *Original scoping, for the record.* The service does
   not currently write the derived tier at all, which is the same gap
   `persist_rendered` closed for thumbnails, and it must be closed
   before `transcode_import` can converge.

   Three concrete pieces, in order:

   * **Thread the source hash to the call site.** `get_transcoded` takes
     `file_id` only, and the table is content-keyed. The hash IS
     available one frame up — `file_retrieval_service::try_transcode`
     is called from a scope holding `dto.content_hash` — so it is a
     parameter to add, not a lookup to invent. Do **not** hash
     `original_content` on the fly: that is a BLAKE3 over the whole
     file on every request.
   * **Give the service a `BlobHandler`.** It has no field for one
     (`cache_dir`, `memory_cache`, `stats`), so this is a constructor
     and DI change — mind the construction order, as
     `ThumbnailService` hit the same thing and solved it with a
     per-call parameter instead.
   * **Write the `.skip` markers as negative rows** — resolved: nullable
     `blob_hash`, see *Negative verdicts*. Thumbnails need the same
     treatment for undecodable and over-`MAX_DECODE_PIXELS` sources,
     which are RAM-only today, so do both together rather than twice.

   Note the cache is keyed `{file_id}:{ext}` in memory and
   `.transcoded/{ext}/{file_id}.{ext}` on disk, so it also carries the
   file-vs-content keying mismatch that `transcode_import` has to
   re-key. Fixing the write path first means the import only has to
   handle history, not a moving target.
8. **`storage.copy_file_satellites` consolidation** — collapse the two
   copy paths onto one helper, with a manifest-aware reference bump.
   **Blocks step 9**: adding a file-keyed table before this means
   writing the same cascade a third and fourth time, into a pair of
   sites that have already drifted once.
9. **`storage.file_attached_blobs`** — **required, not deferred**: the
   client-side generator already ships and PDF thumbnails have no
   server-side regeneration path, so the migration depends on this
   table. Lands with step 5. File-keyed, never in
   `content_derived_blobs`. Register it in `copy_file_satellites` and
   declare its version semantics.
10. **Import jobs + the dual-read fallback** — see the migration
   section. Revised ordering as of 2026-08-26:

   a. **Consolidate onto one `persist_rendered`** (dual-write) — **done
      2026-08-26**. Was the blocker: only one of four render paths wrote
      the derived row, so the import could never converge. Every live
      render now dual-writes. Two paths still pass `None` and stay
      sidecar-only, which is safe *only* because both are reachable
      solely through the `ThumbnailPort` impl and nothing holds a
      `dyn ThumbnailPort` — if either gains a real caller it must take a
      `DedupService` first. See *Prerequisite: one persist function*.
   b. **`thumb_derived_import`** (shipped) and **`thumb_attached_import`**
      (shipped) — two jobs, not one, because the keying differs and that
      difference is the security boundary. A third, `transcode_import`,
      is still needed: `ImageTranscodeService` **already exists** and
      caches `.transcoded/{ext}/{file_id}.{ext}`, so those must be
      **re-keyed** file→content on import (legitimate only because a
      transcode is derivable). Its `.skip` markers import as **negative
      rows** with a NULL `blob_hash` — see *Negative verdicts* — so the
      verdict survives the deletion of `.transcoded/`, which it
      otherwise would not.

      **Not next, and deliberately so.** Two prerequisites, both learned
      the hard way on the thumbnail side:

      * **Format must move into `variant` first.** Transcodes are
        inherently multi-format, and `variant` is keyed on size alone —
        the same gap that keeps JPEG thumbnails on the sidecar (see
        10c). Importing before that means migrating into a schema that
        cannot hold the data without collisions. One migration unblocks
        both.
      * **Step 7 before the import.** `ImageTranscodeService` writes only
        its file-keyed cache today, so an import would run against a
        cache still growing and never reach an empty tail — precisely
        the trap `persist_rendered` had to close for thumbnails.

      Order: format-in-`variant` → step 7 → `transcode_import`.
   c. **Flip the read order**, derived first — **done 2026-08-26**. Two
      things it is not: a two-line swap, and an ETag fix.

      A derived miss must **fall through** to the sidecar, where the old
      code terminated the lookup — while the imports drain, most content
      has a sidecar and no row, so terminating would report "no
      thumbnail" for nearly everything.

      And it is **WebP-only**. `store_derived_blob` writes `image/webp`
      with `variant` keyed on size alone, no format term, so a JPEG
      request matches the WebP row and gets the wrong codec — a
      regression the old ordering hid, because the `.jpg` sidecar won
      first. JPEG clients therefore stay on the sidecar, **and the
      sidecar cannot be deleted for them** until `variant` encodes
      format. That is a new prerequisite for (e), not a detail: it means
      a migration to `(kind, variant, format)` — or a format term inside
      `variant` — has to land before the directories can go.
   d. **Enable deletion** in the import jobs (opt-in, readback-verified)
      — **done 2026-08-27**, both halves, sharing one
      `verify_and_unlink`.

   d2. **Stop writing sidecars.** *(Added 2026-08-27 — the sequence
      above was missing this, and (e)'s gate is unreachable without
      it: dual-write means any render or upload recreates the
      directory seconds after the job removes it, so "no longer
      exists" can never hold.)*

      Two sides, and they differ in what a failed write costs:

      * **Rendered / content-keyed — safe now.** Delete the `fs::write`
        in `persist_rendered`. No gate beyond the read flip, which has
        landed. Existing sidecars are untouched, so a box that has not
        imported yet keeps its fallback for old content; new content
        goes only to the derived tier, which the read path already
        prefers. If the derived store fails the bytes are still served
        and simply not cached — regenerable by definition.
      * **Uploaded / file-keyed — needs a change first.**
        `upload_thumbnail_impl` logs and still returns 201 when
        `store_attached_blob` fails, which is safe *only* because the
        `ext-` sidecar catches it. Remove that sidecar while the store
        is best-effort and a user's uploaded preview can vanish
        silently behind a success response. These are the
        non-regenerable bytes, so **the PUT must fail** before the
        write is removed.

      Order matters: make the attached store fatal, *then* drop its
      sidecar. Reversed, it trades a silent data-loss window for an
      empty directory.

   e. **Remove the fallback read path** once the directory no longer
      *exists* — not merely once it is empty. Two reasons. Empty is a
      momentary property an on-demand render can undo, whereas absence
      is one-way and observable, so the job removes the directory after
      draining it and that absence is the proof. And it is far cheaper
      to test: existence is a single `stat`, while emptiness costs an
      `opendir`/`readdir`/`closedir` — which matters if the fallback
      ever gates on it per read rather than once at boot. The only
      remaining release, and no data is at stake by then.
11. **`DedupService` → `BlobHandler` rename** — decided, mechanical,
    34 files. Standalone commit, `src/AGENTS.md` updated with it. Can
    land at any point; last is easiest, since every earlier slice
    would otherwise rebase across it.

12. **Document the two satellite tables** — `storage.content_derived_blobs`
    and `storage.file_attached_blobs`, with worked examples. **Not
    started.**

    The tables exist and each carries `COMMENT ON` text, but nothing
    explains the pair *together*, and the thing that matters about them
    is the relationship: they hold the same kind of artifact under two
    different keys, and **the keying difference is the security
    boundary**. Someone adding a third artifact type has to get that
    right, and today the only place it is written down is prose
    scattered through this plan and two job doc-comments.

    What the page needs:

    * **Column-by-column structure** of both tables, including the
      parts that are easy to get wrong: `blob_hash` is nullable on
      `content_derived_blobs` (a NULL is a NEGATIVE row — the
      derivation was attempted and is known not to be worth storing),
      paired with `content_type` by a CHECK; `uploaded_by` on
      `file_attached_blobs` is `NOT NULL` with no FK, so provenance
      survives a user deletion, and imported rows carry the nil-UUID
      sentinel rather than a fabricated owner.
    * **Why two tables and not one with a discriminator.**
      `content_derived_blobs` is keyed by the BLAKE3 of the SOURCE, so
      identical content shares one derivation — which is exactly what
      must NOT happen for user-supplied bytes, or one user's uploaded
      preview would be served for every file with matching content.
      That is the poisoning vector; the split is what prevents it, and
      a `kind` column on a single table would not.
    * **Worked examples**, which is what turns the rule into something
      checkable:
      - the same image uploaded twice → one `content_derived_blobs` row,
        two files, one thumbnail blob;
      - a PDF with a client-uploaded preview → one `file_attached_blobs`
        row per file, duplicated on copy, never shared;
      - a screenshot WebP cannot shrink → a negative row with NULL
        `blob_hash`, and what a reader is expected to conclude from it;
      - the same content transcoded and thumbnailed → two rows, same
        `source_hash`, different `kind`.
    * **Lifecycle**: which rows hold a reference (positive
      `blob_hash` bumps `chunk_manifests.ref_count`; a negative row
      holds none), what reaps them (`purge_derived_blobs` on source
      reap; the decrement trigger on `file_attached_blobs`), and which
      consistency job covers which failure — the point being that a
      satellite row whose SOURCE is gone breaks no refcount invariant,
      which is why `satellites_consistency` had to exist at all.
    * **The NULL-handling trap**, worth a paragraph because it has
      already bitten twice: SQL comparison against NULL is NULL, so
      `EXISTS (… WHERE blob_hash = h)` silently excludes negative rows.
      That is correct for refcounts (a negative row holds none) and
      WRONG for dangling checks (it reported every negative row as
      `data_loss`). Anything joining on `blob_hash` must decide which
      of the two it wants.

    * **A diagram**, because the security boundary is a shape before it
      is a rule — two arrows starting from different places is the
      whole argument, and it lands faster than any paragraph:

      ```mermaid
      erDiagram
          FILES  ||--o{ ATTACHED : "file_id — per FILE"
          FILES  }o--|| BLOBS    : "blob_hash (content)"
          BLOBS  ||--o{ DERIVED  : "source_hash — per CONTENT"
          DERIVED }o--o| ARTIFACT : "blob_hash (NULL = negative)"
          ATTACHED }o--|| ARTIFACT : "blob_hash"

          FILES {
              uuid id
              text blob_hash
          }
          DERIVED {
              text source_hash PK
              text kind        PK
              text variant     PK
              text blob_hash   "NULL = not worth deriving"
              text content_type "NULL iff blob_hash NULL"
          }
          ATTACHED {
              uuid file_id  PK
              text kind     PK
              text variant  PK
              text blob_hash
              uuid uploaded_by "no FK; nil = imported"
          }
          ARTIFACT {
              text hash PK
          }
      ```

      What a reader should take from it: `DERIVED` hangs off **content**,
      so two files with identical bytes reach the same row — good for a
      render, catastrophic for an upload. `ATTACHED` hangs off the
      **file**, so those rows are duplicated on copy and never shared.
      Both point at the same artifact space, which is why one table
      could not simply be folded into the other with a `kind` column.

    **Landed** as `docs/architecture/derived-and-attached-blobs.md`,
    beside `backend-storage.md`, which documents the blob layer these
    sit on top of. Operator-facing rather than end-user, so schema and
    SQL belong here in a way they do not in `docs/guide/`.

    Named for the two things rather than for "satellite tables" —
    that is internal shorthand nobody would search for, whereas
    *derived* and *attached* are the words the schema and the import
    jobs already use.

Tracked separately, **not** part of this plan: the
`storage.copy_folder_tree` refcount bug (see the copy section). It is
a production data-loss bug on a path this plan doesn't otherwise
touch, so it wants its own PR and its own reproduction test — but step
8 subsumes the fix, so sequence them together to avoid conflicting
edits to the same function.

Each slice is independently mergeable. Delivery span: rough
estimate ~2 weeks end-to-end.

## Naming clarifications to land alongside this refactor

Two consumer-facing terminology issues that surfaced during the
read-side normalisation (2026-08-02). They're not code-breakers,
but they cost every new implementor a mental round-trip, so
they belong in the tier-2 sweep:

### 1. `DedupService` name is implementation-shaped, not consumer-shaped

From a consumer's perspective the service is "the thing that
reads and writes file content by hash." Deduplication is one
internal responsibility (alongside CDC chunking, ref-counting,
GC). The name `DedupService` narrates HOW it works, not WHAT it
is — new service authors read the name and don't realise they
should be routing every blob read through it.

**Decided 2026-08-16: `BlobHandler`.** (`ContentStore` / `BlobStore`
were the alternatives.) Public surface stays identical; consumers
write `Arc<BlobHandler>` and call `blob_handler.read_blob_bytes(hash)`.
Internal doc-comments document dedup + CDC + GC as strategies.

Scope, measured: **34 files, 209 occurrences** of
`DedupService|dedup_service` under `src/`. Mechanical. Keep it as one
standalone commit so reviewers see "rename" independently from the
substantive changes — it is the noisiest diff in this plan and the
least interesting.

**`src/AGENTS.md` must change in the same commit.** Lines 20-21 name
`Arc<DedupService>` as *the* canonical read abstraction and list its
methods; leaving them would point the rule at a type that no longer
exists — and that rule is what stops new services from taking
`Arc<dyn BlobStorageBackend>` directly. The local variable name
`dedup` is used pervasively in call sites; renaming it to
`blob_handler` is what produces most of the 209 occurrences, so decide
up front whether the variable follows the type (recommended — the
whole point is that readers stop thinking "dedup" when they mean
"read file content").

### 2. `blob` overloaded across two scales

Current usage:

- `storage.blobs` — the physical storage table; rows are BYTES
  written to a backend. Post-CDC, most entries are chunks
  (fragments), not whole files.
- `storage.chunk_manifests` — the CDC manifest that references a
  set of `storage.blobs` rows to reconstitute a file.
- `file.blob_hash` — the hash a file row points at; either a
  whole-file blob (legacy) OR a chunk-manifest (post-CDC).

The word "blob" carries two meanings: *whole-file content* (what
a user thinks of when they say "download the blob") vs
*physical byte-payload on disk* (what the storage backend
holds — may be a whole file, may be a chunk fragment).

Proposed clarification for the tier-2 sweep:

- **Blob** = the abstraction of "content of a file", identified
  by BLAKE3 of the plaintext. Consumers work at this level. What
  `DedupService`/`BlobHandler` returns.
- **Chunk** = a physical byte-payload written to the backend,
  identified by its own BLAKE3. Storage-backend-internal.
- **Manifest** = the map from a Blob to one or more Chunks.

Schema rename (deferred, requires migration):

- `storage.blobs` → `storage.chunks` (that's what it actually holds now)
- `storage.chunk_manifests` → `storage.blob_manifests` (or keep — arguable)
- `BlobStorageBackend` trait → `ChunkStorageBackend` — reads and
  writes physical chunks, not blobs
- **`blobs_consistency` job → `chunks_consistency`** — it iterates
  `storage.blobs`, so it inherits whatever that table is called.

### The job rename has two rules of its own

**Travel with the schema, never ahead of it.** A job named
`chunks_consistency` iterating a table still called `storage.blobs` is
*more* confusing than today's mismatch, not less.

**Never recycle `blobs_consistency`.** Under the corrected taxonomy the
manifest job *is* the blob-level job, so the freed name looks
available — and reusing it would be the worst outcome available. A job
name that survives a release while changing meaning silently breaks
`POST /api/admin/jobs/<name>/trigger` URLs, every historical row in
`background_runs.job_name`, and any dashboard or alert keyed on it.
`manifests_consistency` is unambiguous under either taxonomy; leave it
alone. Net effect: one job renamed, not two swapped.

Budget for the operational cost either way — job names are not internal
identifiers. A rename orphans past runs unless `background_runs.job_name`
is migrated alongside, and any runbook naming the old one breaks. Worth
an alias period or an explicit release note.

### Explicitly NOT renamed

- **The `.blob` on-disk suffix** (`<hash>.blob` in `LocalBlobBackend`
  and `CachedBlobBackend`). Correcting it to `.chunk` would mean
  renaming every file in every deployment's blob store — a migration
  whose cost is wildly out of proportion to the clarity gained, and one
  that can fail halfway. The suffix is an implementation detail no
  consumer parses; leave it.
- **`file.blob_hash`** — semantics stay. It references a Blob via its
  manifest OR (for pre-CDC legacy) points directly at a single-chunk
  Blob whose hash equals its lone chunk's hash.

Scope for this rename: ~23 files touch the SQL, plus a migration
for the table rename. Not free. Ship AFTER the tier-2 write-side
lands so we don't stack schema changes.

## Non-goals

- **Tantivy `.search-index/`** — memory-mapped by design, doesn't
  fit the blob-storage abstraction. Separate future decision:
  keep local, snapshot-to-backend periodically, or retire Tantivy
  for PG-native full-text.
- **`.plugin-logs/`** — ops-local operational data, not user
  content. Stays local.
- **Client thumbnail negotiation protocol** — the wire-level API
  for how clients push their previews. Design piece for the
  photo/mobile team when there's a real feature ask.
- **A per-derived-content cache type.** Explicitly rejected: the
  disk cache is `CachedBlobBackend`, instantiated by the existing
  DI branch. Splitting budgets means a second *instance*, never a
  second implementation.
- **Key/value metadata on files.** A separate plan. It is not a blob
  table: k/v pairs are rows, so they belong in their own schema, NOT
  as a `file_attached_blobs` row with `kind = 'metadata'`. Both
  tables here map a key to exactly one blob hash; arbitrary k/v has
  different cardinality, indexing and query needs.

  But it will hit **the same content-vs-file split**, so it should
  inherit the convention rather than invent one:
  `storage.content_derived_*` for anything that is a pure function of
  the bytes (EXIF, dimensions, duration, extracted text — the split
  already exists as `storage.blob_extracted_text` and `faces.faces`),
  `storage.file_attached_*` for anything user-supplied or
  per-resource (tags, descriptions, custom properties — WebDAV dead
  properties are already this shape). The purity rule in the Keying
  section is not blob-specific; it governs any content-addressed
  store.

  One hazard is sharper for metadata than for thumbnails. Content-
  keying is safe when the datum is a function of bytes the caller
  already holds — a thumbnail of your own file reveals nothing new.
  A content-keyed datum that encodes *another user's input* leaks
  across the dedup boundary twice over: user B reads A's text, and
  learns that someone else holds identical content. So for k/v the
  purity rule carries an anti-enumeration duty too, alongside the
  poisoning one — cf. the dedup blob anti-enumeration policy in
  `project_d7_policy_calls`.

## References

- `docs/architecture/backend-storage.md` — the wrapper stack, header
  format, consistency check, migration semantics that derived
  content inherits.
- `docs/plan/storage-multi-entry.md` — tier-3 configuration model.
- `docs/plan/storage-key-rotation.md` — encryption/rotation applies
  to derived blobs too.
- `src/AGENTS.md` — the read-side rule enforcing backend
  abstraction, the no-hand-crafted-paths rule, and the
  persistent-state-is-backend rule this plan implements.
- `migrations/20260701000000_content_search_index.sql` — the
  content-keying precedent (`storage.blob_extracted_text`) and its
  rationale.
- Memory note `project_services_bypassing_blob_backend` — audit
  history of the pre-normalisation bypasses.