feat(storage): thumb_derived_import — backfill the derived tier from sidecars

First half of step 10. Every server-rendered thumbnail written before
content_derived_blobs existed lives only as
{thumbnails_root}/{size}/{hash}.webp — local-disk state that another
instance cannot see, a backend migration does not carry, and no
consistency job covers. This walks those files into the blob store and
records the mapping, so the derived tier can become authoritative and
the sidecar can be deleted.

A registered JobRegistry tenant rather than a script: the volume is
unbounded, so it needs a cursor, resume, cooperative cancel and run
history, and an operator needs somewhere to watch it. Cursor is
{size_dir}/{filename} over a sorted walk, which totally orders the
traversal.

Idempotent by construction — each file is skipped when a row already
exists, and store_derived_blob is ON CONFLICT DO NOTHING with
release-on-conflict beneath it, so re-runs cannot inflate refcounts.
Re-running is the expected operator behaviour, since Phase 3 (deleting
the sidecars) is gated on a run reporting zero imported.

hash_from_sidecar_name deliberately rejects ext-{file_id}.jpg. Those
bytes are user-supplied and file-keyed; importing them here would
content-key them and share one user's uploaded preview onto every file
with identical content. They belong to thumb_attached_import. Both the
accept and the reject set are under test.

Unreadable files and store failures record a finding and continue: a
sidecar removed by a concurrent GC unlink between listing and read is
expected, not fatal, and the file is left in place for the next run.

Registered unconditionally rather than behind a flag — a migration
nobody can find is a migration nobody runs.
This commit is contained in:
Edouard Vanbelle
2026-08-25 23:57:16 +02:00
parent 95648f2fa3
commit f80a28763e
3 changed files with 332 additions and 0 deletions
+1
View File
@@ -57,6 +57,7 @@ pub mod session_liveness_gauges;
pub mod share_unlock_cookie;
pub mod smtp_email_sender;
pub mod swappable_blob_backend;
pub mod thumb_derived_import_service;
pub mod thumbnail_service;
#[cfg(test)]
mod thumbnail_service_test;