feat(storage): thumb_derived_import — backfill the derived tier from sidecars
First half of step 10. Every server-rendered thumbnail written before
content_derived_blobs existed lives only as
{thumbnails_root}/{size}/{hash}.webp — local-disk state that another
instance cannot see, a backend migration does not carry, and no
consistency job covers. This walks those files into the blob store and
records the mapping, so the derived tier can become authoritative and
the sidecar can be deleted.
A registered JobRegistry tenant rather than a script: the volume is
unbounded, so it needs a cursor, resume, cooperative cancel and run
history, and an operator needs somewhere to watch it. Cursor is
{size_dir}/{filename} over a sorted walk, which totally orders the
traversal.
Idempotent by construction — each file is skipped when a row already
exists, and store_derived_blob is ON CONFLICT DO NOTHING with
release-on-conflict beneath it, so re-runs cannot inflate refcounts.
Re-running is the expected operator behaviour, since Phase 3 (deleting
the sidecars) is gated on a run reporting zero imported.
hash_from_sidecar_name deliberately rejects ext-{file_id}.jpg. Those
bytes are user-supplied and file-keyed; importing them here would
content-key them and share one user's uploaded preview onto every file
with identical content. They belong to thumb_attached_import. Both the
accept and the reject set are under test.
Unreadable files and store failures record a finding and continue: a
sidecar removed by a concurrent GC unlink between listing and read is
expected, not fatal, and the file is left in place for the next run.
Registered unconditionally rather than behind a flag — a migration
nobody can find is a migration nobody runs.
This commit is contained in:
@@ -1476,6 +1476,24 @@ impl AppServiceFactory {
|
||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||
.await;
|
||||
|
||||
// Step 10 migration tenant: backfills `content_derived_blobs` from
|
||||
// the on-disk thumbnail sidecars that predate it. Idempotent, so it
|
||||
// is safe to trigger repeatedly — Phase 3 (deleting the sidecars) is
|
||||
// gated on a run reporting zero imported. Registered unconditionally
|
||||
// rather than behind a flag: a migration nobody can find is a
|
||||
// migration nobody runs.
|
||||
//
|
||||
// `.thumbnails` lives under the storage path, matching
|
||||
// `ThumbnailService::new(&self.storage_path, …)` above.
|
||||
let _ = Arc::new(
|
||||
crate::infrastructure::services::thumb_derived_import_service::ThumbDerivedImport::new(
|
||||
std::path::Path::new(&self.storage_path).join(".thumbnails"),
|
||||
core.dedup_service.clone(),
|
||||
),
|
||||
)
|
||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||
.await;
|
||||
|
||||
// Third recoverable-run tenant. Iterates `storage.files`
|
||||
// and reports parent-folder-trashed cascade misses,
|
||||
// `missing_blob` (data-loss indicator — file references
|
||||
|
||||
Reference in New Issue
Block a user