feat(jobs): run the thumbnail migration at startup, by default

A migration nobody triggers never finishes. Scheduled ticks deliberately
never pass `repair`, so a deployment whose operator never opens the
admin panel re-imported the same sidecars forever and never drained the
directory — and relying on operators to edit `.env` has the same failure
mode one level up.

`OXICLOUD_STARTUP_JOBS` dispatches named jobs once, in the background,
after the scheduler is ready. Entries use the syntax operators already
type at the trigger URL (`name?repair=true`), so the value is literally
the request they would otherwise make by hand. It defaults to both
migration jobs in repair mode, so an untouched deployment migrates and
drains itself.

That is a destructive default and a real exception to
no-silent-auto-repair, so the guard it rests on had to get stronger:
`verify_and_unlink` now compares CONTENT, not length. A blob of the
right size and the wrong bytes used to pass — a key-mapping bug handing
back another file's preview at the same length would have deleted the
original and kept the impostor, and thumbnails cluster tightly enough in
size for that to be a real coincidence. The readback streams from the
backend with no cache in front, so it proves durability rather than that
a write was acknowledged.

Deletion of `.thumbnails/` is attempted first and only falls back to
renaming it `.thumbnails.migrated` when `remove_dir` refuses because a
non-sidecar file is inside (Finder's `.DS_Store`). Either way the
directory stops existing, which lets the read-path probe go back to a
single `stat` on the root instead of walking the size directories.

Validation is fail-fast: an unknown job name or flag panics at boot. A
silently dropped `?repare=true` would leave the job in discovery-only
mode while the operator believed the tier was draining, surfacing months
later as "the migration never finished" with nothing pointing at the
config line.

Interrupted runs resume. Boot recovery flips abandoned rows to Paused
with their cursor, so `run_or_resume` continues rather than rescanning —
a long migration completes across however many restarts it takes. That
is a scoped exception to "we do not auto-resume": here somebody did ask,
in configuration, and not having to ask again is the point.

`StartupJob` holds a `JobRunArgs` rather than re-listing its four
fields, so a fifth flag cannot be added to the scheduler and silently
ignored in configuration.

Jobs named here are ordinary registered jobs — visible in the panel,
triggerable by hand, same runs and findings. Their rows now carry a
`startup` object so an operator can see that a job deletes on every boot
rather than only when someone clicks Run.

Adds docs/config/thumbnail-migration.md: what runs on first boot, how to
snapshot database and storage together beforehand, and how to verify
afterwards with satellites_consistency plus backend_consistency
?deep=true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/infrastructure/scheduler/mod.rs

@@ -37,5 +37,7 @@ pub use recoverable::{
     RecoverableJobHandler, RunOutcome, RunProgress, RunStatus, RunSummary, derive_progress,
     record_or_log, run_or_resume,
 };
-pub use registry::{JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError};
+pub use registry::{
+    JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError, StartupTrigger,
+};
 pub use types::{ErrCause, JobOutcome, JobRunArgs, Mutates};

src/infrastructure/scheduler/registry.rs

@@ -236,11 +236,12 @@ impl JobRegistry {
                     last_outcome,
                     running: state.current_run_start.is_some(),
                     recoverable: entry.handler.is_recoverable(),
-                    // Populated in `list_jobs` handler via a single
-                    // DB round-trip — kept out of the registry
-                    // snapshot to avoid pulling a DB dependency into
-                    // the in-memory scheduler state.
+                    // Both populated in the `list_jobs` handler — one
+                    // from a DB round-trip, one from AppConfig. Kept
+                    // out of the registry snapshot so the in-memory
+                    // scheduler state pulls in neither dependency.
                     paused_run: None,
+                    startup: None,
                 }
             })
             .collect()
@@ -346,6 +347,32 @@ pub struct JobSummary {
     /// picks Resume when the latest row is Paused).
     #[serde(skip_serializing_if = "Option::is_none")]
     pub paused_run: Option<PausedRunBrief>,
+    /// Populated iff `OXICLOUD_STARTUP_JOBS` names this job — the flags
+    /// it will be dispatched with at every boot.
+    ///
+    /// Surfaced because the panel would otherwise be silently wrong
+    /// about the most consequential thing on the row: a job configured
+    /// with `repair=true` deletes files on every restart, and reading
+    /// the row you would think that only happens when someone clicks.
+    /// Filled by the `list_jobs` handler, which has the config; the
+    /// registry deliberately doesn't.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub startup: Option<StartupTrigger>,
+}
+
+/// The flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with.
+///
+/// Mirrors `JobRunArgs` on the wire rather than embedding it, because
+/// this is an API shape the admin panel switches on, and `JobRunArgs`
+/// is an internal dispatch type free to change without a frontend
+/// release.
+#[derive(Debug, Clone, Serialize)]
+pub struct StartupTrigger {
+    pub force: bool,
+    pub deep: bool,
+    pub repair: bool,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub storage: Option<String>,
 }
 
 /// Enough info about a paused recoverable run for the admin panel to

src/infrastructure/services/thumb_derived_import_service.rs

@@ -48,6 +48,15 @@ use crate::infrastructure::services::dedup_service::DedupService;
 
 pub const THUMB_DERIVED_IMPORT_JOB_NAME: &str = "thumb_derived_import";
 
+/// Where the legacy tree is moved when it cannot be deleted.
+///
+/// Deletion is always attempted first — this is the fallback for the one
+/// case `remove_dir` refuses: a file that is not a sidecar sitting in the
+/// directory (Finder's `.DS_Store`, most often). What matters to the read
+/// path is that `.thumbnails` stops existing, so moving the tree aside
+/// achieves the same thing while preserving whatever the stray file was.
+pub(crate) const PARKED_DIR_NAME: &str = ".thumbnails.migrated";
+
 /// Record a sidecar deletion on the audit channel.
 ///
 /// Both import jobs delete user-visible files during a one-way migration, so
@@ -182,13 +191,36 @@ impl ThumbDerivedImport {
         stored_hash: &str,
         path: &std::path::Path,
     ) -> bool {
-        let Ok(meta) = fs::metadata(path).await else {
+        // Compare CONTENT, not length.
+        //
+        // This is the only thing standing between a storage bug and
+        // permanent loss — `thumb_attached_import` deletes user-uploaded
+        // previews that have no render path to rebuild them, and with the
+        // startup-job default it does so on first boot after an upgrade,
+        // in every deployment at once. A guard that load-bearing should
+        // prove the bytes are the bytes.
+        //
+        // Length alone did not. A blob of the right size and the wrong
+        // content passed: a key-mapping bug handing back another file's
+        // preview at the same length would have deleted the original and
+        // kept the impostor, and thumbnails cluster tightly enough in size
+        // for that to be a real coincidence rather than a theoretical one.
+        //
+        // Re-reading the sidecar costs a few KB of I/O, once per file ever
+        // migrated. The import path already has these bytes in hand, but
+        // taking them as an argument would leave the already-imported path
+        // (which has no bytes, only a file) on a weaker check — one code
+        // path, one guarantee.
+        let Ok(sidecar) = fs::read(path).await else {
             return false;
         };
+        // `read_blob_bytes` streams from the backend, reassembling chunks
+        // if the blob is chunked — no cache sits in front of it, so this
+        // proves durability and not merely that a write was acknowledged.
         let Ok(stored) = dedup.read_blob_bytes(stored_hash).await else {
             return false;
         };
-        if stored.is_empty() || stored.len() as u64 != meta.len() {
+        if stored.is_empty() || stored.as_ref() != sidecar.as_slice() {
             return false;
         }
         if fs::remove_file(path).await.is_err() {
@@ -592,16 +624,47 @@ impl RecoverableJobHandler for ThumbDerivedImport {
                     "🧹 legacy sidecar directory removed — the fallback read path \
                      is inert from the next restart"
                 ),
-                Err(e) => tracing::info!(
-                    target: "oxicloud::dedup",
-                    event = "thumb_derived_import.root_kept",
-                    run_id = %store.run_id(),
-                    path = %self.thumbnails_root.display(),
-                    reason = %e,
-                    "legacy sidecar directory not removed; if this says \
-                     'directory not empty' with no sidecars left, something \
-                     else put a file there (a .DS_Store, typically)"
-                ),
+                // Something unrelated to thumbnails is in the directory, so
+                // `remove_dir` refuses. On macOS that is Finder's `.DS_Store`,
+                // and it would otherwise keep the fallback alive forever on
+                // every developer machine.
+                //
+                // Move the whole tree aside instead. The sidecars are already
+                // imported and verified, so nothing here is load-bearing; what
+                // matters is that `.thumbnails` stops existing, because its
+                // absence is what the read path tests. Renaming preserves the
+                // stray file for whoever put it there, and keeps the check a
+                // single `stat` rather than a directory walk.
+                Err(_) => {
+                    // `with_file_name`, NOT `with_extension`: the directory is
+                    // `.thumbnails`, and a leading-dot name has no extension as
+                    // far as `Path` is concerned — its whole name is the stem.
+                    // `with_extension("thumbnails.migrated")` would have
+                    // produced `.thumbnails.thumbnails.migrated`.
+                    let parked = self.thumbnails_root.with_file_name(PARKED_DIR_NAME);
+                    match fs::rename(&self.thumbnails_root, &parked).await {
+                        Ok(()) => tracing::info!(
+                            target: "oxicloud::dedup",
+                            event = "thumb_derived_import.root_parked",
+                            run_id = %store.run_id(),
+                            from = %self.thumbnails_root.display(),
+                            to = %parked.display(),
+                            "🧹 legacy sidecar directory could not be removed (a \
+                             non-sidecar file remains) — moved aside instead. The \
+                             fallback read path is inert from the next restart; \
+                             the directory is safe to delete by hand."
+                        ),
+                        Err(e) => tracing::warn!(
+                            target: "oxicloud::dedup",
+                            event = "thumb_derived_import.root_kept",
+                            run_id = %store.run_id(),
+                            path = %self.thumbnails_root.display(),
+                            reason = %e,
+                            "legacy sidecar directory could be neither removed nor \
+                             moved aside — the fallback read path stays live"
+                        ),
+                    }
+                }
             }
         }
 
@@ -646,6 +709,24 @@ pub(crate) mod tests {
     /// A second hash, for the JPEG sidecar in `legacy_tree`.
     const H2: &str = "c222222222222222222222222222222222222222222222222222222222222222";
 
+    /// The park path must be a SIBLING of `.thumbnails`, not a suffixed
+    /// child of its name.
+    ///
+    /// `Path::with_extension` looks right and is wrong here: a leading-dot
+    /// name has no extension as far as `Path` is concerned — `.thumbnails`
+    /// is entirely stem — so `with_extension("thumbnails.migrated")`
+    /// yields `.thumbnails.thumbnails.migrated`. The rename would still
+    /// have "worked", leaving a directory nobody documented and an
+    /// operator hunting for the name the runbook promised.
+    #[test]
+    fn parked_directory_is_a_sibling_named_thumbnails_migrated() {
+        let root = std::path::Path::new("/srv/storage/.thumbnails");
+        assert_eq!(
+            root.with_file_name(PARKED_DIR_NAME),
+            std::path::Path::new("/srv/storage/.thumbnails.migrated"),
+        );
+    }
+
     /// BOTH codecs are claimed, and the format comes from the extension.
     ///
     /// `.jpg` was previously rejected here, which was correct only while the

src/infrastructure/services/thumbnail_service.rs

@@ -263,27 +263,19 @@ impl ThumbnailService {
     /// written a sidecar since step 10d2, so there is nothing to create them
     /// for.
     ///
-    /// The probe tests the SIZE directories, not the root. On macOS Finder
-    /// drops a `.DS_Store` in the root, which blocks `remove_dir` there
-    /// forever — gating on the root would keep the fallback alive on every
-    /// developer machine for a reason that has nothing to do with thumbnails.
-    /// If no size directory exists, no sidecar can exist.
+    /// One `stat` on the root. The import job guarantees that is enough: it
+    /// removes the directory once drained, and when something unrelated
+    /// keeps `remove_dir` from succeeding — Finder's `.DS_Store`, typically
+    /// — it renames the tree to `.thumbnails.migrated` rather than leaving
+    /// it in place. So `.thumbnails` existing always means "there may be
+    /// sidecars under here", and a stray file cannot pin the fallback open.
     ///
     /// Result is cached for the process lifetime. It can only be stale in the
     /// harmless direction: a drain completing mid-life leaves the flag `true`
     /// until restart, which costs the same failed opens as today. It never
     /// goes `false` while sidecars remain.
     pub async fn initialize(&self) -> std::io::Result<()> {
-        let mut present = false;
-        for size in ThumbnailSize::all() {
-            if fs::metadata(self.thumbnails_root.join(size.dir_name()))
-                .await
-                .is_ok()
-            {
-                present = true;
-                break;
-            }
-        }
+        let present = fs::metadata(&self.thumbnails_root).await.is_ok();
         self.legacy_sidecars.store(present, Ordering::Relaxed);
 
         // Asymmetric on purpose. "Present" is actionable and temporary — it