feat(jobs): run the thumbnail migration at startup, by default

A migration nobody triggers never finishes. Scheduled ticks deliberately
never pass `repair`, so a deployment whose operator never opens the
admin panel re-imported the same sidecars forever and never drained the
directory — and relying on operators to edit `.env` has the same failure
mode one level up.

`OXICLOUD_STARTUP_JOBS` dispatches named jobs once, in the background,
after the scheduler is ready. Entries use the syntax operators already
type at the trigger URL (`name?repair=true`), so the value is literally
the request they would otherwise make by hand. It defaults to both
migration jobs in repair mode, so an untouched deployment migrates and
drains itself.

That is a destructive default and a real exception to
no-silent-auto-repair, so the guard it rests on had to get stronger:
`verify_and_unlink` now compares CONTENT, not length. A blob of the
right size and the wrong bytes used to pass — a key-mapping bug handing
back another file's preview at the same length would have deleted the
original and kept the impostor, and thumbnails cluster tightly enough in
size for that to be a real coincidence. The readback streams from the
backend with no cache in front, so it proves durability rather than that
a write was acknowledged.

Deletion of `.thumbnails/` is attempted first and only falls back to
renaming it `.thumbnails.migrated` when `remove_dir` refuses because a
non-sidecar file is inside (Finder's `.DS_Store`). Either way the
directory stops existing, which lets the read-path probe go back to a
single `stat` on the root instead of walking the size directories.

Validation is fail-fast: an unknown job name or flag panics at boot. A
silently dropped `?repare=true` would leave the job in discovery-only
mode while the operator believed the tier was draining, surfacing months
later as "the migration never finished" with nothing pointing at the
config line.

Interrupted runs resume. Boot recovery flips abandoned rows to Paused
with their cursor, so `run_or_resume` continues rather than rescanning —
a long migration completes across however many restarts it takes. That
is a scoped exception to "we do not auto-resume": here somebody did ask,
in configuration, and not having to ask again is the point.

`StartupJob` holds a `JobRunArgs` rather than re-listing its four
fields, so a fifth flag cannot be added to the scheduler and silently
ignored in configuration.

Jobs named here are ordinary registered jobs — visible in the panel,
triggerable by hand, same runs and findings. Their rows now carry a
`startup` object so an operator can see that a job deletes on every boot
rather than only when someone clicks Run.

Adds docs/config/thumbnail-migration.md: what runs on first boot, how to
snapshot database and storage together beforehand, and how to verify
afterwards with satellites_consistency plus backend_consistency
?deep=true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-08-29 10:23:02 +02:00
parent 03246305f6
commit 577ecb7cef
15 changed files with 829 additions and 32 deletions
+92
View File
@@ -737,6 +737,98 @@ trigger) resumes any `Paused` row per the normal flow.
Consistency-check.md's existing consistency-scoped sweep collapses
into this general one.
### Startup jobs — `OXICLOUD_STARTUP_JOBS`
A comma-separated list of jobs to dispatch once, in the background,
after the scheduler is ready. Each entry is a registered job name,
optionally with the same query syntax the admin trigger URL uses.
**The default is both migration jobs, in repair mode:**
```
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true
```
An explicit value replaces that list; an empty value disables startup
jobs entirely.
**Why it exists.** Scheduled ticks deliberately never pass `repair` — a
job that deletes on its default setting is what no-silent-auto-repair
forbids. But that left the migration jobs unable to finish on their
own: a deployment whose operator never opens the admin panel re-imports
sidecars it already imported, forever, and never drains the directory.
**Why the default deletes anyway.** Relying on operators to edit `.env`
has the same failure mode one level up — the ones who never edit it are
exactly the ones whose migration never completes. So this is a
deliberate exception to no-silent-auto-repair, and it rests on three
properties that must keep holding:
- **Nothing is deleted before its replacement has been read back.**
`verify_and_unlink` imports, reads the blob back through the normal
stack, and only then unlinks; a store that reported success but landed
unreadable keeps its sidecar. This matters most for
`thumb_attached_import`, whose bytes are user-uploaded previews with
no render path — a wrong deletion there is permanent, where a wrong
deletion of a server-rendered thumbnail costs a re-render.
- **Sidecars whose source is gone are deleted without a readback**,
because there is nothing to read back and nothing can reference them
again. Unrecoverable and unreachable are different things; these are
both.
- **Every deletion is audited**, so what a boot removed, and from which
source, is reconstructable afterwards.
The consequence to hold in mind: an upgrade deletes on first boot, in
every deployment at once, with no operator action. A regression in the
readback path would be simultaneous and unrecoverable, so that code is
load-bearing. Operators who want to inspect before committing set
`OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import` —
same jobs, import only.
It is not a "run everything in repair mode" switch. Each job is named
individually and carries its own flags.
**Validation is fail-fast.** An unknown job name panics at boot — the
registry is fully populated by then, so a name that doesn't resolve is a
typo or a stale rename, and ignoring it would leave a migration that
silently never runs. Unknown flags panic too: a dropped `?repare=true`
would leave the job in discovery-only mode while the operator believed
the tier was draining, and the symptom ("it never finished") surfaces
months later with nothing pointing back at the config.
**Dispatch is non-blocking.** `tokio::spawn`, so readiness never waits
on a job that may walk a filesystem for hours. Jobs in the list run
sequentially within that task, not concurrently: they contend for the
same directories and pool, and the exclusivity gate would turn overlap
into a *skipped* run rather than a queued one.
**Interrupted runs resume.** The boot recovery sweep above runs first
and flips every abandoned `Running` row to `Paused` with its cursor
intact; `run_or_resume` then picks Resume over a fresh start. So a
migration killed by a restart continues where it stopped, and completes
across however many restarts it takes.
That is a deliberate exception to "do NOT auto-resume" — scoped to the
named jobs only. The rule protects against a restart silently resuming
work nobody asked for; here somebody did ask, in configuration, and not
having to ask again is the entire point. Every other paused run still
waits for an operator.
A resumed run keeps the flags it started with (`repair` / `deep` are
persisted to `params` on the fresh open and read back on resume), so
editing the config mid-migration does not retroactively change a run
already in flight.
**Safe to leave set.** Each job is idempotent and resumable; once the
tier has drained, a run is a `read_dir` over three directories that
returns nothing — and after the directory is removed, not even that.
**Visible in the admin panel.** These are ordinary registered jobs:
they appear in `GET /api/admin/jobs`, are triggerable by hand, and
record the same runs and findings. Rows named here additionally carry a
`startup` object with the configured flags, so an operator can see that
a job deletes files on every boot rather than only when someone clicks.
### Admin surface (recoverable runs)
Same URL taxonomy as Part 1 — resource-first, action second, all