Edouard Vanbelle
577ecb7cef
A migration nobody triggers never finishes. Scheduled ticks deliberately never pass `repair`, so a deployment whose operator never opens the admin panel re-imported the same sidecars forever and never drained the directory — and relying on operators to edit `.env` has the same failure mode one level up. `OXICLOUD_STARTUP_JOBS` dispatches named jobs once, in the background, after the scheduler is ready. Entries use the syntax operators already type at the trigger URL (`name?repair=true`), so the value is literally the request they would otherwise make by hand. It defaults to both migration jobs in repair mode, so an untouched deployment migrates and drains itself. That is a destructive default and a real exception to no-silent-auto-repair, so the guard it rests on had to get stronger: `verify_and_unlink` now compares CONTENT, not length. A blob of the right size and the wrong bytes used to pass — a key-mapping bug handing back another file's preview at the same length would have deleted the original and kept the impostor, and thumbnails cluster tightly enough in size for that to be a real coincidence. The readback streams from the backend with no cache in front, so it proves durability rather than that a write was acknowledged. Deletion of `.thumbnails/` is attempted first and only falls back to renaming it `.thumbnails.migrated` when `remove_dir` refuses because a non-sidecar file is inside (Finder's `.DS_Store`). Either way the directory stops existing, which lets the read-path probe go back to a single `stat` on the root instead of walking the size directories. Validation is fail-fast: an unknown job name or flag panics at boot. A silently dropped `?repare=true` would leave the job in discovery-only mode while the operator believed the tier was draining, surfacing months later as "the migration never finished" with nothing pointing at the config line. Interrupted runs resume. Boot recovery flips abandoned rows to Paused with their cursor, so `run_or_resume` continues rather than rescanning — a long migration completes across however many restarts it takes. That is a scoped exception to "we do not auto-resume": here somebody did ask, in configuration, and not having to ask again is the point. `StartupJob` holds a `JobRunArgs` rather than re-listing its four fields, so a fifth flag cannot be added to the scheduler and silently ignored in configuration. Jobs named here are ordinary registered jobs — visible in the panel, triggerable by hand, same runs and findings. Their rows now carry a `startup` object so an operator can see that a job deletes on every boot rather than only when someone clicks Run. Adds docs/config/thumbnail-migration.md: what runs on first boot, how to snapshot database and storage together beforehand, and how to verify afterwards with satellites_consistency plus backend_consistency ?deep=true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5.3 KiB
Thumbnail migration runbook
Thumbnails used to live as files under {STORAGE_PATH}/.thumbnails/.
They now live in the content-addressed blob store, alongside file
content. This page is for operators upgrading across that change.
You do not have to do anything. The migration runs itself, in the background, on the first boot after the upgrade. The rest of this page is for operators who want to verify it, take a safety net first, or understand what it did.
What runs, and when
Two background jobs, dispatched once at startup and daily thereafter:
| Job | Migrates | Regenerable if lost? |
|---|---|---|
thumb_derived_import |
Thumbnails the server rendered from file content | Yes — the next request re-renders |
thumb_attached_import |
Previews a client uploaded (ext-{file_id}.jpg) |
No — there is no render path for these |
Both import each sidecar into blob storage, read it back to confirm the
copy is byte-identical, and only then delete the original. When the
directory is empty it is removed, and .thumbnails/ stops existing.
Startup dispatch is non-blocking — the server is ready immediately and the migration proceeds behind it. A run interrupted by a restart resumes from where it stopped, so a large installation finishes over several restarts rather than starting again each time.
This is controlled by OXICLOUD_STARTUP_JOBS, which defaults to:
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true
To import without deleting — migrate now, inspect, delete later:
OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import
The sidecars then stay on disk. Trigger the deletion when you are ready from Admin → Jobs, using each job's Repair action.
To disable startup jobs entirely, set the variable to an empty value.
Taking a safety net first
Recommended for any installation where the uploaded previews matter, and cheap enough to be worth it regardless. Both parts must be captured together — a database that references blobs a storage snapshot predates is worse than neither.
1. Stop the server. A snapshot taken while writes are in flight can catch a blob that exists on disk without its database row, or the reverse.
systemctl stop oxicloud # or: docker compose stop oxicloud
2. Snapshot the database.
pg_dump --format=custom --file=oxicloud-preflight.dump "$DATABASE_URL"
Use --format=custom; restoring it needs pg_restore --disable-triggers,
because the folder table carries a self-referencing foreign key that a
plain SQL restore cannot order correctly.
3. Snapshot the storage directory. At minimum .thumbnails/, which
is what the migration touches:
tar -czf oxicloud-thumbnails-preflight.tar.gz -C "$STORAGE_PATH" .thumbnails
A whole-directory snapshot is better if you have the space — filesystem or volume snapshots (ZFS, LVM, EBS) are ideal, since they are atomic and near-instant:
zfs snapshot tank/oxicloud@preflight
4. Start the server. The migration begins in the background.
Keep both snapshots until you have run the verification below and are satisfied.
Verifying the migration
Two checks, both from Admin → Jobs or the API. Run them after the migration reports no remaining work.
1. Every mapping points at a blob that exists. Run
satellites_consistency. It walks both thumbnail tables and reports any
row whose blob or source is gone. A clean run means nothing was lost in
the bookkeeping.
POST /api/admin/jobs/satellites_consistency/trigger
2. Every blob still hashes to what it claims. Run
backend_consistency with ?deep=true. It reads every blob back from
storage and re-hashes it, which covers the migrated thumbnails along
with everything else. This is a full read of your storage and can take
hours on a large installation — schedule it accordingly.
POST /api/admin/jobs/backend_consistency/trigger?deep=true
A clean pass on both means the thumbnails are readable, correctly referenced, and byte-intact in their new home. At that point the snapshots can be discarded.
Checking it finished
.thumbnails/ is gone. That is the whole test:
ls -d "$STORAGE_PATH/.thumbnails" # No such file or directory
If you instead find .thumbnails.migrated/, the migration completed but
could not remove the directory, because something that is not a
thumbnail was inside it — a .DS_Store from macOS Finder is the usual
culprit. The tree was moved aside instead of deleted. Its contents are
no longer used and it is safe to remove by hand once you have looked at
what is in there.
While either directory is absent, the server skips the legacy read path
entirely, at no cost. While .thumbnails/ is present, reads fall back
to it on a miss, which is what makes the migration invisible to users
while it runs.
If something looks wrong
Every deletion is written to the audit log, naming the job, the file removed and the blob that replaced it. To review what a migration removed:
journalctl -u oxicloud | grep sidecar_deleted
A sidecar is only ever deleted after its replacement has been read back and compared byte-for-byte, so a file that failed that check is still on disk. Those show up as findings on the job's run in Admin → Jobs, with the reason recorded per file.