feat(jobs): run the thumbnail migration at startup, by default
A migration nobody triggers never finishes. Scheduled ticks deliberately never pass `repair`, so a deployment whose operator never opens the admin panel re-imported the same sidecars forever and never drained the directory — and relying on operators to edit `.env` has the same failure mode one level up. `OXICLOUD_STARTUP_JOBS` dispatches named jobs once, in the background, after the scheduler is ready. Entries use the syntax operators already type at the trigger URL (`name?repair=true`), so the value is literally the request they would otherwise make by hand. It defaults to both migration jobs in repair mode, so an untouched deployment migrates and drains itself. That is a destructive default and a real exception to no-silent-auto-repair, so the guard it rests on had to get stronger: `verify_and_unlink` now compares CONTENT, not length. A blob of the right size and the wrong bytes used to pass — a key-mapping bug handing back another file's preview at the same length would have deleted the original and kept the impostor, and thumbnails cluster tightly enough in size for that to be a real coincidence. The readback streams from the backend with no cache in front, so it proves durability rather than that a write was acknowledged. Deletion of `.thumbnails/` is attempted first and only falls back to renaming it `.thumbnails.migrated` when `remove_dir` refuses because a non-sidecar file is inside (Finder's `.DS_Store`). Either way the directory stops existing, which lets the read-path probe go back to a single `stat` on the root instead of walking the size directories. Validation is fail-fast: an unknown job name or flag panics at boot. A silently dropped `?repare=true` would leave the job in discovery-only mode while the operator believed the tier was draining, surfacing months later as "the migration never finished" with nothing pointing at the config line. Interrupted runs resume. Boot recovery flips abandoned rows to Paused with their cursor, so `run_or_resume` continues rather than rescanning — a long migration completes across however many restarts it takes. That is a scoped exception to "we do not auto-resume": here somebody did ask, in configuration, and not having to ask again is the point. `StartupJob` holds a `JobRunArgs` rather than re-listing its four fields, so a fifth flag cannot be added to the scheduler and silently ignored in configuration. Jobs named here are ordinary registered jobs — visible in the panel, triggerable by hand, same runs and findings. Their rows now carry a `startup` object so an operator can see that a job deletes on every boot rather than only when someone clicks Run. Adds docs/config/thumbnail-migration.md: what runs on first boot, how to snapshot database and storage together beforehand, and how to verify afterwards with satellites_consistency plus backend_consistency ?deep=true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
@@ -18,6 +18,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
|
||||
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Direct (non-chunked) uploads stream straight into the blob store and need no spool directory. Placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
||||
| `OXICLOUD_METRICS_LISTEN` | (unset) | Prometheus `/metrics` listener address (e.g. `127.0.0.1:9090`, IPv6 allowed as `[::1]:9090`). **Unset = disabled**: no `/metrics` endpoint is bound and no metrics recorder is installed (zero runtime cost). When set, a separate HTTP listener on this address serves the text-format scrape. **Deliberately NOT merged into the main API** — no auth, CSRF, or DPoP layer in front. Bind to loopback or a private interface unless you intend to expose metrics publicly. Starter counters: `oxicloud_dpop_verify_failed_total{reason}`, `oxicloud_dpop_proof_missing_total`, `oxicloud_dpop_header_missing_on_bound_session_total`, `oxicloud_dpop_replay_detected_total`, `oxicloud_dpop_nonce_challenges_issued_total`. |
|
||||
| `OXICLOUD_STARTUP_JOBS` | `thumb_derived_import?repair=true,thumb_attached_import?repair=true` | Background jobs dispatched once at boot, comma-separated, each `name` or `name?flag=true` using the same syntax as `POST /api/admin/jobs/{name}/trigger`. Flags: `force`, `deep`, `repair`, `storage`. **The default migrates thumbnails out of the legacy `.thumbnails/` directory and deletes the originals**, so the migration completes without anyone triggering it from the admin panel; each sidecar is read back through the normal stack before it is unlinked, and every deletion is audited. An explicit value **replaces** the default; set it empty (`OXICLOUD_STARTUP_JOBS=`) to disable startup jobs, or to `thumb_derived_import,thumb_attached_import` to import without deleting. **Non-blocking** — readiness never waits on a job; entries run sequentially in the background. **Fail-fast** — an unknown job name or flag panics at boot, because a silently-dropped entry means a migration that never runs. A run interrupted by a restart resumes from its cursor on the next boot, so a long migration finishes across restarts. Safe to leave at the default: the jobs are idempotent, and once drained a run does nothing. See [Thumbnail Migration](./thumbnail-migration.md) for the upgrade runbook. |
|
||||
|
||||
## Database
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ OxiCloud is configured entirely via **environment variables** (no config files n
|
||||
- [Deployment & Docker](/config/deployment) — Docker Compose, Kubernetes Helm chart, image details
|
||||
- [Environment Variables](/config/env) — complete reference of all `OXICLOUD_*` variables
|
||||
- [Storage Fine Tuning](/config/storage-fine-tuning) — sizing the upload caps + spool directories; tmpfs vs real disk; NVMe split layouts
|
||||
- [Thumbnail Migration](/config/thumbnail-migration) — upgrading past `.thumbnails/`: what runs on first boot, taking a snapshot first, verifying afterwards
|
||||
- [Authentication](/config/authentication) — JWT auth, login, refresh, password changes, and auth status
|
||||
- [OIDC / SSO](/config/oidc) — single sign-on with Keycloak, Authentik, Authelia, Google, Azure AD
|
||||
- [WOPI (Office Editing)](/config/wopi) — Collabora Online / OnlyOffice integration
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
# Thumbnail migration runbook
|
||||
|
||||
Thumbnails used to live as files under `{STORAGE_PATH}/.thumbnails/`.
|
||||
They now live in the content-addressed blob store, alongside file
|
||||
content. This page is for operators upgrading across that change.
|
||||
|
||||
**You do not have to do anything.** The migration runs itself, in the
|
||||
background, on the first boot after the upgrade. The rest of this page
|
||||
is for operators who want to verify it, take a safety net first, or
|
||||
understand what it did.
|
||||
|
||||
## What runs, and when
|
||||
|
||||
Two background jobs, dispatched once at startup and daily thereafter:
|
||||
|
||||
| Job | Migrates | Regenerable if lost? |
|
||||
|---|---|---|
|
||||
| `thumb_derived_import` | Thumbnails the server rendered from file content | Yes — the next request re-renders |
|
||||
| `thumb_attached_import` | Previews a client uploaded (`ext-{file_id}.jpg`) | **No** — there is no render path for these |
|
||||
|
||||
Both import each sidecar into blob storage, read it back to confirm the
|
||||
copy is byte-identical, and only then delete the original. When the
|
||||
directory is empty it is removed, and `.thumbnails/` stops existing.
|
||||
|
||||
Startup dispatch is non-blocking — the server is ready immediately and
|
||||
the migration proceeds behind it. A run interrupted by a restart resumes
|
||||
from where it stopped, so a large installation finishes over several
|
||||
restarts rather than starting again each time.
|
||||
|
||||
This is controlled by `OXICLOUD_STARTUP_JOBS`, which defaults to:
|
||||
|
||||
```
|
||||
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true
|
||||
```
|
||||
|
||||
To **import without deleting** — migrate now, inspect, delete later:
|
||||
|
||||
```
|
||||
OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import
|
||||
```
|
||||
|
||||
The sidecars then stay on disk. Trigger the deletion when you are ready
|
||||
from **Admin → Jobs**, using each job's Repair action.
|
||||
|
||||
To disable startup jobs entirely, set the variable to an empty value.
|
||||
|
||||
## Taking a safety net first
|
||||
|
||||
Recommended for any installation where the uploaded previews matter, and
|
||||
cheap enough to be worth it regardless. Both parts must be captured
|
||||
together — a database that references blobs a storage snapshot predates
|
||||
is worse than neither.
|
||||
|
||||
**1. Stop the server.** A snapshot taken while writes are in flight can
|
||||
catch a blob that exists on disk without its database row, or the
|
||||
reverse.
|
||||
|
||||
```bash
|
||||
systemctl stop oxicloud # or: docker compose stop oxicloud
|
||||
```
|
||||
|
||||
**2. Snapshot the database.**
|
||||
|
||||
```bash
|
||||
pg_dump --format=custom --file=oxicloud-preflight.dump "$DATABASE_URL"
|
||||
```
|
||||
|
||||
Use `--format=custom`; restoring it needs `pg_restore --disable-triggers`,
|
||||
because the folder table carries a self-referencing foreign key that a
|
||||
plain SQL restore cannot order correctly.
|
||||
|
||||
**3. Snapshot the storage directory.** At minimum `.thumbnails/`, which
|
||||
is what the migration touches:
|
||||
|
||||
```bash
|
||||
tar -czf oxicloud-thumbnails-preflight.tar.gz -C "$STORAGE_PATH" .thumbnails
|
||||
```
|
||||
|
||||
A whole-directory snapshot is better if you have the space — filesystem
|
||||
or volume snapshots (ZFS, LVM, EBS) are ideal, since they are atomic and
|
||||
near-instant:
|
||||
|
||||
```bash
|
||||
zfs snapshot tank/oxicloud@preflight
|
||||
```
|
||||
|
||||
**4. Start the server.** The migration begins in the background.
|
||||
|
||||
Keep both snapshots until you have run the verification below and are
|
||||
satisfied.
|
||||
|
||||
## Verifying the migration
|
||||
|
||||
Two checks, both from **Admin → Jobs** or the API. Run them after the
|
||||
migration reports no remaining work.
|
||||
|
||||
**1. Every mapping points at a blob that exists.** Run
|
||||
`satellites_consistency`. It walks both thumbnail tables and reports any
|
||||
row whose blob or source is gone. A clean run means nothing was lost in
|
||||
the bookkeeping.
|
||||
|
||||
```
|
||||
POST /api/admin/jobs/satellites_consistency/trigger
|
||||
```
|
||||
|
||||
**2. Every blob still hashes to what it claims.** Run
|
||||
`backend_consistency` with `?deep=true`. It reads every blob back from
|
||||
storage and re-hashes it, which covers the migrated thumbnails along
|
||||
with everything else. This is a full read of your storage and can take
|
||||
hours on a large installation — schedule it accordingly.
|
||||
|
||||
```
|
||||
POST /api/admin/jobs/backend_consistency/trigger?deep=true
|
||||
```
|
||||
|
||||
A clean pass on both means the thumbnails are readable, correctly
|
||||
referenced, and byte-intact in their new home. At that point the
|
||||
snapshots can be discarded.
|
||||
|
||||
## Checking it finished
|
||||
|
||||
`.thumbnails/` is gone. That is the whole test:
|
||||
|
||||
```bash
|
||||
ls -d "$STORAGE_PATH/.thumbnails" # No such file or directory
|
||||
```
|
||||
|
||||
If you instead find `.thumbnails.migrated/`, the migration completed but
|
||||
could not remove the directory, because something that is not a
|
||||
thumbnail was inside it — a `.DS_Store` from macOS Finder is the usual
|
||||
culprit. The tree was moved aside instead of deleted. Its contents are
|
||||
no longer used and it is safe to remove by hand once you have looked at
|
||||
what is in there.
|
||||
|
||||
While either directory is absent, the server skips the legacy read path
|
||||
entirely, at no cost. While `.thumbnails/` is present, reads fall back
|
||||
to it on a miss, which is what makes the migration invisible to users
|
||||
while it runs.
|
||||
|
||||
## If something looks wrong
|
||||
|
||||
Every deletion is written to the audit log, naming the job, the file
|
||||
removed and the blob that replaced it. To review what a migration
|
||||
removed:
|
||||
|
||||
```bash
|
||||
journalctl -u oxicloud | grep sidecar_deleted
|
||||
```
|
||||
|
||||
A sidecar is only ever deleted after its replacement has been read back
|
||||
and compared byte-for-byte, so a file that failed that check is still on
|
||||
disk. Those show up as findings on the job's run in **Admin → Jobs**,
|
||||
with the reason recorded per file.
|
||||
@@ -737,6 +737,98 @@ trigger) resumes any `Paused` row per the normal flow.
|
||||
Consistency-check.md's existing consistency-scoped sweep collapses
|
||||
into this general one.
|
||||
|
||||
### Startup jobs — `OXICLOUD_STARTUP_JOBS`
|
||||
|
||||
A comma-separated list of jobs to dispatch once, in the background,
|
||||
after the scheduler is ready. Each entry is a registered job name,
|
||||
optionally with the same query syntax the admin trigger URL uses.
|
||||
|
||||
**The default is both migration jobs, in repair mode:**
|
||||
|
||||
```
|
||||
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true
|
||||
```
|
||||
|
||||
An explicit value replaces that list; an empty value disables startup
|
||||
jobs entirely.
|
||||
|
||||
**Why it exists.** Scheduled ticks deliberately never pass `repair` — a
|
||||
job that deletes on its default setting is what no-silent-auto-repair
|
||||
forbids. But that left the migration jobs unable to finish on their
|
||||
own: a deployment whose operator never opens the admin panel re-imports
|
||||
sidecars it already imported, forever, and never drains the directory.
|
||||
|
||||
**Why the default deletes anyway.** Relying on operators to edit `.env`
|
||||
has the same failure mode one level up — the ones who never edit it are
|
||||
exactly the ones whose migration never completes. So this is a
|
||||
deliberate exception to no-silent-auto-repair, and it rests on three
|
||||
properties that must keep holding:
|
||||
|
||||
- **Nothing is deleted before its replacement has been read back.**
|
||||
`verify_and_unlink` imports, reads the blob back through the normal
|
||||
stack, and only then unlinks; a store that reported success but landed
|
||||
unreadable keeps its sidecar. This matters most for
|
||||
`thumb_attached_import`, whose bytes are user-uploaded previews with
|
||||
no render path — a wrong deletion there is permanent, where a wrong
|
||||
deletion of a server-rendered thumbnail costs a re-render.
|
||||
- **Sidecars whose source is gone are deleted without a readback**,
|
||||
because there is nothing to read back and nothing can reference them
|
||||
again. Unrecoverable and unreachable are different things; these are
|
||||
both.
|
||||
- **Every deletion is audited**, so what a boot removed, and from which
|
||||
source, is reconstructable afterwards.
|
||||
|
||||
The consequence to hold in mind: an upgrade deletes on first boot, in
|
||||
every deployment at once, with no operator action. A regression in the
|
||||
readback path would be simultaneous and unrecoverable, so that code is
|
||||
load-bearing. Operators who want to inspect before committing set
|
||||
`OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import` —
|
||||
same jobs, import only.
|
||||
|
||||
It is not a "run everything in repair mode" switch. Each job is named
|
||||
individually and carries its own flags.
|
||||
|
||||
**Validation is fail-fast.** An unknown job name panics at boot — the
|
||||
registry is fully populated by then, so a name that doesn't resolve is a
|
||||
typo or a stale rename, and ignoring it would leave a migration that
|
||||
silently never runs. Unknown flags panic too: a dropped `?repare=true`
|
||||
would leave the job in discovery-only mode while the operator believed
|
||||
the tier was draining, and the symptom ("it never finished") surfaces
|
||||
months later with nothing pointing back at the config.
|
||||
|
||||
**Dispatch is non-blocking.** `tokio::spawn`, so readiness never waits
|
||||
on a job that may walk a filesystem for hours. Jobs in the list run
|
||||
sequentially within that task, not concurrently: they contend for the
|
||||
same directories and pool, and the exclusivity gate would turn overlap
|
||||
into a *skipped* run rather than a queued one.
|
||||
|
||||
**Interrupted runs resume.** The boot recovery sweep above runs first
|
||||
and flips every abandoned `Running` row to `Paused` with its cursor
|
||||
intact; `run_or_resume` then picks Resume over a fresh start. So a
|
||||
migration killed by a restart continues where it stopped, and completes
|
||||
across however many restarts it takes.
|
||||
|
||||
That is a deliberate exception to "do NOT auto-resume" — scoped to the
|
||||
named jobs only. The rule protects against a restart silently resuming
|
||||
work nobody asked for; here somebody did ask, in configuration, and not
|
||||
having to ask again is the entire point. Every other paused run still
|
||||
waits for an operator.
|
||||
|
||||
A resumed run keeps the flags it started with (`repair` / `deep` are
|
||||
persisted to `params` on the fresh open and read back on resume), so
|
||||
editing the config mid-migration does not retroactively change a run
|
||||
already in flight.
|
||||
|
||||
**Safe to leave set.** Each job is idempotent and resumable; once the
|
||||
tier has drained, a run is a `read_dir` over three directories that
|
||||
returns nothing — and after the directory is removed, not even that.
|
||||
|
||||
**Visible in the admin panel.** These are ordinary registered jobs:
|
||||
they appear in `GET /api/admin/jobs`, are triggerable by hand, and
|
||||
record the same runs and findings. Rows named here additionally carry a
|
||||
`startup` object with the configured flags, so an operator can see that
|
||||
a job deletes files on every boot rather than only when someone clicks.
|
||||
|
||||
### Admin surface (recoverable runs)
|
||||
|
||||
Same URL taxonomy as Part 1 — resource-first, action second, all
|
||||
|
||||
+30
@@ -55,6 +55,36 @@ OXICLOUD_SERVER_HOST=0.0.0.0
|
||||
# Recommended: 127.0.0.1:9090 with node_exporter-style scrapers.
|
||||
#OXICLOUD_METRICS_LISTEN=127.0.0.1:9090
|
||||
|
||||
# ── Startup jobs ──────────────────────────────────────────────────────
|
||||
# Background jobs dispatched once, after the scheduler is ready.
|
||||
# Comma-separated; each entry is a registered job name, optionally with
|
||||
# the same flags the admin trigger URL takes (force, deep, repair,
|
||||
# storage).
|
||||
#
|
||||
# DEFAULT (applied when this variable is unset):
|
||||
# thumb_derived_import?repair=true,thumb_attached_import?repair=true
|
||||
#
|
||||
# Those two migrate thumbnails out of the legacy .thumbnails/ directory
|
||||
# into blob storage and then delete the originals, so the migration
|
||||
# completes without anyone having to trigger it from the admin panel.
|
||||
# Each sidecar is read back through the normal stack before it is
|
||||
# unlinked, and every deletion is written to the audit log.
|
||||
#
|
||||
# Dispatch is non-blocking — startup never waits on a job. A run
|
||||
# interrupted by a restart resumes from its cursor on the next boot, so
|
||||
# a long migration finishes across restarts. Safe to leave at the
|
||||
# default: the jobs are idempotent, and once the directory is drained a
|
||||
# run does nothing at all.
|
||||
#
|
||||
# An unknown job name or flag is a FATAL error at boot, not a warning —
|
||||
# a silently ignored entry means a migration that never runs.
|
||||
#
|
||||
# To disable every startup job, set this to the empty value:
|
||||
#OXICLOUD_STARTUP_JOBS=
|
||||
#
|
||||
# To import without deleting (inspect first, delete later by hand):
|
||||
#OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import
|
||||
|
||||
# ── Upload size caps ──────────────────────────────────────────────────
|
||||
# See docs/config/storage-fine-tuning.md for sizing guidance.
|
||||
|
||||
|
||||
@@ -663,6 +663,19 @@ export interface JobSummary {
|
||||
* for this job. Distinct from `running` — a paused run is
|
||||
* resumable via the same trigger endpoint. */
|
||||
paused_run?: PausedRunBrief;
|
||||
/** Present iff `OXICLOUD_STARTUP_JOBS` names this job — the flags it
|
||||
* is dispatched with at every boot. Worth showing: a job configured
|
||||
* with `repair: true` deletes on every restart, and the row would
|
||||
* otherwise suggest that only happens when someone clicks Run. */
|
||||
startup?: StartupTrigger;
|
||||
}
|
||||
|
||||
/** Flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with. */
|
||||
export interface StartupTrigger {
|
||||
force: boolean;
|
||||
deep: boolean;
|
||||
repair: boolean;
|
||||
storage?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -860,6 +860,31 @@
|
||||
{mutatesLabel(job)}
|
||||
</span>
|
||||
{/if}
|
||||
<!-- Configured to fire at boot. Called out because the
|
||||
row is otherwise silent about the most consequential
|
||||
fact on it: with repair on, this job deletes files
|
||||
on every restart, not only when someone clicks Run.
|
||||
Warn-coloured in that case, neutral otherwise. -->
|
||||
{#if job.startup}
|
||||
<span
|
||||
class="jobs-panel__pill"
|
||||
class:jobs-panel__pill--paused={job.startup.repair}
|
||||
class:jobs-panel__pill--neutral={!job.startup.repair}
|
||||
title={job.startup.repair
|
||||
? t(
|
||||
'admin.jobs.startup_repair_tooltip',
|
||||
'Configured in OXICLOUD_STARTUP_JOBS to run in repair mode at every boot.'
|
||||
)
|
||||
: t(
|
||||
'admin.jobs.startup_tooltip',
|
||||
'Configured in OXICLOUD_STARTUP_JOBS to run at every boot.'
|
||||
)}
|
||||
>
|
||||
{job.startup.repair
|
||||
? t('admin.jobs.startup_repair', 'at boot · repair')
|
||||
: t('admin.jobs.startup', 'at boot')}
|
||||
</span>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="jobs-panel__muted">{cadenceLabel(job)}</td>
|
||||
<td class="jobs-panel__muted">{timeAgo(job.last_run_at)}</td>
|
||||
|
||||
@@ -1282,6 +1282,10 @@
|
||||
"run_mutating_confirm_body": "This job changes stored state when it runs.",
|
||||
"mutates_never": "read-only",
|
||||
"mutates_on_repair_only": "read-only unless repaired",
|
||||
"startup": "at boot",
|
||||
"startup_repair": "at boot · repair",
|
||||
"startup_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run at every boot.",
|
||||
"startup_repair_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run in repair mode at every boot.",
|
||||
"run_variants_menu": "Run variants menu",
|
||||
"run_repair_confirm": "Repair",
|
||||
"triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired",
|
||||
|
||||
@@ -2,6 +2,8 @@ use std::env;
|
||||
use std::path::PathBuf;
|
||||
use std::time::Duration;
|
||||
|
||||
use crate::infrastructure::scheduler::JobRunArgs;
|
||||
|
||||
/// Cache configuration
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CacheConfig {
|
||||
@@ -2280,6 +2282,148 @@ pub struct GrantCleanupConfig {
|
||||
pub interval_hours: u64,
|
||||
}
|
||||
|
||||
/// One job to dispatch once at startup, parsed from an entry of
|
||||
/// `OXICLOUD_STARTUP_JOBS`.
|
||||
///
|
||||
/// **Why this exists.** Scheduled ticks deliberately never pass
|
||||
/// `repair` — a job that deletes on its default setting is the thing
|
||||
/// no-silent-auto-repair forbids. But that leaves the migration jobs in
|
||||
/// a state where an operator who never opens the admin panel imports
|
||||
/// forever and never drains: the sidecars are fully redundant, and
|
||||
/// nothing removes them. Naming the job in configuration IS the
|
||||
/// deliberate operator action; it just gets taken once, at boot,
|
||||
/// instead of every time.
|
||||
///
|
||||
/// Not a general "run everything in repair mode" switch. Each job is
|
||||
/// named individually, and the flags are per job.
|
||||
/// Holds a [`JobRunArgs`] rather than re-listing its fields. They are
|
||||
/// the same four flags with the same meanings, and a copy here would
|
||||
/// have to be found and updated the next time the scheduler grows a
|
||||
/// fifth — silently ignoring it in configuration until someone noticed.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct StartupJob {
|
||||
/// Registered job name — must match `JobHandler::name`.
|
||||
pub name: String,
|
||||
/// Forwarded verbatim to `JobRegistry::trigger`.
|
||||
pub args: JobRunArgs,
|
||||
}
|
||||
|
||||
/// Parse one `OXICLOUD_STARTUP_JOBS` entry: `name`, or
|
||||
/// `name?repair=true&deep=true`.
|
||||
///
|
||||
/// The query syntax is the one an operator already types at
|
||||
/// `POST /api/admin/jobs/{name}/trigger?repair=true`, so the value is
|
||||
/// literally the request they would otherwise make by hand.
|
||||
///
|
||||
/// **Errors on anything it does not recognise**, rather than ignoring
|
||||
/// it. A silently-dropped `?repare=true` typo would leave the job
|
||||
/// running in discovery-only mode forever while the operator believed
|
||||
/// the tier was draining — the failure would surface as "the migration
|
||||
/// never finishes" months later, with nothing in the logs pointing at
|
||||
/// the config. Same reasoning as fail-fast on any broken config.
|
||||
fn parse_startup_job(raw: &str) -> Result<StartupJob, String> {
|
||||
let raw = raw.trim();
|
||||
let (name, query) = match raw.split_once('?') {
|
||||
Some((n, q)) => (n.trim(), q),
|
||||
None => (raw, ""),
|
||||
};
|
||||
if name.is_empty() {
|
||||
return Err("empty job name".to_string());
|
||||
}
|
||||
|
||||
let mut job = StartupJob {
|
||||
name: name.to_string(),
|
||||
args: JobRunArgs::default(),
|
||||
};
|
||||
|
||||
for pair in query.split('&').filter(|p| !p.is_empty()) {
|
||||
let (key, value) = pair
|
||||
.split_once('=')
|
||||
.ok_or_else(|| format!("`{pair}` is not key=value (job `{name}`)"))?;
|
||||
// Booleans accept only `true`/`false` — the same rule the HTTP
|
||||
// trigger enforces, so a value that works in one place works in
|
||||
// the other. See memory `bug_axum_query_bool_only_accepts_true_false`.
|
||||
let as_bool = || match value {
|
||||
"true" => Ok(true),
|
||||
"false" => Ok(false),
|
||||
other => Err(format!(
|
||||
"`{key}={other}` on job `{name}`: expected true or false"
|
||||
)),
|
||||
};
|
||||
match key {
|
||||
"force" => job.args.force = as_bool()?,
|
||||
"deep" => job.args.deep = as_bool()?,
|
||||
"repair" => job.args.repair = as_bool()?,
|
||||
"storage" => job.args.storage = Some(value.to_string()),
|
||||
other => {
|
||||
return Err(format!(
|
||||
"unknown flag `{other}` on job `{name}`: expected force, deep, repair \
|
||||
or storage"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(job)
|
||||
}
|
||||
|
||||
/// What runs at boot when `OXICLOUD_STARTUP_JOBS` is unset.
|
||||
///
|
||||
/// **Both migration jobs, both in repair mode** — they import their
|
||||
/// sidecars and then delete them. Chosen deliberately: an operator who
|
||||
/// never edits `.env` is the normal case, and a migration nobody
|
||||
/// triggers never finishes, so a default that only imports would leave
|
||||
/// every untouched deployment carrying a fully-redundant `.thumbnails/`
|
||||
/// forever.
|
||||
///
|
||||
/// This is a destructive default, which is a real exception to
|
||||
/// no-silent-auto-repair, so what makes it safe has to hold:
|
||||
///
|
||||
/// * **Nothing is deleted before its replacement has been read back.**
|
||||
/// `verify_and_unlink` imports, reads the blob back through the normal
|
||||
/// stack, and only then unlinks. A store that reported success but
|
||||
/// landed unreadable keeps its sidecar. That readback is the whole
|
||||
/// safety argument — it matters most for `thumb_attached_import`,
|
||||
/// whose bytes are user-uploaded previews with no render path, so a
|
||||
/// wrong deletion there is permanent where a wrong deletion of a
|
||||
/// server-rendered thumbnail costs only a re-render.
|
||||
/// * **Sidecars whose source is gone are deleted without a readback**,
|
||||
/// because there is nothing to read back and nothing can ever
|
||||
/// reference them again. Unrecoverable and unreachable are different
|
||||
/// things; these are both.
|
||||
/// * **Every deletion is audited**, so an operator can reconstruct what
|
||||
/// a boot removed and from which source.
|
||||
///
|
||||
/// The consequence to be aware of when changing this: an upgrade
|
||||
/// deletes on first boot, in every deployment at once, with no operator
|
||||
/// action. A regression in the readback path would therefore be
|
||||
/// simultaneous and unrecoverable. Treat that code as load-bearing.
|
||||
///
|
||||
/// Set `OXICLOUD_STARTUP_JOBS=` (empty) to disable startup jobs
|
||||
/// entirely; any explicit value replaces this list rather than adding
|
||||
/// to it.
|
||||
const DEFAULT_STARTUP_JOBS: &str =
|
||||
"thumb_derived_import?repair=true,thumb_attached_import?repair=true";
|
||||
|
||||
/// Parse the whole `OXICLOUD_STARTUP_JOBS` value. Empty → no startup
|
||||
/// jobs (an explicit opt-out); unset → [`DEFAULT_STARTUP_JOBS`].
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// On any malformed entry. A startup-job list that half-parses is worse
|
||||
/// than one that fails: the server would come up looking healthy with a
|
||||
/// migration that never runs.
|
||||
fn parse_startup_jobs(raw: &str) -> Vec<StartupJob> {
|
||||
raw.split(',')
|
||||
.map(str::trim)
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(|entry| {
|
||||
parse_startup_job(entry).unwrap_or_else(|e| {
|
||||
panic!("OXICLOUD_STARTUP_JOBS: {e}");
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
impl Default for GrantCleanupConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
@@ -2543,6 +2687,17 @@ pub struct AppConfig {
|
||||
/// bind to loopback / a private interface without exposing
|
||||
/// metrics publicly.
|
||||
pub metrics_listen: Option<std::net::SocketAddr>,
|
||||
/// Jobs to dispatch once, in the background, after the scheduler is
|
||||
/// ready. Env: `OXICLOUD_STARTUP_JOBS` — comma-separated, each entry
|
||||
/// `name` or `name?repair=true`, mirroring the admin trigger URL.
|
||||
///
|
||||
/// Empty by default. Intended for the migration jobs, whose
|
||||
/// scheduled ticks import but deliberately never delete: naming one
|
||||
/// here is the operator's standing consent to the deletion, given
|
||||
/// once in configuration instead of per run in the panel.
|
||||
///
|
||||
/// Dispatch is non-blocking — readiness never waits on a job.
|
||||
pub startup_jobs: Vec<StartupJob>,
|
||||
/// Cache configuration
|
||||
pub cache: CacheConfig,
|
||||
/// Timeout configuration
|
||||
@@ -2671,6 +2826,7 @@ impl Default for AppConfig {
|
||||
plugins: PluginConfig::default(),
|
||||
faces: FacesConfig::default(),
|
||||
metrics_listen: None,
|
||||
startup_jobs: parse_startup_jobs(DEFAULT_STARTUP_JOBS),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2718,6 +2874,19 @@ impl AppConfig {
|
||||
}
|
||||
}
|
||||
|
||||
// Jobs to fire once at boot. Unset keeps DEFAULT_STARTUP_JOBS (set
|
||||
// by `Default`); any explicit value REPLACES it, and an empty value
|
||||
// is the opt-out.
|
||||
//
|
||||
// Panics on a malformed entry rather than warning: unlike metrics,
|
||||
// a startup job that silently fails to parse leaves a migration
|
||||
// that never runs, and the symptom ("the tier never drained")
|
||||
// surfaces months later with nothing pointing back at the config
|
||||
// line.
|
||||
if let Ok(raw) = env::var("OXICLOUD_STARTUP_JOBS") {
|
||||
config.startup_jobs = parse_startup_jobs(&raw);
|
||||
}
|
||||
|
||||
// Database configuration
|
||||
if let Ok(connection_string) = env::var("OXICLOUD_DB_CONNECTION_STRING") {
|
||||
config.database.connection_string = connection_string;
|
||||
@@ -3783,6 +3952,89 @@ pub fn default_config() -> AppConfig {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn startup_job_parses_name_and_flags() {
|
||||
let jobs = parse_startup_jobs(
|
||||
"thumb_derived_import?repair=true, thumb_attached_import ,blobs_consistency?deep=true&force=false",
|
||||
);
|
||||
assert_eq!(jobs.len(), 3);
|
||||
|
||||
assert_eq!(jobs[0].name, "thumb_derived_import");
|
||||
assert!(jobs[0].args.repair);
|
||||
assert!(!jobs[0].args.deep);
|
||||
|
||||
// Bare name → all flags default off, which is the discovery-only
|
||||
// run. Naming a migration job without `repair` imports and stops.
|
||||
assert_eq!(jobs[1].name, "thumb_attached_import");
|
||||
assert!(!jobs[1].args.repair);
|
||||
|
||||
assert!(jobs[2].args.deep);
|
||||
assert!(!jobs[2].args.force);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn startup_job_accepts_storage_scope() {
|
||||
let jobs = parse_startup_jobs("backend_consistency?storage=s3_prod&deep=true");
|
||||
assert_eq!(jobs[0].args.storage.as_deref(), Some("s3_prod"));
|
||||
assert!(jobs[0].args.deep);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn startup_jobs_empty_value_is_the_opt_out() {
|
||||
assert!(parse_startup_jobs("").is_empty());
|
||||
assert!(parse_startup_jobs(" , ,").is_empty());
|
||||
}
|
||||
|
||||
/// Both migration jobs drain themselves out of the box, deletion
|
||||
/// included. Pinned rather than left implicit because this is a
|
||||
/// destructive default: it deletes on first boot after an upgrade,
|
||||
/// everywhere, with no operator action. Whoever changes this line
|
||||
/// should have to change a test that says so.
|
||||
///
|
||||
/// What keeps it safe is the readback in `verify_and_unlink` — import,
|
||||
/// read the blob back through the normal stack, and only then unlink.
|
||||
/// That matters most for `thumb_attached_import`, whose bytes are
|
||||
/// user-uploaded and have no render path to rebuild them.
|
||||
#[test]
|
||||
fn default_startup_jobs_drain_both_thumbnail_tiers() {
|
||||
let jobs = AppConfig::default().startup_jobs;
|
||||
let names: Vec<&str> = jobs.iter().map(|j| j.name.as_str()).collect();
|
||||
assert_eq!(names, ["thumb_derived_import", "thumb_attached_import"]);
|
||||
assert!(jobs.iter().all(|j| j.args.repair));
|
||||
assert!(jobs.iter().all(|j| !j.args.deep && !j.args.force));
|
||||
}
|
||||
|
||||
/// A misspelled flag must not parse. Silently ignoring `repare=true`
|
||||
/// leaves the job in discovery-only mode while the operator believes
|
||||
/// the tier is draining — a failure that surfaces months later as
|
||||
/// "the migration never finished", with nothing pointing at the
|
||||
/// config line.
|
||||
#[test]
|
||||
#[should_panic(expected = "unknown flag `repare`")]
|
||||
fn startup_job_rejects_a_misspelled_flag() {
|
||||
parse_startup_jobs("thumb_derived_import?repare=true");
|
||||
}
|
||||
|
||||
/// Booleans take only true/false — the same rule the HTTP trigger
|
||||
/// enforces, so a value that works in one place works in the other.
|
||||
#[test]
|
||||
#[should_panic(expected = "expected true or false")]
|
||||
fn startup_job_rejects_a_non_boolean_flag_value() {
|
||||
parse_startup_jobs("thumb_derived_import?repair=yes");
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[should_panic(expected = "not key=value")]
|
||||
fn startup_job_rejects_a_valueless_flag() {
|
||||
parse_startup_jobs("thumb_derived_import?repair");
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[should_panic(expected = "empty job name")]
|
||||
fn startup_job_rejects_flags_with_no_job() {
|
||||
parse_startup_jobs("?repair=true");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_allowlist_accepts_any_email() {
|
||||
let cfg = MagicLinkConfig::default();
|
||||
|
||||
@@ -2835,6 +2835,108 @@ impl AppServiceFactory {
|
||||
registered
|
||||
);
|
||||
|
||||
// `OXICLOUD_STARTUP_JOBS` — dispatch each named job once, now.
|
||||
//
|
||||
// Exists for the migration jobs. Their scheduled ticks import but
|
||||
// never delete (`repair` defaults false, per no-silent-auto-repair),
|
||||
// so a deployment whose operator never opens the admin panel keeps
|
||||
// importing sidecars it already imported and never drains the
|
||||
// directory. Naming the job in configuration IS the deliberate
|
||||
// consent that rule asks for; it is simply given once, at boot,
|
||||
// rather than per run.
|
||||
//
|
||||
// Validated here, dispatched in the background:
|
||||
//
|
||||
// * Unknown names **panic**. The registry is fully populated at this
|
||||
// point, so a name that does not resolve is a typo or a rename, and
|
||||
// the failure mode of ignoring it is a migration that silently
|
||||
// never runs. Fail at boot, where the operator is watching.
|
||||
// * Dispatch is `tokio::spawn` — readiness must never wait on a job
|
||||
// that walks a filesystem for hours.
|
||||
// * Sequential within the task, not concurrent: these jobs contend
|
||||
// for the same directory and DB, and the exclusivity gate would
|
||||
// turn overlap into a skipped run rather than a queued one.
|
||||
// * Safe on every boot, including a crash loop: each is idempotent
|
||||
// and resumable, and once drained a run is a `read_dir` that
|
||||
// returns nothing.
|
||||
//
|
||||
// **Killed mid-run, this resumes from the cursor.** The boot
|
||||
// recovery sweep runs earlier in this function and flips every row
|
||||
// the dead process abandoned in `Running` to `Paused`, keeping its
|
||||
// cursor. `run_or_resume` then picks Resume over a fresh start, so
|
||||
// a job interrupted by a restart continues where it stopped rather
|
||||
// than rescanning from the beginning — and a long migration
|
||||
// completes across however many restarts it takes.
|
||||
//
|
||||
// That is a deliberate exception to `boot_recovery_sweep`'s "we do
|
||||
// not auto-resume; operators trigger the resume explicitly". The
|
||||
// rule exists so a restart never silently resumes work nobody
|
||||
// asked for. Here somebody did ask, in configuration, and the whole
|
||||
// point of the option is not having to ask again. The exception is
|
||||
// scoped to the named jobs; every other paused run still waits for
|
||||
// an operator.
|
||||
//
|
||||
// The resumed run keeps the flags it started with — `repair` and
|
||||
// `deep` are persisted to the run's `params` on the fresh open and
|
||||
// read back on resume — so editing the config mid-migration does
|
||||
// not retroactively change a run already in flight.
|
||||
if !self.config.startup_jobs.is_empty() {
|
||||
let mut planned = Vec::with_capacity(self.config.startup_jobs.len());
|
||||
for job in &self.config.startup_jobs {
|
||||
if app_state.core.job_registry.get(&job.name).await.is_none() {
|
||||
panic!(
|
||||
"OXICLOUD_STARTUP_JOBS names `{}`, which is not a registered job. \
|
||||
Check the spelling against GET /api/admin/jobs.",
|
||||
job.name
|
||||
);
|
||||
}
|
||||
planned.push(job.clone());
|
||||
}
|
||||
|
||||
let registry = app_state.core.job_registry.clone();
|
||||
tokio::spawn(async move {
|
||||
for job in planned {
|
||||
// Audited, not merely logged: a startup job may delete
|
||||
// files, and "who asked for this" must be answerable
|
||||
// afterwards. The answer is the configuration, which is
|
||||
// exactly what this line records.
|
||||
tracing::info!(
|
||||
target: "audit",
|
||||
event = "job.startup_trigger",
|
||||
job = %job.name,
|
||||
force = job.args.force,
|
||||
deep = job.args.deep,
|
||||
repair = job.args.repair,
|
||||
storage = ?job.args.storage,
|
||||
"👮🏻♂️ dispatching `{}` from OXICLOUD_STARTUP_JOBS",
|
||||
job.name,
|
||||
);
|
||||
match registry.trigger(&job.name, &job.args).await {
|
||||
Some(outcome) => tracing::info!(
|
||||
target: "oxicloud::scheduler",
|
||||
event = "job.startup_completed",
|
||||
job = %job.name,
|
||||
outcome = outcome.kind(),
|
||||
"startup job `{}` finished ({})",
|
||||
job.name,
|
||||
outcome.kind(),
|
||||
),
|
||||
// Unreachable — the name was resolved above, and
|
||||
// nothing unregisters. Logged rather than panicking
|
||||
// because this is a detached task by then.
|
||||
None => tracing::error!(
|
||||
target: "oxicloud::scheduler",
|
||||
event = "job.startup_vanished",
|
||||
job = %job.name,
|
||||
"startup job `{}` disappeared from the registry between \
|
||||
validation and dispatch",
|
||||
job.name,
|
||||
),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
Ok(app_state)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -37,5 +37,7 @@ pub use recoverable::{
|
||||
RecoverableJobHandler, RunOutcome, RunProgress, RunStatus, RunSummary, derive_progress,
|
||||
record_or_log, run_or_resume,
|
||||
};
|
||||
pub use registry::{JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError};
|
||||
pub use registry::{
|
||||
JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError, StartupTrigger,
|
||||
};
|
||||
pub use types::{ErrCause, JobOutcome, JobRunArgs, Mutates};
|
||||
|
||||
@@ -236,11 +236,12 @@ impl JobRegistry {
|
||||
last_outcome,
|
||||
running: state.current_run_start.is_some(),
|
||||
recoverable: entry.handler.is_recoverable(),
|
||||
// Populated in `list_jobs` handler via a single
|
||||
// DB round-trip — kept out of the registry
|
||||
// snapshot to avoid pulling a DB dependency into
|
||||
// the in-memory scheduler state.
|
||||
// Both populated in the `list_jobs` handler — one
|
||||
// from a DB round-trip, one from AppConfig. Kept
|
||||
// out of the registry snapshot so the in-memory
|
||||
// scheduler state pulls in neither dependency.
|
||||
paused_run: None,
|
||||
startup: None,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
@@ -346,6 +347,32 @@ pub struct JobSummary {
|
||||
/// picks Resume when the latest row is Paused).
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub paused_run: Option<PausedRunBrief>,
|
||||
/// Populated iff `OXICLOUD_STARTUP_JOBS` names this job — the flags
|
||||
/// it will be dispatched with at every boot.
|
||||
///
|
||||
/// Surfaced because the panel would otherwise be silently wrong
|
||||
/// about the most consequential thing on the row: a job configured
|
||||
/// with `repair=true` deletes files on every restart, and reading
|
||||
/// the row you would think that only happens when someone clicks.
|
||||
/// Filled by the `list_jobs` handler, which has the config; the
|
||||
/// registry deliberately doesn't.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub startup: Option<StartupTrigger>,
|
||||
}
|
||||
|
||||
/// The flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with.
|
||||
///
|
||||
/// Mirrors `JobRunArgs` on the wire rather than embedding it, because
|
||||
/// this is an API shape the admin panel switches on, and `JobRunArgs`
|
||||
/// is an internal dispatch type free to change without a frontend
|
||||
/// release.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct StartupTrigger {
|
||||
pub force: bool,
|
||||
pub deep: bool,
|
||||
pub repair: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
/// Enough info about a paused recoverable run for the admin panel to
|
||||
|
||||
@@ -48,6 +48,15 @@ use crate::infrastructure::services::dedup_service::DedupService;
|
||||
|
||||
pub const THUMB_DERIVED_IMPORT_JOB_NAME: &str = "thumb_derived_import";
|
||||
|
||||
/// Where the legacy tree is moved when it cannot be deleted.
|
||||
///
|
||||
/// Deletion is always attempted first — this is the fallback for the one
|
||||
/// case `remove_dir` refuses: a file that is not a sidecar sitting in the
|
||||
/// directory (Finder's `.DS_Store`, most often). What matters to the read
|
||||
/// path is that `.thumbnails` stops existing, so moving the tree aside
|
||||
/// achieves the same thing while preserving whatever the stray file was.
|
||||
pub(crate) const PARKED_DIR_NAME: &str = ".thumbnails.migrated";
|
||||
|
||||
/// Record a sidecar deletion on the audit channel.
|
||||
///
|
||||
/// Both import jobs delete user-visible files during a one-way migration, so
|
||||
@@ -182,13 +191,36 @@ impl ThumbDerivedImport {
|
||||
stored_hash: &str,
|
||||
path: &std::path::Path,
|
||||
) -> bool {
|
||||
let Ok(meta) = fs::metadata(path).await else {
|
||||
// Compare CONTENT, not length.
|
||||
//
|
||||
// This is the only thing standing between a storage bug and
|
||||
// permanent loss — `thumb_attached_import` deletes user-uploaded
|
||||
// previews that have no render path to rebuild them, and with the
|
||||
// startup-job default it does so on first boot after an upgrade,
|
||||
// in every deployment at once. A guard that load-bearing should
|
||||
// prove the bytes are the bytes.
|
||||
//
|
||||
// Length alone did not. A blob of the right size and the wrong
|
||||
// content passed: a key-mapping bug handing back another file's
|
||||
// preview at the same length would have deleted the original and
|
||||
// kept the impostor, and thumbnails cluster tightly enough in size
|
||||
// for that to be a real coincidence rather than a theoretical one.
|
||||
//
|
||||
// Re-reading the sidecar costs a few KB of I/O, once per file ever
|
||||
// migrated. The import path already has these bytes in hand, but
|
||||
// taking them as an argument would leave the already-imported path
|
||||
// (which has no bytes, only a file) on a weaker check — one code
|
||||
// path, one guarantee.
|
||||
let Ok(sidecar) = fs::read(path).await else {
|
||||
return false;
|
||||
};
|
||||
// `read_blob_bytes` streams from the backend, reassembling chunks
|
||||
// if the blob is chunked — no cache sits in front of it, so this
|
||||
// proves durability and not merely that a write was acknowledged.
|
||||
let Ok(stored) = dedup.read_blob_bytes(stored_hash).await else {
|
||||
return false;
|
||||
};
|
||||
if stored.is_empty() || stored.len() as u64 != meta.len() {
|
||||
if stored.is_empty() || stored.as_ref() != sidecar.as_slice() {
|
||||
return false;
|
||||
}
|
||||
if fs::remove_file(path).await.is_err() {
|
||||
@@ -592,16 +624,47 @@ impl RecoverableJobHandler for ThumbDerivedImport {
|
||||
"🧹 legacy sidecar directory removed — the fallback read path \
|
||||
is inert from the next restart"
|
||||
),
|
||||
Err(e) => tracing::info!(
|
||||
target: "oxicloud::dedup",
|
||||
event = "thumb_derived_import.root_kept",
|
||||
run_id = %store.run_id(),
|
||||
path = %self.thumbnails_root.display(),
|
||||
reason = %e,
|
||||
"legacy sidecar directory not removed; if this says \
|
||||
'directory not empty' with no sidecars left, something \
|
||||
else put a file there (a .DS_Store, typically)"
|
||||
),
|
||||
// Something unrelated to thumbnails is in the directory, so
|
||||
// `remove_dir` refuses. On macOS that is Finder's `.DS_Store`,
|
||||
// and it would otherwise keep the fallback alive forever on
|
||||
// every developer machine.
|
||||
//
|
||||
// Move the whole tree aside instead. The sidecars are already
|
||||
// imported and verified, so nothing here is load-bearing; what
|
||||
// matters is that `.thumbnails` stops existing, because its
|
||||
// absence is what the read path tests. Renaming preserves the
|
||||
// stray file for whoever put it there, and keeps the check a
|
||||
// single `stat` rather than a directory walk.
|
||||
Err(_) => {
|
||||
// `with_file_name`, NOT `with_extension`: the directory is
|
||||
// `.thumbnails`, and a leading-dot name has no extension as
|
||||
// far as `Path` is concerned — its whole name is the stem.
|
||||
// `with_extension("thumbnails.migrated")` would have
|
||||
// produced `.thumbnails.thumbnails.migrated`.
|
||||
let parked = self.thumbnails_root.with_file_name(PARKED_DIR_NAME);
|
||||
match fs::rename(&self.thumbnails_root, &parked).await {
|
||||
Ok(()) => tracing::info!(
|
||||
target: "oxicloud::dedup",
|
||||
event = "thumb_derived_import.root_parked",
|
||||
run_id = %store.run_id(),
|
||||
from = %self.thumbnails_root.display(),
|
||||
to = %parked.display(),
|
||||
"🧹 legacy sidecar directory could not be removed (a \
|
||||
non-sidecar file remains) — moved aside instead. The \
|
||||
fallback read path is inert from the next restart; \
|
||||
the directory is safe to delete by hand."
|
||||
),
|
||||
Err(e) => tracing::warn!(
|
||||
target: "oxicloud::dedup",
|
||||
event = "thumb_derived_import.root_kept",
|
||||
run_id = %store.run_id(),
|
||||
path = %self.thumbnails_root.display(),
|
||||
reason = %e,
|
||||
"legacy sidecar directory could be neither removed nor \
|
||||
moved aside — the fallback read path stays live"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -646,6 +709,24 @@ pub(crate) mod tests {
|
||||
/// A second hash, for the JPEG sidecar in `legacy_tree`.
|
||||
const H2: &str = "c222222222222222222222222222222222222222222222222222222222222222";
|
||||
|
||||
/// The park path must be a SIBLING of `.thumbnails`, not a suffixed
|
||||
/// child of its name.
|
||||
///
|
||||
/// `Path::with_extension` looks right and is wrong here: a leading-dot
|
||||
/// name has no extension as far as `Path` is concerned — `.thumbnails`
|
||||
/// is entirely stem — so `with_extension("thumbnails.migrated")`
|
||||
/// yields `.thumbnails.thumbnails.migrated`. The rename would still
|
||||
/// have "worked", leaving a directory nobody documented and an
|
||||
/// operator hunting for the name the runbook promised.
|
||||
#[test]
|
||||
fn parked_directory_is_a_sibling_named_thumbnails_migrated() {
|
||||
let root = std::path::Path::new("/srv/storage/.thumbnails");
|
||||
assert_eq!(
|
||||
root.with_file_name(PARKED_DIR_NAME),
|
||||
std::path::Path::new("/srv/storage/.thumbnails.migrated"),
|
||||
);
|
||||
}
|
||||
|
||||
/// BOTH codecs are claimed, and the format comes from the extension.
|
||||
///
|
||||
/// `.jpg` was previously rejected here, which was correct only while the
|
||||
|
||||
@@ -263,27 +263,19 @@ impl ThumbnailService {
|
||||
/// written a sidecar since step 10d2, so there is nothing to create them
|
||||
/// for.
|
||||
///
|
||||
/// The probe tests the SIZE directories, not the root. On macOS Finder
|
||||
/// drops a `.DS_Store` in the root, which blocks `remove_dir` there
|
||||
/// forever — gating on the root would keep the fallback alive on every
|
||||
/// developer machine for a reason that has nothing to do with thumbnails.
|
||||
/// If no size directory exists, no sidecar can exist.
|
||||
/// One `stat` on the root. The import job guarantees that is enough: it
|
||||
/// removes the directory once drained, and when something unrelated
|
||||
/// keeps `remove_dir` from succeeding — Finder's `.DS_Store`, typically
|
||||
/// — it renames the tree to `.thumbnails.migrated` rather than leaving
|
||||
/// it in place. So `.thumbnails` existing always means "there may be
|
||||
/// sidecars under here", and a stray file cannot pin the fallback open.
|
||||
///
|
||||
/// Result is cached for the process lifetime. It can only be stale in the
|
||||
/// harmless direction: a drain completing mid-life leaves the flag `true`
|
||||
/// until restart, which costs the same failed opens as today. It never
|
||||
/// goes `false` while sidecars remain.
|
||||
pub async fn initialize(&self) -> std::io::Result<()> {
|
||||
let mut present = false;
|
||||
for size in ThumbnailSize::all() {
|
||||
if fs::metadata(self.thumbnails_root.join(size.dir_name()))
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
present = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
let present = fs::metadata(&self.thumbnails_root).await.is_ok();
|
||||
self.legacy_sidecars.store(present, Ordering::Relaxed);
|
||||
|
||||
// Asymmetric on purpose. "Present" is actionable and temporary — it
|
||||
|
||||
@@ -2562,6 +2562,28 @@ pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse
|
||||
}
|
||||
}
|
||||
|
||||
// Mark the jobs `OXICLOUD_STARTUP_JOBS` dispatches at boot. Without
|
||||
// this the panel is silently wrong about the most consequential thing
|
||||
// on the row: a job configured with `repair=true` deletes files on
|
||||
// every restart, and the row would suggest that only ever happens
|
||||
// when someone clicks Run.
|
||||
for job in summary.iter_mut() {
|
||||
if let Some(configured) = state
|
||||
.core
|
||||
.config
|
||||
.startup_jobs
|
||||
.iter()
|
||||
.find(|s| s.name == job.name)
|
||||
{
|
||||
job.startup = Some(crate::infrastructure::scheduler::StartupTrigger {
|
||||
force: configured.args.force,
|
||||
deep: configured.args.deep,
|
||||
repair: configured.args.repair,
|
||||
storage: configured.args.storage.clone(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
(StatusCode::OK, Json(summary)).into_response()
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user