Ed's diagnosis, and it is the root of three symptoms I had been patching
one at a time: a job has two independent statuses, and the panel was
collapsing them into one column.
* STATE — where the run is in its lifecycle: running, paused,
cancelled, completed, failed.
* OUTCOME — how the work turned out: ok, issues, notices, err.
They are orthogonal. A paused run has no outcome yet. A completed run's
outcome may still be "issues". Conflating them produced, in order:
1. a paused migration rendering as a green "ok" — the outcome was
genuinely ok, the STATE was Paused, and only the outcome was shown;
2. my first fix, which put "blocked" into the OUTCOME column — a
category error, encoding lifecycle into the result axis;
3. a cancelled job still reading "blocked", because that outcome was
cached in memory while the cancel had flipped the row in SQL.
The layout already had both columns. State just never rendered anything
but "running" or "—", so the status axis had no home and the information
leaked into Outcome.
Now:
* State renders `last_run_status`, sourced from the run ROW. Memory
cannot answer this — it is empty after a restart and stale after a
cancel, both of which the row gets right. The retryable reason, when
there is one, is the pill's tooltip.
* Outcome goes back to describing only the work: ok / issues /
notices / err. No lifecycle in it.
`JobSummary` gains `last_run_status`, and `last_run_at` falls back to
the row's `started_at` when memory has none — a restart left the column
reading "never" for a job whose last run was hours earlier.
"never" is now reserved for jobs that genuinely never ran. With a run
row present but no cached outcome the cell reads "—": the honest "no
outcome recorded", rather than a claim the run history immediately
contradicts.
The enrichment query generalises rather than multiplying — it already
fetched Paused rows for the Resume button, so it now takes the latest
row per job via `DISTINCT ON` and derives state, timestamp and paused
brief from it. Sound as "the current run" because the
`one_active_run_per_job` partial unique index permits one non-terminal
row per job and a resume reuses it, so a non-terminal row is always
newest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ed pointed a broken S3 entry at 127.0.0.1 with nothing listening. The
classification worked end to end — the run paused with
`target backend init: Transient Backend: … ConnectionRefused` — but the
job row showed a green **ok** pill and the reason was only visible after
expanding it.
`PausedRetryable` reports `outcome: 'ok'` on the wire, and that is
correct: the run did not fail, and a Resume continues it. But rendering
it as plain "ok" hides the one thing worth acting on. A paused
`backend_migration` is still holding `migration_readonly` and refusing
writes across the whole application, presented as a healthy job.
The row now reads **blocked**, in amber, with the reason as the pill's
title so it is legible without unfolding anything.
Amber rather than red, deliberately: nothing is broken and no data was
lost — the run is waiting for the backend to return. Red reads as "this
job is failing" and invites a Cancel, which for a migration also
discards the copy already done and is the one action that cannot be
undone.
Checked BEFORE the findings branches, too. A run that never finished has
nothing meaningful to say about findings, and "0 issues" on an aborted
scan is a worse answer than "blocked".
`JobOutcome.extra` was typed `unknown`, so the panel could not read the
`retryable` flag the backend already sends. Now a narrow
`JobOutcomeExtra` exposing just the three keys that describe the RUN's
shape rather than its work — the rest stay per-job counters nothing
generic should switch on.
Same class of defect as the known "Ok despite findings" issue: an
outcome that looks like success while hiding the state an operator needs
to see.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`JobRunArgs` was a fixed struct — `force`, `deep`, `storage`, `repair` —
and six places hardcoded that same list: the engine's persist/restore,
the trigger endpoint's query type, the OXICLOUD_STARTUP_JOBS parser, the
frontend API wrapper, the panel's checkboxes, and `StartupTrigger` on
the wire.
Two costs. Adding a parameter meant editing all six, and forgetting one
dropped it silently — most damagingly in persist/restore, where a
resumed run lost it and a `?repair=true` migration came back as
discovery-only after a restart. And the panel offered the same knobs on
every job: only two jobs read `deep`, six read `repair`, so most of
those controls did nothing with no way to tell which.
Now `JobHandler::parameters()` returns `&'static [JobParam]` — name,
type (boolean/string/number), default, and the job's own description of
what it does. `JobRunArgs` holds a map keyed by those names.
Everything reads the declaration:
* `run_or_resume` iterates it to persist and restore, replacing
`const FLAGS` plus a `storage` special case. `storage` stops being
special — it was the one Option<String> among three bools.
* `dispatch` normalises every run against it, which is what makes "a
handler sees its declared parameters with their declared defaults"
true rather than usual. The periodic tick passes an empty
`JobRunArgs::default()`, so a `default: true` parameter would
otherwise read false on every scheduled run.
* The trigger endpoint takes free-form query params and rejects
undeclared ones with a 400 naming the real set, instead of ignoring
them.
* OXICLOUD_STARTUP_JOBS keeps raw pairs (config is parsed before the
registry exists) and validates at dispatch, where the error can name
the job's actual parameters. Still a boot panic, same as an unknown
job name — a typo'd `?repare=true` must not leave a migration
importing forever in discovery mode.
* `JobSummary.parameters` carries it to the panel, whose `supportsDeep`
was a hardcoded name allowlist (`consistency_batch ||
backend_consistency`). A job gaining a deep mode needed a frontend
release; one losing it left a button that silently did nothing. The
menu now renders from the declaration, so a newly-declared boolean
appears with no frontend change.
Three consistency tenants were hand-rolling persist-on-fresh /
restore-on-resume for their own flag, under the same `params` key the
engine already used. Deleted — they read `args.get_bool(…)` now.
Fresh runs also filter to the declaration. `consistency_batch` forwards
its args verbatim to sub-jobs, so a tenant's `params` row could grow
`deep` with no deep mode, and the run-detail view would claim a mode the
job never had.
Two things found while wiring it, both worth knowing:
`RecoverableAdapter` bridges the two traits, and `parameters` has to be
forwarded there or the registry sees `&[]`. Both traits have defaults,
so omitting it compiled cleanly — and the trigger endpoint then rejected
`?repair=true` on the very jobs that declare it, with
OXICLOUD_STARTUP_JOBS panicking at boot. Now covered by
`adapter_forwards_job_metadata_from_inner_handler`.
`TriggerJobQuery` was briefly a newtype over the map. `serde_urlencoded`
cannot deserialize a newtype struct at the top level, so axum's `Query`
rejected EVERY trigger with a 400 — even one with no query string —
before the handler ran. It reads exactly like the new validation
rejecting something, which sent the first diagnosis to the wrong layer.
Now covered by `trigger_query_extracts_from_every_url_shape`.
Wire names are a compatibility surface: `params` rows are keyed by them
and the panel switches on them, so a rename breaks existing run history
the same way renaming a `Mutates` variant does. The JSON shape is pinned
in `snapshot_carries_job_metadata`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A DPoP proof carries a server-issued nonce. With none cached the server
answers `401 use_dpop_nonce`, the client harvests the nonce and retries.
`signAndFetch` already absorbed that, so it was invisible — but it did
it PER REQUEST, with no coordination.
The Service Worker always cold-starts without a nonce. Both mechanisms
that pre-seed the page are unreachable from worker scope: workers have
no `sessionStorage`, and `seedNonceFromCookie` early-returns on
`typeof document === 'undefined'`. The SW also skips requests that
already carry a `DPoP` header, so it never observes the page's
responses and cannot harvest from them either. Browsers terminate idle
workers after ~30s, so this happens routinely, not once.
Uncoordinated, every request issued in that window discovers the nonce
independently: N parallel requests → N challenges → 2N requests. That is
precisely the photo grid, and serving `<img src>` is the SW's main job —
those requests cannot sign themselves, which is why the worker exists.
Each wasted challenge also costs the server a full ECDSA P-256 verify,
because `verify_proof` runs before the nonce check.
Now the first request through owns the discovery and the rest await it,
so N challenges collapse to 1. The wait is capped (5s) and released in a
`finally`: the SW is on the critical path for every thumbnail, so a hung
discovery must degrade to the old behaviour rather than stall the grid
behind a promise that never settles.
Measured on a 763-line e2e server log: 115 `dpop.nonce_challenged`
events across 97 logins.
Scope, deliberately: this does NOT remove the one challenge per worker
lifetime. Doing that needs the nonce persisted where a worker can read
it — IndexedDB already holds the keypair — and it can never replace the
challenge path anyway, since a persisted nonce can be stale. Left out
until the audit line shows it is worth it; the numbers above are now
legible enough to tell.
Not a correctness fix. Nothing was broken and no test failed over this.
The argument is waste, plus signal: 115 challenges per run is noise that
would bury a real one.
`hasNonce()` is exported for the gate — deliberately "will the next
proof carry a nonce", not "is it still valid", since only the server
knows the latter and the challenge path already handles it. Its tests
assert it agrees with what `buildDpopProof` actually emits, because a
wrong answer either reinstates the stampede or stalls every request
behind a bootstrap that is not happening.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found on a sandbox restore. `thumb_derived_import` ran first, imported
and deleted its own hash-named sidecars, then found `remove_dir` refused
because the `ext-*.jpg` previews were still there — those belong to
`thumb_attached_import`. The rename fallback fired, moving the tree to
`.thumbnails.migrated`; the attached job then looked in `.thumbnails/`,
found nothing, and reported zeros.
That stranded the user-uploaded previews, which are the one class of
file here with no render path to rebuild them. The rename exists for
files NEITHER job claims — a `.DS_Store` blocking removal forever — and
it fired for the sibling's work in progress instead. Inverting the job
order does not help: once the tree is renamed, both jobs look at
`.thumbnails/` and find nothing, whatever order they run in.
Teardown is now shared and refuses to act while anything remains that
either job would claim. Both jobs call it, so whichever finishes last
removes the tree in the same boot rather than leaving an empty
directory until the next one. The rename survives for its original
purpose, and now only fires when the remaining files are genuinely
nobody's.
Also drops the daily tick on both imports — they are on-demand now. The
boot run in repair mode IS the migration: nothing has written a sidecar
since step 10d2, so the tail cannot grow afterwards, and a tick could
not finish the job anyway because ticks never pass `repair`. Once
drained it was a `read_dir` returning nothing, every day, forever.
UX: the "at boot" badge moves from beside the job name into the cadence
column. It answers WHEN a job runs, which is what that column is for —
next to the name it read as a property of the job, and the row could
show "on-demand" beside a badge saying otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A migration nobody triggers never finishes. Scheduled ticks deliberately
never pass `repair`, so a deployment whose operator never opens the
admin panel re-imported the same sidecars forever and never drained the
directory — and relying on operators to edit `.env` has the same failure
mode one level up.
`OXICLOUD_STARTUP_JOBS` dispatches named jobs once, in the background,
after the scheduler is ready. Entries use the syntax operators already
type at the trigger URL (`name?repair=true`), so the value is literally
the request they would otherwise make by hand. It defaults to both
migration jobs in repair mode, so an untouched deployment migrates and
drains itself.
That is a destructive default and a real exception to
no-silent-auto-repair, so the guard it rests on had to get stronger:
`verify_and_unlink` now compares CONTENT, not length. A blob of the
right size and the wrong bytes used to pass — a key-mapping bug handing
back another file's preview at the same length would have deleted the
original and kept the impostor, and thumbnails cluster tightly enough in
size for that to be a real coincidence. The readback streams from the
backend with no cache in front, so it proves durability rather than that
a write was acknowledged.
Deletion of `.thumbnails/` is attempted first and only falls back to
renaming it `.thumbnails.migrated` when `remove_dir` refuses because a
non-sidecar file is inside (Finder's `.DS_Store`). Either way the
directory stops existing, which lets the read-path probe go back to a
single `stat` on the root instead of walking the size directories.
Validation is fail-fast: an unknown job name or flag panics at boot. A
silently dropped `?repare=true` would leave the job in discovery-only
mode while the operator believed the tier was draining, surfacing months
later as "the migration never finished" with nothing pointing at the
config line.
Interrupted runs resume. Boot recovery flips abandoned rows to Paused
with their cursor, so `run_or_resume` continues rather than rescanning —
a long migration completes across however many restarts it takes. That
is a scoped exception to "we do not auto-resume": here somebody did ask,
in configuration, and not having to ask again is the point.
`StartupJob` holds a `JobRunArgs` rather than re-listing its four
fields, so a fifth flag cannot be added to the scheduler and silently
ignored in configuration.
Jobs named here are ordinary registered jobs — visible in the panel,
triggerable by hand, same runs and findings. Their rows now carry a
`startup` object so an operator can see that a job deletes on every boot
rather than only when someone clicks Run.
Adds docs/config/thumbnail-migration.md: what runs on first boot, how to
snapshot database and storage together beforehand, and how to verify
afterwards with satellites_consistency plus backend_consistency
?deep=true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`blobs_consistency` probed `blob_exists` once per row and, under
`?deep=true`, read and re-hashed every blob. `backend_consistency`
already reports the same `blob_missing_from_backend` from its
merge-join — so the probe was duplicated work that found strictly less
(a DB walk cannot see backend-only orphans by construction) at N round
-trips instead of one enumeration. Every scheduled sweep paid for it.
All three physical checks move to `backend_consistency`:
* `blob_missing_from_backend` was already there; the duplicate is gone.
* `blob_corrupted` / `blob_unreadable` hook the matched arm of the
merge-join, which holds exactly the key pairs worth reading. Guarded
by `in_range` so a pair past the horizon is not read twice, and
`params.deep` is persisted on a fresh run and read back on resume so
a paused deep scan does not silently continue shallow.
Deep mode belongs there because it is backend work end to end: the
only DB input is the hash. Keeping it in `blobs_consistency` forced
that tenant to carry a backend for one flag.
What remains is the half that needs no backend: `refcount_mismatch`
and its repair. The constructor drops from five parameters to two —
no backend, no storage_entries, no storage_path_fallback — and
`?storage=<name>` / `?deep=true` are now inert there, which the
job description says outright.
`affected_files` is needed by both tenants, so it moves to a shared
`blob_diagnostics` module rather than being copied.
`PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined
in `blobs_consistency` and re-exported, which is backwards once the
DB-only tenant has no entry to scope. The create-grace window goes
with the probe — it existed to avoid flagging a blob whose bytes had
landed before its row, and the refcount comparison reads one
consistent snapshot.
Known cost: `backend_consistency` returns `backend_unenumerable` on
Azure and mid-migration, so on those configs missing bytes now go
unreported where the per-row probe caught them. That argues for the
Azure enumeration impl, not for keeping the probe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
```
oxi.UPLOAD_BATCH_BYTES // get current value
oxi.UPLOAD_BATCH_BYTES=1024*1024 // change values
```
values are stored in localstorage, default is 8*1024*1024
Edouard Vanbelle