cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(jobs): jobs describe themselves — description, mutates, repair_description

Browse Source 
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.

Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:

    fn description(&self) -> &'static str
    fn mutates(&self) -> Mutates          // Never | Always | OnRepairOnly
    fn repair_description(&self) -> Option<&'static str>

`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.

Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.

`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.

All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.

Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-08-29 00:04:12 +02:00
parent b485db46fa
commit 1ea3826660
27 changed files with 906 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plan/job-registry.md
+52
View File
@@ -171,6 +171,58 @@ Native services implement this trait on an existing service type (no
new wrapper) and register a single `Arc<dyn JobHandler>` with the
scheduler.
### Self-description — `description` / `mutates` / `repair_description`
Three defaulted methods on both `JobHandler` and `RecoverableJobHandler`
let a job tell the admin UI what it is. `RecoverableAdapter` forwards
them, since the registry only ever holds `dyn JobHandler`.
```rust
fn description(&self) -> &'static str { "" }
fn mutates(&self) -> Mutates { Mutates::Never }
fn repair_description(&self) -> Option<&'static str> { None }
pub enum Mutates { Never, Always, OnRepairOnly }
```
They surface on `JobSummary` (`GET /api/admin/jobs`) and drive the
panel: `Never` earns a read-only badge and triggers straight through,
`Always` confirms first, `OnRepairOnly` is safe to run and confirms only
when the repair variant is picked. `repair_description.is_some()` is
what renders the repair toggle at all, and its text is the confirmation
copy.
**Why three values and not a boolean.** A job can be read-only by
default and destructive under `?repair=true`; a boolean has to answer
wrongly for one of those two modes, and `false` on something that
deletes files is the dangerous direction to be wrong in. It is also
where the recovery framework is heading — discovery-only default,
mutation behind an opt-in — so a tenant that later grows a repair arm
changes this one value and nothing else.
**Why `Option<&str>` and not `supports_repair: bool` + prose.**
Presence gates the toggle, content supplies the wording. Split across
two methods they can disagree; and the frontend cannot invent the
wording itself, because correcting a counter and unlinking files off
disk are not the same warning. The two are independent, not derived
from each other: the thumbnail imports are `Always` *and*
repair-capable.
`OnRepairOnly` with no `repair_description` is rejected at registration
— it claims to mutate only under a flag it does not support, and would
render as safe with no reachable mutating path.
**Why English in the trait, not `locales/*.json`.** A description that
lives away from the behaviour rots the moment a job changes, invisibly,
and a translator cannot know what `manifests_consistency` reconciles.
i18n can layer on later keyed by job name with these as the fallback,
matching the frontend's `t(key, params, fallback)` — a missing
translation then degrades to English from code rather than to a blank
panel. No rework needed to get there.
Defaults exist so the methods could be added without touching every
job at once; every registered job declares all three today.
### `JobOutcome`
```rust
frontend/src/lib/api/types.ts
+24
View File
@@ -618,8 +618,32 @@ export interface PausedRunBrief {
total?: number;
}
/**
* When a job changes state — `RecoverableJobHandler::mutates()` on the
* backend. Three values rather than a boolean because the interesting
* case is conditional: a job can be read-only by default and destructive
* under `?repair=true`.
*
* - `never` — read-only under every flag. Render a read-only badge; no
* confirmation needed to trigger.
* - `always` — changes state on a plain run. Confirm before triggering.
* - `on_repair_only` — safe to trigger; confirm only when the repair
* toggle is on.
*/
export type Mutates = 'never' | 'always' | 'on_repair_only';
export interface JobSummary {
name: string;
/** One or two sentences on what the job does, in English, authored
* next to the handler. Absent for jobs that haven't declared one —
* omit the line rather than rendering an empty block. */
description?: string;
mutates: Mutates;
/** Present iff `?repair=true` does something beyond a default run;
* describes what it ADDS. Presence is what gates the repair toggle;
* the text is the confirmation copy. Independent of `mutates` — the
* thumbnail import jobs are `always` AND repair-capable. */
repair_description?: string;
interval_ms?: number;
next_run_at?: string;
last_run_at?: string;
frontend/src/lib/components/AdminJobsPanel.svelte
+165 -77
View File
@@ -167,7 +167,7 @@
.slice()
// `consistency_batch` is served by the top-bar
// action buttons; hiding it here removes the
// duplicate table row. `hasBatch` still checks the
// duplicate table row. `batchJob` still reads from the
// full fetched list so the top buttons only render
// when the coordinator is actually registered.
.filter((j) => j.name !== 'consistency_batch')
@@ -180,7 +180,7 @@
// Track whether the coordinator is registered so the
// top-bar buttons can gate on it without checking `jobs`
// (which now filters it out).
hasBatch = fetched.some((j) => j.name === 'consistency_batch');
batchJob = fetched.find((j) => j.name === 'consistency_batch') ?? null;
loadError = null;
} catch (e) {
loadError = errorMessage(e);
@@ -250,13 +250,15 @@
// ─── Expansion toggles ─────────────────────────────────────────────
function toggleJob(name: string) {
if (expandedJob === name) {
function toggleJob(job: JobSummary) {
if (expandedJob === job.name) {
expandedJob = null;
} else {
expandedJob = name;
// Lazy-load on first open, refresh on subsequent opens.
void loadRuns(name);
expandedJob = job.name;
// Lazy-load on first open, refresh on subsequent opens. Only
// recoverable jobs have runs to load — the others expand purely
// to show their description.
if (isRecoverable(job)) void loadRuns(job.name);
}
}
@@ -458,8 +460,9 @@
* Per-severity finding counts from `last_outcome.extra.severity_counts`
* (a JSON object populated by `run_or_resume`). Missing / older
* runs return an empty record — callers should tolerate absent keys.
* The three severity values are the ones consistency tenants emit
* today: `data_loss`, `inconsistent`, `anomaly`.
* Severity values emitted today: `data_loss`, `inconsistent`,
* `anomaly`. The set is open (the column is TEXT), so unknown keys
* must degrade rather than throw.
*/
function lastSeverityCounts(job: JobSummary): Record<string, number> {
if (!job.last_outcome || job.last_outcome.outcome !== 'ok') return {};
@@ -480,6 +483,13 @@
return (s.data_loss ?? 0) + (s.inconsistent ?? 0);
}
/**
* Informational findings. `anomaly` is the wire value; "notice" is
* what the panel calls it — there is no separate `notice` severity.
* A job that acted on what it found (a repair run deleting an
* orphaned sidecar) records the same severity and says so in the
* finding's `detail`.
*/
function anomalyFindingCount(job: JobSummary): number {
return lastSeverityCounts(job).anomaly ?? 0;
}
@@ -636,29 +646,65 @@
return name === 'consistency_batch' || name === 'blobs_consistency';
}
// Jobs whose handler consults `args.repair` and applies a
// corrective UPDATE against the finding it just emitted. Only the
// two ref_count tenants today; `consistency_batch` also accepts
// the flag (fans out to both) and is surfaced separately as the
// top-bar "Repair ref_counts" button. Keep this list narrow —
// adding a job here without a matching backend handler produces a
// silently no-op button that confuses operators.
function supportsRepair(name: string): boolean {
return name === 'blobs_consistency' || name === 'manifests_consistency';
// Whether `?repair=true` does anything for this job — declared by the
// handler itself via `repair_description()`, not by a name allowlist
// here. The allowlist this replaces named only the two ref_count
// tenants and silently omitted every repair-capable job added since,
// so the thumbnail imports could not be run in repair mode from the
// panel at all despite supporting it.
function supportsRepair(job: JobSummary): boolean {
return !!job.repair_description;
}
async function onTriggerWithRepairConfirm(name: string) {
// What the repair adds, in the handler's own words. The backend owns
// this string precisely because the wording differs per job: correcting
// a counter and unlinking files off disk are not the same warning, and
// the frontend has no way to tell them apart.
async function onTriggerWithRepairConfirm(job: JobSummary) {
const ok = await confirmDialog({
title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
message: t(
'admin.jobs.run_repair_confirm_body_scoped',
{ name },
'Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.'
title: t(
'admin.jobs.run_repair_confirm_title_scoped',
{ name: job.name },
'Run {{name}} in repair mode?'
),
message: job.repair_description ?? '',
confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
danger: true
});
if (ok) await onTrigger(name, { repair: true });
if (ok) await onTrigger(job.name, { repair: true });
}
// Confirmation before a plain run of a job that writes. `never` jobs
// trigger straight through — that is the point of the flag — and
// `on_repair_only` jobs are read-only until the repair variant is
// picked, which carries its own confirm.
async function onTriggerGuarded(job: JobSummary) {
if (job.mutates === 'always') {
const ok = await confirmDialog({
title: t('admin.jobs.run_mutating_confirm_title', { name: job.name }, 'Run {{name}}?'),
message:
job.description ||
t('admin.jobs.run_mutating_confirm_body', 'This job changes stored state when it runs.'),
confirmText: t('admin.jobs.run', 'Run'),
danger: true
});
if (!ok) return;
}
await onTrigger(job.name);
}
// Row badge. `never` is the one worth stating outright — it is the
// answer to "is it safe to click this on production?", and it is the
// question an operator asks before every trigger.
function mutatesLabel(job: JobSummary): string | null {
switch (job.mutates) {
case 'never':
return t('admin.jobs.mutates_never', 'read-only');
case 'on_repair_only':
return t('admin.jobs.mutates_on_repair_only', 'read-only unless repaired');
default:
return null;
}
}
function isRunning(job: JobSummary): boolean {
@@ -679,11 +725,13 @@
// coordinator is registered (should always be true post-Slice 5,
// but check defensively so the button doesn't appear on an old
// deployment before this component is upgraded).
// Coordinator registration flag — set imperatively in
// `loadJobs` because `jobs` no longer contains the
// `consistency_batch` row (filtered out to avoid duplicating the
// top-bar action buttons).
let hasBatch = $state(false);
// Held as the whole summary rather than a boolean because the
// top-bar buttons need its `repair_description` — the coordinator
// describes its own repair semantics, same as every table row.
// Set imperatively in `loadJobs` because `jobs` no longer contains
// the `consistency_batch` row (filtered out to avoid duplicating
// the top-bar action buttons).
let batchJob = $state<JobSummary | null>(null);
</script>
<section class="jobs-panel">
@@ -697,10 +745,12 @@
</p>
</div>
<div class="jobs-panel__header-actions">
{#if hasBatch}
{#if batchJob}
{@const batch = batchJob}
<button
class="jobs-panel__btn jobs-panel__btn--primary"
disabled={busyKeys.has('trigger:consistency_batch')}
title={batch.description || undefined}
onclick={() => onTrigger('consistency_batch')}
>
<Icon name="play" />
@@ -718,43 +768,28 @@
<Icon name="play" />
{t('admin.jobs.run_deep', 'Run deep')}
</button>
<!-- Repair goes behind a confirm because it issues corrective
UPDATEs on `storage.blobs.ref_count` and
`storage.chunk_manifests.ref_count`. Content-safe (only
counters change, matching the auditor's computed truth)
and race-safe (each UPDATE recomputes inside the same
statement), but writing-a-lot is still writing-a-lot.
One click, one confirm, one batch dispatched to both
refcount tenants via consistency_batch's arg
propagation. See `?repair=true` on
`POST /api/admin/jobs/{name}/trigger`. -->
<button
class="jobs-panel__btn jobs-panel__btn--warn"
disabled={busyKeys.has('trigger:consistency_batch:repair')}
title={t(
'admin.jobs.run_repair_hint',
'Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.'
)}
onclick={async () => {
const ok = await confirmDialog({
title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
message: t(
'admin.jobs.run_repair_confirm_body',
'Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.'
),
confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
danger: true
});
if (ok) await onTrigger('consistency_batch', { repair: true });
}}
>
<Icon name="cog" />
{t('admin.jobs.run_repair', 'Repair ref_counts')}
</button>
<!-- Repair goes behind a confirm because it fans `?repair=true`
out to every sub-check that acts on it. The confirmation
text comes from the coordinator's own
`repair_description` rather than being written here —
what repair means changes as tenants gain repair arms,
and this button would otherwise keep describing only the
two refcount ones. -->
{#if batch.repair_description}
<button
class="jobs-panel__btn jobs-panel__btn--warn"
disabled={busyKeys.has('trigger:consistency_batch:repair')}
title={batch.repair_description}
onclick={() => onTriggerWithRepairConfirm(batch)}
>
<Icon name="cog" />
{t('admin.jobs.run_repair', 'Repair ref_counts')}
</button>
{/if}
{/if}
<!-- Purge is orthogonal to consistency — it works even
when the batch coordinator isn't registered, so it
lives outside the {#if hasBatch}. Opens a modal so
lives outside the batch block. Opens a modal so
the operator picks a retention window with intent
(no accidental delete-all). -->
<button
@@ -798,7 +833,12 @@
{@const runsErr = runsErrorByJob[job.name]}
{@const runsLoading = runsLoadingByJob[job.name]}
{@const expandedRun = expandedRunByJob[job.name] ?? null}
{@const canExpand = isRecoverable(job)}
<!-- Expandable if there is anything to show: a run history,
a description, or both. Gating on `recoverable` alone
would leave the plain periodic jobs (dedup_gc,
trash_cleanup, …) with no way to reach their
description at all. -->
{@const canExpand = isRecoverable(job) || !!job.description}
<tr class="jobs-panel__row" class:jobs-panel__row--expanded={expandedJob === job.name}>
<td>
{#if canExpand}
@@ -806,7 +846,7 @@
type="button"
class="jobs-panel__expand"
aria-expanded={expandedJob === job.name}
onclick={() => toggleJob(job.name)}
onclick={() => toggleJob(job)}
>
<Icon name={expandedJob === job.name ? 'chevron-down' : 'chevron-right'} />
<span class="jobs-panel__name">{job.name}</span>
@@ -814,6 +854,11 @@
{:else}
<span class="jobs-panel__name jobs-panel__name--flat">{job.name}</span>
{/if}
{#if mutatesLabel(job)}
<span class="jobs-panel__pill jobs-panel__pill--readonly">
{mutatesLabel(job)}
</span>
{/if}
</td>
<td class="jobs-panel__muted">{cadenceLabel(job)}</td>
<td class="jobs-panel__muted">{timeAgo(job.last_run_at)}</td>
@@ -898,15 +943,16 @@
button — no chevron, no menu, no extra
width. Preserves one-click discovery for
the common case. -->
{@const hasRunVariants = supportsDeep(job.name) || supportsRepair(job.name)}
{@const hasRunVariants = supportsDeep(job.name) || supportsRepair(job)}
<span class="jobs-panel__split">
<button
class="jobs-panel__btn jobs-panel__btn--small"
class:jobs-panel__split-main={hasRunVariants}
disabled={busyKeys.has(`trigger:${job.name}`)}
title={job.description || undefined}
onclick={() => {
closeAllRunMenus();
void onTrigger(job.name);
void onTriggerGuarded(job);
}}
>
{t('admin.jobs.run', 'Run')}
@@ -942,19 +988,16 @@
<span>{t('admin.jobs.run_deep', 'Run deep')}</span>
</button>
{/if}
{#if supportsRepair(job.name)}
{#if supportsRepair(job)}
<button
type="button"
class="jobs-panel__run-menu-item jobs-panel__run-menu-item--warn"
role="menuitem"
disabled={busyKeys.has(`trigger:${job.name}:repair`)}
title={t(
'admin.jobs.run_repair_hint',
'Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.'
)}
title={job.repair_description}
onclick={() => {
closeAllRunMenus();
void onTriggerWithRepairConfirm(job.name);
void onTriggerWithRepairConfirm(job);
}}
>
<Icon name="cog" />
@@ -966,7 +1009,11 @@
{/if}
</span>
{/if}
{#if isRunning(job) && canExpand}
<!-- `isRecoverable`, not `canExpand`: the latter now also
covers rows that expand only to show a description,
and those must not gain a Cancel button they never
had. -->
{#if isRunning(job) && isRecoverable(job)}
{#if isRecoverable(job)}
<!-- Recoverable running: [Pause] preserves cursor
for later resume; [Cancel] abandons terminally
@@ -1006,7 +1053,20 @@
</td>
</tr>
{#if expandedJob === job.name}
<!-- First row of the expanded block, and only visible there:
the description answers "what is this job" before the
run history answers "what did it do", and keeping it
folded keeps the collapsed table scannable — 17 rows of
two-line prose is not a table any more. -->
{#if expandedJob === job.name && job.description}
<tr class="jobs-panel__desc-row">
<td colspan="6">
<p class="jobs-panel__description">{job.description}</p>
</td>
</tr>
{/if}
{#if expandedJob === job.name && isRecoverable(job)}
<tr class="jobs-panel__runs">
<td colspan="6">
<div class="jobs-panel__runs-inner">
@@ -1621,6 +1681,34 @@
color: var(--color-text-muted);
}
/* "read-only" sits beside the job name and answers the question an
operator asks before every trigger. Deliberately quiet — it marks
the safe case, so it should not compete with outcome pills. */
.jobs-panel__pill--readonly {
margin-left: 0.4rem;
background: var(--color-bg-subtle);
color: var(--color-text-muted);
font-size: 0.72rem;
font-weight: 400;
vertical-align: middle;
}
/* Opens the expanded block, so it carries the drawer's background and
drops its own separator — the runs table below it is part of the
same block, not a new entry. */
.jobs-panel__desc-row td {
padding-left: 2rem; /* clears the chevron, lines up with the name */
border-bottom-color: transparent;
background: var(--color-bg-subtle);
}
.jobs-panel__description {
margin: 0;
color: var(--color-text-muted);
font-size: 0.8rem;
line-height: 1.4;
}
.jobs-panel__runs {
background: var(--color-bg-subtle);
}
frontend/static/locales/en.json
+5 -3
View File
@@ -1277,9 +1277,11 @@
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
"run_repair": "Repair ref_counts",
"run_repair_hint": "Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.",
"run_repair_confirm_title": "Repair drifted ref_counts?",
"run_repair_confirm_body": "Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.",
"run_repair_confirm_body_scoped": "Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.",
"run_repair_confirm_title_scoped": "Run {{name}} in repair mode?",
"run_mutating_confirm_title": "Run {{name}}?",
"run_mutating_confirm_body": "This job changes stored state when it runs.",
"mutates_never": "read-only",
"mutates_on_repair_only": "read-only unless repaired",
"run_variants_menu": "Run variants menu",
"run_repair_confirm": "Repair",
"triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired",
src/application/services/storage_usage_service.rs
+14 -1
View File
@@ -578,7 +578,7 @@ impl StorageUsageService {
pub const USAGE_RECONCILE_JOB_NAME: &str = "usage_reconcile";
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs, Mutates};
use async_trait::async_trait;
impl StorageUsageService {
@@ -603,6 +603,19 @@ impl JobHandler for StorageUsageService {
USAGE_RECONCILE_JOB_NAME
}
fn description(&self) -> &'static str {
"Recomputes the cached storage counters from the underlying file \
sizes — drives first, then the per-user envelope derived from \
them — and corrects any that drifted. This is the corrective \
counterpart to drives_consistency, which only reports the drift."
}
/// Rewrites the counters it finds wrong. Safe to trigger: it recomputes
/// from the files themselves, so a run is idempotent.
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Runs both reconciliation sweeps — drives first, then users —
/// and reports the total number of rows corrected.
///
src/infrastructure/scheduler/handler.rs
+37 -1
View File
@@ -9,7 +9,7 @@
use async_trait::async_trait;
use super::types::{JobOutcome, JobRunArgs};
use super::types::{JobOutcome, JobRunArgs, Mutates};
/// Implemented by every service that wants to run on a fixed interval
/// through the periodic scheduler.
@@ -92,4 +92,40 @@ pub trait JobHandler: Send + Sync {
fn is_recoverable(&self) -> bool {
false
}
/// What this job does, in one or two sentences, for the admin UI.
///
/// English, in the trait, beside the behaviour it describes — not in
/// `locales/*.json`. A description that lives away from the code rots
/// the moment a job changes, invisibly, and a translator cannot know
/// what `manifests_consistency` reconciles. i18n can layer on later
/// keyed by job name with this as the fallback, so a missing
/// translation degrades to English rather than to a blank panel.
///
/// Defaulted to `""` so adding it to the existing jobs is incremental
/// rather than one breaking change; the UI omits the line when empty.
fn description(&self) -> &'static str {
""
}
/// Whether a run changes state, and under what conditions. See
/// [`Mutates`] for why this is not a boolean.
fn mutates(&self) -> Mutates {
Mutates::Never
}
/// `Some(..)` when `?repair=true` does something beyond a default run,
/// describing what it ADDS; `None` when the flag is inert.
///
/// One method rather than a `supports_repair` boolean plus prose: its
/// presence drives whether the UI offers the toggle, its content drives
/// the confirmation text. A boolean would leave the frontend to invent
/// wording for a destructive action it does not understand.
///
/// Independent of [`Self::mutates`], not derived from it — the thumbnail
/// import jobs are [`Mutates::Always`] *and* repair-capable, inserting
/// rows on a plain run and additionally unlinking sidecars under repair.
fn repair_description(&self) -> Option<&'static str> {
None
}
}
src/infrastructure/scheduler/mod.rs
+1 -1
View File
@@ -38,4 +38,4 @@ pub use recoverable::{
record_or_log, run_or_resume,
};
pub use registry::{JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError};
pub use types::{ErrCause, JobOutcome, JobRunArgs};
pub use types::{ErrCause, JobOutcome, JobRunArgs, Mutates};
src/infrastructure/scheduler/recoverable.rs
+107 -2
View File
@@ -48,7 +48,7 @@ use uuid::Uuid;
use crate::common::errors::DomainError;
use super::handler::JobHandler;
use super::types::{JobOutcome, JobRunArgs};
use super::types::{JobOutcome, JobRunArgs, Mutates};
// ─── Run status ─────────────────────────────────────────────────────────────
@@ -238,6 +238,47 @@ pub trait RecoverableJobHandler: Send + Sync {
/// URL fragment: `POST /api/admin/jobs/{name}/trigger`.
fn name(&self) -> &str;
/// What this job does, for the admin UI.
///
/// English, in the trait, beside the behaviour it describes — not in
/// `locales/*.json`. A description that lives away from the code rots the
/// moment a job changes, invisibly, and a translator cannot know what
/// `manifests_consistency` reconciles. i18n can layer on later keyed by
/// job name, with this as the fallback, so a missing translation degrades
/// to English rather than a blank panel.
///
/// Defaulted so adding it to ~15 existing jobs is incremental rather than
/// one breaking change.
fn description(&self) -> &'static str {
""
}
/// Whether a run changes state, and under what conditions.
///
/// Three values rather than a boolean because there are three cases, and
/// the interesting one is conditional: a job can be read-only by default
/// and destructive under `?repair=true`. A boolean forces that job to
/// answer wrongly for one of its two modes — `false` on something that
/// can delete files is actively misleading.
fn mutates(&self) -> Mutates {
Mutates::Never
}
/// `Some(..)` when `?repair=true` does something beyond a default run,
/// describing what it ADDS; `None` when the flag is inert.
///
/// One method rather than a `supports_repair` boolean plus prose: its
/// presence drives whether the UI offers the toggle, its content drives
/// the confirmation text. A boolean would leave the frontend to invent
/// wording for a destructive action it does not understand.
///
/// Independent of [`Self::mutates`], not derived from it — the import
/// jobs are [`Mutates::Always`] *and* repair-capable, inserting rows on a
/// plain run and additionally unlinking files under repair.
fn repair_description(&self) -> Option<&'static str> {
None
}
/// Long-running scan. See trait-level doc for the contract.
///
/// `store` — bound to THIS run (a single row in
@@ -361,7 +402,15 @@ pub trait JobStore: Send + Sync {
/// `"stale_used_bytes"`, `"missing_blob"`). Never rename across
/// releases; new failure modes get new values.
///
/// `severity` — one of `"data_loss"`, `"inconsistent"`, `"anomaly"`.
/// `severity` — one of:
/// - `"data_loss"` — bytes / rows unreachable or gone.
/// - `"inconsistent"` — counters or materialised values wrong,
/// content intact.
/// - `"anomaly"` — surprising state worth surfacing, no known impact.
/// This is the level the admin panel labels "notices"; there is no
/// separate `notice` severity, and a job that acted on what it found
/// says so in `detail` rather than in a fourth severity that would
/// render identically.
///
/// `resource_id` — the file / folder / drive / blob the finding
/// pertains to. `None` for run-wide findings (e.g. "backend
@@ -993,6 +1042,21 @@ impl JobHandler for RecoverableAdapter {
// downstream.
true
}
// The registry only ever sees `dyn JobHandler`, so the tenant's own
// metadata has to be forwarded through the wrapper or it is invisible
// to `GET /api/admin/jobs`. Silently returning the JobHandler defaults
// here would leave every recoverable job undescribed and reported as
// read-only — including ones that delete files.
fn description(&self) -> &'static str {
self.inner.description()
}
fn mutates(&self) -> Mutates {
self.inner.mutates()
}
fn repair_description(&self) -> Option<&'static str> {
self.inner.repair_description()
}
}
// ─── Ergonomics: JobRegistry extension for recoverable jobs ─────────────────
@@ -1504,6 +1568,47 @@ mod tests {
// ─── Tests ─────────────────────────────────────────────────────────────
/// The registry only ever sees `dyn JobHandler`, so a recoverable
/// tenant's metadata reaches `GET /api/admin/jobs` only if the adapter
/// forwards it. Falling back to the `JobHandler` defaults here would
/// report every recoverable job as undescribed and read-only —
/// including the imports, which delete files under repair.
#[tokio::test]
async fn adapter_forwards_job_metadata_from_inner_handler() {
struct Annotated;
#[async_trait]
impl RecoverableJobHandler for Annotated {
fn name(&self) -> &str {
"annotated"
}
async fn run_resumable(
&self,
_store: &dyn JobStore,
_args: &JobRunArgs,
_resume_cursor: Option<Vec<u8>>,
) -> RunOutcome {
RunOutcome::completed()
}
fn description(&self) -> &'static str {
"walks a thing"
}
fn mutates(&self) -> Mutates {
Mutates::OnRepairOnly
}
fn repair_description(&self) -> Option<&'static str> {
Some("fixes the thing")
}
}
let provider: Arc<dyn JobStoreProvider> = Arc::new(MemProvider::new());
let adapter = RecoverableAdapter::new(Arc::new(Annotated), provider);
let as_handler: &dyn JobHandler = &adapter;
assert_eq!(as_handler.description(), "walks a thing");
assert_eq!(as_handler.mutates(), Mutates::OnRepairOnly);
assert_eq!(as_handler.repair_description(), Some("fixes the thing"));
}
#[tokio::test]
async fn fresh_run_completes_and_marks_status_completed() {
let provider = Arc::new(MemProvider::new());
src/infrastructure/scheduler/registry.rs
+99 -1
View File
@@ -20,7 +20,7 @@ use serde::Serialize;
use tokio::sync::{RwLock, Semaphore};
use super::handler::JobHandler;
use super::types::{JobOutcome, JobRunArgs};
use super::types::{JobOutcome, JobRunArgs, Mutates};
/// A registered job plus its runtime state. Held as `Arc<JobEntry>`
/// inside the registry so the engine can hold a snapshot across an
@@ -135,6 +135,13 @@ impl JobRegistry {
timeout: Option<Duration>,
) -> Result<(), RegisterError> {
let name = handler.name().to_string();
// A job declaring it mutates only under a flag it does not support
// is self-contradictory, and the UI would render it as safe with no
// way to reach the mutating path. Cheap to catch here, invisible
// otherwise.
if handler.mutates() == Mutates::OnRepairOnly && handler.repair_description().is_none() {
return Err(RegisterError::RepairOnlyWithoutRepair(name));
}
let mut guard = self.entries.write().await;
if guard.contains_key(&name) {
return Err(RegisterError::DuplicateName(name));
@@ -220,6 +227,9 @@ impl JobRegistry {
};
JobSummary {
name,
description: entry.handler.description(),
mutates: entry.handler.mutates(),
repair_description: entry.handler.repair_description(),
interval_ms: entry.interval.map(|d| d.as_millis() as u64),
next_run_at: state.next_run_at,
last_run_at,
@@ -283,6 +293,11 @@ impl Default for JobRegistry {
pub enum RegisterError {
#[error("job name already registered: {0}")]
DuplicateName(String),
#[error(
"job {0} declares mutates = OnRepairOnly but no repair_description() — \
it claims to mutate only under a flag it does not support"
)]
RepairOnlyWithoutRepair(String),
}
/// Per-job row in the `GET /api/admin/jobs` response.
@@ -304,6 +319,17 @@ pub enum RegisterError {
#[derive(Debug, Clone, Serialize)]
pub struct JobSummary {
pub name: String,
/// One or two sentences on what the job does. Empty for jobs that
/// haven't declared one yet — the UI omits the line rather than
/// rendering a blank block.
#[serde(skip_serializing_if = "str::is_empty")]
pub description: &'static str,
pub mutates: Mutates,
/// `Some` iff the job does something extra under `?repair=true`.
/// Presence is what gates the repair toggle in the UI; the string
/// is the confirmation text.
#[serde(skip_serializing_if = "Option::is_none")]
pub repair_description: Option<&'static str>,
#[serde(skip_serializing_if = "Option::is_none")]
pub interval_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -393,6 +419,78 @@ mod tests {
assert!(matches!(err, RegisterError::DuplicateName(_)));
}
/// A job declaring `OnRepairOnly` without a `repair_description` has
/// no reachable mutating path — the UI gates the repair toggle on
/// that string's presence, so the job would render as safe and stay
/// read-only forever. Catch it at wiring time rather than let it read
/// as a working configuration.
#[tokio::test]
async fn repair_only_without_repair_description_rejected() {
struct Contradictory;
#[async_trait]
impl JobHandler for Contradictory {
fn name(&self) -> &str {
"contradictory"
}
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
JobOutcome::ok(0)
}
fn mutates(&self) -> Mutates {
Mutates::OnRepairOnly
}
// repair_description() left at its `None` default — the bug.
}
let reg = JobRegistry::new();
let err = reg
.try_register(Arc::new(Contradictory), None, None)
.await
.expect_err("OnRepairOnly without a repair_description must be rejected");
assert!(matches!(err, RegisterError::RepairOnlyWithoutRepair(_)));
}
/// The registry hands `dyn JobHandler` to the admin snapshot, so a
/// tenant's own metadata is only visible if it survives that erasure.
#[tokio::test]
async fn snapshot_carries_job_metadata() {
struct Described;
#[async_trait]
impl JobHandler for Described {
fn name(&self) -> &str {
"described"
}
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
JobOutcome::ok(0)
}
fn description(&self) -> &'static str {
"does a thing"
}
fn mutates(&self) -> Mutates {
Mutates::Always
}
fn repair_description(&self) -> Option<&'static str> {
Some("also deletes the thing")
}
}
let reg = JobRegistry::new();
reg.register(Arc::new(Described), None, None).await;
let snap = reg.snapshot().await;
let row = snap.iter().find(|j| j.name == "described").unwrap();
assert_eq!(row.description, "does a thing");
assert_eq!(row.mutates, Mutates::Always);
assert_eq!(row.repair_description, Some("also deletes the thing"));
// Undeclared jobs stay at the safe defaults so the panel can tell
// "read-only" from "not yet described" — empty string, not prose.
reg.register(handler("bare"), None, None).await;
let snap = reg.snapshot().await;
let bare = snap.iter().find(|j| j.name == "bare").unwrap();
assert_eq!(bare.description, "");
assert_eq!(bare.mutates, Mutates::Never);
assert!(bare.repair_description.is_none());
}
#[tokio::test]
#[should_panic(expected = "DI wiring bug")]
async fn register_panics_on_duplicate() {
src/infrastructure/scheduler/types.rs
+42
View File
@@ -167,10 +167,52 @@ impl fmt::Display for ErrCause {
}
}
/// When a job changes state.
///
/// Drives how the admin UI presents a trigger: `Never` earns a read-only
/// badge, `OnRepairOnly` is safe to run and warns only when the toggle is on,
/// `Always` warns regardless.
///
/// Three values rather than a boolean because there are three cases, and the
/// interesting one is conditional. `false` on a job that can delete files
/// under `?repair=true` is actively misleading; `true` on one that is
/// read-only by default is equally wrong. `OnRepairOnly` names the case a
/// boolean cannot, and it is where the recovery framework is heading —
/// discovery-only by default, mutation behind an explicit opt-in — so a
/// consistency tenant that later grows a repair arm changes this one value
/// and nothing else.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum Mutates {
/// Read-only under every flag. All consistency tenants.
Never,
/// Changes state on a plain run. GC, janitors, the import jobs.
Always,
/// Read-only by default; mutates only under `?repair=true`. Pairing this
/// with `repair_description() == None` is contradictory — a job claiming
/// it mutates only under a flag it does not support.
OnRepairOnly,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn mutates_serialises_snake_case() {
// The admin UI switches on these strings — a rename is a breaking
// change to the panel, not just to Rust callers.
assert_eq!(serde_json::to_string(&Mutates::Never).unwrap(), "\"never\"");
assert_eq!(
serde_json::to_string(&Mutates::Always).unwrap(),
"\"always\""
);
assert_eq!(
serde_json::to_string(&Mutates::OnRepairOnly).unwrap(),
"\"on_repair_only\""
);
}
#[test]
fn joboutcome_kind_label() {
assert_eq!(JobOutcome::ok(0).kind(), "ok");
src/infrastructure/services/backend_consistency_service.rs
+8
View File
@@ -147,6 +147,14 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
BACKEND_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Merge-joins the storage backend's blob enumeration against \
storage.blobs, both ordered by hash, so one pass yields the delta \
in both directions: bytes on the backend no DB row claims, and \
rows whose bytes are gone. Read-only — nothing is uploaded or \
deleted."
}
/// Approximate total: on a healthy install every backend blob
/// has a `storage.blobs` row, so the DB count is a proxy for
/// the backend count. The fraction deviating from 1.0 at run
src/infrastructure/services/backend_migration_service.rs
+16 -2
View File
@@ -59,8 +59,8 @@ use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::NamedStorageEntry;
use crate::common::errors::DomainError;
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
use crate::infrastructure::services::encrypted_blob_backend::{EncryptedBlobBackend, HeadCheck};
use crate::infrastructure::services::entry_backend::{
@@ -188,6 +188,20 @@ impl RecoverableJobHandler for BackendMigrationService {
BACKEND_MIGRATION_JOB_NAME
}
fn description(&self) -> &'static str {
"Copies every blob payload from the backend the server booted with \
to the one the current storage settings describe. Covers legacy \
whole-file blobs and CDC chunks in a single walk. Resumable — a \
paused or crashed run continues from its cursor rather than \
restarting."
}
/// Writes bytes to the target backend. Source bytes are left in place —
/// the copy is additive, so an aborted migration loses nothing.
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Definitive count — one row per blob. `SELECT COUNT(*) FROM
/// storage.blobs` on a modern PG is a sub-second index-only scan
/// even at millions of rows.
src/infrastructure/services/backend_rotate_service.rs
+16 -2
View File
@@ -62,8 +62,8 @@ use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::NamedStorageEntry;
use crate::common::migration_progress::MigrationProgress;
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
use crate::infrastructure::services::encrypted_blob_backend::BlobFormat;
use crate::infrastructure::services::entry_backend::build_entry_backend_typed;
@@ -135,6 +135,20 @@ impl RecoverableJobHandler for BackendRotateService {
BACKEND_ROTATE_JOB_NAME
}
fn description(&self) -> &'static str {
"Brings every blob's on-disk format in line with the storage \
entry's current head key: encrypts plaintext, re-encrypts under a \
rotated key, decrypts when the head is 'none', and upgrades \
legacy blobs to v1. Blobs already in the right format are skipped, \
so re-running after a key change is cheap."
}
/// Rewrites blobs **in place**. Unlike a migration this has no additive
/// fallback — the previous ciphertext is gone once a blob is rewritten.
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Definitive count — one row per blob. Same query as
/// `backend_migration::count_total`; the two walk the same rows.
async fn count_total(&self) -> Option<u64> {
src/infrastructure/services/blobs_consistency_service.rs
+25 -2
View File
@@ -67,8 +67,8 @@ use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, Ref
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::NamedStorageEntry;
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
use crate::infrastructure::services::entry_backend::build_entry_backend;
@@ -223,6 +223,29 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
BLOBS_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Walks storage.blobs and checks each row against the reference- \
counting invariants dedup_gc relies on, and probes the backend \
once per row for missing bytes. That probe costs one backend \
call per blob — backend_consistency finds the same missing \
bytes in a single merge-join pass, and orphaned bytes too, so \
prefer it on large installs. Add ?deep=true to re-read and \
re-hash every blob for bit-rot; that is a full read of storage."
}
fn mutates(&self) -> Mutates {
Mutates::OnRepairOnly
}
fn repair_description(&self) -> Option<&'static str> {
Some(
"Rewrites drifted ref_count values to the recomputed truth. \
Does not delete blobs or resurrect missing bytes — an \
over-counted blob simply becomes eligible for the next \
dedup_gc sweep.",
)
}
/// Definitive count. `storage.blobs` PK scan is index-only;
/// even at millions of rows it's sub-second on modern PG.
async fn count_total(&self) -> Option<u64> {
src/infrastructure/services/consistency_batch_service.rs
+23 -1
View File
@@ -59,7 +59,7 @@ use std::sync::{Arc, Weak};
use async_trait::async_trait;
use serde_json::json;
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs, Mutates};
pub const CONSISTENCY_BATCH_JOB_NAME: &str = "consistency_batch";
@@ -90,6 +90,28 @@ impl JobHandler for ConsistencyBatch {
CONSISTENCY_BATCH_JOB_NAME
}
fn description(&self) -> &'static str {
"Runs every registered consistency check in sequence — one click \
for 'check everything'. New tenants are picked up automatically \
by name, so nothing needs updating here when one is added. Flags \
are forwarded to each sub-job."
}
/// Read-only on a plain run because every tenant it dispatches is, but
/// `?repair=true` reaches whichever of them act on it — so the batch
/// inherits the strongest mode any sub-job can be put into.
fn mutates(&self) -> Mutates {
Mutates::OnRepairOnly
}
fn repair_description(&self) -> Option<&'static str> {
Some(
"Forwards ?repair=true to every sub-check, so the ones that \
support it fix what they find (today: refcount drift on blobs \
and manifests) instead of only reporting it.",
)
}
async fn run(&self, args: &JobRunArgs) -> JobOutcome {
// Upgrade the Weak. Only fails if the registry has been
// dropped — which can only happen during process shutdown,
src/infrastructure/services/dedup_service.rs
+14
View File
@@ -3675,6 +3675,20 @@ impl crate::infrastructure::scheduler::JobHandler for DedupService {
DEDUP_GC_JOB_NAME
}
fn description(&self) -> &'static str {
"Reclaims blobs and chunk manifests that no file, thumbnail or \
preview references any more, once they are past the orphan grace \
window. Trash cleanup already runs this as its tail step; \
triggering it here is for reclaiming immediately rather than at \
the next tick. Add ?force=true to skip the grace window."
}
/// Deletes bytes. `force` is its accelerator, not a repair flag —
/// there is nothing this job reports without also acting on it.
fn mutates(&self) -> crate::infrastructure::scheduler::Mutates {
crate::infrastructure::scheduler::Mutates::Always
}
/// Runs one `garbage_collect` sweep — the same reclamation that
/// `TrashCleanupService` invokes inline as its tail step, exposed
/// through the scheduler so operators can trigger it uniformly via
src/infrastructure/services/drives_consistency_service.rs
+7
View File
@@ -72,6 +72,13 @@ impl RecoverableJobHandler for DrivesConsistencyCheck {
DRIVES_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Compares each drive's cached used_bytes against the actual sum of \
its file sizes and reports the drift. Read-only — usage_reconcile \
is what corrects the counter; this surfaces WHEN it drifts so the \
cause can be traced (missed delta, silent failure, race)."
}
/// Definitive count — one row per drive, table is tiny (dozens per
/// install), COUNT(*) is trivially fast. Enables progress bar on
/// the admin UI.
src/infrastructure/services/files_consistency_service.rs
+7
View File
@@ -148,6 +148,13 @@ impl RecoverableJobHandler for FilesConsistencyCheck {
FILES_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Walks storage.files and reports rows whose parent-folder state, \
blob reference or denormalised size has drifted from what the \
join with folders and blobs says is true. Read-only — the fixes \
live in other jobs (trash cascade, dedup_gc, blob resurrection)."
}
/// Definitive count — one row per file. This is the largest table
/// of the trio (millions on big installs); COUNT(*) is still an
/// index-only scan but can take ~seconds. The tradeoff is worth
src/infrastructure/services/folders_consistency_service.rs
+8
View File
@@ -120,6 +120,14 @@ impl RecoverableJobHandler for FoldersConsistencyCheck {
FOLDERS_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Walks storage.folders and reports rows whose materialised path \
and lpath have drifted from what walking the parent_id chain \
produces. Any write path that bypasses the ltree cascade trigger \
can leave these wrong, which silently breaks subtree queries. \
Read-only."
}
/// Definitive count — one row per folder. Larger table than drives
/// but the COUNT(*) is still index-only on PG. On multi-million-row
/// deployments this is ~100ms at run start; acceptable given the
src/infrastructure/services/grant_cleanup_service.rs
+13 -1
View File
@@ -23,7 +23,7 @@ use tracing::{error, info};
use crate::application::ports::authorization_ports::AuthorizationEngine;
use crate::common::errors::DomainError;
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs, Mutates};
use crate::infrastructure::services::pg_acl_engine::PgAclEngine;
use async_trait::async_trait;
@@ -122,6 +122,18 @@ impl JobHandler for GrantCleanupService {
GRANT_CLEANUP_JOB_NAME
}
fn description(&self) -> &'static str {
"Deletes expired role grants once they are past the retention \
window. Expired grants never leak permission — every AuthZ check \
filters on expires_at — they just accumulate. The window keeps \
'what happened to my access?' answerable for a few weeks after \
expiry."
}
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Runs one purge. `count` on the returned `JobOutcome::Ok` is
/// the number of `role_grants` rows physically deleted;
/// `extra.grace_days` records which grace was applied so admin
src/infrastructure/services/manifests_consistency_service.rs
+22 -2
View File
@@ -46,8 +46,8 @@ use sqlx::PgPool;
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
pub const MANIFESTS_CONSISTENCY_JOB_NAME: &str = "manifests_consistency";
@@ -139,6 +139,26 @@ impl RecoverableJobHandler for ManifestsConsistencyCheck {
MANIFESTS_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Reconciles storage.chunk_manifests.ref_count against its actual \
referrers. There are two reference counters — a chunk reference \
lands on storage.blobs.ref_count, a whole-Blob reference on the \
manifest — and only the first was ever verified; this covers the \
other half."
}
fn mutates(&self) -> Mutates {
Mutates::OnRepairOnly
}
fn repair_description(&self) -> Option<&'static str> {
Some(
"Rewrites drifted manifest ref_count values to the recomputed \
truth. Nothing is deleted here — a corrected count only makes \
the manifest eligible for a later dedup_gc sweep.",
)
}
async fn count_total(&self) -> Option<u64> {
let row: Result<(i64,), sqlx::Error> =
sqlx::query_as("SELECT COUNT(*) FROM storage.chunk_manifests")
src/infrastructure/services/satellites_consistency_service.rs
+9
View File
@@ -204,6 +204,15 @@ impl RecoverableJobHandler for SatellitesConsistencyCheck {
SATELLITES_CONSISTENCY_JOB_NAME
}
fn description(&self) -> &'static str {
"Walks both satellite tables — content_derived_blobs (thumbnails \
keyed by source content) and file_attached_blobs (previews keyed \
by file) — and reports mappings whose source or target no longer \
exists. Nothing else finds these: every other job reasons from a \
Blob outwards, and a satellite row pointing at a deleted source \
breaks none of their invariants. Read-only."
}
async fn count_total(&self) -> Option<u64> {
sqlx::query_as::<_, (i64,)>(
"SELECT (SELECT COUNT(*) FROM storage.content_derived_blobs)
src/infrastructure/services/session_cleanup_service.rs
+12 -1
View File
@@ -35,7 +35,7 @@ use tracing::{error, info};
use crate::domain::repositories::session_repository::SessionRepository;
use crate::infrastructure::repositories::SessionPgRepository;
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs, Mutates};
/// How long a session row survives past its `expires_at` before this
/// janitor deletes it. Enough time for a security review of a
@@ -84,6 +84,17 @@ impl JobHandler for SessionCleanupService {
Self::JOB_NAME
}
fn description(&self) -> &'static str {
"Deletes session rows long past their expiry. They cannot \
authenticate — expiry is checked at every auth path — but the row \
keeps a forensic trail (which user, from which IP, minted how) \
for a retention window after expiry, then becomes dead weight."
}
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Runs one bulk-delete of long-expired session rows. `count` on
/// the returned `JobOutcome::Ok` is the number of rows dropped
/// this tick; `extra` records the retention window operators can
src/infrastructure/services/thumb_attached_import_service.rs
+64 -18
View File
@@ -46,8 +46,8 @@ use uuid::Uuid;
use crate::application::ports::thumbnail_ports::ThumbnailSize;
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
use crate::infrastructure::services::dedup_service::DedupService;
// The readback-then-unlink rule is shared, not copied: two versions of it
@@ -167,6 +167,28 @@ impl RecoverableJobHandler for ThumbAttachedImport {
THUMB_ATTACHED_IMPORT_JOB_NAME
}
fn description(&self) -> &'static str {
"Migrates USER-UPLOADED previews (ext-{file_id}.jpg) into \
file-keyed blob storage. Until a row exists, copying a file loses \
its preview: the sidecar is keyed by file id and no copy path \
duplicates it. These bytes have no server-side render path, so \
unlike rendered thumbnails they cannot be regenerated."
}
fn mutates(&self) -> Mutates {
Mutates::Always
}
fn repair_description(&self) -> Option<&'static str> {
Some(
"Also DELETES each sidecar once its replacement has been read \
back. Previews whose file no longer exists are deleted without \
a readback — nothing can reference them again. Irreversible, \
and these bytes cannot be regenerated, so the readback is the \
only safeguard.",
)
}
async fn count_total(&self) -> Option<u64> {
let mut total = 0u64;
for size in ThumbnailSize::all() {
@@ -308,10 +330,12 @@ impl RecoverableJobHandler for ThumbAttachedImport {
// there is no row and no blob to read back, and nothing to
// regenerate from either.
orphaned += 1;
let mut removed = false;
if delete_imported {
let path = self.thumbnails_root.join(&dir_name).join(&name);
if fs::remove_file(&path).await.is_ok() {
deleted += 1;
removed = true;
// Explicit: nothing to verify against, so this
// bypasses verify_and_unlink. Worth auditing
// loudest of all — these bytes were
@@ -325,22 +349,35 @@ impl RecoverableJobHandler for ThumbAttachedImport {
&path,
);
}
} else {
record_or_log(
store,
THUMB_ATTACHED_IMPORT_JOB_NAME,
"attached_sidecar_orphan",
"anomaly",
None,
serde_json::json!({
"path": position,
"file_id": file_id_str,
"note": "no storage.files row; unimportable, and deleted on a \
repair run since nothing can reference it again",
}),
)
.await;
}
// Recorded in BOTH modes — see the twin in
// thumb_derived_import. Deleting a non-regenerable
// user-uploaded preview and reporting nothing is the
// worst version of this: the one outcome an operator
// needs in the run drawer was the one it withheld.
record_or_log(
store,
THUMB_ATTACHED_IMPORT_JOB_NAME,
"attached_sidecar_orphan",
// `anomaly` renders as "notices"; `detail.deleted`
// is what says whether the run acted. See the
// derived twin.
"anomaly",
None,
serde_json::json!({
"path": position,
"file_id": file_id_str,
"deleted": removed,
"note": if removed {
"no storage.files row; sidecar was unimportable and has been \
deleted — nothing can reference it again"
} else {
"no storage.files row; unimportable, and deleted on a repair \
run since nothing can reference it again"
},
}),
)
.await;
} else {
let path = self.thumbnails_root.join(&dir_name).join(&name);
match fs::read(&path).await {
@@ -458,7 +495,16 @@ impl RecoverableJobHandler for ThumbAttachedImport {
{unverified} kept unverified"
);
RunOutcome::completed()
// Same reasoning as the derived twin: what the run did belongs on
// the run row, not only in the process log.
RunOutcome::completed_with(serde_json::json!({
"imported": imported,
"already_present": already,
"deleted": deleted,
"unverified": unverified,
"orphaned": orphaned,
"failed": failed,
}))
}
}
src/infrastructure/services/thumb_derived_import_service.rs
+70 -18
View File
@@ -41,8 +41,8 @@ use tokio::fs;
use crate::application::ports::thumbnail_ports::{ThumbnailFormat, ThumbnailSize};
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
RunOutcome, RunStatus, record_or_log,
};
use crate::infrastructure::services::dedup_service::DedupService;
@@ -230,6 +230,29 @@ impl RecoverableJobHandler for ThumbDerivedImport {
THUMB_DERIVED_IMPORT_JOB_NAME
}
fn description(&self) -> &'static str {
"Migrates server-rendered thumbnails from the legacy .thumbnails/ \
directory into content-addressed blob storage. Local-disk sidecars \
are invisible to other instances and are not carried by a backend \
migration; importing them is what lets that directory be deleted."
}
/// `Always`: a plain run inserts rows and writes blobs. Repair-capable on
/// top of that, which is why the two are independent.
fn mutates(&self) -> Mutates {
Mutates::Always
}
fn repair_description(&self) -> Option<&'static str> {
Some(
"Also DELETES each sidecar once its replacement has been read \
back from blob storage, and removes the directory when empty. \
Files whose source no longer exists are deleted without a \
readback — they cannot be imported and nothing can reference \
them. Irreversible.",
)
}
async fn count_total(&self) -> Option<u64> {
let mut total = 0u64;
for size in ThumbnailSize::all() {
@@ -390,10 +413,12 @@ impl RecoverableJobHandler for ThumbDerivedImport {
// since there is nothing to read back and nothing to
// regenerate from.
dead_source += 1;
let mut removed = false;
if delete_imported {
let path = self.thumbnails_root.join(dir_name).join(&name);
if fs::remove_file(&path).await.is_ok() {
deleted += 1;
removed = true;
// Audited explicitly: this unlink bypasses
// verify_and_unlink, which has nothing to verify
// against here.
@@ -405,22 +430,39 @@ impl RecoverableJobHandler for ThumbDerivedImport {
&path,
);
}
} else {
record_or_log(
store,
THUMB_DERIVED_IMPORT_JOB_NAME,
"sidecar_source_gone",
"anomaly",
None,
serde_json::json!({
"path": position,
"source_hash": hash,
"note": "source Blob no longer exists; the thumbnail is \
unimportable and is deleted on a repair run",
}),
)
.await;
}
// Recorded in BOTH modes. The finding used to be the
// `else` of the deletion, so a repair run unlinked files
// and reported a clean sweep — the audit stream held the
// only trace, and the run drawer an operator actually
// looks at said zero. A deletion is the outcome most
// worth a finding, not least.
record_or_log(
store,
THUMB_DERIVED_IMPORT_JOB_NAME,
"sidecar_source_gone",
// `anomaly` in both modes — it is what the panel
// renders as "notices", and `detail.deleted` carries
// whether the run left the sidecar alone or removed
// it. A separate severity for the deleted case would
// render identically and split one badge across two
// values.
"anomaly",
None,
serde_json::json!({
"path": position,
"source_hash": hash,
"deleted": removed,
"note": if removed {
"source Blob no longer exists; sidecar was unimportable and \
has been deleted"
} else {
"source Blob no longer exists; the thumbnail is unimportable \
and is deleted on a repair run"
},
}),
)
.await;
} else {
let path = self.thumbnails_root.join(dir_name).join(&name);
match fs::read(&path).await {
@@ -553,7 +595,17 @@ impl RecoverableJobHandler for ThumbDerivedImport {
{dead_source} skipped (source gone)"
);
RunOutcome::completed()
// Surfaced on the run row, not just in the process log. A repair run
// that unlinks hundreds of files while reporting only a finding
// total tells an operator nothing about what it did with them.
RunOutcome::completed_with(serde_json::json!({
"imported": imported,
"already_present": already,
"deleted": deleted,
"unverified": unverified,
"dead_source": dead_source,
"failed": failed,
}))
}
}
src/infrastructure/services/trash_cleanup_service.rs
+12 -1
View File
@@ -6,7 +6,7 @@ use tracing::{debug, error, info, instrument};
use crate::common::errors::Result;
use crate::domain::repositories::trash_repository::TrashRepository;
use crate::infrastructure::repositories::pg::trash_db_repository::TrashDbRepository;
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs, Mutates};
use crate::infrastructure::services::dedup_service::DedupService;
use async_trait::async_trait;
@@ -177,6 +177,17 @@ impl JobHandler for TrashCleanupService {
Self::JOB_NAME
}
fn description(&self) -> &'static str {
"Permanently deletes trashed items past the retention window, then \
runs a dedup GC sweep as its tail step to reclaim blobs the \
deletions dropped to zero references. This is the periodic tick \
that keeps storage bounded."
}
fn mutates(&self) -> Mutates {
Mutates::Always
}
/// Runs one bulk-delete-expired + GC sweep. `count` on the returned
/// `JobOutcome::Ok` is the total number of rows this tick removed
/// from the trash (files + folders); `extra` carries GC reclaim
tests/api/admin_jobs.hurl
+34
View File
@@ -113,6 +113,40 @@ jsonpath "$[*].name" contains "consistency_batch"
jsonpath "$[*].name" contains "backend_migration"
jsonpath "$[*].name" contains "backend_rotate"
# Job metadata — `description` / `mutates` / `repair_description`.
# The admin panel keys the read-only badge and the repair toggle off
# these, so a handler that stops declaring them degrades the UI
# silently: a mutating job renders as safe to click, and a repair-
# capable one loses its toggle entirely. That second failure is the
# bug this replaced — a name-based allowlist in the panel that never
# grew past the two refcount tenants, leaving the thumbnail imports
# unrunnable in repair mode from the UI.
#
# Per-job pins use scalar equality on a single-match filter — NOT
# `count`, which trips Hurl's "filter matched one item → scalar, not
# list" quirk. See memory `hurl-jsonpath-filter-empty-result`.
#
# `mutates` is a closed enum the UI switches on, so all three wire
# spellings are pinned; a rename would break the panel silently.
jsonpath "$..mutates" contains "never"
jsonpath "$..mutates" contains "always"
jsonpath "$..mutates" contains "on_repair_only"
# Read-only tenant — safe to trigger, earns the read-only badge.
jsonpath "$[?(@.name=='files_consistency')].mutates" == "never"
# Repairs refcounts under ?repair=true, read-only otherwise.
jsonpath "$[?(@.name=='blobs_consistency')].mutates" == "on_repair_only"
# Destructive on a plain run AND repair-capable — the combination a
# boolean could not express, and the reason `Mutates` has three values
# rather than two.
jsonpath "$[?(@.name=='thumb_derived_import')].mutates" == "always"
# Floors, not totals: every job registered today declares a
# description, and five declare a repair arm (both imports, both
# refcount tenants, consistency_batch). New tenants only push these
# up. Registration itself already rejects `on_repair_only` without a
# repair_description, so the contradictory pairing can't reach here.
jsonpath "$..description" count >= 10
jsonpath "$..repair_description" count >= 5
# ─────────────────────────────────────────────────────────────
# Step 3 — Trigger `trash_cleanup`. Envelope shape: