Edouard Vanbelle
1ea3826660
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
337 lines
17 KiB
Plaintext
337 lines
17 KiB
Plaintext
# =============================================================
|
|
# OxiCloud — JobRegistry admin surface
|
|
# =============================================================
|
|
# Pins `GET /api/admin/jobs` and `POST /api/admin/jobs/{name}/trigger`
|
|
# — the production admin surface for the periodic-job scheduler
|
|
# (`docs/plan/job-registry.md` Part 1).
|
|
#
|
|
# Coverage:
|
|
# * Listing returns the four registered tenants
|
|
# (trash_cleanup, usage_reconcile, dedup_gc, grant_cleanup).
|
|
# * Scheduled jobs report `interval_ms`; on-demand jobs
|
|
# (`dedup_gc`) omit it via `skip_serializing_if=None`.
|
|
# * Triggering a job updates its `last_outcome` in the next list.
|
|
# * Unknown job → 404 (anti-enum on the trigger URL).
|
|
# * Non-admin caller → 403 from the admin middleware layer
|
|
# (no bespoke check in `list_jobs`/`trigger_job` itself).
|
|
# =============================================================
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Setup — admin login + jobs_bob (re-)provisioning
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/auth/login
|
|
Content-Type: application/json
|
|
{ "username": "{{username}}", "password": "{{password}}" }
|
|
|
|
HTTP 200
|
|
[Captures]
|
|
admin_token: jsonpath "$.access_token"
|
|
|
|
|
|
# Anti-enum registration.
|
|
POST {{base_url}}/api/auth/register
|
|
Content-Type: application/json
|
|
{
|
|
"username": "jobs_bob",
|
|
"email": "jobs_bob@example.com",
|
|
"password": "JobsBobPassword1!"
|
|
}
|
|
|
|
HTTP 200
|
|
|
|
|
|
POST {{base_url}}/api/auth/login
|
|
Content-Type: application/json
|
|
{ "username": "jobs_bob", "password": "JobsBobPassword1!" }
|
|
|
|
HTTP 200
|
|
[Captures]
|
|
bob_token: jsonpath "$.access_token"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 1 — Admin lists jobs. All four Part 1 tenants must appear.
|
|
# Scheduled jobs (trash_cleanup, usage_reconcile, grant_cleanup)
|
|
# report `interval_ms`; on-demand jobs (`dedup_gc`) omit it via
|
|
# serde's `skip_serializing_if = "Option::is_none"`.
|
|
#
|
|
# JSONPath idiom (per `feedback_hurl_jsonpath_filter_empty`): the
|
|
# single-match `[?(...)]` filter unwraps to a scalar so `count`
|
|
# fails; `count == 0` also fails on the zero-match case ("no
|
|
# value"). Aggregate `$..field` + `contains` is the reliable
|
|
# primitive Ed's memory endorses.
|
|
# ─────────────────────────────────────────────────────────────
|
|
GET {{base_url}}/api/admin/jobs
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
# Presence asserts — each scheduled tenant we EXPECT must be
|
|
# listed. Failure of any one means the registration wiring was
|
|
# dropped; adding a new tenant elsewhere never breaks these.
|
|
#
|
|
# Deliberately shape-not-cardinality: an earlier version of this
|
|
# step hardcoded `count == 12` / `count == 4` totals, which broke
|
|
# on EVERY new registration in unrelated PRs. Duplicate-name
|
|
# registration is already trapped at boot with a panic (see
|
|
# `TrashCleanupService::register` — the registry rejects same-name
|
|
# reinserts loudly), so the total-count gate added nothing that
|
|
# `contains` didn't already cover, at the cost of drive-by
|
|
# maintenance every quarter.
|
|
jsonpath "$[*].name" contains "trash_cleanup"
|
|
jsonpath "$[*].name" contains "usage_reconcile"
|
|
jsonpath "$[*].name" contains "dedup_gc"
|
|
jsonpath "$[*].name" contains "grant_cleanup"
|
|
jsonpath "$[*].name" contains "session_cleanup"
|
|
|
|
# Schema asserts — the response shape itself. Every scheduled
|
|
# tenant carries an `interval_ms` value; recursive-descent
|
|
# `contains` on the specific intervals we ship pins the
|
|
# non-default cadences without caring about the ordering or the
|
|
# total number of scheduled tenants.
|
|
# TrashCleanup / GrantCleanup / SessionCleanup → 24 h = 86_400_000 ms
|
|
# StorageReconcile → 600 s = 600_000 ms
|
|
jsonpath "$..interval_ms" contains 86400000
|
|
jsonpath "$..interval_ms" contains 600000
|
|
|
|
# Floor asserts — catastrophic-regression guard, not a fragile
|
|
# total. Bump the floors in lockstep with the `contains "..."`
|
|
# presence checks above: 5 named tenants total (trash_cleanup,
|
|
# usage_reconcile, dedup_gc, grant_cleanup, session_cleanup) →
|
|
# ≥ 5 `running` entries; 4 of those are scheduled (dedup_gc is
|
|
# on-demand and carries no `interval_ms`) → ≥ 4 `interval_ms`
|
|
# values. New unrelated tenants only PUSH the actual counts UP,
|
|
# never below these floors, so they don't break on drive-by
|
|
# additions.
|
|
jsonpath "$..running" count >= 5
|
|
jsonpath "$..interval_ms" count >= 4
|
|
jsonpath "$[*].name" contains "drives_consistency"
|
|
jsonpath "$[*].name" contains "folders_consistency"
|
|
jsonpath "$[*].name" contains "files_consistency"
|
|
jsonpath "$[*].name" contains "consistency_batch"
|
|
jsonpath "$[*].name" contains "backend_migration"
|
|
jsonpath "$[*].name" contains "backend_rotate"
|
|
|
|
# Job metadata — `description` / `mutates` / `repair_description`.
|
|
# The admin panel keys the read-only badge and the repair toggle off
|
|
# these, so a handler that stops declaring them degrades the UI
|
|
# silently: a mutating job renders as safe to click, and a repair-
|
|
# capable one loses its toggle entirely. That second failure is the
|
|
# bug this replaced — a name-based allowlist in the panel that never
|
|
# grew past the two refcount tenants, leaving the thumbnail imports
|
|
# unrunnable in repair mode from the UI.
|
|
#
|
|
# Per-job pins use scalar equality on a single-match filter — NOT
|
|
# `count`, which trips Hurl's "filter matched one item → scalar, not
|
|
# list" quirk. See memory `hurl-jsonpath-filter-empty-result`.
|
|
#
|
|
# `mutates` is a closed enum the UI switches on, so all three wire
|
|
# spellings are pinned; a rename would break the panel silently.
|
|
jsonpath "$..mutates" contains "never"
|
|
jsonpath "$..mutates" contains "always"
|
|
jsonpath "$..mutates" contains "on_repair_only"
|
|
# Read-only tenant — safe to trigger, earns the read-only badge.
|
|
jsonpath "$[?(@.name=='files_consistency')].mutates" == "never"
|
|
# Repairs refcounts under ?repair=true, read-only otherwise.
|
|
jsonpath "$[?(@.name=='blobs_consistency')].mutates" == "on_repair_only"
|
|
# Destructive on a plain run AND repair-capable — the combination a
|
|
# boolean could not express, and the reason `Mutates` has three values
|
|
# rather than two.
|
|
jsonpath "$[?(@.name=='thumb_derived_import')].mutates" == "always"
|
|
# Floors, not totals: every job registered today declares a
|
|
# description, and five declare a repair arm (both imports, both
|
|
# refcount tenants, consistency_batch). New tenants only push these
|
|
# up. Registration itself already rejects `on_repair_only` without a
|
|
# repair_description, so the contradictory pairing can't reach here.
|
|
jsonpath "$..description" count >= 10
|
|
jsonpath "$..repair_description" count >= 5
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 3 — Trigger `trash_cleanup`. Envelope shape:
|
|
# `{ ok, outcome: { outcome, count, extra } }`.
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/admin/jobs/trash_cleanup/trigger
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
jsonpath "$.ok" == true
|
|
jsonpath "$.outcome.outcome" == "ok"
|
|
# Trash may or may not have expired items; count is a non-negative
|
|
# integer either way. `isNumber` on a 1-element JSONPath extract
|
|
# needs the filter idiom too — assert exists via the wrapper key.
|
|
jsonpath "$.outcome.count" exists
|
|
jsonpath "$.outcome.extra.files_purged" exists
|
|
jsonpath "$.outcome.extra.folders_purged" exists
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4 — Re-list. At least one job (`trash_cleanup`, just
|
|
# triggered above) now has a `last_outcome` populated
|
|
# with `outcome=ok`. Aggregate JSONPath — recursive
|
|
# descent collects every last_outcome.outcome value
|
|
# across the response.
|
|
# ─────────────────────────────────────────────────────────────
|
|
GET {{base_url}}/api/admin/jobs
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
jsonpath "$..last_outcome.outcome" contains "ok"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4b — Trigger `folders_consistency`. Second Part 2
|
|
# recoverable tenant. Goes through the same
|
|
# RecoverableAdapter → PgJobStoreProvider path as
|
|
# drives_consistency (opens a run row, walks the
|
|
# `storage.folders` cursor, marks Completed). Success
|
|
# envelope shape identical.
|
|
#
|
|
# Captures the run_id so Step 4b-findings can pin the
|
|
# `GET /runs/{id}/findings` endpoint.
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/admin/jobs/folders_consistency/trigger
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Captures]
|
|
folders_run_id: jsonpath "$.outcome.extra.run_id"
|
|
[Asserts]
|
|
jsonpath "$.ok" == true
|
|
jsonpath "$.outcome.outcome" == "ok"
|
|
jsonpath "$.outcome.count" exists
|
|
jsonpath "$.outcome.extra.completed" == true
|
|
jsonpath "$.outcome.extra.run_id" exists
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4b-findings — List findings for the run we just kicked.
|
|
# The response body is a JSON array (possibly empty on a clean
|
|
# test DB — the fresh Hurl DB has no folder-tree drift). Assert:
|
|
# * 200 on a real run_id.
|
|
# * response is an array (isCollection covers both empty + non-
|
|
# empty). Structural shape of individual finding rows is pinned
|
|
# by the drives_consistency_service integration test which seeds
|
|
# drift and asserts kind/severity/detail — not repeated here.
|
|
# ─────────────────────────────────────────────────────────────
|
|
GET {{base_url}}/api/admin/jobs/folders_consistency/runs/{{folders_run_id}}/findings
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
jsonpath "$" isCollection
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4b-findings-404 — Findings for a run_id that doesn't
|
|
# exist. Endpoint returns 404, not 200 [] — otherwise a broken
|
|
# link from the admin UI would look like "no findings" instead
|
|
# of "run missing".
|
|
# ─────────────────────────────────────────────────────────────
|
|
GET {{base_url}}/api/admin/jobs/folders_consistency/runs/00000000-0000-0000-0000-000000000000/findings
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 404
|
|
[Asserts]
|
|
jsonpath "$.error" == "run not found"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4b2 — Trigger `files_consistency`. Third Part 2 recoverable
|
|
# tenant. Iterates `storage.files` and self-joins folder
|
|
# + blob. Same envelope shape as the earlier consistency
|
|
# tenants.
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/admin/jobs/files_consistency/trigger
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
jsonpath "$.ok" == true
|
|
jsonpath "$.outcome.outcome" == "ok"
|
|
jsonpath "$.outcome.count" exists
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 4c — Trigger `consistency_batch`. Coordinator (plain
|
|
# JobHandler) — snapshots the registry, filters names
|
|
# ending `_consistency`, sequentially triggers each.
|
|
# `extra.per_check` carries a per-child outcome map. Batch
|
|
# itself always returns ok — child failures live inside
|
|
# per_check. `?deep=true` propagates as `extra.deep`.
|
|
#
|
|
# NO assertion on `outcome.count`. The batch auto-discovers
|
|
# tenants via `.ends_with("_consistency")`, so a hardcoded
|
|
# total breaks every time one is added — it broke on
|
|
# `manifests_consistency` and again on
|
|
# `satellites_consistency`, each time asserting arithmetic
|
|
# rather than behaviour. Per the house rule: `contains` per
|
|
# item, never a total.
|
|
#
|
|
# What matters is that every child SUCCEEDED, which
|
|
# `err == 0` states directly and without a magic number, plus
|
|
# a named check per tenant below so a job silently dropping
|
|
# out of the batch is still caught.
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/admin/jobs/consistency_batch/trigger?deep=true
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 200
|
|
[Asserts]
|
|
jsonpath "$.ok" == true
|
|
jsonpath "$.outcome.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.deep" == true
|
|
# Zero failures, whatever the tenant count happens to be. `ok` is not
|
|
# asserted against a number for the same reason `count` is not.
|
|
jsonpath "$.outcome.extra.err" == 0
|
|
# per_check is keyed by child job name. `manifests_consistency` was added
|
|
# by the refcount_cascade fix — see docs/plan/derived-blobs.md and
|
|
# `[[bug_dual_refcount_divergence]]` for why the second counter needed
|
|
# its own tenant. Auto-picked by `consistency_batch` via `.ends_with(
|
|
# "_consistency")` (no explicit list in the batch service).
|
|
jsonpath "$.outcome.extra.per_check.drives_consistency.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.per_check.folders_consistency.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.per_check.manifests_consistency.outcome" == "ok"
|
|
jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok"
|
|
# Finds satellite mappings whose Blob is gone — the one class every
|
|
# refcount-based check above reports as healthy, because the row holds a
|
|
# valid reference with an exactly correct count while pinning an artifact
|
|
# that can never be reclaimed.
|
|
jsonpath "$.outcome.extra.per_check.satellites_consistency.outcome" == "ok"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 5 — Trigger a job that doesn't exist. 404 anti-enum on
|
|
# `JobRegistry::trigger` returning `None`.
|
|
# ─────────────────────────────────────────────────────────────
|
|
POST {{base_url}}/api/admin/jobs/no_such_job/trigger
|
|
Authorization: Bearer {{admin_token}}
|
|
|
|
HTTP 404
|
|
[Asserts]
|
|
jsonpath "$.error" == "job not registered"
|
|
jsonpath "$.name" == "no_such_job"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────
|
|
# Step 6 — Non-admin caller is denied by the `/api/admin/*`
|
|
# middleware layer. The handler itself has no bespoke
|
|
# role check — reaching it at all means the caller is
|
|
# admin (same shape as `dedup_admin_gate.hurl` pins).
|
|
# ─────────────────────────────────────────────────────────────
|
|
GET {{base_url}}/api/admin/jobs
|
|
Authorization: Bearer {{bob_token}}
|
|
|
|
HTTP 403
|
|
|
|
|
|
POST {{base_url}}/api/admin/jobs/trash_cleanup/trigger
|
|
Authorization: Bearer {{bob_token}}
|
|
|
|
HTTP 403
|