test(api): the blackhole trigger is detached — bound the poll, not the dispatch

Two fixes from Ed's run.

## The assertion I called load-bearing was measuring nothing

`backend_migration` is a DETACHED job: the trigger spawns the handler
and returns 202 in milliseconds, carrying no outcome and no run_id.
I had modelled it on `admin_jobs.hurl`, where the jobs are synchronous
and the response IS the outcome.

So `duration < 120000` on the trigger would have passed against the
ORIGINAL unbounded behaviour — it timed the dispatch, not the
migration. The one assert the file existed for proved nothing.

The bound is now a polling budget: `/runs?limit=1` with `retry: 60`,
`retry-interval: 2000`. 120s, then hurl fails on the last assert.
Against a 15-minute hang the row sits in `Running` and the budget
exhausts, which is the failure this file is for. `run_id` comes from
`$[0].id` (runs are `ORDER BY started_at DESC`), since the 202 body
has none.

## A count assert on a registry, again

`storage_multi_entry.hurl` asserted `$.entries count == 3` and
`s3_blackhole` made it 4. The failure reads "expected 3, got 4",
naming neither the entry that appeared nor whether it belonged.

Replaced with per-name `contains`, which is what a registry wants:
membership asserted per item, so declaring a new entry does not break
an unrelated file. Positional asserts stay — entry ORDER is a separate
property and a real one, since the boot fallback picks `[0]` when no
active pointer exists.

Its comment also said "Two entries declared" while asserting three:
the drift a count invites, visible in the same three lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-09-08 08:51:40 +02:00
parent 2ff8a77331
commit 49979108f7
2 changed files with 65 additions and 51 deletions
+50 -49
View File
@@ -67,68 +67,69 @@ admin_token: jsonpath "$.access_token"
# ─────────────────────────────────────────────────────────────
# Step 2 — Trigger the migration at the black hole.
# Step 2 — Dispatch the migration at the black hole.
#
# The trigger is synchronous, so the response IS the outcome.
# `backend_migration` is a DETACHED job: the handler is spawned and the
# call returns 202 immediately, so this response carries no outcome and
# no run_id. Step 3 polls for both.
#
# `outcome.outcome == "ok"` is deliberate and not a contradiction: a
# retryable pause is carried as `Ok` because the handler did its job and
# stopped cleanly at a checkpoint. `extra.paused` / `extra.retryable`
# are what distinguish it, which is exactly why the scheduler log line
# had to stop projecting a paused run as a clean one.
#
# The `duration` assert is the heart of this file. Everything else here
# would also pass against the old hanging behaviour — given fifteen
# minutes. This is the only assertion that fails if the bound is ever
# removed, so treat it as load-bearing rather than a performance nicety.
#
# 120s: comfortably above the observed ~31s (the SDK's own attempts
# stacked on the 10s connect timeout) and far below the ~15min the
# unbounded socket would take. Deliberately loose — a slow CI runner
# must not make this flaky, and the failure it guards against is three
# orders of magnitude away, not adjacent.
# That is also why the bound is asserted as a polling budget rather
# than with `duration` — the trigger returns in milliseconds no matter
# how long the backend hangs, so timing THIS request would prove
# nothing at all.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/jobs/backend_migration/trigger?storage=s3_blackhole
Authorization: Bearer {{admin_token}}
HTTP 202
[Asserts]
jsonpath "$.ok" == true
jsonpath "$.detached" == true
# ─────────────────────────────────────────────────────────────
# Step 3 — Poll until the run reaches Paused.
#
# THE LOAD-BEARING STEP. Runs are newest-first, so `$[0]` is ours.
#
# The retry budget IS the bounded-time assertion: 60 attempts × 2s =
# 120s, after which hurl fails with the last assert error. Against the
# old hanging behaviour the row would sit in `Running` for ~15 minutes
# and this step would exhaust its budget — which is the entire point of
# the file. Every other assertion here would eventually pass even
# unbounded; this one would not.
#
# 120s is deliberately loose: comfortably above the observed ~31s (the
# SDK's own attempts stacked on the 10s connect timeout) and three
# orders of magnitude below the unbounded socket. A slow CI runner must
# not make this flaky, and the failure it guards is nowhere near the
# threshold.
#
# `Paused`, not `Failed`: that distinction is the whole plan. Failed is
# terminal and needs a human; Paused resumes and finishes the migration
# once the backend returns.
# ─────────────────────────────────────────────────────────────
GET {{base_url}}/api/admin/jobs/backend_migration/runs?limit=1
Authorization: Bearer {{admin_token}}
[Options]
retry: 60
retry-interval: 2000
HTTP 200
[Captures]
blackhole_run_id: jsonpath "$.outcome.extra.run_id"
blackhole_run_id: jsonpath "$[0].id"
[Asserts]
duration < 120000
jsonpath "$.ok" == true
jsonpath "$.outcome.outcome" == "ok"
jsonpath "$.outcome.extra.paused" == true
jsonpath "$.outcome.extra.retryable" == true
jsonpath "$.outcome.extra.run_id" exists
jsonpath "$[0].status" == "Paused"
# The reason must name what went wrong, not merely that something did.
# An operator reading only this string has to be able to tell an
# unreachable backend from a wrong bucket — the first is worth waiting
# out, the second never resolves on its own.
jsonpath "$.outcome.extra.reason" contains "Transient Backend"
jsonpath "$.outcome.extra.reason" contains "target backend init"
# ─────────────────────────────────────────────────────────────
# Step 3 — The run row must agree with the outcome.
#
# `Paused`, not `Failed`: the distinction is the whole plan. Failed is
# terminal and needs a human to decide what happened; Paused resumes
# and finishes the migration once the backend returns.
#
# `completed_at` must be absent — the run is not over. A paused row
# carrying a completion timestamp would make every "how long did this
# take" query lie, and would read as finished in the admin panel.
# ─────────────────────────────────────────────────────────────
GET {{base_url}}/api/admin/jobs/backend_migration/runs/{{blackhole_run_id}}
Authorization: Bearer {{admin_token}}
HTTP 200
[Asserts]
jsonpath "$.status" == "Paused"
jsonpath "$.error_message" exists
jsonpath "$.error_message" contains "Transient Backend"
jsonpath "$.completed_at" not exists
jsonpath "$[0].error_message" contains "Transient Backend"
jsonpath "$[0].error_message" contains "target backend init"
# Absent, not null — the run is not over. A paused row carrying a
# completion timestamp would make every "how long did this take" query
# lie, and would read as finished in the admin panel.
jsonpath "$[0].completed_at" not exists
# ─────────────────────────────────────────────────────────────
+15 -2
View File
@@ -43,8 +43,21 @@ Authorization: Bearer {{admin_token}}
HTTP 200
[Asserts]
# Two entries declared, in _ENTRIES order.
jsonpath "$.entries" count == 3
# Membership, asserted per entry rather than by counting them.
#
# A `count ==` here breaks every time anyone declares a new entry in
# server.env, and the failure says "expected 3, got 4" — which names
# neither the entry that appeared nor whether it was supposed to. This
# is a registry; it grows. `s3_blackhole` was the fourth and broke it
# exactly this way.
jsonpath "$.entries[*].name" contains "local_main"
jsonpath "$.entries[*].name" contains "s3_stub"
jsonpath "$.entries[*].name" contains "azurite"
jsonpath "$.entries[*].name" contains "s3_blackhole"
# Position, which is a SEPARATE property from membership and is
# deliberately still pinned: entries come back in _ENTRIES order, and
# the boot fallback below depends on `[0]` being the first declared.
jsonpath "$.entries[0].name" == "local_main"
jsonpath "$.entries[1].name" == "s3_stub"
jsonpath "$.entries[2].name" == "azurite"