test(api): pin that an unreachable backend pauses in bounded time

The regression test `docs/plan/jobs-handling-recoverable-error.md`
§Testing asks for: assert the run reaches Paused, that `error_message`
names the cause, and that it does so in bounded time rather than
hanging.

## The endpoint has to HANG, not refuse

`s3_stub` already existed and points at `127.0.0.1:9999`, where nothing
listens. That connection is REFUSED — ECONNREFUSED, immediately — and
that path was never broken. A test built on it would pass with no
timeout configured anywhere, which is worse than no test: it would read
as coverage of exactly the failure it cannot see.

So `s3_blackhole` points at `192.0.2.1`, TEST-NET-1 (RFC 5737),
reserved for documentation and guaranteed unrouted. A SYN goes
unanswered — no RST, no ICMP — which is the failure that used to hang
until the OS abandoned TCP retransmission ~15 minutes later, with the
job neither running nor failed the whole time.

Ed's suggestion, and it is the right fixture: a server that never
answers is reproducible in a way that unplugging a cable is not.

## The load-bearing assertion is `duration`

Every other assert in the file would also pass against the old hanging
behaviour, given fifteen minutes. `duration < 120000` is the only one
that fails if the bound is ever removed. The threshold is deliberately
loose — three orders of magnitude from the failure it guards, so a slow
runner cannot make it flaky.

## Why it is safe in the shared suite

The run fails at `target.initialize()`, which is BEFORE
`migration_readonly` is engaged, so this file cannot leave the server
read-only for whatever runs next. A mid-copy failure would have held
the freeze — that is why this shape was chosen.

Teardown is mandatory rather than tidy: `open_or_start` picks up the
latest non-terminal row, so a Paused row left behind would be RESUMED
by the next `backend_migration` trigger in the suite, silently
retargeting an unrelated test at the black hole. The file cancels its
own run and asserts the row reached Cancelled.

Placed second-to-last. It is the slowest file in the suite by design —
it waits out an unreachable endpoint to prove the wait is bounded — so
that cost lands after everything else has reported. Azurite stays last
for the reason its own comment gives.

Not yet executed: the suite tears down containers and Ed usually has a
run in flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-09-08 08:38:08 +02:00
parent d99b718d43
commit 2ff8a77331
3 changed files with 203 additions and 1 deletions
+7
View File
@@ -227,6 +227,13 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
"$API_DIR/nfc_normalization.hurl" \
"$API_DIR/wopi_authz.hurl" \
"$API_DIR/wopi_shared_drive.hurl" \
`# Second-to-last. The slowest file in the suite BY DESIGN: it waits` \
`# out an unreachable endpoint (~31s) to prove the wait is bounded, so` \
`# that cost belongs at the end rather than in the middle. It leaves no` \
`# read-only freeze behind — the migration fails at target init, before` \
`# the gate is engaged — and cancels its own run, so the shared DB is` \
`# clean for whatever follows.` \
"$API_DIR/backend_migration_blackhole.hurl" \
`# LAST, deliberately — and kept last even though it no longer cuts` \
`# the storage pointer over. It is the only scenario that depends on a` \
`# second service (Azurite on 10000), so if that container is missing` \