test(api): pin that an unreachable backend pauses in bounded time
The regression test `docs/plan/jobs-handling-recoverable-error.md` §Testing asks for: assert the run reaches Paused, that `error_message` names the cause, and that it does so in bounded time rather than hanging. ## The endpoint has to HANG, not refuse `s3_stub` already existed and points at `127.0.0.1:9999`, where nothing listens. That connection is REFUSED — ECONNREFUSED, immediately — and that path was never broken. A test built on it would pass with no timeout configured anywhere, which is worse than no test: it would read as coverage of exactly the failure it cannot see. So `s3_blackhole` points at `192.0.2.1`, TEST-NET-1 (RFC 5737), reserved for documentation and guaranteed unrouted. A SYN goes unanswered — no RST, no ICMP — which is the failure that used to hang until the OS abandoned TCP retransmission ~15 minutes later, with the job neither running nor failed the whole time. Ed's suggestion, and it is the right fixture: a server that never answers is reproducible in a way that unplugging a cable is not. ## The load-bearing assertion is `duration` Every other assert in the file would also pass against the old hanging behaviour, given fifteen minutes. `duration < 120000` is the only one that fails if the bound is ever removed. The threshold is deliberately loose — three orders of magnitude from the failure it guards, so a slow runner cannot make it flaky. ## Why it is safe in the shared suite The run fails at `target.initialize()`, which is BEFORE `migration_readonly` is engaged, so this file cannot leave the server read-only for whatever runs next. A mid-copy failure would have held the freeze — that is why this shape was chosen. Teardown is mandatory rather than tidy: `open_or_start` picks up the latest non-terminal row, so a Paused row left behind would be RESUMED by the next `backend_migration` trigger in the suite, silently retargeting an unrelated test at the black hole. The file cancels its own run and asserts the row reached Cancelled. Placed second-to-last. It is the slowest file in the suite by design — it waits out an unreachable endpoint to prove the wait is bounded — so that cost lands after everything else has reported. Azurite stays last for the reason its own comment gives. Not yet executed: the suite tears down containers and Ed usually has a run in flight. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+31
-1
@@ -42,7 +42,7 @@ OXICLOUD_NEXTCLOUD_ENABLED=true
|
||||
# `s3_stub` is declared but never activated — it lets storage_config.hurl
|
||||
# assert the entries table has more than one row without needing a
|
||||
# real S3 backend.
|
||||
OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite
|
||||
OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite,s3_blackhole
|
||||
OXICLOUD_STORAGE_local_main_BACKEND=local
|
||||
OXICLOUD_STORAGE_s3_stub_BACKEND=s3
|
||||
OXICLOUD_STORAGE_s3_stub_S3_BUCKET=oxicloud-test-stub
|
||||
@@ -51,6 +51,36 @@ OXICLOUD_STORAGE_s3_stub_S3_ENDPOINT_URL=http://127.0.0.1:9999
|
||||
OXICLOUD_STORAGE_s3_stub_S3_ACCESS_KEY=stub
|
||||
OXICLOUD_STORAGE_s3_stub_S3_SECRET_KEY=stub
|
||||
|
||||
# `s3_blackhole` — an endpoint that never answers, as opposed to
|
||||
# `s3_stub` above which refuses instantly.
|
||||
#
|
||||
# The distinction is the entire point. `127.0.0.1:9999` has nothing
|
||||
# listening, so a connection is REFUSED: the kernel returns ECONNREFUSED
|
||||
# immediately and the SDK reports an error straight away. That path was
|
||||
# always handled. A test built on it would pass even with no timeout
|
||||
# configured anywhere.
|
||||
#
|
||||
# `192.0.2.1` is TEST-NET-1 (RFC 5737), reserved for documentation and
|
||||
# guaranteed not to be routed. A SYN to it goes unanswered — no RST, no
|
||||
# ICMP — which is the failure that used to hang: with no bound, a socket
|
||||
# read blocks until the OS abandons retransmission, on the order of
|
||||
# fifteen minutes, during which the job is neither running nor failed.
|
||||
#
|
||||
# Declared but NEVER activated, like the other two. `backend_migration
|
||||
# ?storage=s3_blackhole` reaches it explicitly.
|
||||
#
|
||||
# If a CI network answers 192.0.2.1 with ICMP unreachable, the failure
|
||||
# degrades to the refused shape and `backend_migration_blackhole.hurl`
|
||||
# still passes — both classify transient and both pause. It would simply
|
||||
# stop pinning the timeout specifically. The assert on elapsed time in
|
||||
# that file is what would notice.
|
||||
OXICLOUD_STORAGE_s3_blackhole_BACKEND=s3
|
||||
OXICLOUD_STORAGE_s3_blackhole_S3_BUCKET=oxicloud-blackhole
|
||||
OXICLOUD_STORAGE_s3_blackhole_S3_REGION=us-east-1
|
||||
OXICLOUD_STORAGE_s3_blackhole_S3_ENDPOINT_URL=http://192.0.2.1:9999
|
||||
OXICLOUD_STORAGE_s3_blackhole_S3_ACCESS_KEY=blackhole
|
||||
OXICLOUD_STORAGE_s3_blackhole_S3_SECRET_KEY=blackhole
|
||||
|
||||
# `azurite` — a REAL, reachable Azure backend, unlike `s3_stub` above.
|
||||
# It points at the Azurite emulator started by spawn-db.sh, which speaks
|
||||
# the actual Blob REST API, so this is the only way to exercise the Azure
|
||||
|
||||
Reference in New Issue
Block a user