feat(azure): enumerate blobs, and fail instead of degrading when that breaks

`AzureBlobBackend` inherited the trait's `operation_not_supported`
default for `list_blob_hashes`, so every `backend_consistency` run on
Azure fell back to a per-row probe. That fallback walks `storage.blobs`
asking "are these bytes there", which structurally cannot find orphans:
bytes no row claims are invisible to anything starting from the
database, because you need a hash to ask about one and discovering
unknown hashes IS enumeration. Azure had half the coverage of local and
S3, in the direction that wastes space.

## Enumeration

The obstacle was the cursor contract. The caller advances ONE cursor
across both sides of the merge-join, feeding the same value to the
backend and to `WHERE hash > $1`, so the cursor IS a blob hash. S3
satisfies that with `StartAfter`. Azure has no equivalent on this SDK:
REST 2023-05-03 added `startFrom`, but `azure_storage_blobs` 0.21 never
sends it — `ListBlobs` exposes only prefix, delimiter, max_results and
an opaque marker that cannot be derived from a hash.

Resume rides on `prefix` instead. Names are `{hash[0..2]}/{hash}.blob`,
which partitions the container into 256 shards that are themselves in
hash order, so walking 00/…ff/ yields exactly the global order the
merge-join needs and a cursor names the shard to restart in.
Re-listing on resume is bounded by shard width rather than by the whole
container — the cost a client-side skip over a flat listing would pay on
every page. `marker` pages within one call and never escapes as the
cursor, the same treatment the S3 impl gives its continuation token.

One asymmetry against S3, deliberate: constraining to `{2-hex}/` means
foreign files outside that shape never reach `unknowns`. Safe in the
direction that matters — an orphan is a blob we wrote and stopped
referencing, so it always has the canonical name — and it buys O(N)
enumeration instead of O(N²/limit).

`hash_from_blob_name` mirrors S3's parser, shard-equals-prefix check
included: without it a mis-sharded name would round-trip to a
`blob_name` we never wrote, reporting a live blob no read path can find.
Tested for round-trip, for nine non-canonical shapes, and for the
ordering premise the merge-join rests on.

## Removing the fallback

With Azure enumerating, nothing shipped answers
`operation_not_supported`. The fallback's other stated justification —
mid-migration — never applied: it named a `MigrationBlobBackend` that
does not exist, and `SwappableBlobBackend::list_blob_hashes` forwards to
whatever is currently active, as do the Encrypted, Cached and Retry
wrappers.

What still reached it was a transient failure (auth blip, throttle,
network) relabelled as a capability limit, on a run that then read as
clean while having silently lost orphan coverage. So it was not merely
dead, it produced the wrong outcome — the only one it could. It also had
zero test coverage across 227 lines.

Now any `Err` from `list_blob_hashes` fails the run. That is louder than
an anomaly on a green run, which was the fallback's own goal. The trait
default still returns `operation_not_supported`, so a genuinely
unenumerable backend would fail every run — detectable, not silent, and
the point at which to bring the fallback back with tests.

## Also here

A doc note on `get_blob_range_stream` explaining why the Azurite
migration hang is not worked around: `azure_core` 0.21's
`Range::as_headers` attaches `x-ms-range-get-content-crc64` to any range
under 4 MiB with no opt-out, Azurite 500s on it, and `azure_core`
retries a deterministic error forever. The reachable path is
`backend_migration` → `EncryptedBlobBackend::head_check` →
`get_blob_range_stream(hash, 0, HEADER_SIZE)` — a pre-write probe on the
TARGET, so it fires on the first blob while `migration_readonly` refuses
writes app-wide. Working around it would trade production read
amplification for emulator support; the fix is the official SDK, where
`range_get_content_crc64` is an explicit field.

`RUSTSEC-2026-0275` is ignored on the same reasoning — `azure_core` 0.21
logs the `authorization` header at debug, the advisory's "upgrade to
>=0.22.0" names a version that does not exist, and the real remedy is
that same migration. Reachable only via an explicit
`RUST_LOG=…,azure_core=debug`; the entry says not to run that against a
real account.

`docs/plan/jobs-handling-recoverable-error.md` covers the other half —
a bounded retry should have turned that hang into a Paused run with a
reason, whatever the SDK does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/plan/jobs-handling-recoverable-error.md

@@ -0,0 +1,265 @@
+# Recoverable errors in jobs — retry, then pause
+
+**Status: not started.** Design settled 2026-08-31, from a live
+diagnosis (see [Motivating incident](#motivating-incident)).
+
+A job that hits a failing backend today has two possible endings, and
+neither is right for an outage: it fails the run (throwing away a
+partially-complete migration, since `Failed` is terminal and only
+`Paused` resumes), or it hangs forever inside an SDK retry loop with no
+log line and no way to act on it.
+
+This plan adds the third: **retry a bounded number of times, then pause
+with the reason recorded**, so an operator resumes when the provider
+recovers and the job continues from its cursor.
+
+---
+
+## Motivating incident
+
+`backend_migration ?storage=azurite` hung indefinitely. Diagnosis, after
+several wrong theories:
+
+- The job issued a ranged GET carrying `x-ms-range-get-content-crc64`.
+- Azurite answered **500** (real Azure supports CRC64 range validation;
+  the emulator does not).
+- `azure_core`'s retry policy classifies 500 as retryable and loops.
+- The response was deterministic, so every retry failed identically.
+- The job never advanced, never failed, and emitted no per-blob line —
+  while holding `migration_readonly`, refusing writes **across the whole
+  application**.
+
+The exact chain was pinned down later (2026-09-02) and is worth having,
+because it is not where you would look — the copy itself is innocent:
+
+```
+backend_migration_service.rs   target.head_check(hash)   ← pre-write probe
+  → EncryptedBlobBackend::head_check
+  → get_blob_range_stream(hash, 0, HEADER_SIZE)          ← ~40 bytes
+  → azure_core Range::as_headers                         ← adds the CRC64
+      (src/request_options/range.rs: any range < 4 MiB)     header, no opt-out
+```
+
+`copy_blob` reads from the SOURCE, which is local in a local→Azure
+migration, so it never touches an Azure range. What hangs is the format
+probe against the TARGET, on the first blob, before a byte is copied.
+
+A workaround exists — unranged `get()` for small requests, truncate
+client-side — and was **rejected 2026-09-02**: it pays for an emulator
+with production read amplification and puts new offset arithmetic on the
+read path. See the note on `AzureBlobBackend::get_blob_range_stream`.
+The fix is the official SDK, where `range_get_content_crc64` is an
+explicit field. **This plan is unaffected either way** — a bounded retry
+would have turned the hang into a Paused run with a reason, which is the
+point.
+
+Two properties made it invisible: the 500 was only visible at
+`azure_core=debug`, and nothing bounded the retry. The same shape would
+occur against real Azure or S3 on any persistent 5xx; it is not an
+emulator quirk. `AzureBlobBackend` configures no retry policy and no
+timeout at all — `grep "retry\|timeout\|ClientOptions"` on
+`azure_blob_backend.rs` returns nothing.
+
+Full evidence chain, including the theories ruled out and what each
+cost, is in the memory note `bug-azure-put-blob-hangs-no-timeout`.
+
+---
+
+## Why this belongs at the top level
+
+Every recoverable job goes through `run_or_resume`, which already owns
+the run lifecycle: it opens the row, persists and restores
+[`JobRunArgs`](./job-registry.md) (`63820c3c`), dispatches the handler,
+and writes the terminal state. Retry-and-pause is the same kind of
+concern — policy about *how a run behaves*, not about what any one job
+does.
+
+Implemented there, `backend_rotate`, `transcode_import`, the thumbnail
+imports and anything added later inherit it. Implemented per-job, it
+gets written once per job and drifts.
+
+It also survives the pending official-Azure-SDK migration untouched,
+where per-SDK retry tuning would have to be redone.
+
+**The blocker is that the engine cannot currently act on what it is
+told.** A handler returns `RunOutcome::{Completed, Paused, Failed}`, so
+a transient backend error is already flattened into `Failed` before the
+engine sees it — "the provider is down" and "this data is wrong" are
+indistinguishable. Closing that is what makes a top-level
+implementation possible, and it is step 1.
+
+---
+
+## Step 1 — errors say whether they are retryable
+
+Today both backends wrap SDK errors into
+`DomainError::internal_error("Azure", format!("…{e}"))`, so the status
+code survives only inside a formatted string. Recovering it means
+string-matching, which is exactly the kind of fragility that turns into
+a silent behaviour change when an SDK reformats its `Display`.
+
+Carry the distinction on the type instead — a `retryable` flag, or a
+kind the storage ports set deliberately.
+
+**Retryable** (environment, may clear on its own):
+- HTTP 5xx, 429 / `SlowDown` / throttling
+- connect timeouts, connection resets, DNS failure
+
+**Permanent** (will fail identically forever):
+- 4xx other than 429 — 401/403 (credentials), 404 (missing container)
+- decode failures, checksum mismatch
+- `operation_not_supported`
+
+> **The Azurite 500 is a permanent error wearing a retryable status
+> code.** No classification by status alone gets this right, which is
+> the case for a bounded cap rather than "retry until it works". The cap
+> is the safety net for exactly the errors the taxonomy misjudges.
+
+**Do not double-retry.** The AWS SDK already retries internally with its
+own backoff, so a second layer above it multiplies. Check what the S3
+backend inherits before adding anything, and consider making the
+engine's cap the *outer* bound with SDK retries reduced or disabled.
+
+## Step 2 — an outcome the engine can act on
+
+`RunOutcome` grows a variant meaning "the environment failed, this is
+worth trying again later":
+
+```rust
+RunOutcome::PausedRetryable { cursor: Vec<u8>, reason: String }
+```
+
+Distinct from all three existing outcomes, and the distinction is the
+point:
+
+| outcome | meaning | resumes? |
+|---|---|---|
+| `Failed` | the data or the request is wrong | no — terminal |
+| `Paused` | an operator asked it to stop | yes |
+| `PausedRetryable` | the environment failed | yes, and says why |
+
+The row lands as `Paused` either way, so resume works unchanged. What
+differs is `error_message`, which must let the panel — and an operator —
+tell "I paused this" from "the provider went down". Without that
+distinction a paused run is an unexplained one.
+
+## Step 3 — the engine implements the policy
+
+In `run_or_resume`:
+
+- bounded exponential backoff, ~5 attempts
+- **log each failed attempt at `warn` on our side.** The incident took
+  several runs to diagnose because the 500 was visible only at
+  `azure_core=debug`. One line per exhausted operation, naming status,
+  target and attempt count.
+- on exhaustion, write the row as `Paused` with the reason in
+  `error_message`
+
+Handlers then return the retryable outcome and get the policy for free.
+
+---
+
+## Step 4 — `migration_readonly`, the sharp edge
+
+`backend_migration` holds a gate that refuses writes **application-wide**
+until cutover. What happens to it on pause is a correctness question,
+not a cosmetic one.
+
+### Start conservative: keep the gate held while paused
+
+Correct, because no writes during the pause means the cursor stays valid
+and resume-from-cursor is sound.
+
+It is also **strictly better than today**, which is the thing to
+remember when the read-only window looks unattractive: the application
+is *already* read-only while the job hangs — there is simply no way to
+see why or act. Same lock, now with a reason and two operator actions.
+
+**Cancel must clear the gate.** It ends the run with no swap, so the
+source stays active and writes must return. The `failed > 0` path
+already clears readonly on the reasoning that "users shouldn't be locked
+out because of a partial run" — cancel is the escape hatch operators
+will reach for during an outage, and it must work.
+
+**Make the state loud.** Writes refused app-wide should be obvious in
+the admin panel, not discovered by reading a run row. The progress
+snapshot already feeds the header middleware; a paused migration wants
+the same visibility, saying why and offering resume/cancel.
+
+### Why NOT to release the gate on pause (yet)
+
+Tempting — ops should not be locked out during a provider outage — and
+unsafe as the job stands:
+
+1. Release readonly; users write again. New blobs land on the source,
+   which is still active, and are absent from the target.
+2. Resume continues **from the cursor**, a position in a hash-ordered
+   walk.
+3. A blob written during the pause whose hash sorts *before* that cursor
+   is never visited.
+4. The run completes, flips the pointer, and reads for that hash 404
+   against a target that never received it.
+
+This is why the existing `failed > 0` path can safely clear readonly: it
+**ends** the run, and an operator retrying starts a new run with a fresh
+cursor, so everything is rescanned. Pause-and-resume is what
+reintroduces the gap.
+
+Releasing on pause becomes safe only alongside one of:
+
+- **resume rescans from the beginning** rather than trusting the cursor
+  — cheap, because the walk short-circuits on blobs already present in
+  the target, so a second pass costs a lookup per blob, not a copy; or
+- **a final catch-up pass under readonly before the swap**, with the
+  pointer flipping only when a complete pass finds nothing new.
+
+### Where this should end up
+
+The second option is the standard online-migration shape, and it is
+where this job wants to go regardless: copy the bulk **without** holding
+the gate, engage it only for a short final catch-up plus the swap.
+
+That removes what made the incident damaging — writes refused app-wide
+for the entire duration of a long copy — and turns releasing-on-pause
+into a free consequence rather than a correctness fix. Worth doing as a
+follow-up, from the safer base this plan establishes.
+
+---
+
+## Scope: Azure and S3 both
+
+The only reason S3 looks healthy is that the endpoint behaves. A
+persistent 5xx from S3 hangs identically — the gap is the absence of a
+bound, not anything Azure-specific.
+
+Putting the policy above the `BlobStorageBackend` trait covers Azure,
+S3, local and anything added later with one implementation. Per-backend
+SDK retry tuning stays a separate, optional refinement — and for Azure
+specifically it should wait for the official SDK, since `azure_core`
+0.21 is archived and queued for replacement.
+
+## Testing
+
+The `backend_consistency_azure.hurl` scenario and its Azurite service
+are already wired (`tests/common/docker-compose.test.yml`,
+`spawn-db.sh`), which gives a backend that reliably produces the
+failure: Azurite 500s on the CRC64 ranged GET every time. That makes it
+a genuine fixture for this work rather than a flake —
+**deterministically unretryable-but-retryable-looking**, which is the
+hard case.
+
+Note the scenario no longer triggers a migration — it audits Azurite
+without cutting over, for the reason in its header. Reaching the fixture
+means triggering `backend_migration ?storage=azurite` explicitly, which
+is exactly the hang this plan is meant to convert into a Paused run.
+Doing that inside the shared suite is what ordering it last was for; it
+can go back once the outcome is bounded.
+
+Assert the run reaches `Paused`, that `error_message` names the cause,
+and that it does so in bounded time rather than hanging.
+
+`POST /api/admin/settings/storage/test` with `entry_name` is the
+pre-flight worth keeping in mind: synchronous, does a real write/read
+round-trip, and isolates "the backend is misconfigured" from "the job
+is broken". It passing while the migration hung is what ruled out
+credentials, container and the write path during the incident.