2 Commits

Author	SHA1	Message	Date
Edouard Vanbelle	ef626127c6	doc: mark jobs-handling-recoverable-error as implemented ... It still said "Status: not started" after the whole thing shipped and was validated by hand against a real S3 endpoint in both directions. Steps 1–4 marked DONE, §Testing marked DONE with the two places the implementation departed from what the section anticipated: * The fixture is an unreachable ADDRESS, not Azurite. Azurite's deterministic 500 is a *failure*, and failures were never the hard case — they surface and get classified. What hung was a peer that never answers. Also records that the existing `s3_stub` (`127.0.0.1:9999`) cannot serve this: nothing listens, so the connection is refused instantly and a test built on it would pass with no timeout configured anywhere. * The bound is a polling budget, not a request duration. `backend_migration` is detached — the trigger returns 202 in milliseconds however long the backend hangs, so timing it proves nothing. That mistake was made and caught in review. The header also records the two things the DESIGN did not anticipate, because they explain why the policy alone would not have been enough: classification cannot see a call that never returns (no error to classify), and `NotFound` was being returned for every read failure at nine sites — including `blob_exists`, the migration's first probe of the source, which made a transient outage look like an absent blob and could flip the pointer to an incomplete target. Remaining work left explicitly open: the online-migration shape, the stacked-retry tuning, and the one unreproduced `scanned_count` over-report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-08 08:56:15 +02:00
Edouard Vanbelle	2b52d233f0	feat(azure): enumerate blobs, and fail instead of degrading when that breaks ... `AzureBlobBackend` inherited the trait's `operation_not_supported` default for `list_blob_hashes`, so every `backend_consistency` run on Azure fell back to a per-row probe. That fallback walks `storage.blobs` asking "are these bytes there", which structurally cannot find orphans: bytes no row claims are invisible to anything starting from the database, because you need a hash to ask about one and discovering unknown hashes IS enumeration. Azure had half the coverage of local and S3, in the direction that wastes space. ## Enumeration The obstacle was the cursor contract. The caller advances ONE cursor across both sides of the merge-join, feeding the same value to the backend and to `WHERE hash > $1`, so the cursor IS a blob hash. S3 satisfies that with `StartAfter`. Azure has no equivalent on this SDK: REST 2023-05-03 added `startFrom`, but `azure_storage_blobs` 0.21 never sends it — `ListBlobs` exposes only prefix, delimiter, max_results and an opaque marker that cannot be derived from a hash. Resume rides on `prefix` instead. Names are `{hash[0..2]}/{hash}.blob`, which partitions the container into 256 shards that are themselves in hash order, so walking 00/…ff/ yields exactly the global order the merge-join needs and a cursor names the shard to restart in. Re-listing on resume is bounded by shard width rather than by the whole container — the cost a client-side skip over a flat listing would pay on every page. `marker` pages within one call and never escapes as the cursor, the same treatment the S3 impl gives its continuation token. One asymmetry against S3, deliberate: constraining to `{2-hex}/` means foreign files outside that shape never reach `unknowns`. Safe in the direction that matters — an orphan is a blob we wrote and stopped referencing, so it always has the canonical name — and it buys O(N) enumeration instead of O(N²/limit). `hash_from_blob_name` mirrors S3's parser, shard-equals-prefix check included: without it a mis-sharded name would round-trip to a `blob_name` we never wrote, reporting a live blob no read path can find. Tested for round-trip, for nine non-canonical shapes, and for the ordering premise the merge-join rests on. ## Removing the fallback With Azure enumerating, nothing shipped answers `operation_not_supported`. The fallback's other stated justification — mid-migration — never applied: it named a `MigrationBlobBackend` that does not exist, and `SwappableBlobBackend::list_blob_hashes` forwards to whatever is currently active, as do the Encrypted, Cached and Retry wrappers. What still reached it was a transient failure (auth blip, throttle, network) relabelled as a capability limit, on a run that then read as clean while having silently lost orphan coverage. So it was not merely dead, it produced the wrong outcome — the only one it could. It also had zero test coverage across 227 lines. Now any `Err` from `list_blob_hashes` fails the run. That is louder than an anomaly on a green run, which was the fallback's own goal. The trait default still returns `operation_not_supported`, so a genuinely unenumerable backend would fail every run — detectable, not silent, and the point at which to bring the fallback back with tests. ## Also here A doc note on `get_blob_range_stream` explaining why the Azurite migration hang is not worked around: `azure_core` 0.21's `Range::as_headers` attaches `x-ms-range-get-content-crc64` to any range under 4 MiB with no opt-out, Azurite 500s on it, and `azure_core` retries a deterministic error forever. The reachable path is `backend_migration` → `EncryptedBlobBackend::head_check` → `get_blob_range_stream(hash, 0, HEADER_SIZE)` — a pre-write probe on the TARGET, so it fires on the first blob while `migration_readonly` refuses writes app-wide. Working around it would trade production read amplification for emulator support; the fix is the official SDK, where `range_get_content_crc64` is an explicit field. `RUSTSEC-2026-0275` is ignored on the same reasoning — `azure_core` 0.21 logs the `authorization` header at debug, the advisory's "upgrade to >=0.22.0" names a version that does not exist, and the real remedy is that same migration. Reachable only via an explicit `RUST_LOG=…,azure_core=debug`; the entry says not to run that against a real account. `docs/plan/jobs-handling-recoverable-error.md` covers the other half — a bounded retry should have turned that hang into a Paused run with a reason, whatever the SDK does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 19:24:32 +02:00