feat(azure): enumerate blobs, and fail instead of degrading when that breaks
`AzureBlobBackend` inherited the trait's `operation_not_supported`
default for `list_blob_hashes`, so every `backend_consistency` run on
Azure fell back to a per-row probe. That fallback walks `storage.blobs`
asking "are these bytes there", which structurally cannot find orphans:
bytes no row claims are invisible to anything starting from the
database, because you need a hash to ask about one and discovering
unknown hashes IS enumeration. Azure had half the coverage of local and
S3, in the direction that wastes space.
## Enumeration
The obstacle was the cursor contract. The caller advances ONE cursor
across both sides of the merge-join, feeding the same value to the
backend and to `WHERE hash > $1`, so the cursor IS a blob hash. S3
satisfies that with `StartAfter`. Azure has no equivalent on this SDK:
REST 2023-05-03 added `startFrom`, but `azure_storage_blobs` 0.21 never
sends it — `ListBlobs` exposes only prefix, delimiter, max_results and
an opaque marker that cannot be derived from a hash.
Resume rides on `prefix` instead. Names are `{hash[0..2]}/{hash}.blob`,
which partitions the container into 256 shards that are themselves in
hash order, so walking 00/…ff/ yields exactly the global order the
merge-join needs and a cursor names the shard to restart in.
Re-listing on resume is bounded by shard width rather than by the whole
container — the cost a client-side skip over a flat listing would pay on
every page. `marker` pages within one call and never escapes as the
cursor, the same treatment the S3 impl gives its continuation token.
One asymmetry against S3, deliberate: constraining to `{2-hex}/` means
foreign files outside that shape never reach `unknowns`. Safe in the
direction that matters — an orphan is a blob we wrote and stopped
referencing, so it always has the canonical name — and it buys O(N)
enumeration instead of O(N²/limit).
`hash_from_blob_name` mirrors S3's parser, shard-equals-prefix check
included: without it a mis-sharded name would round-trip to a
`blob_name` we never wrote, reporting a live blob no read path can find.
Tested for round-trip, for nine non-canonical shapes, and for the
ordering premise the merge-join rests on.
## Removing the fallback
With Azure enumerating, nothing shipped answers
`operation_not_supported`. The fallback's other stated justification —
mid-migration — never applied: it named a `MigrationBlobBackend` that
does not exist, and `SwappableBlobBackend::list_blob_hashes` forwards to
whatever is currently active, as do the Encrypted, Cached and Retry
wrappers.
What still reached it was a transient failure (auth blip, throttle,
network) relabelled as a capability limit, on a run that then read as
clean while having silently lost orphan coverage. So it was not merely
dead, it produced the wrong outcome — the only one it could. It also had
zero test coverage across 227 lines.
Now any `Err` from `list_blob_hashes` fails the run. That is louder than
an anomaly on a green run, which was the fallback's own goal. The trait
default still returns `operation_not_supported`, so a genuinely
unenumerable backend would fail every run — detectable, not silent, and
the point at which to bring the fallback back with tests.
## Also here
A doc note on `get_blob_range_stream` explaining why the Azurite
migration hang is not worked around: `azure_core` 0.21's
`Range::as_headers` attaches `x-ms-range-get-content-crc64` to any range
under 4 MiB with no opt-out, Azurite 500s on it, and `azure_core`
retries a deterministic error forever. The reachable path is
`backend_migration` → `EncryptedBlobBackend::head_check` →
`get_blob_range_stream(hash, 0, HEADER_SIZE)` — a pre-write probe on the
TARGET, so it fires on the first blob while `migration_readonly` refuses
writes app-wide. Working around it would trade production read
amplification for emulator support; the fix is the official SDK, where
`range_get_content_crc64` is an explicit field.
`RUSTSEC-2026-0275` is ignored on the same reasoning — `azure_core` 0.21
logs the `authorization` header at debug, the advisory's "upgrade to
>=0.22.0" names a version that does not exist, and the real remedy is
that same migration. Reachable only via an explicit
`RUST_LOG=…,azure_core=debug`; the entry says not to run that against a
real account.
`docs/plan/jobs-handling-recoverable-error.md` covers the other half —
a bounded retry should have turned that hang into a Paused run with a
reason, whatever the SDK does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -51,6 +51,33 @@ ignore = [
|
||||
# No direct security impact; no upgrade path exists.
|
||||
"RUSTSEC-2024-0384",
|
||||
|
||||
# azure_core 0.21.0 writes the `authorization` header value to logs —
|
||||
# `policies/transport.rs` does `debug!("…{request:#?}")` over the whole
|
||||
# request. For a SharedKey entry that value is the per-request HMAC
|
||||
# signature; for a SAS entry it is the token. Severity 6.5 (medium).
|
||||
#
|
||||
# The advisory says "upgrade to >=0.22.0". That version does not exist:
|
||||
# `azure_core` jumped 0.21 → 1.x, and `azure_storage_blobs` never left
|
||||
# 0.21.0 before being archived. So the stated remedy IS the official-SDK
|
||||
# migration, tracked separately alongside the quick-xml pair above.
|
||||
#
|
||||
# Not reachable at our log levels: the line is `debug!` on the
|
||||
# `azure_core::policies::transport` target, and the default filter is
|
||||
# `info`. It fires only if an operator explicitly asks for
|
||||
# `RUST_LOG=…,azure_core=debug`, which is not hypothetical — that is the
|
||||
# invocation used to diagnose the Azurite migration hang. **Do not run
|
||||
# `azure_core=debug` against a real Azure account**; it prints request
|
||||
# signatures to the terminal. Against Azurite it only exposes the
|
||||
# published dev key's signatures.
|
||||
#
|
||||
# A subscriber-level directive pinning that target off was prototyped
|
||||
# and rejected 2026-09-02 — not worth carrying a filter hack for a
|
||||
# dependency being replaced.
|
||||
#
|
||||
# Un-ignore trigger: the azure_storage_blob 1.x migration lands
|
||||
# (`cargo tree -i azure_core@0.21` returns no rows).
|
||||
"RUSTSEC-2026-0275",
|
||||
|
||||
# quick-xml 0.31.0 — transitive via azure_core 0.21.0 (unofficial SDK,
|
||||
# now archived). Our direct dep is already on 0.41.0; the 0.31 copy is
|
||||
# only reachable through the azure_storage_blobs chain, which parses
|
||||
|
||||
Reference in New Issue
Block a user