cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #704 from EdouardVanbelle/fix/azure-enumeration

Browse Source
This commit is contained in:
Dionisio Pozo
2026-09-03 15:15:14 +02:00
committed by GitHub
parent 4cd4f5a8f6 04e0df0c89
commit 24a060c0cc
10 changed files with 1069 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cargo/audit.toml
+27
View File
@@ -51,6 +51,33 @@ ignore = [
# No direct security impact; no upgrade path exists.
"RUSTSEC-2024-0384",
# azure_core 0.21.0 writes the `authorization` header value to logs —
# `policies/transport.rs` does `debug!("…{request:#?}")` over the whole
# request. For a SharedKey entry that value is the per-request HMAC
# signature; for a SAS entry it is the token. Severity 6.5 (medium).
#
# The advisory says "upgrade to >=0.22.0". That version does not exist:
# `azure_core` jumped 0.21 → 1.x, and `azure_storage_blobs` never left
# 0.21.0 before being archived. So the stated remedy IS the official-SDK
# migration, tracked separately alongside the quick-xml pair above.
#
# Not reachable at our log levels: the line is `debug!` on the
# `azure_core::policies::transport` target, and the default filter is
# `info`. It fires only if an operator explicitly asks for
# `RUST_LOG=…,azure_core=debug`, which is not hypothetical — that is the
# invocation used to diagnose the Azurite migration hang. **Do not run
# `azure_core=debug` against a real Azure account**; it prints request
# signatures to the terminal. Against Azurite it only exposes the
# published dev key's signatures.
#
# A subscriber-level directive pinning that target off was prototyped
# and rejected 2026-09-02 — not worth carrying a filter hack for a
# dependency being replaced.
#
# Un-ignore trigger: the azure_storage_blob 1.x migration lands
# (`cargo tree -i azure_core@0.21` returns no rows).
"RUSTSEC-2026-0275",
# quick-xml 0.31.0 — transitive via azure_core 0.21.0 (unofficial SDK,
# now archived). Our direct dep is already on 0.41.0; the 0.31 copy is
# only reachable through the azure_storage_blobs chain, which parses
docs/plan/jobs-handling-recoverable-error.md
+265
View File
@@ -0,0 +1,265 @@
# Recoverable errors in jobs — retry, then pause
**Status: not started.** Design settled 2026-08-31, from a live
diagnosis (see [Motivating incident](#motivating-incident)).
A job that hits a failing backend today has two possible endings, and
neither is right for an outage: it fails the run (throwing away a
partially-complete migration, since `Failed` is terminal and only
`Paused` resumes), or it hangs forever inside an SDK retry loop with no
log line and no way to act on it.
This plan adds the third: **retry a bounded number of times, then pause
with the reason recorded**, so an operator resumes when the provider
recovers and the job continues from its cursor.
---
## Motivating incident
`backend_migration ?storage=azurite` hung indefinitely. Diagnosis, after
several wrong theories:
- The job issued a ranged GET carrying `x-ms-range-get-content-crc64`.
- Azurite answered **500** (real Azure supports CRC64 range validation;
the emulator does not).
- `azure_core`'s retry policy classifies 500 as retryable and loops.
- The response was deterministic, so every retry failed identically.
- The job never advanced, never failed, and emitted no per-blob line —
while holding `migration_readonly`, refusing writes **across the whole
application**.
The exact chain was pinned down later (2026-09-02) and is worth having,
because it is not where you would look — the copy itself is innocent:
```
backend_migration_service.rs target.head_check(hash) ← pre-write probe
→ EncryptedBlobBackend::head_check
→ get_blob_range_stream(hash, 0, HEADER_SIZE) ← ~40 bytes
→ azure_core Range::as_headers ← adds the CRC64
(src/request_options/range.rs: any range < 4 MiB) header, no opt-out
```
`copy_blob` reads from the SOURCE, which is local in a local→Azure
migration, so it never touches an Azure range. What hangs is the format
probe against the TARGET, on the first blob, before a byte is copied.
A workaround exists — unranged `get()` for small requests, truncate
client-side — and was **rejected 2026-09-02**: it pays for an emulator
with production read amplification and puts new offset arithmetic on the
read path. See the note on `AzureBlobBackend::get_blob_range_stream`.
The fix is the official SDK, where `range_get_content_crc64` is an
explicit field. **This plan is unaffected either way** — a bounded retry
would have turned the hang into a Paused run with a reason, which is the
point.
Two properties made it invisible: the 500 was only visible at
`azure_core=debug`, and nothing bounded the retry. The same shape would
occur against real Azure or S3 on any persistent 5xx; it is not an
emulator quirk. `AzureBlobBackend` configures no retry policy and no
timeout at all — `grep "retry\|timeout\|ClientOptions"` on
`azure_blob_backend.rs` returns nothing.
Full evidence chain, including the theories ruled out and what each
cost, is in the memory note `bug-azure-put-blob-hangs-no-timeout`.
---
## Why this belongs at the top level
Every recoverable job goes through `run_or_resume`, which already owns
the run lifecycle: it opens the row, persists and restores
[`JobRunArgs`](./job-registry.md) (`63820c3c`), dispatches the handler,
and writes the terminal state. Retry-and-pause is the same kind of
concern — policy about *how a run behaves*, not about what any one job
does.
Implemented there, `backend_rotate`, `transcode_import`, the thumbnail
imports and anything added later inherit it. Implemented per-job, it
gets written once per job and drifts.
It also survives the pending official-Azure-SDK migration untouched,
where per-SDK retry tuning would have to be redone.
**The blocker is that the engine cannot currently act on what it is
told.** A handler returns `RunOutcome::{Completed, Paused, Failed}`, so
a transient backend error is already flattened into `Failed` before the
engine sees it — "the provider is down" and "this data is wrong" are
indistinguishable. Closing that is what makes a top-level
implementation possible, and it is step 1.
---
## Step 1 — errors say whether they are retryable
Today both backends wrap SDK errors into
`DomainError::internal_error("Azure", format!("…{e}"))`, so the status
code survives only inside a formatted string. Recovering it means
string-matching, which is exactly the kind of fragility that turns into
a silent behaviour change when an SDK reformats its `Display`.
Carry the distinction on the type instead — a `retryable` flag, or a
kind the storage ports set deliberately.
**Retryable** (environment, may clear on its own):
- HTTP 5xx, 429 / `SlowDown` / throttling
- connect timeouts, connection resets, DNS failure
**Permanent** (will fail identically forever):
- 4xx other than 429 — 401/403 (credentials), 404 (missing container)
- decode failures, checksum mismatch
- `operation_not_supported`
> **The Azurite 500 is a permanent error wearing a retryable status
> code.** No classification by status alone gets this right, which is
> the case for a bounded cap rather than "retry until it works". The cap
> is the safety net for exactly the errors the taxonomy misjudges.
**Do not double-retry.** The AWS SDK already retries internally with its
own backoff, so a second layer above it multiplies. Check what the S3
backend inherits before adding anything, and consider making the
engine's cap the *outer* bound with SDK retries reduced or disabled.
## Step 2 — an outcome the engine can act on
`RunOutcome` grows a variant meaning "the environment failed, this is
worth trying again later":
```rust
RunOutcome::PausedRetryable { cursor: Vec<u8>, reason: String }
```
Distinct from all three existing outcomes, and the distinction is the
point:
| outcome | meaning | resumes? |
|---|---|---|
| `Failed` | the data or the request is wrong | no — terminal |
| `Paused` | an operator asked it to stop | yes |
| `PausedRetryable` | the environment failed | yes, and says why |
The row lands as `Paused` either way, so resume works unchanged. What
differs is `error_message`, which must let the panel — and an operator —
tell "I paused this" from "the provider went down". Without that
distinction a paused run is an unexplained one.
## Step 3 — the engine implements the policy
In `run_or_resume`:
- bounded exponential backoff, ~5 attempts
- **log each failed attempt at `warn` on our side.** The incident took
several runs to diagnose because the 500 was visible only at
`azure_core=debug`. One line per exhausted operation, naming status,
target and attempt count.
- on exhaustion, write the row as `Paused` with the reason in
`error_message`
Handlers then return the retryable outcome and get the policy for free.
---
## Step 4 — `migration_readonly`, the sharp edge
`backend_migration` holds a gate that refuses writes **application-wide**
until cutover. What happens to it on pause is a correctness question,
not a cosmetic one.
### Start conservative: keep the gate held while paused
Correct, because no writes during the pause means the cursor stays valid
and resume-from-cursor is sound.
It is also **strictly better than today**, which is the thing to
remember when the read-only window looks unattractive: the application
is *already* read-only while the job hangs — there is simply no way to
see why or act. Same lock, now with a reason and two operator actions.
**Cancel must clear the gate.** It ends the run with no swap, so the
source stays active and writes must return. The `failed > 0` path
already clears readonly on the reasoning that "users shouldn't be locked
out because of a partial run" — cancel is the escape hatch operators
will reach for during an outage, and it must work.
**Make the state loud.** Writes refused app-wide should be obvious in
the admin panel, not discovered by reading a run row. The progress
snapshot already feeds the header middleware; a paused migration wants
the same visibility, saying why and offering resume/cancel.
### Why NOT to release the gate on pause (yet)
Tempting — ops should not be locked out during a provider outage — and
unsafe as the job stands:
1. Release readonly; users write again. New blobs land on the source,
which is still active, and are absent from the target.
2. Resume continues **from the cursor**, a position in a hash-ordered
walk.
3. A blob written during the pause whose hash sorts *before* that cursor
is never visited.
4. The run completes, flips the pointer, and reads for that hash 404
against a target that never received it.
This is why the existing `failed > 0` path can safely clear readonly: it
**ends** the run, and an operator retrying starts a new run with a fresh
cursor, so everything is rescanned. Pause-and-resume is what
reintroduces the gap.
Releasing on pause becomes safe only alongside one of:
- **resume rescans from the beginning** rather than trusting the cursor
— cheap, because the walk short-circuits on blobs already present in
the target, so a second pass costs a lookup per blob, not a copy; or
- **a final catch-up pass under readonly before the swap**, with the
pointer flipping only when a complete pass finds nothing new.
### Where this should end up
The second option is the standard online-migration shape, and it is
where this job wants to go regardless: copy the bulk **without** holding
the gate, engage it only for a short final catch-up plus the swap.
That removes what made the incident damaging — writes refused app-wide
for the entire duration of a long copy — and turns releasing-on-pause
into a free consequence rather than a correctness fix. Worth doing as a
follow-up, from the safer base this plan establishes.
---
## Scope: Azure and S3 both
The only reason S3 looks healthy is that the endpoint behaves. A
persistent 5xx from S3 hangs identically — the gap is the absence of a
bound, not anything Azure-specific.
Putting the policy above the `BlobStorageBackend` trait covers Azure,
S3, local and anything added later with one implementation. Per-backend
SDK retry tuning stays a separate, optional refinement — and for Azure
specifically it should wait for the official SDK, since `azure_core`
0.21 is archived and queued for replacement.
## Testing
The `backend_consistency_azure.hurl` scenario and its Azurite service
are already wired (`tests/common/docker-compose.test.yml`,
`spawn-db.sh`), which gives a backend that reliably produces the
failure: Azurite 500s on the CRC64 ranged GET every time. That makes it
a genuine fixture for this work rather than a flake —
**deterministically unretryable-but-retryable-looking**, which is the
hard case.
Note the scenario no longer triggers a migration — it audits Azurite
without cutting over, for the reason in its header. Reaching the fixture
means triggering `backend_migration ?storage=azurite` explicitly, which
is exactly the hang this plan is meant to convert into a Paused run.
Doing that inside the shared suite is what ordering it last was for; it
can go back once the outcome is bounded.
Assert the run reaches `Paused`, that `error_message` names the cause,
and that it does so in bounded time rather than hanging.
`POST /api/admin/settings/storage/test` with `entry_name` is the
pre-flight worth keeping in mind: synchronous, does a real write/read
round-trip, and isolates "the backend is misconfigured" from "the job
is broken". It passing while the migration hung is what ruled out
credentials, container and the write path during the incident.
src/infrastructure/services/azure_blob_backend.rs
+318 -11
View File
@@ -13,7 +13,8 @@ use futures::{StreamExt, TryStreamExt};
use tokio::fs;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
BackendBlobEntry, BackendUnknownEntry, BlobListPage, BlobStorageBackend, BlobStream,
StorageHealthStatus,
};
use crate::common::config::AzureStorageConfig;
use crate::domain::errors::{DomainError, ErrorKind};
@@ -55,6 +56,29 @@ impl AzureBlobBackend {
}
}
/// Inverse of [`Self::blob_name`] — the hash a listing entry names,
/// or `None` when the entry is not one of ours.
///
/// Mirrors `S3BlobBackend::hash_from_object_key`, including the check
/// that the shard equals the hash's own first two characters: without
/// it, `blob_name(hash)` would not reproduce the name we just parsed,
/// and a mis-sharded object would be reported as a live blob that no
/// read path can find.
fn hash_from_blob_name(name: &str) -> Option<String> {
let (prefix, rest) = name.split_once('/')?;
if prefix.len() != 2 || !prefix.chars().all(|c| c.is_ascii_hexdigit()) {
return None;
}
let stem = rest.strip_suffix(".blob")?;
if stem.len() != 64 || !stem.chars().all(|c| c.is_ascii_hexdigit()) {
return None;
}
if !stem.starts_with(prefix) {
return None;
}
Some(stem.to_string())
}
/// Compute the blob name for a given hash.
fn blob_name(hash: &str) -> String {
let prefix = &hash[0..2];
@@ -243,6 +267,37 @@ impl BlobStorageBackend for AzureBlobBackend {
})
}
/// # Known incompatibility: sub-4 MiB ranges break on Azurite
///
/// `azure_core` 0.21's `Range::as_headers`
/// (`src/request_options/range.rs`) attaches
/// `x-ms-range-get-content-crc64: true` to **any range shorter than
/// 4 MiB**, unconditionally and with no opt-out. Real Azure honours
/// it; Azurite answers 500. `azure_core` then classifies 500 as
/// retryable and loops on a deterministic error, forever.
///
/// The reachable path is `backend_migration` →
/// `EncryptedBlobBackend::head_check` →
/// `get_blob_range_stream(hash, 0, HEADER_SIZE)`. `HEADER_SIZE` is a
/// few dozen bytes, and it runs against the TARGET before each write,
/// so a local→Azurite migration hangs on its first blob while holding
/// `migration_readonly` — writes refused application-wide.
///
/// **Deliberately not worked around here.** The available workaround
/// is to issue an unranged `get()` for small requests (its 16 MiB
/// `initial_range` clears the threshold, so the header is never sent)
/// and truncate client-side. That is correct against real Azure but
/// pays for an emulator with production cost: a ~40-byte format probe
/// becomes a whole-blob transfer, and it puts new offset arithmetic
/// on the read path, where a mistake serves wrong bytes silently
/// rather than failing.
///
/// The real fix is the official `azure_storage_blob` 1.x, where
/// `range_get_content_crc64` is an explicit field on
/// `BlobClientDownloadOptions` — leave it unset and the request is
/// never made. Until then, the Azurite suite exercises enumeration
/// and round-trips but not migration; see
/// `tests/api/backend_consistency_azure.hurl`.
fn get_blob_range_stream(
&self,
hash: &str,
@@ -394,6 +449,197 @@ impl BlobStorageBackend for AzureBlobBackend {
})
}
/// Enumerate blob hashes in lexicographic order, so
/// `backend_consistency` can merge-join against `storage.blobs`
/// instead of degrading to a per-row probe that structurally cannot
/// see orphans.
///
/// ## Why this is a shard walk and not one flat listing
///
/// **The cursor IS a blob hash**, not a provider token. The caller
/// forces that: it advances ONE cursor across both sides of the join,
/// feeding the same value here and to `WHERE hash > $1` in SQL. S3
/// satisfies it with `start_after(object_key(cursor))`.
///
/// Azure has no `StartAfter`. REST API 2023-05-03 added `startFrom`,
/// which would be the direct equivalent — but this SDK
/// (`azure_storage_blobs` 0.21, archived) never sends it: `ListBlobs`
/// exposes only `prefix`, `delimiter`, `max_results` and `marker`,
/// and `marker` is an opaque continuation token that cannot be
/// derived from a hash.
///
/// So resume rides on `prefix` instead. Names are
/// `{hash[0..2]}/{hash}.blob`, which partitions the container into
/// 256 shards that are themselves in hash order. Walking
/// `00/` … `ff/` therefore yields exactly the global hash order, and
/// a cursor names the shard to restart in. Re-listing on resume is
/// bounded by shard width — 1/256th of the container — rather than
/// by the whole container, which is what a client-side skip over a
/// flat listing would cost on every single page.
///
/// `marker` is used only INSIDE one call, to page within a shard, and
/// never escapes as the cursor — the same treatment the S3 impl gives
/// its continuation token.
///
/// ## What this does NOT see, unlike S3
///
/// S3 lists the bucket with no prefix, so any foreign object lands in
/// `unknowns`. Constraining to `{2-hex}/` means foreign names outside
/// that shape are invisible here.
///
/// That asymmetry is deliberate and safe in the direction that
/// matters: an orphan is a blob **we** wrote and later stopped
/// referencing, so it always has the canonical name and is always
/// enumerated. Only genuinely foreign files — another workload
/// sharing the container — can be missed, and they are informational
/// notices, never findings. Trading them for O(N) enumeration instead
/// of O(N²/limit) is worth it.
fn list_blob_hashes(
&self,
cursor: Option<String>,
limit: usize,
) -> Pin<Box<dyn std::future::Future<Output = Result<BlobListPage, DomainError>> + Send + '_>>
{
Box::pin(async move {
// A run of foreign entries can't produce a resume cursor, and
// buffering the container to find one blob is worse than
// failing. Mirrors the S3 impl's bound, and like it is on
// entries accumulated rather than requests made: request
// count scales with the caller's `limit`, so a request cap
// would fire on a healthy container merely because the caller
// paged finely.
const MAX_UNKNOWNS: usize = 10_000;
// A shard is `{2-hex}/`, so the space is 0x00..=0xff.
const LAST_SHARD: u16 = 0xff;
let mut blobs: Vec<BackendBlobEntry> = Vec::new();
let mut unknowns: Vec<BackendUnknownEntry> = Vec::new();
// Resume in the cursor's own shard — its remaining entries
// still sort after it, and the client-side skip below drops
// the ones that don't. A malformed cursor is a bug in the
// caller's checkpoint, and silently restarting from `00`
// would re-report every blob as new, so refuse it.
let mut shard: u16 = match cursor.as_deref() {
Some(c) => u16::from(u8::from_str_radix(c.get(0..2).unwrap_or(""), 16).map_err(
|_| {
DomainError::internal_error(
"Blob",
format!(
"Azure enumeration cursor '{c}' is not a blob hash — it must \
start with the two hex characters naming its shard"
),
)
},
)?),
None => 0,
};
// Azure caps a page at 5000; asking for the caller's `limit`
// keeps a small page cheap. `MaxResults` rejects zero, and a
// caller asking for nothing still needs a well-formed
// request — and, more importantly, must not be answered with
// an empty page and a `None` cursor, which would read as
// "container fully enumerated, nothing here".
let want = limit.max(1);
let page_size = want.min(5000) as u32;
'shards: while shard <= LAST_SHARD {
let prefix = format!("{shard:02x}/");
// `Pageable` follows `next_marker` itself, so one stream
// covers the whole shard however many round-trips it takes.
let mut pages = self
.container_client
.list_blobs()
.prefix(prefix)
.max_results(std::num::NonZeroU32::new(page_size).expect("clamped above 0"))
.into_stream();
while let Some(page) = pages.next().await {
let page = page.map_err(|e| {
DomainError::internal_error(
"Blob",
format!(
"Azure ListBlobs failed on shard {shard:02x} of container '{}': {e}",
self.container_name
),
)
})?;
for blob in page.blobs.blobs() {
let name = blob.name.clone();
// `OffsetDateTime` → chrono, for the caller's
// grace window. A value outside chrono's range
// degrades to `None`, which the port documents as
// "treat as old enough" — the conservative side,
// since it only ever suppresses a finding on a
// freshly-written blob.
let mtime = chrono::DateTime::<chrono::Utc>::from_timestamp(
blob.properties.last_modified.unix_timestamp(),
blob.properties.last_modified.nanosecond(),
);
match Self::hash_from_blob_name(&name) {
Some(hash) => {
// `prefix` is inclusive of the cursor's own
// entry and of everything before it in the
// shard. Without this skip the caller sees
// a hash it already consumed and the
// merge-join never advances past it.
if cursor.as_deref().is_some_and(|c| hash.as_str() <= c) {
continue;
}
blobs.push(BackendBlobEntry { hash, mtime });
}
// Not ours — a foreign workload sharing the
// container. Surfaced rather than dropped so
// an operator can see it.
None => unknowns.push(BackendUnknownEntry { path: name, mtime }),
}
}
if blobs.len() >= want {
break 'shards;
}
if unknowns.len() >= MAX_UNKNOWNS {
return Err(DomainError::internal_error(
"Blob",
format!(
"Azure enumeration accumulated {} non-blob entrie(s) without \
filling a page, so no resume cursor can be produced. Container \
'{}' likely holds a large foreign namespace — give OxiCloud a \
dedicated container.",
unknowns.len(),
self.container_name,
),
));
}
}
shard += 1;
}
// Exhausting every shard is the ONLY end of enumeration.
// Stopping early because one shard was empty would truncate
// the sweep and report the rest of the container as absent,
// so `shard > LAST_SHARD` — not "this page was empty" — is
// what produces `None`.
let next_cursor = if shard > LAST_SHARD {
None
} else {
blobs.last().map(|entry| entry.hash.clone())
};
Ok(BlobListPage {
blobs,
unknowns,
next_cursor,
})
})
}
fn backend_type(&self) -> &'static str {
"azure"
}
@@ -406,14 +652,75 @@ impl BlobStorageBackend for AzureBlobBackend {
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
None
}
// TODO: implement `list_blob_hashes` via
// `container_client.list_blobs()` (`azure_storage_blobs`
// paginator). Same filter as local + S3 impls:
// `<xx>/<64-hex>.blob` naming. Currently inherits the trait
// default which returns `operation_not_supported` — the
// `backend_consistency` tenant handles that by emitting a
// single run-level `backend_unenumerable` finding and
// completing without per-blob probes. Ship as a follow-up once
// there's an Azure test environment to validate against.
}
#[cfg(test)]
mod tests {
use super::*;
const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9";
/// The enumeration cursor is fed straight back in as a shard prefix,
/// so a name that does not round-trip would resume in the wrong shard
/// and silently skip everything between.
#[test]
fn blob_name_round_trips_through_hash_from_blob_name() {
let name = AzureBlobBackend::blob_name(H);
assert_eq!(name, format!("0a/{H}.blob"));
assert_eq!(
AzureBlobBackend::hash_from_blob_name(&name).as_deref(),
Some(H)
);
}
/// Each of these would otherwise be treated as a hash — and the
/// resume path slices `[0..2]` off it to pick the next shard.
#[test]
fn non_canonical_names_are_rejected() {
let cases = [
"0a/junk.tmp".to_string(), // spool file
"junk.tmp".to_string(), // no shard
"0a/junk".to_string(), // no suffix
"thumbnails/abc.jpg".to_string(), // foreign namespace
format!("0a/{H}.blob.corrupt"), // sidecar
format!("0a/{H}"), // suffix missing
format!("zz/{H}.blob"), // non-hex shard
format!("ff/{H}.blob"), // shard != hash prefix
format!("0a/{}.blob", &H[..63]), // wrong length
];
for name in &cases {
assert_eq!(
AzureBlobBackend::hash_from_blob_name(name),
None,
"must not be read as a blob: {name}"
);
}
}
/// The shard walk relies on `{hash[0..2]}/…` ordering lexicographic
/// names into exactly the order `ORDER BY hash` produces. If the
/// shard were not the hash's own prefix the two sequences would
/// interleave differently and the merge-join would emit phantom
/// findings in BOTH directions.
#[test]
fn shard_order_matches_hash_order() {
let hashes = ["00aa", "0a1b", "0aff", "b0cd", "ffff"]
.map(|p| format!("{p}{}", "0".repeat(60)))
.to_vec();
let mut names: Vec<String> = hashes
.iter()
.map(|h| AzureBlobBackend::blob_name(h))
.collect();
names.sort();
let recovered: Vec<String> = names
.iter()
.filter_map(|n| AzureBlobBackend::hash_from_blob_name(n))
.collect();
let mut sorted_hashes = hashes.clone();
sorted_hashes.sort();
assert_eq!(recovered, sorted_hashes);
}
}
src/infrastructure/services/backend_consistency_service.rs
+55 -54
View File
@@ -49,15 +49,38 @@
//! that ever admitted uppercase or variable length would break this
//! silently and in both directions at once.
//!
//! ### Run-level check
//! ### When enumeration fails
//!
//! * `backend_unenumerable` (severity `anomaly`) — the backend
//! returned `operation_not_supported` on the first
//! `list_blob_hashes` call. Currently this fires when a
//! `MigrationBlobBackend` is active (refuses enumeration
//! mid-migration by design) or on an Azure backend (Azure impl
//! deferred). Informational — operators know they can't rely on
//! this scan under that config.
//! **The run fails.** There is no degraded mode.
//!
//! There used to be: an error on the first `list_blob_hashes` call
//! emitted a `backend_unenumerable` anomaly and fell back to
//! `probe_each_row`, one `blob_exists` per `storage.blobs` row. That
//! recovered `blob_missing_from_backend` (the direction that loses
//! FILES) but never `orphan_blob`, since bytes no row claims are
//! invisible to anything starting from the database.
//!
//! It was written for two cases, and neither exists:
//!
//! * **Azure** — enumerates since the 256-way shard walk (see
//! `AzureBlobBackend::list_blob_hashes` for why it needs one to do
//! what S3 gets from `StartAfter`).
//! * **Mid-migration** — never applied. That justification named a
//! `MigrationBlobBackend` that does not exist;
//! `SwappableBlobBackend::list_blob_hashes` forwards to whatever is
//! currently active, as do the Encrypted, Cached and Retry wrappers.
//! Do not reintroduce the claim without grepping for the impl.
//!
//! So the only thing still reaching it was a *transient* failure — auth
//! blip, throttle, network — being relabelled as a capability limit and
//! silently costing orphan coverage. A failed run is louder than an
//! anomaly on an otherwise-clean-looking scan, which was the fallback's
//! own stated goal.
//!
//! The trait default still returns `operation_not_supported`, so a
//! future write-only or read-only-mirror backend would fail every run
//! here. **That is when the fallback should come back — with tests.**
//! It had none, which is the other half of why it went.
//!
//! ### Grace window
//!
@@ -116,10 +139,6 @@ const BATCH_SIZE: usize = 500;
/// `blobs_consistency` + `dedup_gc`.
const CREATE_GRACE: Duration = Duration::hours(1);
/// Cap on affected-blob examples surfaced in the run-level
/// `backend_unenumerable` finding. Keeps the finding detail bounded.
const _MAX_EXAMPLES: usize = 5;
pub struct BackendConsistencyCheck {
pool: Arc<PgPool>,
/// Default backend to enumerate when `args.storage` is `None` —
@@ -172,10 +191,12 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
"Merge-joins the storage backend's blob enumeration against \
storage.blobs, both ordered by hash, so one pass yields the delta \
in both directions: bytes on the backend no DB row claims, and \
rows whose bytes are gone. Add ?deep=true to also read every \
matched blob back and re-hash it, catching silent bit-rot — that \
is a full read of storage and can take hours. Read-only in both \
modes: nothing is uploaded or deleted."
rows whose bytes are gone. If the backend cannot be enumerated \
the run fails rather than reporting partial coverage. \
Add ?deep=true to also read every matched blob back and re-hash \
it, catching silent bit-rot — that is a full read of storage and \
can take hours. Read-only in every mode: nothing is uploaded or \
deleted."
}
/// Approximate total: on a healthy install every backend blob
@@ -377,45 +398,25 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
let page = match backend.list_blob_hashes(cursor.clone(), BATCH_SIZE).await {
Ok(v) => v,
Err(e) => {
// Backend refuses / can't enumerate. First-batch
// failure = we emit ONE run-level anomaly and
// complete cleanly (the run stays useful — the
// operator learns why nothing was checked
// instead of getting a red error). Mid-scan
// failure = we fail the run.
let is_first_batch = cursor.is_none() && finding_count == 0;
if is_first_batch {
// No local increment — the local
// `finding_count` is only used for the
// completion log below, but this branch
// returns immediately. The finding IS
// persisted + counted in `stats.finding_count`
// by `record_or_log` → `store.record_finding`.
record_or_log(
store,
BACKEND_CONSISTENCY_JOB_NAME,
"backend_unenumerable",
"anomaly",
None,
serde_json::json!({
"backend": backend.backend_type(),
"error": format!("{e}"),
"note": "backend refused enumeration; no per-blob orphan probes attempted",
}),
)
.await;
tracing::info!(
target: "oxicloud::consistency",
event = "backend_consistency.unenumerable",
run_id = %store.run_id(),
backend = backend.backend_type(),
"backend refused enumeration (typical during migration or on backends without list support)"
);
return RunOutcome::completed();
}
// Fail loudly, first batch or not.
//
// A first-batch failure used to degrade to
// `probe_each_row` instead. That was written for
// backends which genuinely cannot enumerate, and none
// ship today — see the module docs for why the two it
// named do not apply. What was left reaching it was a
// transient error relabelled as a capability limit, on
// a run that then looked clean while having lost orphan
// coverage entirely.
//
// Whether the enumeration died on page 1 or page 900,
// the audit did not complete, and the operator needs to
// know that rather than read a green run.
return RunOutcome::Failed {
message: format!("backend list failed mid-scan: {e}"),
message: format!(
"backend enumeration failed on {}: {e}",
backend.backend_type()
),
};
}
};
tests/api/backend_consistency_azure.hurl
+261
View File
@@ -0,0 +1,261 @@
# =============================================================
# OxiCloud – backend_consistency against the Azure backend
#
# Runs against **Azurite**, the Azure Blob emulator started by
# `tests/common/spawn-db.sh`. It speaks the real Blob REST API, so this
# is the only way to exercise the Azure path without an account. The
# `azurite` storage entry is declared in `server.env` but never
# activated — the suite's active backend stays local, and this file
# reaches Azure explicitly through `?storage=azurite`.
#
# ## What it pins
#
# That Azure enumeration works against a real Blob REST implementation —
# SharedKey signing, prefix/marker paging, the 256-way shard walk and its
# termination. `AzureBlobBackend::list_blob_hashes` has unit tests for
# its name parser and ordering, but nothing else in the tree speaks the
# protocol.
#
# A failure surfaces as `ok: false`, because an enumeration error now
# fails the run. It used to degrade to a per-row probe — walk
# `storage.blobs`, ask "are these bytes there" — which found only the
# DB→backend direction and left orphans undetectable, since bytes no row
# claims are invisible to anything starting from the database. That
# fallback is gone; the module docs on `backend_consistency_service.rs`
# say why.
#
# ## What it deliberately does NOT assert
#
# Any finding count. The Azurite container starts empty and the run's
# grace window is one hour, so a freshly-uploaded blob is skipped in
# both directions by design — an audit here can only ever report zero.
# Asserting "zero findings" would pass whether enumeration worked or
# returned nothing at all.
#
# Real orphan/missing coverage needs blobs on the backend older than the
# grace window, which needs either a cutover into Azurite or a way to
# backdate `last_modified`. See "Why there is no cutover here" below.
#
# Prerequisites: setup.hurl must have run (admin user exists), and
# Azurite must be listening on 10000.
#
# Run:
# hurl --variables-file tests/api/test.env --file-root tests \
# --test tests/api/backend_consistency_azure.hurl
# =============================================================
# ─────────────────────────────────────────────────────────────
# Step 1 – Login
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/login
Content-Type: application/json
{
"username": "{{username}}",
"password": "{{password}}"
}
HTTP 200
[Captures]
token: jsonpath "$.access_token"
# ─────────────────────────────────────────────────────────────
# Step 2 – Upload a file so the registry is not empty.
#
# The bytes land on the ACTIVE (local) backend and never reach Azurite,
# so this does not feed step 3. It feeds step 4: a local sweep over an
# empty `storage.blobs` would satisfy every assertion there while
# comparing nothing, and the control is only a control if it had
# something to compare.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/folders
Authorization: Bearer {{token}}
Content-Type: application/json
{
"name": "hurl-azure-consistency"
}
HTTP 201
[Captures]
folder_id: jsonpath "$.id"
POST {{base_url}}/api/files/upload
Authorization: Bearer {{token}}
[MultipartFormData]
folder_id: {{folder_id}}
file: file,fixtures/red-image.png; image/png
HTTP 201
[Captures]
file_id: jsonpath "$.id"
# ─────────────────────────────────────────────────────────────
# Step 2b – Pre-flight: does the Azure backend work at all?
#
# Cheap, synchronous, and it isolates the failure. It separates "Azurite
# is missing, wedged, or misconfigured" from "enumeration is broken",
# which step 3 alone cannot: a dead container and a broken
# `list_blob_hashes` both surface there as the per-row fallback.
#
# `entry_name` resolves against OXICLOUD_STORAGE_ENTRIES, so this
# exercises the same entry step 3 audits rather than an ad-hoc config.
# It does a health check AND a write/read round-trip, and
# `phase_reached` names how far it got, so a failure points at a step
# rather than at the suite.
#
# Commonest cause of a failure here: the container does not exist.
# `AzureBlobBackend::initialize` verifies rather than creates, so
# `spawn-db.sh` provisions it with a hand-signed PUT.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/settings/storage/test
Authorization: Bearer {{token}}
Content-Type: application/json
{
"entry_name": "azurite"
}
HTTP 200
[Asserts]
jsonpath "$.connected" == true
jsonpath "$.roundtrip_passed" == true
jsonpath "$.bytes_written" != 0
jsonpath "$.bytes_read" != 0
# ─────────────────────────────────────────────────────────────
# Step 3 – Audit the Azure entry.
#
# `?storage=azurite` builds a backend for that named entry directly,
# bypassing the active-backend pointer — which is also the answer for
# auditing either side mid-migration.
#
# ## Why there is no cutover here — DECIDED, do not retry casually
#
# An earlier version ran `backend_migration ?storage=azurite` first, to
# put real bytes in the container. It hangs, on the first blob, and the
# cause is in `azure_core` 0.21 rather than in anything OxiCloud does.
#
# The chain, all verified in source:
#
# backend_migration_service.rs target.head_check(hash)
# → EncryptedBlobBackend::head_check
# → get_blob_range_stream(hash, 0, HEADER_SIZE) ~40 bytes
# → azure_core Range::as_headers adds x-ms-range-get-content-crc64
# to ANY range under 4 MiB
# → Azurite answers 500
# → azure_core classifies 500 retryable, response is deterministic,
# so it retries forever — while migration_readonly refuses writes
# application-wide.
#
# `head_check` is a pre-write format probe on the TARGET, so it fires
# before the first byte is copied. Nothing about the migration job is
# wrong; it works against real S3.
#
# **A workaround exists and was rejected** (2026-09-02): issue an
# unranged `get()` for small requests — its 16 MiB initial range clears
# the 4 MiB threshold, so the header is never sent — and truncate
# client-side. Correct against real Azure, but it pays for an emulator
# with production cost (a 40-byte probe becomes a whole-blob transfer)
# and puts new offset arithmetic on the read path, where a mistake
# serves wrong bytes silently instead of failing. See the note on
# `AzureBlobBackend::get_blob_range_stream`.
#
# So the cutover comes back with the official `azure_storage_blob` 1.x,
# where `range_get_content_crc64` is an explicit field to leave unset —
# and with it the orphan/missing assertions this file cannot make today.
# `docs/plan/jobs-handling-recoverable-error.md` covers the other half:
# the run should have paused with a reason instead of hanging, whatever
# the SDK does.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=azurite
Authorization: Bearer {{token}}
HTTP 200
[Asserts]
# THIS is the assertion that matters, and it is a real one. Enumeration
# failure is no longer degraded into a completed run — any Err from
# `list_blob_hashes` now fails the run — so a broken Azure enumeration
# surfaces right here as `ok: false`, whatever went wrong: signing,
# paging, the shard walk, the cursor.
#
# Before the fallback was deleted this needed a proxy assert on
# `extra_stats.mode`, because a broken enumeration completed "cleanly"
# with half its coverage silently gone.
jsonpath "$.ok" == true
jsonpath "$.outcome.outcome" == "ok"
# No unexpected notices. `backend_unenumerable` is gone with the
# fallback, so what this now guards against is a stray
# `unknown_backend_file` — a non-canonical name in the blob namespace,
# which on a container we provision ourselves means something wrote
# where it should not have.
jsonpath "$.outcome.extra.severity_counts.anomaly" not exists
# ── Known weakness, stated rather than hidden ────────────────────────
#
# A run that enumerates SUCCESSFULLY but returns nothing still passes.
# `scanned_count` would catch that, and it is deliberately not asserted
# here: the container is empty, so the job early-returns before its
# first checkpoint and 0 is the correct answer. Step 4 carries that
# assert instead, on the one entry that does hold blobs.
#
# Making it positive HERE needs the corpus on Azure, which needs the
# cutover — see the header. Planting blobs by hand is not a substitute:
# the DB side would then be walked against a backend that does not hold
# the corpus, and every local blob would report `blob_missing_from_backend`.
# ─────────────────────────────────────────────────────────────
# Step 4 – The same job against the LOCAL entry behaves identically.
#
# Azure and local now take the same code path, so this is no longer a
# contrast — it is the control. If a future change re-degrades Azure,
# this passing while step 3 fails localises the break to the Azure
# backend rather than to the job.
#
# It also carries the one POSITIVE assert this file can make. Local
# holds the blob step 2 uploaded, so the sweep gets past the empty-page
# early return and checkpoints — which is the only thing here that
# distinguishes "the merge-join compared something" from "the merge-join
# was handed nothing and completed".
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=local_main
Authorization: Bearer {{token}}
HTTP 200
[Asserts]
jsonpath "$.ok" == true
jsonpath "$.outcome.outcome" == "ok"
# `scanned_count` accumulates via `checkpoint`, which the early return
# skips — so a non-zero value means blobs were enumerated AND paired
# against `storage.blobs`, not merely that the run ended cleanly.
jsonpath "$.outcome.extra.scanned_count" != 0
# ─────────────────────────────────────────────────────────────
# Step 5 – Teardown. Hurl files share one database, so a folder left
# behind changes what later scenarios see.
# ─────────────────────────────────────────────────────────────
DELETE {{base_url}}/api/folders/{{folder_id}}
Authorization: Bearer {{token}}
HTTP 204
GET {{base_url}}/api/trash/resources
Authorization: Bearer {{token}}
HTTP 200
[Captures]
trash_id: jsonpath "$.items[?(@.resource.id == '{{folder_id}}')].resource.id"
DELETE {{base_url}}/api/trash/{{trash_id}}
Authorization: Bearer {{token}}
HTTP 200
tests/api/run.sh
+9 -1
View File
@@ -225,7 +225,15 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
"$API_DIR/webdav_permissions.hurl" \
"$API_DIR/webdav_nested_move_cascade.hurl" \
"$API_DIR/wopi_authz.hurl" \
"$API_DIR/wopi_shared_drive.hurl"
"$API_DIR/wopi_shared_drive.hurl" \
`# LAST, deliberately — and kept last even though it no longer cuts` \
`# the storage pointer over. It is the only scenario that depends on a` \
`# second service (Azurite on 10000), so if that container is missing` \
`# or wedged the failure lands after everything else has reported,` \
`# rather than in the middle of an otherwise-green run. It is also` \
`# where a cutover comes back once the official Azure SDK lands (see` \
`# the file header), and that WILL need to be last.` \
"$API_DIR/backend_consistency_azure.hurl"
#bash "$API_DIR/dedup_bulk_upload.sh"
tests/api/storage_multi_entry.hurl
+4 -1
View File
@@ -44,19 +44,22 @@ Authorization: Bearer {{admin_token}}
HTTP 200
[Asserts]
# Two entries declared, in _ENTRIES order.
jsonpath "$.entries" count == 2
jsonpath "$.entries" count == 3
jsonpath "$.entries[0].name" == "local_main"
jsonpath "$.entries[1].name" == "s3_stub"
jsonpath "$.entries[2].name" == "azurite"
# Backend types match the declarations.
jsonpath "$.entries[0].backend" == "local"
jsonpath "$.entries[1].backend" == "s3"
jsonpath "$.entries[2].backend" == "azure"
# Active pointer: fresh DB has no `active_backend_name` row, so the
# boot fallback picks the FIRST entry in _ENTRIES.
jsonpath "$.active_entry_name" == "local_main"
jsonpath "$.entries[0].is_active" == true
jsonpath "$.entries[1].is_active" == false
jsonpath "$.entries[2].is_active" == false
# Read-only mode off on a fresh boot (no in-flight migration, no
# stale flag in DB).
tests/common/docker-compose.test.yml
+29
View File
@@ -14,3 +14,32 @@ services:
interval: 2s
timeout: 5s
retries: 10
# Azure Blob Storage emulator. Speaks the real Blob REST API, so it is
# the only way to exercise the Azure backend without an account — and
# the account it does ship is the fixed, publicly-documented dev pair
# (`devstoreaccount1` + the well-known key), so nothing here is a
# credential worth protecting.
#
# Blob service only: `--blobHost` with no queue/table, since the backend
# uses blobs exclusively and the other two would just be open ports.
#
# tmpfs for the same reason postgres uses it — a run must start from an
# empty container, or a previous run's blobs read as orphans in the very
# audit this exists to test.
azurite-test:
image: mcr.microsoft.com/azure-storage/azurite:3.36.0
command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data --skipApiVersionCheck
ports:
- "10000:10000"
tmpfs:
- /data
healthcheck:
# Azurite answers 400 to an unauthenticated root GET, which is proof
# enough that the HTTP listener is up — the SDK handshake is what
# validates credentials, and that happens later in the backend's
# own `initialize`.
test: ["CMD-SHELL", "nc -z 127.0.0.1 10000 || exit 1"]
interval: 2s
timeout: 5s
retries: 15
tests/common/server.env
+19 -1
View File
@@ -42,7 +42,7 @@ OXICLOUD_NEXTCLOUD_ENABLED=true
# `s3_stub` is declared but never activated — it lets storage_config.hurl
# assert the entries table has more than one row without needing a
# real S3 backend.
OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub
OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite
OXICLOUD_STORAGE_local_main_BACKEND=local
OXICLOUD_STORAGE_s3_stub_BACKEND=s3
OXICLOUD_STORAGE_s3_stub_S3_BUCKET=oxicloud-test-stub
@@ -51,6 +51,24 @@ OXICLOUD_STORAGE_s3_stub_S3_ENDPOINT_URL=http://127.0.0.1:9999
OXICLOUD_STORAGE_s3_stub_S3_ACCESS_KEY=stub
OXICLOUD_STORAGE_s3_stub_S3_SECRET_KEY=stub
# `azurite` — a REAL, reachable Azure backend, unlike `s3_stub` above.
# It points at the Azurite emulator started by spawn-db.sh, which speaks
# the actual Blob REST API, so this is the only way to exercise the Azure
# path without an account.
#
# Declared but NOT activated: the active backend stays local_main, so the
# rest of the suite is untouched. Tests reach it explicitly through
# `?storage=azurite` on the jobs that accept an entry name — which is
# exactly what that parameter was added for.
#
# The credentials are Azurite's fixed, publicly-documented development
# pair. They are in every Microsoft quickstart; nothing here is secret.
OXICLOUD_STORAGE_azurite_BACKEND=azure
OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_NAME=devstoreaccount1
OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
OXICLOUD_STORAGE_azurite_AZURE_CONTAINER=oxicloud-test
OXICLOUD_STORAGE_azurite_AZURE_ENDPOINT_URL=http://127.0.0.1:10000/devstoreaccount1
RUST_LOG="warn,audit=info,sqlx::migrate=info"
#RUST_LOG="warn,audit=info,oxicloud::quota=debug"
#RUST_LOG=debug
tests/common/spawn-db.sh
+82 -1
View File
@@ -52,7 +52,57 @@ wait_for_postgres_ready() {
exit 1
}
echo "[setup] Starting test postgres..."
# Create the blob container Azurite serves.
#
# `AzureBlobBackend::initialize` VERIFIES the container exists and fails
# with a 404 if it does not — it deliberately does not create one, since
# auto-creating would turn a typo'd container name into a silently
# working empty container. So the harness provisions it, exactly as an
# operator would in production.
#
# Signed by hand rather than shelling out to the Azure CLI: `az` would
# mean pulling a ~700 MB image into every CI run to issue one PUT. This
# needs only curl and openssl, both already required.
#
# Two things make this fiddly enough to be worth commenting:
# * The key is base64 and HMAC needs raw bytes, so it is decoded and
# re-encoded as hex for `-macopt hexkey:`.
# * The canonicalized resource repeats the account name —
# `/{account}/{account}/{container}` — because the emulator puts the
# account in the URL path where real Azure puts it in the host. This
# is the classic Azurite signing trap; getting it wrong yields 403,
# not a hint.
#
# Idempotent by result: a second run gets 409 ContainerAlreadyExists,
# which is success for our purposes.
create_azurite_container() {
local acc=devstoreaccount1
local cont=oxicloud-test
# Azurite's fixed development key — hardcoded in the image, published
# by Microsoft, and valid against nothing but a local emulator.
local key='Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=='
local ver=2021-08-06
local date_hdr hexkey sts sig code
date_hdr=$(LC_ALL=C TZ=GMT date '+%a, %d %b %Y %H:%M:%S GMT')
hexkey=$(printf '%s' "$key" | base64 -d | od -An -tx1 | tr -d ' \n')
# Twelve leading empty lines are the unused standard headers
# (Content-*, Date, If-*, Range) the SharedKey scheme requires in
# fixed positions.
sts="PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:${date_hdr}\nx-ms-version:${ver}\n/${acc}/${acc}/${cont}\nrestype:container"
sig=$(printf '%b' "$sts" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${hexkey}" -binary | base64)
code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \
-H "x-ms-date: ${date_hdr}" \
-H "x-ms-version: ${ver}" \
-H "Authorization: SharedKey ${acc}:${sig}" \
"http://127.0.0.1:10000/${acc}/${cont}?restype=container" 2>/dev/null)
case "$code" in
201) echo "[setup] Azurite container '${cont}' created." ;;
409) echo "[setup] Azurite container '${cont}' already exists." ;;
*) echo "[setup] WARNING: could not create Azurite container '${cont}' (HTTP ${code}) — Azure-backed tests will fail." >&2 ;;
esac
}
echo "[setup] Starting test postgres + azurite..."
docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
docker compose -f "$COMPOSE_FILE" up -d
echo "[setup] Waiting for postgres on port 5433..."
@@ -60,3 +110,34 @@ wait_for_port 127.0.0.1 5433
echo "[setup] Waiting for postgres to accept queries..."
wait_for_postgres_ready
echo "[setup] Postgres is ready."
# Azurite backs the `azurite` storage entry, which `backend_consistency`
# audits via `?storage=azurite`. Only a port wait: the entry is declared
# but never activated, so nothing in the suite touches it until a test
# names it, and by then the listener has had the whole server boot to
# settle. A failure here should not take down a run that mostly does not
# use it — so this warns rather than exits, and the Azure test fails on
# its own terms with a clearer message than "setup timed out".
#
# Its own loop rather than `wait_for_port`: that helper calls `exit 1` on
# timeout, and `exit` inside a function ends the script whatever context
# it was called from — so wrapping it in an `if` would not degrade, it
# would just fail later and less clearly.
echo "[setup] Waiting for azurite on port 10000..."
azurite_deadline=$(( $(date +%s) + 30 ))
until nc -z 127.0.0.1 10000 2>/dev/null; do
if [[ $(date +%s) -ge $azurite_deadline ]]; then
echo "[setup] WARNING: azurite did not come up — Azure-backed tests will fail." >&2
break
fi
sleep 0.5
done
if nc -z 127.0.0.1 10000 2>/dev/null; then
# Port open only means the listener is up — NOT that the tests can use
# it. Saying "ready" here was misleading: the container was still
# missing, so the first Azure job failed with a 404 while setup had
# already reported success. `create_azurite_container` is what makes
# the claim true, so it is the one that gets to announce it.
echo "[setup] Azurite listening; provisioning container..."
create_azurite_container
fi