diff --git a/.cargo/audit.toml b/.cargo/audit.toml index 9e21fc99..24e0b57b 100644 --- a/.cargo/audit.toml +++ b/.cargo/audit.toml @@ -51,6 +51,33 @@ ignore = [ # No direct security impact; no upgrade path exists. "RUSTSEC-2024-0384", + # azure_core 0.21.0 writes the `authorization` header value to logs — + # `policies/transport.rs` does `debug!("…{request:#?}")` over the whole + # request. For a SharedKey entry that value is the per-request HMAC + # signature; for a SAS entry it is the token. Severity 6.5 (medium). + # + # The advisory says "upgrade to >=0.22.0". That version does not exist: + # `azure_core` jumped 0.21 → 1.x, and `azure_storage_blobs` never left + # 0.21.0 before being archived. So the stated remedy IS the official-SDK + # migration, tracked separately alongside the quick-xml pair above. + # + # Not reachable at our log levels: the line is `debug!` on the + # `azure_core::policies::transport` target, and the default filter is + # `info`. It fires only if an operator explicitly asks for + # `RUST_LOG=…,azure_core=debug`, which is not hypothetical — that is the + # invocation used to diagnose the Azurite migration hang. **Do not run + # `azure_core=debug` against a real Azure account**; it prints request + # signatures to the terminal. Against Azurite it only exposes the + # published dev key's signatures. + # + # A subscriber-level directive pinning that target off was prototyped + # and rejected 2026-09-02 — not worth carrying a filter hack for a + # dependency being replaced. + # + # Un-ignore trigger: the azure_storage_blob 1.x migration lands + # (`cargo tree -i azure_core@0.21` returns no rows). + "RUSTSEC-2026-0275", + # quick-xml 0.31.0 — transitive via azure_core 0.21.0 (unofficial SDK, # now archived). Our direct dep is already on 0.41.0; the 0.31 copy is # only reachable through the azure_storage_blobs chain, which parses diff --git a/docs/plan/jobs-handling-recoverable-error.md b/docs/plan/jobs-handling-recoverable-error.md new file mode 100644 index 00000000..3337c0d4 --- /dev/null +++ b/docs/plan/jobs-handling-recoverable-error.md @@ -0,0 +1,265 @@ +# Recoverable errors in jobs — retry, then pause + +**Status: not started.** Design settled 2026-08-31, from a live +diagnosis (see [Motivating incident](#motivating-incident)). + +A job that hits a failing backend today has two possible endings, and +neither is right for an outage: it fails the run (throwing away a +partially-complete migration, since `Failed` is terminal and only +`Paused` resumes), or it hangs forever inside an SDK retry loop with no +log line and no way to act on it. + +This plan adds the third: **retry a bounded number of times, then pause +with the reason recorded**, so an operator resumes when the provider +recovers and the job continues from its cursor. + +--- + +## Motivating incident + +`backend_migration ?storage=azurite` hung indefinitely. Diagnosis, after +several wrong theories: + +- The job issued a ranged GET carrying `x-ms-range-get-content-crc64`. +- Azurite answered **500** (real Azure supports CRC64 range validation; + the emulator does not). +- `azure_core`'s retry policy classifies 500 as retryable and loops. +- The response was deterministic, so every retry failed identically. +- The job never advanced, never failed, and emitted no per-blob line — + while holding `migration_readonly`, refusing writes **across the whole + application**. + +The exact chain was pinned down later (2026-09-02) and is worth having, +because it is not where you would look — the copy itself is innocent: + +``` +backend_migration_service.rs target.head_check(hash) ← pre-write probe + → EncryptedBlobBackend::head_check + → get_blob_range_stream(hash, 0, HEADER_SIZE) ← ~40 bytes + → azure_core Range::as_headers ← adds the CRC64 + (src/request_options/range.rs: any range < 4 MiB) header, no opt-out +``` + +`copy_blob` reads from the SOURCE, which is local in a local→Azure +migration, so it never touches an Azure range. What hangs is the format +probe against the TARGET, on the first blob, before a byte is copied. + +A workaround exists — unranged `get()` for small requests, truncate +client-side — and was **rejected 2026-09-02**: it pays for an emulator +with production read amplification and puts new offset arithmetic on the +read path. See the note on `AzureBlobBackend::get_blob_range_stream`. +The fix is the official SDK, where `range_get_content_crc64` is an +explicit field. **This plan is unaffected either way** — a bounded retry +would have turned the hang into a Paused run with a reason, which is the +point. + +Two properties made it invisible: the 500 was only visible at +`azure_core=debug`, and nothing bounded the retry. The same shape would +occur against real Azure or S3 on any persistent 5xx; it is not an +emulator quirk. `AzureBlobBackend` configures no retry policy and no +timeout at all — `grep "retry\|timeout\|ClientOptions"` on +`azure_blob_backend.rs` returns nothing. + +Full evidence chain, including the theories ruled out and what each +cost, is in the memory note `bug-azure-put-blob-hangs-no-timeout`. + +--- + +## Why this belongs at the top level + +Every recoverable job goes through `run_or_resume`, which already owns +the run lifecycle: it opens the row, persists and restores +[`JobRunArgs`](./job-registry.md) (`63820c3c`), dispatches the handler, +and writes the terminal state. Retry-and-pause is the same kind of +concern — policy about *how a run behaves*, not about what any one job +does. + +Implemented there, `backend_rotate`, `transcode_import`, the thumbnail +imports and anything added later inherit it. Implemented per-job, it +gets written once per job and drifts. + +It also survives the pending official-Azure-SDK migration untouched, +where per-SDK retry tuning would have to be redone. + +**The blocker is that the engine cannot currently act on what it is +told.** A handler returns `RunOutcome::{Completed, Paused, Failed}`, so +a transient backend error is already flattened into `Failed` before the +engine sees it — "the provider is down" and "this data is wrong" are +indistinguishable. Closing that is what makes a top-level +implementation possible, and it is step 1. + +--- + +## Step 1 — errors say whether they are retryable + +Today both backends wrap SDK errors into +`DomainError::internal_error("Azure", format!("…{e}"))`, so the status +code survives only inside a formatted string. Recovering it means +string-matching, which is exactly the kind of fragility that turns into +a silent behaviour change when an SDK reformats its `Display`. + +Carry the distinction on the type instead — a `retryable` flag, or a +kind the storage ports set deliberately. + +**Retryable** (environment, may clear on its own): +- HTTP 5xx, 429 / `SlowDown` / throttling +- connect timeouts, connection resets, DNS failure + +**Permanent** (will fail identically forever): +- 4xx other than 429 — 401/403 (credentials), 404 (missing container) +- decode failures, checksum mismatch +- `operation_not_supported` + +> **The Azurite 500 is a permanent error wearing a retryable status +> code.** No classification by status alone gets this right, which is +> the case for a bounded cap rather than "retry until it works". The cap +> is the safety net for exactly the errors the taxonomy misjudges. + +**Do not double-retry.** The AWS SDK already retries internally with its +own backoff, so a second layer above it multiplies. Check what the S3 +backend inherits before adding anything, and consider making the +engine's cap the *outer* bound with SDK retries reduced or disabled. + +## Step 2 — an outcome the engine can act on + +`RunOutcome` grows a variant meaning "the environment failed, this is +worth trying again later": + +```rust +RunOutcome::PausedRetryable { cursor: Vec, reason: String } +``` + +Distinct from all three existing outcomes, and the distinction is the +point: + +| outcome | meaning | resumes? | +|---|---|---| +| `Failed` | the data or the request is wrong | no — terminal | +| `Paused` | an operator asked it to stop | yes | +| `PausedRetryable` | the environment failed | yes, and says why | + +The row lands as `Paused` either way, so resume works unchanged. What +differs is `error_message`, which must let the panel — and an operator — +tell "I paused this" from "the provider went down". Without that +distinction a paused run is an unexplained one. + +## Step 3 — the engine implements the policy + +In `run_or_resume`: + +- bounded exponential backoff, ~5 attempts +- **log each failed attempt at `warn` on our side.** The incident took + several runs to diagnose because the 500 was visible only at + `azure_core=debug`. One line per exhausted operation, naming status, + target and attempt count. +- on exhaustion, write the row as `Paused` with the reason in + `error_message` + +Handlers then return the retryable outcome and get the policy for free. + +--- + +## Step 4 — `migration_readonly`, the sharp edge + +`backend_migration` holds a gate that refuses writes **application-wide** +until cutover. What happens to it on pause is a correctness question, +not a cosmetic one. + +### Start conservative: keep the gate held while paused + +Correct, because no writes during the pause means the cursor stays valid +and resume-from-cursor is sound. + +It is also **strictly better than today**, which is the thing to +remember when the read-only window looks unattractive: the application +is *already* read-only while the job hangs — there is simply no way to +see why or act. Same lock, now with a reason and two operator actions. + +**Cancel must clear the gate.** It ends the run with no swap, so the +source stays active and writes must return. The `failed > 0` path +already clears readonly on the reasoning that "users shouldn't be locked +out because of a partial run" — cancel is the escape hatch operators +will reach for during an outage, and it must work. + +**Make the state loud.** Writes refused app-wide should be obvious in +the admin panel, not discovered by reading a run row. The progress +snapshot already feeds the header middleware; a paused migration wants +the same visibility, saying why and offering resume/cancel. + +### Why NOT to release the gate on pause (yet) + +Tempting — ops should not be locked out during a provider outage — and +unsafe as the job stands: + +1. Release readonly; users write again. New blobs land on the source, + which is still active, and are absent from the target. +2. Resume continues **from the cursor**, a position in a hash-ordered + walk. +3. A blob written during the pause whose hash sorts *before* that cursor + is never visited. +4. The run completes, flips the pointer, and reads for that hash 404 + against a target that never received it. + +This is why the existing `failed > 0` path can safely clear readonly: it +**ends** the run, and an operator retrying starts a new run with a fresh +cursor, so everything is rescanned. Pause-and-resume is what +reintroduces the gap. + +Releasing on pause becomes safe only alongside one of: + +- **resume rescans from the beginning** rather than trusting the cursor + — cheap, because the walk short-circuits on blobs already present in + the target, so a second pass costs a lookup per blob, not a copy; or +- **a final catch-up pass under readonly before the swap**, with the + pointer flipping only when a complete pass finds nothing new. + +### Where this should end up + +The second option is the standard online-migration shape, and it is +where this job wants to go regardless: copy the bulk **without** holding +the gate, engage it only for a short final catch-up plus the swap. + +That removes what made the incident damaging — writes refused app-wide +for the entire duration of a long copy — and turns releasing-on-pause +into a free consequence rather than a correctness fix. Worth doing as a +follow-up, from the safer base this plan establishes. + +--- + +## Scope: Azure and S3 both + +The only reason S3 looks healthy is that the endpoint behaves. A +persistent 5xx from S3 hangs identically — the gap is the absence of a +bound, not anything Azure-specific. + +Putting the policy above the `BlobStorageBackend` trait covers Azure, +S3, local and anything added later with one implementation. Per-backend +SDK retry tuning stays a separate, optional refinement — and for Azure +specifically it should wait for the official SDK, since `azure_core` +0.21 is archived and queued for replacement. + +## Testing + +The `backend_consistency_azure.hurl` scenario and its Azurite service +are already wired (`tests/common/docker-compose.test.yml`, +`spawn-db.sh`), which gives a backend that reliably produces the +failure: Azurite 500s on the CRC64 ranged GET every time. That makes it +a genuine fixture for this work rather than a flake — +**deterministically unretryable-but-retryable-looking**, which is the +hard case. + +Note the scenario no longer triggers a migration — it audits Azurite +without cutting over, for the reason in its header. Reaching the fixture +means triggering `backend_migration ?storage=azurite` explicitly, which +is exactly the hang this plan is meant to convert into a Paused run. +Doing that inside the shared suite is what ordering it last was for; it +can go back once the outcome is bounded. + +Assert the run reaches `Paused`, that `error_message` names the cause, +and that it does so in bounded time rather than hanging. + +`POST /api/admin/settings/storage/test` with `entry_name` is the +pre-flight worth keeping in mind: synchronous, does a real write/read +round-trip, and isolates "the backend is misconfigured" from "the job +is broken". It passing while the migration hung is what ruled out +credentials, container and the write path during the incident. diff --git a/src/infrastructure/services/azure_blob_backend.rs b/src/infrastructure/services/azure_blob_backend.rs index 622af000..e8a18b43 100644 --- a/src/infrastructure/services/azure_blob_backend.rs +++ b/src/infrastructure/services/azure_blob_backend.rs @@ -13,7 +13,8 @@ use futures::{StreamExt, TryStreamExt}; use tokio::fs; use crate::application::ports::blob_storage_ports::{ - BlobStorageBackend, BlobStream, StorageHealthStatus, + BackendBlobEntry, BackendUnknownEntry, BlobListPage, BlobStorageBackend, BlobStream, + StorageHealthStatus, }; use crate::common::config::AzureStorageConfig; use crate::domain::errors::{DomainError, ErrorKind}; @@ -55,6 +56,29 @@ impl AzureBlobBackend { } } + /// Inverse of [`Self::blob_name`] — the hash a listing entry names, + /// or `None` when the entry is not one of ours. + /// + /// Mirrors `S3BlobBackend::hash_from_object_key`, including the check + /// that the shard equals the hash's own first two characters: without + /// it, `blob_name(hash)` would not reproduce the name we just parsed, + /// and a mis-sharded object would be reported as a live blob that no + /// read path can find. + fn hash_from_blob_name(name: &str) -> Option { + let (prefix, rest) = name.split_once('/')?; + if prefix.len() != 2 || !prefix.chars().all(|c| c.is_ascii_hexdigit()) { + return None; + } + let stem = rest.strip_suffix(".blob")?; + if stem.len() != 64 || !stem.chars().all(|c| c.is_ascii_hexdigit()) { + return None; + } + if !stem.starts_with(prefix) { + return None; + } + Some(stem.to_string()) + } + /// Compute the blob name for a given hash. fn blob_name(hash: &str) -> String { let prefix = &hash[0..2]; @@ -243,6 +267,37 @@ impl BlobStorageBackend for AzureBlobBackend { }) } + /// # Known incompatibility: sub-4 MiB ranges break on Azurite + /// + /// `azure_core` 0.21's `Range::as_headers` + /// (`src/request_options/range.rs`) attaches + /// `x-ms-range-get-content-crc64: true` to **any range shorter than + /// 4 MiB**, unconditionally and with no opt-out. Real Azure honours + /// it; Azurite answers 500. `azure_core` then classifies 500 as + /// retryable and loops on a deterministic error, forever. + /// + /// The reachable path is `backend_migration` → + /// `EncryptedBlobBackend::head_check` → + /// `get_blob_range_stream(hash, 0, HEADER_SIZE)`. `HEADER_SIZE` is a + /// few dozen bytes, and it runs against the TARGET before each write, + /// so a local→Azurite migration hangs on its first blob while holding + /// `migration_readonly` — writes refused application-wide. + /// + /// **Deliberately not worked around here.** The available workaround + /// is to issue an unranged `get()` for small requests (its 16 MiB + /// `initial_range` clears the threshold, so the header is never sent) + /// and truncate client-side. That is correct against real Azure but + /// pays for an emulator with production cost: a ~40-byte format probe + /// becomes a whole-blob transfer, and it puts new offset arithmetic + /// on the read path, where a mistake serves wrong bytes silently + /// rather than failing. + /// + /// The real fix is the official `azure_storage_blob` 1.x, where + /// `range_get_content_crc64` is an explicit field on + /// `BlobClientDownloadOptions` — leave it unset and the request is + /// never made. Until then, the Azurite suite exercises enumeration + /// and round-trips but not migration; see + /// `tests/api/backend_consistency_azure.hurl`. fn get_blob_range_stream( &self, hash: &str, @@ -394,6 +449,197 @@ impl BlobStorageBackend for AzureBlobBackend { }) } + /// Enumerate blob hashes in lexicographic order, so + /// `backend_consistency` can merge-join against `storage.blobs` + /// instead of degrading to a per-row probe that structurally cannot + /// see orphans. + /// + /// ## Why this is a shard walk and not one flat listing + /// + /// **The cursor IS a blob hash**, not a provider token. The caller + /// forces that: it advances ONE cursor across both sides of the join, + /// feeding the same value here and to `WHERE hash > $1` in SQL. S3 + /// satisfies it with `start_after(object_key(cursor))`. + /// + /// Azure has no `StartAfter`. REST API 2023-05-03 added `startFrom`, + /// which would be the direct equivalent — but this SDK + /// (`azure_storage_blobs` 0.21, archived) never sends it: `ListBlobs` + /// exposes only `prefix`, `delimiter`, `max_results` and `marker`, + /// and `marker` is an opaque continuation token that cannot be + /// derived from a hash. + /// + /// So resume rides on `prefix` instead. Names are + /// `{hash[0..2]}/{hash}.blob`, which partitions the container into + /// 256 shards that are themselves in hash order. Walking + /// `00/` … `ff/` therefore yields exactly the global hash order, and + /// a cursor names the shard to restart in. Re-listing on resume is + /// bounded by shard width — 1/256th of the container — rather than + /// by the whole container, which is what a client-side skip over a + /// flat listing would cost on every single page. + /// + /// `marker` is used only INSIDE one call, to page within a shard, and + /// never escapes as the cursor — the same treatment the S3 impl gives + /// its continuation token. + /// + /// ## What this does NOT see, unlike S3 + /// + /// S3 lists the bucket with no prefix, so any foreign object lands in + /// `unknowns`. Constraining to `{2-hex}/` means foreign names outside + /// that shape are invisible here. + /// + /// That asymmetry is deliberate and safe in the direction that + /// matters: an orphan is a blob **we** wrote and later stopped + /// referencing, so it always has the canonical name and is always + /// enumerated. Only genuinely foreign files — another workload + /// sharing the container — can be missed, and they are informational + /// notices, never findings. Trading them for O(N) enumeration instead + /// of O(N²/limit) is worth it. + fn list_blob_hashes( + &self, + cursor: Option, + limit: usize, + ) -> Pin> + Send + '_>> + { + Box::pin(async move { + // A run of foreign entries can't produce a resume cursor, and + // buffering the container to find one blob is worse than + // failing. Mirrors the S3 impl's bound, and like it is on + // entries accumulated rather than requests made: request + // count scales with the caller's `limit`, so a request cap + // would fire on a healthy container merely because the caller + // paged finely. + const MAX_UNKNOWNS: usize = 10_000; + + // A shard is `{2-hex}/`, so the space is 0x00..=0xff. + const LAST_SHARD: u16 = 0xff; + + let mut blobs: Vec = Vec::new(); + let mut unknowns: Vec = Vec::new(); + + // Resume in the cursor's own shard — its remaining entries + // still sort after it, and the client-side skip below drops + // the ones that don't. A malformed cursor is a bug in the + // caller's checkpoint, and silently restarting from `00` + // would re-report every blob as new, so refuse it. + let mut shard: u16 = match cursor.as_deref() { + Some(c) => u16::from(u8::from_str_radix(c.get(0..2).unwrap_or(""), 16).map_err( + |_| { + DomainError::internal_error( + "Blob", + format!( + "Azure enumeration cursor '{c}' is not a blob hash — it must \ + start with the two hex characters naming its shard" + ), + ) + }, + )?), + None => 0, + }; + + // Azure caps a page at 5000; asking for the caller's `limit` + // keeps a small page cheap. `MaxResults` rejects zero, and a + // caller asking for nothing still needs a well-formed + // request — and, more importantly, must not be answered with + // an empty page and a `None` cursor, which would read as + // "container fully enumerated, nothing here". + let want = limit.max(1); + let page_size = want.min(5000) as u32; + + 'shards: while shard <= LAST_SHARD { + let prefix = format!("{shard:02x}/"); + // `Pageable` follows `next_marker` itself, so one stream + // covers the whole shard however many round-trips it takes. + let mut pages = self + .container_client + .list_blobs() + .prefix(prefix) + .max_results(std::num::NonZeroU32::new(page_size).expect("clamped above 0")) + .into_stream(); + + while let Some(page) = pages.next().await { + let page = page.map_err(|e| { + DomainError::internal_error( + "Blob", + format!( + "Azure ListBlobs failed on shard {shard:02x} of container '{}': {e}", + self.container_name + ), + ) + })?; + + for blob in page.blobs.blobs() { + let name = blob.name.clone(); + // `OffsetDateTime` → chrono, for the caller's + // grace window. A value outside chrono's range + // degrades to `None`, which the port documents as + // "treat as old enough" — the conservative side, + // since it only ever suppresses a finding on a + // freshly-written blob. + let mtime = chrono::DateTime::::from_timestamp( + blob.properties.last_modified.unix_timestamp(), + blob.properties.last_modified.nanosecond(), + ); + + match Self::hash_from_blob_name(&name) { + Some(hash) => { + // `prefix` is inclusive of the cursor's own + // entry and of everything before it in the + // shard. Without this skip the caller sees + // a hash it already consumed and the + // merge-join never advances past it. + if cursor.as_deref().is_some_and(|c| hash.as_str() <= c) { + continue; + } + blobs.push(BackendBlobEntry { hash, mtime }); + } + // Not ours — a foreign workload sharing the + // container. Surfaced rather than dropped so + // an operator can see it. + None => unknowns.push(BackendUnknownEntry { path: name, mtime }), + } + } + + if blobs.len() >= want { + break 'shards; + } + + if unknowns.len() >= MAX_UNKNOWNS { + return Err(DomainError::internal_error( + "Blob", + format!( + "Azure enumeration accumulated {} non-blob entrie(s) without \ + filling a page, so no resume cursor can be produced. Container \ + '{}' likely holds a large foreign namespace — give OxiCloud a \ + dedicated container.", + unknowns.len(), + self.container_name, + ), + )); + } + } + + shard += 1; + } + + // Exhausting every shard is the ONLY end of enumeration. + // Stopping early because one shard was empty would truncate + // the sweep and report the rest of the container as absent, + // so `shard > LAST_SHARD` — not "this page was empty" — is + // what produces `None`. + let next_cursor = if shard > LAST_SHARD { + None + } else { + blobs.last().map(|entry| entry.hash.clone()) + }; + + Ok(BlobListPage { + blobs, + unknowns, + next_cursor, + }) + }) + } + fn backend_type(&self) -> &'static str { "azure" } @@ -406,14 +652,75 @@ impl BlobStorageBackend for AzureBlobBackend { fn local_blob_path(&self, _hash: &str) -> Option { None } - - // TODO: implement `list_blob_hashes` via - // `container_client.list_blobs()` (`azure_storage_blobs` - // paginator). Same filter as local + S3 impls: - // `/<64-hex>.blob` naming. Currently inherits the trait - // default which returns `operation_not_supported` — the - // `backend_consistency` tenant handles that by emitting a - // single run-level `backend_unenumerable` finding and - // completing without per-blob probes. Ship as a follow-up once - // there's an Azure test environment to validate against. +} + +#[cfg(test)] +mod tests { + use super::*; + + const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + + /// The enumeration cursor is fed straight back in as a shard prefix, + /// so a name that does not round-trip would resume in the wrong shard + /// and silently skip everything between. + #[test] + fn blob_name_round_trips_through_hash_from_blob_name() { + let name = AzureBlobBackend::blob_name(H); + assert_eq!(name, format!("0a/{H}.blob")); + assert_eq!( + AzureBlobBackend::hash_from_blob_name(&name).as_deref(), + Some(H) + ); + } + + /// Each of these would otherwise be treated as a hash — and the + /// resume path slices `[0..2]` off it to pick the next shard. + #[test] + fn non_canonical_names_are_rejected() { + let cases = [ + "0a/junk.tmp".to_string(), // spool file + "junk.tmp".to_string(), // no shard + "0a/junk".to_string(), // no suffix + "thumbnails/abc.jpg".to_string(), // foreign namespace + format!("0a/{H}.blob.corrupt"), // sidecar + format!("0a/{H}"), // suffix missing + format!("zz/{H}.blob"), // non-hex shard + format!("ff/{H}.blob"), // shard != hash prefix + format!("0a/{}.blob", &H[..63]), // wrong length + ]; + for name in &cases { + assert_eq!( + AzureBlobBackend::hash_from_blob_name(name), + None, + "must not be read as a blob: {name}" + ); + } + } + + /// The shard walk relies on `{hash[0..2]}/…` ordering lexicographic + /// names into exactly the order `ORDER BY hash` produces. If the + /// shard were not the hash's own prefix the two sequences would + /// interleave differently and the merge-join would emit phantom + /// findings in BOTH directions. + #[test] + fn shard_order_matches_hash_order() { + let hashes = ["00aa", "0a1b", "0aff", "b0cd", "ffff"] + .map(|p| format!("{p}{}", "0".repeat(60))) + .to_vec(); + + let mut names: Vec = hashes + .iter() + .map(|h| AzureBlobBackend::blob_name(h)) + .collect(); + names.sort(); + + let recovered: Vec = names + .iter() + .filter_map(|n| AzureBlobBackend::hash_from_blob_name(n)) + .collect(); + + let mut sorted_hashes = hashes.clone(); + sorted_hashes.sort(); + assert_eq!(recovered, sorted_hashes); + } } diff --git a/src/infrastructure/services/backend_consistency_service.rs b/src/infrastructure/services/backend_consistency_service.rs index 2b94f819..4ba04798 100644 --- a/src/infrastructure/services/backend_consistency_service.rs +++ b/src/infrastructure/services/backend_consistency_service.rs @@ -49,15 +49,38 @@ //! that ever admitted uppercase or variable length would break this //! silently and in both directions at once. //! -//! ### Run-level check +//! ### When enumeration fails //! -//! * `backend_unenumerable` (severity `anomaly`) — the backend -//! returned `operation_not_supported` on the first -//! `list_blob_hashes` call. Currently this fires when a -//! `MigrationBlobBackend` is active (refuses enumeration -//! mid-migration by design) or on an Azure backend (Azure impl -//! deferred). Informational — operators know they can't rely on -//! this scan under that config. +//! **The run fails.** There is no degraded mode. +//! +//! There used to be: an error on the first `list_blob_hashes` call +//! emitted a `backend_unenumerable` anomaly and fell back to +//! `probe_each_row`, one `blob_exists` per `storage.blobs` row. That +//! recovered `blob_missing_from_backend` (the direction that loses +//! FILES) but never `orphan_blob`, since bytes no row claims are +//! invisible to anything starting from the database. +//! +//! It was written for two cases, and neither exists: +//! +//! * **Azure** — enumerates since the 256-way shard walk (see +//! `AzureBlobBackend::list_blob_hashes` for why it needs one to do +//! what S3 gets from `StartAfter`). +//! * **Mid-migration** — never applied. That justification named a +//! `MigrationBlobBackend` that does not exist; +//! `SwappableBlobBackend::list_blob_hashes` forwards to whatever is +//! currently active, as do the Encrypted, Cached and Retry wrappers. +//! Do not reintroduce the claim without grepping for the impl. +//! +//! So the only thing still reaching it was a *transient* failure — auth +//! blip, throttle, network — being relabelled as a capability limit and +//! silently costing orphan coverage. A failed run is louder than an +//! anomaly on an otherwise-clean-looking scan, which was the fallback's +//! own stated goal. +//! +//! The trait default still returns `operation_not_supported`, so a +//! future write-only or read-only-mirror backend would fail every run +//! here. **That is when the fallback should come back — with tests.** +//! It had none, which is the other half of why it went. //! //! ### Grace window //! @@ -116,10 +139,6 @@ const BATCH_SIZE: usize = 500; /// `blobs_consistency` + `dedup_gc`. const CREATE_GRACE: Duration = Duration::hours(1); -/// Cap on affected-blob examples surfaced in the run-level -/// `backend_unenumerable` finding. Keeps the finding detail bounded. -const _MAX_EXAMPLES: usize = 5; - pub struct BackendConsistencyCheck { pool: Arc, /// Default backend to enumerate when `args.storage` is `None` — @@ -172,10 +191,12 @@ impl RecoverableJobHandler for BackendConsistencyCheck { "Merge-joins the storage backend's blob enumeration against \ storage.blobs, both ordered by hash, so one pass yields the delta \ in both directions: bytes on the backend no DB row claims, and \ - rows whose bytes are gone. Add ?deep=true to also read every \ - matched blob back and re-hash it, catching silent bit-rot — that \ - is a full read of storage and can take hours. Read-only in both \ - modes: nothing is uploaded or deleted." + rows whose bytes are gone. If the backend cannot be enumerated \ + the run fails rather than reporting partial coverage. \ + Add ?deep=true to also read every matched blob back and re-hash \ + it, catching silent bit-rot — that is a full read of storage and \ + can take hours. Read-only in every mode: nothing is uploaded or \ + deleted." } /// Approximate total: on a healthy install every backend blob @@ -377,45 +398,25 @@ impl RecoverableJobHandler for BackendConsistencyCheck { let page = match backend.list_blob_hashes(cursor.clone(), BATCH_SIZE).await { Ok(v) => v, Err(e) => { - // Backend refuses / can't enumerate. First-batch - // failure = we emit ONE run-level anomaly and - // complete cleanly (the run stays useful — the - // operator learns why nothing was checked - // instead of getting a red error). Mid-scan - // failure = we fail the run. - - let is_first_batch = cursor.is_none() && finding_count == 0; - if is_first_batch { - // No local increment — the local - // `finding_count` is only used for the - // completion log below, but this branch - // returns immediately. The finding IS - // persisted + counted in `stats.finding_count` - // by `record_or_log` → `store.record_finding`. - record_or_log( - store, - BACKEND_CONSISTENCY_JOB_NAME, - "backend_unenumerable", - "anomaly", - None, - serde_json::json!({ - "backend": backend.backend_type(), - "error": format!("{e}"), - "note": "backend refused enumeration; no per-blob orphan probes attempted", - }), - ) - .await; - tracing::info!( - target: "oxicloud::consistency", - event = "backend_consistency.unenumerable", - run_id = %store.run_id(), - backend = backend.backend_type(), - "backend refused enumeration (typical during migration or on backends without list support)" - ); - return RunOutcome::completed(); - } + // Fail loudly, first batch or not. + // + // A first-batch failure used to degrade to + // `probe_each_row` instead. That was written for + // backends which genuinely cannot enumerate, and none + // ship today — see the module docs for why the two it + // named do not apply. What was left reaching it was a + // transient error relabelled as a capability limit, on + // a run that then looked clean while having lost orphan + // coverage entirely. + // + // Whether the enumeration died on page 1 or page 900, + // the audit did not complete, and the operator needs to + // know that rather than read a green run. return RunOutcome::Failed { - message: format!("backend list failed mid-scan: {e}"), + message: format!( + "backend enumeration failed on {}: {e}", + backend.backend_type() + ), }; } }; diff --git a/tests/api/backend_consistency_azure.hurl b/tests/api/backend_consistency_azure.hurl new file mode 100644 index 00000000..5338c2bb --- /dev/null +++ b/tests/api/backend_consistency_azure.hurl @@ -0,0 +1,261 @@ +# ============================================================= +# OxiCloud – backend_consistency against the Azure backend +# +# Runs against **Azurite**, the Azure Blob emulator started by +# `tests/common/spawn-db.sh`. It speaks the real Blob REST API, so this +# is the only way to exercise the Azure path without an account. The +# `azurite` storage entry is declared in `server.env` but never +# activated — the suite's active backend stays local, and this file +# reaches Azure explicitly through `?storage=azurite`. +# +# ## What it pins +# +# That Azure enumeration works against a real Blob REST implementation — +# SharedKey signing, prefix/marker paging, the 256-way shard walk and its +# termination. `AzureBlobBackend::list_blob_hashes` has unit tests for +# its name parser and ordering, but nothing else in the tree speaks the +# protocol. +# +# A failure surfaces as `ok: false`, because an enumeration error now +# fails the run. It used to degrade to a per-row probe — walk +# `storage.blobs`, ask "are these bytes there" — which found only the +# DB→backend direction and left orphans undetectable, since bytes no row +# claims are invisible to anything starting from the database. That +# fallback is gone; the module docs on `backend_consistency_service.rs` +# say why. +# +# ## What it deliberately does NOT assert +# +# Any finding count. The Azurite container starts empty and the run's +# grace window is one hour, so a freshly-uploaded blob is skipped in +# both directions by design — an audit here can only ever report zero. +# Asserting "zero findings" would pass whether enumeration worked or +# returned nothing at all. +# +# Real orphan/missing coverage needs blobs on the backend older than the +# grace window, which needs either a cutover into Azurite or a way to +# backdate `last_modified`. See "Why there is no cutover here" below. +# +# Prerequisites: setup.hurl must have run (admin user exists), and +# Azurite must be listening on 10000. +# +# Run: +# hurl --variables-file tests/api/test.env --file-root tests \ +# --test tests/api/backend_consistency_azure.hurl +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Upload a file so the registry is not empty. +# +# The bytes land on the ACTIVE (local) backend and never reach Azurite, +# so this does not feed step 3. It feeds step 4: a local sweep over an +# empty `storage.blobs` would satisfy every assertion there while +# comparing nothing, and the control is only a control if it had +# something to compare. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-azure-consistency" +} + +HTTP 201 +[Captures] +folder_id: jsonpath "$.id" + + +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +folder_id: {{folder_id}} +file: file,fixtures/red-image.png; image/png + +HTTP 201 +[Captures] +file_id: jsonpath "$.id" + + +# ───────────────────────────────────────────────────────────── +# Step 2b – Pre-flight: does the Azure backend work at all? +# +# Cheap, synchronous, and it isolates the failure. It separates "Azurite +# is missing, wedged, or misconfigured" from "enumeration is broken", +# which step 3 alone cannot: a dead container and a broken +# `list_blob_hashes` both surface there as the per-row fallback. +# +# `entry_name` resolves against OXICLOUD_STORAGE_ENTRIES, so this +# exercises the same entry step 3 audits rather than an ad-hoc config. +# It does a health check AND a write/read round-trip, and +# `phase_reached` names how far it got, so a failure points at a step +# rather than at the suite. +# +# Commonest cause of a failure here: the container does not exist. +# `AzureBlobBackend::initialize` verifies rather than creates, so +# `spawn-db.sh` provisions it with a hand-signed PUT. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/settings/storage/test +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "entry_name": "azurite" +} + +HTTP 200 +[Asserts] +jsonpath "$.connected" == true +jsonpath "$.roundtrip_passed" == true +jsonpath "$.bytes_written" != 0 +jsonpath "$.bytes_read" != 0 + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Audit the Azure entry. +# +# `?storage=azurite` builds a backend for that named entry directly, +# bypassing the active-backend pointer — which is also the answer for +# auditing either side mid-migration. +# +# ## Why there is no cutover here — DECIDED, do not retry casually +# +# An earlier version ran `backend_migration ?storage=azurite` first, to +# put real bytes in the container. It hangs, on the first blob, and the +# cause is in `azure_core` 0.21 rather than in anything OxiCloud does. +# +# The chain, all verified in source: +# +# backend_migration_service.rs target.head_check(hash) +# → EncryptedBlobBackend::head_check +# → get_blob_range_stream(hash, 0, HEADER_SIZE) ~40 bytes +# → azure_core Range::as_headers adds x-ms-range-get-content-crc64 +# to ANY range under 4 MiB +# → Azurite answers 500 +# → azure_core classifies 500 retryable, response is deterministic, +# so it retries forever — while migration_readonly refuses writes +# application-wide. +# +# `head_check` is a pre-write format probe on the TARGET, so it fires +# before the first byte is copied. Nothing about the migration job is +# wrong; it works against real S3. +# +# **A workaround exists and was rejected** (2026-09-02): issue an +# unranged `get()` for small requests — its 16 MiB initial range clears +# the 4 MiB threshold, so the header is never sent — and truncate +# client-side. Correct against real Azure, but it pays for an emulator +# with production cost (a 40-byte probe becomes a whole-blob transfer) +# and puts new offset arithmetic on the read path, where a mistake +# serves wrong bytes silently instead of failing. See the note on +# `AzureBlobBackend::get_blob_range_stream`. +# +# So the cutover comes back with the official `azure_storage_blob` 1.x, +# where `range_get_content_crc64` is an explicit field to leave unset — +# and with it the orphan/missing assertions this file cannot make today. +# `docs/plan/jobs-handling-recoverable-error.md` covers the other half: +# the run should have paused with a reason instead of hanging, whatever +# the SDK does. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=azurite +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] + +# THIS is the assertion that matters, and it is a real one. Enumeration +# failure is no longer degraded into a completed run — any Err from +# `list_blob_hashes` now fails the run — so a broken Azure enumeration +# surfaces right here as `ok: false`, whatever went wrong: signing, +# paging, the shard walk, the cursor. +# +# Before the fallback was deleted this needed a proxy assert on +# `extra_stats.mode`, because a broken enumeration completed "cleanly" +# with half its coverage silently gone. +jsonpath "$.ok" == true +jsonpath "$.outcome.outcome" == "ok" + +# No unexpected notices. `backend_unenumerable` is gone with the +# fallback, so what this now guards against is a stray +# `unknown_backend_file` — a non-canonical name in the blob namespace, +# which on a container we provision ourselves means something wrote +# where it should not have. +jsonpath "$.outcome.extra.severity_counts.anomaly" not exists + +# ── Known weakness, stated rather than hidden ──────────────────────── +# +# A run that enumerates SUCCESSFULLY but returns nothing still passes. +# `scanned_count` would catch that, and it is deliberately not asserted +# here: the container is empty, so the job early-returns before its +# first checkpoint and 0 is the correct answer. Step 4 carries that +# assert instead, on the one entry that does hold blobs. +# +# Making it positive HERE needs the corpus on Azure, which needs the +# cutover — see the header. Planting blobs by hand is not a substitute: +# the DB side would then be walked against a backend that does not hold +# the corpus, and every local blob would report `blob_missing_from_backend`. + + +# ───────────────────────────────────────────────────────────── +# Step 4 – The same job against the LOCAL entry behaves identically. +# +# Azure and local now take the same code path, so this is no longer a +# contrast — it is the control. If a future change re-degrades Azure, +# this passing while step 3 fails localises the break to the Azure +# backend rather than to the job. +# +# It also carries the one POSITIVE assert this file can make. Local +# holds the blob step 2 uploaded, so the sweep gets past the empty-page +# early return and checkpoints — which is the only thing here that +# distinguishes "the merge-join compared something" from "the merge-join +# was handed nothing and completed". +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=local_main +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.ok" == true +jsonpath "$.outcome.outcome" == "ok" + +# `scanned_count` accumulates via `checkpoint`, which the early return +# skips — so a non-zero value means blobs were enumerated AND paired +# against `storage.blobs`, not merely that the run ended cleanly. +jsonpath "$.outcome.extra.scanned_count" != 0 + + +# ───────────────────────────────────────────────────────────── +# Step 5 – Teardown. Hurl files share one database, so a folder left +# behind changes what later scenarios see. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/folders/{{folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_id: jsonpath "$.items[?(@.resource.id == '{{folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_id}} +Authorization: Bearer {{token}} + +HTTP 200 diff --git a/tests/api/run.sh b/tests/api/run.sh index 1c483ea1..9afd51bf 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -225,7 +225,15 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/webdav_permissions.hurl" \ "$API_DIR/webdav_nested_move_cascade.hurl" \ "$API_DIR/wopi_authz.hurl" \ - "$API_DIR/wopi_shared_drive.hurl" + "$API_DIR/wopi_shared_drive.hurl" \ + `# LAST, deliberately — and kept last even though it no longer cuts` \ + `# the storage pointer over. It is the only scenario that depends on a` \ + `# second service (Azurite on 10000), so if that container is missing` \ + `# or wedged the failure lands after everything else has reported,` \ + `# rather than in the middle of an otherwise-green run. It is also` \ + `# where a cutover comes back once the official Azure SDK lands (see` \ + `# the file header), and that WILL need to be last.` \ + "$API_DIR/backend_consistency_azure.hurl" #bash "$API_DIR/dedup_bulk_upload.sh" diff --git a/tests/api/storage_multi_entry.hurl b/tests/api/storage_multi_entry.hurl index 49bfe561..847e93fb 100644 --- a/tests/api/storage_multi_entry.hurl +++ b/tests/api/storage_multi_entry.hurl @@ -44,19 +44,22 @@ Authorization: Bearer {{admin_token}} HTTP 200 [Asserts] # Two entries declared, in _ENTRIES order. -jsonpath "$.entries" count == 2 +jsonpath "$.entries" count == 3 jsonpath "$.entries[0].name" == "local_main" jsonpath "$.entries[1].name" == "s3_stub" +jsonpath "$.entries[2].name" == "azurite" # Backend types match the declarations. jsonpath "$.entries[0].backend" == "local" jsonpath "$.entries[1].backend" == "s3" +jsonpath "$.entries[2].backend" == "azure" # Active pointer: fresh DB has no `active_backend_name` row, so the # boot fallback picks the FIRST entry in _ENTRIES. jsonpath "$.active_entry_name" == "local_main" jsonpath "$.entries[0].is_active" == true jsonpath "$.entries[1].is_active" == false +jsonpath "$.entries[2].is_active" == false # Read-only mode off on a fresh boot (no in-flight migration, no # stale flag in DB). diff --git a/tests/common/docker-compose.test.yml b/tests/common/docker-compose.test.yml index cd6e5fb3..4412eaeb 100644 --- a/tests/common/docker-compose.test.yml +++ b/tests/common/docker-compose.test.yml @@ -14,3 +14,32 @@ services: interval: 2s timeout: 5s retries: 10 + + # Azure Blob Storage emulator. Speaks the real Blob REST API, so it is + # the only way to exercise the Azure backend without an account — and + # the account it does ship is the fixed, publicly-documented dev pair + # (`devstoreaccount1` + the well-known key), so nothing here is a + # credential worth protecting. + # + # Blob service only: `--blobHost` with no queue/table, since the backend + # uses blobs exclusively and the other two would just be open ports. + # + # tmpfs for the same reason postgres uses it — a run must start from an + # empty container, or a previous run's blobs read as orphans in the very + # audit this exists to test. + azurite-test: + image: mcr.microsoft.com/azure-storage/azurite:3.36.0 + command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data --skipApiVersionCheck + ports: + - "10000:10000" + tmpfs: + - /data + healthcheck: + # Azurite answers 400 to an unauthenticated root GET, which is proof + # enough that the HTTP listener is up — the SDK handshake is what + # validates credentials, and that happens later in the backend's + # own `initialize`. + test: ["CMD-SHELL", "nc -z 127.0.0.1 10000 || exit 1"] + interval: 2s + timeout: 5s + retries: 15 diff --git a/tests/common/server.env b/tests/common/server.env index 5c2b7018..a24a4127 100644 --- a/tests/common/server.env +++ b/tests/common/server.env @@ -42,7 +42,7 @@ OXICLOUD_NEXTCLOUD_ENABLED=true # `s3_stub` is declared but never activated — it lets storage_config.hurl # assert the entries table has more than one row without needing a # real S3 backend. -OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub +OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite OXICLOUD_STORAGE_local_main_BACKEND=local OXICLOUD_STORAGE_s3_stub_BACKEND=s3 OXICLOUD_STORAGE_s3_stub_S3_BUCKET=oxicloud-test-stub @@ -51,6 +51,24 @@ OXICLOUD_STORAGE_s3_stub_S3_ENDPOINT_URL=http://127.0.0.1:9999 OXICLOUD_STORAGE_s3_stub_S3_ACCESS_KEY=stub OXICLOUD_STORAGE_s3_stub_S3_SECRET_KEY=stub +# `azurite` — a REAL, reachable Azure backend, unlike `s3_stub` above. +# It points at the Azurite emulator started by spawn-db.sh, which speaks +# the actual Blob REST API, so this is the only way to exercise the Azure +# path without an account. +# +# Declared but NOT activated: the active backend stays local_main, so the +# rest of the suite is untouched. Tests reach it explicitly through +# `?storage=azurite` on the jobs that accept an entry name — which is +# exactly what that parameter was added for. +# +# The credentials are Azurite's fixed, publicly-documented development +# pair. They are in every Microsoft quickstart; nothing here is secret. +OXICLOUD_STORAGE_azurite_BACKEND=azure +OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_NAME=devstoreaccount1 +OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== +OXICLOUD_STORAGE_azurite_AZURE_CONTAINER=oxicloud-test +OXICLOUD_STORAGE_azurite_AZURE_ENDPOINT_URL=http://127.0.0.1:10000/devstoreaccount1 + RUST_LOG="warn,audit=info,sqlx::migrate=info" #RUST_LOG="warn,audit=info,oxicloud::quota=debug" #RUST_LOG=debug diff --git a/tests/common/spawn-db.sh b/tests/common/spawn-db.sh index 0db7cff3..ec67f59c 100755 --- a/tests/common/spawn-db.sh +++ b/tests/common/spawn-db.sh @@ -52,7 +52,57 @@ wait_for_postgres_ready() { exit 1 } -echo "[setup] Starting test postgres..." +# Create the blob container Azurite serves. +# +# `AzureBlobBackend::initialize` VERIFIES the container exists and fails +# with a 404 if it does not — it deliberately does not create one, since +# auto-creating would turn a typo'd container name into a silently +# working empty container. So the harness provisions it, exactly as an +# operator would in production. +# +# Signed by hand rather than shelling out to the Azure CLI: `az` would +# mean pulling a ~700 MB image into every CI run to issue one PUT. This +# needs only curl and openssl, both already required. +# +# Two things make this fiddly enough to be worth commenting: +# * The key is base64 and HMAC needs raw bytes, so it is decoded and +# re-encoded as hex for `-macopt hexkey:`. +# * The canonicalized resource repeats the account name — +# `/{account}/{account}/{container}` — because the emulator puts the +# account in the URL path where real Azure puts it in the host. This +# is the classic Azurite signing trap; getting it wrong yields 403, +# not a hint. +# +# Idempotent by result: a second run gets 409 ContainerAlreadyExists, +# which is success for our purposes. +create_azurite_container() { + local acc=devstoreaccount1 + local cont=oxicloud-test + # Azurite's fixed development key — hardcoded in the image, published + # by Microsoft, and valid against nothing but a local emulator. + local key='Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==' + local ver=2021-08-06 + local date_hdr hexkey sts sig code + date_hdr=$(LC_ALL=C TZ=GMT date '+%a, %d %b %Y %H:%M:%S GMT') + hexkey=$(printf '%s' "$key" | base64 -d | od -An -tx1 | tr -d ' \n') + # Twelve leading empty lines are the unused standard headers + # (Content-*, Date, If-*, Range) the SharedKey scheme requires in + # fixed positions. + sts="PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:${date_hdr}\nx-ms-version:${ver}\n/${acc}/${acc}/${cont}\nrestype:container" + sig=$(printf '%b' "$sts" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${hexkey}" -binary | base64) + code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \ + -H "x-ms-date: ${date_hdr}" \ + -H "x-ms-version: ${ver}" \ + -H "Authorization: SharedKey ${acc}:${sig}" \ + "http://127.0.0.1:10000/${acc}/${cont}?restype=container" 2>/dev/null) + case "$code" in + 201) echo "[setup] Azurite container '${cont}' created." ;; + 409) echo "[setup] Azurite container '${cont}' already exists." ;; + *) echo "[setup] WARNING: could not create Azurite container '${cont}' (HTTP ${code}) — Azure-backed tests will fail." >&2 ;; + esac +} + +echo "[setup] Starting test postgres + azurite..." docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true docker compose -f "$COMPOSE_FILE" up -d echo "[setup] Waiting for postgres on port 5433..." @@ -60,3 +110,34 @@ wait_for_port 127.0.0.1 5433 echo "[setup] Waiting for postgres to accept queries..." wait_for_postgres_ready echo "[setup] Postgres is ready." + +# Azurite backs the `azurite` storage entry, which `backend_consistency` +# audits via `?storage=azurite`. Only a port wait: the entry is declared +# but never activated, so nothing in the suite touches it until a test +# names it, and by then the listener has had the whole server boot to +# settle. A failure here should not take down a run that mostly does not +# use it — so this warns rather than exits, and the Azure test fails on +# its own terms with a clearer message than "setup timed out". +# +# Its own loop rather than `wait_for_port`: that helper calls `exit 1` on +# timeout, and `exit` inside a function ends the script whatever context +# it was called from — so wrapping it in an `if` would not degrade, it +# would just fail later and less clearly. +echo "[setup] Waiting for azurite on port 10000..." +azurite_deadline=$(( $(date +%s) + 30 )) +until nc -z 127.0.0.1 10000 2>/dev/null; do + if [[ $(date +%s) -ge $azurite_deadline ]]; then + echo "[setup] WARNING: azurite did not come up — Azure-backed tests will fail." >&2 + break + fi + sleep 0.5 +done +if nc -z 127.0.0.1 10000 2>/dev/null; then + # Port open only means the listener is up — NOT that the tests can use + # it. Saying "ready" here was misleading: the container was still + # missing, so the first Azure job failed with a 404 while setup had + # already reported success. `create_azurite_container` is what makes + # the claim true, so it is the one that gets to announce it. + echo "[setup] Azurite listening; provisioning container..." + create_azurite_container +fi