test(consistency): exercise the Azure backend against Azurite

Adds an Azurite service and a scenario that audits the Azure backend
through `?storage=azurite`. It is the only coverage of that code path in
the tree: `AzureBlobBackend` has unit tests for its name parser and
ordering, but nothing else speaks the protocol, and a paid account is
not an option for CI. Azurite implements the real Blob REST API, so this
exercises SharedKey signing, prefix/marker paging, and the 256-way shard
walk with its termination.

## Harness

`docker-compose.test.yml` gains an azurite service on 10000 (tmpfs, so
it dies with the stack). `spawn-db.sh` provisions the container itself,
because `AzureBlobBackend::initialize` verifies rather than creates —
signed by hand with curl + openssl rather than pulling a ~700 MB `az`
image for one PUT. Two traps are commented there: the account key is
base64 but HMAC wants raw bytes, and the canonicalized resource repeats
the account name (`/{acc}/{acc}/{container}`) because the emulator puts
in the path what real Azure puts in the host. Getting that wrong yields
403, not a hint.

The `azurite` entry is declared in `server.env` but never activated, so
the suite's active backend stays local and only this file reaches Azure.

## What it asserts, and what it cannot

A failure surfaces as `ok: false`, because an enumeration error now
fails the run rather than degrading to a per-row probe.

It deliberately asserts no finding count. The container starts empty and
the job's grace window is an hour, so a freshly-uploaded blob is skipped
in both directions by design — an audit here can only report zero, and
"zero findings" would pass whether enumeration worked or returned
nothing. The one positive assert, `scanned_count != 0`, therefore sits
on the local control, which does hold blobs; `scanned_count` accumulates
via `checkpoint`, which the empty-page early return skips.

## No cutover, deliberately

Putting real bytes in the container means `backend_migration
?storage=azurite`, which hangs on the first blob: `head_check` issues a
~40-byte ranged GET, `azure_core` 0.21 attaches
`x-ms-range-get-content-crc64` to anything under 4 MiB, Azurite 500s,
and the deterministic error is retried forever while
`migration_readonly` refuses writes app-wide. The full chain and the
rejected workaround are in the file header. The scenario is still
ordered last in `run.sh` — it is the only one needing a second service,
and the cutover comes back there once the official SDK lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tests/api/backend_consistency_azure.hurl

@@ -0,0 +1,261 @@
+# =============================================================
+# OxiCloud – backend_consistency against the Azure backend
+#
+# Runs against **Azurite**, the Azure Blob emulator started by
+# `tests/common/spawn-db.sh`. It speaks the real Blob REST API, so this
+# is the only way to exercise the Azure path without an account. The
+# `azurite` storage entry is declared in `server.env` but never
+# activated — the suite's active backend stays local, and this file
+# reaches Azure explicitly through `?storage=azurite`.
+#
+# ## What it pins
+#
+# That Azure enumeration works against a real Blob REST implementation —
+# SharedKey signing, prefix/marker paging, the 256-way shard walk and its
+# termination. `AzureBlobBackend::list_blob_hashes` has unit tests for
+# its name parser and ordering, but nothing else in the tree speaks the
+# protocol.
+#
+# A failure surfaces as `ok: false`, because an enumeration error now
+# fails the run. It used to degrade to a per-row probe — walk
+# `storage.blobs`, ask "are these bytes there" — which found only the
+# DB→backend direction and left orphans undetectable, since bytes no row
+# claims are invisible to anything starting from the database. That
+# fallback is gone; the module docs on `backend_consistency_service.rs`
+# say why.
+#
+# ## What it deliberately does NOT assert
+#
+# Any finding count. The Azurite container starts empty and the run's
+# grace window is one hour, so a freshly-uploaded blob is skipped in
+# both directions by design — an audit here can only ever report zero.
+# Asserting "zero findings" would pass whether enumeration worked or
+# returned nothing at all.
+#
+# Real orphan/missing coverage needs blobs on the backend older than the
+# grace window, which needs either a cutover into Azurite or a way to
+# backdate `last_modified`. See "Why there is no cutover here" below.
+#
+# Prerequisites: setup.hurl must have run (admin user exists), and
+# Azurite must be listening on 10000.
+#
+# Run:
+#   hurl --variables-file tests/api/test.env --file-root tests \
+#        --test tests/api/backend_consistency_azure.hurl
+# =============================================================
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 1 – Login
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/auth/login
+Content-Type: application/json
+{
+  "username": "{{username}}",
+  "password": "{{password}}"
+}
+
+HTTP 200
+[Captures]
+token: jsonpath "$.access_token"
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 2 – Upload a file so the registry is not empty.
+#
+# The bytes land on the ACTIVE (local) backend and never reach Azurite,
+# so this does not feed step 3. It feeds step 4: a local sweep over an
+# empty `storage.blobs` would satisfy every assertion there while
+# comparing nothing, and the control is only a control if it had
+# something to compare.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/folders
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+  "name": "hurl-azure-consistency"
+}
+
+HTTP 201
+[Captures]
+folder_id: jsonpath "$.id"
+
+
+POST {{base_url}}/api/files/upload
+Authorization: Bearer {{token}}
+[MultipartFormData]
+folder_id: {{folder_id}}
+file: file,fixtures/red-image.png; image/png
+
+HTTP 201
+[Captures]
+file_id: jsonpath "$.id"
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 2b – Pre-flight: does the Azure backend work at all?
+#
+# Cheap, synchronous, and it isolates the failure. It separates "Azurite
+# is missing, wedged, or misconfigured" from "enumeration is broken",
+# which step 3 alone cannot: a dead container and a broken
+# `list_blob_hashes` both surface there as the per-row fallback.
+#
+# `entry_name` resolves against OXICLOUD_STORAGE_ENTRIES, so this
+# exercises the same entry step 3 audits rather than an ad-hoc config.
+# It does a health check AND a write/read round-trip, and
+# `phase_reached` names how far it got, so a failure points at a step
+# rather than at the suite.
+#
+# Commonest cause of a failure here: the container does not exist.
+# `AzureBlobBackend::initialize` verifies rather than creates, so
+# `spawn-db.sh` provisions it with a hand-signed PUT.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/settings/storage/test
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+  "entry_name": "azurite"
+}
+
+HTTP 200
+[Asserts]
+jsonpath "$.connected" == true
+jsonpath "$.roundtrip_passed" == true
+jsonpath "$.bytes_written" != 0
+jsonpath "$.bytes_read" != 0
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 3 – Audit the Azure entry.
+#
+# `?storage=azurite` builds a backend for that named entry directly,
+# bypassing the active-backend pointer — which is also the answer for
+# auditing either side mid-migration.
+#
+# ## Why there is no cutover here — DECIDED, do not retry casually
+#
+# An earlier version ran `backend_migration ?storage=azurite` first, to
+# put real bytes in the container. It hangs, on the first blob, and the
+# cause is in `azure_core` 0.21 rather than in anything OxiCloud does.
+#
+# The chain, all verified in source:
+#
+#   backend_migration_service.rs      target.head_check(hash)
+#     → EncryptedBlobBackend::head_check
+#     → get_blob_range_stream(hash, 0, HEADER_SIZE)   ~40 bytes
+#     → azure_core Range::as_headers  adds x-ms-range-get-content-crc64
+#                                     to ANY range under 4 MiB
+#     → Azurite answers 500
+#     → azure_core classifies 500 retryable, response is deterministic,
+#       so it retries forever — while migration_readonly refuses writes
+#       application-wide.
+#
+# `head_check` is a pre-write format probe on the TARGET, so it fires
+# before the first byte is copied. Nothing about the migration job is
+# wrong; it works against real S3.
+#
+# **A workaround exists and was rejected** (2026-09-02): issue an
+# unranged `get()` for small requests — its 16 MiB initial range clears
+# the 4 MiB threshold, so the header is never sent — and truncate
+# client-side. Correct against real Azure, but it pays for an emulator
+# with production cost (a 40-byte probe becomes a whole-blob transfer)
+# and puts new offset arithmetic on the read path, where a mistake
+# serves wrong bytes silently instead of failing. See the note on
+# `AzureBlobBackend::get_blob_range_stream`.
+#
+# So the cutover comes back with the official `azure_storage_blob` 1.x,
+# where `range_get_content_crc64` is an explicit field to leave unset —
+# and with it the orphan/missing assertions this file cannot make today.
+# `docs/plan/jobs-handling-recoverable-error.md` covers the other half:
+# the run should have paused with a reason instead of hanging, whatever
+# the SDK does.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=azurite
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+
+# THIS is the assertion that matters, and it is a real one. Enumeration
+# failure is no longer degraded into a completed run — any Err from
+# `list_blob_hashes` now fails the run — so a broken Azure enumeration
+# surfaces right here as `ok: false`, whatever went wrong: signing,
+# paging, the shard walk, the cursor.
+#
+# Before the fallback was deleted this needed a proxy assert on
+# `extra_stats.mode`, because a broken enumeration completed "cleanly"
+# with half its coverage silently gone.
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+
+# No unexpected notices. `backend_unenumerable` is gone with the
+# fallback, so what this now guards against is a stray
+# `unknown_backend_file` — a non-canonical name in the blob namespace,
+# which on a container we provision ourselves means something wrote
+# where it should not have.
+jsonpath "$.outcome.extra.severity_counts.anomaly" not exists
+
+# ── Known weakness, stated rather than hidden ────────────────────────
+#
+# A run that enumerates SUCCESSFULLY but returns nothing still passes.
+# `scanned_count` would catch that, and it is deliberately not asserted
+# here: the container is empty, so the job early-returns before its
+# first checkpoint and 0 is the correct answer. Step 4 carries that
+# assert instead, on the one entry that does hold blobs.
+#
+# Making it positive HERE needs the corpus on Azure, which needs the
+# cutover — see the header. Planting blobs by hand is not a substitute:
+# the DB side would then be walked against a backend that does not hold
+# the corpus, and every local blob would report `blob_missing_from_backend`.
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 4 – The same job against the LOCAL entry behaves identically.
+#
+# Azure and local now take the same code path, so this is no longer a
+# contrast — it is the control. If a future change re-degrades Azure,
+# this passing while step 3 fails localises the break to the Azure
+# backend rather than to the job.
+#
+# It also carries the one POSITIVE assert this file can make. Local
+# holds the blob step 2 uploaded, so the sweep gets past the empty-page
+# early return and checkpoints — which is the only thing here that
+# distinguishes "the merge-join compared something" from "the merge-join
+# was handed nothing and completed".
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=local_main
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+
+# `scanned_count` accumulates via `checkpoint`, which the early return
+# skips — so a non-zero value means blobs were enumerated AND paired
+# against `storage.blobs`, not merely that the run ended cleanly.
+jsonpath "$.outcome.extra.scanned_count" != 0
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 5 – Teardown. Hurl files share one database, so a folder left
+# behind changes what later scenarios see.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/folders/{{folder_id}}
+Authorization: Bearer {{token}}
+
+HTTP 204
+
+
+GET {{base_url}}/api/trash/resources
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Captures]
+trash_id: jsonpath "$.items[?(@.resource.id == '{{folder_id}}')].resource.id"
+
+
+DELETE {{base_url}}/api/trash/{{trash_id}}
+Authorization: Bearer {{token}}
+
+HTTP 200

tests/api/run.sh

@@ -225,7 +225,15 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
   "$API_DIR/webdav_permissions.hurl" \
   "$API_DIR/webdav_nested_move_cascade.hurl" \
   "$API_DIR/wopi_authz.hurl" \
-  "$API_DIR/wopi_shared_drive.hurl"
+  "$API_DIR/wopi_shared_drive.hurl" \
+  `# LAST, deliberately — and kept last even though it no longer cuts` \
+  `# the storage pointer over. It is the only scenario that depends on a` \
+  `# second service (Azurite on 10000), so if that container is missing` \
+  `# or wedged the failure lands after everything else has reported,` \
+  `# rather than in the middle of an otherwise-green run. It is also` \
+  `# where a cutover comes back once the official Azure SDK lands (see` \
+  `# the file header), and that WILL need to be last.` \
+  "$API_DIR/backend_consistency_azure.hurl"
 
 #bash "$API_DIR/dedup_bulk_upload.sh"
 

tests/api/storage_multi_entry.hurl

@@ -44,19 +44,22 @@ Authorization: Bearer {{admin_token}}
 HTTP 200
 [Asserts]
 # Two entries declared, in _ENTRIES order.
-jsonpath "$.entries" count == 2
+jsonpath "$.entries" count == 3
 jsonpath "$.entries[0].name" == "local_main"
 jsonpath "$.entries[1].name" == "s3_stub"
+jsonpath "$.entries[2].name" == "azurite"
 
 # Backend types match the declarations.
 jsonpath "$.entries[0].backend" == "local"
 jsonpath "$.entries[1].backend" == "s3"
+jsonpath "$.entries[2].backend" == "azure"
 
 # Active pointer: fresh DB has no `active_backend_name` row, so the
 # boot fallback picks the FIRST entry in _ENTRIES.
 jsonpath "$.active_entry_name" == "local_main"
 jsonpath "$.entries[0].is_active" == true
 jsonpath "$.entries[1].is_active" == false
+jsonpath "$.entries[2].is_active" == false
 
 # Read-only mode off on a fresh boot (no in-flight migration, no
 # stale flag in DB).

tests/common/docker-compose.test.yml

@@ -14,3 +14,32 @@ services:
       interval: 2s
       timeout: 5s
       retries: 10
+
+  # Azure Blob Storage emulator. Speaks the real Blob REST API, so it is
+  # the only way to exercise the Azure backend without an account — and
+  # the account it does ship is the fixed, publicly-documented dev pair
+  # (`devstoreaccount1` + the well-known key), so nothing here is a
+  # credential worth protecting.
+  #
+  # Blob service only: `--blobHost` with no queue/table, since the backend
+  # uses blobs exclusively and the other two would just be open ports.
+  #
+  # tmpfs for the same reason postgres uses it — a run must start from an
+  # empty container, or a previous run's blobs read as orphans in the very
+  # audit this exists to test.
+  azurite-test:
+    image: mcr.microsoft.com/azure-storage/azurite:3.36.0
+    command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data --skipApiVersionCheck
+    ports:
+      - "10000:10000"
+    tmpfs:
+      - /data
+    healthcheck:
+      # Azurite answers 400 to an unauthenticated root GET, which is proof
+      # enough that the HTTP listener is up — the SDK handshake is what
+      # validates credentials, and that happens later in the backend's
+      # own `initialize`.
+      test: ["CMD-SHELL", "nc -z 127.0.0.1 10000 || exit 1"]
+      interval: 2s
+      timeout: 5s
+      retries: 15

tests/common/server.env

@@ -42,7 +42,7 @@ OXICLOUD_NEXTCLOUD_ENABLED=true
 # `s3_stub` is declared but never activated — it lets storage_config.hurl
 # assert the entries table has more than one row without needing a
 # real S3 backend.
-OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub
+OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite
 OXICLOUD_STORAGE_local_main_BACKEND=local
 OXICLOUD_STORAGE_s3_stub_BACKEND=s3
 OXICLOUD_STORAGE_s3_stub_S3_BUCKET=oxicloud-test-stub
@@ -51,6 +51,24 @@ OXICLOUD_STORAGE_s3_stub_S3_ENDPOINT_URL=http://127.0.0.1:9999
 OXICLOUD_STORAGE_s3_stub_S3_ACCESS_KEY=stub
 OXICLOUD_STORAGE_s3_stub_S3_SECRET_KEY=stub
 
+# `azurite` — a REAL, reachable Azure backend, unlike `s3_stub` above.
+# It points at the Azurite emulator started by spawn-db.sh, which speaks
+# the actual Blob REST API, so this is the only way to exercise the Azure
+# path without an account.
+#
+# Declared but NOT activated: the active backend stays local_main, so the
+# rest of the suite is untouched. Tests reach it explicitly through
+# `?storage=azurite` on the jobs that accept an entry name — which is
+# exactly what that parameter was added for.
+#
+# The credentials are Azurite's fixed, publicly-documented development
+# pair. They are in every Microsoft quickstart; nothing here is secret.
+OXICLOUD_STORAGE_azurite_BACKEND=azure
+OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_NAME=devstoreaccount1
+OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
+OXICLOUD_STORAGE_azurite_AZURE_CONTAINER=oxicloud-test
+OXICLOUD_STORAGE_azurite_AZURE_ENDPOINT_URL=http://127.0.0.1:10000/devstoreaccount1
+
 RUST_LOG="warn,audit=info,sqlx::migrate=info"
 #RUST_LOG="warn,audit=info,oxicloud::quota=debug"
 #RUST_LOG=debug

tests/common/spawn-db.sh

@@ -52,7 +52,57 @@ wait_for_postgres_ready() {
   exit 1
 }
 
-echo "[setup] Starting test postgres..."
+# Create the blob container Azurite serves.
+#
+# `AzureBlobBackend::initialize` VERIFIES the container exists and fails
+# with a 404 if it does not — it deliberately does not create one, since
+# auto-creating would turn a typo'd container name into a silently
+# working empty container. So the harness provisions it, exactly as an
+# operator would in production.
+#
+# Signed by hand rather than shelling out to the Azure CLI: `az` would
+# mean pulling a ~700 MB image into every CI run to issue one PUT. This
+# needs only curl and openssl, both already required.
+#
+# Two things make this fiddly enough to be worth commenting:
+#   * The key is base64 and HMAC needs raw bytes, so it is decoded and
+#     re-encoded as hex for `-macopt hexkey:`.
+#   * The canonicalized resource repeats the account name —
+#     `/{account}/{account}/{container}` — because the emulator puts the
+#     account in the URL path where real Azure puts it in the host. This
+#     is the classic Azurite signing trap; getting it wrong yields 403,
+#     not a hint.
+#
+# Idempotent by result: a second run gets 409 ContainerAlreadyExists,
+# which is success for our purposes.
+create_azurite_container() {
+  local acc=devstoreaccount1
+  local cont=oxicloud-test
+  # Azurite's fixed development key — hardcoded in the image, published
+  # by Microsoft, and valid against nothing but a local emulator.
+  local key='Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=='
+  local ver=2021-08-06
+  local date_hdr hexkey sts sig code
+  date_hdr=$(LC_ALL=C TZ=GMT date '+%a, %d %b %Y %H:%M:%S GMT')
+  hexkey=$(printf '%s' "$key" | base64 -d | od -An -tx1 | tr -d ' \n')
+  # Twelve leading empty lines are the unused standard headers
+  # (Content-*, Date, If-*, Range) the SharedKey scheme requires in
+  # fixed positions.
+  sts="PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:${date_hdr}\nx-ms-version:${ver}\n/${acc}/${acc}/${cont}\nrestype:container"
+  sig=$(printf '%b' "$sts" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${hexkey}" -binary | base64)
+  code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \
+    -H "x-ms-date: ${date_hdr}" \
+    -H "x-ms-version: ${ver}" \
+    -H "Authorization: SharedKey ${acc}:${sig}" \
+    "http://127.0.0.1:10000/${acc}/${cont}?restype=container" 2>/dev/null)
+  case "$code" in
+    201) echo "[setup] Azurite container '${cont}' created." ;;
+    409) echo "[setup] Azurite container '${cont}' already exists." ;;
+    *)   echo "[setup] WARNING: could not create Azurite container '${cont}' (HTTP ${code}) — Azure-backed tests will fail." >&2 ;;
+  esac
+}
+
+echo "[setup] Starting test postgres + azurite..."
 docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
 docker compose -f "$COMPOSE_FILE" up -d
 echo "[setup] Waiting for postgres on port 5433..."
@@ -60,3 +110,34 @@ wait_for_port 127.0.0.1 5433
 echo "[setup] Waiting for postgres to accept queries..."
 wait_for_postgres_ready
 echo "[setup] Postgres is ready."
+
+# Azurite backs the `azurite` storage entry, which `backend_consistency`
+# audits via `?storage=azurite`. Only a port wait: the entry is declared
+# but never activated, so nothing in the suite touches it until a test
+# names it, and by then the listener has had the whole server boot to
+# settle. A failure here should not take down a run that mostly does not
+# use it — so this warns rather than exits, and the Azure test fails on
+# its own terms with a clearer message than "setup timed out".
+#
+# Its own loop rather than `wait_for_port`: that helper calls `exit 1` on
+# timeout, and `exit` inside a function ends the script whatever context
+# it was called from — so wrapping it in an `if` would not degrade, it
+# would just fail later and less clearly.
+echo "[setup] Waiting for azurite on port 10000..."
+azurite_deadline=$(( $(date +%s) + 30 ))
+until nc -z 127.0.0.1 10000 2>/dev/null; do
+  if [[ $(date +%s) -ge $azurite_deadline ]]; then
+    echo "[setup] WARNING: azurite did not come up — Azure-backed tests will fail." >&2
+    break
+  fi
+  sleep 0.5
+done
+if nc -z 127.0.0.1 10000 2>/dev/null; then
+  # Port open only means the listener is up — NOT that the tests can use
+  # it. Saying "ready" here was misleading: the container was still
+  # missing, so the first Azure job failed with a 404 while setup had
+  # already reported success. `create_azurite_container` is what makes
+  # the claim true, so it is the one that gets to announce it.
+  echo "[setup] Azurite listening; provisioning container..."
+  create_azurite_container
+fi