refactor(consistency): blobs_consistency reads only the database

`blobs_consistency` probed `blob_exists` once per row and, under
`?deep=true`, read and re-hashed every blob. `backend_consistency`
already reports the same `blob_missing_from_backend` from its
merge-join — so the probe was duplicated work that found strictly less
(a DB walk cannot see backend-only orphans by construction) at N round
-trips instead of one enumeration. Every scheduled sweep paid for it.

All three physical checks move to `backend_consistency`:

* `blob_missing_from_backend` was already there; the duplicate is gone.
* `blob_corrupted` / `blob_unreadable` hook the matched arm of the
  merge-join, which holds exactly the key pairs worth reading. Guarded
  by `in_range` so a pair past the horizon is not read twice, and
  `params.deep` is persisted on a fresh run and read back on resume so
  a paused deep scan does not silently continue shallow.

Deep mode belongs there because it is backend work end to end: the
only DB input is the hash. Keeping it in `blobs_consistency` forced
that tenant to carry a backend for one flag.

What remains is the half that needs no backend: `refcount_mismatch`
and its repair. The constructor drops from five parameters to two —
no backend, no storage_entries, no storage_path_fallback — and
`?storage=<name>` / `?deep=true` are now inert there, which the
job description says outright.

`affected_files` is needed by both tenants, so it moves to a shared
`blob_diagnostics` module rather than being copied.
`PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined
in `blobs_consistency` and re-exported, which is backwards once the
DB-only tenant has no entry to scope. The create-grace window goes
with the probe — it existed to avoid flagging a blob whose bytes had
landed before its row, and the refcount comparison reads one
consistent snapshot.

Known cost: `backend_consistency` returns `backend_unenumerable` on
Azure and mid-migration, so on those configs missing bytes now go
unreported where the per-row probe caught them. That argues for the
Azure enumeration impl, not for keeping the probe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-08-29 01:44:08 +02:00
parent 1ea3826660
commit f1f327a6c4
11 changed files with 361 additions and 474 deletions
+2 -2
View File
@@ -949,8 +949,8 @@ Findings each job reports today, and where the new tables land:
| # | Edge | Direction | Mechanism | Status |
|---|---|---|---|---|
| 1 | backend → `storage.blobs` | orphan bytes | `orphan_blob` (backend_consistency) | ✓ |
| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` | ✓ |
| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` | ✓ |
| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` (backend_consistency) | ✓ |
| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` (backend_consistency, `?deep=true`) | ✓ |
| 4 | manifest → chunks | chunk reaped | `chunk_missing` (files_consistency) | ✓ |
| 5 | `files` → Blob | dangling | `missing_blob` (files_consistency) | ✓ |
| 6 | `storage.blobs.ref_count` | recompute | `refcount_mismatch` | ✓ chunk level only |
+11 -1
View File
@@ -179,7 +179,17 @@ Guardrail:
head pair's fingerprint differs from the second entry's — signals "you
added new keys but haven't rotated legacy blobs yet".
* A **legacy-blob counter** is surfaced in the admin panel per storage entry.
The counter is maintained by `blobs_consistency`: during its normal walk it
> **Retarget (post-split).** This plan names `blobs_consistency` as the
> host for the magic-byte branch throughout. That is no longer the right
> tenant: `blobs_consistency` became database-only and opens no backend,
> while `backend_consistency` owns every physical check and already holds
> the enumeration. Read `backend_consistency` wherever the sections below
> say `blobs_consistency`. Nothing else about the design changes — the
> magic-byte check still rides along on an existing walk, and still lands
> in the run's `stats` bag.
The counter is maintained by that scan: during its normal walk it
branches on the magic-byte check and records the legacy count as a run
statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The
admin panel reads the most recent count and displays it. Refresh cadence is
+4 -2
View File
@@ -407,10 +407,12 @@ Per slice, plus these end-to-end scenarios in Hurl:
6. **In-place encryption rotation refused**: two entries, same S3 bucket,
different encryption keys. Trigger migration → refuses with the specific
error message pointing at the encryption case and the two-step workaround.
7. **`?storage=<name>` on blobs_consistency**: run against `s3_prod` before
7. **`?storage=<name>` on backend_consistency**: run against `s3_prod` before
cutover. Full walk, `probed_storage` in run row. Then cutover, then rerun
against `local_main` — verifies old backend still has everything.
8. **Unknown storage name**: `POST /admin/jobs/blobs_consistency/trigger?storage=nope`
(This scenario named `blobs_consistency` until that tenant became
database-only; entry scoping belongs to whichever job opens a backend.)
8. **Unknown storage name**: `POST /admin/jobs/backend_consistency/trigger?storage=nope`
→ 400 with known-names list. No run row created.
9. **Missing entry at boot**: `active_backend_name = "gone"` but `_ENTRIES`
doesn't include it → boot aborts with the specific message pointing at