refactor(consistency): blobs_consistency reads only the database
`blobs_consistency` probed `blob_exists` once per row and, under `?deep=true`, read and re-hashed every blob. `backend_consistency` already reports the same `blob_missing_from_backend` from its merge-join — so the probe was duplicated work that found strictly less (a DB walk cannot see backend-only orphans by construction) at N round -trips instead of one enumeration. Every scheduled sweep paid for it. All three physical checks move to `backend_consistency`: * `blob_missing_from_backend` was already there; the duplicate is gone. * `blob_corrupted` / `blob_unreadable` hook the matched arm of the merge-join, which holds exactly the key pairs worth reading. Guarded by `in_range` so a pair past the horizon is not read twice, and `params.deep` is persisted on a fresh run and read back on resume so a paused deep scan does not silently continue shallow. Deep mode belongs there because it is backend work end to end: the only DB input is the hash. Keeping it in `blobs_consistency` forced that tenant to carry a backend for one flag. What remains is the half that needs no backend: `refcount_mismatch` and its repair. The constructor drops from five parameters to two — no backend, no storage_entries, no storage_path_fallback — and `?storage=<name>` / `?deep=true` are now inert there, which the job description says outright. `affected_files` is needed by both tenants, so it moves to a shared `blob_diagnostics` module rather than being copied. `PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined in `blobs_consistency` and re-exported, which is backwards once the DB-only tenant has no entry to scope. The create-grace window goes with the probe — it existed to avoid flagging a blob whose bytes had landed before its row, and the refcount comparison reads one consistent snapshot. Known cost: `backend_consistency` returns `backend_unenumerable` on Azure and mid-migration, so on those configs missing bytes now go unreported where the per-row probe caught them. That argues for the Azure enumeration impl, not for keeping the probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -234,27 +234,38 @@ can be safely dropped.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Blob consistency (`blobs_consistency`)
|
## 4. Blob consistency — two jobs, split by what they read
|
||||||
|
|
||||||
Read-only recoverable job that walks `storage.blobs` and reports
|
The registry side and the physical side are separate tenants. They
|
||||||
divergence between the DB registry and the physical backend.
|
used to be one, with `blobs_consistency` probing the backend once per
|
||||||
|
row; that probe found strictly less than the merge-join below, at N
|
||||||
|
round-trips instead of one enumeration, so it was removed.
|
||||||
|
|
||||||
### Shallow mode (default)
|
### `blobs_consistency` — database only
|
||||||
|
|
||||||
Per row:
|
Walks `storage.blobs` and compares `ref_count` against the reference
|
||||||
|
count computed from `storage.files.blob_hash` +
|
||||||
|
`chunk_manifests.chunk_hashes[]`. On mismatch: `refcount_mismatch`
|
||||||
|
(severity `inconsistent`), repairable under `?repair=true`.
|
||||||
|
|
||||||
- `blob_exists(hash)` on the active backend → if false, record
|
It opens no backend and makes no network call. `?storage=<name>` and
|
||||||
`blob_missing_from_backend` (severity `data_loss`)
|
`?deep=true` are inert. Cost is one aggregate SQL per row.
|
||||||
- Compare `ref_count` against the actual reference count computed
|
|
||||||
from `SUM` over `storage.files.blob_hash` + `chunk_manifests.chunk_hashes[]`
|
|
||||||
→ if mismatch, record `refcount_mismatch` (severity `inconsistent`)
|
|
||||||
|
|
||||||
Cost: one existence probe + one aggregate SQL per row. Fast on
|
### `backend_consistency` — everything physical
|
||||||
S3/Azure (single HEAD).
|
|
||||||
|
|
||||||
### Deep mode (`?deep=true`)
|
Merge-joins the backend's enumeration against `storage.blobs`, both
|
||||||
|
ordered by hash, yielding both deltas in one pass:
|
||||||
|
|
||||||
Adds a full read of every blob:
|
- bytes with no registry row → `orphan_blob` (severity `inconsistent`)
|
||||||
|
- a registry row with no bytes → `blob_missing_from_backend`
|
||||||
|
(severity `data_loss`)
|
||||||
|
|
||||||
|
`?storage=<name>` scopes it to any declared entry rather than the live
|
||||||
|
backend.
|
||||||
|
|
||||||
|
### Deep mode (`?deep=true`, on `backend_consistency`)
|
||||||
|
|
||||||
|
For every hash present on both sides, adds a full read:
|
||||||
|
|
||||||
- Stream the blob through `EncryptedBlobBackend::get_blob_stream`
|
- Stream the blob through `EncryptedBlobBackend::get_blob_stream`
|
||||||
(strips header, decrypts if needed, applies BLAKE3 rescue for
|
(strips header, decrypts if needed, applies BLAKE3 rescue for
|
||||||
|
|||||||
@@ -100,12 +100,16 @@ This one-shot repair command re-runs the same env-parse the server does at boot,
|
|||||||
|
|
||||||
### Auditing entries other than the active one
|
### Auditing entries other than the active one
|
||||||
|
|
||||||
`blobs_consistency` and `backend_consistency` (recoverable jobs on the Jobs tab) accept `?storage=<name>` to probe any declared entry — not just the live one. Use this to verify a migration target before cutover, or to audit an old backend after cutover but before decommissioning:
|
`backend_consistency` (a recoverable job on the Jobs tab) accepts `?storage=<name>` to audit any declared entry — not just the live one. Use this to verify a migration target before cutover, or to audit an old backend after cutover but before decommissioning:
|
||||||
|
|
||||||
```
|
```
|
||||||
POST /api/admin/jobs/blobs_consistency/trigger?storage=<name>
|
POST /api/admin/jobs/backend_consistency/trigger?storage=<name>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Add `?deep=true` to also read every blob back and re-hash it, which catches silent bit-rot. That is a full read of the entry and can take hours.
|
||||||
|
|
||||||
|
`blobs_consistency` does *not* accept `?storage=<name>`: it only reads the database, so there is no entry for it to scope.
|
||||||
|
|
||||||
Unknown names 400 at the HTTP layer.
|
Unknown names 400 at the HTTP layer.
|
||||||
|
|
||||||
## Data Storage
|
## Data Storage
|
||||||
|
|||||||
@@ -949,8 +949,8 @@ Findings each job reports today, and where the new tables land:
|
|||||||
| # | Edge | Direction | Mechanism | Status |
|
| # | Edge | Direction | Mechanism | Status |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 1 | backend → `storage.blobs` | orphan bytes | `orphan_blob` (backend_consistency) | ✓ |
|
| 1 | backend → `storage.blobs` | orphan bytes | `orphan_blob` (backend_consistency) | ✓ |
|
||||||
| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` | ✓ |
|
| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` (backend_consistency) | ✓ |
|
||||||
| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` | ✓ |
|
| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` (backend_consistency, `?deep=true`) | ✓ |
|
||||||
| 4 | manifest → chunks | chunk reaped | `chunk_missing` (files_consistency) | ✓ |
|
| 4 | manifest → chunks | chunk reaped | `chunk_missing` (files_consistency) | ✓ |
|
||||||
| 5 | `files` → Blob | dangling | `missing_blob` (files_consistency) | ✓ |
|
| 5 | `files` → Blob | dangling | `missing_blob` (files_consistency) | ✓ |
|
||||||
| 6 | `storage.blobs.ref_count` | recompute | `refcount_mismatch` | ✓ chunk level only |
|
| 6 | `storage.blobs.ref_count` | recompute | `refcount_mismatch` | ✓ chunk level only |
|
||||||
|
|||||||
@@ -179,7 +179,17 @@ Guardrail:
|
|||||||
head pair's fingerprint differs from the second entry's — signals "you
|
head pair's fingerprint differs from the second entry's — signals "you
|
||||||
added new keys but haven't rotated legacy blobs yet".
|
added new keys but haven't rotated legacy blobs yet".
|
||||||
* A **legacy-blob counter** is surfaced in the admin panel per storage entry.
|
* A **legacy-blob counter** is surfaced in the admin panel per storage entry.
|
||||||
The counter is maintained by `blobs_consistency`: during its normal walk it
|
|
||||||
|
> **Retarget (post-split).** This plan names `blobs_consistency` as the
|
||||||
|
> host for the magic-byte branch throughout. That is no longer the right
|
||||||
|
> tenant: `blobs_consistency` became database-only and opens no backend,
|
||||||
|
> while `backend_consistency` owns every physical check and already holds
|
||||||
|
> the enumeration. Read `backend_consistency` wherever the sections below
|
||||||
|
> say `blobs_consistency`. Nothing else about the design changes — the
|
||||||
|
> magic-byte check still rides along on an existing walk, and still lands
|
||||||
|
> in the run's `stats` bag.
|
||||||
|
|
||||||
|
The counter is maintained by that scan: during its normal walk it
|
||||||
branches on the magic-byte check and records the legacy count as a run
|
branches on the magic-byte check and records the legacy count as a run
|
||||||
statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The
|
statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The
|
||||||
admin panel reads the most recent count and displays it. Refresh cadence is
|
admin panel reads the most recent count and displays it. Refresh cadence is
|
||||||
|
|||||||
@@ -407,10 +407,12 @@ Per slice, plus these end-to-end scenarios in Hurl:
|
|||||||
6. **In-place encryption rotation refused**: two entries, same S3 bucket,
|
6. **In-place encryption rotation refused**: two entries, same S3 bucket,
|
||||||
different encryption keys. Trigger migration → refuses with the specific
|
different encryption keys. Trigger migration → refuses with the specific
|
||||||
error message pointing at the encryption case and the two-step workaround.
|
error message pointing at the encryption case and the two-step workaround.
|
||||||
7. **`?storage=<name>` on blobs_consistency**: run against `s3_prod` before
|
7. **`?storage=<name>` on backend_consistency**: run against `s3_prod` before
|
||||||
cutover. Full walk, `probed_storage` in run row. Then cutover, then rerun
|
cutover. Full walk, `probed_storage` in run row. Then cutover, then rerun
|
||||||
against `local_main` — verifies old backend still has everything.
|
against `local_main` — verifies old backend still has everything.
|
||||||
8. **Unknown storage name**: `POST /admin/jobs/blobs_consistency/trigger?storage=nope`
|
(This scenario named `blobs_consistency` until that tenant became
|
||||||
|
database-only; entry scoping belongs to whichever job opens a backend.)
|
||||||
|
8. **Unknown storage name**: `POST /admin/jobs/backend_consistency/trigger?storage=nope`
|
||||||
→ 400 with known-names list. No run row created.
|
→ 400 with known-names list. No run row created.
|
||||||
9. **Missing entry at boot**: `active_backend_name = "gone"` but `_ENTRIES`
|
9. **Missing entry at boot**: `active_backend_name = "gone"` but `_ENTRIES`
|
||||||
doesn't include it → boot aborts with the specific message pointing at
|
doesn't include it → boot aborts with the specific message pointing at
|
||||||
|
|||||||
@@ -637,13 +637,14 @@
|
|||||||
// Jobs that respect `?deep=true`:
|
// Jobs that respect `?deep=true`:
|
||||||
// * `consistency_batch` — propagates deep to every child that
|
// * `consistency_batch` — propagates deep to every child that
|
||||||
// understands it
|
// understands it
|
||||||
// * `blobs_consistency` — deep mode re-reads + re-hashes every
|
// * `backend_consistency` — deep mode re-reads + re-hashes every
|
||||||
// blob for silent bit-rot detection (severity `data_loss`).
|
// matched blob for silent bit-rot detection (severity
|
||||||
// Full read of storage; can take hours on big installs — the
|
// `data_loss`). Full read of storage; can take hours on big
|
||||||
// "Run" (normal) button on the same row does the cheap
|
// installs — the "Run" button on the same row does the
|
||||||
// existence probes only.
|
// enumeration merge-join only. This was `blobs_consistency`
|
||||||
|
// until that tenant became database-only.
|
||||||
function supportsDeep(name: string): boolean {
|
function supportsDeep(name: string): boolean {
|
||||||
return name === 'consistency_batch' || name === 'blobs_consistency';
|
return name === 'consistency_batch' || name === 'backend_consistency';
|
||||||
}
|
}
|
||||||
|
|
||||||
// Whether `?repair=true` does anything for this job — declared by the
|
// Whether `?repair=true` does anything for this job — declared by the
|
||||||
|
|||||||
+12
-15
@@ -1536,24 +1536,21 @@ impl AppServiceFactory {
|
|||||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||||
.await;
|
.await;
|
||||||
|
|
||||||
// Fourth recoverable-run tenant. Iterates `storage.blobs`
|
// Fourth recoverable-run tenant. Iterates `storage.blobs` and
|
||||||
// and verifies each row against the physical backend AND
|
// checks the reference-counting invariant `dedup_gc` relies on:
|
||||||
// against the reference-counting invariants that `dedup_gc`
|
// `refcount_mismatch` (inconsistent — an under-count lets GC reap
|
||||||
// relies on. Three per-row checks (subject-iteration in
|
// a live blob, an over-count pins a dead one), repairable under
|
||||||
// action): `blob_missing_from_backend` (data_loss, bytes
|
// `?repair=true`.
|
||||||
// gone from disk), `refcount_mismatch` (inconsistent,
|
//
|
||||||
// dedup counter drift), and `blob_corrupted` (data_loss,
|
// DB-only, and takes no backend. Physical checks — missing bytes,
|
||||||
// deep mode only — bit-rot). Complements
|
// orphaned bytes, bit-rot — all belong to `backend_consistency`,
|
||||||
// `files_consistency` without doubling work: probing
|
// which merge-joins the backend enumeration against this same
|
||||||
// per-unique-blob preserves dedup savings vs probing
|
// table in one pass. This tenant used to probe the backend once
|
||||||
// per-file-chunk. See memory
|
// per row for missing bytes, which found strictly less than the
|
||||||
// `project_cdc_dual_storage_registries` for the rationale.
|
// merge-join at N round-trips instead of one enumeration.
|
||||||
let _ = Arc::new(
|
let _ = Arc::new(
|
||||||
crate::infrastructure::services::blobs_consistency_service::BlobsConsistencyCheck::new(
|
crate::infrastructure::services::blobs_consistency_service::BlobsConsistencyCheck::new(
|
||||||
maintenance_pool.clone(),
|
maintenance_pool.clone(),
|
||||||
core.blob_backend.clone(),
|
|
||||||
core.config.storage_entries.clone(),
|
|
||||||
self.storage_path.clone(),
|
|
||||||
// Same registry instance GC reaps from — see
|
// Same registry instance GC reaps from — see
|
||||||
// DedupService::reference_registry.
|
// DedupService::reference_registry.
|
||||||
core.dedup_service.reference_registry(),
|
core.dedup_service.reference_registry(),
|
||||||
|
|||||||
@@ -10,7 +10,8 @@
|
|||||||
//! backend listing, so it was invisible here by construction. That half
|
//! backend listing, so it was invisible here by construction. That half
|
||||||
//! was left to `blobs_consistency`'s per-row HEAD probe, which does not
|
//! was left to `blobs_consistency`'s per-row HEAD probe, which does not
|
||||||
//! survive the row counts this plan produces — see
|
//! survive the row counts this plan produces — see
|
||||||
//! `docs/plan/derived-blobs.md`.
|
//! `docs/plan/derived-blobs.md`. That probe is now gone: this tenant
|
||||||
|
//! owns every backend-side check, and `blobs_consistency` is DB-only.
|
||||||
//!
|
//!
|
||||||
//! ### Per-row checks
|
//! ### Per-row checks
|
||||||
//!
|
//!
|
||||||
@@ -22,6 +23,22 @@
|
|||||||
//! * `blob_missing_from_backend` (severity `data_loss`) — a registry
|
//! * `blob_missing_from_backend` (severity `data_loss`) — a registry
|
||||||
//! row whose bytes are absent. The opposite direction and the more
|
//! row whose bytes are absent. The opposite direction and the more
|
||||||
//! serious one: an orphan wastes space, this loses a file.
|
//! serious one: an orphan wastes space, this loses a file.
|
||||||
|
//! * `blob_corrupted` (severity `data_loss`, `?deep=true` only) —
|
||||||
|
//! the key exists on both sides but the bytes behind it no longer
|
||||||
|
//! hash to it. Silent bit-rot.
|
||||||
|
//! * `blob_unreadable` (severity `data_loss`, `?deep=true` only) —
|
||||||
|
//! the key exists but the bytes cannot be read at all: decrypt
|
||||||
|
//! failure (missing key), transport error, permissions. Same impact
|
||||||
|
//! as corruption from a file's point of view, different remedy,
|
||||||
|
//! hence a separate kind. Triage on the recorded `error`.
|
||||||
|
//!
|
||||||
|
//! ### Deep mode
|
||||||
|
//!
|
||||||
|
//! The last two moved here from `blobs_consistency`, which used to
|
||||||
|
//! carry a backend solely for them. Re-hashing is backend work end to
|
||||||
|
//! end — the only DB input is the hash — and this walk already holds
|
||||||
|
//! the matched key pairs, which is exactly the set worth reading. It
|
||||||
|
//! costs a full read of every blob, so it is opt-in.
|
||||||
//!
|
//!
|
||||||
//! ### Why the two orderings agree
|
//! ### Why the two orderings agree
|
||||||
//!
|
//!
|
||||||
@@ -71,14 +88,18 @@ use crate::infrastructure::scheduler::{
|
|||||||
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, ProgressKind, RecoverableJobHandler,
|
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, ProgressKind, RecoverableJobHandler,
|
||||||
RunOutcome, RunStatus, record_or_log,
|
RunOutcome, RunStatus, record_or_log,
|
||||||
};
|
};
|
||||||
|
use crate::infrastructure::services::blob_diagnostics::affected_files;
|
||||||
|
|
||||||
pub const BACKEND_CONSISTENCY_JOB_NAME: &str = "backend_consistency";
|
pub const BACKEND_CONSISTENCY_JOB_NAME: &str = "backend_consistency";
|
||||||
|
|
||||||
/// Same `params` JSONB key `blobs_consistency` uses — kept identical
|
/// `params` JSONB key under which the entry name being enumerated is
|
||||||
/// so operators grepping run rows see the same convention across
|
/// stashed on a Fresh run (matches `TARGET_NAME_PARAM` on
|
||||||
/// both storage-audit tenants.
|
/// `backend_migration`). Resumed runs re-read it so a paused audit
|
||||||
pub const PROBED_STORAGE_PARAM: &str =
|
/// survives restart without the admin re-specifying the target.
|
||||||
crate::infrastructure::services::blobs_consistency_service::PROBED_STORAGE_PARAM;
|
///
|
||||||
|
/// Defined here rather than in `blobs_consistency`, which no longer
|
||||||
|
/// touches a backend and so has no entry to scope.
|
||||||
|
pub const PROBED_STORAGE_PARAM: &str = "probed_storage";
|
||||||
|
|
||||||
/// Batch size for backend enumeration + DB probe. 500 is enough to
|
/// Batch size for backend enumeration + DB probe. 500 is enough to
|
||||||
/// amortise the DB round-trip while keeping the cancel-poll cadence
|
/// amortise the DB round-trip while keeping the cancel-poll cadence
|
||||||
@@ -151,8 +172,10 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
|
|||||||
"Merge-joins the storage backend's blob enumeration against \
|
"Merge-joins the storage backend's blob enumeration against \
|
||||||
storage.blobs, both ordered by hash, so one pass yields the delta \
|
storage.blobs, both ordered by hash, so one pass yields the delta \
|
||||||
in both directions: bytes on the backend no DB row claims, and \
|
in both directions: bytes on the backend no DB row claims, and \
|
||||||
rows whose bytes are gone. Read-only — nothing is uploaded or \
|
rows whose bytes are gone. Add ?deep=true to also read every \
|
||||||
deleted."
|
matched blob back and re-hash it, catching silent bit-rot — that \
|
||||||
|
is a full read of storage and can take hours. Read-only in both \
|
||||||
|
modes: nothing is uploaded or deleted."
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Approximate total: on a healthy install every backend blob
|
/// Approximate total: on a healthy install every backend blob
|
||||||
@@ -259,6 +282,49 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Deep mode — read every matched blob back and re-hash it, rather
|
||||||
|
// than trusting that a key present on both sides means the bytes
|
||||||
|
// behind it are still the bytes that key names.
|
||||||
|
//
|
||||||
|
// It lives here rather than in `blobs_consistency` because it is
|
||||||
|
// a backend operation end to end: the only DB input is the hash,
|
||||||
|
// which this merge-join already holds. Keeping it there forced
|
||||||
|
// that tenant to carry a backend for one flag, which is the
|
||||||
|
// overlap this split removes.
|
||||||
|
//
|
||||||
|
// Persisted to `params.deep` on a Fresh run so a Resume picks up
|
||||||
|
// the same mode (a Paused deep scan must not silently continue
|
||||||
|
// shallow) and the admin run-detail view can show what the scan
|
||||||
|
// actually verified. Written BEFORE the walk so a crash mid-batch
|
||||||
|
// still leaves the marker.
|
||||||
|
let deep = if is_fresh {
|
||||||
|
let v = if args.deep { "true" } else { "false" };
|
||||||
|
if let Err(e) = store.set_string_param("deep", v).await {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("failed to persist deep flag to params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
args.deep
|
||||||
|
} else {
|
||||||
|
match store.get_string_param("deep").await {
|
||||||
|
Ok(Some(v)) => v == "true",
|
||||||
|
Ok(None) => false,
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("read `deep` from params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if deep {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "backend_consistency.deep_mode_active",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
"deep mode: re-reading + re-hashing every matched blob (bit-rot detection)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// Cursor = opaque backend continuation token, UTF-8-encoded.
|
// Cursor = opaque backend continuation token, UTF-8-encoded.
|
||||||
// Each backend defines its own format (local = shard/hash,
|
// Each backend defines its own format (local = shard/hash,
|
||||||
// S3 = ListObjectsV2 continuation token, Azure = list
|
// S3 = ListObjectsV2 continuation token, Azure = list
|
||||||
@@ -469,7 +535,20 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
|
|||||||
let mut di = db_hashes.iter().peekable();
|
let mut di = db_hashes.iter().peekable();
|
||||||
loop {
|
loop {
|
||||||
match (bi.peek(), di.peek()) {
|
match (bi.peek(), di.peek()) {
|
||||||
|
// Present on both sides. Shallow: nothing to say — the
|
||||||
|
// key exists where the registry claims. Deep: the key
|
||||||
|
// matching says nothing about the bytes behind it, so
|
||||||
|
// read them back and re-hash.
|
||||||
|
//
|
||||||
|
// Guarded by `in_range` so a pair past the horizon is
|
||||||
|
// not read twice — the cursor stops at the horizon, so
|
||||||
|
// that pair comes round again next batch and is
|
||||||
|
// verified then.
|
||||||
(Some(b), Some(d)) if b.hash == **d => {
|
(Some(b), Some(d)) if b.hash == **d => {
|
||||||
|
if deep && in_range(&b.hash) {
|
||||||
|
finding_count +=
|
||||||
|
self.verify_bytes(store, backend.as_ref(), &b.hash).await;
|
||||||
|
}
|
||||||
bi.next();
|
bi.next();
|
||||||
di.next();
|
di.next();
|
||||||
}
|
}
|
||||||
@@ -581,3 +660,106 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl BackendConsistencyCheck {
|
||||||
|
/// Deep-mode per-blob verification. Reads the blob back, re-hashes it,
|
||||||
|
/// and records what it finds. Returns the number of findings recorded
|
||||||
|
/// (0 or 1) so the caller's counter stays the single tally.
|
||||||
|
///
|
||||||
|
/// Moved here from `blobs_consistency` along with the rest of the
|
||||||
|
/// backend-touching work: the merge-join already holds a verified
|
||||||
|
/// key pair, which is exactly the set worth reading.
|
||||||
|
async fn verify_bytes(
|
||||||
|
&self,
|
||||||
|
store: &dyn JobStore,
|
||||||
|
backend: &dyn BlobStorageBackend,
|
||||||
|
hash: &str,
|
||||||
|
) -> u64 {
|
||||||
|
match recompute_hash(backend, hash).await {
|
||||||
|
// The bytes still hash to the key they are filed under.
|
||||||
|
Ok(computed) if computed == hash => 0,
|
||||||
|
// Silent bit-rot. `computed_hash` is reported rather than a
|
||||||
|
// bare "mismatch" because the value is diagnostic: a one-bit
|
||||||
|
// flip, a truncation and a whole-object swap leave distinct
|
||||||
|
// signatures.
|
||||||
|
Ok(computed) => {
|
||||||
|
let affected = affected_files(self.pool.as_ref(), hash).await;
|
||||||
|
record_or_log(
|
||||||
|
store,
|
||||||
|
BACKEND_CONSISTENCY_JOB_NAME,
|
||||||
|
"blob_corrupted",
|
||||||
|
"data_loss",
|
||||||
|
None,
|
||||||
|
serde_json::json!({
|
||||||
|
"hash": hash,
|
||||||
|
"computed_hash": computed,
|
||||||
|
"backend": backend.backend_type(),
|
||||||
|
"affected_files": affected,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
1
|
||||||
|
}
|
||||||
|
// Bytes are there by key but cannot be read at all: decrypt
|
||||||
|
// failure (missing key), transport error, permissions. Same
|
||||||
|
// impact as corruption from a file's point of view — the
|
||||||
|
// content is inaccessible — but a different remedy, which is
|
||||||
|
// why it is a separate kind rather than folded into
|
||||||
|
// `blob_corrupted`. Operators triage on `error`.
|
||||||
|
Err(e) => {
|
||||||
|
let affected = affected_files(self.pool.as_ref(), hash).await;
|
||||||
|
record_or_log(
|
||||||
|
store,
|
||||||
|
BACKEND_CONSISTENCY_JOB_NAME,
|
||||||
|
"blob_unreadable",
|
||||||
|
"data_loss",
|
||||||
|
None,
|
||||||
|
serde_json::json!({
|
||||||
|
"hash": hash,
|
||||||
|
"backend": backend.backend_type(),
|
||||||
|
"affected_files": affected,
|
||||||
|
"error": e.to_string(),
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
tracing::warn!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "backend_consistency.blob_unreadable",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
hash = %hash,
|
||||||
|
error = %e,
|
||||||
|
"🚨 blob unreadable in deep mode — recorded finding, continuing"
|
||||||
|
);
|
||||||
|
1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deep-mode helper — read the blob from the backend and recompute its
|
||||||
|
/// BLAKE3 hash. Returns the recomputed hex string; callers compare it
|
||||||
|
/// against the expected hash themselves. Returning the actual hash (not
|
||||||
|
/// a bool) lets the finding surface WHAT the bytes now hash to, which is
|
||||||
|
/// diagnostic gold: a one-bit flip has a very different signature from a
|
||||||
|
/// chunk-boundary corruption or a truncated read. `Err(_)` on any
|
||||||
|
/// backend-side error — the caller records that as `blob_unreadable`
|
||||||
|
/// rather than as corruption.
|
||||||
|
async fn recompute_hash(
|
||||||
|
backend: &dyn BlobStorageBackend,
|
||||||
|
expected_hash: &str,
|
||||||
|
) -> Result<String, crate::common::errors::DomainError> {
|
||||||
|
use crate::common::errors::DomainError;
|
||||||
|
use futures::StreamExt;
|
||||||
|
|
||||||
|
let mut stream = backend.get_blob_stream(expected_hash).await?;
|
||||||
|
let mut hasher = blake3::Hasher::new();
|
||||||
|
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
let bytes = chunk.map_err(|e| {
|
||||||
|
DomainError::internal_error("BackendConsistency", format!("stream read: {e}"))
|
||||||
|
})?;
|
||||||
|
hasher.update(&bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(hasher.finalize().to_hex().to_string())
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
//! Reverse-lookup helpers shared by the storage consistency tenants.
|
||||||
|
//!
|
||||||
|
//! `blobs_consistency` (DB-side: refcount drift) and
|
||||||
|
//! `backend_consistency` (backend-side: missing / orphaned / corrupted
|
||||||
|
//! bytes) both answer the same operator question when they emit a
|
||||||
|
//! finding — *which files does this hash break?* — so the query lives
|
||||||
|
//! here rather than in whichever tenant happened to need it first.
|
||||||
|
|
||||||
|
use sqlx::PgPool;
|
||||||
|
|
||||||
|
/// Cap on reverse-lookup file names surfaced in a finding's detail.
|
||||||
|
/// Keeps detail JSON bounded when a broken blob is referenced by
|
||||||
|
/// hundreds of files.
|
||||||
|
const AFFECTED_FILES_SAMPLE: i64 = 5;
|
||||||
|
|
||||||
|
/// Sample of file names that reference this blob — either directly
|
||||||
|
/// (`files.blob_hash = $hash`, legacy pre-CDC) or transitively via a
|
||||||
|
/// manifest (`chunk_hashes @> ARRAY[$hash]`, the post-CDC dominant
|
||||||
|
/// path). Capped so a chunk shared by 10 000 files doesn't blow up the
|
||||||
|
/// finding detail JSON. Order is arbitrary — this samples for
|
||||||
|
/// diagnosis, it does not enumerate.
|
||||||
|
///
|
||||||
|
/// Returns an empty vec on query error: a finding with no sample is
|
||||||
|
/// still a finding, and failing the sweep because the diagnostic
|
||||||
|
/// garnish didn't load would trade the whole scan for a nicety.
|
||||||
|
pub(crate) async fn affected_files(pool: &PgPool, hash: &str) -> Vec<String> {
|
||||||
|
let rows: Vec<(String,)> = sqlx::query_as(
|
||||||
|
r#"
|
||||||
|
SELECT DISTINCT f.name
|
||||||
|
FROM storage.files f
|
||||||
|
WHERE f.blob_hash = $1
|
||||||
|
OR EXISTS (
|
||||||
|
SELECT 1 FROM storage.chunk_manifests m
|
||||||
|
WHERE m.file_hash = f.blob_hash
|
||||||
|
AND $1 = ANY(m.chunk_hashes)
|
||||||
|
)
|
||||||
|
LIMIT $2
|
||||||
|
"#,
|
||||||
|
)
|
||||||
|
.bind(hash)
|
||||||
|
.bind(AFFECTED_FILES_SAMPLE)
|
||||||
|
.fetch_all(pool)
|
||||||
|
.await
|
||||||
|
.unwrap_or_default();
|
||||||
|
rows.into_iter().map(|(n,)| n).collect()
|
||||||
|
}
|
||||||
@@ -1,32 +1,12 @@
|
|||||||
//! Fourth tenant of Part 2 (recoverable-run engine).
|
//! Fourth tenant of Part 2 (recoverable-run engine).
|
||||||
//!
|
//!
|
||||||
//! Iterates `storage.blobs` — the content-addressable registry —
|
//! Iterates `storage.blobs` — the content-addressable registry — and
|
||||||
//! and verifies each row against the physical backend AND against
|
//! checks the reference-counting invariant `dedup_gc` relies on.
|
||||||
//! the reference-counting invariants that `dedup_gc` relies on.
|
|
||||||
//!
|
//!
|
||||||
//! Three per-row checks (subject-iteration principle in action —
|
//! **Database only.** It opens no backend and makes no network call;
|
||||||
//! one walk, multiple branches):
|
//! `?storage=<name>` and `?deep=true` are both inert here.
|
||||||
//!
|
//!
|
||||||
//! * `blob_missing_from_backend` (severity `data_loss`) — the DB
|
//! One per-row check:
|
||||||
//! row says the hash exists but `BlobStorageBackend::blob_exists`
|
|
||||||
//! returns false. Bytes gone from disk / S3 / Azure. Any file
|
|
||||||
//! whose manifest references this hash (or whose whole-file
|
|
||||||
//! `blob_hash` points at it) will fail to read.
|
|
||||||
//!
|
|
||||||
//! * `blob_corrupted` (severity `data_loss`, deep mode only) —
|
|
||||||
//! bytes exist on the backend but their BLAKE3 no longer matches
|
|
||||||
//! the hash under which they're indexed. Silent bit-rot. Only
|
|
||||||
//! runs when the operator passes `?deep=true` because it costs a
|
|
||||||
//! full read of every blob.
|
|
||||||
//!
|
|
||||||
//! * `blob_unreadable` (severity `data_loss`, deep mode only) —
|
|
||||||
//! `blob_exists` returned true but the read pipeline errored (can't
|
|
||||||
//! decrypt, network glitch, permission error, etc.). Distinct from
|
|
||||||
//! `blob_corrupted` (which requires successful read + hash mismatch);
|
|
||||||
//! here we can't get bytes out at all. Same operator impact — any
|
|
||||||
//! file referencing this hash is inaccessible — but the remedy
|
|
||||||
//! differs (key recovery, retry, or blob replacement, depending on
|
|
||||||
//! the recorded `error` field).
|
|
||||||
//!
|
//!
|
||||||
//! * `refcount_mismatch` (severity `inconsistent`) —
|
//! * `refcount_mismatch` (severity `inconsistent`) —
|
||||||
//! `storage.blobs.ref_count` disagrees with the actual reference
|
//! `storage.blobs.ref_count` disagrees with the actual reference
|
||||||
@@ -36,85 +16,57 @@
|
|||||||
//! a blob is being pinned longer than needed. Content-safe either
|
//! a blob is being pinned longer than needed. Content-safe either
|
||||||
//! way (the storage.blobs row is fine, the counter is wrong).
|
//! way (the storage.blobs row is fine, the counter is wrong).
|
||||||
//!
|
//!
|
||||||
//! ### Complements `files_consistency`
|
//! ### Why nothing physical lives here any more
|
||||||
//!
|
//!
|
||||||
//! `files_consistency` (Slice 6/10) iterates files and verifies DB
|
//! This tenant used to probe `BlobStorageBackend::blob_exists` once
|
||||||
//! integrity. `blobs_consistency` iterates the storage registry and
|
//! per row for `blob_missing_from_backend`, and under `?deep=true`
|
||||||
//! verifies physical existence + counter integrity. Together they
|
//! read and re-hashed every blob for `blob_corrupted` /
|
||||||
//! cover both sides of the reference graph. Neither doubles the
|
//! `blob_unreadable`.
|
||||||
//! other's work — probing per-blob (here) instead of per-file-chunk
|
|
||||||
//! preserves dedup savings: a chunk shared by 5 files gets probed
|
|
||||||
//! ONCE.
|
|
||||||
//!
|
//!
|
||||||
//! ### Not covered here
|
//! All three moved to `backend_consistency`, which merge-joins the
|
||||||
|
//! backend's enumeration against this same table in one ordered pass.
|
||||||
|
//! It reports the same missing bytes, plus the backend-only orphans a
|
||||||
|
//! DB walk cannot see by construction, at one enumeration instead of
|
||||||
|
//! N round-trips — and a deep pass there re-hashes the matched pairs
|
||||||
|
//! it already holds. Keeping the probe here bought nothing and made
|
||||||
|
//! every scheduled sweep pay for it.
|
||||||
//!
|
//!
|
||||||
//! * **Orphan bytes on the backend** (files on disk with no DB row)
|
//! What is left is the half that needs no backend at all: a counter,
|
||||||
//! — belongs in the future `backend_consistency` tenant which
|
//! and the two tables that determine what it should be.
|
||||||
//! iterates the backend itself. Requires the `list_blob_hashes`
|
//!
|
||||||
//! trait extension and per-backend enumeration impls.
|
//! ### Elsewhere in the graph
|
||||||
|
//!
|
||||||
|
//! * **Physical existence, orphan bytes, bit-rot** —
|
||||||
|
//! `backend_consistency`.
|
||||||
|
//! * **File-side DB integrity** (parent folder, blob reference,
|
||||||
|
//! denormalised size) — `files_consistency`.
|
||||||
//! * **Manifest-level integrity** (`storage.chunk_manifests` rows
|
//! * **Manifest-level integrity** (`storage.chunk_manifests` rows
|
||||||
//! pointing at reaped chunks) — already covered by
|
//! pointing at reaped chunks) — `files_consistency::chunk_missing`.
|
||||||
//! `files_consistency::chunk_missing`.
|
//! * **The OTHER refcount** (`chunk_manifests.ref_count`, which every
|
||||||
|
//! whole-Blob reference lands on) — `manifests_consistency`.
|
||||||
|
|
||||||
use std::path::PathBuf;
|
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
use chrono::{DateTime, Duration, Utc};
|
|
||||||
use sqlx::PgPool;
|
use sqlx::PgPool;
|
||||||
|
|
||||||
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
|
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
|
||||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
|
||||||
use crate::common::config::NamedStorageEntry;
|
|
||||||
use crate::infrastructure::scheduler::{
|
use crate::infrastructure::scheduler::{
|
||||||
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
|
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, Mutates, RecoverableJobHandler,
|
||||||
RunOutcome, RunStatus, record_or_log,
|
RunOutcome, RunStatus, record_or_log,
|
||||||
};
|
};
|
||||||
use crate::infrastructure::services::entry_backend::build_entry_backend;
|
use crate::infrastructure::services::blob_diagnostics::affected_files;
|
||||||
|
|
||||||
pub const BLOBS_CONSISTENCY_JOB_NAME: &str = "blobs_consistency";
|
pub const BLOBS_CONSISTENCY_JOB_NAME: &str = "blobs_consistency";
|
||||||
|
|
||||||
/// `params` JSONB key under which the entry name being probed is
|
|
||||||
/// stashed on a Fresh run (matches `TARGET_NAME_PARAM` on
|
|
||||||
/// `backend_migration`). Resumed runs re-read it so a paused audit
|
|
||||||
/// survives restart without the admin re-specifying the target.
|
|
||||||
pub const PROBED_STORAGE_PARAM: &str = "probed_storage";
|
|
||||||
|
|
||||||
/// Rows per batch. Blobs are numerous (millions on a busy install)
|
/// Rows per batch. Blobs are numerous (millions on a busy install)
|
||||||
/// but per-row work is one indexed backend probe + one indexed SQL
|
/// but per-row work is now a single indexed SQL ref-count comparison,
|
||||||
/// ref-count query. 200 balances cancel-poll cadence against
|
/// with no backend round-trip. 200 balances cancel-poll cadence
|
||||||
/// round-trip amortisation.
|
/// against round-trip amortisation.
|
||||||
const BATCH_SIZE: i64 = 200;
|
const BATCH_SIZE: i64 = 200;
|
||||||
|
|
||||||
/// Grace window — rows created within this window are skipped by
|
|
||||||
/// the physical-existence probe because the write path is
|
|
||||||
/// durability-before-visibility: `dedup_service` writes bytes, then
|
|
||||||
/// registers the row a few ms later. A scan catching a row
|
|
||||||
/// mid-write would false-positive it as `blob_missing_from_backend`.
|
|
||||||
/// Same shape `dedup_gc` uses (see its `grace_secs`).
|
|
||||||
const CREATE_GRACE: Duration = Duration::hours(1);
|
|
||||||
|
|
||||||
/// Cap on reverse-lookup file names surfaced in a finding's detail.
|
|
||||||
/// Keeps detail JSON size bounded when a broken blob is referenced
|
|
||||||
/// by hundreds of files.
|
|
||||||
const AFFECTED_FILES_SAMPLE: i64 = 5;
|
|
||||||
|
|
||||||
pub struct BlobsConsistencyCheck {
|
pub struct BlobsConsistencyCheck {
|
||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
/// The default backend to probe when `args.storage` is `None` —
|
|
||||||
/// the currently-active LIVE backend, injected at DI time. Runs
|
|
||||||
/// with `?storage=<name>` build a fresh backend for the named
|
|
||||||
/// entry instead (via [`build_entry_backend`]).
|
|
||||||
backend: Arc<dyn BlobStorageBackend>,
|
|
||||||
/// Snapshot of `AppConfig.storage_entries` used to resolve
|
|
||||||
/// `args.storage` to a `NamedStorageEntry`. Empty for the
|
|
||||||
/// legacy zero-entries path — `?storage=<name>` runs then
|
|
||||||
/// fail-fast with a clear "no entries declared" message.
|
|
||||||
storage_entries: Vec<NamedStorageEntry>,
|
|
||||||
/// Ambient `AppConfig.storage_path` — used as the `root_dir`
|
|
||||||
/// fallback for a Local target entry with no `_ROOT_DIR`. Same
|
|
||||||
/// fallback rule the boot path uses.
|
|
||||||
storage_path_fallback: PathBuf,
|
|
||||||
/// The chunk-level page query, assembled once from the blob-reference
|
/// The chunk-level page query, assembled once from the blob-reference
|
||||||
/// registry so this recompute and `dedup_gc` agree on what "referenced"
|
/// registry so this recompute and `dedup_gc` agree on what "referenced"
|
||||||
/// means. Built at construction rather than per page so the sweep runs a
|
/// means. Built at construction rather than per page so the sweep runs a
|
||||||
@@ -166,7 +118,6 @@ fn chunk_page_sql(registry: &BlobReferenceRegistry) -> String {
|
|||||||
b.hash AS hash,
|
b.hash AS hash,
|
||||||
b.size AS size,
|
b.size AS size,
|
||||||
b.ref_count AS ref_count,
|
b.ref_count AS ref_count,
|
||||||
b.created_at AS created_at,
|
|
||||||
({expected})::bigint AS actual_ref_count
|
({expected})::bigint AS actual_ref_count
|
||||||
FROM storage.blobs b
|
FROM storage.blobs b
|
||||||
WHERE ($1::text IS NULL OR b.hash > $1)
|
WHERE ($1::text IS NULL OR b.hash > $1)
|
||||||
@@ -176,18 +127,9 @@ fn chunk_page_sql(registry: &BlobReferenceRegistry) -> String {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl BlobsConsistencyCheck {
|
impl BlobsConsistencyCheck {
|
||||||
pub fn new(
|
pub fn new(pool: Arc<PgPool>, reference_registry: Arc<BlobReferenceRegistry>) -> Self {
|
||||||
pool: Arc<PgPool>,
|
|
||||||
backend: Arc<dyn BlobStorageBackend>,
|
|
||||||
storage_entries: Vec<NamedStorageEntry>,
|
|
||||||
storage_path_fallback: PathBuf,
|
|
||||||
reference_registry: Arc<BlobReferenceRegistry>,
|
|
||||||
) -> Self {
|
|
||||||
Self {
|
Self {
|
||||||
pool,
|
pool,
|
||||||
backend,
|
|
||||||
storage_entries,
|
|
||||||
storage_path_fallback,
|
|
||||||
chunk_page_sql: chunk_page_sql(&reference_registry),
|
chunk_page_sql: chunk_page_sql(&reference_registry),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -209,7 +151,6 @@ struct BlobRow {
|
|||||||
hash: String,
|
hash: String,
|
||||||
size: i64,
|
size: i64,
|
||||||
ref_count: i32,
|
ref_count: i32,
|
||||||
created_at: DateTime<Utc>,
|
|
||||||
/// Real reference count derived from the actual references —
|
/// Real reference count derived from the actual references —
|
||||||
/// files' whole-file `blob_hash` PLUS every chunk hash across
|
/// files' whole-file `blob_hash` PLUS every chunk hash across
|
||||||
/// `storage.chunk_manifests`. Compared to `ref_count` (the
|
/// `storage.chunk_manifests`. Compared to `ref_count` (the
|
||||||
@@ -224,13 +165,12 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn description(&self) -> &'static str {
|
fn description(&self) -> &'static str {
|
||||||
"Walks storage.blobs and checks each row against the reference- \
|
"Walks storage.blobs and reports rows whose ref_count disagrees \
|
||||||
counting invariants dedup_gc relies on, and probes the backend \
|
with the references that actually exist. An under-count lets \
|
||||||
once per row for missing bytes. That probe costs one backend \
|
dedup_gc reap a blob that is still in use; an over-count pins \
|
||||||
call per blob — backend_consistency finds the same missing \
|
one nothing needs. Database only — it never touches the storage \
|
||||||
bytes in a single merge-join pass, and orphaned bytes too, so \
|
backend, so it is cheap and safe to run at any time. Missing, \
|
||||||
prefer it on large installs. Add ?deep=true to re-read and \
|
orphaned or corrupted bytes are backend_consistency's job."
|
||||||
re-hash every blob for bit-rot; that is a full read of storage."
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn mutates(&self) -> Mutates {
|
fn mutates(&self) -> Mutates {
|
||||||
@@ -272,78 +212,12 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
args: &JobRunArgs,
|
args: &JobRunArgs,
|
||||||
resume_cursor: Option<Vec<u8>>,
|
resume_cursor: Option<Vec<u8>>,
|
||||||
) -> RunOutcome {
|
) -> RunOutcome {
|
||||||
// Resolve the backend to probe. Two paths, mirroring the
|
// No backend is resolved here, and `?storage=<name>` is inert:
|
||||||
// Fresh/Resumed split the backend_migration handler uses:
|
// this tenant reads nothing but the database. Everything physical
|
||||||
|
// — existence, orphan bytes, bit-rot — belongs to
|
||||||
|
// `backend_consistency`, which finds it in one enumeration pass
|
||||||
|
// instead of one probe per row.
|
||||||
//
|
//
|
||||||
// * Fresh + args.storage=Some — probe that named entry
|
|
||||||
// instead of the live backend. Stamp probed_storage in
|
|
||||||
// params so a mid-audit restart resumes against the same
|
|
||||||
// entry without re-input.
|
|
||||||
// * Fresh + args.storage=None — probe the live backend
|
|
||||||
// (today's default; audit of what the app is actually
|
|
||||||
// using).
|
|
||||||
// * Resumed — read probed_storage from params; None means
|
|
||||||
// the original run was against the live backend.
|
|
||||||
let is_fresh = resume_cursor.is_none();
|
|
||||||
let probed_storage: Option<String> = if is_fresh {
|
|
||||||
let name = args.storage.clone();
|
|
||||||
if let Some(n) = &name
|
|
||||||
&& let Err(e) = store.set_string_param(PROBED_STORAGE_PARAM, n).await
|
|
||||||
{
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!("persist {PROBED_STORAGE_PARAM} to params: {e}"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
name
|
|
||||||
} else {
|
|
||||||
match store.get_string_param(PROBED_STORAGE_PARAM).await {
|
|
||||||
Ok(v) => v,
|
|
||||||
Err(e) => {
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!("read {PROBED_STORAGE_PARAM} from params: {e}"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
let backend: Arc<dyn BlobStorageBackend> = match &probed_storage {
|
|
||||||
None => self.backend.clone(),
|
|
||||||
Some(name) => match self.storage_entries.iter().find(|e| &e.name == name) {
|
|
||||||
Some(entry) => build_entry_backend(entry, &self.storage_path_fallback),
|
|
||||||
None => {
|
|
||||||
let available = if self.storage_entries.is_empty() {
|
|
||||||
"(none)".to_string()
|
|
||||||
} else {
|
|
||||||
self.storage_entries
|
|
||||||
.iter()
|
|
||||||
.map(|e| e.name.as_str())
|
|
||||||
.collect::<Vec<_>>()
|
|
||||||
.join(", ")
|
|
||||||
};
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!(
|
|
||||||
"storage entry `{name}` not found in OXICLOUD_STORAGE_ENTRIES. \
|
|
||||||
Available: [{available}]"
|
|
||||||
),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
},
|
|
||||||
};
|
|
||||||
if let Err(e) = backend.initialize().await {
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!("probed backend init: {e}"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
if let Some(name) = &probed_storage {
|
|
||||||
tracing::info!(
|
|
||||||
target: "audit",
|
|
||||||
event = "blobs_consistency.probe_scoped",
|
|
||||||
run_id = %store.run_id(),
|
|
||||||
probed_storage = %name,
|
|
||||||
"blobs_consistency probing entry `{name}` (via ?storage=<name>) instead of \
|
|
||||||
live backend"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Snapshot "is this a Fresh run?" BEFORE the resume_cursor
|
// Snapshot "is this a Fresh run?" BEFORE the resume_cursor
|
||||||
// match consumes it — otherwise the `is_none()` check later
|
// match consumes it — otherwise the `is_none()` check later
|
||||||
// borrows a partially-moved value. Fresh = no cursor bytes
|
// borrows a partially-moved value. Fresh = no cursor bytes
|
||||||
@@ -375,60 +249,14 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
// `extra_stats` so operators see "found N, fixed M" in one line.
|
// `extra_stats` so operators see "found N, fixed M" in one line.
|
||||||
let mut repaired_count = 0u64;
|
let mut repaired_count = 0u64;
|
||||||
|
|
||||||
// Deep mode is a per-run flag with two consumers:
|
// `?deep=true` is not handled here. Re-reading and re-hashing
|
||||||
// 1. This handler — decides whether to re-hash bytes.
|
// bytes is backend work end to end, so it moved to
|
||||||
// 2. The admin panel — needs to display whether the run
|
// `backend_consistency`, where the merge-join already holds the
|
||||||
// was deep so operators know what the scan actually
|
// matched key pairs worth verifying. A deep flag on this tenant
|
||||||
// verified.
|
// would be a flag with nothing to do.
|
||||||
//
|
|
||||||
// On a Fresh run we take it from `deep` (the trigger
|
|
||||||
// endpoint stamps `?deep=true` onto the args) and stash it
|
|
||||||
// in `params.deep` so:
|
|
||||||
// * Resume picks up the same mode (would previously become
|
|
||||||
// non-deep on Resume — a Paused deep scan silently lost
|
|
||||||
// its `deep` intent).
|
|
||||||
// * The admin panel run-detail view can render
|
|
||||||
// `params.deep = "true"` alongside `target_name`,
|
|
||||||
// `progress_kind`, etc.
|
|
||||||
//
|
|
||||||
// Persist BEFORE the walk so a mid-fresh-batch crash still
|
|
||||||
// leaves a Paused row with the right mode marker.
|
|
||||||
let deep = if is_fresh {
|
|
||||||
let deep = args.deep;
|
|
||||||
let v = if deep { "true" } else { "false" };
|
|
||||||
if let Err(e) = store.set_string_param("deep", v).await {
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!("failed to persist deep flag to params: {e}"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
deep
|
|
||||||
} else {
|
|
||||||
// Resumed run — read the persisted flag. Default to
|
|
||||||
// false (fast mode) if the row is a pre-K3.5 Paused
|
|
||||||
// scan without the param stashed.
|
|
||||||
match store.get_string_param("deep").await {
|
|
||||||
Ok(Some(v)) => v == "true",
|
|
||||||
Ok(None) => false,
|
|
||||||
Err(e) => {
|
|
||||||
return RunOutcome::Failed {
|
|
||||||
message: format!("read `deep` from params: {e}"),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
if deep {
|
// Repair mode persisted to `params.repair` so the admin run-detail
|
||||||
tracing::info!(
|
// view can display it. Fresh persists what the trigger asked for;
|
||||||
target: "oxicloud::consistency",
|
|
||||||
event = "blobs_consistency.deep_mode_active",
|
|
||||||
run_id = %store.run_id(),
|
|
||||||
"deep mode: re-reading + re-hashing every blob (bit-rot detection)"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Repair mode: same shape as `deep` above so the admin run-
|
|
||||||
// detail view can display `params.repair = "true"` alongside
|
|
||||||
// `params.deep`. Fresh persists what the trigger asked for;
|
|
||||||
// Resume reads back so a paused repair scan stays a repair
|
// Resume reads back so a paused repair scan stays a repair
|
||||||
// scan (a mid-scan crash mustn't silently downgrade to
|
// scan (a mid-scan crash mustn't silently downgrade to
|
||||||
// discovery-only for the remaining rows).
|
// discovery-only for the remaining rows).
|
||||||
@@ -512,7 +340,6 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
finding_count = finding_count,
|
finding_count = finding_count,
|
||||||
repaired_count = repaired_count,
|
repaired_count = repaired_count,
|
||||||
repair_requested = repair,
|
repair_requested = repair,
|
||||||
deep = deep,
|
|
||||||
"blobs_consistency completed with {} finding(s), {} repaired",
|
"blobs_consistency completed with {} finding(s), {} repaired",
|
||||||
finding_count,
|
finding_count,
|
||||||
repaired_count
|
repaired_count
|
||||||
@@ -523,12 +350,12 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
let grace_cutoff = Utc::now() - CREATE_GRACE;
|
// No grace window here any more. It existed to keep the
|
||||||
|
// physical probe from flagging a blob whose bytes had landed
|
||||||
|
// but whose row hadn't — a write-path race this tenant no
|
||||||
|
// longer looks at. The refcount comparison reads one
|
||||||
|
// consistent DB snapshot, so there is nothing to wait for.
|
||||||
for row in &rows {
|
for row in &rows {
|
||||||
// (1) refcount_mismatch — content-safe check, cheap,
|
|
||||||
// always runs. Emitted BEFORE the physical probe so
|
|
||||||
// a broken-and-miscounted blob shows both findings.
|
|
||||||
if row.ref_count as i64 != row.actual_ref_count {
|
if row.ref_count as i64 != row.actual_ref_count {
|
||||||
finding_count += 1;
|
finding_count += 1;
|
||||||
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
|
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
|
||||||
@@ -610,135 +437,6 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Skip physical probes for rows within the write
|
|
||||||
// grace window — writes-in-flight would false-positive.
|
|
||||||
if row.created_at > grace_cutoff {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
// (2) blob_missing_from_backend — normal mode
|
|
||||||
// physical existence probe. Fails-open on backend
|
|
||||||
// error (log + skip): a transient S3 network blip
|
|
||||||
// shouldn't produce a flood of false data_loss
|
|
||||||
// findings.
|
|
||||||
let exists = match backend.blob_exists(&row.hash).await {
|
|
||||||
Ok(v) => v,
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(
|
|
||||||
target: "oxicloud::consistency",
|
|
||||||
event = "blobs_consistency.blob_exists_error",
|
|
||||||
run_id = %store.run_id(),
|
|
||||||
hash = %row.hash,
|
|
||||||
error = %e,
|
|
||||||
"blob_exists probe failed; skipping this row"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
if !exists {
|
|
||||||
finding_count += 1;
|
|
||||||
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
|
|
||||||
record_or_log(
|
|
||||||
store,
|
|
||||||
BLOBS_CONSISTENCY_JOB_NAME,
|
|
||||||
"blob_missing_from_backend",
|
|
||||||
"data_loss",
|
|
||||||
None,
|
|
||||||
serde_json::json!({
|
|
||||||
"hash": row.hash,
|
|
||||||
"size": row.size,
|
|
||||||
"ref_count": row.ref_count,
|
|
||||||
"affected_files": affected,
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
// No point re-hashing bytes that aren't there.
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
// (3) blob_corrupted — DEEP MODE only. Read the
|
|
||||||
// whole blob, recompute BLAKE3, compare to the hash
|
|
||||||
// it's indexed under. Any mismatch = silent bit-rot.
|
|
||||||
//
|
|
||||||
// Finding fields:
|
|
||||||
// * `hash` — expected hash (the key the blob is
|
|
||||||
// indexed under in `storage.blobs`).
|
|
||||||
// * `computed_hash` — what BLAKE3 of the current
|
|
||||||
// bytes actually produces. Diagnostic: a
|
|
||||||
// one-bit flip vs a truncation vs a whole-file
|
|
||||||
// swap all leave distinctive signatures.
|
|
||||||
// `expected_hash` was NOT reused as a name to
|
|
||||||
// avoid mistaking it for "the hash we expect to
|
|
||||||
// see on disk (i.e. what will fix this)".
|
|
||||||
if deep {
|
|
||||||
match recompute_hash(backend.as_ref(), &row.hash).await {
|
|
||||||
Ok(computed_hash) if computed_hash == row.hash => {}
|
|
||||||
Ok(computed_hash) => {
|
|
||||||
finding_count += 1;
|
|
||||||
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
|
|
||||||
record_or_log(
|
|
||||||
store,
|
|
||||||
BLOBS_CONSISTENCY_JOB_NAME,
|
|
||||||
"blob_corrupted",
|
|
||||||
"data_loss",
|
|
||||||
None,
|
|
||||||
serde_json::json!({
|
|
||||||
"hash": row.hash,
|
|
||||||
"computed_hash": computed_hash,
|
|
||||||
"size": row.size,
|
|
||||||
"ref_count": row.ref_count,
|
|
||||||
"affected_files": affected,
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
// Blob can't be read at all — record as
|
|
||||||
// `blob_unreadable`. Distinct from
|
|
||||||
// `blob_corrupted` (hash mismatch = we
|
|
||||||
// can read but content differs): here
|
|
||||||
// we can't get bytes out to hash. Common
|
|
||||||
// causes: decrypt failure (missing key),
|
|
||||||
// network glitch on S3/Azure, missing
|
|
||||||
// file on Local, permission error.
|
|
||||||
//
|
|
||||||
// Recorded as `data_loss` because from
|
|
||||||
// the file's perspective the outcome is
|
|
||||||
// the same as corruption: content is
|
|
||||||
// inaccessible. Admins triage the error
|
|
||||||
// string to distinguish transient
|
|
||||||
// (retry-safe) from permanent (needs
|
|
||||||
// key recovery or blob replacement).
|
|
||||||
finding_count += 1;
|
|
||||||
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
|
|
||||||
record_or_log(
|
|
||||||
store,
|
|
||||||
BLOBS_CONSISTENCY_JOB_NAME,
|
|
||||||
"blob_unreadable",
|
|
||||||
"data_loss",
|
|
||||||
None,
|
|
||||||
serde_json::json!({
|
|
||||||
"hash": row.hash,
|
|
||||||
"size": row.size,
|
|
||||||
"ref_count": row.ref_count,
|
|
||||||
"affected_files": affected,
|
|
||||||
"error": e.to_string(),
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
tracing::warn!(
|
|
||||||
target: "oxicloud::consistency",
|
|
||||||
event = "blobs_consistency.blob_unreadable",
|
|
||||||
run_id = %store.run_id(),
|
|
||||||
hash = %row.hash,
|
|
||||||
error = %e,
|
|
||||||
"🚨 blob unreadable in deep mode — recorded finding, continuing"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Advance cursor + checkpoint.
|
// Advance cursor + checkpoint.
|
||||||
@@ -759,7 +457,6 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
finding_count = finding_count,
|
finding_count = finding_count,
|
||||||
repaired_count = repaired_count,
|
repaired_count = repaired_count,
|
||||||
repair_requested = repair,
|
repair_requested = repair,
|
||||||
deep = deep,
|
|
||||||
"blobs_consistency completed with {} finding(s), {} repaired",
|
"blobs_consistency completed with {} finding(s), {} repaired",
|
||||||
finding_count,
|
finding_count,
|
||||||
repaired_count
|
repaired_count
|
||||||
@@ -773,69 +470,6 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Sample of file names that reference this blob — either directly
|
|
||||||
/// (`files.blob_hash = $hash`, legacy pre-CDC) or transitively via a
|
|
||||||
/// manifest (`chunk_hashes @> ARRAY[$hash]`, post-CDC dominant path).
|
|
||||||
/// Capped so a chunk shared by 10 000 files doesn't blow up the
|
|
||||||
/// finding detail JSON. Order is arbitrary — sampling for
|
|
||||||
/// diagnosis, not enumeration.
|
|
||||||
async fn affected_files(pool: &PgPool, hash: &str) -> Vec<String> {
|
|
||||||
let rows: Vec<(String,)> = sqlx::query_as(
|
|
||||||
r#"
|
|
||||||
SELECT DISTINCT f.name
|
|
||||||
FROM storage.files f
|
|
||||||
WHERE f.blob_hash = $1
|
|
||||||
OR EXISTS (
|
|
||||||
SELECT 1 FROM storage.chunk_manifests m
|
|
||||||
WHERE m.file_hash = f.blob_hash
|
|
||||||
AND $1 = ANY(m.chunk_hashes)
|
|
||||||
)
|
|
||||||
LIMIT $2
|
|
||||||
"#,
|
|
||||||
)
|
|
||||||
.bind(hash)
|
|
||||||
.bind(AFFECTED_FILES_SAMPLE)
|
|
||||||
.fetch_all(pool)
|
|
||||||
.await
|
|
||||||
.unwrap_or_default();
|
|
||||||
rows.into_iter().map(|(n,)| n).collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Deep-mode helper — read the blob from the backend and recompute
|
|
||||||
/// its BLAKE3 hash. Returns `Ok(true)` when the recomputed hash
|
|
||||||
/// matches `expected_hash` (byte for byte), `Ok(false)` on mismatch
|
|
||||||
/// (bit-rot), `Err(_)` on any backend-side error (network blip,
|
|
||||||
/// permission issue) — callers log-and-skip errors since a transient
|
|
||||||
/// failure isn't a corruption signal.
|
|
||||||
/// Deep-mode helper — read the blob from the backend and recompute
|
|
||||||
/// its BLAKE3 hash. Returns the recomputed hex string; callers
|
|
||||||
/// compare against the expected hash themselves. Returning the
|
|
||||||
/// actual hash (not just a bool) lets the finding surface WHAT the
|
|
||||||
/// bytes now hash to, which is diagnostic gold: a specific one-bit
|
|
||||||
/// flip has a very different signature from a chunk-boundary
|
|
||||||
/// corruption or a truncated read. `Err(_)` on backend-side error
|
|
||||||
/// (network blip, permission issue) — callers log-and-skip since
|
|
||||||
/// transient failure isn't a corruption signal.
|
|
||||||
async fn recompute_hash(
|
|
||||||
backend: &dyn BlobStorageBackend,
|
|
||||||
expected_hash: &str,
|
|
||||||
) -> Result<String, crate::common::errors::DomainError> {
|
|
||||||
use crate::common::errors::DomainError;
|
|
||||||
use futures::StreamExt;
|
|
||||||
|
|
||||||
let mut stream = backend.get_blob_stream(expected_hash).await?;
|
|
||||||
let mut hasher = blake3::Hasher::new();
|
|
||||||
|
|
||||||
while let Some(chunk) = stream.next().await {
|
|
||||||
let bytes = chunk.map_err(|e| {
|
|
||||||
DomainError::internal_error("BlobsConsistency", format!("stream read: {e}"))
|
|
||||||
})?;
|
|
||||||
hasher.update(&bytes);
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(hasher.finalize().to_hex().to_string())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
@@ -865,7 +499,6 @@ mod tests {
|
|||||||
b.hash AS hash,
|
b.hash AS hash,
|
||||||
b.size AS size,
|
b.size AS size,
|
||||||
b.ref_count AS ref_count,
|
b.ref_count AS ref_count,
|
||||||
b.created_at AS created_at,
|
|
||||||
((SELECT COUNT(*) FROM storage.files cnt_f
|
((SELECT COUNT(*) FROM storage.files cnt_f
|
||||||
WHERE cnt_f.blob_hash = b.hash
|
WHERE cnt_f.blob_hash = b.hash
|
||||||
AND NOT EXISTS (
|
AND NOT EXISTS (
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ pub mod azure_blob_backend;
|
|||||||
pub mod backend_consistency_service;
|
pub mod backend_consistency_service;
|
||||||
pub mod backend_migration_service;
|
pub mod backend_migration_service;
|
||||||
pub mod backend_rotate_service;
|
pub mod backend_rotate_service;
|
||||||
|
pub mod blob_diagnostics;
|
||||||
pub mod blobs_consistency_service;
|
pub mod blobs_consistency_service;
|
||||||
pub mod cached_blob_backend;
|
pub mod cached_blob_backend;
|
||||||
pub mod chunked_upload_service;
|
pub mod chunked_upload_service;
|
||||||
|
|||||||
Reference in New Issue
Block a user