Edouard Vanbelle
f1f327a6c4
`blobs_consistency` probed `blob_exists` once per row and, under `?deep=true`, read and re-hashed every blob. `backend_consistency` already reports the same `blob_missing_from_backend` from its merge-join — so the probe was duplicated work that found strictly less (a DB walk cannot see backend-only orphans by construction) at N round -trips instead of one enumeration. Every scheduled sweep paid for it. All three physical checks move to `backend_consistency`: * `blob_missing_from_backend` was already there; the duplicate is gone. * `blob_corrupted` / `blob_unreadable` hook the matched arm of the merge-join, which holds exactly the key pairs worth reading. Guarded by `in_range` so a pair past the horizon is not read twice, and `params.deep` is persisted on a fresh run and read back on resume so a paused deep scan does not silently continue shallow. Deep mode belongs there because it is backend work end to end: the only DB input is the hash. Keeping it in `blobs_consistency` forced that tenant to carry a backend for one flag. What remains is the half that needs no backend: `refcount_mismatch` and its repair. The constructor drops from five parameters to two — no backend, no storage_entries, no storage_path_fallback — and `?storage=<name>` / `?deep=true` are now inert there, which the job description says outright. `affected_files` is needed by both tenants, so it moves to a shared `blob_diagnostics` module rather than being copied. `PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined in `blobs_consistency` and re-exported, which is backwards once the DB-only tenant has no entry to scope. The create-grace window goes with the probe — it existed to avoid flagging a blob whose bytes had landed before its row, and the refcount comparison reads one consistent snapshot. Known cost: `backend_consistency` returns `backend_unenumerable` on Azure and mid-migration, so on those configs missing bytes now go unreported where the per-row probe caught them. That argues for the Azure enumeration impl, not for keeping the probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
47 lines
1.8 KiB
Rust
47 lines
1.8 KiB
Rust
//! Reverse-lookup helpers shared by the storage consistency tenants.
|
|
//!
|
|
//! `blobs_consistency` (DB-side: refcount drift) and
|
|
//! `backend_consistency` (backend-side: missing / orphaned / corrupted
|
|
//! bytes) both answer the same operator question when they emit a
|
|
//! finding — *which files does this hash break?* — so the query lives
|
|
//! here rather than in whichever tenant happened to need it first.
|
|
|
|
use sqlx::PgPool;
|
|
|
|
/// Cap on reverse-lookup file names surfaced in a finding's detail.
|
|
/// Keeps detail JSON bounded when a broken blob is referenced by
|
|
/// hundreds of files.
|
|
const AFFECTED_FILES_SAMPLE: i64 = 5;
|
|
|
|
/// Sample of file names that reference this blob — either directly
|
|
/// (`files.blob_hash = $hash`, legacy pre-CDC) or transitively via a
|
|
/// manifest (`chunk_hashes @> ARRAY[$hash]`, the post-CDC dominant
|
|
/// path). Capped so a chunk shared by 10 000 files doesn't blow up the
|
|
/// finding detail JSON. Order is arbitrary — this samples for
|
|
/// diagnosis, it does not enumerate.
|
|
///
|
|
/// Returns an empty vec on query error: a finding with no sample is
|
|
/// still a finding, and failing the sweep because the diagnostic
|
|
/// garnish didn't load would trade the whole scan for a nicety.
|
|
pub(crate) async fn affected_files(pool: &PgPool, hash: &str) -> Vec<String> {
|
|
let rows: Vec<(String,)> = sqlx::query_as(
|
|
r#"
|
|
SELECT DISTINCT f.name
|
|
FROM storage.files f
|
|
WHERE f.blob_hash = $1
|
|
OR EXISTS (
|
|
SELECT 1 FROM storage.chunk_manifests m
|
|
WHERE m.file_hash = f.blob_hash
|
|
AND $1 = ANY(m.chunk_hashes)
|
|
)
|
|
LIMIT $2
|
|
"#,
|
|
)
|
|
.bind(hash)
|
|
.bind(AFFECTED_FILES_SAMPLE)
|
|
.fetch_all(pool)
|
|
.await
|
|
.unwrap_or_default();
|
|
rows.into_iter().map(|(n,)| n).collect()
|
|
}
|