feat(consistency): reconcile chunk_manifests.ref_count
Step 3 (prerequisite 2) of docs/plan/derived-blobs.md, and the last one
before the thumbnail slice.
There are two reference counters and only one was ever verified.
add_reference bumps chunk_manifests.ref_count first and only falls back
to storage.blobs.ref_count, so a reference lands on whichever counter
its hash names: chunk references feed storage.blobs and are reconciled
by blobs_consistency::refcount_mismatch, while Blob references — every
CDC file, and every derived artifact once those exist — feed
chunk_manifests.ref_count, which nothing reconciled.
That gap was survivable only because dedup_gc's reap predicate carried a
second clause ("no storage.files row references this manifest") that
quietly compensated for drift on the bulk-delete paths where ref_count
is never decremented. Generalising that clause to the reference registry
in 1c8ead49 — so thumbnails stop being reaped — removed the
compensation, which is precisely why the counter now needs checking
directly. The two changes have to ship together.
Adds manifests_consistency, a recoverable job reporting
manifest_refcount_mismatch (severity inconsistent). The finding carries
reap_risk so an operator can triage: an under-count means GC reaps a
manifest whose content is still reachable, taking its chunks with it,
while an over-count merely pins storage.
A separate job rather than a second phase of blobs_consistency: one
subject per job, as the other five consistency tenants do, and it avoids
changing the cursor format of an existing recoverable job — which would
strand any run paused across the deploy.
The page query is assembled from the same registry dedup_gc reaps from
(via DedupService::reference_registry), built once at construction, and
pinned by a golden test. Two invariants the test guards: the files term
carries no NOT EXISTS guard — that guard keeps CDC rows out of the
*chunk* level and here would count nothing — and chunk_hashes appears
nowhere, since a manifest citing its own chunks is not a referrer of
itself.
fmt, clippy --all-features --all-targets, and 3 new unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1477,6 +1477,22 @@ impl AppServiceFactory {
|
||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||
.await;
|
||||
|
||||
// Reconciles `chunk_manifests.ref_count` — the SECOND reference
|
||||
// counter, and the one nothing verified before. add_reference bumps
|
||||
// it first and only falls back to storage.blobs.ref_count, so every
|
||||
// CDC file (and every derived artifact, once those land) counts here
|
||||
// rather than at the chunk level. Uses the same registry dedup_gc
|
||||
// reaps from, so the two cannot disagree.
|
||||
// See docs/plan/derived-blobs.md.
|
||||
let _ = Arc::new(
|
||||
crate::infrastructure::services::manifests_consistency_service::ManifestsConsistencyCheck::new(
|
||||
maintenance_pool.clone(),
|
||||
core.dedup_service.reference_registry(),
|
||||
),
|
||||
)
|
||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||
.await;
|
||||
|
||||
// Third recoverable-run tenant. Iterates `storage.files`
|
||||
// and reports parent-folder-trashed cascade misses,
|
||||
// `missing_blob` (data-loss indicator — file references
|
||||
|
||||
Reference in New Issue
Block a user