feat(storage): add BlobReferenceSource port + registry

Step 1 of docs/plan/derived-blobs.md. Makes "who references this blob
hash" an extension point instead of SQL hardcoded in two places
(dedup_gc's reap predicate and blobs_consistency's refcount recompute,
both naming storage.files and storage.chunk_manifests directly). Adding
a blob-owning table without teaching those two risks silent orphaning:
GC sees ref_count = 0 and reaps live content.

Behaviour is unchanged — this commit only introduces the port and the
two sources that reproduce today's SQL. Wiring follows.

Two levels, not one. add_reference bumps chunk_manifests.ref_count first
and only falls back to storage.blobs.ref_count, so a reference lands on
whichever counter its hash names and the two must be recomputed
separately. RefLevel is a parameter rather than a property of a source,
because storage.files legitimately contributes at both: a manifest-less
legacy row references a chunk, a CDC row references a Blob. The
NOT EXISTS guard on the chunk-level files term is load-bearing — for a
single-chunk file the whole-file hash equals its lone chunk's hash, so
without it the row is counted at both levels.

SQL fragments rather than a per-hash count. blobs_consistency recomputes
with one query per page, the expected count inlined as correlated
subqueries; asking each source for a count per hash would turn that into
sources x rows round-trips. So sources emit a fragment the registry sums
into the existing page query, and count_references exists only for the
on-demand path where the candidate set is already filtered to
ref_count = 0.

Fragments use their own aliases (cnt_f, cnt_m) rather than the sweeps'
outer-row aliases (b, m). A fragment reusing `m` would shadow the outer
alias in the manifest sweep and silently correlate against itself;
there is a test for it.

The SQL builders are free functions so the shape can be asserted without
constructing a pool — sqlx's connect_lazy still needs a Tokio context,
and the fragments are pure string assembly anyway.

9 unit tests. fmt, clippy --all-features --all-targets, build clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-08-23 14:33:27 +02:00
parent 07bb38eacb
commit a708389d19
4 changed files with 608 additions and 0 deletions
@@ -0,0 +1,278 @@
//! `BlobReferenceSource` — the extension point that teaches ref-counting
//! and the consistency jobs about a table holding blob references.
//!
//! Before this port, "who references this hash" was hardcoded SQL in two
//! places (`dedup_gc`'s reap predicate and `blobs_consistency`'s refcount
//! recompute), both naming `storage.files` and `storage.chunk_manifests`
//! directly. Any new blob-owning table therefore risked silent orphaning:
//! `dedup_gc` sees `ref_count = 0`, or a manifest with no `storage.files`
//! row behind it, and reaps live content.
//!
//! See `docs/plan/derived-blobs.md` for the design and the coverage matrix.
//!
//! # Two levels, and why a source may span both
//!
//! [`DedupService::add_reference`] bumps `chunk_manifests.ref_count` first
//! and only falls back to `storage.blobs.ref_count`. So a reference lands
//! on whichever counter its hash names, and the two must be recomputed
//! separately — mixing them double-counts, systematically:
//!
//! * A **Blob** (`chunk_manifests.file_hash`) is "the content of a file".
//! * A **Chunk** (`storage.blobs.hash`) is a physical byte payload.
//! * For a single-chunk Blob the two hashes are **equal**, because both are
//! BLAKE3 over the same bytes. That aliasing is why today's chunk-level
//! recompute carries a `NOT EXISTS` clause, and why every fragment here
//! must be level-correct rather than merely plausible.
//!
//! A source is not confined to one level: [`RefLevel::Chunk`] and
//! [`RefLevel::Manifest`] fragments are requested independently, and
//! `storage.files` legitimately contributes to both — a manifest-less
//! legacy row references a chunk, a CDC row references a Blob.
//!
//! # Why SQL fragments rather than a per-hash count
//!
//! `blobs_consistency` recomputes refcounts with **one query per page**,
//! the expected count inlined as correlated subqueries. Asking each source
//! for a count per hash would turn that into `sources × rows` round-trips —
//! a catastrophic regression on a table with millions of rows. So sources
//! contribute a *fragment* that the registry sums into the existing page
//! query, and [`BlobReferenceSource::count_references`] exists only for the
//! on-demand path (`dedup_gc` checking a single reap candidate, where the
//! candidate set is already filtered to `ref_count = 0`).
use std::sync::Arc;
use async_trait::async_trait;
use crate::domain::errors::DomainError;
/// Which counter a source's references land on.
///
/// Not a property of the source — see the module docs; the same source may
/// contribute at both levels.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum RefLevel {
/// References a physical chunk. Feeds `storage.blobs.ref_count`.
Chunk,
/// References a Blob via its manifest. Feeds
/// `chunk_manifests.ref_count`.
Manifest,
}
impl RefLevel {
/// Both levels, for callers that sweep each in turn.
pub const ALL: [RefLevel; 2] = [RefLevel::Chunk, RefLevel::Manifest];
/// Stable name for logs and consistency-finding fields.
pub fn as_str(self) -> &'static str {
match self {
RefLevel::Chunk => "chunk",
RefLevel::Manifest => "manifest",
}
}
}
/// One table that holds references to blob hashes.
///
/// Implementors are registered on [`BlobReferenceRegistry`] during DI.
/// Adding a blob-owning table **without** registering it is the failure
/// this port exists to prevent.
#[async_trait]
pub trait BlobReferenceSource: Send + Sync {
/// Short stable identifier for logs and consistency-finding `source`
/// fields — `"files"`, `"chunks"`, `"content_derived"`, …
///
/// Stable across releases: log aggregators key off it.
fn source_name(&self) -> &'static str;
/// A correlated-subquery fragment counting this source's references
/// **at `level`** to `outer_hash_expr`, or `None` when this source
/// holds no references at that level.
///
/// `outer_hash_expr` is the SQL expression naming the hash of the row
/// being recomputed — `"b.hash"` when sweeping `storage.blobs`,
/// `"m.file_hash"` when sweeping `storage.chunk_manifests`. The
/// fragment must be a parenthesised scalar subquery so the registry can
/// join fragments with `+`.
///
/// **Identifiers only.** `outer_hash_expr` is supplied by the sweep, never
/// by a request; no fragment may interpolate caller input.
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String>;
/// Count of references this source holds on `blob_hash`, across both
/// levels.
///
/// **On-demand path only** — `dedup_gc` checking a single reap
/// candidate. The consistency sweeps must use [`Self::ref_count_sql`];
/// calling this per row would turn one query per page into
/// `sources × rows` round-trips.
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;
/// Iterate the hashes this source references, paged by the
/// implementation's natural cursor (typically a primary key).
///
/// Used by `backend_consistency` to walk the backend against the union
/// of all sources. Returns the page plus the cursor to resume from,
/// `None` when exhausted.
async fn list_referenced_blobs(
&self,
cursor: Option<Vec<u8>>,
limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;
/// Notification that `dedup_gc` reaped this blob.
///
/// Sources maintaining a denormalised refcount can clean up here. Most
/// leave the default noop — the mapping row is normally deleted by the
/// owning service's `on_blob_deleted` hook instead.
fn on_blob_reaped(&self, _blob_hash: &str) {}
}
/// The set of registered [`BlobReferenceSource`]s.
///
/// Assembled once during DI and shared (`Arc`) by `dedup_gc` and the
/// consistency jobs, so all three agree on what "referenced" means.
#[derive(Default)]
pub struct BlobReferenceRegistry {
sources: Vec<Arc<dyn BlobReferenceSource>>,
}
impl BlobReferenceRegistry {
pub fn new() -> Self {
Self::default()
}
/// Register a source. Order is irrelevant — fragments are summed and
/// counts added.
pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>) {
self.sources.push(source);
}
pub fn sources(&self) -> &[Arc<dyn BlobReferenceSource>] {
&self.sources
}
/// The summed SQL expression counting every source's references at
/// `level` to `outer_hash_expr`.
///
/// Returns `"0"` when no source contributes at this level, which keeps
/// the caller's query valid without a special case.
pub fn ref_count_expr(&self, level: RefLevel, outer_hash_expr: &str) -> String {
let fragments: Vec<String> = self
.sources
.iter()
.filter_map(|s| s.ref_count_sql(level, outer_hash_expr))
.collect();
if fragments.is_empty() {
"0".to_string()
} else {
fragments.join("\n + ")
}
}
/// Total references held on `hash` across every source.
///
/// On-demand path only — see [`BlobReferenceSource::count_references`].
pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError> {
let mut total = 0u64;
for source in &self.sources {
total = total.saturating_add(source.count_references(hash).await?);
}
Ok(total)
}
/// Fan out a reap notification to every source.
pub fn notify_reaped(&self, blob_hash: &str) {
for source in &self.sources {
source.on_blob_reaped(blob_hash);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
struct Stub {
name: &'static str,
chunk: Option<&'static str>,
manifest: Option<&'static str>,
count: u64,
}
#[async_trait]
impl BlobReferenceSource for Stub {
fn source_name(&self) -> &'static str {
self.name
}
fn ref_count_sql(&self, level: RefLevel, outer: &str) -> Option<String> {
let tmpl = match level {
RefLevel::Chunk => self.chunk?,
RefLevel::Manifest => self.manifest?,
};
Some(tmpl.replace("{outer}", outer))
}
async fn count_references(&self, _blob_hash: &str) -> Result<u64, DomainError> {
Ok(self.count)
}
async fn list_referenced_blobs(
&self,
_cursor: Option<Vec<u8>>,
_limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
Ok((Vec::new(), None))
}
}
fn registry() -> BlobReferenceRegistry {
let mut r = BlobReferenceRegistry::new();
r.register(Arc::new(Stub {
name: "a",
chunk: Some("(SELECT 1 WHERE {outer} = 'x')"),
manifest: None,
count: 2,
}));
r.register(Arc::new(Stub {
name: "b",
chunk: Some("(SELECT 2 WHERE {outer} = 'y')"),
manifest: Some("(SELECT 3 WHERE {outer} = 'z')"),
count: 5,
}));
r
}
#[test]
fn chunk_level_sums_every_contributing_source() {
let expr = registry().ref_count_expr(RefLevel::Chunk, "b.hash");
assert!(expr.contains("b.hash = 'x'"), "{expr}");
assert!(expr.contains("b.hash = 'y'"), "{expr}");
assert!(expr.contains('+'), "fragments must be summed: {expr}");
}
/// A source returning `None` for a level must contribute nothing there —
/// this is what keeps manifest-only tables out of the chunk recompute,
/// where they would double-count against the single-chunk hash alias.
#[test]
fn manifest_level_skips_non_contributing_sources() {
let expr = registry().ref_count_expr(RefLevel::Manifest, "m.file_hash");
assert!(expr.contains("m.file_hash = 'z'"), "{expr}");
assert!(!expr.contains('+'), "only one source contributes: {expr}");
}
/// An empty level must still yield a valid scalar expression, so callers
/// need no special case before a registry is fully populated.
#[test]
fn empty_level_yields_zero_literal() {
let r = BlobReferenceRegistry::new();
assert_eq!(r.ref_count_expr(RefLevel::Chunk, "b.hash"), "0");
}
#[tokio::test]
async fn total_references_adds_across_sources() {
assert_eq!(registry().total_references("deadbeef").await.unwrap(), 7);
}
}
+1
View File
@@ -1,6 +1,7 @@
pub mod auth_ports; pub mod auth_ports;
pub mod authorization_ports; pub mod authorization_ports;
pub mod blob_lifecycle; pub mod blob_lifecycle;
pub mod blob_reference_ports;
pub mod blob_storage_ports; pub mod blob_storage_ports;
pub mod cache_ports; pub mod cache_ports;
pub mod calendar_ports; pub mod calendar_ports;
@@ -0,0 +1,328 @@
//! The two implicit blob-reference sources, made explicit.
//!
//! Before this module, "who references this hash" lived as hardcoded SQL
//! inside `blobs_consistency`'s refcount recompute and `dedup_gc`'s reap
//! predicate. These two implementations reproduce that SQL **exactly** —
//! the fragments below sum to today's `actual_ref_count` expression — so
//! the registry can be wired in without changing any observed count.
//!
//! See `docs/plan/derived-blobs.md` and
//! [`crate::application::ports::blob_reference_ports`].
use std::sync::Arc;
use async_trait::async_trait;
use sqlx::{PgPool, Row};
use uuid::Uuid;
use crate::application::ports::blob_reference_ports::{BlobReferenceSource, RefLevel};
use crate::domain::errors::DomainError;
/// Aliases used inside the emitted fragments.
///
/// Deliberately distinct from the aliases the sweeps use for their outer
/// row (`b` for `storage.blobs`, `m` for `storage.chunk_manifests`): a
/// fragment reusing `m` would shadow the outer alias in the manifest-level
/// sweep and silently correlate against itself.
const FILES_ALIAS: &str = "cnt_f";
const MANIFEST_ALIAS: &str = "cnt_m";
/// Fragment for [`FilesReferenceSource`], as a free function so the SQL
/// shape can be tested without constructing a pool — it is a property of
/// the module, not of an instance.
fn files_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
let f = FILES_ALIAS;
match level {
// Legacy whole-file blobs only — CDC files are counted at the
// manifest level, and counting them here too would double up on the
// single-chunk hash alias.
RefLevel::Chunk => Some(format!(
"(SELECT COUNT(*) FROM storage.files {f}
WHERE {f}.blob_hash = {outer_hash_expr}
AND NOT EXISTS (
SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS}
WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash
))"
)),
RefLevel::Manifest => Some(format!(
"(SELECT COUNT(*) FROM storage.files {f}
WHERE {f}.blob_hash = {outer_hash_expr})"
)),
}
}
/// Fragment for [`ChunksReferenceSource`]. See [`files_ref_sql`].
fn chunks_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
match level {
RefLevel::Chunk => {
let m = MANIFEST_ALIAS;
Some(format!(
"(SELECT COUNT(*) FROM storage.chunk_manifests {m}
WHERE {outer_hash_expr} = ANY({m}.chunk_hashes))"
))
}
// A manifest is never referenced by another manifest.
RefLevel::Manifest => None,
}
}
// ─── storage.files ───────────────────────────────────────────────────────
/// References held by `storage.files.blob_hash`.
///
/// Contributes at **both** levels, which is why `RefLevel` is a parameter
/// rather than a property of the source:
///
/// * [`RefLevel::Manifest`] — a CDC file's `blob_hash` names a manifest.
/// * [`RefLevel::Chunk`] — a pre-CDC legacy file, whose `blob_hash` names a
/// whole-file blob with no manifest behind it. The `NOT EXISTS` guard is
/// load-bearing: for a single-chunk file the whole-file hash *equals* its
/// lone chunk's hash, so without it the row would be counted at both
/// levels.
pub struct FilesReferenceSource {
pool: Arc<PgPool>,
}
impl FilesReferenceSource {
pub fn new(pool: Arc<PgPool>) -> Self {
Self { pool }
}
}
#[async_trait]
impl BlobReferenceSource for FilesReferenceSource {
fn source_name(&self) -> &'static str {
"files"
}
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
files_ref_sql(level, outer_hash_expr)
}
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
// No level split here: the question is "how many file rows name this
// exact hash", and a hash names either a manifest or a legacy blob,
// never both at once from the caller's point of view.
let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.files WHERE blob_hash = $1")
.bind(blob_hash)
.fetch_one(self.pool.as_ref())
.await
.map_err(|e| {
DomainError::internal_error("BlobRefSource", format!("files count: {e}"))
})?;
Ok(n.max(0) as u64)
}
async fn list_referenced_blobs(
&self,
cursor: Option<Vec<u8>>,
limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
// Paged by the file's own PK so the cursor is stable under concurrent
// inserts; `blob_hash` is not unique and would skip or repeat rows.
let after: Option<Uuid> = match cursor {
Some(bytes) => Some(decode_uuid_cursor(&bytes)?),
None => None,
};
let rows = sqlx::query(
"SELECT id, blob_hash FROM storage.files
WHERE ($1::uuid IS NULL OR id > $1)
ORDER BY id
LIMIT $2",
)
.bind(after)
.bind(limit as i64)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("files page: {e}")))?;
let next = rows
.last()
.map(|r| r.get::<Uuid, _>("id").as_bytes().to_vec())
.filter(|_| rows.len() == limit);
let hashes = rows
.iter()
.map(|r| r.get::<String, _>("blob_hash"))
.collect();
Ok((hashes, next))
}
}
// ─── storage.chunk_manifests ─────────────────────────────────────────────
/// References held by `storage.chunk_manifests.chunk_hashes[]`.
///
/// Chunk level only — a manifest never references another manifest, so
/// [`RefLevel::Manifest`] yields `None` and this source contributes nothing
/// to the manifest recompute.
pub struct ChunksReferenceSource {
pool: Arc<PgPool>,
}
impl ChunksReferenceSource {
pub fn new(pool: Arc<PgPool>) -> Self {
Self { pool }
}
}
#[async_trait]
impl BlobReferenceSource for ChunksReferenceSource {
fn source_name(&self) -> &'static str {
"chunks"
}
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
chunks_ref_sql(level, outer_hash_expr)
}
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
let n: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM storage.chunk_manifests WHERE $1 = ANY(chunk_hashes)",
)
.bind(blob_hash)
.fetch_one(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks count: {e}")))?;
Ok(n.max(0) as u64)
}
async fn list_referenced_blobs(
&self,
cursor: Option<Vec<u8>>,
limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
// Paged by the manifest PK, not by the unnested chunk hash: a single
// manifest expands to many hashes, so the page boundary has to fall
// between manifests or the cursor cannot be resumed unambiguously.
let after: Option<String> = match cursor {
Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| {
DomainError::internal_error("BlobRefSource", format!("bad chunk cursor: {e}"))
})?),
None => None,
};
let rows = sqlx::query(
"SELECT file_hash, chunk_hashes FROM storage.chunk_manifests
WHERE ($1::text IS NULL OR file_hash > $1)
ORDER BY file_hash
LIMIT $2",
)
.bind(after)
.bind(limit as i64)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks page: {e}")))?;
let next = rows
.last()
.map(|r| r.get::<String, _>("file_hash").into_bytes())
.filter(|_| rows.len() == limit);
let hashes = rows
.iter()
.flat_map(|r| r.get::<Vec<String>, _>("chunk_hashes"))
.collect();
Ok((hashes, next))
}
}
fn decode_uuid_cursor(bytes: &[u8]) -> Result<Uuid, DomainError> {
let raw: [u8; 16] = bytes.try_into().map_err(|_| {
DomainError::internal_error(
"BlobRefSource",
format!("bad uuid cursor: expected 16 bytes, got {}", bytes.len()),
)
})?;
Ok(Uuid::from_bytes(raw))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::application::ports::blob_reference_ports::BlobReferenceRegistry;
/// The registry sums whatever the sources emit; these helpers exercise the
/// same code path without needing a pool, since `ref_count_sql` is pure.
fn summed(level: RefLevel, outer: &str) -> String {
let frags: Vec<String> = [files_ref_sql(level, outer), chunks_ref_sql(level, outer)]
.into_iter()
.flatten()
.collect();
if frags.is_empty() {
"0".to_string()
} else {
frags.join("\n + ")
}
}
/// The chunk-level expression must reproduce the two terms
/// `blobs_consistency` inlines today: legacy-only files (guarded by
/// NOT EXISTS) plus manifests citing the chunk.
#[test]
fn chunk_level_reproduces_todays_two_terms() {
let expr = summed(RefLevel::Chunk, "b.hash");
assert!(expr.contains("storage.files"), "{expr}");
assert!(
expr.contains("NOT EXISTS"),
"legacy term must keep the CDC guard: {expr}"
);
assert!(
expr.contains("= ANY(cnt_m.chunk_hashes)"),
"chunk term missing: {expr}"
);
assert!(expr.contains("b.hash"), "must correlate on the outer row");
assert!(expr.contains('+'), "both terms must be summed: {expr}");
}
/// Only `storage.files` references a manifest, so the manifest-level
/// expression is the single files term with no `NOT EXISTS` guard — the
/// guard exists to keep CDC rows *out* of the chunk level, and applying
/// it here would count nothing at all.
#[test]
fn manifest_level_is_files_only_and_unguarded() {
let expr = summed(RefLevel::Manifest, "m.file_hash");
assert!(expr.contains("storage.files"), "{expr}");
assert!(!expr.contains("NOT EXISTS"), "{expr}");
assert!(
!expr.contains("chunk_hashes"),
"chunks must not contribute at manifest level: {expr}"
);
assert!(!expr.contains('+'), "only one source contributes: {expr}");
assert!(expr.contains("m.file_hash"));
}
/// Fragments must not use the aliases the sweeps use for their outer row
/// (`b` for storage.blobs, `m` for chunk_manifests), or the manifest sweep
/// would shadow its own alias and silently correlate against itself.
#[test]
fn fragments_avoid_outer_row_aliases() {
for level in RefLevel::ALL {
let expr = summed(level, "m.file_hash");
for bad in [
"storage.files f",
"storage.files b",
"chunk_manifests m ",
"chunk_manifests b",
] {
assert!(!expr.contains(bad), "alias collision at {level:?}: {expr}");
}
}
}
/// A source declining a level must drop out of the sum entirely, which is
/// what keeps manifest-only tables out of the chunk recompute where the
/// single-chunk hash alias would double-count them.
#[test]
fn chunks_source_declines_manifest_level() {
assert!(chunks_ref_sql(RefLevel::Manifest, "m.file_hash").is_none());
assert!(chunks_ref_sql(RefLevel::Chunk, "b.hash").is_some());
}
/// Guards the registry contract the sweeps rely on: an empty level still
/// yields a valid scalar expression.
#[test]
fn empty_registry_yields_zero_literal() {
let r = BlobReferenceRegistry::new();
assert_eq!(r.ref_count_expr(RefLevel::Manifest, "m.file_hash"), "0");
}
}
@@ -1,5 +1,6 @@
mod address_book_pg_repository; mod address_book_pg_repository;
mod app_password_pg_repository; mod app_password_pg_repository;
pub mod blob_reference_sources;
mod calendar_event_pg_repository; mod calendar_event_pg_repository;
mod calendar_pg_repository; mod calendar_pg_repository;
mod contact_group_pg_repository; mod contact_group_pg_repository;