Oxicloud/docs/plan/consistency-check.md

Plan — Resumable consistency checks + StatefulAdapter contract

⚠️ PARTIALLY SUPERSEDED (Ed 2026-07-28). The current shipping design organises consistency checks by the subject they iterate (drives / folders / files / storage), NOT by concern (blob / thumbnail / used_bytes). Each *_consistency job is a direct RecoverableJobHandler impl on the Part 2 engine — no ConsistencyCheck trait, no StatefulAdapter supertrait, no per- subsystem check registry. Cursor = row PK of the iterated subject.

See instead:

• Memory: project_consistency_jobs_landscape — the current taxonomy.
• docs/plan/job-registry.md Part 2 §Native tenants — updated table.
• docs/architecture/jobs.md — implementor guide.

Sections below discuss BlobConsistencyCheck, ThumbnailConsistencyCheck, UsedBytesConsistencyCheck etc. as separate impls of a ConsistencyCheck trait. That IS retired. Read those sections for the invariants (grace-window trap, cursor discipline, findings idempotency) — they still apply. Ignore the trait shapes / registration wiring — the Part 2 engine covers those uniformly.

Context

OxiCloud persists state in several independent subsystems: content-addressable blobs on disk / S3, thumbnails (server-generated per blob + user-uploaded per file), text-extraction cache, audio metadata cache, storage.folders + file_metadata, storage.trash, storage.drives.used_bytes, WebDAV dead properties, and more. Each has invariants that can silently drift:

• A blob on disk with no file_blobs reference (leak — wasted disk).
• A file_blobs row whose bytes are gone from storage (data loss — GET returns 500).
• A drives.used_bytes counter that no longer matches SUM(size).
• A folders.parent_id pointing at a deleted row (historical raw-SQL fix).
• A thumbnail cache entry with no live file id (leak) or a missing entry the user actually uploaded (data loss).

Today the only "consistency" primitives are targeted point solutions — dedup_service GC (orphan blob reap with 1 h grace), storage_usage_service reconciliation (rebuild used_bytes from SUM(size)), and the trash cleaner. None of them SURFACE inconsistencies for operators; they act blindly and best-effort. tests/api/storage_cleanup_check.sh polls with a 5 s window and races the 1 h GC grace (memory note project_dedup_gc_test_trigger).

At scale — Ed's example: 1000 users × ~1000 files each = 1M files — an ad-hoc "is my disk usage accurate?" check must be resumable across restarts, cooperative on cancellation, and non-blocking to live traffic. A batch job that starts over from scratch after a container restart or SIGTERM never completes.

This plan lands:

1. Two traits — ConsistencyCheck (one check) and StatefulAdapter (marker + registration on every state-owning port).
2. A contract every new adapter must satisfy at compile time — via a supertrait bound on existing state-owning ports, no new adapter compiles without declaring its consistency contract.
3. An educational surface in trait doc-comments — decision axes (severity, direction, grace, cursor) and canonical-example pointers.
4. Consistency-specific persistence — jobs.run_findings, idempotent-on-(run_id, kind, resource_id).
5. A first check — BlobConsistencyCheck (both directions, blob-keyed cursor, severity split).

Layer boundary — the runtime is not in this plan. The resumable execution engine (cursor persistence, exclusivity, cancel protocol, crash recovery, jobs.recoverable_runs schema, JobStore, RunOutcome, run_or_resume) lives in docs/plan/job-registry.md Part 2. This plan describes what ConsistencyCheck implementors write and how the check-specific bits (findings, severity, StatefulAdapter supertrait wiring) compose on top of that engine.

Order: ships after the job-registry Part 2 engine lands. Consistency closes an operator-visibility gap today, but it depends on Part 2's RecoverableJobHandler + JobStore + jobs.recoverable_runs primitives — those come first. Once both are in, consistency runs are admin-triggered v1, becoming periodic-triggered when a JobRegistry (Part 1) tenant wraps run_or_resume for each ConsistencyCheck.

Design decisions

Two-trait split

ConsistencyCheck = one check (implements run_resumable). StatefulAdapter = a subsystem that CONTRIBUTES checks (one or more).

This split is load-bearing:

• Some subsystems emit multiple checks (ThumbnailStore emits four — server-generated × 2 directions, user-uploaded × 2 directions).
• Some checks are composed across adapters (UsedBytesConsistencyCheck reads from both FileMetadataRepository and DriveRepository).

Bundling them into one trait would over-constrain the shape.

Compile-time enforcement via supertrait bound

StatefulAdapter is added as a supertrait on every port that persists state:

pub trait BlobStorage: StatefulAdapter { … }
pub trait ThumbnailStore: StatefulAdapter { … }
pub trait FileBlobReadRepository: StatefulAdapter { … }
pub trait FolderRepository: StatefulAdapter { … }

Any new impl of these ports — a new S3-alike backend, a new mock in tests, a plugin-provided storage backend — will not compile without providing subsystem() and consistency_checks(). The compiler is the enforcement; reviewers cannot merge a stateful adapter without an answer to "what can go wrong with this state, and how do you check it?"

The severity axis

Every finding carries a Severity so the admin UI can order results and operators can dismiss the low-impact ones without hiding real risk.

Severity	Meaning	Examples
DataLoss	User-visible impact (500 on GET, missing user bytes)	Missing blob for a live file_blobs row; missing user-uploaded thumbnail
Reclaimable	Disk waste, no user impact	Orphan blob on storage, orphan thumbnail file
Regenerable	Auto-heals on next request	Missing server-generated thumbnail (server rebuilds), missing text-index row
Drift	Accounting mismatch, no user impact	drives.used_bytes vs SUM(size)

Rule of thumb: if a human user notices, it's DataLoss. If only the disk accountant notices, it's Reclaimable or Drift. If the next automatic regeneration will fix it, it's Regenerable.

Bidirectional in every check

Every check emits BOTH directions where they exist:

• Backward (storage → DB) — orphan detection. Wasted disk. Reclaimable.
• Forward (DB → storage) — missing detection. User-visible data loss. DataLoss. Higher severity — a single missing content-addressable blob silently breaks every file that referenced it.

Skipping the forward direction is the single most common consistency-check mistake. It's easy because Pass 1 (list storage, cross-check DB) LOOKS complete. Pass 2 (list DB, cross-check storage) is where data-loss surfaces.

Report shape for missing findings — blob-level, not file-level

MissingInStorage { blob_hash, ref_count, affected_file_ids: Vec<Uuid> }. One row per missing blob with the fan-out of broken files. Operator gets a triage-ordered "biggest impact first" list. File-per-line reports lose that ordering.

Two-pass discipline eliminates the need for maintenance mode

Pass 1 — build candidate list from a snapshot read (storage listing for orphan direction, DB SELECT for missing direction). Exclude anything younger than grace_window.

Pass 2 — per candidate, re-read the OTHER side's state right before flagging. If it transitioned (ref went up, blob just landed, row was deleted, etc.), silently drop.

Race matrix — orphan direction:

• Upload lands mid-scan (dedup hit → ref_count ↑ after we sampled) — grace window skips young objects.
• Last ref deleted mid-scan (ref_count → 0, GC not yet) — cross-reference blobs.orphaned_at; expected transient state, not flagged.
• Deep hash on partial upload — deep mode runs only on rows older than a LONGER grace (24 h).

Race matrix — missing direction:

• Blob written but DB row not yet inserted (young file looks missing at flag time) — grace window skips DB rows younger than 1 h.
• File deleted mid-scan — Pass 2 re-reads file_metadata by id; if gone, drop the finding.
• Blob just now landed — Pass 2 re-verifies storage HEAD; if now present, drop.

Nothing before "byte-exact whole-table snapshot verification" needs a quiescent server. Reserve concurrent_safe() = false for that one.

Resumability — runs live in Part 2's recoverable_runs

Consistency runs are ordinary RecoverableJobHandlers. The runtime plumbing — cursor persistence, exclusivity, cancel protocol, crash recovery, jobs.recoverable_runs schema, JobStore trait, RunOutcome, run_or_resume helper — lives in docs/plan/job-registry.md Part 2. This plan does not redefine any of it.

How consistency slots into the shared table:

• Each consistency check registers under job_name = 'consistency_<check_name>' (e.g. 'consistency_blobs', 'consistency_thumbnails'). Naming convention lets a single SELECT DISTINCT ON (job_name) return the last run of every check alongside every other background job.
• Consistency's per-check knobs — grace_window_secs, batch_size, concurrent_safe — live inside recoverable_runs.params JSONB at run-start time. The check reads them back via serde_json::from_value(store.params()?).
• recoverable_runs.stats accumulates {"scanned_count": …, "findings_this_run": …}; readers call (stats->>'scanned_count')::bigint.

The findings themselves are Layer C (this plan) — they don't generalise to storage-migration or reextract:

CREATE TABLE jobs.run_findings (
    id             UUID PRIMARY KEY,
    run_id         UUID NOT NULL REFERENCES jobs.recoverable_runs(id) ON DELETE CASCADE,
    kind           TEXT NOT NULL,             -- OrphanBlob / MissingBlob / ...
    severity       TEXT NOT NULL,             -- DataLoss / Reclaimable / ...
    resource_id    TEXT NOT NULL,
    detail         JSONB,
    found_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (run_id, kind, resource_id)        -- idempotent re-scan on resume
);
CREATE INDEX ON jobs.run_findings (run_id, severity);

FK on recoverable_runs.id links a finding back to the run that produced it; ON DELETE CASCADE clears findings when their run row is pruned by a future retention job.

jobs.* is a NEW schema, created by Part 2's migration — keep it distinct from auth.* / storage.* / admin.* so operational tables don't pollute domain schemas.

Non-obvious traps

Recorded here (and in the trait doc-comments) because every one has been learned the hard way in similar systems:

1. Grace window uses scan_started_at, NOT NOW(). A resumable scan spanning 6 h must snapshot its grace boundary at start. Otherwise items uploaded 30 min in flip from "young, skip" (Pass 1's view) to "old, flag" (Pass 2's view) mid-flight — the scan produces false positives against itself.
2. Cursor is per-check, opaque bytes. Blob check cursors on BLAKE3 hash (fixed 64 hex chars — natural lex order). Thumbnail cursors on file_id UUID. Folder-tree cursor on ltree path. The trait treats it as Vec<u8>; each impl serializes what it needs.
3. Findings are idempotent on (run_id, kind, resource_id). Resume revisit must not double-count. Pass 2 can also DELETE findings that transitioned (was MissingBlob, blob has since landed → drop the finding, not the whole run).
4. Cooperative cancellation ONLY. Between batches, poll recoverable_runs.status. A tokio::spawn abort mid-batch leaks — cursor unpersisted, findings half-written. Cancel path writes status='Paused' + current cursor before returning.
5. Crash recovery on boot. Any status='Running' at server start = server died mid-scan. Auto-transition to Paused; DON'T auto-resume (the bug that killed the last run may still be present). Admin decides.
6. Batch size 1000 items or 30 s, whichever comes first. Cursor commit per-row makes DB write cost dominate at 1M items; longer batches leak more progress on crash.
7. Two directions don't share a cursor. BlobConsistencyCheck orphan side walks storage listing (S3 continuation token / readdir); missing side walks file_blobs by hash. Sequence them (orphan phase → missing phase); cursor encodes current phase. ThumbnailConsistencyCheck is worse — four phases (2 subspaces × 2 directions), each with its own natural cursor. Cursor encodes (subspace, direction, key).

Trait shapes

ConsistencyCheck

/// A single consistency check with a resumable, cursor-based scan.
///
/// # For implementors
///
/// Every implementation must decide five things before the first line of
/// code. Answer them in comments at the top of the impl:
///
/// 1. **Direction.** Backward (storage → DB) surfaces orphans; forward
///    (DB → storage) surfaces missing. Most checks do BOTH — sequence
///    them and encode the current phase in the cursor.
///
/// 2. **Severity per finding kind.** `DataLoss` (user impact) /
///    `Reclaimable` (disk waste) / `Regenerable` (auto-heals) /
///    `Drift` (accounting). The single most common mistake is treating
///    a missing user-uploaded thumbnail as `Regenerable` — it's not,
///    the server can't recreate what the user provided. It's `DataLoss`.
///
/// 3. **Cursor format.** Opaque `Vec<u8>` to the framework. Yours to
///    serialize. Content-addressable blobs → 32-byte BLAKE3. UUID rows →
///    16-byte UUID. Path rows → the path bytes. Multi-phase check →
///    prepend a phase byte.
///
/// 4. **Grace window.** Default 1 h (matches dedup GC). Deep checks
///    (hash verification) use 24 h. Grace ALWAYS refers to
///    `scan_started_at`, never `NOW()` — see trap #1 below.
///
/// 5. **Batch boundary.** 1000 items or 30 s. Call `store.checkpoint`
///    and `store.should_cancel` between batches — cancellation is
///    cooperative, never task-abort.
///
/// # Two-pass discipline
///
/// Pass 1 — build candidate list from a snapshot read, excluding items
/// younger than `grace_window`.
///
/// Pass 2 — per candidate, re-read the OTHER side's state right before
/// flagging. If it transitioned (ref went up, blob just landed, row was
/// deleted), silently drop.
///
/// Pass 1 alone LOOKS complete but produces false positives on every
/// race. Never skip Pass 2.
///
/// # Canonical example
///
/// See `BlobConsistencyCheck` in
/// `src/infrastructure/services/consistency/blob_check.rs` — it exercises
/// every axis (both directions, both severities, grace window, cursor,
/// cooperative cancel, blob-level report shape for missing findings).
#[async_trait]
pub trait ConsistencyCheck: Send + Sync {
    /// Machine-readable name — appears in the admin endpoint slug and in
    /// audit `event` values. Lowercase snake_case, one per check.
    fn name(&self) -> &'static str;

    fn grace_window(&self) -> Duration { Duration::from_secs(3600) }

    /// `true` (default) → safe to run against live traffic; the check
    /// respects grace window + two-pass re-verify. Only false for a
    /// check that genuinely needs a quiescent DB (whole-table snapshot
    /// verification of hashes) — not required for anything in v1-v5.
    fn concurrent_safe(&self) -> bool { true }

    /// `cursor: None` → fresh run. `Some(bytes)` → resume from last
    /// persisted checkpoint. Impls MUST:
    /// - call `store.checkpoint(cursor).await` between batches
    ///   (~1000 items or 30 s, whichever comes first);
    /// - call `store.should_cancel().await` between batches — return
    ///   `RunOutcome::Paused { cursor }` when it returns `true`;
    /// - use `store.scan_started_at()` (not `now()`) as the grace
    ///   window reference.
    async fn run_resumable(
        &self,
        opts: &CheckOptions,
        cursor: Option<Vec<u8>>,
        store: &dyn CheckStore,
    ) -> Result<RunOutcome, DomainError>;
}

#[derive(Debug)]
pub enum RunOutcome {
    Completed,
    Paused { cursor: Vec<u8> },
    Failed(DomainError),
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Severity {
    DataLoss,     // user impact — top of triage
    Reclaimable,  // disk waste, no user impact
    Regenerable,  // auto-heals on next request
    Drift,        // accounting mismatch, no user impact
}

pub struct Inconsistency {
    pub kind: &'static str,       // "OrphanBlob", "MissingBlob", ...
    pub severity: Severity,
    pub resource_id: String,      // opaque
    pub detail: serde_json::Value,
}

CheckStore

The framework hands each check a CheckStore — the only side effect a check performs on shared state.

#[async_trait]
pub trait CheckStore: Send + Sync {
    fn run_id(&self) -> Uuid;
    fn scan_started_at(&self) -> chrono::DateTime<chrono::Utc>;

    /// Persist the cursor + last-progress timestamp. Called between
    /// batches. If a crash happens after this returns, the next resume
    /// starts from `cursor`.
    async fn checkpoint(&self, cursor: Vec<u8>, scanned_count: u64)
        -> Result<(), DomainError>;

    /// Poll the run's `status` column. Returns `true` when an admin
    /// requested cancellation. The check MUST return `Paused` with the
    /// current cursor.
    async fn should_cancel(&self) -> Result<bool, DomainError>;

    /// Upsert a finding. `UNIQUE (run_id, kind, resource_id)` in the
    /// schema means re-scanning the same resource on resume is safe.
    async fn record_finding(&self, finding: Inconsistency)
        -> Result<(), DomainError>;

    /// Delete a previously-recorded finding — used when Pass 2 sees
    /// the resource transitioned out of the inconsistent state.
    async fn drop_finding(&self, kind: &str, resource_id: &str)
        -> Result<(), DomainError>;
}

StatefulAdapter

/// Marker + registration trait for any adapter that persists state OUTSIDE
/// process memory: blobs on disk / S3, DB tables, on-disk caches, message
/// queues you own.
///
/// Added as a SUPERTRAIT on every state-owning port
/// (`trait BlobStorage: StatefulAdapter`, `trait ThumbnailStore:
/// StatefulAdapter`, `trait FolderRepository: StatefulAdapter`, …), which
/// means: NO NEW ADAPTER CAN COMPILE without declaring its consistency
/// contract. The compiler is the enforcement; these doc-comments are the
/// education.
///
/// # For implementors adding a new stateful adapter
///
/// You cannot skip this trait. If you're reading this because your PR
/// won't compile, work through:
///
/// 1. **Am I actually stateful?** State means "bytes or rows outside
///    process memory that can desync from other subsystems". Config,
///    caches keyed by session, and derived indexes are NOT stateful
///    for this purpose (they can be dropped and rebuilt). If you're
///    not stateful, drop the `StatefulAdapter` impl entirely — but
///    then your port shouldn't have `StatefulAdapter` as a supertrait
///    either, so this compile error means the port author already
///    decided you were.
///
/// 2. **What are the DIRECTIONS of drift I can detect?** Almost every
///    stateful adapter has both:
///    - Backward (my storage → the DB that references it): orphans.
///    - Forward (the DB → my storage): missing.
///    Return one check that covers both by sequencing phases, OR two
///    checks (one per direction). The former is easier to operate.
///
/// 3. **What's the SEVERITY of each finding?** See `Severity` in
///    `consistency_check.rs`. Missing user-uploaded data is `DataLoss`;
///    missing server-derived data is `Regenerable`; orphan bytes are
///    `Reclaimable`; accounting drift is `Drift`.
///
/// 4. **What CURSOR fits my walk?** Content-addressable → hash prefix.
///    UUID-keyed → UUID lex. Path-keyed → path bytes. Whatever you pick,
///    it's opaque `Vec<u8>` to the framework — decode inside your check.
///
/// See `BlobConsistencyCheck` for the canonical impl to copy-adapt.
pub trait StatefulAdapter: Send + Sync {
    /// Subsystem slug — appears in the JobRegistry-registered
    /// `job_name` (`consistency_<subsystem>`, e.g. `consistency_blobs`)
    /// and in audit log `event` values. Lowercase snake_case, unique
    /// per adapter. Convention: `"blobs"`, `"thumbnails"`, `"trash"`,
    /// `"folder_tree"`, `"used_bytes"`.
    fn subsystem(&self) -> &'static str;

    /// REQUIRED (no default impl). Return every consistency check
    /// this adapter contributes. Most adapters return exactly one.
    /// Multi-keying subsystems return more — `ThumbnailStore` returns
    /// FOUR checks (server-generated + user-uploaded, each in both
    /// directions).
    ///
    /// Returning `vec![]` is a red flag. If your adapter has state but
    /// no check, either:
    /// - Your state is fully covered by another adapter's check
    ///   (rare — document exactly WHERE in a comment on this method).
    /// - You haven't written the check yet — return
    ///   `vec![]` with a `TODO(consistency): add <Name>ConsistencyCheck`
    ///   comment, ship the trait wiring, add the check in a follow-up PR.
    ///
    /// Reviewers will grep `TODO(consistency)` and ask when it lands.
    fn consistency_checks(&self) -> Vec<Arc<dyn ConsistencyCheck>>;
}

ConsistencyRegistry

/// Collects `StatefulAdapter`s at wire-up time. Instantiated once in
/// `AppServiceFactory`, exposed on `AppState`, consumed by the admin
/// handler + (when JobRegistry lands) the scheduler.
pub struct ConsistencyRegistry {
    adapters: Vec<Arc<dyn StatefulAdapter>>,
}

impl ConsistencyRegistry {
    pub fn register(&mut self, adapter: Arc<dyn StatefulAdapter>) {
        // Trait bound forces `subsystem()` + `consistency_checks()` to exist.
        self.adapters.push(adapter);
    }

    /// Every check contributed by every registered adapter, flat.
    pub fn all_checks(&self) -> Vec<Arc<dyn ConsistencyCheck>> {
        self.adapters
            .iter()
            .flat_map(|a| a.consistency_checks())
            .collect()
    }

    pub fn get(&self, name: &str) -> Option<Arc<dyn ConsistencyCheck>> {
        self.all_checks().into_iter().find(|c| c.name() == name)
    }
}

Admin surface

Consistency runs are ordinary RecoverableJobHandlers (see docs/plan/job-registry.md Part 2), so most operator actions reach them through the shared scheduler surface:

GET    /api/admin/jobs
       → summary list — consistency runs appear as
         `job_name = "consistency_<name>"`

POST   /api/admin/jobs/consistency_{name}/trigger
       → 200 { ok, outcome: { run_id, status } }
         (starts a new run or resumes the latest Paused one — see
         Part 2's `run_or_resume`)

POST   /api/admin/jobs/consistency_{name}/cancel
       → 200 { run_id, status: "CancelRequested" }
       (cooperative — check finishes its current batch and returns Paused)

GET    /api/admin/jobs/consistency_{name}/runs?status=<status>
       → 200 [{ id, status, scanned_count, last_progress_at, … }]

GET    /api/admin/jobs/consistency_{name}/runs/{id}
       → 200 { run: {...}, findings: [...paginated] }

Findings enrichment on runs/{id} is consistency-specific — read from jobs.run_findings and joined into the response. Everything else is generic Part 2 behaviour.

Production surface — always on, audit-logged. No feature-flag gate.

Approach

1. Traits + framework in isolation

src/application/ports/consistency.rs

• Define ConsistencyCheck, RunOutcome, Severity, Inconsistency, CheckStore, StatefulAdapter, CheckOptions.
• Full doc-comments as sketched above — these are the educational surface, don't cut them.

src/infrastructure/services/consistency/mod.rs

• ConsistencyRegistry (data structure only).
• PgCheckStore — impl of CheckStore reading/writing jobs.recoverable_runs (filtered to job_name LIKE 'consistency_%')
  • jobs.run_findings.
• run_check(check, cursor, store) — the runner that calls run_resumable, applies timeout, records outcome.

2. Schema migration

migrations/YYYYMMDDHHMMSS_background_runs_admin_schema.sql — creates the merged jobs.recoverable_runs table shared with the JobRegistry plan. Consistency checks own the jobs.run_findings table alone and reference recoverable_runs.id via FK.

CREATE SCHEMA IF NOT EXISTS admin;

CREATE TABLE jobs.recoverable_runs (
    id                 UUID PRIMARY KEY,
    job_name           TEXT NOT NULL,               -- 'consistency_blobs', 'backend_migration', 'reextract_audio', ...
    status             TEXT NOT NULL,               -- Running / Paused / Completed / Failed / CancelRequested
    started_at         TIMESTAMPTZ NOT NULL,
    last_progress_at   TIMESTAMPTZ NOT NULL,
    completed_at       TIMESTAMPTZ,
    cursor             BYTEA,
    stats              JSONB NOT NULL DEFAULT '{}'::jsonb,   -- e.g. {"scanned_count": 12345}
    params             JSONB NOT NULL DEFAULT '{}'::jsonb,   -- e.g. {"grace_window_secs": 3600}
    error_message      TEXT
);
CREATE UNIQUE INDEX one_active_run_per_job
    ON jobs.recoverable_runs (job_name)
    WHERE status IN ('Running', 'Paused');
CREATE INDEX ON jobs.recoverable_runs (last_progress_at) WHERE status = 'Running';

CREATE TABLE jobs.run_findings (
    id             UUID PRIMARY KEY,
    run_id         UUID NOT NULL REFERENCES jobs.recoverable_runs(id) ON DELETE CASCADE,
    kind           TEXT NOT NULL,
    severity       TEXT NOT NULL,
    resource_id    TEXT NOT NULL,
    detail         JSONB NOT NULL DEFAULT '{}'::jsonb,
    found_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (run_id, kind, resource_id)
);
CREATE INDEX ON jobs.run_findings (run_id, severity);

3. Supertrait bounds on existing state-owning ports

Add StatefulAdapter as a supertrait on:

• src/application/ports/storage_ports.rs::BlobStorage (or wherever the blob-storage port lives).
• src/application/ports/thumbnails.rs::ThumbnailStore (both server-generated and user-uploaded paths).
• src/application/ports/text_extraction.rs::TextExtractionCache.
• src/application/ports/audio_metadata.rs::AudioMetadataCache (if a distinct port exists).
• src/domain/repositories/file_blob_read_repository.rs::FileBlobReadRepository (via the port trait it exposes to application services).
• src/domain/repositories/folder_repository.rs::FolderRepository.
• src/domain/repositories/trash_repository.rs::TrashRepository.
• src/infrastructure/services/webdav_dead_property_store.rs (DeadPropertyStore — has its own leak class per the deferred-rekey memory note).

Each of these will trigger compile errors in its impls. Each impl gets a two-line stub:

impl StatefulAdapter for LocalFsBlobStorage {
    fn subsystem(&self) -> &'static str { "blobs" }
    fn consistency_checks(&self) -> Vec<Arc<dyn ConsistencyCheck>> {
        // TODO(consistency): add BlobConsistencyCheck once impl lands.
        vec![]
    }
}

Ship this PR without the actual checks. Grep TODO(consistency) = punch list.

4. First real check — BlobConsistencyCheck

src/infrastructure/services/consistency/blob_check.rs

• Depends on BlobStorage (storage listing) + FileBlobReadRepository (DB SELECT).
• Phase 1 (orphan direction): walk storage listing, cursor on hash prefix. Batch of 1000, checkpoint, cancel-poll. For each batch: SELECT ref_count FROM storage.file_blobs WHERE hash IN (…). Pass 2 re-verifies at flag time. Severity: Reclaimable.
• Phase 2 (missing direction): walk file_blobs ordered by hash, cursor on hash. Batch of 1000. For each row: HEAD on storage backend. If missing AND row hasn't disappeared AND row is older than grace_window, flag MissingInStorage with affected_file_ids from a JOIN to file_metadata. Severity: DataLoss.
• Cursor format: [phase: u8, hash_key: 32 bytes].
• LocalFsBlobStorage::consistency_checks() returns vec![Arc::new(BlobConsistencyCheck::new(self.clone(), ...))].

5. Admin handlers

src/interfaces/api/handlers/admin_handler.rs

• start_consistency_check(name, force) — insert an jobs.recoverable_runs row with job_name = 'consistency_<name>' and status = 'Running', spawn a tokio task calling run_check, return run_id. Concurrent triggers hit the partial unique index and short-circuit to returning the surviving row.
• cancel_run(id) — UPDATE status = 'CancelRequested'.
• resume_run(id) — verify status == 'Paused', spawn task with the persisted cursor.
• list_runs(filter) — SELECT with filters + paginate.
• get_run(id) — SELECT run + paginated findings.

Same admin-guard as the JobRegistry surface (trigger_job, list_jobs). Production surface — always on, audit-logged, no feature-flag gate.

6. Boot-time crashed-run recovery

In AppServiceFactory init, after DB pool is up:

sqlx::query!(
    "UPDATE jobs.recoverable_runs
        SET status = 'Paused',
            error_message = COALESCE(error_message, 'server restart mid-run')
      WHERE job_name LIKE 'consistency_%'
        AND (status = 'Running' OR status = 'CancelRequested')"
).execute(&pool).await?;

Filtering on job_name LIKE 'consistency_%' scopes the sweep to consistency runs; other tenants of recoverable_runs (storage migration, reextract-*) run the same auto-Pause sweep from their own boot-time hook. The JobRegistry supervisor's boot check may generalise this into a single scheduler-wide sweep — until then, one per subsystem is fine.

Do NOT auto-resume — the bug that killed the last run may still be there. Log a warning if any rows were flipped so operators notice.

7. Hurl regression — tests/api/consistency_check.hurl

• Setup: login admin, seed one file (which creates one blob).
• Trigger blobs check with force=true (grace_days=0). Poll runs list until status='Completed'. Assert 0 findings.
• Manually orphan a blob (SQL: DELETE FROM file_metadata WHERE …, leave file_blobs + storage in place). Trigger check again. Assert 1 finding with kind='OrphanInStorage', severity='Reclaimable'.
• Manually break a blob (SQL: leave file_blobs alone, wipe the storage backend for that hash — actually, use the storage service's test hook if one exists; otherwise skip this in Hurl and cover in integration tests).
• Cancel a run mid-scan (large seed, poll for scanned_count > 0, POST cancel, poll until status='Paused'). Resume. Assert scanned_count after resume > checkpoint.

8. Follow-up PRs (remaining checks)

Priority order:

#	Check	Direction	Complexity
1	BlobConsistencyCheck	both	high (canonical)
2	ThumbnailConsistencyCheck	both × 2 subspaces = 4 sub-scans	high
3	UsedBytesConsistencyCheck	pure SQL	low — wrap existing reconciliation diff
4	FolderTreeConsistencyCheck	pure SQL	low — closure over folders.parent_id
5	Deep-hash sub-mode on BlobConsistencyCheck	forward	medium — 24 h grace, opt-in
6	DeadPropertyConsistencyCheck	forward	low, blocked on rekey (see project_webdav_dead_properties_drive_rekey)
7	TrashConsistencyCheck	both	medium — trash rows vs file_metadata soft-delete flags

Each is a separate PR against the stable trait. TODO(consistency) count decreases by one per PR.

Critical files

Create:

• src/application/ports/consistency.rs (~250 lines — traits + doc-comments)
• src/infrastructure/services/consistency/mod.rs (~40 lines — pub types)
• src/infrastructure/services/consistency/registry.rs (~80 lines)
• src/infrastructure/services/consistency/pg_check_store.rs (~150 lines)
• src/infrastructure/services/consistency/runner.rs (~100 lines)
• src/infrastructure/services/consistency/blob_check.rs (~300 lines — canonical impl)
• migrations/YYYYMMDDHHMMSS_consistency_check_admin_schema.sql (~30 lines)
• tests/api/consistency_check.hurl (~150 lines)

Modify (add supertrait bound):

• src/application/ports/storage_ports.rs — BlobStorage: StatefulAdapter.
• src/application/ports/thumbnails.rs — ThumbnailStore: StatefulAdapter.
• src/application/ports/text_extraction.rs.
• src/application/ports/audio_metadata.rs (if applicable).
• src/domain/repositories/file_blob_read_repository.rs.
• src/domain/repositories/folder_repository.rs.
• src/domain/repositories/trash_repository.rs.
• src/infrastructure/services/webdav_dead_property_store.rs.

Modify (add StatefulAdapter stubs):

• Every impl of the above ports. Each gets subsystem() + a vec![] stub with TODO(consistency).

Modify (wire up admin surface):

• src/common/di.rs — build Arc<ConsistencyRegistry>, expose on AppState, register every stateful adapter.
• src/interfaces/api/handlers/admin_handler.rs — five handlers.
• src/interfaces/api/routes.rs — five routes.
• src/interfaces/api/mod.rs — utoipa paths.
• tests/api/run.sh — register consistency_check.hurl.

Reused existing utilities

• Admin-guard + audit-log pattern at src/interfaces/api/handlers/admin_handler.rs::trigger_job — same shape for the new endpoints (production surface, always-on, audit-logged; no feature-flag gate).
• Dedup GC's orphan-detection logic (dedup_service.rs) — the algorithmic template for BlobConsistencyCheck's orphan phase. Reference impl, not a callsite — the check needs its own two-pass discipline; GC currently reap-and-forgets.
• Reconciliation SQL diff in storage_usage_service.rs — becomes UsedBytesConsistencyCheck almost verbatim, wrapped in report-only mode.
• AGENTS.md audit convention — every finding double-logs to target: "audit", event: "consistency.{check}.finding", plus operational log to target: "oxicloud::consistency".

Verification

1. Compile: cargo check --all-features --all-targets + cargo clippy -- -D warnings clean.
2. Schema: just fe-nothing … cargo run starts; migration lands the admin schema; psql -c "\dt admin.*" shows the two tables.
3. Boot line: consistency: N adapter(s) registered, M check(s) available. Grep TODO(consistency) in the source; count should equal M in v1 minus the shipped BlobConsistencyCheck.
4. Hurl (tests/api/consistency_check.hurl):
   • clean state → 0 findings
   • forced orphan → 1 OrphanInStorage finding, severity Reclaimable
   • cancel + resume round-trip preserves scanned_count
5. Crash recovery: kill server mid-scan (kill -9); restart; confirm the row is Paused with error_message='server restart mid-run'; POST resume; check completes.
6. Trait enforcement: add a new dummy adapter impl of BlobStorage without StatefulAdapter — compile MUST fail. Add the stub; compile succeeds. This is the load-bearing property of the design.
7. Grace-window sanity: run against a fresh 10 s window; upload a file mid-scan; confirm the young blob does NOT surface as MissingInStorage (grace window covers it).
8. AuthZ gate: non-admin caller hits POST /api/admin/jobs/consistency_blobs/trigger → 403 from the admin middleware, audit line records the rejection.

Out of scope

• JobRegistry integration. Consistency checks are admin-triggered in v1. When docs/plan/job-registry.md lands, JobRegistry will consume ConsistencyRegistry::all_checks() for scheduled execution — no code change needed here.
• Auto-repair. Findings are reported, not fixed. Repair primitives live in the existing services (dedup GC's reaper, storage_usage reconciler); a future admin surface could trigger targeted repair after human review.
• Distributed scheduling. Single-process. If OxiCloud ever runs multi-node, add SELECT … FOR UPDATE SKIP LOCKED on the run rows.
• Byte-exact whole-table snapshot verification. The concurrent_safe = false case — reserved for a future DeepBlobConsistencyCheck that requires either pg_export_snapshot + S3-consistent list OR a read-only mode. Not needed for v1-v5.
• Cursor pagination on the GET /runs/{id} findings list. Simple offset/limit for v1. Add cursor only if operators actually hit a 10k-findings run.
• Findings retention. Runs + findings accumulate forever until an operator manually deletes. Add a background cleaner once volume actually matters — most likely alongside JobRegistry.
• Auto-scheduling in v1. No tokio::spawn interval loop. Admin triggers only. Every scheduled invocation goes through JobRegistry when it lands.

Related memory notes

• feedback_no_abbreviated_env_vars — full-word env var names if any land (e.g. OXICLOUD_CONSISTENCY_BATCH_SIZE, not OXICLOUD_CC_BATCH).
• project_consistency_check_trait — the memory that captures this design's decisions and the traps that shape the trait.
• project_dedup_gc_test_trigger — motivates the check's grace-window discipline; also the source of the algorithmic template for the orphan-blob direction.
• project_webdav_dead_properties_drive_rekey — DeadPropertyStore will get a check, but only after the rekey lands.
• bug_thumbnail_dedup, bug_folder_cascade_hooks_missing — surface the four-sub-scan complexity of ThumbnailConsistencyCheck.
• bug_orphan_seed_null_orphaned_at_flaky — reminds implementors that the orphan-blob direction MUST check orphaned_at, not just ref_count = 0.