Merge pull request #688 from EdouardVanbelle/fix/copy_folder_ref_count_issue
This commit is contained in:
+1091
-71
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,246 @@
|
|||||||
|
# Plan — Hidden system drive for user-owned objects
|
||||||
|
|
||||||
|
**Status:** design captured 2026-08-21. Not implemented. Sibling to
|
||||||
|
`docs/plan/derived-blobs.md`, which answers "where does *derived*
|
||||||
|
content live"; this one answers "where do *user-owned binaries* live".
|
||||||
|
The two share one rule — **point at a file, never at a blob** — and
|
||||||
|
that rule is the reason neither needs new blob-referencing tables.
|
||||||
|
|
||||||
|
## Problem — binaries in the users row
|
||||||
|
|
||||||
|
`auth.users.image TEXT` (migration `20260526000000_add_user_image.sql`)
|
||||||
|
holds the avatar inline, up to 512 KiB. It is the wrong home, and the
|
||||||
|
cost is already measured rather than theoretical:
|
||||||
|
|
||||||
|
- **It TOASTs, and every wide read pays.** The repository comment on
|
||||||
|
`get_users_by_ids` records a group fan-out that "detoasted + shipped
|
||||||
|
+ parsed M avatars purely to discard them", fixed by adding a narrow
|
||||||
|
projection (`benches/ROUND12.md §Q1`, `ROUND13.md §Q1`). That
|
||||||
|
workaround exists *because* the column is in the wrong place; the
|
||||||
|
rule it leaves behind — "add a wide sibling rather than widening
|
||||||
|
this one back" — is a permanent tax on every future query.
|
||||||
|
- **Base64 inflation.** A ~384 KB image becomes ~512 KB of TEXT.
|
||||||
|
- **No dedup.** N users sharing a default or IdP-supplied avatar cost
|
||||||
|
N copies.
|
||||||
|
- **None of the storage stack applies** — no `EncryptedBlobBackend`,
|
||||||
|
no backend migration, no key rotation, no local cache, no
|
||||||
|
consistency coverage.
|
||||||
|
- **Backups and replication carry it.** Binary weight lands in the
|
||||||
|
logical dump and on every replica, forever.
|
||||||
|
|
||||||
|
The same pressure is coming for the UI background and a signature
|
||||||
|
image, so this needs a general answer, not another column.
|
||||||
|
|
||||||
|
## The rule — point at a file, never at a blob
|
||||||
|
|
||||||
|
```sql
|
||||||
|
ALTER TABLE auth.users
|
||||||
|
ADD COLUMN avatar_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
|
||||||
|
ADD COLUMN background_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
|
||||||
|
ADD COLUMN signature_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL;
|
||||||
|
```
|
||||||
|
|
||||||
|
`storage.files` is already a `BlobReferenceSource`, already covered by
|
||||||
|
every consistency edge, already GC-integrated, already copy- and
|
||||||
|
version-aware. A pointer to a *file* therefore adds **zero new
|
||||||
|
reference sources and zero new consistency edges** — `auth.users`
|
||||||
|
holds no blob reference at all, only a pointer to a row that does.
|
||||||
|
Deleting the file decrements the blob refcount through the existing
|
||||||
|
file-deletion path.
|
||||||
|
|
||||||
|
Point at `id`, never at a name or path: a rename must not break a
|
||||||
|
profile.
|
||||||
|
|
||||||
|
For a small fixed set, **columns beat a table.** A table becomes right
|
||||||
|
only when the object set is open-ended, and it would cost exactly what
|
||||||
|
the pointer avoids.
|
||||||
|
|
||||||
|
### Rejected alternatives
|
||||||
|
|
||||||
|
| Option | Why not |
|
||||||
|
|---|---|
|
||||||
|
| **New table → blobs/chunks** | Needs a new `BlobReferenceSource`, a new fragment in the manifest sweep, and a new dangling check. Precisely the complexity the pointer removes. |
|
||||||
|
| **Direct backend paths** (`profile/{uuid}/avatar.png`) | `backend_migration` enumerates *blobs*, so a Local→S3 cutover **silently drops every avatar**. `EncryptedBlobBackend` is hash-keyed, so writes either bypass encryption or need a parallel path. `backend_consistency` raises `unknown_backend_file` (severity `anomaly`, "non-canonical file in blob namespace") per object on every sweep. And a fixed key overwritten in place breaks the immutability everything else rests on, killing `Cache-Control: immutable`. |
|
||||||
|
| **Reserved folder in the user's own drive** (`.profile/`, `.oxiprofile/`) | The user can write to it, so there are **two write paths and only one is validated** — the upload endpoint's format/size checks are bypassable over WebDAV and sync. It is in the sync tree, so "hidden" is not hidden in the protocols that matter. Existence-by-path drags back the extension problem. And any reserved name squats a namespace users own — `.profile` is a POSIX shell file, so it collides by default for anyone syncing a Linux home. |
|
||||||
|
| **One drive per user** | Per-user creation at signup, backfill for existing users, per-user quota exemption, and a cascade on account deletion. All avoidable — see below. |
|
||||||
|
|
||||||
|
Underneath all of it: these objects are owned by the **application on
|
||||||
|
the user's behalf**, not by the user as documents. Putting them in a
|
||||||
|
document tree conflates the two, and every problem above follows.
|
||||||
|
|
||||||
|
## The hidden system drive
|
||||||
|
|
||||||
|
**One shared drive, not one per user.** `kind = 'system'`, alongside
|
||||||
|
today's `CHECK (kind IN ('personal', 'shared'))`. Files inside are
|
||||||
|
owned by their respective users via the normal `created_by` /
|
||||||
|
ownership columns; the drive is only a container.
|
||||||
|
|
||||||
|
Sharing one drive drops per-user creation at signup, backfill for
|
||||||
|
existing users, and per-user quota exemption. Deleting a user becomes
|
||||||
|
a query over `storage.files` rather than a drive cascade.
|
||||||
|
|
||||||
|
Properties it needs:
|
||||||
|
|
||||||
|
- **Hidden at drive enumeration.** This is the single filter point,
|
||||||
|
and it is why the drive beats a folder: a folder must be filtered in
|
||||||
|
directory listings, search results, recent items, trash, photo
|
||||||
|
indexing and sync deltas, whereas a drive is filtered once where
|
||||||
|
drives are listed. Every surface must honour it — REST, WebDAV,
|
||||||
|
NextCloud, search, quota reporting. **A missed filter is the
|
||||||
|
characteristic bug of this design**, so it deserves a test per
|
||||||
|
surface rather than per call site.
|
||||||
|
- **Trash disabled.** Otherwise every replaced avatar lands in a trash
|
||||||
|
nobody can see, holding a blob reference that GC cannot reclaim
|
||||||
|
while retention keeps it alive — invisible storage growth with no
|
||||||
|
signal. Deletion here is immediate.
|
||||||
|
- **Exempt from the user quota envelope.** Nobody should pay quota for
|
||||||
|
their own avatar.
|
||||||
|
- **Created at install, fail-fast at boot.** If the drive is missing,
|
||||||
|
panic rather than silently disabling profile objects — a silently
|
||||||
|
absent avatar surface is worse than a refusal to start.
|
||||||
|
- **Visible to admins.** Ops need to see it for storage accounting
|
||||||
|
even though it is hidden from users.
|
||||||
|
|
||||||
|
## Visibility — per kind, in code
|
||||||
|
|
||||||
|
Reads go through a service method carrying an explicit policy, audited
|
||||||
|
like any other authorization decision. Because the column set is fixed
|
||||||
|
and small, the policy is a `match`, not stored data — there is nothing
|
||||||
|
to misconfigure, and adding a column forces adding an arm:
|
||||||
|
|
||||||
|
| Object | Who may read | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| `avatar` | **the same rule as profile visibility** | Not "any authenticated user". `AGENTS.md` has `user_profile.rejected` return **404, never 403**, for an external caller with no relationship, specifically so existence cannot be confirmed. An avatar endpoint answering 200 for any caller is an oracle around that control. |
|
||||||
|
| `background` | owner only | Nobody else has a reason to fetch it. |
|
||||||
|
| `signature` | owner only, plus the document render path | A handwritten signature is forgery material. It is "public" only in the sense that it appears on documents you may already read — which argues for rendering it into those documents, not exposing it as a directly-readable object. |
|
||||||
|
|
||||||
|
Note the consequence: the drive's own permission model is **not** what
|
||||||
|
governs these reads. The object lives in a drive and is read through a
|
||||||
|
different door. That is a deliberate choice, not an oversight — record
|
||||||
|
it so nobody later "fixes" it by granting cross-user drive access.
|
||||||
|
|
||||||
|
## What must NOT live here
|
||||||
|
|
||||||
|
> **If losing control of it is a security incident rather than a
|
||||||
|
> cosmetic bug, it stays in the database.** The system drive is for
|
||||||
|
> user-facing binaries.
|
||||||
|
|
||||||
|
So E2E/Vault key material — public key bundle, passphrase-wrapped
|
||||||
|
private key, recovery kit — stays in `auth.users` columns. Four
|
||||||
|
reasons, the last decisive:
|
||||||
|
|
||||||
|
1. **Failure-mode asymmetry.** The characteristic bug here is a missed
|
||||||
|
listing filter. For a wallpaper that is cosmetic; for key material
|
||||||
|
it is disclosure.
|
||||||
|
2. **Atomicity.** Rotating a passphrase rewraps the private key
|
||||||
|
*together with* the credential change. A DB column makes that one
|
||||||
|
transaction; a file write plus a column update cannot be atomic.
|
||||||
|
3. **Size.** A few KB — blob storage buys nothing.
|
||||||
|
4. **`EncryptedBlobBackend` encrypts under a key the server holds.**
|
||||||
|
For E2E material the whole premise is that the server *cannot*
|
||||||
|
decrypt. Routing a wrapped private key through the blob layer
|
||||||
|
encrypts it twice, once under a key the operator controls, adding
|
||||||
|
no protection while creating the impression of it.
|
||||||
|
|
||||||
|
Users who want to store genuinely private *files* already have the
|
||||||
|
personal drive, with the full AuthZ engine behind it. There is no gap.
|
||||||
|
|
||||||
|
## Migrating the avatar off `auth.users.image`
|
||||||
|
|
||||||
|
Volume is one row per user, so unlike the thumbnail migration this
|
||||||
|
needs **no read-through phase** — a single batch job is enough.
|
||||||
|
|
||||||
|
**Phase 1.** Add the pointer columns and the system drive. Write path
|
||||||
|
switches to files; read path prefers `avatar_file_id` and falls back
|
||||||
|
to `image` when null.
|
||||||
|
|
||||||
|
**Phase 2.** `profile_image_import`, a registered `JobRegistry` job
|
||||||
|
(subject-first naming, per convention). For each user with a non-null
|
||||||
|
`image`:
|
||||||
|
|
||||||
|
1. Decode the data URI; skip and log if it does not parse, rather than
|
||||||
|
failing the batch.
|
||||||
|
2. `store_from_stream` the decoded bytes → derived blob + manifest.
|
||||||
|
3. Insert a `storage.files` row in the system drive, owned by that
|
||||||
|
user.
|
||||||
|
4. Set `avatar_file_id`.
|
||||||
|
|
||||||
|
Idempotent (`WHERE avatar_file_id IS NULL`), resumable via a user-id
|
||||||
|
cursor, and reports imported / skipped-unparseable / failed counts.
|
||||||
|
|
||||||
|
**Phase 3.** Drop `auth.users.image` and the fallback, gated on the
|
||||||
|
job reporting zero remaining. Dropping the column is what actually
|
||||||
|
reclaims the TOAST weight and retires the narrow-projection rule in
|
||||||
|
`get_users_by_ids`.
|
||||||
|
|
||||||
|
**IdP-sourced avatars.** OIDC login already refreshes the avatar
|
||||||
|
("same IdP avatar, already verified" — `user_pg_repository.rs:1435`).
|
||||||
|
That path must be converted at Phase 1, not Phase 3, or it keeps
|
||||||
|
writing to a column the migration is draining.
|
||||||
|
|
||||||
|
## Object catalogue
|
||||||
|
|
||||||
|
**Now:** avatar, UI background, signature image.
|
||||||
|
|
||||||
|
**Strong future candidates** — these are what justify a drive rather
|
||||||
|
than three columns and a corner:
|
||||||
|
|
||||||
|
| Object | Why it fits |
|
||||||
|
|---|---|
|
||||||
|
| **Data exports** (GDPR takeout, drive-export zip) | Generated async, large, downloadable, should expire. Today there is nowhere to put them. |
|
||||||
|
| **Staged imports** (Google Takeout, NextCloud export) | Multi-step ingestion needs durability beyond a temp file. |
|
||||||
|
| **Share-page branding / logo** | Per-user or per-org, served on public share pages. |
|
||||||
|
|
||||||
|
**Same problem, different owner — this drive does not help:**
|
||||||
|
`carddav.contacts.photo_url TEXT` (contact photos, today a URL or an
|
||||||
|
inlined data URI) and CalDAV `ATTACH` event attachments. They are keyed
|
||||||
|
by contact and by event, not by user. But the *pointer* generalises:
|
||||||
|
`contacts.photo_file_id UUID REFERENCES storage.files(id)` solves them
|
||||||
|
with no new blob-referencing table either — they simply live in the
|
||||||
|
address book's or calendar's own drive rather than here. Own plan.
|
||||||
|
|
||||||
|
## Operational details
|
||||||
|
|
||||||
|
- **Replace must delete.** Write new file → update pointer →
|
||||||
|
hard-delete the previous file. `ON DELETE SET NULL` protects the
|
||||||
|
pointer when a file vanishes, but nothing deletes the old file
|
||||||
|
because the pointer moved.
|
||||||
|
- **Validation lives at the endpoint** and is now the only write path,
|
||||||
|
which is the point of not using a user-writable location. Enforce
|
||||||
|
format, dimensions and size there.
|
||||||
|
- **Account deletion** deletes the user's files in the system drive
|
||||||
|
explicitly; the pointer columns are on the row being deleted anyway.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- **Secrets of any kind.** See the discriminator above.
|
||||||
|
- **Per-user system drives.** One shared drive; revisit only if
|
||||||
|
per-user quota or trash semantics ever become necessary.
|
||||||
|
- **Contact photos and event attachments.** Same pointer pattern,
|
||||||
|
different owner, different drive — separate plan.
|
||||||
|
- **A generic "user objects" API.** The column set is fixed and small
|
||||||
|
on purpose. Reach for a table only when it demonstrably is not.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Who owns the system drive row itself, and what does
|
||||||
|
`drives_consistency` expect of a drive with no human owner?
|
||||||
|
- Does the signature object survive the "owner only" rule, or does
|
||||||
|
document rendering need a broader read path than expected?
|
||||||
|
- Should exports live here or in a short-lived namespace with its own
|
||||||
|
expiry, given they are the only candidate with a natural TTL?
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/plan/derived-blobs.md` — the sibling plan; shares the
|
||||||
|
point-at-a-file rule and documents the consistency coverage matrix
|
||||||
|
these objects inherit for free.
|
||||||
|
- `migrations/20260526000000_add_user_image.sql` — the column being
|
||||||
|
retired.
|
||||||
|
- `migrations/20260802100000_drives_schema_additive.sql` — the
|
||||||
|
`kind IN ('personal','shared')` constraint this extends.
|
||||||
|
- `src/AGENTS.md` — the backend-abstraction rules, and the
|
||||||
|
anti-enumeration pattern the avatar visibility rule follows.
|
||||||
|
- Memory `project_drive_naming_and_vault_reservation` — "Vault"
|
||||||
|
reserved for the future E2E kind whose key material this plan
|
||||||
|
explicitly keeps out of the drive.
|
||||||
@@ -60,11 +60,21 @@ export function listJobs(): Promise<JobSummary[]> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* `POST /api/admin/jobs/{name}/trigger?force=X&deep=X` — dispatch a job
|
* `POST /api/admin/jobs/{name}/trigger?force=X&deep=X&repair=X` —
|
||||||
* on-demand. `force` bypasses per-tenant idempotency checks (e.g.
|
* dispatch a job on-demand.
|
||||||
* `trash_cleanup` skipping when nothing is due). `deep` opts into slow
|
*
|
||||||
* variants (currently only `storage_consistency`, propagated by
|
* - `force` bypasses per-tenant idempotency checks (e.g. `trash_cleanup`
|
||||||
* `consistency_batch` to every child).
|
* skipping when nothing is due).
|
||||||
|
* - `deep` opts into slow variants (currently only `storage_consistency`,
|
||||||
|
* propagated by `consistency_batch` to every child).
|
||||||
|
* - `repair` opts into corrective action on the refcount consistency
|
||||||
|
* tenants (`blobs_consistency`, `manifests_consistency`, and
|
||||||
|
* `consistency_batch` which fans out to both). Content-safe: only the
|
||||||
|
* stored counter changes to match the auditor's computed value. Race-
|
||||||
|
* safe: the corrective UPDATE recomputes the auditor formula in the
|
||||||
|
* same statement, so a concurrent write can't leave a stale value.
|
||||||
|
* Default `false` preserves discovery-only behaviour — surface a
|
||||||
|
* confirm-first flow when calling with `repair: true`.
|
||||||
*
|
*
|
||||||
* Throws on 4xx / 5xx with the backend's error message when present.
|
* Throws on 4xx / 5xx with the backend's error message when present.
|
||||||
* A 404 means the job name isn't registered — surface that specifically
|
* A 404 means the job name isn't registered — surface that specifically
|
||||||
@@ -72,11 +82,12 @@ export function listJobs(): Promise<JobSummary[]> {
|
|||||||
*/
|
*/
|
||||||
export async function triggerJob(
|
export async function triggerJob(
|
||||||
name: string,
|
name: string,
|
||||||
opts: { force?: boolean; deep?: boolean; storage?: string } = {}
|
opts: { force?: boolean; deep?: boolean; storage?: string; repair?: boolean } = {}
|
||||||
): Promise<TriggerResponse> {
|
): Promise<TriggerResponse> {
|
||||||
const params = new URLSearchParams();
|
const params = new URLSearchParams();
|
||||||
if (opts.force) params.set('force', 'true');
|
if (opts.force) params.set('force', 'true');
|
||||||
if (opts.deep) params.set('deep', 'true');
|
if (opts.deep) params.set('deep', 'true');
|
||||||
|
if (opts.repair) params.set('repair', 'true');
|
||||||
// `storage` scopes tenants that respect JobRunArgs.storage —
|
// `storage` scopes tenants that respect JobRunArgs.storage —
|
||||||
// currently blobs_consistency / backend_consistency (probes the
|
// currently blobs_consistency / backend_consistency (probes the
|
||||||
// named entry instead of the live backend). See
|
// named entry instead of the live backend). See
|
||||||
|
|||||||
@@ -21,6 +21,7 @@
|
|||||||
import { SvelteSet } from 'svelte/reactivity';
|
import { SvelteSet } from 'svelte/reactivity';
|
||||||
import Icon from '$lib/icons/Icon.svelte';
|
import Icon from '$lib/icons/Icon.svelte';
|
||||||
import Modal from '$lib/components/Modal.svelte';
|
import Modal from '$lib/components/Modal.svelte';
|
||||||
|
import { confirmDialog } from '$lib/stores/dialogs.svelte';
|
||||||
import { t } from '$lib/i18n/index.svelte';
|
import { t } from '$lib/i18n/index.svelte';
|
||||||
import { errorMessage } from '$lib/utils/errors';
|
import { errorMessage } from '$lib/utils/errors';
|
||||||
import { ui } from '$lib/stores/ui.svelte';
|
import { ui } from '$lib/stores/ui.svelte';
|
||||||
@@ -66,6 +67,43 @@
|
|||||||
else busyKeys.delete(key);
|
else busyKeys.delete(key);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Per-job "Run" split-button menu state. Keyed by job name so
|
||||||
|
// two rows can open their menus independently (though the
|
||||||
|
// outside-click handler below closes all on any click outside
|
||||||
|
// any menu — matching the /files upload dropdown pattern). Only
|
||||||
|
// rows with `supportsDeep` OR `supportsRepair` render a chevron;
|
||||||
|
// the plain-Run rows (drives/folders/files/backend/… consistency,
|
||||||
|
// trash_cleanup, dedup_gc, …) show a bare "Run" button with no
|
||||||
|
// menu, keeping the common case one-click.
|
||||||
|
let runMenuOpen = $state<Record<string, boolean>>({});
|
||||||
|
function toggleRunMenu(name: string) {
|
||||||
|
runMenuOpen = { ...runMenuOpen, [name]: !runMenuOpen[name] };
|
||||||
|
}
|
||||||
|
function closeAllRunMenus() {
|
||||||
|
runMenuOpen = {};
|
||||||
|
}
|
||||||
|
// Global outside-click + Escape dismiss. Only registered while at
|
||||||
|
// least one menu is open — a background admin tab doesn't hold
|
||||||
|
// listeners.
|
||||||
|
$effect(() => {
|
||||||
|
const anyOpen = Object.values(runMenuOpen).some((v) => v);
|
||||||
|
if (!anyOpen) return;
|
||||||
|
const onDown = (e: MouseEvent) => {
|
||||||
|
if (!(e.target as HTMLElement).closest('.jobs-panel__split')) {
|
||||||
|
closeAllRunMenus();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const onKey = (e: KeyboardEvent) => {
|
||||||
|
if (e.key === 'Escape') closeAllRunMenus();
|
||||||
|
};
|
||||||
|
window.addEventListener('pointerdown', onDown);
|
||||||
|
window.addEventListener('keydown', onKey);
|
||||||
|
return () => {
|
||||||
|
window.removeEventListener('pointerdown', onDown);
|
||||||
|
window.removeEventListener('keydown', onKey);
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
// Purge-modal state. Null = closed; otherwise carries the
|
// Purge-modal state. Null = closed; otherwise carries the
|
||||||
// draft retention days the operator's picking. Kept separate
|
// draft retention days the operator's picking. Kept separate
|
||||||
// from the top-bar action state so mouse-away doesn't lose
|
// from the top-bar action state so mouse-away doesn't lose
|
||||||
@@ -234,8 +272,10 @@
|
|||||||
|
|
||||||
// ─── Actions ───────────────────────────────────────────────────────
|
// ─── Actions ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
async function onTrigger(name: string, opts: { deep?: boolean } = {}) {
|
async function onTrigger(name: string, opts: { deep?: boolean; repair?: boolean } = {}) {
|
||||||
const key = `trigger:${name}${opts.deep ? ':deep' : ''}`;
|
// Key suffix has to keep every dispatched variant distinct so the
|
||||||
|
// button-disabled state of one doesn't lock out another mid-flight.
|
||||||
|
const key = `trigger:${name}${opts.deep ? ':deep' : ''}${opts.repair ? ':repair' : ''}`;
|
||||||
markBusy(key, true);
|
markBusy(key, true);
|
||||||
try {
|
try {
|
||||||
// Fire the trigger + a follow-up loadJobs after a short delay
|
// Fire the trigger + a follow-up loadJobs after a short delay
|
||||||
@@ -265,10 +305,49 @@
|
|||||||
if (!res.outcome) {
|
if (!res.outcome) {
|
||||||
// dispatched (detached) — no outcome to render
|
// dispatched (detached) — no outcome to render
|
||||||
} else if (res.outcome.outcome === 'ok') {
|
} else if (res.outcome.outcome === 'ok') {
|
||||||
ui.notify(
|
// Repair runs surface a rollup so the operator sees
|
||||||
t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
|
// whether corrective UPDATEs actually fired. `extra`
|
||||||
'success'
|
// carries `repaired_count` on the two refcount tenants
|
||||||
);
|
// directly, and nested under `per_check[*].extra` when
|
||||||
|
// dispatched via `consistency_batch`. Sum across the
|
||||||
|
// per_check dict if present, else read the top-level.
|
||||||
|
let repairedTotal = 0;
|
||||||
|
let sawRepair = false;
|
||||||
|
const extra = (res.outcome.extra ?? {}) as {
|
||||||
|
repair_requested?: boolean;
|
||||||
|
repaired_count?: number;
|
||||||
|
per_check?: Record<
|
||||||
|
string,
|
||||||
|
{ extra?: { repair_requested?: boolean; repaired_count?: number } }
|
||||||
|
>;
|
||||||
|
};
|
||||||
|
if (extra.repair_requested) {
|
||||||
|
sawRepair = true;
|
||||||
|
repairedTotal += extra.repaired_count ?? 0;
|
||||||
|
}
|
||||||
|
if (extra.per_check) {
|
||||||
|
for (const child of Object.values(extra.per_check)) {
|
||||||
|
if (child?.extra?.repair_requested) {
|
||||||
|
sawRepair = true;
|
||||||
|
repairedTotal += child.extra.repaired_count ?? 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (sawRepair) {
|
||||||
|
ui.notify(
|
||||||
|
t(
|
||||||
|
'admin.jobs.triggered_ok_repair',
|
||||||
|
{ name, n: repairedTotal },
|
||||||
|
'{{name}}: {{n}} counter(s) repaired'
|
||||||
|
),
|
||||||
|
'success'
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
ui.notify(
|
||||||
|
t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
|
||||||
|
'success'
|
||||||
|
);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
ui.notify(
|
ui.notify(
|
||||||
t(
|
t(
|
||||||
@@ -557,6 +636,31 @@
|
|||||||
return name === 'consistency_batch' || name === 'blobs_consistency';
|
return name === 'consistency_batch' || name === 'blobs_consistency';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Jobs whose handler consults `args.repair` and applies a
|
||||||
|
// corrective UPDATE against the finding it just emitted. Only the
|
||||||
|
// two ref_count tenants today; `consistency_batch` also accepts
|
||||||
|
// the flag (fans out to both) and is surfaced separately as the
|
||||||
|
// top-bar "Repair ref_counts" button. Keep this list narrow —
|
||||||
|
// adding a job here without a matching backend handler produces a
|
||||||
|
// silently no-op button that confuses operators.
|
||||||
|
function supportsRepair(name: string): boolean {
|
||||||
|
return name === 'blobs_consistency' || name === 'manifests_consistency';
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onTriggerWithRepairConfirm(name: string) {
|
||||||
|
const ok = await confirmDialog({
|
||||||
|
title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
|
||||||
|
message: t(
|
||||||
|
'admin.jobs.run_repair_confirm_body_scoped',
|
||||||
|
{ name },
|
||||||
|
'Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.'
|
||||||
|
),
|
||||||
|
confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
|
||||||
|
danger: true
|
||||||
|
});
|
||||||
|
if (ok) await onTrigger(name, { repair: true });
|
||||||
|
}
|
||||||
|
|
||||||
function isRunning(job: JobSummary): boolean {
|
function isRunning(job: JobSummary): boolean {
|
||||||
return job.running;
|
return job.running;
|
||||||
}
|
}
|
||||||
@@ -614,6 +718,39 @@
|
|||||||
<Icon name="play" />
|
<Icon name="play" />
|
||||||
{t('admin.jobs.run_deep', 'Run deep')}
|
{t('admin.jobs.run_deep', 'Run deep')}
|
||||||
</button>
|
</button>
|
||||||
|
<!-- Repair goes behind a confirm because it issues corrective
|
||||||
|
UPDATEs on `storage.blobs.ref_count` and
|
||||||
|
`storage.chunk_manifests.ref_count`. Content-safe (only
|
||||||
|
counters change, matching the auditor's computed truth)
|
||||||
|
and race-safe (each UPDATE recomputes inside the same
|
||||||
|
statement), but writing-a-lot is still writing-a-lot.
|
||||||
|
One click, one confirm, one batch dispatched to both
|
||||||
|
refcount tenants via consistency_batch's arg
|
||||||
|
propagation. See `?repair=true` on
|
||||||
|
`POST /api/admin/jobs/{name}/trigger`. -->
|
||||||
|
<button
|
||||||
|
class="jobs-panel__btn jobs-panel__btn--warn"
|
||||||
|
disabled={busyKeys.has('trigger:consistency_batch:repair')}
|
||||||
|
title={t(
|
||||||
|
'admin.jobs.run_repair_hint',
|
||||||
|
'Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.'
|
||||||
|
)}
|
||||||
|
onclick={async () => {
|
||||||
|
const ok = await confirmDialog({
|
||||||
|
title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
|
||||||
|
message: t(
|
||||||
|
'admin.jobs.run_repair_confirm_body',
|
||||||
|
'Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.'
|
||||||
|
),
|
||||||
|
confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
|
||||||
|
danger: true
|
||||||
|
});
|
||||||
|
if (ok) await onTrigger('consistency_batch', { repair: true });
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Icon name="cog" />
|
||||||
|
{t('admin.jobs.run_repair', 'Repair ref_counts')}
|
||||||
|
</button>
|
||||||
{/if}
|
{/if}
|
||||||
<!-- Purge is orthogonal to consistency — it works even
|
<!-- Purge is orthogonal to consistency — it works even
|
||||||
when the batch coordinator isn't registered, so it
|
when the batch coordinator isn't registered, so it
|
||||||
@@ -754,22 +891,80 @@
|
|||||||
{t('admin.jobs.cancel', 'Cancel')}
|
{t('admin.jobs.cancel', 'Cancel')}
|
||||||
</button>
|
</button>
|
||||||
{:else}
|
{:else}
|
||||||
<button
|
<!-- Split-button: primary "Run" fires the default
|
||||||
class="jobs-panel__btn jobs-panel__btn--small"
|
trigger; the chevron opens a menu with the
|
||||||
disabled={busyKeys.has(`trigger:${job.name}`)}
|
tenant-specific variants (Run deep / Repair).
|
||||||
onclick={() => onTrigger(job.name)}
|
Rows without any variant render a bare Run
|
||||||
>
|
button — no chevron, no menu, no extra
|
||||||
{t('admin.jobs.run', 'Run')}
|
width. Preserves one-click discovery for
|
||||||
</button>
|
the common case. -->
|
||||||
{#if supportsDeep(job.name)}
|
{@const hasRunVariants = supportsDeep(job.name) || supportsRepair(job.name)}
|
||||||
|
<span class="jobs-panel__split">
|
||||||
<button
|
<button
|
||||||
class="jobs-panel__btn jobs-panel__btn--small"
|
class="jobs-panel__btn jobs-panel__btn--small"
|
||||||
disabled={busyKeys.has(`trigger:${job.name}:deep`)}
|
class:jobs-panel__split-main={hasRunVariants}
|
||||||
onclick={() => onTrigger(job.name, { deep: true })}
|
disabled={busyKeys.has(`trigger:${job.name}`)}
|
||||||
|
onclick={() => {
|
||||||
|
closeAllRunMenus();
|
||||||
|
void onTrigger(job.name);
|
||||||
|
}}
|
||||||
>
|
>
|
||||||
{t('admin.jobs.run_deep', 'Run deep')}
|
{t('admin.jobs.run', 'Run')}
|
||||||
</button>
|
</button>
|
||||||
{/if}
|
{#if hasRunVariants}
|
||||||
|
<button
|
||||||
|
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__split-toggle"
|
||||||
|
aria-haspopup="menu"
|
||||||
|
aria-expanded={runMenuOpen[job.name] ?? false}
|
||||||
|
aria-label={t('admin.jobs.run_variants_menu', 'Run variants menu')}
|
||||||
|
onclick={() => toggleRunMenu(job.name)}
|
||||||
|
>
|
||||||
|
<Icon name="caret-down" />
|
||||||
|
</button>
|
||||||
|
{#if runMenuOpen[job.name]}
|
||||||
|
<div class="jobs-panel__run-menu" role="menu">
|
||||||
|
{#if supportsDeep(job.name)}
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class="jobs-panel__run-menu-item"
|
||||||
|
role="menuitem"
|
||||||
|
disabled={busyKeys.has(`trigger:${job.name}:deep`)}
|
||||||
|
title={t(
|
||||||
|
'admin.jobs.run_deep_hint',
|
||||||
|
'Also runs slow variants (blob re-hash, bitrot detection).'
|
||||||
|
)}
|
||||||
|
onclick={() => {
|
||||||
|
closeAllRunMenus();
|
||||||
|
void onTrigger(job.name, { deep: true });
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Icon name="search" />
|
||||||
|
<span>{t('admin.jobs.run_deep', 'Run deep')}</span>
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
|
{#if supportsRepair(job.name)}
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class="jobs-panel__run-menu-item jobs-panel__run-menu-item--warn"
|
||||||
|
role="menuitem"
|
||||||
|
disabled={busyKeys.has(`trigger:${job.name}:repair`)}
|
||||||
|
title={t(
|
||||||
|
'admin.jobs.run_repair_hint',
|
||||||
|
'Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.'
|
||||||
|
)}
|
||||||
|
onclick={() => {
|
||||||
|
closeAllRunMenus();
|
||||||
|
void onTriggerWithRepairConfirm(job.name);
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Icon name="cog" />
|
||||||
|
<span>{t('admin.jobs.run_repair', 'Repair')}</span>
|
||||||
|
</button>
|
||||||
|
{/if}
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
{/if}
|
||||||
|
</span>
|
||||||
{/if}
|
{/if}
|
||||||
{#if isRunning(job) && canExpand}
|
{#if isRunning(job) && canExpand}
|
||||||
{#if isRecoverable(job)}
|
{#if isRecoverable(job)}
|
||||||
@@ -1304,6 +1499,90 @@
|
|||||||
color: var(--color-danger-text-alt);
|
color: var(--color-danger-text-alt);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Warn variant — used for actions that mutate data but are content-
|
||||||
|
safe / reversible-in-outcome (e.g. Repair ref_counts). Signals
|
||||||
|
"read the tooltip and the confirm before clicking" without the
|
||||||
|
danger red reserved for destructive delete-style buttons. */
|
||||||
|
.jobs-panel__btn--warn {
|
||||||
|
border-color: var(--color-warning-border);
|
||||||
|
color: var(--color-warning-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Split-button — inline flex holding a primary "Run" (fires default
|
||||||
|
action) and a chevron (opens the variants menu). `position:
|
||||||
|
relative` anchors the menu below the toggle. Only rendered on
|
||||||
|
rows whose job supports at least one variant; plain-Run rows
|
||||||
|
sidestep this whole structure. */
|
||||||
|
.jobs-panel__split {
|
||||||
|
display: inline-flex;
|
||||||
|
position: relative;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Attached-button trick: main loses its right border-radius, toggle
|
||||||
|
loses its left. Toggle also loses its left border so the two
|
||||||
|
don't render a double-thick divider. */
|
||||||
|
.jobs-panel__split-main {
|
||||||
|
border-top-right-radius: 0;
|
||||||
|
border-bottom-right-radius: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.jobs-panel__split-toggle {
|
||||||
|
border-top-left-radius: 0;
|
||||||
|
border-bottom-left-radius: 0;
|
||||||
|
border-left: none;
|
||||||
|
padding-left: 0.35rem;
|
||||||
|
padding-right: 0.35rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The variants menu — dropdown below the toggle, right-aligned so
|
||||||
|
it doesn't overflow the Actions column edge into the next row's
|
||||||
|
badge cell. Shadow + surface bg mirror the /files upload
|
||||||
|
dropdown (`upload-dropdown-menu`); using local CSS here rather
|
||||||
|
than the ported class so the jobs-panel keeps its scoped styling. */
|
||||||
|
.jobs-panel__run-menu {
|
||||||
|
position: absolute;
|
||||||
|
top: calc(100% + 2px);
|
||||||
|
right: 0;
|
||||||
|
z-index: 30;
|
||||||
|
min-width: 10rem;
|
||||||
|
background: var(--color-bg-surface);
|
||||||
|
border: 1px solid var(--color-border);
|
||||||
|
border-radius: var(--radius-md, 6px);
|
||||||
|
box-shadow: var(--shadow-md);
|
||||||
|
padding: 0.25rem 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.jobs-panel__run-menu-item {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
width: 100%;
|
||||||
|
padding: 0.4rem 0.75rem;
|
||||||
|
background: transparent;
|
||||||
|
border: none;
|
||||||
|
text-align: left;
|
||||||
|
font: inherit;
|
||||||
|
color: var(--color-text);
|
||||||
|
cursor: pointer;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.jobs-panel__run-menu-item:hover:not(:disabled) {
|
||||||
|
background: var(--color-bg-hover);
|
||||||
|
}
|
||||||
|
|
||||||
|
.jobs-panel__run-menu-item:disabled {
|
||||||
|
opacity: 0.5;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Warn colour on the menu item mirrors the button variant so the
|
||||||
|
Repair option carries the same "attention-worthy but not
|
||||||
|
destructive" visual weight as its top-bar counterpart. */
|
||||||
|
.jobs-panel__run-menu-item--warn {
|
||||||
|
color: var(--color-warning-text);
|
||||||
|
}
|
||||||
|
|
||||||
.jobs-panel__pill {
|
.jobs-panel__pill {
|
||||||
display: inline-block;
|
display: inline-block;
|
||||||
padding: 0.1rem 0.5rem;
|
padding: 0.1rem 0.5rem;
|
||||||
|
|||||||
@@ -1275,6 +1275,14 @@
|
|||||||
"run_all_consistency": "Run all consistency checks",
|
"run_all_consistency": "Run all consistency checks",
|
||||||
"run_deep": "Run deep",
|
"run_deep": "Run deep",
|
||||||
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
|
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
|
||||||
|
"run_repair": "Repair ref_counts",
|
||||||
|
"run_repair_hint": "Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.",
|
||||||
|
"run_repair_confirm_title": "Repair drifted ref_counts?",
|
||||||
|
"run_repair_confirm_body": "Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.",
|
||||||
|
"run_repair_confirm_body_scoped": "Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.",
|
||||||
|
"run_variants_menu": "Run variants menu",
|
||||||
|
"run_repair_confirm": "Repair",
|
||||||
|
"triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired",
|
||||||
"col_name": "Name",
|
"col_name": "Name",
|
||||||
"col_cadence": "Cadence",
|
"col_cadence": "Cadence",
|
||||||
"col_last_run": "Last run",
|
"col_last_run": "Last run",
|
||||||
|
|||||||
@@ -1197,6 +1197,14 @@
|
|||||||
"run_all_consistency": "Exécuter tous les contrôles de cohérence",
|
"run_all_consistency": "Exécuter tous les contrôles de cohérence",
|
||||||
"run_deep": "Analyse approfondie",
|
"run_deep": "Analyse approfondie",
|
||||||
"run_deep_hint": "Exécute également des variantes lentes (re-hachage de blob, détection bitrot).",
|
"run_deep_hint": "Exécute également des variantes lentes (re-hachage de blob, détection bitrot).",
|
||||||
|
"run_repair": "Réparer les compteurs",
|
||||||
|
"run_repair_hint": "Corrige les compteurs de références (blobs + manifestes) désynchronisés détectés par l'audit. Sûr pour les données — seuls les compteurs changent, pas le contenu.",
|
||||||
|
"run_repair_confirm_title": "Réparer les compteurs de références ?",
|
||||||
|
"run_repair_confirm_body": "Lance l'audit sur chaque blob et manifeste, puis applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu des blobs et les enregistrements de fichiers ne sont pas touchés. Vous pouvez exécuter cela à tout moment ; un passage en lecture seule s'exécute d'abord pour visualiser l'écart avant que la réparation ne l'écrase.",
|
||||||
|
"run_repair_confirm_body_scoped": "Lance {{name}} et applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu et les enregistrements de fichiers ne sont pas touchés.",
|
||||||
|
"run_variants_menu": "Menu des variantes d'exécution",
|
||||||
|
"run_repair_confirm": "Réparer",
|
||||||
|
"triggered_ok_repair": "{{name}} : {{n}} compteur(s) réparé(s)",
|
||||||
"col_name": "Nom",
|
"col_name": "Nom",
|
||||||
"col_cadence": "Cadence",
|
"col_cadence": "Cadence",
|
||||||
"col_last_run": "Dernière exécution",
|
"col_last_run": "Dernière exécution",
|
||||||
|
|||||||
@@ -0,0 +1,211 @@
|
|||||||
|
-- Fix: `storage.copy_folder_tree` never incremented `chunk_manifests.ref_count`.
|
||||||
|
--
|
||||||
|
-- The function bumped only `storage.blobs`:
|
||||||
|
--
|
||||||
|
-- UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
|
||||||
|
-- FROM (...) hc WHERE b.hash = hc.blob_hash;
|
||||||
|
--
|
||||||
|
-- but a CDC file's `blob_hash` names a MANIFEST, not a chunk. For any
|
||||||
|
-- multi-chunk file that predicate matches nothing, so a folder copy took
|
||||||
|
-- NO reference. Delete the original afterwards and `remove_reference`
|
||||||
|
-- walks the manifest to 0, `dedup_gc` reaps the manifest and every chunk
|
||||||
|
-- behind it — and the copy is unreadable. Silent data loss on an ordinary
|
||||||
|
-- UI operation.
|
||||||
|
--
|
||||||
|
-- Single-chunk files escaped by accident: their whole-file hash equals
|
||||||
|
-- their lone chunk's hash, so the UPDATE did match — bumping the wrong
|
||||||
|
-- counter, which shows up as a manifest under-count plus a blob
|
||||||
|
-- over-count rather than as loss.
|
||||||
|
--
|
||||||
|
-- Reproduced on a 5 MiB / 18-chunk file copied through the UI:
|
||||||
|
-- `chunk_manifests.ref_count` stayed at 1 while two `storage.files` rows
|
||||||
|
-- referenced it; `manifests_consistency` reported
|
||||||
|
-- `manifest_refcount_mismatch` with `delta: 1, reap_risk: true`.
|
||||||
|
--
|
||||||
|
-- This migration only rewrites the reference-counting block; everything
|
||||||
|
-- else is `20260902000001_copy_folder_tree_drop_user_id.sql` verbatim.
|
||||||
|
--
|
||||||
|
-- NOTE: existing drift is NOT repaired here. Run `manifests_consistency`
|
||||||
|
-- to find it — a data fix belongs with the recovery framework, not in a
|
||||||
|
-- schema migration that cannot know which counter is authoritative.
|
||||||
|
|
||||||
|
CREATE OR REPLACE FUNCTION storage.copy_folder_tree(
|
||||||
|
p_source_id UUID,
|
||||||
|
p_target_parent_id UUID, -- NULL = copy to root (keeps source drive)
|
||||||
|
p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name
|
||||||
|
) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$
|
||||||
|
DECLARE
|
||||||
|
v_root_lpath ltree;
|
||||||
|
v_root_depth INT;
|
||||||
|
v_max_depth INT;
|
||||||
|
v_level INT;
|
||||||
|
v_folders BIGINT := 0;
|
||||||
|
v_files BIGINT := 0;
|
||||||
|
v_inserted BIGINT;
|
||||||
|
v_new_root UUID;
|
||||||
|
v_dest_drive_id UUID;
|
||||||
|
BEGIN
|
||||||
|
-- Validate source exists.
|
||||||
|
SELECT fo.lpath, nlevel(fo.lpath)
|
||||||
|
INTO v_root_lpath, v_root_depth
|
||||||
|
FROM storage.folders fo
|
||||||
|
WHERE fo.id = p_source_id AND NOT fo.is_trashed;
|
||||||
|
|
||||||
|
IF v_root_lpath IS NULL THEN
|
||||||
|
RAISE EXCEPTION 'Source folder not found: %', p_source_id
|
||||||
|
USING ERRCODE = 'P0002'; -- no_data_found
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- Resolve destination drive_id once up front (cross-drive copy path).
|
||||||
|
IF p_target_parent_id IS NULL THEN
|
||||||
|
SELECT fo.drive_id INTO v_dest_drive_id
|
||||||
|
FROM storage.folders fo
|
||||||
|
WHERE fo.id = p_source_id;
|
||||||
|
ELSE
|
||||||
|
SELECT fo.drive_id INTO v_dest_drive_id
|
||||||
|
FROM storage.folders fo
|
||||||
|
WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed;
|
||||||
|
IF v_dest_drive_id IS NULL THEN
|
||||||
|
RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id
|
||||||
|
USING ERRCODE = 'P0002';
|
||||||
|
END IF;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- Temp mapping: every folder in the subtree → new UUID.
|
||||||
|
CREATE TEMP TABLE IF NOT EXISTS _copy_map(
|
||||||
|
old_id UUID PRIMARY KEY,
|
||||||
|
new_id UUID NOT NULL DEFAULT gen_random_uuid()
|
||||||
|
) ON COMMIT DROP;
|
||||||
|
TRUNCATE _copy_map;
|
||||||
|
|
||||||
|
INSERT INTO _copy_map(old_id)
|
||||||
|
SELECT fo.id
|
||||||
|
FROM storage.folders fo
|
||||||
|
WHERE NOT fo.is_trashed
|
||||||
|
AND fo.lpath <@ v_root_lpath;
|
||||||
|
|
||||||
|
SELECT cm.new_id INTO v_new_root
|
||||||
|
FROM _copy_map cm WHERE cm.old_id = p_source_id;
|
||||||
|
|
||||||
|
SELECT MAX(nlevel(fo.lpath))
|
||||||
|
INTO v_max_depth
|
||||||
|
FROM storage.folders fo
|
||||||
|
JOIN _copy_map cm ON fo.id = cm.old_id;
|
||||||
|
|
||||||
|
-- ── Insert folders level by level ──
|
||||||
|
-- Post-D7: `user_id` intentionally omitted from the column list so
|
||||||
|
-- copied rows leave the (now-nullable) column NULL. Provenance is
|
||||||
|
-- carried by `created_by` / `updated_by` (§14 columns) — preserved
|
||||||
|
-- from source so authorship survives the copy.
|
||||||
|
FOR v_level IN v_root_depth .. v_max_depth LOOP
|
||||||
|
INSERT INTO storage.folders(
|
||||||
|
id, name, parent_id,
|
||||||
|
drive_id, created_by, updated_by
|
||||||
|
)
|
||||||
|
SELECT cm.new_id,
|
||||||
|
CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL
|
||||||
|
THEN p_dest_name ELSE fo.name END,
|
||||||
|
CASE WHEN fo.id = p_source_id THEN p_target_parent_id
|
||||||
|
ELSE pm.new_id END,
|
||||||
|
v_dest_drive_id,
|
||||||
|
fo.created_by,
|
||||||
|
fo.updated_by
|
||||||
|
FROM storage.folders fo
|
||||||
|
JOIN _copy_map cm ON fo.id = cm.old_id
|
||||||
|
LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id
|
||||||
|
WHERE NOT fo.is_trashed
|
||||||
|
AND nlevel(fo.lpath) = v_level;
|
||||||
|
|
||||||
|
GET DIAGNOSTICS v_inserted = ROW_COUNT;
|
||||||
|
v_folders := v_folders + v_inserted;
|
||||||
|
END LOOP;
|
||||||
|
|
||||||
|
-- Temp mapping for files src→dst (dst ids pre-allocated so we can
|
||||||
|
-- reference them in the dead-property duplication below).
|
||||||
|
CREATE TEMP TABLE IF NOT EXISTS _copy_file_map(
|
||||||
|
old_id UUID PRIMARY KEY,
|
||||||
|
new_id UUID NOT NULL DEFAULT gen_random_uuid()
|
||||||
|
) ON COMMIT DROP;
|
||||||
|
TRUNCATE _copy_file_map;
|
||||||
|
|
||||||
|
INSERT INTO _copy_file_map(old_id)
|
||||||
|
SELECT f.id
|
||||||
|
FROM storage.files f
|
||||||
|
JOIN _copy_map cm ON f.folder_id = cm.old_id
|
||||||
|
WHERE NOT f.is_trashed;
|
||||||
|
|
||||||
|
-- ── Batch copy all files (zero-copy: same blob_hash) ──
|
||||||
|
-- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`.
|
||||||
|
INSERT INTO storage.files(
|
||||||
|
id, name, folder_id, blob_hash, size, mime_type,
|
||||||
|
media_sort_date, drive_id, created_by, updated_by
|
||||||
|
)
|
||||||
|
SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size,
|
||||||
|
f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by,
|
||||||
|
f.updated_by
|
||||||
|
FROM storage.files f
|
||||||
|
JOIN _copy_map cm ON f.folder_id = cm.old_id
|
||||||
|
JOIN _copy_file_map fm ON fm.old_id = f.id
|
||||||
|
WHERE NOT f.is_trashed;
|
||||||
|
|
||||||
|
GET DIAGNOSTICS v_files = ROW_COUNT;
|
||||||
|
|
||||||
|
-- Batch increment reference counts — MANIFEST FIRST, blobs only as
|
||||||
|
-- fallback. This mirrors `DedupService::add_reference`, and the order
|
||||||
|
-- is the whole point:
|
||||||
|
--
|
||||||
|
-- A CDC file's `blob_hash` names a MANIFEST (`chunk_manifests.file_hash`),
|
||||||
|
-- not a chunk. The previous version of this block updated only
|
||||||
|
-- `storage.blobs`, so for a multi-chunk file the predicate
|
||||||
|
-- `b.hash = hc.blob_hash` matched ZERO rows and the copy took no
|
||||||
|
-- reference at all. Deleting the original then walked the manifest's
|
||||||
|
-- ref_count to 0, dedup_gc reaped the manifest and every chunk behind
|
||||||
|
-- it, and the copy became unreadable. Reproduced via the UI folder
|
||||||
|
-- copy on a 5 MiB (18-chunk) file: ref_count stayed 1 with two files
|
||||||
|
-- referencing it.
|
||||||
|
--
|
||||||
|
-- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing.
|
||||||
|
-- For a SINGLE-chunk file the whole-file hash equals its lone chunk's
|
||||||
|
-- hash, so without it the copy would be counted at both levels and
|
||||||
|
-- turn an under-count into an over-count.
|
||||||
|
IF v_files > 0 THEN
|
||||||
|
WITH hc AS (
|
||||||
|
SELECT f.blob_hash, COUNT(*)::int AS cnt
|
||||||
|
FROM storage.files f
|
||||||
|
JOIN _copy_map cm ON f.folder_id = cm.new_id
|
||||||
|
WHERE NOT f.is_trashed
|
||||||
|
GROUP BY f.blob_hash
|
||||||
|
),
|
||||||
|
bumped AS (
|
||||||
|
UPDATE storage.chunk_manifests m
|
||||||
|
SET ref_count = m.ref_count + hc.cnt
|
||||||
|
FROM hc
|
||||||
|
WHERE m.file_hash = hc.blob_hash
|
||||||
|
RETURNING m.file_hash
|
||||||
|
)
|
||||||
|
UPDATE storage.blobs b
|
||||||
|
SET ref_count = b.ref_count + hc.cnt,
|
||||||
|
-- Matches add_reference: a blob resurrected inside its GC
|
||||||
|
-- grace window must lose its orphan stamp.
|
||||||
|
orphaned_at = NULL
|
||||||
|
FROM hc
|
||||||
|
WHERE b.hash = hc.blob_hash
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM bumped WHERE file_hash = hc.blob_hash);
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- Duplicate dead properties per RFC 4918 §8.8 — id-keyed store.
|
||||||
|
INSERT INTO storage.webdav_dead_properties
|
||||||
|
(folder_id, namespace, local_name, value)
|
||||||
|
SELECT cm.new_id, dp.namespace, dp.local_name, dp.value
|
||||||
|
FROM storage.webdav_dead_properties dp
|
||||||
|
JOIN _copy_map cm ON dp.folder_id = cm.old_id;
|
||||||
|
|
||||||
|
INSERT INTO storage.webdav_dead_properties
|
||||||
|
(file_id, namespace, local_name, value)
|
||||||
|
SELECT fm.new_id, dp.namespace, dp.local_name, dp.value
|
||||||
|
FROM storage.webdav_dead_properties dp
|
||||||
|
JOIN _copy_file_map fm ON dp.file_id = fm.old_id;
|
||||||
|
|
||||||
|
RETURN QUERY SELECT v_new_root::text, v_folders, v_files;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
-- Fix: `trg_files_decrement_blob_ref` decremented the wrong counter for
|
||||||
|
-- CDC files.
|
||||||
|
--
|
||||||
|
-- The original trigger (2026-03-07 initial schema) unconditionally ran:
|
||||||
|
--
|
||||||
|
-- UPDATE storage.blobs
|
||||||
|
-- SET ref_count = GREATEST(ref_count - 1, 0)
|
||||||
|
-- WHERE hash = OLD.blob_hash;
|
||||||
|
--
|
||||||
|
-- That's correct for a legacy whole-file blob, where `OLD.blob_hash`
|
||||||
|
-- names a `storage.blobs` row directly. For a CDC file, `OLD.blob_hash`
|
||||||
|
-- names a `storage.chunk_manifests.file_hash` — the blob table row (if
|
||||||
|
-- one exists at all) holds a DIFFERENT counter, incremented by the
|
||||||
|
-- MANIFEST's presence in its own `chunk_hashes[]`, not by the file.
|
||||||
|
--
|
||||||
|
-- Consequences before this fix:
|
||||||
|
-- 1. `storage.chunk_manifests.ref_count` never decremented on file
|
||||||
|
-- DELETE → over-count grows unboundedly across delete/purge
|
||||||
|
-- cycles.
|
||||||
|
-- 2. `storage.blobs.ref_count` decremented for hashes it shouldn't
|
||||||
|
-- (CDC whole-file hashes) → the counter drops toward 0 while the
|
||||||
|
-- manifest still legitimately references the chunk. GC then reaps
|
||||||
|
-- a live blob → downloadable-then-404 data loss.
|
||||||
|
--
|
||||||
|
-- Both bugs surfaced by `tests/api/refcount_cascade.hurl` on the
|
||||||
|
-- 135-byte fixture (single-chunk CDC file, worst case for confusion
|
||||||
|
-- because the whole-file hash equals its lone chunk's hash). The
|
||||||
|
-- 2026-08-22 sandbox drift (`storage.blobs.ref_count = 0`,
|
||||||
|
-- `actual_auditor = 1`) is the same bug at rest.
|
||||||
|
--
|
||||||
|
-- Sibling fix: `20261016000000_copy_folder_tree_manifest_refcount.sql`
|
||||||
|
-- fixed the mirror-image INCREMENT bug in `storage.copy_folder_tree`.
|
||||||
|
-- This migration closes the decrement half.
|
||||||
|
--
|
||||||
|
-- Cross-references:
|
||||||
|
-- - `DedupService::add_reference` (dedup_service.rs:1703) — app-layer
|
||||||
|
-- twin for the increment direction: manifest first, blob fallback.
|
||||||
|
-- - `manifests_consistency` tenant (2026-08-23) — surfaces any
|
||||||
|
-- residual drift after this fix lands.
|
||||||
|
--
|
||||||
|
-- ── DESIGN NOTE — decrement only, no manifest reap here ──
|
||||||
|
--
|
||||||
|
-- The trigger DELIBERATELY does not delete manifests or walk chunks on
|
||||||
|
-- a last-ref decrement. Both actions used to live inside
|
||||||
|
-- `DedupService::cleanup_if_orphaned` and its callee
|
||||||
|
-- `remove_manifest_reference`, and both fire `fire_blob_hooks` —
|
||||||
|
-- the Rust callback that reaps disk artefacts keyed by the whole-file
|
||||||
|
-- content hash (thumbnails, face embeddings, audio tags, media
|
||||||
|
-- metadata). SQL triggers can't invoke Rust callbacks, so if this
|
||||||
|
-- trigger reaped the manifest itself, dedup_gc Phase 1
|
||||||
|
-- (`dedup_service.rs:2660-2772`) — the ONLY code path that knows to
|
||||||
|
-- fire `fire_blob_hooks` for a reaped manifest's `file_hash` — would
|
||||||
|
-- find nothing to do on its next sweep, and every derived artefact
|
||||||
|
-- would leak on disk. `storage_cleanup_check.sh`'s "N thumbnail
|
||||||
|
-- file(s) remain on disk" gate catches this class immediately.
|
||||||
|
--
|
||||||
|
-- Contract: trigger decrements the correct counter atomically inside
|
||||||
|
-- the DELETE txn. GC (`dedup_gc`) is responsible for:
|
||||||
|
-- • finding manifests whose ref_count hit 0 (or that no reference
|
||||||
|
-- source references, covering bulk-delete paths),
|
||||||
|
-- • deleting them,
|
||||||
|
-- • decrementing each chunk in `chunk_hashes[]`,
|
||||||
|
-- • firing `fire_blob_hooks(file_hash)` so Rust callbacks reap
|
||||||
|
-- derived disk artefacts,
|
||||||
|
-- • the corresponding legacy-blob path for ref_count = 0 blobs.
|
||||||
|
--
|
||||||
|
-- NOTE: pre-existing drift is NOT repaired here. Run `manifests_
|
||||||
|
-- consistency` + `blobs_consistency` after deploy; feed the findings
|
||||||
|
-- into the recovery framework.
|
||||||
|
|
||||||
|
CREATE OR REPLACE FUNCTION storage.decrement_blob_ref()
|
||||||
|
RETURNS trigger AS $$
|
||||||
|
BEGIN
|
||||||
|
-- Manifest-first, mirroring the increment side. We touch ONE
|
||||||
|
-- counter and return — the manifest reap + chunk walk + hook
|
||||||
|
-- firing lives in `dedup_gc` where Rust callbacks can run.
|
||||||
|
IF EXISTS (
|
||||||
|
SELECT 1 FROM storage.chunk_manifests
|
||||||
|
WHERE file_hash = OLD.blob_hash
|
||||||
|
) THEN
|
||||||
|
UPDATE storage.chunk_manifests
|
||||||
|
SET ref_count = GREATEST(ref_count - 1, 0)
|
||||||
|
WHERE file_hash = OLD.blob_hash;
|
||||||
|
ELSE
|
||||||
|
-- Legacy whole-file blob path: no manifest, blob is referenced
|
||||||
|
-- directly by this file row. Preserves the original behaviour
|
||||||
|
-- verbatim for the pre-CDC path.
|
||||||
|
UPDATE storage.blobs
|
||||||
|
SET ref_count = GREATEST(ref_count - 1, 0),
|
||||||
|
orphaned_at = CASE
|
||||||
|
WHEN GREATEST(ref_count - 1, 0) = 0
|
||||||
|
THEN now()
|
||||||
|
ELSE orphaned_at
|
||||||
|
END
|
||||||
|
WHERE hash = OLD.blob_hash;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
RETURN OLD;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
COMMENT ON FUNCTION storage.decrement_blob_ref() IS
|
||||||
|
'Decrement the correct ref_count when a file is deleted. '
|
||||||
|
'Manifest-aware (2026-10-17): dispatches to chunk_manifests.ref_count '
|
||||||
|
'when the file''s blob_hash names a manifest, else to '
|
||||||
|
'storage.blobs.ref_count for legacy whole-file blobs. Decrement-only: '
|
||||||
|
'physical cleanup + Rust lifecycle hooks fire from dedup_gc, which '
|
||||||
|
'can invoke callbacks a SQL trigger cannot.';
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
-- One-time repair of ref_count drift accumulated under the pre-fix
|
||||||
|
-- copy/delete code paths.
|
||||||
|
--
|
||||||
|
-- Why this is atomic with the upgrade rather than a manual admin action
|
||||||
|
-- ─────────────────────────────────────────────────────────────────────
|
||||||
|
-- The two prior migrations on this branch:
|
||||||
|
-- * `20261016000000_copy_folder_tree_manifest_refcount.sql`
|
||||||
|
-- (fix the INCREMENT path — copy was bumping the wrong counter for
|
||||||
|
-- CDC files)
|
||||||
|
-- * `20261017000000_file_delete_trigger_manifest_aware.sql`
|
||||||
|
-- (fix the DECREMENT path — trigger was decrementing the wrong
|
||||||
|
-- counter for CDC files; folder-cascade + trash-empty paths
|
||||||
|
-- inherited that drift silently)
|
||||||
|
-- both close the bugs going forward, but production DBs upgrading
|
||||||
|
-- through this branch may carry accumulated drift from every prior
|
||||||
|
-- copy → delete cycle a CDC file went through. Under-count is the
|
||||||
|
-- dangerous direction: the next `dedup_gc` pass would reap a live
|
||||||
|
-- blob → user-facing 404 → silent data loss.
|
||||||
|
--
|
||||||
|
-- Waiting for an operator to open the admin panel and click "Repair
|
||||||
|
-- ref_counts" is the wrong default for a data-loss-preventing fix.
|
||||||
|
-- Ed's rule (`[[feedback_no_silent_auto_repair]]`): consistency
|
||||||
|
-- tenants must default to discovery-only so future bugs surface — but
|
||||||
|
-- fixing KNOWN pre-existing drift on the upgrade itself is the
|
||||||
|
-- bounded exception, because at that specific moment the source of
|
||||||
|
-- drift is known + closed, and there is no upstream mystery to
|
||||||
|
-- preserve.
|
||||||
|
--
|
||||||
|
-- Content-safety guarantees:
|
||||||
|
-- * Only counter columns change (`storage.chunk_manifests.ref_count`,
|
||||||
|
-- `storage.blobs.ref_count`). No file rows, no blob rows, no
|
||||||
|
-- manifest rows, no chunk arrays, no backend files.
|
||||||
|
-- * The corrective UPDATE sets `stored = actual` where `actual` is
|
||||||
|
-- computed from the SAME auditor formulas that
|
||||||
|
-- `manifests_consistency` / `blobs_consistency` use, so this
|
||||||
|
-- migration and those tenants agree by construction.
|
||||||
|
-- * Race-safe against concurrent writes (migrations run
|
||||||
|
-- single-connection at startup before the server serves any
|
||||||
|
-- traffic; nobody else is writing).
|
||||||
|
-- * Idempotent — fresh installs and already-clean DBs no-op (both
|
||||||
|
-- `stored` and `actual` are equal, the `WHERE <>` filters
|
||||||
|
-- everything out).
|
||||||
|
--
|
||||||
|
-- The panel button + `?repair=true` on the trigger endpoints stay for
|
||||||
|
-- FUTURE drift (regression detector; not for repeat use on this
|
||||||
|
-- accumulated set).
|
||||||
|
|
||||||
|
DO $$
|
||||||
|
DECLARE
|
||||||
|
v_m_fixed int;
|
||||||
|
v_b_fixed int;
|
||||||
|
BEGIN
|
||||||
|
-- Manifest counter: `actual` = # files whose blob_hash names this
|
||||||
|
-- manifest's file_hash. Same formula as
|
||||||
|
-- `manifests_consistency_service::manifest_page_sql` (via the
|
||||||
|
-- BlobReferenceRegistry at RefLevel::Manifest) — inline here
|
||||||
|
-- because migrations can't call Rust.
|
||||||
|
UPDATE storage.chunk_manifests m
|
||||||
|
SET ref_count = (SELECT COUNT(*) FROM storage.files
|
||||||
|
WHERE blob_hash = m.file_hash)
|
||||||
|
WHERE m.ref_count <> (SELECT COUNT(*) FROM storage.files
|
||||||
|
WHERE blob_hash = m.file_hash);
|
||||||
|
GET DIAGNOSTICS v_m_fixed = ROW_COUNT;
|
||||||
|
|
||||||
|
-- Blob counter: two-term formula mirroring
|
||||||
|
-- `blobs_consistency_service.rs:395-408`:
|
||||||
|
-- (files pointing at this blob AND having NO manifest for their
|
||||||
|
-- blob_hash — legacy whole-file path)
|
||||||
|
-- + (manifests including this hash as a chunk in chunk_hashes[])
|
||||||
|
UPDATE storage.blobs b
|
||||||
|
SET ref_count = (
|
||||||
|
(SELECT COUNT(*) FROM storage.files f
|
||||||
|
WHERE f.blob_hash = b.hash
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests m
|
||||||
|
WHERE m.file_hash = f.blob_hash))
|
||||||
|
+ (SELECT COUNT(*) FROM storage.chunk_manifests m
|
||||||
|
WHERE b.hash = ANY(m.chunk_hashes))
|
||||||
|
)
|
||||||
|
WHERE b.ref_count <> (
|
||||||
|
(SELECT COUNT(*) FROM storage.files f
|
||||||
|
WHERE f.blob_hash = b.hash
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests m
|
||||||
|
WHERE m.file_hash = f.blob_hash))
|
||||||
|
+ (SELECT COUNT(*) FROM storage.chunk_manifests m
|
||||||
|
WHERE b.hash = ANY(m.chunk_hashes))
|
||||||
|
);
|
||||||
|
GET DIAGNOSTICS v_b_fixed = ROW_COUNT;
|
||||||
|
|
||||||
|
-- Landed in the deploy log so an operator upgrading a huge instance
|
||||||
|
-- can see the migration did work — silent no-op on fresh installs.
|
||||||
|
RAISE NOTICE '[refcount_repair] fixed % manifest(s), % blob(s)',
|
||||||
|
v_m_fixed, v_b_fixed;
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
@@ -0,0 +1,313 @@
|
|||||||
|
//! `BlobReferenceSource` — the extension point that teaches ref-counting
|
||||||
|
//! and the consistency jobs about a table holding blob references.
|
||||||
|
//!
|
||||||
|
//! Before this port, "who references this hash" was hardcoded SQL in two
|
||||||
|
//! places (`dedup_gc`'s reap predicate and `blobs_consistency`'s refcount
|
||||||
|
//! recompute), both naming `storage.files` and `storage.chunk_manifests`
|
||||||
|
//! directly. Any new blob-owning table therefore risked silent orphaning:
|
||||||
|
//! `dedup_gc` sees `ref_count = 0`, or a manifest with no `storage.files`
|
||||||
|
//! row behind it, and reaps live content.
|
||||||
|
//!
|
||||||
|
//! See `docs/plan/derived-blobs.md` for the design and the coverage matrix.
|
||||||
|
//!
|
||||||
|
//! # Two levels, and why a source may span both
|
||||||
|
//!
|
||||||
|
//! [`DedupService::add_reference`] bumps `chunk_manifests.ref_count` first
|
||||||
|
//! and only falls back to `storage.blobs.ref_count`. So a reference lands
|
||||||
|
//! on whichever counter its hash names, and the two must be recomputed
|
||||||
|
//! separately — mixing them double-counts, systematically:
|
||||||
|
//!
|
||||||
|
//! * A **Blob** (`chunk_manifests.file_hash`) is "the content of a file".
|
||||||
|
//! * A **Chunk** (`storage.blobs.hash`) is a physical byte payload.
|
||||||
|
//! * For a single-chunk Blob the two hashes are **equal**, because both are
|
||||||
|
//! BLAKE3 over the same bytes. That aliasing is why today's chunk-level
|
||||||
|
//! recompute carries a `NOT EXISTS` clause, and why every fragment here
|
||||||
|
//! must be level-correct rather than merely plausible.
|
||||||
|
//!
|
||||||
|
//! A source is not confined to one level: [`RefLevel::Chunk`] and
|
||||||
|
//! [`RefLevel::Manifest`] fragments are requested independently, and
|
||||||
|
//! `storage.files` legitimately contributes to both — a manifest-less
|
||||||
|
//! legacy row references a chunk, a CDC row references a Blob.
|
||||||
|
//!
|
||||||
|
//! # Why SQL fragments rather than a per-hash count
|
||||||
|
//!
|
||||||
|
//! `blobs_consistency` recomputes refcounts with **one query per page**,
|
||||||
|
//! the expected count inlined as correlated subqueries. Asking each source
|
||||||
|
//! for a count per hash would turn that into `sources × rows` round-trips —
|
||||||
|
//! a catastrophic regression on a table with millions of rows. So sources
|
||||||
|
//! contribute a *fragment* that the registry sums into the existing page
|
||||||
|
//! query, and [`BlobReferenceSource::count_references`] exists only for the
|
||||||
|
//! on-demand path (`dedup_gc` checking a single reap candidate, where the
|
||||||
|
//! candidate set is already filtered to `ref_count = 0`).
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Which counter a source's references land on.
|
||||||
|
///
|
||||||
|
/// Not a property of the source — see the module docs; the same source may
|
||||||
|
/// contribute at both levels.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||||
|
pub enum RefLevel {
|
||||||
|
/// References a physical chunk. Feeds `storage.blobs.ref_count`.
|
||||||
|
Chunk,
|
||||||
|
/// References a Blob via its manifest. Feeds
|
||||||
|
/// `chunk_manifests.ref_count`.
|
||||||
|
Manifest,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RefLevel {
|
||||||
|
/// Both levels, for callers that sweep each in turn.
|
||||||
|
pub const ALL: [RefLevel; 2] = [RefLevel::Chunk, RefLevel::Manifest];
|
||||||
|
|
||||||
|
/// Stable name for logs and consistency-finding fields.
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
RefLevel::Chunk => "chunk",
|
||||||
|
RefLevel::Manifest => "manifest",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One table that holds references to blob hashes.
|
||||||
|
///
|
||||||
|
/// Implementors are registered on [`BlobReferenceRegistry`] during DI.
|
||||||
|
/// Adding a blob-owning table **without** registering it is the failure
|
||||||
|
/// this port exists to prevent.
|
||||||
|
#[async_trait]
|
||||||
|
pub trait BlobReferenceSource: Send + Sync {
|
||||||
|
/// Short stable identifier for logs and consistency-finding `source`
|
||||||
|
/// fields — `"files"`, `"chunks"`, `"content_derived"`, …
|
||||||
|
///
|
||||||
|
/// Stable across releases: log aggregators key off it.
|
||||||
|
fn source_name(&self) -> &'static str;
|
||||||
|
|
||||||
|
/// A correlated-subquery fragment counting this source's references
|
||||||
|
/// **at `level`** to `outer_hash_expr`, or `None` when this source
|
||||||
|
/// holds no references at that level.
|
||||||
|
///
|
||||||
|
/// `outer_hash_expr` is the SQL expression naming the hash of the row
|
||||||
|
/// being recomputed — `"b.hash"` when sweeping `storage.blobs`,
|
||||||
|
/// `"m.file_hash"` when sweeping `storage.chunk_manifests`. The
|
||||||
|
/// fragment must be a parenthesised scalar subquery so the registry can
|
||||||
|
/// join fragments with `+`.
|
||||||
|
///
|
||||||
|
/// **Identifiers only.** `outer_hash_expr` is supplied by the sweep, never
|
||||||
|
/// by a request; no fragment may interpolate caller input.
|
||||||
|
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String>;
|
||||||
|
|
||||||
|
/// Existence form of [`Self::ref_count_sql`] — a boolean fragment, true
|
||||||
|
/// when this source holds at least one reference at `level`.
|
||||||
|
///
|
||||||
|
/// Defaults to `(<count>) > 0`. Override when the source can express a
|
||||||
|
/// short-circuiting `EXISTS`, which the planner can stop at the first
|
||||||
|
/// matching row: `dedup_gc`'s reap predicate runs this per candidate
|
||||||
|
/// manifest, and a heavily-deduplicated blob has many referrers, so
|
||||||
|
/// counting all of them where existence would do is a real regression.
|
||||||
|
fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
self.ref_count_sql(level, outer_hash_expr)
|
||||||
|
.map(|fragment| format!("{fragment} > 0"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Count of references this source holds on `blob_hash`, across both
|
||||||
|
/// levels.
|
||||||
|
///
|
||||||
|
/// **On-demand path only** — `dedup_gc` checking a single reap
|
||||||
|
/// candidate. The consistency sweeps must use [`Self::ref_count_sql`];
|
||||||
|
/// calling this per row would turn one query per page into
|
||||||
|
/// `sources × rows` round-trips.
|
||||||
|
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;
|
||||||
|
|
||||||
|
/// Iterate the hashes this source references, paged by the
|
||||||
|
/// implementation's natural cursor (typically a primary key).
|
||||||
|
///
|
||||||
|
/// Used by `backend_consistency` to walk the backend against the union
|
||||||
|
/// of all sources. Returns the page plus the cursor to resume from,
|
||||||
|
/// `None` when exhausted.
|
||||||
|
async fn list_referenced_blobs(
|
||||||
|
&self,
|
||||||
|
cursor: Option<Vec<u8>>,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;
|
||||||
|
|
||||||
|
/// Notification that `dedup_gc` reaped this blob.
|
||||||
|
///
|
||||||
|
/// Sources maintaining a denormalised refcount can clean up here. Most
|
||||||
|
/// leave the default noop — the mapping row is normally deleted by the
|
||||||
|
/// owning service's `on_blob_deleted` hook instead.
|
||||||
|
fn on_blob_reaped(&self, _blob_hash: &str) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The set of registered [`BlobReferenceSource`]s.
|
||||||
|
///
|
||||||
|
/// Assembled once during DI and shared (`Arc`) by `dedup_gc` and the
|
||||||
|
/// consistency jobs, so all three agree on what "referenced" means.
|
||||||
|
#[derive(Default)]
|
||||||
|
pub struct BlobReferenceRegistry {
|
||||||
|
sources: Vec<Arc<dyn BlobReferenceSource>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobReferenceRegistry {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Register a source. Order is irrelevant — fragments are summed and
|
||||||
|
/// counts added.
|
||||||
|
pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>) {
|
||||||
|
self.sources.push(source);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn sources(&self) -> &[Arc<dyn BlobReferenceSource>] {
|
||||||
|
&self.sources
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The summed SQL expression counting every source's references at
|
||||||
|
/// `level` to `outer_hash_expr`.
|
||||||
|
///
|
||||||
|
/// Returns `"0"` when no source contributes at this level, which keeps
|
||||||
|
/// the caller's query valid without a special case.
|
||||||
|
pub fn ref_count_expr(&self, level: RefLevel, outer_hash_expr: &str) -> String {
|
||||||
|
let fragments: Vec<String> = self
|
||||||
|
.sources
|
||||||
|
.iter()
|
||||||
|
.filter_map(|s| s.ref_count_sql(level, outer_hash_expr))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
if fragments.is_empty() {
|
||||||
|
"0".to_string()
|
||||||
|
} else {
|
||||||
|
fragments.join("\n + ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Predicate selecting rows that **no** registered source references at
|
||||||
|
/// `level` — i.e. reap candidates.
|
||||||
|
///
|
||||||
|
/// Returns `None` when no source contributes at this level, and callers
|
||||||
|
/// **must** treat that as "refuse to act" rather than substituting a
|
||||||
|
/// default. The natural default would be the sum-equals-zero form, which
|
||||||
|
/// on an empty registry reduces to `0 = 0` — vacuously true for every
|
||||||
|
/// row, i.e. "delete everything". Returning `None` makes that
|
||||||
|
/// unrepresentable at the call site instead of merely discouraged.
|
||||||
|
pub fn no_reference_predicate(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
let fragments: Vec<String> = self
|
||||||
|
.sources
|
||||||
|
.iter()
|
||||||
|
.filter_map(|s| s.ref_exists_sql(level, outer_hash_expr))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
if fragments.is_empty() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(format!("NOT ({})", fragments.join("\n OR ")))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Total references held on `hash` across every source.
|
||||||
|
///
|
||||||
|
/// On-demand path only — see [`BlobReferenceSource::count_references`].
|
||||||
|
pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError> {
|
||||||
|
let mut total = 0u64;
|
||||||
|
for source in &self.sources {
|
||||||
|
total = total.saturating_add(source.count_references(hash).await?);
|
||||||
|
}
|
||||||
|
Ok(total)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fan out a reap notification to every source.
|
||||||
|
pub fn notify_reaped(&self, blob_hash: &str) {
|
||||||
|
for source in &self.sources {
|
||||||
|
source.on_blob_reaped(blob_hash);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
struct Stub {
|
||||||
|
name: &'static str,
|
||||||
|
chunk: Option<&'static str>,
|
||||||
|
manifest: Option<&'static str>,
|
||||||
|
count: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl BlobReferenceSource for Stub {
|
||||||
|
fn source_name(&self) -> &'static str {
|
||||||
|
self.name
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ref_count_sql(&self, level: RefLevel, outer: &str) -> Option<String> {
|
||||||
|
let tmpl = match level {
|
||||||
|
RefLevel::Chunk => self.chunk?,
|
||||||
|
RefLevel::Manifest => self.manifest?,
|
||||||
|
};
|
||||||
|
Some(tmpl.replace("{outer}", outer))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn count_references(&self, _blob_hash: &str) -> Result<u64, DomainError> {
|
||||||
|
Ok(self.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn list_referenced_blobs(
|
||||||
|
&self,
|
||||||
|
_cursor: Option<Vec<u8>>,
|
||||||
|
_limit: usize,
|
||||||
|
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||||
|
Ok((Vec::new(), None))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn registry() -> BlobReferenceRegistry {
|
||||||
|
let mut r = BlobReferenceRegistry::new();
|
||||||
|
r.register(Arc::new(Stub {
|
||||||
|
name: "a",
|
||||||
|
chunk: Some("(SELECT 1 WHERE {outer} = 'x')"),
|
||||||
|
manifest: None,
|
||||||
|
count: 2,
|
||||||
|
}));
|
||||||
|
r.register(Arc::new(Stub {
|
||||||
|
name: "b",
|
||||||
|
chunk: Some("(SELECT 2 WHERE {outer} = 'y')"),
|
||||||
|
manifest: Some("(SELECT 3 WHERE {outer} = 'z')"),
|
||||||
|
count: 5,
|
||||||
|
}));
|
||||||
|
r
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn chunk_level_sums_every_contributing_source() {
|
||||||
|
let expr = registry().ref_count_expr(RefLevel::Chunk, "b.hash");
|
||||||
|
assert!(expr.contains("b.hash = 'x'"), "{expr}");
|
||||||
|
assert!(expr.contains("b.hash = 'y'"), "{expr}");
|
||||||
|
assert!(expr.contains('+'), "fragments must be summed: {expr}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A source returning `None` for a level must contribute nothing there —
|
||||||
|
/// this is what keeps manifest-only tables out of the chunk recompute,
|
||||||
|
/// where they would double-count against the single-chunk hash alias.
|
||||||
|
#[test]
|
||||||
|
fn manifest_level_skips_non_contributing_sources() {
|
||||||
|
let expr = registry().ref_count_expr(RefLevel::Manifest, "m.file_hash");
|
||||||
|
assert!(expr.contains("m.file_hash = 'z'"), "{expr}");
|
||||||
|
assert!(!expr.contains('+'), "only one source contributes: {expr}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An empty level must still yield a valid scalar expression, so callers
|
||||||
|
/// need no special case before a registry is fully populated.
|
||||||
|
#[test]
|
||||||
|
fn empty_level_yields_zero_literal() {
|
||||||
|
let r = BlobReferenceRegistry::new();
|
||||||
|
assert_eq!(r.ref_count_expr(RefLevel::Chunk, "b.hash"), "0");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn total_references_adds_across_sources() {
|
||||||
|
assert_eq!(registry().total_references("deadbeef").await.unwrap(), 7);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
pub mod auth_ports;
|
pub mod auth_ports;
|
||||||
pub mod authorization_ports;
|
pub mod authorization_ports;
|
||||||
pub mod blob_lifecycle;
|
pub mod blob_lifecycle;
|
||||||
|
pub mod blob_reference_ports;
|
||||||
pub mod blob_storage_ports;
|
pub mod blob_storage_ports;
|
||||||
pub mod cache_ports;
|
pub mod cache_ports;
|
||||||
pub mod calendar_ports;
|
pub mod calendar_ports;
|
||||||
|
|||||||
+37
-1
@@ -440,6 +440,22 @@ impl AppServiceFactory {
|
|||||||
// `blob_backend` into DedupService.
|
// `blob_backend` into DedupService.
|
||||||
let blob_backend_for_consistency = blob_backend.clone();
|
let blob_backend_for_consistency = blob_backend.clone();
|
||||||
|
|
||||||
|
// Every table holding blob references. Built ONCE and shared by the
|
||||||
|
// GC reap predicate and the consistency recompute so the two cannot
|
||||||
|
// disagree about what "referenced" means — a disagreement reaps live
|
||||||
|
// content. New blob-owning tables register here.
|
||||||
|
// See docs/plan/derived-blobs.md.
|
||||||
|
let blob_reference_registry = {
|
||||||
|
use crate::infrastructure::repositories::pg::blob_reference_sources::{
|
||||||
|
ChunksReferenceSource, FilesReferenceSource,
|
||||||
|
};
|
||||||
|
let mut registry =
|
||||||
|
crate::application::ports::blob_reference_ports::BlobReferenceRegistry::new();
|
||||||
|
registry.register(Arc::new(FilesReferenceSource::new(db_pool.clone())));
|
||||||
|
registry.register(Arc::new(ChunksReferenceSource::new(db_pool.clone())));
|
||||||
|
Arc::new(registry)
|
||||||
|
};
|
||||||
|
|
||||||
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
|
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
|
||||||
let dedup_service = Arc::new(
|
let dedup_service = Arc::new(
|
||||||
crate::infrastructure::services::dedup_service::DedupService::new(
|
crate::infrastructure::services::dedup_service::DedupService::new(
|
||||||
@@ -447,7 +463,8 @@ impl AppServiceFactory {
|
|||||||
db_pool.clone(),
|
db_pool.clone(),
|
||||||
maintenance_pool.clone(),
|
maintenance_pool.clone(),
|
||||||
)
|
)
|
||||||
.with_blob_lifecycle(blob_lifecycle),
|
.with_blob_lifecycle(blob_lifecycle)
|
||||||
|
.with_reference_registry(blob_reference_registry.clone()),
|
||||||
);
|
);
|
||||||
dedup_service.initialize().await?;
|
dedup_service.initialize().await?;
|
||||||
|
|
||||||
@@ -1460,6 +1477,22 @@ impl AppServiceFactory {
|
|||||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||||
.await;
|
.await;
|
||||||
|
|
||||||
|
// Reconciles `chunk_manifests.ref_count` — the SECOND reference
|
||||||
|
// counter, and the one nothing verified before. add_reference bumps
|
||||||
|
// it first and only falls back to storage.blobs.ref_count, so every
|
||||||
|
// CDC file (and every derived artifact, once those land) counts here
|
||||||
|
// rather than at the chunk level. Uses the same registry dedup_gc
|
||||||
|
// reaps from, so the two cannot disagree.
|
||||||
|
// See docs/plan/derived-blobs.md.
|
||||||
|
let _ = Arc::new(
|
||||||
|
crate::infrastructure::services::manifests_consistency_service::ManifestsConsistencyCheck::new(
|
||||||
|
maintenance_pool.clone(),
|
||||||
|
core.dedup_service.reference_registry(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||||
|
.await;
|
||||||
|
|
||||||
// Third recoverable-run tenant. Iterates `storage.files`
|
// Third recoverable-run tenant. Iterates `storage.files`
|
||||||
// and reports parent-folder-trashed cascade misses,
|
// and reports parent-folder-trashed cascade misses,
|
||||||
// `missing_blob` (data-loss indicator — file references
|
// `missing_blob` (data-loss indicator — file references
|
||||||
@@ -1493,6 +1526,9 @@ impl AppServiceFactory {
|
|||||||
core.blob_backend.clone(),
|
core.blob_backend.clone(),
|
||||||
core.config.storage_entries.clone(),
|
core.config.storage_entries.clone(),
|
||||||
self.storage_path.clone(),
|
self.storage_path.clone(),
|
||||||
|
// Same registry instance GC reaps from — see
|
||||||
|
// DedupService::reference_registry.
|
||||||
|
core.dedup_service.reference_registry(),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
|
||||||
|
|||||||
@@ -0,0 +1,353 @@
|
|||||||
|
//! The two implicit blob-reference sources, made explicit.
|
||||||
|
//!
|
||||||
|
//! Before this module, "who references this hash" lived as hardcoded SQL
|
||||||
|
//! inside `blobs_consistency`'s refcount recompute and `dedup_gc`'s reap
|
||||||
|
//! predicate. These two implementations reproduce that SQL **exactly** —
|
||||||
|
//! the fragments below sum to today's `actual_ref_count` expression — so
|
||||||
|
//! the registry can be wired in without changing any observed count.
|
||||||
|
//!
|
||||||
|
//! See `docs/plan/derived-blobs.md` and
|
||||||
|
//! [`crate::application::ports::blob_reference_ports`].
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use sqlx::{PgPool, Row};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_reference_ports::{BlobReferenceSource, RefLevel};
|
||||||
|
use crate::domain::errors::DomainError;
|
||||||
|
|
||||||
|
/// Aliases used inside the emitted fragments.
|
||||||
|
///
|
||||||
|
/// Deliberately distinct from the aliases the sweeps use for their outer
|
||||||
|
/// row (`b` for `storage.blobs`, `m` for `storage.chunk_manifests`): a
|
||||||
|
/// fragment reusing `m` would shadow the outer alias in the manifest-level
|
||||||
|
/// sweep and silently correlate against itself.
|
||||||
|
const FILES_ALIAS: &str = "cnt_f";
|
||||||
|
const MANIFEST_ALIAS: &str = "cnt_m";
|
||||||
|
|
||||||
|
/// Fragment for [`FilesReferenceSource`], as a free function so the SQL
|
||||||
|
/// shape can be tested without constructing a pool — it is a property of
|
||||||
|
/// the module, not of an instance.
|
||||||
|
fn files_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
let f = FILES_ALIAS;
|
||||||
|
match level {
|
||||||
|
// Legacy whole-file blobs only — CDC files are counted at the
|
||||||
|
// manifest level, and counting them here too would double up on the
|
||||||
|
// single-chunk hash alias.
|
||||||
|
RefLevel::Chunk => Some(format!(
|
||||||
|
"(SELECT COUNT(*) FROM storage.files {f}
|
||||||
|
WHERE {f}.blob_hash = {outer_hash_expr}
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS}
|
||||||
|
WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash
|
||||||
|
))"
|
||||||
|
)),
|
||||||
|
RefLevel::Manifest => Some(format!(
|
||||||
|
"(SELECT COUNT(*) FROM storage.files {f}
|
||||||
|
WHERE {f}.blob_hash = {outer_hash_expr})"
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Short-circuiting existence form of [`files_ref_sql`].
|
||||||
|
///
|
||||||
|
/// `dedup_gc` evaluates this per candidate manifest, so counting every
|
||||||
|
/// referrer where existence would do is a real cost on a heavily-deduplicated
|
||||||
|
/// blob. This is also the exact shape the reap predicate used before the
|
||||||
|
/// registry existed, so wiring it in changes no plan.
|
||||||
|
fn files_exists_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
let f = FILES_ALIAS;
|
||||||
|
match level {
|
||||||
|
RefLevel::Chunk => Some(format!(
|
||||||
|
"EXISTS (SELECT 1 FROM storage.files {f} \
|
||||||
|
WHERE {f}.blob_hash = {outer_hash_expr} \
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS} \
|
||||||
|
WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash))"
|
||||||
|
)),
|
||||||
|
RefLevel::Manifest => Some(format!(
|
||||||
|
"EXISTS (SELECT 1 FROM storage.files {f} WHERE {f}.blob_hash = {outer_hash_expr})"
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fragment for [`ChunksReferenceSource`]. See [`files_ref_sql`].
|
||||||
|
fn chunks_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
match level {
|
||||||
|
RefLevel::Chunk => {
|
||||||
|
let m = MANIFEST_ALIAS;
|
||||||
|
Some(format!(
|
||||||
|
"(SELECT COUNT(*) FROM storage.chunk_manifests {m}
|
||||||
|
WHERE {outer_hash_expr} = ANY({m}.chunk_hashes))"
|
||||||
|
))
|
||||||
|
}
|
||||||
|
// A manifest is never referenced by another manifest.
|
||||||
|
RefLevel::Manifest => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── storage.files ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// References held by `storage.files.blob_hash`.
|
||||||
|
///
|
||||||
|
/// Contributes at **both** levels, which is why `RefLevel` is a parameter
|
||||||
|
/// rather than a property of the source:
|
||||||
|
///
|
||||||
|
/// * [`RefLevel::Manifest`] — a CDC file's `blob_hash` names a manifest.
|
||||||
|
/// * [`RefLevel::Chunk`] — a pre-CDC legacy file, whose `blob_hash` names a
|
||||||
|
/// whole-file blob with no manifest behind it. The `NOT EXISTS` guard is
|
||||||
|
/// load-bearing: for a single-chunk file the whole-file hash *equals* its
|
||||||
|
/// lone chunk's hash, so without it the row would be counted at both
|
||||||
|
/// levels.
|
||||||
|
pub struct FilesReferenceSource {
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FilesReferenceSource {
|
||||||
|
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||||
|
Self { pool }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl BlobReferenceSource for FilesReferenceSource {
|
||||||
|
fn source_name(&self) -> &'static str {
|
||||||
|
"files"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
files_ref_sql(level, outer_hash_expr)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
files_exists_sql(level, outer_hash_expr)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
|
||||||
|
// No level split here: the question is "how many file rows name this
|
||||||
|
// exact hash", and a hash names either a manifest or a legacy blob,
|
||||||
|
// never both at once from the caller's point of view.
|
||||||
|
let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.files WHERE blob_hash = $1")
|
||||||
|
.bind(blob_hash)
|
||||||
|
.fetch_one(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobRefSource", format!("files count: {e}"))
|
||||||
|
})?;
|
||||||
|
Ok(n.max(0) as u64)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn list_referenced_blobs(
|
||||||
|
&self,
|
||||||
|
cursor: Option<Vec<u8>>,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||||
|
// Paged by the file's own PK so the cursor is stable under concurrent
|
||||||
|
// inserts; `blob_hash` is not unique and would skip or repeat rows.
|
||||||
|
let after: Option<Uuid> = match cursor {
|
||||||
|
Some(bytes) => Some(decode_uuid_cursor(&bytes)?),
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let rows = sqlx::query(
|
||||||
|
"SELECT id, blob_hash FROM storage.files
|
||||||
|
WHERE ($1::uuid IS NULL OR id > $1)
|
||||||
|
ORDER BY id
|
||||||
|
LIMIT $2",
|
||||||
|
)
|
||||||
|
.bind(after)
|
||||||
|
.bind(limit as i64)
|
||||||
|
.fetch_all(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("files page: {e}")))?;
|
||||||
|
|
||||||
|
let next = rows
|
||||||
|
.last()
|
||||||
|
.map(|r| r.get::<Uuid, _>("id").as_bytes().to_vec())
|
||||||
|
.filter(|_| rows.len() == limit);
|
||||||
|
let hashes = rows
|
||||||
|
.iter()
|
||||||
|
.map(|r| r.get::<String, _>("blob_hash"))
|
||||||
|
.collect();
|
||||||
|
Ok((hashes, next))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── storage.chunk_manifests ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// References held by `storage.chunk_manifests.chunk_hashes[]`.
|
||||||
|
///
|
||||||
|
/// Chunk level only — a manifest never references another manifest, so
|
||||||
|
/// [`RefLevel::Manifest`] yields `None` and this source contributes nothing
|
||||||
|
/// to the manifest recompute.
|
||||||
|
pub struct ChunksReferenceSource {
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ChunksReferenceSource {
|
||||||
|
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||||
|
Self { pool }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl BlobReferenceSource for ChunksReferenceSource {
|
||||||
|
fn source_name(&self) -> &'static str {
|
||||||
|
"chunks"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||||
|
chunks_ref_sql(level, outer_hash_expr)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
|
||||||
|
let n: i64 = sqlx::query_scalar(
|
||||||
|
"SELECT COUNT(*) FROM storage.chunk_manifests WHERE $1 = ANY(chunk_hashes)",
|
||||||
|
)
|
||||||
|
.bind(blob_hash)
|
||||||
|
.fetch_one(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks count: {e}")))?;
|
||||||
|
Ok(n.max(0) as u64)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn list_referenced_blobs(
|
||||||
|
&self,
|
||||||
|
cursor: Option<Vec<u8>>,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||||
|
// Paged by the manifest PK, not by the unnested chunk hash: a single
|
||||||
|
// manifest expands to many hashes, so the page boundary has to fall
|
||||||
|
// between manifests or the cursor cannot be resumed unambiguously.
|
||||||
|
let after: Option<String> = match cursor {
|
||||||
|
Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| {
|
||||||
|
DomainError::internal_error("BlobRefSource", format!("bad chunk cursor: {e}"))
|
||||||
|
})?),
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let rows = sqlx::query(
|
||||||
|
"SELECT file_hash, chunk_hashes FROM storage.chunk_manifests
|
||||||
|
WHERE ($1::text IS NULL OR file_hash > $1)
|
||||||
|
ORDER BY file_hash
|
||||||
|
LIMIT $2",
|
||||||
|
)
|
||||||
|
.bind(after)
|
||||||
|
.bind(limit as i64)
|
||||||
|
.fetch_all(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks page: {e}")))?;
|
||||||
|
|
||||||
|
let next = rows
|
||||||
|
.last()
|
||||||
|
.map(|r| r.get::<String, _>("file_hash").into_bytes())
|
||||||
|
.filter(|_| rows.len() == limit);
|
||||||
|
let hashes = rows
|
||||||
|
.iter()
|
||||||
|
.flat_map(|r| r.get::<Vec<String>, _>("chunk_hashes"))
|
||||||
|
.collect();
|
||||||
|
Ok((hashes, next))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_uuid_cursor(bytes: &[u8]) -> Result<Uuid, DomainError> {
|
||||||
|
let raw: [u8; 16] = bytes.try_into().map_err(|_| {
|
||||||
|
DomainError::internal_error(
|
||||||
|
"BlobRefSource",
|
||||||
|
format!("bad uuid cursor: expected 16 bytes, got {}", bytes.len()),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
Ok(Uuid::from_bytes(raw))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::application::ports::blob_reference_ports::BlobReferenceRegistry;
|
||||||
|
|
||||||
|
/// The registry sums whatever the sources emit; these helpers exercise the
|
||||||
|
/// same code path without needing a pool, since `ref_count_sql` is pure.
|
||||||
|
fn summed(level: RefLevel, outer: &str) -> String {
|
||||||
|
let frags: Vec<String> = [files_ref_sql(level, outer), chunks_ref_sql(level, outer)]
|
||||||
|
.into_iter()
|
||||||
|
.flatten()
|
||||||
|
.collect();
|
||||||
|
if frags.is_empty() {
|
||||||
|
"0".to_string()
|
||||||
|
} else {
|
||||||
|
frags.join("\n + ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The chunk-level expression must reproduce the two terms
|
||||||
|
/// `blobs_consistency` inlines today: legacy-only files (guarded by
|
||||||
|
/// NOT EXISTS) plus manifests citing the chunk.
|
||||||
|
#[test]
|
||||||
|
fn chunk_level_reproduces_todays_two_terms() {
|
||||||
|
let expr = summed(RefLevel::Chunk, "b.hash");
|
||||||
|
assert!(expr.contains("storage.files"), "{expr}");
|
||||||
|
assert!(
|
||||||
|
expr.contains("NOT EXISTS"),
|
||||||
|
"legacy term must keep the CDC guard: {expr}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
expr.contains("= ANY(cnt_m.chunk_hashes)"),
|
||||||
|
"chunk term missing: {expr}"
|
||||||
|
);
|
||||||
|
assert!(expr.contains("b.hash"), "must correlate on the outer row");
|
||||||
|
assert!(expr.contains('+'), "both terms must be summed: {expr}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Only `storage.files` references a manifest, so the manifest-level
|
||||||
|
/// expression is the single files term with no `NOT EXISTS` guard — the
|
||||||
|
/// guard exists to keep CDC rows *out* of the chunk level, and applying
|
||||||
|
/// it here would count nothing at all.
|
||||||
|
#[test]
|
||||||
|
fn manifest_level_is_files_only_and_unguarded() {
|
||||||
|
let expr = summed(RefLevel::Manifest, "m.file_hash");
|
||||||
|
assert!(expr.contains("storage.files"), "{expr}");
|
||||||
|
assert!(!expr.contains("NOT EXISTS"), "{expr}");
|
||||||
|
assert!(
|
||||||
|
!expr.contains("chunk_hashes"),
|
||||||
|
"chunks must not contribute at manifest level: {expr}"
|
||||||
|
);
|
||||||
|
assert!(!expr.contains('+'), "only one source contributes: {expr}");
|
||||||
|
assert!(expr.contains("m.file_hash"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fragments must not use the aliases the sweeps use for their outer row
|
||||||
|
/// (`b` for storage.blobs, `m` for chunk_manifests), or the manifest sweep
|
||||||
|
/// would shadow its own alias and silently correlate against itself.
|
||||||
|
#[test]
|
||||||
|
fn fragments_avoid_outer_row_aliases() {
|
||||||
|
for level in RefLevel::ALL {
|
||||||
|
let expr = summed(level, "m.file_hash");
|
||||||
|
for bad in [
|
||||||
|
"storage.files f",
|
||||||
|
"storage.files b",
|
||||||
|
"chunk_manifests m ",
|
||||||
|
"chunk_manifests b",
|
||||||
|
] {
|
||||||
|
assert!(!expr.contains(bad), "alias collision at {level:?}: {expr}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A source declining a level must drop out of the sum entirely, which is
|
||||||
|
/// what keeps manifest-only tables out of the chunk recompute where the
|
||||||
|
/// single-chunk hash alias would double-count them.
|
||||||
|
#[test]
|
||||||
|
fn chunks_source_declines_manifest_level() {
|
||||||
|
assert!(chunks_ref_sql(RefLevel::Manifest, "m.file_hash").is_none());
|
||||||
|
assert!(chunks_ref_sql(RefLevel::Chunk, "b.hash").is_some());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Guards the registry contract the sweeps rely on: an empty level still
|
||||||
|
/// yields a valid scalar expression.
|
||||||
|
#[test]
|
||||||
|
fn empty_registry_yields_zero_literal() {
|
||||||
|
let r = BlobReferenceRegistry::new();
|
||||||
|
assert_eq!(r.ref_count_expr(RefLevel::Manifest, "m.file_hash"), "0");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1024,10 +1024,21 @@ impl FileWritePort for FileBlobWriteRepository {
|
|||||||
DomainError::internal_error("FileBlobWrite", format!("fetch blob_hash: {e}"))
|
DomainError::internal_error("FileBlobWrite", format!("fetch blob_hash: {e}"))
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
// DELETE fires trg_files_decrement_blob_ref → storage.blobs.ref_count--
|
// DELETE fires `trg_files_decrement_blob_ref` — post-2026-08-23
|
||||||
|
// it dispatches manifest-first (see migration
|
||||||
|
// `20261017000000_file_delete_trigger_manifest_aware.sql`):
|
||||||
|
// decrements `chunk_manifests.ref_count` if the hash names a
|
||||||
|
// manifest (walking chunks on last-ref), else falls back to
|
||||||
|
// `storage.blobs.ref_count`. Counter state after this call is
|
||||||
|
// already correct.
|
||||||
self.delete_file(file_id).await?;
|
self.delete_file(file_id).await?;
|
||||||
|
|
||||||
// If the blob is now unreferenced, remove disk file + thumbnails.
|
// Physical cleanup only. `cleanup_if_orphaned` was previously
|
||||||
|
// manifest-aware and did counter compensation for the old
|
||||||
|
// trigger's over-decrement; after the trigger rewrite it's a
|
||||||
|
// legacy-blob-eager-reap helper — safe to keep calling
|
||||||
|
// unconditionally (no-op for CDC hashes; reaps legacy blobs
|
||||||
|
// that reached ref_count = 0).
|
||||||
if let Some(hash) = blob_hash {
|
if let Some(hash) = blob_hash {
|
||||||
self.dedup.cleanup_if_orphaned(&hash).await;
|
self.dedup.cleanup_if_orphaned(&hash).await;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
mod address_book_pg_repository;
|
mod address_book_pg_repository;
|
||||||
mod app_password_pg_repository;
|
mod app_password_pg_repository;
|
||||||
|
pub mod blob_reference_sources;
|
||||||
mod calendar_event_pg_repository;
|
mod calendar_event_pg_repository;
|
||||||
mod calendar_pg_repository;
|
mod calendar_pg_repository;
|
||||||
mod contact_group_pg_repository;
|
mod contact_group_pg_repository;
|
||||||
|
|||||||
@@ -45,11 +45,25 @@ use serde::{Deserialize, Serialize};
|
|||||||
/// of the entry to probe instead of the currently-active backend.
|
/// of the entry to probe instead of the currently-active backend.
|
||||||
/// `None` falls through to the live backend (today's behaviour).
|
/// `None` falls through to the live backend (today's behaviour).
|
||||||
/// - Others — ignored.
|
/// - Others — ignored.
|
||||||
|
///
|
||||||
|
/// Semantics of `repair` (added 2026-10-17 for the refcount fix):
|
||||||
|
/// - `blobs_consistency` / `manifests_consistency` — when `true`,
|
||||||
|
/// after each `refcount_mismatch` / `manifest_refcount_mismatch`
|
||||||
|
/// finding is recorded, apply the corrective UPDATE that sets the
|
||||||
|
/// stored counter to the auditor's computed `actual_ref_count`.
|
||||||
|
/// Content-safe: the row itself is fine, only the counter is
|
||||||
|
/// wrong. Race-safe: each UPDATE recomputes the auditor formula
|
||||||
|
/// in the same statement, so a concurrent write can't leave a
|
||||||
|
/// stale value. Default `false` preserves discovery-only
|
||||||
|
/// behaviour. Also propagates through `consistency_batch` to
|
||||||
|
/// both tenants — one `?repair=true` call fixes both counters.
|
||||||
|
/// - Others — ignored.
|
||||||
#[derive(Debug, Clone, Default)]
|
#[derive(Debug, Clone, Default)]
|
||||||
pub struct JobRunArgs {
|
pub struct JobRunArgs {
|
||||||
pub force: bool,
|
pub force: bool,
|
||||||
pub deep: bool,
|
pub deep: bool,
|
||||||
pub storage: Option<String>,
|
pub storage: Option<String>,
|
||||||
|
pub repair: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Uniform outcome the supervisor logs and stores for every job dispatch.
|
/// Uniform outcome the supervisor logs and stores for every job dispatch.
|
||||||
|
|||||||
@@ -63,6 +63,7 @@ use async_trait::async_trait;
|
|||||||
use chrono::{DateTime, Duration, Utc};
|
use chrono::{DateTime, Duration, Utc};
|
||||||
use sqlx::PgPool;
|
use sqlx::PgPool;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
|
||||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
use crate::common::config::NamedStorageEntry;
|
use crate::common::config::NamedStorageEntry;
|
||||||
use crate::infrastructure::scheduler::{
|
use crate::infrastructure::scheduler::{
|
||||||
@@ -114,6 +115,64 @@ pub struct BlobsConsistencyCheck {
|
|||||||
/// fallback for a Local target entry with no `_ROOT_DIR`. Same
|
/// fallback for a Local target entry with no `_ROOT_DIR`. Same
|
||||||
/// fallback rule the boot path uses.
|
/// fallback rule the boot path uses.
|
||||||
storage_path_fallback: PathBuf,
|
storage_path_fallback: PathBuf,
|
||||||
|
/// The chunk-level page query, assembled once from the blob-reference
|
||||||
|
/// registry so this recompute and `dedup_gc` agree on what "referenced"
|
||||||
|
/// means. Built at construction rather than per page so the sweep runs a
|
||||||
|
/// fixed statement — same reasoning as `DedupService::manifest_reap_sql`.
|
||||||
|
/// See `docs/plan/derived-blobs.md`.
|
||||||
|
chunk_page_sql: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The chunk-level page query, with `actual_ref_count` summed from the
|
||||||
|
/// registered reference sources.
|
||||||
|
///
|
||||||
|
/// `storage.blobs.ref_count` semantics — the invariant `dedup_service`
|
||||||
|
/// actually maintains:
|
||||||
|
///
|
||||||
|
/// ```text
|
||||||
|
/// ref_count = (number of chunk_manifests whose chunk_hashes[] contains
|
||||||
|
/// this hash)
|
||||||
|
/// + (number of files.blob_hash pointing at this hash on the
|
||||||
|
/// LEGACY whole-file path — files with NO manifest for their
|
||||||
|
/// blob_hash)
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// Naively `COUNT(files) + COUNT(manifests referring)` double-counts
|
||||||
|
/// single-chunk CDC files: where a file's whole-file hash equals its lone
|
||||||
|
/// chunk's hash (anything under one CDC chunk), the file appears BOTH in
|
||||||
|
/// `files.blob_hash` and in the manifest's `chunk_hashes[]`. The
|
||||||
|
/// `NOT EXISTS` guard inside `FilesReferenceSource`'s chunk-level fragment
|
||||||
|
/// excludes CDC-path files from the legacy term so the two don't overlap.
|
||||||
|
///
|
||||||
|
/// The GIN index on `chunk_hashes` (migration
|
||||||
|
/// `20260628000000_delta_upload_gin_index`) keeps the `= ANY(chunk_hashes)`
|
||||||
|
/// probe cheap.
|
||||||
|
///
|
||||||
|
/// # Panics
|
||||||
|
///
|
||||||
|
/// If no source contributes at [`RefLevel::Chunk`] — a wiring bug that
|
||||||
|
/// would make every blob look unreferenced and flag the whole table as
|
||||||
|
/// `refcount_mismatch`.
|
||||||
|
fn chunk_page_sql(registry: &BlobReferenceRegistry) -> String {
|
||||||
|
let expected = registry.ref_count_expr(RefLevel::Chunk, "b.hash");
|
||||||
|
assert!(
|
||||||
|
expected != "0",
|
||||||
|
"no chunk-level blob reference source registered: every blob would \
|
||||||
|
appear unreferenced"
|
||||||
|
);
|
||||||
|
|
||||||
|
format!(
|
||||||
|
"SELECT
|
||||||
|
b.hash AS hash,
|
||||||
|
b.size AS size,
|
||||||
|
b.ref_count AS ref_count,
|
||||||
|
b.created_at AS created_at,
|
||||||
|
({expected})::bigint AS actual_ref_count
|
||||||
|
FROM storage.blobs b
|
||||||
|
WHERE ($1::text IS NULL OR b.hash > $1)
|
||||||
|
ORDER BY b.hash
|
||||||
|
LIMIT $2"
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
impl BlobsConsistencyCheck {
|
impl BlobsConsistencyCheck {
|
||||||
@@ -122,12 +181,14 @@ impl BlobsConsistencyCheck {
|
|||||||
backend: Arc<dyn BlobStorageBackend>,
|
backend: Arc<dyn BlobStorageBackend>,
|
||||||
storage_entries: Vec<NamedStorageEntry>,
|
storage_entries: Vec<NamedStorageEntry>,
|
||||||
storage_path_fallback: PathBuf,
|
storage_path_fallback: PathBuf,
|
||||||
|
reference_registry: Arc<BlobReferenceRegistry>,
|
||||||
) -> Self {
|
) -> Self {
|
||||||
Self {
|
Self {
|
||||||
pool,
|
pool,
|
||||||
backend,
|
backend,
|
||||||
storage_entries,
|
storage_entries,
|
||||||
storage_path_fallback,
|
storage_path_fallback,
|
||||||
|
chunk_page_sql: chunk_page_sql(&reference_registry),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -286,6 +347,10 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
// stats.finding_count — actual persistence happens in
|
// stats.finding_count — actual persistence happens in
|
||||||
// `record_finding` on each emission).
|
// `record_finding` on each emission).
|
||||||
let mut finding_count = 0u64;
|
let mut finding_count = 0u64;
|
||||||
|
// Only touched when `args.repair == true`. Symmetric with
|
||||||
|
// `manifests_consistency`; reported in completion log +
|
||||||
|
// `extra_stats` so operators see "found N, fixed M" in one line.
|
||||||
|
let mut repaired_count = 0u64;
|
||||||
|
|
||||||
// Deep mode is a per-run flag with two consumers:
|
// Deep mode is a per-run flag with two consumers:
|
||||||
// 1. This handler — decides whether to re-hash bytes.
|
// 1. This handler — decides whether to re-hash bytes.
|
||||||
@@ -338,6 +403,41 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Repair mode: same shape as `deep` above so the admin run-
|
||||||
|
// detail view can display `params.repair = "true"` alongside
|
||||||
|
// `params.deep`. Fresh persists what the trigger asked for;
|
||||||
|
// Resume reads back so a paused repair scan stays a repair
|
||||||
|
// scan (a mid-scan crash mustn't silently downgrade to
|
||||||
|
// discovery-only for the remaining rows).
|
||||||
|
let repair = if is_fresh {
|
||||||
|
let v = if args.repair { "true" } else { "false" };
|
||||||
|
if let Err(e) = store.set_string_param("repair", v).await {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("failed to persist repair flag to params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
args.repair
|
||||||
|
} else {
|
||||||
|
match store.get_string_param("repair").await {
|
||||||
|
Ok(Some(v)) => v == "true",
|
||||||
|
Ok(None) => false,
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("read `repair` from params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if repair {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "blobs_consistency.repair_mode_active",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
"repair mode: refcount_mismatch findings will trigger corrective UPDATE"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
loop {
|
loop {
|
||||||
// Cooperative cancel poll between batches.
|
// Cooperative cancel poll between batches.
|
||||||
match store.status().await {
|
match store.status().await {
|
||||||
@@ -364,58 +464,14 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fetch the next batch. Per-row `actual_ref_count`
|
// Fetch the next batch. `actual_ref_count` is summed from the
|
||||||
// computed inline via correlated subqueries — one for
|
// registered reference sources — see `chunk_page_sql`, which
|
||||||
// legacy whole-file references (`files.blob_hash`), one
|
// documents the invariant and the single-chunk double-count trap.
|
||||||
// for CDC chunk references (`chunk_manifests.chunk_hashes`).
|
let rows: Vec<BlobRow> = match sqlx::query_as(&self.chunk_page_sql)
|
||||||
// GIN index on `chunk_hashes` (migration
|
.bind(cursor.as_deref())
|
||||||
// 20260628000000_delta_upload_gin_index) makes the
|
.bind(BATCH_SIZE)
|
||||||
// `= ANY(chunk_hashes)` probe cheap.
|
.fetch_all(self.pool.as_ref())
|
||||||
// `storage.blobs.ref_count` semantics — what the invariant
|
.await
|
||||||
// dedup_service maintains actually is:
|
|
||||||
//
|
|
||||||
// ref_count = (number of chunk_manifests whose
|
|
||||||
// chunk_hashes[] contains this hash)
|
|
||||||
// + (number of files.blob_hash pointing at
|
|
||||||
// this hash on the LEGACY whole-file path
|
|
||||||
// — i.e. files with NO manifest for their
|
|
||||||
// blob_hash)
|
|
||||||
//
|
|
||||||
// Naively `COUNT(files) + COUNT(manifests referring)`
|
|
||||||
// double-counts single-chunk CDC files: for a file whose
|
|
||||||
// whole-file hash == its single chunk's hash (any file
|
|
||||||
// small enough to fit in one CDC chunk — under ~256 KB
|
|
||||||
// average), the file appears BOTH in `files.blob_hash`
|
|
||||||
// AND in the manifest's `chunk_hashes[]`. The `NOT
|
|
||||||
// EXISTS` clause below excludes CDC-path files from the
|
|
||||||
// legacy count so the two terms don't overlap.
|
|
||||||
let rows: Vec<BlobRow> = match sqlx::query_as(
|
|
||||||
r#"
|
|
||||||
SELECT
|
|
||||||
b.hash AS hash,
|
|
||||||
b.size AS size,
|
|
||||||
b.ref_count AS ref_count,
|
|
||||||
b.created_at AS created_at,
|
|
||||||
(
|
|
||||||
(SELECT COUNT(*) FROM storage.files f
|
|
||||||
WHERE f.blob_hash = b.hash
|
|
||||||
AND NOT EXISTS (
|
|
||||||
SELECT 1 FROM storage.chunk_manifests m
|
|
||||||
WHERE m.file_hash = f.blob_hash
|
|
||||||
))
|
|
||||||
+ (SELECT COUNT(*) FROM storage.chunk_manifests m
|
|
||||||
WHERE b.hash = ANY(m.chunk_hashes))
|
|
||||||
)::bigint AS actual_ref_count
|
|
||||||
FROM storage.blobs b
|
|
||||||
WHERE ($1::text IS NULL OR b.hash > $1)
|
|
||||||
ORDER BY b.hash
|
|
||||||
LIMIT $2
|
|
||||||
"#,
|
|
||||||
)
|
|
||||||
.bind(cursor.as_deref())
|
|
||||||
.bind(BATCH_SIZE)
|
|
||||||
.fetch_all(self.pool.as_ref())
|
|
||||||
.await
|
|
||||||
{
|
{
|
||||||
Ok(r) => r,
|
Ok(r) => r,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
@@ -431,11 +487,17 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
event = "blobs_consistency.completed",
|
event = "blobs_consistency.completed",
|
||||||
run_id = %store.run_id(),
|
run_id = %store.run_id(),
|
||||||
finding_count = finding_count,
|
finding_count = finding_count,
|
||||||
|
repaired_count = repaired_count,
|
||||||
|
repair_requested = repair,
|
||||||
deep = deep,
|
deep = deep,
|
||||||
"blobs_consistency completed with {} finding(s)",
|
"blobs_consistency completed with {} finding(s), {} repaired",
|
||||||
finding_count
|
finding_count,
|
||||||
|
repaired_count
|
||||||
);
|
);
|
||||||
return RunOutcome::completed();
|
return RunOutcome::completed_with(serde_json::json!({
|
||||||
|
"repair_requested": repair,
|
||||||
|
"repaired_count": repaired_count,
|
||||||
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
let grace_cutoff = Utc::now() - CREATE_GRACE;
|
let grace_cutoff = Utc::now() - CREATE_GRACE;
|
||||||
@@ -463,6 +525,67 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
}),
|
}),
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
|
|
||||||
|
// Repair pass — content-safe corrective UPDATE. Sets
|
||||||
|
// `stored` to the value the auditor's two-term formula
|
||||||
|
// would compute at UPDATE time (subquery mirrors
|
||||||
|
// `chunk_page_sql`'s `actual_ref_count`), so a
|
||||||
|
// concurrent write between our page fetch and this
|
||||||
|
// UPDATE can't leave a stale value — the subquery
|
||||||
|
// re-reads inside the same statement. The
|
||||||
|
// `<> (subquery)` guard makes the UPDATE a no-op if
|
||||||
|
// the drift has healed, making this idempotent under
|
||||||
|
// retry.
|
||||||
|
if repair {
|
||||||
|
let expected = "( \
|
||||||
|
(SELECT COUNT(*) FROM storage.files f \
|
||||||
|
WHERE f.blob_hash = b.hash \
|
||||||
|
AND NOT EXISTS ( \
|
||||||
|
SELECT 1 FROM storage.chunk_manifests m \
|
||||||
|
WHERE m.file_hash = f.blob_hash \
|
||||||
|
)) \
|
||||||
|
+ (SELECT COUNT(*) FROM storage.chunk_manifests m \
|
||||||
|
WHERE b.hash = ANY(m.chunk_hashes)) \
|
||||||
|
)";
|
||||||
|
let update_sql = format!(
|
||||||
|
"UPDATE storage.blobs b \
|
||||||
|
SET ref_count = {expected} \
|
||||||
|
WHERE b.hash = $1 \
|
||||||
|
AND b.ref_count <> {expected}",
|
||||||
|
);
|
||||||
|
match sqlx::query(&update_sql)
|
||||||
|
.bind(&row.hash)
|
||||||
|
.execute(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(res) if res.rows_affected() > 0 => {
|
||||||
|
repaired_count += 1;
|
||||||
|
tracing::info!(
|
||||||
|
target: "audit",
|
||||||
|
event = "blobs_consistency.repaired",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
hash = %row.hash,
|
||||||
|
stored_was = row.ref_count,
|
||||||
|
actual = row.actual_ref_count,
|
||||||
|
"🩹 blob ref_count repaired"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(_) => {
|
||||||
|
// No row touched — concurrent repair or
|
||||||
|
// self-healing drift. Silent no-op.
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "blobs_consistency.repair_failed",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
hash = %row.hash,
|
||||||
|
error = %e,
|
||||||
|
"blob ref_count repair UPDATE failed — finding stays"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Skip physical probes for rows within the write
|
// Skip physical probes for rows within the write
|
||||||
@@ -611,11 +734,17 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
|
|||||||
event = "blobs_consistency.completed",
|
event = "blobs_consistency.completed",
|
||||||
run_id = %store.run_id(),
|
run_id = %store.run_id(),
|
||||||
finding_count = finding_count,
|
finding_count = finding_count,
|
||||||
|
repaired_count = repaired_count,
|
||||||
|
repair_requested = repair,
|
||||||
deep = deep,
|
deep = deep,
|
||||||
"blobs_consistency completed with {} finding(s)",
|
"blobs_consistency completed with {} finding(s), {} repaired",
|
||||||
finding_count
|
finding_count,
|
||||||
|
repaired_count
|
||||||
);
|
);
|
||||||
return RunOutcome::completed();
|
return RunOutcome::completed_with(serde_json::json!({
|
||||||
|
"repair_requested": repair,
|
||||||
|
"repaired_count": repaired_count,
|
||||||
|
}));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -683,3 +812,64 @@ async fn recompute_hash(
|
|||||||
|
|
||||||
Ok(hasher.finalize().to_hex().to_string())
|
Ok(hasher.finalize().to_hex().to_string())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::infrastructure::repositories::pg::blob_reference_sources::{
|
||||||
|
ChunksReferenceSource, FilesReferenceSource,
|
||||||
|
};
|
||||||
|
|
||||||
|
fn default_registry() -> BlobReferenceRegistry {
|
||||||
|
let pool = Arc::new(
|
||||||
|
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
|
||||||
|
.connect_lazy("postgres://invalid/invalid")
|
||||||
|
.expect("lazy pool never connects"),
|
||||||
|
);
|
||||||
|
let mut registry = BlobReferenceRegistry::new();
|
||||||
|
registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
|
||||||
|
registry.register(Arc::new(ChunksReferenceSource::new(pool)));
|
||||||
|
registry
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Golden test for the chunk-level recompute. Pins the statement
|
||||||
|
/// byte-for-byte because it is assembled from the registry rather than
|
||||||
|
/// written as a literal — the reviewer should read the SQL here.
|
||||||
|
///
|
||||||
|
/// This expression must stay equal to what the query computed before the
|
||||||
|
/// registry existed: the legacy-files term guarded by `NOT EXISTS`, plus
|
||||||
|
/// the manifests-citing-this-chunk term. If a change makes those two
|
||||||
|
/// overlap, every single-chunk CDC file is counted twice and the whole
|
||||||
|
/// table reports `refcount_mismatch`.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn chunk_page_statement_is_stable() {
|
||||||
|
let sql = chunk_page_sql(&default_registry());
|
||||||
|
let expected = r#"SELECT
|
||||||
|
b.hash AS hash,
|
||||||
|
b.size AS size,
|
||||||
|
b.ref_count AS ref_count,
|
||||||
|
b.created_at AS created_at,
|
||||||
|
((SELECT COUNT(*) FROM storage.files cnt_f
|
||||||
|
WHERE cnt_f.blob_hash = b.hash
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM storage.chunk_manifests cnt_m
|
||||||
|
WHERE cnt_m.file_hash = cnt_f.blob_hash
|
||||||
|
))
|
||||||
|
+ (SELECT COUNT(*) FROM storage.chunk_manifests cnt_m
|
||||||
|
WHERE b.hash = ANY(cnt_m.chunk_hashes)))::bigint AS actual_ref_count
|
||||||
|
FROM storage.blobs b
|
||||||
|
WHERE ($1::text IS NULL OR b.hash > $1)
|
||||||
|
ORDER BY b.hash
|
||||||
|
LIMIT $2"#;
|
||||||
|
assert_eq!(sql, expected, "chunk page statement changed:\n{sql}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With no chunk-level source every blob would look unreferenced and the
|
||||||
|
/// sweep would report the entire table as `refcount_mismatch`. Refuse to
|
||||||
|
/// build the statement instead.
|
||||||
|
#[test]
|
||||||
|
#[should_panic(expected = "no chunk-level blob reference source")]
|
||||||
|
fn empty_registry_refuses_to_build_page_statement() {
|
||||||
|
let _ = chunk_page_sql(&BlobReferenceRegistry::new());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -178,6 +178,7 @@ impl JobHandler for ConsistencyBatch {
|
|||||||
"per_check": per_check,
|
"per_check": per_check,
|
||||||
"deep": args.deep,
|
"deep": args.deep,
|
||||||
"force": args.force,
|
"force": args.force,
|
||||||
|
"repair": args.repair,
|
||||||
"ok": ok_count,
|
"ok": ok_count,
|
||||||
"err": err_count,
|
"err": err_count,
|
||||||
}),
|
}),
|
||||||
|
|||||||
@@ -55,6 +55,7 @@ use std::sync::Arc;
|
|||||||
use tokio_util::io::StreamReader;
|
use tokio_util::io::StreamReader;
|
||||||
|
|
||||||
use crate::application::ports::blob_lifecycle::BlobLifecycleHook;
|
use crate::application::ports::blob_lifecycle::BlobLifecycleHook;
|
||||||
|
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
|
||||||
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||||
use crate::application::ports::dedup_ports::{
|
use crate::application::ports::dedup_ports::{
|
||||||
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
||||||
@@ -424,6 +425,51 @@ async fn populate_integrity_blob_sizes<'a>(
|
|||||||
IntegrityBlobSizes { hashes, sizes }
|
IntegrityBlobSizes { hashes, sizes }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Build the manifest reap statement from the registered reference sources.
|
||||||
|
///
|
||||||
|
/// A manifest is collectible when either:
|
||||||
|
/// * `ref_count` reached 0 via `cleanup_if_orphaned` on the single-file
|
||||||
|
/// delete path, **or**
|
||||||
|
/// * nothing references it any more — the bulk-delete path (user cascade,
|
||||||
|
/// `empty_trash`), where the PG trigger only touches `storage.blobs` and
|
||||||
|
/// the per-file `cleanup_if_orphaned` call is skipped, so `ref_count` is
|
||||||
|
/// never decremented and the second clause is the only thing that reaps.
|
||||||
|
///
|
||||||
|
/// The second clause used to name `storage.files` directly, which hardcoded
|
||||||
|
/// "files is the only thing that can reference a manifest". Any new referring
|
||||||
|
/// table — thumbnails via `storage.content_derived_blobs`, previews via
|
||||||
|
/// `storage.file_attached_blobs` — would then have its manifests reaped on the
|
||||||
|
/// next sweep *despite a correct `ref_count`*: clause one false, clause two
|
||||||
|
/// true, `OR` fires, bytes gone. See `docs/plan/derived-blobs.md`.
|
||||||
|
///
|
||||||
|
/// # Panics
|
||||||
|
///
|
||||||
|
/// If no source contributes at [`RefLevel::Manifest`]. That is a wiring bug,
|
||||||
|
/// and it must be loud: with no source, "nothing references it" is vacuously
|
||||||
|
/// true for every row and this statement would delete every manifest in the
|
||||||
|
/// database. `DedupService::new` always registers `FilesReferenceSource`, so
|
||||||
|
/// the only way to reach this is to pass a deliberately empty registry.
|
||||||
|
fn manifest_reap_sql(registry: &BlobReferenceRegistry) -> String {
|
||||||
|
let orphaned = registry
|
||||||
|
.no_reference_predicate(RefLevel::Manifest, "m.file_hash")
|
||||||
|
.expect(
|
||||||
|
"no manifest-level blob reference source registered: the reap \
|
||||||
|
predicate would match every manifest",
|
||||||
|
);
|
||||||
|
|
||||||
|
format!(
|
||||||
|
"DELETE FROM storage.chunk_manifests
|
||||||
|
WHERE ctid = ANY(
|
||||||
|
SELECT ctid
|
||||||
|
FROM storage.chunk_manifests m
|
||||||
|
WHERE m.ref_count <= 0
|
||||||
|
OR {orphaned}
|
||||||
|
LIMIT $1
|
||||||
|
)
|
||||||
|
RETURNING file_hash, chunk_hashes, total_size"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
pub struct DedupService {
|
pub struct DedupService {
|
||||||
/// Pluggable blob storage backend (local FS, S3, …).
|
/// Pluggable blob storage backend (local FS, S3, …).
|
||||||
backend: Arc<dyn BlobStorageBackend>,
|
backend: Arc<dyn BlobStorageBackend>,
|
||||||
@@ -442,6 +488,16 @@ pub struct DedupService {
|
|||||||
/// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
|
/// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
|
||||||
/// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
|
/// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
|
||||||
manifest_cache: moka::future::Cache<String, Arc<ChunkManifest>>,
|
manifest_cache: moka::future::Cache<String, Arc<ChunkManifest>>,
|
||||||
|
/// Every table that holds blob references, so GC agrees with the
|
||||||
|
/// consistency jobs on what "referenced" means. Defaults to the two
|
||||||
|
/// built-in sources; DI replaces it once more tables exist. Never
|
||||||
|
/// optional — an empty registry would make "nothing references it"
|
||||||
|
/// vacuously true and the manifest sweep would reap everything.
|
||||||
|
reference_registry: Arc<BlobReferenceRegistry>,
|
||||||
|
/// The manifest reap statement, built once from `reference_registry`.
|
||||||
|
/// Kept as a field so `garbage_collect` runs a fixed statement rather
|
||||||
|
/// than assembling SQL inside a delete loop — see `manifest_reap_sql`.
|
||||||
|
manifest_reap_sql: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl DedupService {
|
impl DedupService {
|
||||||
@@ -455,15 +511,32 @@ impl DedupService {
|
|||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
maintenance_pool: Arc<PgPool>,
|
maintenance_pool: Arc<PgPool>,
|
||||||
) -> Self {
|
) -> Self {
|
||||||
|
let registry = Arc::new(Self::default_reference_registry(pool.clone()));
|
||||||
Self {
|
Self {
|
||||||
backend,
|
backend,
|
||||||
pool,
|
pool,
|
||||||
maintenance_pool,
|
maintenance_pool,
|
||||||
blob_lifecycle: None,
|
blob_lifecycle: None,
|
||||||
manifest_cache: Self::build_manifest_cache(),
|
manifest_cache: Self::build_manifest_cache(),
|
||||||
|
reference_registry: registry.clone(),
|
||||||
|
manifest_reap_sql: manifest_reap_sql(®istry),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The two sources that were implicit before the registry existed.
|
||||||
|
/// Keeping this as the default means every construction path — including
|
||||||
|
/// tests — has a manifest-level source, so the reap predicate can never
|
||||||
|
/// degenerate to "nothing references anything".
|
||||||
|
fn default_reference_registry(pool: Arc<PgPool>) -> BlobReferenceRegistry {
|
||||||
|
use crate::infrastructure::repositories::pg::blob_reference_sources::{
|
||||||
|
ChunksReferenceSource, FilesReferenceSource,
|
||||||
|
};
|
||||||
|
let mut registry = BlobReferenceRegistry::new();
|
||||||
|
registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
|
||||||
|
registry.register(Arc::new(ChunksReferenceSource::new(pool)));
|
||||||
|
registry
|
||||||
|
}
|
||||||
|
|
||||||
/// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
|
/// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
|
||||||
/// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
|
/// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
|
||||||
fn build_manifest_cache() -> moka::future::Cache<String, Arc<ChunkManifest>> {
|
fn build_manifest_cache() -> moka::future::Cache<String, Arc<ChunkManifest>> {
|
||||||
@@ -476,6 +549,25 @@ impl DedupService {
|
|||||||
.build()
|
.build()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Registers the blob-reference registry used by the manifest reap
|
||||||
|
/// predicate. Without it `garbage_collect` skips manifest collection
|
||||||
|
/// entirely — see `docs/plan/derived-blobs.md`.
|
||||||
|
pub fn with_reference_registry(mut self, registry: Arc<BlobReferenceRegistry>) -> Self {
|
||||||
|
self.manifest_reap_sql = manifest_reap_sql(®istry);
|
||||||
|
self.reference_registry = registry;
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The registry backing the reap predicate.
|
||||||
|
///
|
||||||
|
/// Exposed so `blobs_consistency` recomputes refcounts from the *same*
|
||||||
|
/// source set GC reaps from. If the two ever diverged, the sweep would
|
||||||
|
/// bless counts the collector disagrees with — and the collector wins,
|
||||||
|
/// destructively.
|
||||||
|
pub fn reference_registry(&self) -> Arc<BlobReferenceRegistry> {
|
||||||
|
self.reference_registry.clone()
|
||||||
|
}
|
||||||
|
|
||||||
/// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
|
/// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
|
||||||
pub fn with_blob_lifecycle(mut self, lifecycle: Arc<BlobLifecycleService>) -> Self {
|
pub fn with_blob_lifecycle(mut self, lifecycle: Arc<BlobLifecycleService>) -> Self {
|
||||||
self.blob_lifecycle = Some(lifecycle);
|
self.blob_lifecycle = Some(lifecycle);
|
||||||
@@ -510,12 +602,15 @@ impl DedupService {
|
|||||||
.connect_lazy("postgres://invalid:5432/none")
|
.connect_lazy("postgres://invalid:5432/none")
|
||||||
.unwrap(),
|
.unwrap(),
|
||||||
);
|
);
|
||||||
|
let stub_registry = Arc::new(Self::default_reference_registry(stub_pool.clone()));
|
||||||
Self {
|
Self {
|
||||||
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
|
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
|
||||||
pool: stub_pool.clone(),
|
pool: stub_pool.clone(),
|
||||||
maintenance_pool: stub_pool,
|
maintenance_pool: stub_pool.clone(),
|
||||||
blob_lifecycle: None,
|
blob_lifecycle: None,
|
||||||
manifest_cache: Self::build_manifest_cache(),
|
manifest_cache: Self::build_manifest_cache(),
|
||||||
|
reference_registry: stub_registry.clone(),
|
||||||
|
manifest_reap_sql: manifest_reap_sql(&stub_registry),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -523,6 +618,32 @@ impl DedupService {
|
|||||||
pub async fn initialize(&self) -> Result<(), DomainError> {
|
pub async fn initialize(&self) -> Result<(), DomainError> {
|
||||||
self.backend.initialize().await?;
|
self.backend.initialize().await?;
|
||||||
|
|
||||||
|
// The reap statement is assembled from the registered reference
|
||||||
|
// sources, so it is not greppable in the source tree. It DELETES
|
||||||
|
// manifests, so log it unconditionally at info rather than hiding it
|
||||||
|
// behind a filter an operator has to know to enable — if what GC
|
||||||
|
// considers "referenced" ever changes, that must be visible on the
|
||||||
|
// next boot without anyone going looking.
|
||||||
|
//
|
||||||
|
// Whitespace-collapsed to a single field so a multi-line query does
|
||||||
|
// not sprawl across the boot log; expand it with
|
||||||
|
// `sed 's/ AND / AND\n /g'` or just paste it into psql.
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::dedup",
|
||||||
|
sources = ?self
|
||||||
|
.reference_registry
|
||||||
|
.sources()
|
||||||
|
.iter()
|
||||||
|
.map(|s| s.source_name())
|
||||||
|
.collect::<Vec<_>>(),
|
||||||
|
statement = %self
|
||||||
|
.manifest_reap_sql
|
||||||
|
.split_whitespace()
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(" "),
|
||||||
|
"🧹 manifest reap predicate registered"
|
||||||
|
);
|
||||||
|
|
||||||
let blob_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
let blob_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
|
||||||
.fetch_one(self.pool.as_ref())
|
.fetch_one(self.pool.as_ref())
|
||||||
.await
|
.await
|
||||||
@@ -1838,66 +1959,48 @@ impl DedupService {
|
|||||||
pub async fn cleanup_if_orphaned(&self, hash: &str) {
|
pub async fn cleanup_if_orphaned(&self, hash: &str) {
|
||||||
let short = &hash[..hash.len().min(12)];
|
let short = &hash[..hash.len().min(12)];
|
||||||
|
|
||||||
// ── CDC manifest path (must run FIRST) ───────────────────
|
// 2026-08-23 refactor: this function used to compensate for the
|
||||||
// For single-chunk CDC files file_hash == chunk_hash, so the PG
|
// OLD PG trigger `trg_files_decrement_blob_ref` unconditionally
|
||||||
// trigger on storage.files already decremented storage.blobs.ref_count
|
// decrementing `storage.blobs.ref_count`, which was wrong for
|
||||||
// when this function is called. try_dedup_hit increments
|
// CDC files (their `blob_hash` names a `chunk_manifests.file_hash`,
|
||||||
// chunk_manifests.ref_count but NOT storage.blobs.ref_count, so
|
// not a chunk-in-a-manifest). The compensation branches would:
|
||||||
// blobs.ref_count can reach 0 while the manifest still has ref_count > 1
|
// * Decrement `chunk_manifests.ref_count` a SECOND time (the
|
||||||
// (other files sharing the same blob). Checking the manifest first
|
// trigger having wrongly touched blobs, not the manifest);
|
||||||
// prevents premature blob + manifest deletion.
|
// * Undo the trigger's blob decrement (rc > 1 branch);
|
||||||
let manifest = sqlx::query_as::<_, (i32, Vec<String>)>(
|
// * Call `remove_manifest_reference` (rc <= 1 branch), which
|
||||||
"SELECT ref_count, chunk_hashes \
|
// deletes manifest + dereferences chunks — again duplicating
|
||||||
FROM storage.chunk_manifests WHERE file_hash = $1",
|
// work the trigger should own.
|
||||||
)
|
//
|
||||||
.bind(hash)
|
// Migration `20261017000000_file_delete_trigger_manifest_aware.sql`
|
||||||
.fetch_optional(self.pool.as_ref())
|
// rewrote the trigger to be manifest-aware, so it now correctly
|
||||||
.await
|
// decrements EITHER the manifest OR the blob depending on which
|
||||||
.unwrap_or(None);
|
// one the hash names, walks chunks on last-ref manifest delete,
|
||||||
|
// and leaves the counters in a consistent state without any
|
||||||
if let Some((ref_count, chunk_hashes)) = manifest {
|
// compensation call. Running the old compensation ON TOP of the
|
||||||
if ref_count <= 1 {
|
// new trigger causes double-decrement / double-delete and is
|
||||||
// Last reference — remove manifest and all its chunks.
|
// exactly what broke `dedup_blob_cleanup.hurl` step 7
|
||||||
if let Err(e) = self
|
// (`ref_count == 1` observed 0 after purging one of two dedup
|
||||||
.remove_manifest_reference(hash, ref_count, &chunk_hashes)
|
// uploads).
|
||||||
.await
|
//
|
||||||
{
|
// What remains here: **physical cleanup only**. If the trigger
|
||||||
tracing::warn!("cleanup_if_orphaned: manifest cleanup failed for {short}: {e}");
|
// brought a LEGACY whole-file blob to ref_count = 0 and no
|
||||||
}
|
// manifest still references it (either directly via file_hash or
|
||||||
} else {
|
// indirectly as a chunk in another manifest's chunk_hashes[]),
|
||||||
// Other files still share this blob: just decrement the manifest
|
// reap the DB row and the backend file eagerly. For CDC chunks
|
||||||
// counter and undo the PG trigger's premature chunk ref_count
|
// whose ref_count reached 0 via the trigger's last-ref manifest
|
||||||
// decrement (blobs.ref_count is chunk-level; the manifest is the
|
// path, `dedup_gc` handles physical reap with a grace window
|
||||||
// authoritative file-level counter).
|
// against re-upload races.
|
||||||
sqlx::query(
|
//
|
||||||
"UPDATE storage.chunk_manifests \
|
// Callers can keep invoking `cleanup_if_orphaned` unconditionally
|
||||||
SET ref_count = ref_count - 1 WHERE file_hash = $1",
|
// — for CDC paths it's a cheap no-op (manifest still exists OR
|
||||||
)
|
// the hash never had a blob row), for legacy paths it reaps.
|
||||||
.bind(hash)
|
|
||||||
.execute(self.pool.as_ref())
|
|
||||||
.await
|
|
||||||
.ok();
|
|
||||||
// Undo the PG trigger's decrement of storage.blobs.ref_count.
|
|
||||||
// The trigger fired with blob_hash = file_hash, so only the row
|
|
||||||
// WHERE hash = file_hash is affected. For single-chunk files
|
|
||||||
// file_hash == chunk_hash and that row exists; for multi-chunk
|
|
||||||
// files file_hash is not in storage.blobs, making this a no-op.
|
|
||||||
sqlx::query("UPDATE storage.blobs SET ref_count = ref_count + 1 WHERE hash = $1")
|
|
||||||
.bind(hash)
|
|
||||||
.execute(self.pool.as_ref())
|
|
||||||
.await
|
|
||||||
.ok();
|
|
||||||
tracing::debug!(
|
|
||||||
"cleanup_if_orphaned: manifest {short} ref_count {ref_count}→{}",
|
|
||||||
ref_count - 1
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── Legacy blob path (no manifest) ───────────────────────
|
|
||||||
let deleted_blob = sqlx::query_scalar::<_, String>(
|
let deleted_blob = sqlx::query_scalar::<_, String>(
|
||||||
"DELETE FROM storage.blobs WHERE hash = $1 AND ref_count <= 0 RETURNING hash",
|
"DELETE FROM storage.blobs \
|
||||||
|
WHERE hash = $1 \
|
||||||
|
AND ref_count <= 0 \
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests \
|
||||||
|
WHERE $1 = ANY(chunk_hashes)) \
|
||||||
|
RETURNING hash",
|
||||||
)
|
)
|
||||||
.bind(hash)
|
.bind(hash)
|
||||||
.fetch_optional(self.pool.as_ref())
|
.fetch_optional(self.pool.as_ref())
|
||||||
@@ -1909,7 +2012,7 @@ impl DedupService {
|
|||||||
tracing::warn!("cleanup_if_orphaned: disk delete failed for {short}: {e}");
|
tracing::warn!("cleanup_if_orphaned: disk delete failed for {short}: {e}");
|
||||||
}
|
}
|
||||||
self.fire_blob_hooks(hash);
|
self.fire_blob_hooks(hash);
|
||||||
tracing::info!("cleanup_if_orphaned: removed orphaned blob {short}");
|
tracing::info!("cleanup_if_orphaned: removed orphaned legacy blob {short}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2558,10 +2661,18 @@ impl DedupService {
|
|||||||
// A manifest is collectible when:
|
// A manifest is collectible when:
|
||||||
// • ref_count has been decremented to 0 by cleanup_if_orphaned
|
// • ref_count has been decremented to 0 by cleanup_if_orphaned
|
||||||
// on the single-file-delete service path, OR
|
// on the single-file-delete service path, OR
|
||||||
// • no `storage.files.blob_hash` references its file_hash
|
// • NO registered reference source references its file_hash
|
||||||
// (covers bulk-delete paths: user cascade, empty_trash —
|
// (covers bulk-delete paths: user cascade, empty_trash —
|
||||||
// where the PG trigger only touches storage.blobs and the
|
// where the PG trigger only touches storage.blobs and the
|
||||||
// per-file cleanup_if_orphaned call is skipped).
|
// per-file cleanup_if_orphaned call is skipped).
|
||||||
|
//
|
||||||
|
// The second clause used to name `storage.files` directly. That
|
||||||
|
// hardcoded "files is the only thing that can reference a manifest",
|
||||||
|
// so any new referring table (thumbnails via
|
||||||
|
// storage.content_derived_blobs, …) would see its manifests reaped
|
||||||
|
// on the next sweep despite a correct ref_count — the first clause
|
||||||
|
// is false, the second true, and the OR fires. It is now the union
|
||||||
|
// of every registered source; see docs/plan/derived-blobs.md.
|
||||||
loop {
|
loop {
|
||||||
// Keep the historically cheap DELETE-only shape for the dominant
|
// Keep the historically cheap DELETE-only shape for the dominant
|
||||||
// no-work sweep. Embedding it in the delete/aggregate/update CTE
|
// no-work sweep. Embedding it in the delete/aggregate/update CTE
|
||||||
@@ -2570,23 +2681,14 @@ impl DedupService {
|
|||||||
// update. From two onward, aggregate in-process and issue one UPDATE:
|
// update. From two onward, aggregate in-process and issue one UPDATE:
|
||||||
// the measured crossover is already positive at two, while 500 and
|
// the measured crossover is already positive at two, while 500 and
|
||||||
// 1,000 manifests improve by 60.03x and 51.16x respectively.
|
// 1,000 manifests improve by 60.03x and 51.16x respectively.
|
||||||
let batch: Vec<(String, Vec<String>, i64)> = sqlx::query_as(
|
// Assembled once at construction (see `manifest_reap_sql`), not
|
||||||
"DELETE FROM storage.chunk_manifests
|
// per sweep: no string work in the hot path, a stable statement for
|
||||||
WHERE ctid = ANY(
|
// prepared-statement caching, and a byte-for-byte golden test.
|
||||||
SELECT ctid FROM storage.chunk_manifests m
|
let batch: Vec<(String, Vec<String>, i64)> = sqlx::query_as(&self.manifest_reap_sql)
|
||||||
WHERE m.ref_count <= 0
|
.bind(BATCH_SIZE)
|
||||||
OR NOT EXISTS (
|
.fetch_all(self.maintenance_pool.as_ref())
|
||||||
SELECT 1 FROM storage.files f
|
.await
|
||||||
WHERE f.blob_hash = m.file_hash
|
.map_err(|e| DomainError::internal_error("Dedup", format!("GC manifests: {e}")))?;
|
||||||
)
|
|
||||||
LIMIT $1
|
|
||||||
)
|
|
||||||
RETURNING file_hash, chunk_hashes, total_size",
|
|
||||||
)
|
|
||||||
.bind(BATCH_SIZE)
|
|
||||||
.fetch_all(self.maintenance_pool.as_ref())
|
|
||||||
.await
|
|
||||||
.map_err(|e| DomainError::internal_error("Dedup", format!("GC manifests: {e}")))?;
|
|
||||||
|
|
||||||
if batch.is_empty() {
|
if batch.is_empty() {
|
||||||
break;
|
break;
|
||||||
@@ -3267,6 +3369,41 @@ impl crate::infrastructure::scheduler::JobHandler for DedupService {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
|
/// Golden test for the statement `garbage_collect` runs against production
|
||||||
|
/// data. It is assembled from the registered reference sources rather than
|
||||||
|
/// written as a literal, so this pins the whole thing byte-for-byte — the
|
||||||
|
/// point being that a reviewer reads the SQL *here* instead of mentally
|
||||||
|
/// evaluating the registry.
|
||||||
|
///
|
||||||
|
/// If this fails after adding a source, read the diff carefully: the new
|
||||||
|
/// branch must appear inside the `NOT (...)` group, ORed with the others.
|
||||||
|
/// A branch landing outside that group inverts the predicate for every
|
||||||
|
/// other source and reaps live manifests.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn manifest_reap_statement_is_stable() {
|
||||||
|
let sql = DedupService::new_stub().manifest_reap_sql;
|
||||||
|
let expected = r#"DELETE FROM storage.chunk_manifests
|
||||||
|
WHERE ctid = ANY(
|
||||||
|
SELECT ctid
|
||||||
|
FROM storage.chunk_manifests m
|
||||||
|
WHERE m.ref_count <= 0
|
||||||
|
OR NOT (EXISTS (SELECT 1 FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash))
|
||||||
|
LIMIT $1
|
||||||
|
)
|
||||||
|
RETURNING file_hash, chunk_hashes, total_size"#;
|
||||||
|
assert_eq!(sql, expected, "reap statement changed:\n{sql}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reap predicate must never match a manifest that some source still
|
||||||
|
/// references. With an empty registry `NOT (...)` would have no operands,
|
||||||
|
/// so the builder refuses rather than emitting a statement that deletes
|
||||||
|
/// every manifest in the database.
|
||||||
|
#[test]
|
||||||
|
#[should_panic(expected = "no manifest-level blob reference source")]
|
||||||
|
fn empty_registry_refuses_to_build_reap_statement() {
|
||||||
|
let _ = manifest_reap_sql(&BlobReferenceRegistry::new());
|
||||||
|
}
|
||||||
use std::collections::HashSet;
|
use std::collections::HashSet;
|
||||||
use tempfile::NamedTempFile;
|
use tempfile::NamedTempFile;
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,462 @@
|
|||||||
|
//! Reconciles `storage.chunk_manifests.ref_count` against its actual
|
||||||
|
//! referrers.
|
||||||
|
//!
|
||||||
|
//! ### Why this exists
|
||||||
|
//!
|
||||||
|
//! There are **two** reference counters, and only one of them was ever
|
||||||
|
//! verified. `DedupService::add_reference` bumps
|
||||||
|
//! `chunk_manifests.ref_count` first and only falls back to
|
||||||
|
//! `storage.blobs.ref_count`, so a reference lands on whichever counter
|
||||||
|
//! its hash names:
|
||||||
|
//!
|
||||||
|
//! * a **chunk** reference → `storage.blobs.ref_count`, reconciled by
|
||||||
|
//! `blobs_consistency::refcount_mismatch`;
|
||||||
|
//! * a **Blob** reference (a CDC file, and now every derived artifact) →
|
||||||
|
//! `chunk_manifests.ref_count`, reconciled by **nothing** before this
|
||||||
|
//! job existed.
|
||||||
|
//!
|
||||||
|
//! That gap was survivable only because `dedup_gc`'s reap predicate had a
|
||||||
|
//! second clause — "no `storage.files` row references this manifest" —
|
||||||
|
//! which quietly compensated for drift on the bulk-delete paths where
|
||||||
|
//! `ref_count` is never decremented. Generalising that clause to the
|
||||||
|
//! reference registry (so thumbnails stop being reaped) removes the
|
||||||
|
//! compensation, which is exactly why the manifest counter now has to be
|
||||||
|
//! checked directly. See `docs/plan/derived-blobs.md`.
|
||||||
|
//!
|
||||||
|
//! ### The check
|
||||||
|
//!
|
||||||
|
//! * `manifest_refcount_mismatch` (severity `inconsistent`) —
|
||||||
|
//! `chunk_manifests.ref_count` disagrees with the number of registered
|
||||||
|
//! referrers. An **under**-count is the dangerous direction: GC reaps a
|
||||||
|
//! manifest whose content is still reachable, taking its chunks with it.
|
||||||
|
//! An over-count merely pins storage. Content-safe to report either way
|
||||||
|
//! — the manifest row and its chunks are intact, the counter is wrong.
|
||||||
|
//!
|
||||||
|
//! ### Why a separate job rather than a phase of `blobs_consistency`
|
||||||
|
//!
|
||||||
|
//! One subject per job, per the subject-iteration principle the other five
|
||||||
|
//! consistency tenants follow. It also avoids changing the cursor format of
|
||||||
|
//! an existing *recoverable* job, which would strand any run paused across
|
||||||
|
//! the deploy.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use async_trait::async_trait;
|
||||||
|
use sqlx::PgPool;
|
||||||
|
|
||||||
|
use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
|
||||||
|
use crate::infrastructure::scheduler::{
|
||||||
|
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
|
||||||
|
RunStatus, record_or_log,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub const MANIFESTS_CONSISTENCY_JOB_NAME: &str = "manifests_consistency";
|
||||||
|
|
||||||
|
/// Rows per batch. Each row costs one indexed subquery per registered
|
||||||
|
/// source; 200 matches `blobs_consistency` so the cancel-poll cadence is
|
||||||
|
/// the same for an operator watching either job.
|
||||||
|
const BATCH_SIZE: i64 = 200;
|
||||||
|
|
||||||
|
/// The page query, with `actual_ref_count` summed from the registered
|
||||||
|
/// reference sources at [`RefLevel::Manifest`].
|
||||||
|
///
|
||||||
|
/// Only sources that reference a **Blob** contribute — `storage.files`
|
||||||
|
/// today, plus `storage.content_derived_blobs` and
|
||||||
|
/// `storage.file_attached_blobs` once they exist.
|
||||||
|
/// `ChunksReferenceSource` returns `None` here: a manifest is never
|
||||||
|
/// referenced by another manifest, and including it would count this
|
||||||
|
/// manifest's own chunks as referrers of itself.
|
||||||
|
///
|
||||||
|
/// # Panics
|
||||||
|
///
|
||||||
|
/// If no source contributes at [`RefLevel::Manifest`] — a wiring bug that
|
||||||
|
/// would report every manifest as mismatched.
|
||||||
|
fn manifest_page_sql(registry: &BlobReferenceRegistry) -> String {
|
||||||
|
let expected = registry.ref_count_expr(RefLevel::Manifest, "m.file_hash");
|
||||||
|
assert!(
|
||||||
|
expected != "0",
|
||||||
|
"no manifest-level blob reference source registered: every manifest \
|
||||||
|
would appear unreferenced"
|
||||||
|
);
|
||||||
|
|
||||||
|
format!(
|
||||||
|
"SELECT
|
||||||
|
m.file_hash AS file_hash,
|
||||||
|
m.ref_count AS ref_count,
|
||||||
|
m.total_size AS total_size,
|
||||||
|
m.chunk_count AS chunk_count,
|
||||||
|
({expected})::bigint AS actual_ref_count
|
||||||
|
FROM storage.chunk_manifests m
|
||||||
|
WHERE ($1::text IS NULL OR m.file_hash > $1)
|
||||||
|
ORDER BY m.file_hash
|
||||||
|
LIMIT $2"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct ManifestsConsistencyCheck {
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
/// Built once from the blob-reference registry so this recompute and
|
||||||
|
/// `dedup_gc`'s reap predicate answer "what references this manifest"
|
||||||
|
/// identically. Assembled at construction rather than per page so the
|
||||||
|
/// sweep runs a fixed statement.
|
||||||
|
page_sql: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ManifestsConsistencyCheck {
|
||||||
|
pub fn new(pool: Arc<PgPool>, reference_registry: Arc<BlobReferenceRegistry>) -> Self {
|
||||||
|
Self {
|
||||||
|
pool,
|
||||||
|
page_sql: manifest_page_sql(&reference_registry),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Chainable self-registration. On-demand only — operators fire it
|
||||||
|
/// from `POST /api/admin/jobs/manifests_consistency/trigger`.
|
||||||
|
pub async fn register_recoverable_job(
|
||||||
|
self: Arc<Self>,
|
||||||
|
registry: &JobRegistry,
|
||||||
|
provider: &Arc<dyn JobStoreProvider>,
|
||||||
|
) -> Arc<Self> {
|
||||||
|
registry
|
||||||
|
.register_recoverable_job(self.clone(), provider.clone(), None)
|
||||||
|
.await;
|
||||||
|
self
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, sqlx::FromRow)]
|
||||||
|
struct ManifestRow {
|
||||||
|
file_hash: String,
|
||||||
|
ref_count: i32,
|
||||||
|
total_size: i64,
|
||||||
|
chunk_count: i32,
|
||||||
|
actual_ref_count: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[async_trait]
|
||||||
|
impl RecoverableJobHandler for ManifestsConsistencyCheck {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
MANIFESTS_CONSISTENCY_JOB_NAME
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn count_total(&self) -> Option<u64> {
|
||||||
|
let row: Result<(i64,), sqlx::Error> =
|
||||||
|
sqlx::query_as("SELECT COUNT(*) FROM storage.chunk_manifests")
|
||||||
|
.fetch_one(self.pool.as_ref())
|
||||||
|
.await;
|
||||||
|
match row {
|
||||||
|
Ok((n,)) => Some(n.max(0) as u64),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::debug!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.count_total_failed",
|
||||||
|
error = %e,
|
||||||
|
"count_total failed — run will not surface a progress bar"
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_resumable(
|
||||||
|
&self,
|
||||||
|
store: &dyn JobStore,
|
||||||
|
args: &JobRunArgs,
|
||||||
|
resume_cursor: Option<Vec<u8>>,
|
||||||
|
) -> RunOutcome {
|
||||||
|
let is_fresh = resume_cursor.is_none();
|
||||||
|
|
||||||
|
// Cursor: the last `file_hash` as UTF-8. Same convention as
|
||||||
|
// `blobs_consistency`, which also pages a hash-keyed table.
|
||||||
|
let mut cursor: Option<String> = match resume_cursor {
|
||||||
|
None => None,
|
||||||
|
Some(bytes) if bytes.is_empty() => None,
|
||||||
|
Some(bytes) => match String::from_utf8(bytes) {
|
||||||
|
Ok(s) => Some(s),
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("invalid cursor: not valid UTF-8: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// Persist the repair flag into `params.repair` so the admin
|
||||||
|
// run-detail view can display whether the run was a discovery
|
||||||
|
// scan or an active repair. Fresh takes it from args; Resume
|
||||||
|
// reads back so a paused repair scan stays a repair scan (a
|
||||||
|
// mid-scan crash mustn't silently downgrade the remaining
|
||||||
|
// rows to discovery-only). Same shape as
|
||||||
|
// `blobs_consistency_service.rs`'s `deep` handling — see the
|
||||||
|
// reasoning documented there.
|
||||||
|
let repair = if is_fresh {
|
||||||
|
let v = if args.repair { "true" } else { "false" };
|
||||||
|
if let Err(e) = store.set_string_param("repair", v).await {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("failed to persist repair flag to params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
args.repair
|
||||||
|
} else {
|
||||||
|
match store.get_string_param("repair").await {
|
||||||
|
Ok(Some(v)) => v == "true",
|
||||||
|
Ok(None) => false,
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("read `repair` from params: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if repair {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.repair_mode_active",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
"repair mode: manifest_refcount_mismatch findings will trigger corrective UPDATE"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut finding_count = 0u64;
|
||||||
|
// Only relevant when `repair == true`. Reported inline in
|
||||||
|
// the completion log + the `extra_stats` payload so operators
|
||||||
|
// can see "we found N and fixed M" in one line.
|
||||||
|
let mut repaired_count = 0u64;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
// Cooperative cancel poll between batches.
|
||||||
|
match store.status().await {
|
||||||
|
Ok(RunStatus::CancelRequested) => {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.cancelled",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
finding_count = finding_count,
|
||||||
|
"manifests_consistency cancelled cooperatively, pausing"
|
||||||
|
);
|
||||||
|
return RunOutcome::Paused {
|
||||||
|
cursor: cursor
|
||||||
|
.as_ref()
|
||||||
|
.map(|s| s.as_bytes().to_vec())
|
||||||
|
.unwrap_or_default(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
Ok(_) => {}
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("status poll: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let rows: Vec<ManifestRow> = match sqlx::query_as(&self.page_sql)
|
||||||
|
.bind(cursor.as_deref())
|
||||||
|
.bind(BATCH_SIZE)
|
||||||
|
.fetch_all(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(r) => r,
|
||||||
|
Err(e) => {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("batch fetch: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if rows.is_empty() {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.completed",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
finding_count = finding_count,
|
||||||
|
repaired_count = repaired_count,
|
||||||
|
repair_requested = repair,
|
||||||
|
"manifests_consistency completed with {} finding(s), {} repaired",
|
||||||
|
finding_count,
|
||||||
|
repaired_count
|
||||||
|
);
|
||||||
|
return RunOutcome::completed_with(serde_json::json!({
|
||||||
|
"repair_requested": repair,
|
||||||
|
"repaired_count": repaired_count,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
for row in &rows {
|
||||||
|
if row.ref_count as i64 == row.actual_ref_count {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
finding_count += 1;
|
||||||
|
let delta = row.actual_ref_count - row.ref_count as i64;
|
||||||
|
record_or_log(
|
||||||
|
store,
|
||||||
|
MANIFESTS_CONSISTENCY_JOB_NAME,
|
||||||
|
"manifest_refcount_mismatch",
|
||||||
|
"inconsistent",
|
||||||
|
None, // a hash isn't a UUID; the identifier lives in detail
|
||||||
|
serde_json::json!({
|
||||||
|
"file_hash": row.file_hash,
|
||||||
|
"stored": row.ref_count,
|
||||||
|
"actual": row.actual_ref_count,
|
||||||
|
"delta": delta,
|
||||||
|
"total_size": row.total_size,
|
||||||
|
"chunk_count": row.chunk_count,
|
||||||
|
// Under-count is the dangerous direction: GC reaps a
|
||||||
|
// manifest whose content is still reachable.
|
||||||
|
"reap_risk": delta > 0,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
// Repair pass — content-safe corrective UPDATE. The
|
||||||
|
// stored counter is set to what the auditor formula
|
||||||
|
// would compute at UPDATE time (subquery matches
|
||||||
|
// `manifest_page_sql`'s `actual_ref_count` predicate),
|
||||||
|
// so a concurrent file insert/delete between our page
|
||||||
|
// fetch and this UPDATE can't leave a stale value —
|
||||||
|
// the subquery re-reads inside the same statement.
|
||||||
|
// The `<> (subquery)` guard makes the UPDATE a no-op
|
||||||
|
// if the value is already correct, so this is
|
||||||
|
// idempotent under retry.
|
||||||
|
if repair {
|
||||||
|
match sqlx::query(
|
||||||
|
"UPDATE storage.chunk_manifests m \
|
||||||
|
SET ref_count = ( \
|
||||||
|
SELECT COUNT(*) FROM storage.files \
|
||||||
|
WHERE blob_hash = m.file_hash \
|
||||||
|
) \
|
||||||
|
WHERE m.file_hash = $1 \
|
||||||
|
AND m.ref_count <> ( \
|
||||||
|
SELECT COUNT(*) FROM storage.files \
|
||||||
|
WHERE blob_hash = m.file_hash \
|
||||||
|
)",
|
||||||
|
)
|
||||||
|
.bind(&row.file_hash)
|
||||||
|
.execute(self.pool.as_ref())
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(res) if res.rows_affected() > 0 => {
|
||||||
|
repaired_count += 1;
|
||||||
|
tracing::info!(
|
||||||
|
target: "audit",
|
||||||
|
event = "manifests_consistency.repaired",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
file_hash = %row.file_hash,
|
||||||
|
stored_was = row.ref_count,
|
||||||
|
actual = row.actual_ref_count,
|
||||||
|
"🩹 manifest ref_count repaired"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(_) => {
|
||||||
|
// Row not touched — either another concurrent
|
||||||
|
// repair fixed it first, or the drift healed
|
||||||
|
// itself between page fetch and UPDATE.
|
||||||
|
// Silent no-op.
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.repair_failed",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
file_hash = %row.file_hash,
|
||||||
|
error = %e,
|
||||||
|
"manifest ref_count repair UPDATE failed — finding stays"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Advance cursor + checkpoint.
|
||||||
|
let last_hash = rows
|
||||||
|
.last()
|
||||||
|
.map(|r| r.file_hash.clone())
|
||||||
|
.expect("non-empty rows");
|
||||||
|
cursor = Some(last_hash.clone());
|
||||||
|
let batch_len = rows.len() as u64;
|
||||||
|
if let Err(e) = store.checkpoint(last_hash.into_bytes(), batch_len).await {
|
||||||
|
return RunOutcome::Failed {
|
||||||
|
message: format!("checkpoint: {e}"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (rows.len() as i64) < BATCH_SIZE {
|
||||||
|
tracing::info!(
|
||||||
|
target: "oxicloud::consistency",
|
||||||
|
event = "manifests_consistency.completed",
|
||||||
|
run_id = %store.run_id(),
|
||||||
|
finding_count = finding_count,
|
||||||
|
repaired_count = repaired_count,
|
||||||
|
repair_requested = repair,
|
||||||
|
"manifests_consistency completed with {} finding(s), {} repaired",
|
||||||
|
finding_count,
|
||||||
|
repaired_count
|
||||||
|
);
|
||||||
|
return RunOutcome::completed_with(serde_json::json!({
|
||||||
|
"repair_requested": repair,
|
||||||
|
"repaired_count": repaired_count,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::infrastructure::repositories::pg::blob_reference_sources::{
|
||||||
|
ChunksReferenceSource, FilesReferenceSource,
|
||||||
|
};
|
||||||
|
|
||||||
|
fn default_registry() -> BlobReferenceRegistry {
|
||||||
|
let pool = Arc::new(
|
||||||
|
sqlx::pool::PoolOptions::<sqlx::Postgres>::new()
|
||||||
|
.connect_lazy("postgres://invalid/invalid")
|
||||||
|
.expect("lazy pool never connects"),
|
||||||
|
);
|
||||||
|
let mut registry = BlobReferenceRegistry::new();
|
||||||
|
registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
|
||||||
|
registry.register(Arc::new(ChunksReferenceSource::new(pool)));
|
||||||
|
registry
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Golden test — the statement is assembled from the registry, so pin it
|
||||||
|
/// byte-for-byte and read the SQL here rather than deriving it mentally.
|
||||||
|
///
|
||||||
|
/// Two invariants a future source must not break: the files term carries
|
||||||
|
/// **no** `NOT EXISTS` guard (that guard exists to keep CDC rows out of
|
||||||
|
/// the *chunk* level; applying it here would count nothing), and
|
||||||
|
/// `chunk_hashes` appears nowhere — a manifest citing its own chunks is
|
||||||
|
/// not a referrer of itself.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn manifest_page_statement_is_stable() {
|
||||||
|
let sql = manifest_page_sql(&default_registry());
|
||||||
|
let expected = r#"SELECT
|
||||||
|
m.file_hash AS file_hash,
|
||||||
|
m.ref_count AS ref_count,
|
||||||
|
m.total_size AS total_size,
|
||||||
|
m.chunk_count AS chunk_count,
|
||||||
|
((SELECT COUNT(*) FROM storage.files cnt_f
|
||||||
|
WHERE cnt_f.blob_hash = m.file_hash))::bigint AS actual_ref_count
|
||||||
|
FROM storage.chunk_manifests m
|
||||||
|
WHERE ($1::text IS NULL OR m.file_hash > $1)
|
||||||
|
ORDER BY m.file_hash
|
||||||
|
LIMIT $2"#;
|
||||||
|
assert_eq!(sql, expected, "manifest page statement changed:\n{sql}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn chunks_source_contributes_nothing_at_manifest_level() {
|
||||||
|
let sql = manifest_page_sql(&default_registry());
|
||||||
|
assert!(
|
||||||
|
!sql.contains("chunk_hashes"),
|
||||||
|
"a manifest must not count its own chunks as referrers: {sql}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
#[should_panic(expected = "no manifest-level blob reference source")]
|
||||||
|
fn empty_registry_refuses_to_build_page_statement() {
|
||||||
|
let _ = manifest_page_sql(&BlobReferenceRegistry::new());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -31,6 +31,7 @@ pub mod last_seen_tracker;
|
|||||||
pub mod local_blob_backend;
|
pub mod local_blob_backend;
|
||||||
pub mod local_fs_mount_provider;
|
pub mod local_fs_mount_provider;
|
||||||
pub mod login_lockout_service;
|
pub mod login_lockout_service;
|
||||||
|
pub mod manifests_consistency_service;
|
||||||
pub mod media_metadata_service;
|
pub mod media_metadata_service;
|
||||||
pub mod mock_email_sender;
|
pub mod mock_email_sender;
|
||||||
pub mod mount_provider_factory;
|
pub mod mount_provider_factory;
|
||||||
|
|||||||
@@ -2575,6 +2575,12 @@ pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse
|
|||||||
/// `deep=true` opts into slow variants — `consistency_batch` fans it
|
/// `deep=true` opts into slow variants — `consistency_batch` fans it
|
||||||
/// out to sub-jobs; `storage_consistency` (when implemented) will
|
/// out to sub-jobs; `storage_consistency` (when implemented) will
|
||||||
/// re-BLAKE3 each blob for bitrot detection. See `JobRunArgs.deep`.
|
/// re-BLAKE3 each blob for bitrot detection. See `JobRunArgs.deep`.
|
||||||
|
///
|
||||||
|
/// `repair=true` opts into corrective action on the refcount
|
||||||
|
/// consistency tenants (`blobs_consistency`, `manifests_consistency`,
|
||||||
|
/// and `consistency_batch` which fans out to both). Default `false`
|
||||||
|
/// preserves discovery-only. See `JobRunArgs.repair` for the
|
||||||
|
/// content-safety and race-safety guarantees.
|
||||||
#[derive(serde::Deserialize)]
|
#[derive(serde::Deserialize)]
|
||||||
pub struct TriggerJobQuery {
|
pub struct TriggerJobQuery {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
@@ -2591,6 +2597,8 @@ pub struct TriggerJobQuery {
|
|||||||
/// `AppConfig.storage_entries`.
|
/// `AppConfig.storage_entries`.
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub storage: Option<String>,
|
pub storage: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
pub repair: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `POST /api/admin/jobs/{name}/trigger` — dispatch one run off-schedule.
|
/// `POST /api/admin/jobs/{name}/trigger` — dispatch one run off-schedule.
|
||||||
@@ -2629,15 +2637,18 @@ pub async fn trigger_job(
|
|||||||
job = %name,
|
job = %name,
|
||||||
force = query.force,
|
force = query.force,
|
||||||
deep = query.deep,
|
deep = query.deep,
|
||||||
"👮🏻♂️ Admin triggered job {} (force={}, deep={})",
|
repair = query.repair,
|
||||||
|
"👮🏻♂️ Admin triggered job {} (force={}, deep={}, repair={})",
|
||||||
name,
|
name,
|
||||||
query.force,
|
query.force,
|
||||||
query.deep,
|
query.deep,
|
||||||
|
query.repair,
|
||||||
);
|
);
|
||||||
let args = JobRunArgs {
|
let args = JobRunArgs {
|
||||||
force: query.force,
|
force: query.force,
|
||||||
deep: query.deep,
|
deep: query.deep,
|
||||||
storage: query.storage.clone(),
|
storage: query.storage.clone(),
|
||||||
|
repair: query.repair,
|
||||||
};
|
};
|
||||||
|
|
||||||
// Jobs that can run for hours (backend_migration, future
|
// Jobs that can run for hours (backend_migration, future
|
||||||
|
|||||||
+17
-11
@@ -225,9 +225,10 @@ jsonpath "$.outcome.count" exists
|
|||||||
# Step 4c — Trigger `consistency_batch`. Coordinator (plain
|
# Step 4c — Trigger `consistency_batch`. Coordinator (plain
|
||||||
# JobHandler) — snapshots the registry, filters names
|
# JobHandler) — snapshots the registry, filters names
|
||||||
# ending `_consistency`, sequentially triggers each.
|
# ending `_consistency`, sequentially triggers each.
|
||||||
# `outcome.count` = number of children dispatched (5 as
|
# `outcome.count` = number of children dispatched (6 as
|
||||||
# of Slice 10: drives + folders + files + blobs +
|
# of the refcount_cascade fix: drives + folders +
|
||||||
# backend). `extra.per_check` carries a per-child outcome
|
# files + blobs + manifests + backend). `extra.per_check`
|
||||||
|
# carries a per-child outcome
|
||||||
# map. Batch itself always returns ok — child failures
|
# map. Batch itself always returns ok — child failures
|
||||||
# live inside per_check. `?deep=true` propagates as
|
# live inside per_check. `?deep=true` propagates as
|
||||||
# `extra.deep`.
|
# `extra.deep`.
|
||||||
@@ -239,16 +240,21 @@ HTTP 200
|
|||||||
[Asserts]
|
[Asserts]
|
||||||
jsonpath "$.ok" == true
|
jsonpath "$.ok" == true
|
||||||
jsonpath "$.outcome.outcome" == "ok"
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
jsonpath "$.outcome.count" == 5
|
jsonpath "$.outcome.count" == 6
|
||||||
jsonpath "$.outcome.extra.deep" == true
|
jsonpath "$.outcome.extra.deep" == true
|
||||||
jsonpath "$.outcome.extra.ok" == 5
|
jsonpath "$.outcome.extra.ok" == 6
|
||||||
jsonpath "$.outcome.extra.err" == 0
|
jsonpath "$.outcome.extra.err" == 0
|
||||||
# per_check is keyed by child job name.
|
# per_check is keyed by child job name. `manifests_consistency` was added
|
||||||
jsonpath "$.outcome.extra.per_check.drives_consistency.outcome" == "ok"
|
# by the refcount_cascade fix — see docs/plan/derived-blobs.md and
|
||||||
jsonpath "$.outcome.extra.per_check.folders_consistency.outcome" == "ok"
|
# `[[bug_dual_refcount_divergence]]` for why the second counter needed
|
||||||
jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok"
|
# its own tenant. Auto-picked by `consistency_batch` via `.ends_with(
|
||||||
jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok"
|
# "_consistency")` (no explicit list in the batch service).
|
||||||
jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok"
|
jsonpath "$.outcome.extra.per_check.drives_consistency.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.extra.per_check.folders_consistency.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.extra.per_check.manifests_consistency.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok"
|
||||||
|
|
||||||
|
|
||||||
# ─────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -0,0 +1,614 @@
|
|||||||
|
# =============================================================
|
||||||
|
# OxiCloud – Copy-folder ref_count regression
|
||||||
|
# =============================================================
|
||||||
|
# Regression test for the ref_count drift found on Ed's sandbox
|
||||||
|
# (2026-08-22) and traced to the copy-folder path. When a folder
|
||||||
|
# is copied, every file inside gets duplicated as a NEW file row
|
||||||
|
# pointing at the SAME blob(s) — dedup wins bytes-on-disk, but
|
||||||
|
# `storage.blobs.ref_count` MUST bump by the number of new refs.
|
||||||
|
# If it doesn't, dedup GC will reap a blob that a live file row
|
||||||
|
# still references → dangling reference → user gets 404 on
|
||||||
|
# download of the copied file.
|
||||||
|
#
|
||||||
|
# Covers two paths so the CDC boundary can't hide a regression:
|
||||||
|
#
|
||||||
|
# 1. Small file (`refcount-cascade-small.txt`) → single legacy whole-
|
||||||
|
# file blob. `storage.blobs.ref_count` counted directly on
|
||||||
|
# the file's content hash.
|
||||||
|
# 2. 2 MB file (`refcount-cascade-cdc.bin`) → FastCDC produces multiple
|
||||||
|
# distinct chunks. Whole-file `content_hash` still resolves
|
||||||
|
# through the dedup API.
|
||||||
|
#
|
||||||
|
# Both fixtures are DEDICATED — unique content so ref_count
|
||||||
|
# assertions are absolute (== 1, == 2). Do NOT reuse these
|
||||||
|
# fixtures in other hurl files or absolute assertions here will
|
||||||
|
# flake.
|
||||||
|
#
|
||||||
|
# Deletion is TWO STEPS in OxiCloud:
|
||||||
|
# `DELETE /api/folders/{id}` → moves to trash (ref_count
|
||||||
|
# unchanged; children still
|
||||||
|
# reference the blob).
|
||||||
|
# `DELETE /api/trash/{id}` → permanent purge; NOW
|
||||||
|
# ref_count decrements. If it
|
||||||
|
# hits 0, blob row is deleted
|
||||||
|
# synchronously (`exists=false`).
|
||||||
|
# So the test purges trash after every folder-delete step —
|
||||||
|
# skipping that would make the assertions wrong regardless of
|
||||||
|
# whether the copy-side bug is present.
|
||||||
|
#
|
||||||
|
# `/api/dedup/check/{hash}` returns `ref_count` only for admin
|
||||||
|
# callers (regular users get `null` for anti-enumeration). This
|
||||||
|
# suite requires the admin token; setup.hurl seeds it.
|
||||||
|
#
|
||||||
|
# Run:
|
||||||
|
# hurl --variables-file tests/api/test.env --test \
|
||||||
|
# tests/api/refcount_cascade.hurl
|
||||||
|
# =============================================================
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 1 — Login (admin, required for ref_count in dedup API)
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/auth/login
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"username": "{{username}}",
|
||||||
|
"password": "{{password}}"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Captures]
|
||||||
|
token: jsonpath "$.access_token"
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 2 — Baseline sweeps: run BOTH consistency tenants that
|
||||||
|
# check ref_count invariants and capture their finding
|
||||||
|
# counts. Later checkpoints assert equality with these
|
||||||
|
# baselines instead of `== 0` — so stale findings from
|
||||||
|
# previous tests don't flunk this one; only NEW drift
|
||||||
|
# introduced by our copy/delete does.
|
||||||
|
#
|
||||||
|
# Two tenants because there are two counters (see
|
||||||
|
# `[[bug_dual_refcount_divergence]]`):
|
||||||
|
#
|
||||||
|
# - `blobs_consistency` — reconciles
|
||||||
|
# `storage.blobs.ref_count` against its auditor
|
||||||
|
# formula. Catches chunk-level drift.
|
||||||
|
# - `manifests_consistency` — reconciles
|
||||||
|
# `storage.chunk_manifests.ref_count` against its
|
||||||
|
# auditor formula. Catches whole-file drift (the
|
||||||
|
# path the FE + `/api/dedup/check` surface reads).
|
||||||
|
#
|
||||||
|
# Trigger returns `outcome.count = stats.finding_count`
|
||||||
|
# for recoverable tenants (see
|
||||||
|
# `scheduler/recoverable.rs::JobOutcome::ok_with` in
|
||||||
|
# the Completed branch). `outcome.outcome == "ok"`
|
||||||
|
# means the run walked the whole subject; it does NOT
|
||||||
|
# mean zero findings.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Captures]
|
||||||
|
baseline_blobs_findings: jsonpath "$.outcome.count"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Captures]
|
||||||
|
baseline_manifests_findings: jsonpath "$.outcome.count"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================
|
||||||
|
# Scenario A — small file (single legacy whole-file blob)
|
||||||
|
# =============================================================
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A1 — Create source folder
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/folders
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"name": "hurl-ref-source-small"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
src_small_id: jsonpath "$.id"
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A2 — Upload the small fixture. Content is unique to this
|
||||||
|
# test, so ref_count starts at exactly 1 (no dedup
|
||||||
|
# collision with any other fixture in the suite).
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/files/upload
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
[MultipartFormData]
|
||||||
|
folder_id: {{src_small_id}}
|
||||||
|
file: file,fixtures/refcount-cascade-small.txt; text/plain
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
small_file_id: jsonpath "$.id"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.content_hash" == "2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3"
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A3 — Baseline: exactly one live reference.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A4 — Create target folder + copy source into it. The batch
|
||||||
|
# endpoint is the single entry point every FE / WebDAV
|
||||||
|
# code path funnels through.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/folders
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"name": "hurl-ref-target-small"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
tgt_small_id: jsonpath "$.id"
|
||||||
|
|
||||||
|
POST {{base_url}}/api/batch/folders/copy
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"folder_ids": ["{{src_small_id}}"],
|
||||||
|
"target_folder_id": "{{tgt_small_id}}"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Captures]
|
||||||
|
copy_small_root: jsonpath "$.successful[0].new_root_folder_id"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.stats.successful" == 1
|
||||||
|
jsonpath "$.stats.failed" == 0
|
||||||
|
jsonpath "$.successful[0].files_copied" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A5 — COPY-SIDE ASSERTION: ref_count must be exactly 2. The
|
||||||
|
# pre-fix bug left this at 1 — a subsequent GC would then
|
||||||
|
# reap the blob out from under the copied file.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A5b — Sweep both consistency tenants that check ref_count.
|
||||||
|
# Complements the single-hash probe above: if the copy
|
||||||
|
# path miscounted some OTHER blob or manifest, the single-
|
||||||
|
# hash probe wouldn't catch it. Delta vs the baselines
|
||||||
|
# isolates NEW drift from ambient.
|
||||||
|
#
|
||||||
|
# Both tenants required — one counter each; see
|
||||||
|
# [[bug_dual_refcount_divergence]] for why the copy path
|
||||||
|
# must maintain both symmetrically.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A6 — Soft-delete the copied folder tree (moves to trash).
|
||||||
|
# ref_count is EXPECTED to stay at 2 — trashed files
|
||||||
|
# still reference the blob per the `NOT is_trashed` gap
|
||||||
|
# the auditor deliberately closed (see
|
||||||
|
# `[[project_by_hash_drop_is_trashed_filter]]`). Asserting
|
||||||
|
# == 2 here makes the trash-vs-permanent boundary explicit
|
||||||
|
# so a future refactor that changed the semantics would
|
||||||
|
# surface at THIS line, not several steps downstream.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/folders/{{copy_small_root}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A7 — Purge the copy folder permanently.
|
||||||
|
# `DELETE /api/trash/{id}` accepts the ORIGINAL resource
|
||||||
|
# id as the path param (verified in `trash_handler.rs::
|
||||||
|
# delete_permanently`) — no need to GET+filter the trash
|
||||||
|
# listing to translate. If soft-delete silently failed,
|
||||||
|
# the ref_count assertion two lines below catches it.
|
||||||
|
# ref_count must drop to exactly 1 (the source folder
|
||||||
|
# still holds its file). Guards the DECREMENT half of
|
||||||
|
# the invariant: a double-decrement here would go to 0
|
||||||
|
# and the next GC wipes the still-live original's blob.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/trash/{{copy_small_root}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A8 — Soft-delete + purge the SOURCE folder (last holder).
|
||||||
|
# ref_count hits 0 → blob row deleted synchronously →
|
||||||
|
# `exists == false` on the next probe. Mirror pattern to
|
||||||
|
# `dedup_blob_cleanup.hurl` step 10, applied to folder-
|
||||||
|
# scoped delete.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/folders/{{src_small_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
|
||||||
|
DELETE {{base_url}}/api/trash/{{src_small_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == false
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# A9 — End-of-Scenario-A sweep on BOTH tenants. Scenario A
|
||||||
|
# introduced two file rows (source + copy), then deleted
|
||||||
|
# both. Net effect on the DB is zero — so the drift count
|
||||||
|
# on each counter must be exactly the baseline.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================
|
||||||
|
# Scenario B — 2 MB multi-chunk file (CDC manifest path)
|
||||||
|
#
|
||||||
|
# Coverage: this scenario now sweeps BOTH `blobs_consistency`
|
||||||
|
# (chunk-level ref_counts on `storage.blobs.ref_count`) and
|
||||||
|
# `manifests_consistency` (whole-file ref_counts on
|
||||||
|
# `storage.chunk_manifests.ref_count`). Same-shape assertions
|
||||||
|
# as Scenario A — see the baseline capture block near the top
|
||||||
|
# of this file and [[bug_dual_refcount_divergence]] for why
|
||||||
|
# both are needed.
|
||||||
|
# =============================================================
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B1 — Source folder for the multi-chunk scenario. Isolated
|
||||||
|
# from Scenario A so deletion order can't mask a bug (e.g.
|
||||||
|
# a shared blob whose counter goes negative).
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/folders
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"name": "hurl-ref-source-cdc"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
src_cdc_id: jsonpath "$.id"
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B2 — Upload the 2 MB dedicated fixture. Deterministic per-
|
||||||
|
# position content → FastCDC produces multiple distinct
|
||||||
|
# chunks (no chunk-level dedup with any other fixture).
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/files/upload
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
[MultipartFormData]
|
||||||
|
folder_id: {{src_cdc_id}}
|
||||||
|
file: file,fixtures/refcount-cascade-cdc.bin; application/octet-stream
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
cdc_file_id: jsonpath "$.id"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.content_hash" == "fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83"
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B3 — Baseline. If exists=false here, the whole-file hash
|
||||||
|
# isn't registered in `storage.blobs` for CDC uploads on
|
||||||
|
# this build — swap to a chunk-hash probe or a
|
||||||
|
# blobs_consistency-driven assertion. See
|
||||||
|
# `docs/plan/recovery.md`.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B4 — Target folder + folder copy.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/folders
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"name": "hurl-ref-target-cdc"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
tgt_cdc_id: jsonpath "$.id"
|
||||||
|
|
||||||
|
POST {{base_url}}/api/batch/folders/copy
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"folder_ids": ["{{src_cdc_id}}"],
|
||||||
|
"target_folder_id": "{{tgt_cdc_id}}"
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Captures]
|
||||||
|
copy_cdc_root: jsonpath "$.successful[0].new_root_folder_id"
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.stats.successful" == 1
|
||||||
|
jsonpath "$.successful[0].files_copied" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B5 — CDC copy-side assertion: ref_count == 2.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B5b — Full-DB sweep on BOTH tenants after CDC copy. Multi-
|
||||||
|
# chunk path exercises the manifest side of the invariant
|
||||||
|
# — a bug that skips one chunk out of N would leak that
|
||||||
|
# chunk without touching the whole-file assertion above.
|
||||||
|
# `blobs_consistency` catches chunk-level drift;
|
||||||
|
# `manifests_consistency` catches whole-file drift.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B6 — Soft-delete copy: ref_count stays at 2.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/folders/{{copy_cdc_root}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B7 — Purge copy from trash directly by folder id: ref_count → 1.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/trash/{{copy_cdc_root}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == true
|
||||||
|
jsonpath "$.ref_count" == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B8 — Soft-delete + purge SOURCE: ref_count → 0, blob purged.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
DELETE {{base_url}}/api/folders/{{src_cdc_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
|
||||||
|
DELETE {{base_url}}/api/trash/{{src_cdc_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.exists" == false
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# B9 — End-of-Scenario-B sweep on BOTH tenants. All Scenario B
|
||||||
|
# rows gone; both counter drift counts must be back at
|
||||||
|
# baseline.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
|
jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================
|
||||||
|
# Cleanup — soft-delete + purge the two empty target folders so
|
||||||
|
# the run leaves nothing behind. Same direct-by-id pattern as
|
||||||
|
# above; no trash-listing filter needed.
|
||||||
|
# =============================================================
|
||||||
|
DELETE {{base_url}}/api/folders/{{tgt_small_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
DELETE {{base_url}}/api/trash/{{tgt_small_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
DELETE {{base_url}}/api/folders/{{tgt_cdc_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
DELETE {{base_url}}/api/trash/{{tgt_cdc_id}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
HTTP 200
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Final — force `dedup_gc` synchronously so orphaned manifests +
|
||||||
|
# blobs actually get reaped and their Rust blob-lifecycle hooks
|
||||||
|
# fire (which is what deletes disk thumbnails / face embeddings
|
||||||
|
# / audio tags keyed by the whole-file hash).
|
||||||
|
#
|
||||||
|
# The trigger `trg_files_decrement_blob_ref` deliberately only
|
||||||
|
# adjusts counters (see migration `20261017000000_file_delete_
|
||||||
|
# trigger_manifest_aware.sql`) — a SQL trigger can't invoke Rust
|
||||||
|
# callbacks. Physical cleanup + hook firing lives in `dedup_gc`
|
||||||
|
# Phase 1 (see `dedup_service.rs:2660-2772`), which picks up
|
||||||
|
# manifests at ref_count <= 0 and calls
|
||||||
|
# `fire_blob_hooks(file_hash)` per reap.
|
||||||
|
#
|
||||||
|
# Without this trigger the test would technically pass (the
|
||||||
|
# ref_count assertions all hold; the `exists == false` checks
|
||||||
|
# are user-scoped and don't need the DB row gone), but
|
||||||
|
# `storage_cleanup_check.sh` running after us would then find
|
||||||
|
# 6 orphan thumbnails on disk and fail the whole api-test run.
|
||||||
|
# Making the test self-contained keeps the diagnostic tight —
|
||||||
|
# if orphans remain after this trigger, the bug is in GC or
|
||||||
|
# hooks, not in our cleanup order.
|
||||||
|
#
|
||||||
|
# `?force=true` bypasses the orphan-grace window (safe: this
|
||||||
|
# test has no concurrent uploader that could race the reap).
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/admin/jobs/dedup_gc/trigger?force=true
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 200
|
||||||
|
[Asserts]
|
||||||
|
jsonpath "$.ok" == true
|
||||||
|
jsonpath "$.outcome.outcome" == "ok"
|
||||||
Executable
+141
@@ -0,0 +1,141 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================
|
||||||
|
# refcount_cascade.hurl — post-failure diagnostic
|
||||||
|
# =============================================================
|
||||||
|
# When `refcount_cascade.hurl` asserts a specific
|
||||||
|
# `ref_count` value and the API returns something else, this
|
||||||
|
# script inspects the two DB tables the API surface consults
|
||||||
|
# to distinguish which side is broken:
|
||||||
|
#
|
||||||
|
# 1. `storage.chunk_manifests.ref_count` — queried FIRST by
|
||||||
|
# `dedup_service::get_blob_metadata` (`dedup_service.rs`
|
||||||
|
# :1543-1552). If a manifest row exists for the hash,
|
||||||
|
# the API returns THIS ref_count.
|
||||||
|
# 2. `storage.blobs.ref_count` — legacy whole-file fallback,
|
||||||
|
# returned only when NO manifest row exists.
|
||||||
|
#
|
||||||
|
# Small files (< CDC min chunk size) still get a manifest row
|
||||||
|
# — one degenerate chunk with `chunk_hashes = [file_hash]` —
|
||||||
|
# so BOTH tables carry a ref_count for the same hash. When the
|
||||||
|
# copy or purge path only updates one of the two, the counters
|
||||||
|
# diverge.
|
||||||
|
#
|
||||||
|
# The `actual_auditor` column is the source of truth: the
|
||||||
|
# `blobs_consistency` auditor formula counting live references
|
||||||
|
# from `storage.files` + `chunk_manifests.chunk_hashes[]`
|
||||||
|
# (`blobs_consistency_service.rs:395-408`). Both stored values
|
||||||
|
# should equal this.
|
||||||
|
#
|
||||||
|
# Bug interpretation matrix (S=stored, M=manifest, A=auditor):
|
||||||
|
#
|
||||||
|
# S == M == A → consistent (test bug, unlikely)
|
||||||
|
# S < A and M == A → blob decrement over-fires
|
||||||
|
# S == A and M > A → manifest decrement missed
|
||||||
|
# S > A and M > A → decrement missed both sides
|
||||||
|
# S < A and M < A → double-decrement both sides
|
||||||
|
# S > A and M == A → increment missed on blob side
|
||||||
|
# S == A and M < A → increment missed on manifest
|
||||||
|
# (S=0, M=2, A=1) [seen 8/23]→ blob double-decrement +
|
||||||
|
# manifest never decremented
|
||||||
|
#
|
||||||
|
# The 2026-08-22 sandbox drift showed the raw shape
|
||||||
|
# (`stored=0, actual=1`) that this diagnostic now separates
|
||||||
|
# per-table.
|
||||||
|
#
|
||||||
|
# Called from `run.sh` immediately on hurl failure — see the
|
||||||
|
# dedicated `if ! hurl …; then bash …_diag.sh; fi` block.
|
||||||
|
# =============================================================
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||||
|
|
||||||
|
# Same connection string every other test-time script uses.
|
||||||
|
export PGPASSWORD=oxicloud_test
|
||||||
|
PSQL=(psql -h 127.0.0.1 -p 5433 -U oxicloud_test -d oxicloud_test
|
||||||
|
--set ON_ERROR_STOP=1 --pset pager=off)
|
||||||
|
|
||||||
|
SMALL_HASH='2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3'
|
||||||
|
CDC_HASH='fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83'
|
||||||
|
|
||||||
|
log() { echo "[ref_count-diag] $*"; }
|
||||||
|
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
|
log "refcount_cascade.hurl failed — running diagnostic."
|
||||||
|
log "Shows blob.ref_count AND manifest.ref_count side by side —"
|
||||||
|
log "the API queries manifest first (dedup_service.rs:1543), so"
|
||||||
|
log "if the two diverge, the API surface + auditor + on-disk"
|
||||||
|
log "state all report different numbers. See script header."
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
|
|
||||||
|
# Full picture for each fixture hash — one row per hash.
|
||||||
|
# LEFT JOINs so a hash present in only one table still surfaces
|
||||||
|
# (the other column comes back NULL, which is itself diagnostic).
|
||||||
|
"${PSQL[@]}" <<SQL
|
||||||
|
SELECT
|
||||||
|
h.hash,
|
||||||
|
b.ref_count AS blob_stored,
|
||||||
|
m.ref_count AS manifest_stored,
|
||||||
|
(
|
||||||
|
(SELECT COUNT(*) FROM storage.files f
|
||||||
|
WHERE f.blob_hash = h.hash
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM storage.chunk_manifests mm
|
||||||
|
WHERE mm.file_hash = f.blob_hash
|
||||||
|
))
|
||||||
|
+ (SELECT COUNT(*) FROM storage.chunk_manifests mm
|
||||||
|
WHERE h.hash = ANY(mm.chunk_hashes))
|
||||||
|
) AS actual_auditor,
|
||||||
|
(SELECT COUNT(*) FROM storage.files f
|
||||||
|
WHERE f.blob_hash = h.hash) AS all_files_rows,
|
||||||
|
(SELECT COUNT(*) FROM storage.files f
|
||||||
|
WHERE f.blob_hash = h.hash AND f.is_trashed = TRUE)
|
||||||
|
AS trashed_files,
|
||||||
|
b.size AS blob_size,
|
||||||
|
m.total_size AS manifest_size
|
||||||
|
FROM (VALUES ('$SMALL_HASH'), ('$CDC_HASH')) AS h(hash)
|
||||||
|
LEFT JOIN storage.blobs b ON b.hash = h.hash
|
||||||
|
LEFT JOIN storage.chunk_manifests m ON m.file_hash = h.hash;
|
||||||
|
SQL
|
||||||
|
psql_status=$?
|
||||||
|
|
||||||
|
if [[ $psql_status -ne 0 ]]; then
|
||||||
|
log "psql query failed (exit $psql_status) — DB may already be torn down"
|
||||||
|
exit 0 # don't mask the hurl failure with a psql error
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Show the offending file rows too — helps confirm whether the
|
||||||
|
# copy's file row exists and where (source folder vs copy).
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
|
log "File rows referencing the fixture hashes (name → folder id)"
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
|
"${PSQL[@]}" <<SQL
|
||||||
|
SELECT
|
||||||
|
f.blob_hash,
|
||||||
|
f.name,
|
||||||
|
f.folder_id,
|
||||||
|
f.is_trashed,
|
||||||
|
f.trashed_at,
|
||||||
|
f.created_at
|
||||||
|
FROM storage.files f
|
||||||
|
WHERE f.blob_hash IN ('$SMALL_HASH', '$CDC_HASH')
|
||||||
|
ORDER BY f.blob_hash, f.created_at;
|
||||||
|
SQL
|
||||||
|
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
|
log "Interpretation (compare blob_stored, manifest_stored, actual_auditor):"
|
||||||
|
log " all three equal → consistent (test bug, unlikely)"
|
||||||
|
log " blob=A, manifest>A → manifest decrement missed"
|
||||||
|
log " blob<A, manifest=A → blob decrement over-fires"
|
||||||
|
log " blob<A, manifest>A → double-decrement on blob +"
|
||||||
|
log " no-decrement on manifest"
|
||||||
|
log " (matches the 2026-08-23 case:"
|
||||||
|
log " blob=0, manifest=2, actual=1)"
|
||||||
|
log " blob>A, manifest=A → blob increment missed"
|
||||||
|
log " blob=A, manifest<A → manifest increment missed"
|
||||||
|
log " Either column NULL → row absent from that table"
|
||||||
|
log ""
|
||||||
|
log "The API surface (GET /api/dedup/check/{hash}) queries manifest"
|
||||||
|
log "FIRST — that's why hurl saw manifest_stored while auditor +"
|
||||||
|
log "on-disk state ran off blob_stored."
|
||||||
|
log "─────────────────────────────────────────────────────────"
|
||||||
@@ -224,6 +224,22 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
|
|||||||
|
|
||||||
#bash "$API_DIR/dedup_bulk_upload.sh"
|
#bash "$API_DIR/dedup_bulk_upload.sh"
|
||||||
|
|
||||||
|
# ── 4b. copy-folder ref_count regression — dedicated block ──────────────
|
||||||
|
# Ref_count mismatches surface as "expected 1 got 2" at hurl-assert
|
||||||
|
# level, which doesn't tell you WHICH half of the invariant broke
|
||||||
|
# (cascade delete missed rows vs decrement hook didn't fire). The
|
||||||
|
# `_diag.sh` script inspects `storage.blobs.ref_count` and the
|
||||||
|
# auditor's `actual_ref_count` formula on the two fixture hashes to
|
||||||
|
# pin the mode. Extracted from the main hurl array so `set -e`
|
||||||
|
# doesn't skip the diagnostic on failure — the `if !` guard runs
|
||||||
|
# the diag first, THEN exits with hurl's failure code so CI still
|
||||||
|
# reports the regression.
|
||||||
|
if ! hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test --jobs 1 \
|
||||||
|
"$API_DIR/refcount_cascade.hurl"; then
|
||||||
|
bash "$API_DIR/refcount_cascade_diag.sh" || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
bash "$API_DIR/storage_cleanup_check.sh"
|
bash "$API_DIR/storage_cleanup_check.sh"
|
||||||
|
|
||||||
# ── 5. OPAQUE crypto handshake — the parts Hurl can't drive ─────────────
|
# ── 5. OPAQUE crypto handshake — the parts Hurl can't drive ─────────────
|
||||||
|
|||||||
BIN
Binary file not shown.
+1
@@ -0,0 +1 @@
|
|||||||
|
copy-folder-ref-count regression fixture — do not reuse in other tests (hash must stay unique so ref_count assertions are absolute).
|
||||||
Reference in New Issue
Block a user