From a7b7045ed2268223af2c20e056e5d7e4f649a844 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sun, 23 Aug 2026 18:35:54 +0200 Subject: [PATCH 01/66] docs(plan): record the blobs_consistency -> chunks_consistency rename MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows from the blob/chunk taxonomy already in this section: the job iterates storage.blobs, which post-CDC holds chunks, so it inherits whatever that table ends up called. Two rules attached, because a job name is not an internal identifier — it appears in POST /api/admin/jobs//trigger, in background_runs.job_name, and in whatever dashboards operators built: * Travel with the schema rename, never ahead of it. A job called chunks_consistency iterating a table still called storage.blobs is more confusing than today's mismatch. * Never recycle `blobs_consistency`. Under the corrected taxonomy the manifest job IS the blob-level job, so the freed name looks available — and a name that survives a release while changing meaning silently breaks admin URLs and orphans run history. manifests_consistency is unambiguous either way, so exactly one job gets renamed rather than two swapping. Also records what is explicitly NOT renamed: the `.blob` on-disk suffix, where correcting it to `.chunk` would mean renaming every file in every deployment's blob store — a migration that can fail halfway, for clarity no consumer benefits from since nothing parses the suffix. And file.blob_hash, whose semantics are unchanged. Docs only. Co-Authored-By: Claude Opus 5 (1M context) --- docs/plan/derived-blobs.md | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 2cb6c2cc..149915b9 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1257,10 +1257,40 @@ Schema rename (deferred, requires migration): - `storage.chunk_manifests` → `storage.blob_manifests` (or keep — arguable) - `BlobStorageBackend` trait → `ChunkStorageBackend` — reads and writes physical chunks, not blobs +- **`blobs_consistency` job → `chunks_consistency`** — it iterates + `storage.blobs`, so it inherits whatever that table is called. -`file.blob_hash` semantics stay — references a Blob via its -manifest OR (for pre-CDC legacy) points directly at a single-chunk -Blob whose hash equals its lone chunk's hash. +### The job rename has two rules of its own + +**Travel with the schema, never ahead of it.** A job named +`chunks_consistency` iterating a table still called `storage.blobs` is +*more* confusing than today's mismatch, not less. + +**Never recycle `blobs_consistency`.** Under the corrected taxonomy the +manifest job *is* the blob-level job, so the freed name looks +available — and reusing it would be the worst outcome available. A job +name that survives a release while changing meaning silently breaks +`POST /api/admin/jobs//trigger` URLs, every historical row in +`background_runs.job_name`, and any dashboard or alert keyed on it. +`manifests_consistency` is unambiguous under either taxonomy; leave it +alone. Net effect: one job renamed, not two swapped. + +Budget for the operational cost either way — job names are not internal +identifiers. A rename orphans past runs unless `background_runs.job_name` +is migrated alongside, and any runbook naming the old one breaks. Worth +an alias period or an explicit release note. + +### Explicitly NOT renamed + +- **The `.blob` on-disk suffix** (`.blob` in `LocalBlobBackend` + and `CachedBlobBackend`). Correcting it to `.chunk` would mean + renaming every file in every deployment's blob store — a migration + whose cost is wildly out of proportion to the clarity gained, and one + that can fail halfway. The suffix is an implementation detail no + consumer parses; leave it. +- **`file.blob_hash`** — semantics stay. It references a Blob via its + manifest OR (for pre-CDC legacy) points directly at a single-chunk + Blob whose hash equals its lone chunk's hash. Scope for this rename: ~23 files touch the SQL, plus a migration for the table rename. Not free. Ship AFTER the tier-2 write-side From a7938344dd341ae4c2c67649434771a3cc0e1779 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sun, 23 Aug 2026 19:13:19 +0200 Subject: [PATCH 02/66] feat(storage): add content_derived_blobs table + reference source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping table for server-derived artifacts and registers it as a blob-reference source — deliberately BEFORE anything writes to it, which is the ordering the plan requires: dedup_gc's reap predicate has to know the table exists, or the first sweep after the first thumbnail deletes it. No writer yet, so this is inert: the table is empty and every added SQL term counts zero. The point is that the machinery is in place first. storage.content_derived_blobs maps (source_hash, kind, variant) to the derived blob_hash. The two hash columns mean different things and the migration says so at length: source_hash is a DEPENDENT pointer holding no reference (the file keeps the source alive), while blob_hash is a reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash would pin every source Blob for as long as a thumbnail existed. ContentDerivedReferenceSource contributes at the manifest level only. A derived artifact's blob_hash names a Blob, never a chunk, and contributing at the chunk level would double-count — a thumbnail is almost always single-chunk, so its manifest hash equals its lone chunk's hash, the same aliasing trap the legacy-files term guards against with NOT EXISTS. There is a test for the invariant, and the chunk-level golden test passing UNCHANGED is independent confirmation. Collapses three definitions of "what references a blob" into one. Adding the source revealed that DI assembled its own registry while DedupService::new built a different default, and the two consistency test helpers built a third — so the golden tests would have pinned SQL production never runs. There is now a single `built_in_registry(pool)`; DI reads it back via DedupService::reference_registry() rather than assembling its own. The reap-predicate golden test caught the change exactly as designed, and the new branch landed inside the NOT (...) group ORed with files — so a manifest is reaped only when NEITHER source references it. A branch landing outside that group would have inverted the predicate for every other source; that is why the test pins the whole statement rather than asserting substrings. fmt, clippy --all-features --all-targets, and 15 unit tests clean. fix(migrations): order content_derived_blobs after the refcount fixes Renames 20261015000000_content_derived_blobs.sql to 20261018000000_content_derived_blobs.sql. The file was authored before the rebase onto fix/copy_folder_ref_count_issue, so its version sorted BEFORE migrations that now precede it in history: 20261016000000_copy_folder_tree_manifest_refcount.sql 20261017000000_file_delete_trigger_manifest_aware.sql 20261017000002_repair_existing_refcount_drift.sql Filename order and commit order disagreeing is the problem, not any dependency — the table is standalone and creates nothing those migrations touch. But an installation that has already applied through …17000002 would then be offered a LOWER unapplied version, which sqlx either applies out of order or rejects on its version check, and a fresh install would get an ordering no upgrade path ever produces. Reproducibility between the two is the whole point of the version prefix. Kept as its own commit rather than amending 01d90524, since interactive rebase isn't available here and rewriting mid-branch while the ref_count work is still being rebased elsewhere would churn hashes again. Worth squashing into 01d90524 at merge. No content change — pure rename, verified nothing references the old filename. Co-Authored-By: Claude Opus 5 (1M context) --- .../20261018000000_content_derived_blobs.sql | 69 ++++++++++ src/common/di.rs | 19 +-- .../repositories/pg/blob_reference_sources.rs | 129 +++++++++++++++++- .../services/blobs_consistency_service.rs | 8 +- src/infrastructure/services/dedup_service.rs | 24 ++-- .../services/manifests_consistency_service.rs | 11 +- 6 files changed, 214 insertions(+), 46 deletions(-) create mode 100644 migrations/20261018000000_content_derived_blobs.sql diff --git a/migrations/20261018000000_content_derived_blobs.sql b/migrations/20261018000000_content_derived_blobs.sql new file mode 100644 index 00000000..2baee5a2 --- /dev/null +++ b/migrations/20261018000000_content_derived_blobs.sql @@ -0,0 +1,69 @@ +-- Derived content as blobs — tier-2 refactor, step 5. +-- See `docs/plan/derived-blobs.md`. +-- +-- Maps a source Blob to the artifacts derived FROM it: thumbnails today, +-- transcodes next. Both the mapping key and the value are BLAKE3 hashes, +-- but they mean different things: +-- +-- * `source_hash` — the Blob the artifact was derived from. A +-- *dependent* reference: it keeps nothing alive (the file does), and +-- when that Blob dies these rows are deleted with it. +-- * `blob_hash` — the derived Blob itself. A reference *holder*: it +-- bumps `chunk_manifests.ref_count`, which is why +-- `ContentDerivedReferenceSource` must be registered before the first +-- row is written, or `dedup_gc` reaps the content on its next sweep. +-- +-- KEYING — the rule this table exists to enforce: +-- +-- Bytes that are a pure deterministic function of the source content +-- belong here, content-keyed, and dedupe across every file holding +-- that content. Bytes that are user-supplied or user-chosen do NOT: +-- they must be file-keyed, because content-keying them lets one user's +-- upload be served for another user's identical file. Client-uploaded +-- previews (PDF page 1, video poster frames) are the live example and +-- belong in a separate file-keyed table. +-- +-- `variant` is opaque text. New axes go INSIDE it, never into new +-- columns: 'preview-avif' beside 'preview', '720p-av1' beside '720p'. +-- That is what keeps this table from growing a column per rendering +-- parameter. +-- +-- No FK on either hash column, for the reason +-- `20260701000000_content_search_index.sql` already documents: a hash +-- resolves to either `storage.blobs` (legacy whole blob) or +-- `storage.chunk_manifests` (CDC file hash), so the reference cannot be +-- expressed as a single FK. Orphans are reclaimed by GC and reported by +-- the consistency jobs instead. +-- +-- No `size` column: the bytes are content-addressed, so their length is +-- an immutable fact the blob layer already owns via `blob_hash`. +-- `content_type` IS stored — the thumbnail handler byte-sniffs every +-- response today, and this retires that. + +CREATE TABLE IF NOT EXISTS storage.content_derived_blobs ( + source_hash VARCHAR(64) NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('thumbnail', 'transcode')), + variant TEXT NOT NULL, + blob_hash VARCHAR(64) NOT NULL, + content_type TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (source_hash, kind, variant) +); + +-- Reverse lookup: "what still references this derived Blob?" — used by +-- the manifest-level refcount recompute in `manifests_consistency` and by +-- `dedup_gc`'s reap predicate. +CREATE INDEX IF NOT EXISTS idx_content_derived_blobs_blob_hash + ON storage.content_derived_blobs (blob_hash); + +COMMENT ON TABLE storage.content_derived_blobs IS + 'Server-derived artifacts (thumbnails, transcodes) keyed by the BLAKE3 of their SOURCE content. Content-keyed on purpose: identical content shares one derivation. User-supplied bytes must NOT be stored here — see docs/plan/derived-blobs.md.'; + +COMMENT ON COLUMN storage.content_derived_blobs.source_hash IS + 'The Blob this was derived from. Dependent reference — holds no ref_count; rows are deleted when the source Blob is reaped.'; + +COMMENT ON COLUMN storage.content_derived_blobs.blob_hash IS + 'The derived Blob. Reference HOLDER — bumps chunk_manifests.ref_count via DedupService::add_reference.'; + +COMMENT ON COLUMN storage.content_derived_blobs.variant IS + 'Opaque rendering discriminator (icon | preview | large | 720p...). New axes go inside this string, never into new columns.'; diff --git a/src/common/di.rs b/src/common/di.rs index 330134ef..4630da69 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -440,22 +440,6 @@ impl AppServiceFactory { // `blob_backend` into DedupService. let blob_backend_for_consistency = blob_backend.clone(); - // Every table holding blob references. Built ONCE and shared by the - // GC reap predicate and the consistency recompute so the two cannot - // disagree about what "referenced" means — a disagreement reaps live - // content. New blob-owning tables register here. - // See docs/plan/derived-blobs.md. - let blob_reference_registry = { - use crate::infrastructure::repositories::pg::blob_reference_sources::{ - ChunksReferenceSource, FilesReferenceSource, - }; - let mut registry = - crate::application::ports::blob_reference_ports::BlobReferenceRegistry::new(); - registry.register(Arc::new(FilesReferenceSource::new(db_pool.clone()))); - registry.register(Arc::new(ChunksReferenceSource::new(db_pool.clone()))); - Arc::new(registry) - }; - // Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index) let dedup_service = Arc::new( crate::infrastructure::services::dedup_service::DedupService::new( @@ -463,8 +447,7 @@ impl AppServiceFactory { db_pool.clone(), maintenance_pool.clone(), ) - .with_blob_lifecycle(blob_lifecycle) - .with_reference_registry(blob_reference_registry.clone()), + .with_blob_lifecycle(blob_lifecycle), ); dedup_service.initialize().await?; diff --git a/src/infrastructure/repositories/pg/blob_reference_sources.rs b/src/infrastructure/repositories/pg/blob_reference_sources.rs index 4424671b..ea3c1ac1 100644 --- a/src/infrastructure/repositories/pg/blob_reference_sources.rs +++ b/src/infrastructure/repositories/pg/blob_reference_sources.rs @@ -15,7 +15,9 @@ use async_trait::async_trait; use sqlx::{PgPool, Row}; use uuid::Uuid; -use crate::application::ports::blob_reference_ports::{BlobReferenceSource, RefLevel}; +use crate::application::ports::blob_reference_ports::{ + BlobReferenceRegistry, BlobReferenceSource, RefLevel, +}; use crate::domain::errors::DomainError; /// Aliases used inside the emitted fragments. @@ -26,6 +28,7 @@ use crate::domain::errors::DomainError; /// sweep and silently correlate against itself. const FILES_ALIAS: &str = "cnt_f"; const MANIFEST_ALIAS: &str = "cnt_m"; +const DERIVED_ALIAS: &str = "cnt_d"; /// Fragment for [`FilesReferenceSource`], as a free function so the SQL /// shape can be tested without constructing a pool — it is a property of @@ -87,6 +90,51 @@ fn chunks_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option { } } +/// Fragment for [`ContentDerivedReferenceSource`]. +/// +/// **Manifest level only.** A derived artifact's `blob_hash` names a Blob +/// (its own manifest), never a chunk. Contributing at the chunk level would +/// double-count, because a thumbnail is almost always single-chunk and its +/// manifest hash therefore equals its lone chunk's hash. +fn content_derived_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option { + match level { + RefLevel::Chunk => None, + RefLevel::Manifest => Some(format!( + "(SELECT COUNT(*) FROM storage.content_derived_blobs {DERIVED_ALIAS} \ + WHERE {DERIVED_ALIAS}.blob_hash = {outer_hash_expr})" + )), + } +} + +/// Short-circuiting existence form, used by `dedup_gc`'s reap predicate. +fn content_derived_exists_sql(level: RefLevel, outer_hash_expr: &str) -> Option { + match level { + RefLevel::Chunk => None, + RefLevel::Manifest => Some(format!( + "EXISTS (SELECT 1 FROM storage.content_derived_blobs {DERIVED_ALIAS} \ + WHERE {DERIVED_ALIAS}.blob_hash = {outer_hash_expr})" + )), + } +} + +/// Every built-in blob-reference source, in one place. +/// +/// THE definition of "what references a blob". `DedupService::new` uses it +/// as its construction default and hands it to the consistency jobs via +/// `reference_registry()`, so GC and the sweeps cannot disagree — and the +/// golden tests that pin the generated SQL exercise the same set production +/// runs, rather than a test-local approximation of it. +pub fn built_in_registry(pool: Arc) -> BlobReferenceRegistry { + let mut registry = BlobReferenceRegistry::new(); + registry.register(Arc::new(FilesReferenceSource::new(pool.clone()))); + registry.register(Arc::new(ChunksReferenceSource::new(pool.clone()))); + // Registered before anything writes a derived blob: dedup_gc's reap + // predicate must already know this table exists, or the first sweep + // after the first thumbnail deletes it. + registry.register(Arc::new(ContentDerivedReferenceSource::new(pool))); + registry +} + // ─── storage.files ─────────────────────────────────────────────────────── /// References held by `storage.files.blob_hash`. @@ -261,6 +309,85 @@ fn decode_uuid_cursor(bytes: &[u8]) -> Result { Ok(Uuid::from_bytes(raw)) } +// ─── storage.content_derived_blobs ─────────────────────────────────────── + +/// References held by `storage.content_derived_blobs.blob_hash` — the +/// DERIVED artifact, not the source it came from. +/// +/// **`source_hash` is deliberately not a reference.** It is a dependent +/// pointer: the source Blob is kept alive by the file that owns it, and when +/// that Blob is reaped these rows go with it. Counting `source_hash` here +/// would pin every source Blob for as long as a thumbnail existed. +pub struct ContentDerivedReferenceSource { + pool: Arc, +} + +impl ContentDerivedReferenceSource { + pub fn new(pool: Arc) -> Self { + Self { pool } + } +} + +#[async_trait] +impl BlobReferenceSource for ContentDerivedReferenceSource { + fn source_name(&self) -> &'static str { + "content_derived" + } + + fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option { + content_derived_ref_sql(level, outer_hash_expr) + } + + fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option { + content_derived_exists_sql(level, outer_hash_expr) + } + + async fn count_references(&self, blob_hash: &str) -> Result { + let n: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM storage.content_derived_blobs WHERE blob_hash = $1", + ) + .bind(blob_hash) + .fetch_one(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("BlobRefSource", format!("derived count: {e}")))?; + Ok(n.max(0) as u64) + } + + async fn list_referenced_blobs( + &self, + cursor: Option>, + limit: usize, + ) -> Result<(Vec, Option>), DomainError> { + // Paged by `blob_hash` itself — unlike files it IS the value we + // return, and DISTINCT keeps a Blob shared by several variants from + // appearing more than once per page. + let after: Option = match cursor { + Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| { + DomainError::internal_error("BlobRefSource", format!("bad derived cursor: {e}")) + })?), + None => None, + }; + + let rows: Vec<(String,)> = sqlx::query_as( + "SELECT DISTINCT blob_hash FROM storage.content_derived_blobs + WHERE ($1::text IS NULL OR blob_hash > $1) + ORDER BY blob_hash + LIMIT $2", + ) + .bind(after) + .bind(limit as i64) + .fetch_all(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("BlobRefSource", format!("derived page: {e}")))?; + + let next = rows + .last() + .map(|(h,)| h.clone().into_bytes()) + .filter(|_| rows.len() == limit); + Ok((rows.into_iter().map(|(h,)| h).collect(), next)) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/infrastructure/services/blobs_consistency_service.rs b/src/infrastructure/services/blobs_consistency_service.rs index a2eae950..bab26707 100644 --- a/src/infrastructure/services/blobs_consistency_service.rs +++ b/src/infrastructure/services/blobs_consistency_service.rs @@ -816,9 +816,6 @@ async fn recompute_hash( #[cfg(test)] mod tests { use super::*; - use crate::infrastructure::repositories::pg::blob_reference_sources::{ - ChunksReferenceSource, FilesReferenceSource, - }; fn default_registry() -> BlobReferenceRegistry { let pool = Arc::new( @@ -826,10 +823,7 @@ mod tests { .connect_lazy("postgres://invalid/invalid") .expect("lazy pool never connects"), ); - let mut registry = BlobReferenceRegistry::new(); - registry.register(Arc::new(FilesReferenceSource::new(pool.clone()))); - registry.register(Arc::new(ChunksReferenceSource::new(pool))); - registry + crate::infrastructure::repositories::pg::blob_reference_sources::built_in_registry(pool) } /// Golden test for the chunk-level recompute. Pins the statement diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index 243f037e..38cf180f 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -523,18 +523,17 @@ impl DedupService { } } - /// The two sources that were implicit before the registry existed. - /// Keeping this as the default means every construction path — including - /// tests — has a manifest-level source, so the reap predicate can never - /// degenerate to "nothing references anything". + /// Every built-in blob-reference source, in one place. + /// + /// This is THE definition of "what references a blob" — DI does not + /// assemble its own, it reads this one back via + /// [`Self::reference_registry`] and hands it to the consistency jobs, so + /// GC and the sweeps cannot disagree. Keeping it as the construction + /// default also means every path — including tests — has a + /// manifest-level source, so the reap predicate can never degenerate to + /// "nothing references anything". fn default_reference_registry(pool: Arc) -> BlobReferenceRegistry { - use crate::infrastructure::repositories::pg::blob_reference_sources::{ - ChunksReferenceSource, FilesReferenceSource, - }; - let mut registry = BlobReferenceRegistry::new(); - registry.register(Arc::new(FilesReferenceSource::new(pool.clone()))); - registry.register(Arc::new(ChunksReferenceSource::new(pool))); - registry + crate::infrastructure::repositories::pg::blob_reference_sources::built_in_registry(pool) } /// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one @@ -3388,7 +3387,8 @@ mod tests { SELECT ctid FROM storage.chunk_manifests m WHERE m.ref_count <= 0 - OR NOT (EXISTS (SELECT 1 FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash)) + OR NOT (EXISTS (SELECT 1 FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash) + OR EXISTS (SELECT 1 FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash)) LIMIT $1 ) RETURNING file_hash, chunk_hashes, total_size"#; diff --git a/src/infrastructure/services/manifests_consistency_service.rs b/src/infrastructure/services/manifests_consistency_service.rs index 3ee5b016..3c2d1248 100644 --- a/src/infrastructure/services/manifests_consistency_service.rs +++ b/src/infrastructure/services/manifests_consistency_service.rs @@ -404,9 +404,6 @@ impl RecoverableJobHandler for ManifestsConsistencyCheck { #[cfg(test)] mod tests { use super::*; - use crate::infrastructure::repositories::pg::blob_reference_sources::{ - ChunksReferenceSource, FilesReferenceSource, - }; fn default_registry() -> BlobReferenceRegistry { let pool = Arc::new( @@ -414,10 +411,7 @@ mod tests { .connect_lazy("postgres://invalid/invalid") .expect("lazy pool never connects"), ); - let mut registry = BlobReferenceRegistry::new(); - registry.register(Arc::new(FilesReferenceSource::new(pool.clone()))); - registry.register(Arc::new(ChunksReferenceSource::new(pool))); - registry + crate::infrastructure::repositories::pg::blob_reference_sources::built_in_registry(pool) } /// Golden test — the statement is assembled from the registry, so pin it @@ -437,7 +431,8 @@ mod tests { m.total_size AS total_size, m.chunk_count AS chunk_count, ((SELECT COUNT(*) FROM storage.files cnt_f - WHERE cnt_f.blob_hash = m.file_hash))::bigint AS actual_ref_count + WHERE cnt_f.blob_hash = m.file_hash) + + (SELECT COUNT(*) FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash))::bigint AS actual_ref_count FROM storage.chunk_manifests m WHERE ($1::text IS NULL OR m.file_hash > $1) ORDER BY m.file_hash From 1c488b7df5b60360840abea9ec2b1eb93242d19c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sun, 23 Aug 2026 21:22:26 +0200 Subject: [PATCH 03/66] feat(thumbnails): also store derived thumbnails as blobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO stored through DedupService and recorded in storage.content_derived_blobs. The sidecar write stays and reads are untouched, so nothing user-visible changes. That split is deliberate. This is the first commit in the plan that changes runtime behaviour on a hot path, so it fills the table while reads still come from disk: the rows can be inspected against real data before anything depends on them, and a rollback at any point leaves working thumbnails. The read path and sidecar removal follow separately. DedupService::store_derived_blob does the whole contract in one place, so no caller has to remember the accounting: * writes the bytes through the normal CDC path, so derived blobs inherit the backend, encryption, migration and key rotation that source content already gets; * records (source_hash, kind, variant) -> blob_hash; * releases the reference store_from_stream took IF the mapping already existed. Two instances racing to render the same thumbnail must leave ref_count at 1, not 2 — otherwise every re-render inflates it and pins the blob forever. ThumbnailService deliberately does NOT gain a DedupService field: it implements BlobLifecycleHook, and holding one would close the cycle DedupService -> BlobLifecycleService -> hook -> DedupService that the existing comment warns about. The handle is passed per call instead, which every eager path already has. The tier-3 write is best-effort and logged. A failure must not cost the user a thumbnail that is already on disk and in the moka cache; `derived_import` sweeps anything missed. The sidecar write keeps its existing failure behaviour and now `continue`s, so a disk failure no longer falls through to the cache insert. Nothing reads these rows yet, so the only observable effect is rows appearing in the table and the manifest ref_count they hold — which `manifests_consistency` will now count, since ContentDerivedReferenceSource was registered in 8d4052e1 before any writer existed. fmt, clippy --all-features --all-targets, and 35 unit tests across the touched modules clean. Co-Authored-By: Claude Opus 5 (1M context) --- src/infrastructure/services/dedup_service.rs | 68 +++++++++++++++++++ .../services/thumbnail_service.rs | 60 +++++++++++++--- 2 files changed, 117 insertions(+), 11 deletions(-) diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index 38cf180f..ae013741 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -557,6 +557,74 @@ impl DedupService { self } + /// Store a server-derived artifact and record the mapping from the + /// content it was derived from. + /// + /// One call does the whole contract, so no caller has to remember the + /// accounting: + /// + /// 1. writes the bytes through the normal CDC path — derived blobs get + /// the same backend, encryption, migration and rotation as any other + /// content, and `store_from_stream` takes exactly one reference; + /// 2. records `(source_hash, kind, variant) -> blob_hash`; + /// 3. **releases that reference if the mapping already existed**, because + /// the row that would justify it is not ours — two instances racing + /// to render the same thumbnail must leave `ref_count` at 1, not 2. + /// + /// `bytes` is expected to be small (a thumbnail is 3-90 KB, below + /// `CDC_MIN_CHUNK`, so this is a single chunk). See + /// `docs/plan/derived-blobs.md`. + /// + /// Returns the derived blob hash. + pub async fn store_derived_blob( + &self, + source_hash: &str, + kind: &str, + variant: &str, + content_type: &str, + bytes: Bytes, + ) -> Result { + let stored = self + .store_from_stream( + stream::once(async move { Ok::(bytes) }), + Some(content_type.to_string()), + ) + .await?; + let derived_hash = stored.hash().to_string(); + + let inserted = sqlx::query( + "INSERT INTO storage.content_derived_blobs + (source_hash, kind, variant, blob_hash, content_type) + VALUES ($1, $2, $3, $4, $5) + ON CONFLICT (source_hash, kind, variant) DO NOTHING", + ) + .bind(source_hash) + .bind(kind) + .bind(variant) + .bind(&derived_hash) + .bind(content_type) + .execute(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("Dedup", format!("record derived blob: {e}")))? + .rows_affected(); + + if inserted == 0 { + // Someone else already mapped this variant. Our reference has no + // row behind it; leaving it would inflate ref_count on every + // re-render and pin the blob forever. + if let Err(e) = self.remove_reference(&derived_hash).await { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to release duplicate derived-blob reference for {}", + &derived_hash[..derived_hash.len().min(12)], + ); + } + } + + Ok(derived_hash) + } + /// The registry backing the reap predicate. /// /// Exposed so `blobs_consistency` recomputes refcounts from the *same* diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 09b36a32..1a6c0ced 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -1115,7 +1115,7 @@ impl ThumbnailService { } }; - self.render_and_persist_all_webp(&file_id, &blob_hash, original_data) + self.render_and_persist_all_webp(&file_id, &blob_hash, original_data, Some(&dedup)) .await; tracing::info!("✅ Background thumbnail generation complete: {}", file_id); @@ -1126,7 +1126,21 @@ impl ThumbnailService { /// blob_hash (disk `{hash}.webp` + moka). Shared by the image upload path and /// the video path (which passes the extracted frame as the source), so both /// produce identical, dedup-able, content-negotiable thumbnails. - async fn render_and_persist_all_webp(&self, file_id: &str, blob_hash: &str, source: Bytes) { + /// `dedup` is `Some` on every path that has a handle, which is every + /// eager background path. When present each rendered size is ALSO stored + /// as a derived blob and recorded in `storage.content_derived_blobs`. + /// + /// The sidecar write is deliberately kept: this slice fills the table + /// while reads still come from disk, so a rollback at any point leaves + /// working thumbnails and the table can be inspected against real data + /// before anything depends on it. See `docs/plan/derived-blobs.md`. + async fn render_and_persist_all_webp( + &self, + file_id: &str, + blob_hash: &str, + source: Bytes, + dedup: Option<&DedupService>, + ) { let results = tokio::task::spawn_blocking(move || { Self::render_all_thumbnails_from_data(source.as_ref(), ThumbnailFormat::Webp) }) @@ -1151,15 +1165,39 @@ impl ThumbnailService { } if let Err(e) = fs::write(&thumb_path, &bytes).await { tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e); - } else { - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format: ThumbnailFormat::Webp, - }; - self.cache.insert(cache_key, bytes).await; - tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size); + continue; } + + // Tier-3 copy. Best-effort and logged: a failure here must not + // cost the user their thumbnail, which is already on disk and in + // the cache. `derived_import` sweeps anything missed. + if let Some(dedup) = dedup + && let Err(e) = dedup + .store_derived_blob( + blob_hash, + "thumbnail", + size.dir_name(), + "image/webp", + bytes.clone(), + ) + .await + { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to record derived blob for {} {:?}", + file_id, + size, + ); + } + + let cache_key = ThumbnailCacheKey { + file_id: file_id.to_string(), + size, + format: ThumbnailFormat::Webp, + }; + self.cache.insert(cache_key, bytes).await; + tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size); } } @@ -1251,7 +1289,7 @@ impl ThumbnailService { Ok(p) => p, Err(_) => return, }; - self.render_and_persist_all_webp(&file_id, &blob_hash, frame) + self.render_and_persist_all_webp(&file_id, &blob_hash, frame, Some(&dedup)) .await; tracing::info!("✅ Video thumbnail generation complete: {}", file_id); }); From 60b94e1183eca985117cc9209363ed4cd0347d11 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sun, 23 Aug 2026 23:10:00 +0200 Subject: [PATCH 04/66] feat(thumbnails): serve derived blobs when the sidecar cannot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 5, read path — Option 2 of the two shapes discussed: the derived blob is consulted LAST, after the sidecar, not first. Read order is now moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob For every thumbnail already on disk the new branch is never reached, so the database stays off the hot path and a fault in it cannot break a working gallery. It answers only what disk cannot: a thumbnail rendered by another instance, or a box whose sidecar was never populated. Legacy content keeps serving from disk until `derived_import` migrates it. That inverts the plan's stated order deliberately. Derived-blob-first is right for the END state, because it is what lets the sidecar be deleted; sidecar-first is right transitionally, because the risky reordering should happen after the table has been seen serving real reads. The flip belongs in the release that removes the sidecar, and the comment at the branch says so. The existing precedence is preserved and now documented: the file-keyed client upload (ext-) is checked BEFORE the content-keyed server render. That ordering is a security property, not a preference — content-keyed artifacts are shared across every file with that content, so checking the file-keyed one first is what keeps one user's uploaded preview from ever being served for another user's identical file. Shape notes: * `find_derived_blob` lands on DedupPort/DedupService as the read counterpart of `store_derived_blob`, so ThumbnailService needs no pool field — and therefore ThumbnailService::new, DI and three tests are untouched. * It carries `content_type`, which is what will retire the byte-sniffing in the handlers once reads are table-primary. * The parameter is `Option<&DedupService>`, concrete rather than `&dyn DedupPort`: DedupPort uses native `async fn` and so is not dyn-compatible, and ThumbnailPort is never used as a trait object (checked) — both handlers hold the concrete Arc. `None` means sidecar-only, which is exactly today's behaviour and what the abstract port impl passes. fmt, clippy --all-features --all-targets, 35 unit tests clean. Co-Authored-By: Claude Opus 5 (1M context) --- src/application/ports/dedup_ports.rs | 21 ++++++++ src/common/stubs.rs | 9 ++++ src/infrastructure/services/dedup_service.rs | 36 ++++++++++++++ .../services/thumbnail_service.rs | 48 +++++++++++++++++-- src/interfaces/api/handlers/file_handler.rs | 17 ++++++- src/interfaces/nextcloud/preview_handler.rs | 1 + 6 files changed, 126 insertions(+), 6 deletions(-) diff --git a/src/application/ports/dedup_ports.rs b/src/application/ports/dedup_ports.rs index 36deed77..046ca45c 100644 --- a/src/application/ports/dedup_ports.rs +++ b/src/application/ports/dedup_ports.rs @@ -24,6 +24,16 @@ pub struct BlobMetadataDto { pub content_type: Option, } +/// A stored server-derived artifact: which blob holds it, and what it is. +/// +/// `content_type` is carried so the read path can set the response header +/// without byte-sniffing the payload, which is what it does today. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DerivedBlobRef { + pub blob_hash: String, + pub content_type: String, +} + /// Result of a deduplication store operation. #[derive(Debug, Clone)] pub enum DedupResultDto { @@ -83,6 +93,17 @@ pub trait DedupPort: Send + Sync + 'static { /// Check if a blob with the given hash exists. async fn blob_exists(&self, hash: &str) -> bool; + /// Look up a server-derived artifact by the content it was derived from. + /// + /// The read counterpart of `store_derived_blob`. Returns `None` when no + /// such variant has been derived yet — the caller then renders it. + async fn find_derived_blob( + &self, + source_hash: &str, + kind: &str, + variant: &str, + ) -> Option; + /// Get metadata for a blob. async fn get_blob_metadata(&self, hash: &str) -> Option; diff --git a/src/common/stubs.rs b/src/common/stubs.rs index 952874f7..63a0ac32 100644 --- a/src/common/stubs.rs +++ b/src/common/stubs.rs @@ -756,6 +756,15 @@ impl DedupPort for StubDedupPort { false } + async fn find_derived_blob( + &self, + _source_hash: &str, + _kind: &str, + _variant: &str, + ) -> Option { + None + } + async fn get_blob_metadata(&self, _hash: &str) -> Option { None } diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index ae013741..ff9e1191 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -625,6 +625,33 @@ impl DedupService { Ok(derived_hash) } + /// Look up a derived artifact by its source content. Read counterpart of + /// [`Self::store_derived_blob`]. + pub async fn find_derived_blob( + &self, + source_hash: &str, + kind: &str, + variant: &str, + ) -> Option { + sqlx::query_as::<_, (String, String)>( + "SELECT blob_hash, content_type FROM storage.content_derived_blobs + WHERE source_hash = $1 AND kind = $2 AND variant = $3", + ) + .bind(source_hash) + .bind(kind) + .bind(variant) + .fetch_optional(self.pool.as_ref()) + .await + .ok() + .flatten() + .map(|(blob_hash, content_type)| { + crate::application::ports::dedup_ports::DerivedBlobRef { + blob_hash, + content_type, + } + }) + } + /// The registry backing the reap predicate. /// /// Exposed so `blobs_consistency` recomputes refcounts from the *same* @@ -3295,6 +3322,15 @@ impl DedupPort for DedupService { self.blob_exists(hash).await } + async fn find_derived_blob( + &self, + source_hash: &str, + kind: &str, + variant: &str, + ) -> Option { + self.find_derived_blob(source_hash, kind, variant).await + } + async fn get_blob_metadata(&self, hash: &str) -> Option { self.get_blob_metadata(hash).await } diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 1a6c0ced..52fee4af 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -478,6 +478,11 @@ impl ThumbnailService { blob_hash: Option<&str>, size: ThumbnailSize, format: ThumbnailFormat, + // Concrete, and optional: `ThumbnailPort` is never used as a trait + // object (checked), and `DedupPort` uses native `async fn` so it is + // not dyn-compatible anyway. `None` means sidecar-only — exactly + // today's behaviour, which is what the port impl wants. + dedup: Option<&DedupService>, ) -> Option { // 1. Check in-memory cache let cache_key = ThumbnailCacheKey { @@ -520,10 +525,42 @@ impl ThumbnailService { let bytes = Bytes::from(data); // Populate in-memory cache for next hit self.cache.insert(cache_key, bytes.clone()).await; - Some(bytes) - } else { - None + return Some(bytes); } + + // 4. Tier-3 derived blob. Deliberately LAST while the sidecar still + // exists: for every thumbnail already on disk this branch is never + // reached, so the DB stays off the hot path and a fault here cannot + // break a working gallery. It answers only what disk cannot — another + // instance's render, or a box whose sidecar was never populated. + // + // The order flips (derived blob first, sidecar as fallback) in the + // release that removes the sidecar; see docs/plan/derived-blobs.md. + let dedup = dedup?; + let derived = dedup + .find_derived_blob(hash, "thumbnail", size.dir_name()) + .await?; + use futures::StreamExt; + let mut stream = dedup.read_blob_stream(&derived.blob_hash).await.ok()?; + let mut buf = Vec::new(); + while let Some(chunk) = stream.next().await { + match chunk { + Ok(part) => buf.extend_from_slice(&part), + Err(e) => { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "derived thumbnail read failed for {} {:?}", + file_id, + size, + ); + return None; + } + } + } + let bytes = Bytes::from(buf); + self.cache.insert(cache_key, bytes.clone()).await; + Some(bytes) } /// Store an externally-generated thumbnail (e.g. client-side video frame). @@ -1605,7 +1642,10 @@ impl ThumbnailPort for ThumbnailService { blob_hash: Option<&str>, size: PortThumbnailSize, ) -> Option { - self.get_cached_thumbnail(file_id, blob_hash, size.into(), ThumbnailFormat::Webp) + // `None` — the abstract port has no DedupService handle, so it stays + // sidecar-only. Callers wanting the tier-3 fallback use the concrete + // method, which both handlers already do. + self.get_cached_thumbnail(file_id, blob_hash, size.into(), ThumbnailFormat::Webp, None) .await } diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index f3cc381e..58b576f0 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -484,7 +484,13 @@ impl FileHandler { // Try moka (RAM) → disk before touching the database. // If the thumbnail exists it was authorized at creation time. if let Some(data) = thumbnail_service - .get_cached_thumbnail(&id, None, thumb_size.into(), format) + .get_cached_thumbnail( + &id, + None, + thumb_size.into(), + format, + Some(&state.core.dedup_service), + ) .await { return Response::builder() @@ -541,7 +547,13 @@ impl FileHandler { } }; if let Some(data) = thumbnail_service - .get_cached_thumbnail(&id, Some(&blob_hash), thumb_size.into(), format) + .get_cached_thumbnail( + &id, + Some(&blob_hash), + thumb_size.into(), + format, + Some(&state.core.dedup_service), + ) .await { return Response::builder() @@ -572,6 +584,7 @@ impl FileHandler { Some(&blob_hash), thumb_size.into(), ThumbnailFormat::Webp, + Some(&state.core.dedup_service), ) .await { diff --git a/src/interfaces/nextcloud/preview_handler.rs b/src/interfaces/nextcloud/preview_handler.rs index c137112b..8a32a0d6 100644 --- a/src/interfaces/nextcloud/preview_handler.rs +++ b/src/interfaces/nextcloud/preview_handler.rs @@ -204,6 +204,7 @@ pub async fn handle_preview( Some(&blob_hash), thumb_size.into(), ThumbnailFormat::Jpeg, + Some(&state.core.dedup_service), ) .await { From c276d3861ef0e4ae8fe25db0ac09538e7a00c6e2 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 00:53:46 +0200 Subject: [PATCH 05/66] fix(thumbnails): release derived blobs when their source is reaped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught by the api-test storage check: 15 blob files left on disk after a full cleanup. Since 3736b577 thumbnails are stored as derived blobs, and each content_derived_blobs row holds a manifest reference — but nothing ever deleted those rows, so the reference outlived the source and GC could never reclaim the bytes. The plan specifies this cascade; I implemented the write and read paths and missed it. Adds `purge_derived_blobs`, the delete counterpart of `store_derived_blob`: deletes every row derived from a source hash and releases the reference each held. It lives on DedupService alongside its store/find siblings because ThumbnailService cannot hold a DedupService — it implements BlobLifecycleHook, and holding one would close the DedupService -> BlobLifecycleService -> hook -> DedupService cycle the existing comment warns about. All five reap sites now go through `reap_blob`, which purges then fires the lifecycle hooks, so no path can drop a blob without first releasing what was derived from it. Previously each site called fire_blob_hooks directly, which only cleaned the sidecar files ThumbnailService owns. `reap_blob` is boxed because it is mutually recursive with `remove_reference`: releasing a thumbnail's reference can reap the thumbnail's own blob, which re-enters here. It terminates after one level — nothing is derived from a thumbnail, so the inner purge finds no rows. That bound is a property of the data, not an invariant the code enforces, so it is stated at the definition. fmt, clippy --all-features --all-targets, unit tests clean. The api-test storage check is the real verdict — it is what found this. Co-Authored-By: Claude Opus 5 (1M context) --- src/infrastructure/services/dedup_service.rs | 68 ++++++++++++++++++-- 1 file changed, 63 insertions(+), 5 deletions(-) diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index ff9e1191..41bda424 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -674,6 +674,64 @@ impl DedupService { } } + /// Everything that must happen when a blob is permanently reaped: + /// drop the artifacts derived FROM it, then notify the lifecycle hooks. + /// + /// Boxed because it is mutually recursive with `remove_reference`: + /// releasing a thumbnail's reference can reap the thumbnail's own blob, + /// which comes back through here. It terminates after one level — + /// nothing is derived from a thumbnail, so the inner purge finds no rows. + fn reap_blob<'a>( + &'a self, + hash: &'a str, + ) -> Pin + Send + 'a>> { + Box::pin(async move { + self.purge_derived_blobs(hash).await; + self.fire_blob_hooks(hash); + }) + } + + /// Delete every artifact derived from `source_hash` and release the + /// manifest references those rows held. + /// + /// The delete counterpart of [`Self::store_derived_blob`]. Without it a + /// thumbnail pins its own blob forever: the mapping row keeps + /// `chunk_manifests.ref_count` at 1 with no file behind it, so GC never + /// reclaims the bytes and a full delete leaves orphans on disk. + async fn purge_derived_blobs(&self, source_hash: &str) { + let derived: Vec<(String,)> = match sqlx::query_as( + "DELETE FROM storage.content_derived_blobs + WHERE source_hash = $1 + RETURNING blob_hash", + ) + .bind(source_hash) + .fetch_all(self.pool.as_ref()) + .await + { + Ok(rows) => rows, + Err(e) => { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to purge derived blobs for {}", + &source_hash[..source_hash.len().min(12)], + ); + return; + } + }; + + for (blob_hash,) in derived { + if let Err(e) = self.remove_reference(&blob_hash).await { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to release derived blob {}", + &blob_hash[..blob_hash.len().min(12)], + ); + } + } + } + fn fire_blob_hooks(&self, hash: &str) { if let Some(lc) = &self.blob_lifecycle { lc.on_blob_deleted(hash); @@ -1937,7 +1995,7 @@ impl DedupService { self.manifest_cache.invalidate(file_hash).await; // File content is gone — drop its blob-keyed thumbnails now. - self.fire_blob_hooks(file_hash); + self.reap_blob(file_hash).await; tracing::info!( "MANIFEST DELETED: {} ({} chunks dereferenced; orphans reclaimed by GC)", @@ -2016,7 +2074,7 @@ impl DedupService { } // Bug 3 fix: notify hooks — e.g. thumbnail cleanup keyed by hash - self.fire_blob_hooks(hash); + self.reap_blob(hash).await; tracing::info!("BLOB DELETED: {} (no more references)", &hash[..12]); Ok(true) @@ -2105,7 +2163,7 @@ impl DedupService { if let Err(e) = self.backend.delete_blob(hash).await { tracing::warn!("cleanup_if_orphaned: disk delete failed for {short}: {e}"); } - self.fire_blob_hooks(hash); + self.reap_blob(hash).await; tracing::info!("cleanup_if_orphaned: removed orphaned legacy blob {short}"); } } @@ -2854,7 +2912,7 @@ impl DedupService { // chunk-keyed hook never finds them. Symptom: orphan webp // under `.thumbnails/{icon,preview,large}/.webp` // after a user-cascade-delete of a video upload. - self.fire_blob_hooks(file_hash); + self.reap_blob(file_hash).await; total_bytes += *size as u64; tracing::debug!( @@ -2937,7 +2995,7 @@ impl DedupService { .await; for (hash, size) in &deleted { - self.fire_blob_hooks(hash); + self.reap_blob(hash).await; total_bytes += *size as u64; } total_deleted += n as u64; From a8223cab65fc8d87b0c2eea826dd7c76a250b71f Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 01:05:48 +0200 Subject: [PATCH 06/66] test(storage-check): drain the GC cascade instead of one pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One dedup_gc pass cannot fully drain now that thumbnails are derived blobs. Reaping a source releases the references its derived artifacts hold (each content_derived_blobs row pins a manifest), and those releases happen mid-sweep — the derived chunks are stamped orphaned as the pass is already walking past them, because remove_manifest_reference deliberately does not unlink, to avoid racing a concurrent upload re-referencing the same chunk. They are collectible only on the NEXT sweep, which is why the check saw 15 leftover blobs. Loops until a pass reclaims nothing rather than hardcoding two. Two is correct only while the derivation graph is one level deep — a thumbnail is derived from a file, nothing is derived from a thumbnail. That is a property of the data, not an invariant the code enforces, so a fixed count would silently under-drain the day transcodes-of-thumbnails or E2E-wrapped derivatives exist, and the failure would surface as a confusing leftover-file assertion rather than the design change it is. Bounded at 3 with a warning if it does not settle. Sleeps between passes. The JobRegistry serialises runs of the same job, so a back-to-back trigger risks rejection as already-running — which returns 0 reaped and would exit the loop early, declaring success with blobs still on disk. A false pass is worse than a slow one. It also gives the previous pass's detached unlink tasks (spawned by on_blob_deleted, awaited by nothing) time to land. Deliberately NOT fixed in production code: derived chunks land inside the 1-hour orphan grace, so a second immediate sweep would collect nothing there and the next scheduled run picks them up. A fixpoint loop in garbage_collect would be dead code outside force=true, which is only this test. Co-Authored-By: Claude Opus 5 (1M context) --- tests/api/storage_cleanup_check.sh | 58 +++++++++++++++++++++++++++--- 1 file changed, 53 insertions(+), 5 deletions(-) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index ec416797..a8c6745e 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -287,11 +287,59 @@ curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/usage_reconcile/trigger" > || fail "usage_reconcile trigger failed" log "Reconciliation sweep triggered." -GC_RESULT=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/dedup_gc/trigger?force=true") -[[ -z "$GC_RESULT" ]] && fail "trigger-gc returned an empty body" -GC_BLOBS=$(echo "$GC_RESULT" | jq -r '.outcome.count') -GC_BYTES=$(echo "$GC_RESULT" | jq -r '.outcome.extra.bytes_reclaimed') -log "GC reaped $GC_BLOBS blob(s), $GC_BYTES byte(s) freed." +# One GC pass is NOT enough, and this is by design rather than a bug. +# Reaping a source blob releases the references its DERIVED artifacts hold +# (thumbnails live in storage.content_derived_blobs and each row pins a +# manifest). Those releases happen mid-sweep, so the derived chunks are +# only stamped orphaned as the pass is already walking past them — +# `remove_manifest_reference` deliberately does not unlink, to avoid racing +# a concurrent upload re-referencing the same chunk. They become +# collectible on the NEXT sweep. +# +# Loop until a pass reclaims nothing rather than hardcoding two passes. +# Two is correct only while the derivation graph is one level deep — a +# thumbnail is derived from a file and nothing is derived from a thumbnail. +# That is a property of the data, not an invariant the code enforces, so a +# fixed count would silently under-drain the day transcodes-of-thumbnails +# or E2E-wrapped derivatives appear, and the failure would surface as a +# confusing leftover-file assertion rather than as the design change it is. +# +# Production does NOT need this loop: derived chunks land inside the 1-hour +# orphan grace, so a second immediate pass would collect nothing and the +# next scheduled sweep picks them up. It is only `force=true` (grace 0) +# that can drain a cascade in one go, which is exactly this test. +GC_TOTAL_BLOBS=0 +GC_TOTAL_BYTES=0 +GC_DRAINED=0 +for gc_pass in 1 2 3; do + GC_RESULT=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/dedup_gc/trigger?force=true") + [[ -z "$GC_RESULT" ]] && fail "trigger-gc returned an empty body (pass $gc_pass)" + GC_BLOBS=$(echo "$GC_RESULT" | jq -r '.outcome.count // 0') + GC_BYTES=$(echo "$GC_RESULT" | jq -r '.outcome.extra.bytes_reclaimed // 0') + GC_TOTAL_BLOBS=$((GC_TOTAL_BLOBS + GC_BLOBS)) + GC_TOTAL_BYTES=$((GC_TOTAL_BYTES + GC_BYTES)) + log "GC pass $gc_pass reaped $GC_BLOBS blob(s), $GC_BYTES byte(s) freed." + if [[ "$GC_BLOBS" -eq 0 ]]; then + GC_DRAINED=1 + break + fi + # Breathe before the next trigger, for two reasons: + # + # * The JobRegistry serialises runs of the same job. Firing the next + # trigger before the previous run has fully unwound risks it being + # rejected as already-running — which would come back as 0 reaped + # and exit this loop early, declaring success with blobs still on + # disk. A false pass is worse than a slow one. + # * `on_blob_deleted` spawns detached unlink tasks that nothing + # awaits, so some of the previous pass's disk work may still be in + # flight. + sleep 1 +done +if [[ "$GC_DRAINED" -ne 1 ]]; then + log "WARNING: GC still reaping after 3 passes — the derivation graph may" + log " be deeper than one level; raise the bound and check why." +fi +log "GC total: $GC_TOTAL_BLOBS blob(s), $GC_TOTAL_BYTES byte(s) freed." # ── 4. Disk verification ────────────────────────────────────────────────────── From 7f5ee7401f9eca9e189b797e6f5cb66b7cb2733d Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 21:25:08 +0200 Subject: [PATCH 07/66] refactor(storage): make blob enumeration ordered and hash-cursored MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precondition for the merge-join in backend_consistency (step 6 / option A of docs/plan/derived-blobs.md), landed separately because it is independently useful and carries the risk. Two contract changes on BlobStorageBackend::list_blob_hashes: 1. Entries MUST be in ascending hash order. Every shipped backend already did this — local sorts within each shard and walks 00..ff, and since the shard IS the hash prefix that is globally sorted; S3 and Azure list lexicographically by key and blobs// sorts identically to . It was accidental, and a future backend enumerating in any other order would have silently made the merge-join emit bogus blob_missing_from_backend findings at data_loss severity. 2. The cursor is the last hash returned, not an opaque backend token. This is what lets a caller resume from a checkpoint it already holds — the merge-join keeps one cursor for both the DB walk and the backend walk instead of a compound one, which in turn means blobs_consistency's existing cursor format survives and no paused run is stranded. Local already derived its position from a hash; it now emits the bare hash instead of "/", and still accepts both legacy forms so a run paused across this deploy resumes. The bare-shard form works through the same path unchanged, since "3f" sorts before every 64-char hash beginning "3f". S3 moves from continuation_token to StartAfter, which supports this natively. One non-obvious case handled: a page can contain only non-canonical keys (.tmp spool files, .corrupt sidecars), which are filtered into `unknowns`, leaving `blobs` empty — a naive blobs.last() would return no cursor and silently end enumeration while is_truncated said otherwise, making an audit job under-report. It now falls back to the last key seen; StartAfter is a string comparison, so a non-hash resume point is fine. "Cursor is a hash" constrains what callers may synthesise, not what backends may return. Azure is unaffected — it does not implement list_blob_hashes (TODO, inherits the NotSupported default). Adds the first test for enumeration at all: ordering across shards with deliberately out-of-order inserts, complete paged traversal, and resume from a caller-synthesised cursor. NOT verified against real S3 — no bucket available here. The local path is covered by the new test; the StartAfter change is reasoned from the API contract and needs exercising against a real bucket before it is relied on. Co-Authored-By: Claude Opus 5 (1M context) --- src/application/ports/blob_storage_ports.rs | 21 +++- .../services/local_blob_backend.rs | 95 +++++++++++++++++-- .../services/s3_blob_backend.rs | 44 ++++++++- 3 files changed, 145 insertions(+), 15 deletions(-) diff --git a/src/application/ports/blob_storage_ports.rs b/src/application/ports/blob_storage_ports.rs index 262a1321..ae8732d3 100644 --- a/src/application/ports/blob_storage_ports.rs +++ b/src/application/ports/blob_storage_ports.rs @@ -244,11 +244,28 @@ pub trait BlobStorageBackend: Send + Sync + 'static { /// /// * `cursor` — opaque continuation token from a prior call, or /// `None` to start from the beginning. Format is per-backend - /// (local = last path visited; S3 = continuation token; Azure - /// = list marker); callers treat it as opaque. + /// — **the last blob hash returned by the previous page**. + /// Enumeration resumes strictly AFTER that hash. + /// + /// This is deliberately NOT an opaque backend token. Callers may + /// synthesise a cursor from any hash they hold, which is what lets a + /// consistency sweep merge-join this stream against a + /// `storage.blobs` walk and resume both sides from one checkpoint. + /// An opaque token would force the backend side to re-enumerate from + /// the beginning on every resume. /// * `limit` — soft cap on batch size; backends may return /// fewer (e.g. end of a shard directory). /// + /// **Entries MUST be returned in ascending hash order**, and pages must + /// be contiguous in that order. Every shipped backend already satisfies + /// this — local sorts within each shard and walks shards `00`..`ff` + /// (the shard IS the hash prefix, so that is globally sorted); S3 and + /// Azure list lexicographically by key, and `blobs//` sorts + /// identically to ``. It is stated here because the merge-join in + /// `backend_consistency` depends on it: an unordered backend would + /// silently emit bogus `blob_missing_from_backend` findings at + /// `data_loss` severity. + /// /// Returns `(entries, next_cursor)`. `next_cursor = None` means /// enumeration is complete. Each `BackendBlobEntry` carries the /// hash + optional mtime for grace-window filtering. diff --git a/src/infrastructure/services/local_blob_backend.rs b/src/infrastructure/services/local_blob_backend.rs index 20ed4eaf..4f0fac15 100644 --- a/src/infrastructure/services/local_blob_backend.rs +++ b/src/infrastructure/services/local_blob_backend.rs @@ -778,12 +778,30 @@ impl BlobStorageBackend for LocalBlobBackend { let blob_root = self.blob_root.clone(); Box::pin(async move { + // Cursor is the last hash returned (see the port contract). The + // shard is derivable from it — the shard name IS the hash's first + // two chars — so no composite is needed. + // + // Both legacy forms still resume correctly, so a consistency run + // paused across this deploy is not stranded: + // * "/" — what this backend used to emit; the + // hash half is taken and the shard re-derived from it. + // * "" — a bare 2-char shard. It flows through the same + // path: "3f" sorts BEFORE every 64-char hash beginning "3f", + // so using it as start_after skips nothing. let (start_shard, start_after_hash): (String, Option) = match cursor { None => (String::from("00"), None), - Some(c) => match c.split_once('/') { - Some((sh, h)) => (sh.to_string(), Some(h.to_string())), - None => (c, None), - }, + Some(c) => { + let hash = c.split_once('/').map(|(_, h)| h).unwrap_or(c.as_str()); + if hash.len() >= 2 { + (hash[..2].to_string(), Some(hash.to_string())) + } else { + // Under 2 chars — not a hash and not a shard. Should + // be unreachable; start from the beginning rather + // than index out of bounds. + (String::from("00"), None) + } + } }; let mut blobs: Vec = Vec::with_capacity(limit); @@ -879,11 +897,8 @@ impl BlobStorageBackend for LocalBlobBackend { continue; } if blobs.len() >= limit { - next_cursor = Some(format!( - "{}/{}", - prefix, - blobs.last().map(|e| e.hash.as_str()).unwrap_or("") - )); + // Just the hash — the shard is recoverable from it. + next_cursor = blobs.last().map(|e| e.hash.clone()); return Ok(BlobListPage { blobs, unknowns, @@ -1008,4 +1023,66 @@ mod tests { ); assert_eq!(hash_prefix_slot("gg"), None); } + + /// The port contract now REQUIRES ascending hash order and a cursor that + /// is the last hash returned. `backend_consistency`'s merge-join depends + /// on both: an out-of-order page would make it emit bogus + /// `blob_missing_from_backend` findings at `data_loss` severity, and a + /// non-hash cursor would stop a caller resuming from its own checkpoint. + /// + /// Nothing covered enumeration before this, so both properties were + /// accidental. + #[tokio::test] + async fn list_blob_hashes_is_ordered_and_hash_cursor_resumes() { + let dir = TempDir::new().unwrap(); + let backend = LocalBlobBackend::new(dir.path()); + backend.initialize().await.unwrap(); + + // Deliberately inserted out of order and across several shards, so a + // passing result cannot come from insertion order. + let mut written: Vec = ["f0", "0a", "9c", "0b", "ff", "12"] + .iter() + .map(|p| fake_hash(p)) + .collect(); + for h in &written { + backend + .put_blob_from_bytes(h, Bytes::from_static(b"x")) + .await + .unwrap(); + } + written.sort(); + + // Page with limit 2 so the cursor is exercised repeatedly. + let mut seen: Vec = Vec::new(); + let mut cursor: Option = None; + for _ in 0..20 { + let page = backend.list_blob_hashes(cursor.clone(), 2).await.unwrap(); + seen.extend(page.blobs.iter().map(|e| e.hash.clone())); + match page.next_cursor { + Some(c) => cursor = Some(c), + None => break, + } + } + + assert_eq!(seen, written, "enumeration must be complete and ascending"); + + // A cursor the CALLER synthesises from a hash it already holds must + // work — that is the property the merge-join resume relies on, and + // what an opaque backend token could not provide. + let midpoint = &written[2]; + let resumed = backend + .list_blob_hashes(Some(midpoint.clone()), 100) + .await + .unwrap(); + let expected: Vec = written[3..].to_vec(); + assert_eq!( + resumed + .blobs + .iter() + .map(|e| e.hash.clone()) + .collect::>(), + expected, + "resume must start STRICTLY after the given hash" + ); + } } diff --git a/src/infrastructure/services/s3_blob_backend.rs b/src/infrastructure/services/s3_blob_backend.rs index c58ef2ce..9dfc2591 100644 --- a/src/infrastructure/services/s3_blob_backend.rs +++ b/src/infrastructure/services/s3_blob_backend.rs @@ -497,8 +497,13 @@ impl BlobStorageBackend for S3BlobBackend { .list_objects_v2() .bucket(&self.bucket) .max_keys(limit.min(1000) as i32); + // Resume after a HASH, not a continuation token (port contract). + // ListObjectsV2 supports this natively via StartAfter, and it is + // what lets a caller resume the backend side of a merge-join from + // a checkpoint it holds — a continuation token would force a + // re-enumeration from the start on every resume. if let Some(c) = cursor { - req = req.continuation_token(c); + req = req.start_after(Self::object_key(&c)); } let resp = req.send().await.map_err(|e| { @@ -512,6 +517,9 @@ impl BlobStorageBackend for S3BlobBackend { let objects = resp.contents.unwrap_or_default(); let mut blobs: Vec = Vec::with_capacity(objects.len()); let mut unknowns: Vec = Vec::new(); + // Last key of the page regardless of classification — the resume + // fallback for an all-unknowns page (see next_cursor below). + let mut last_key: Option = None; for obj in objects { let Some(key) = obj.key else { continue }; @@ -539,13 +547,41 @@ impl BlobStorageBackend for S3BlobBackend { }); match is_canonical { - Some(hash) => blobs.push(BackendBlobEntry { hash, mtime }), - None => unknowns.push(BackendUnknownEntry { path: key, mtime }), + Some(hash) => { + last_key = Some(hash.clone()); + blobs.push(BackendBlobEntry { hash, mtime }) + } + None => { + last_key = Some(key.clone()); + unknowns.push(BackendUnknownEntry { path: key, mtime }) + } } } + // Resume point: the last hash of this page, not the continuation + // token — see the StartAfter note above. + // + // `blobs.last()` alone is NOT sufficient. A page can legitimately + // contain only non-canonical keys (`.tmp` spool files, `.corrupt` + // sidecars), which are filtered into `unknowns`; `blobs` is then + // empty and a naive `blobs.last()` yields None, silently ending + // enumeration while `is_truncated` says otherwise. A consistency + // sweep would under-report rather than fail — the worst shape of + // bug for an audit job. + // + // So fall back to the last KEY seen. StartAfter is a plain string + // comparison, so any key works as a resume point; it need not be + // a hash. The port contract's "cursor is a hash" is what CALLERS + // may synthesise, not a restriction on what backends may return. let next_cursor = if resp.is_truncated.unwrap_or(false) { - resp.next_continuation_token + match blobs.last() { + Some(entry) => Some(entry.hash.clone()), + None => last_key.map(|k| { + // Strip the `blobs//` prefix: object_key() re-adds + // it when this comes back as a cursor. + k.rsplit('/').next().unwrap_or(&k).to_string() + }), + } } else { None }; From 7261b5b175ab728750fd155158718ffb122b65fd Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 22:08:30 +0200 Subject: [PATCH 08/66] fix(s3): never return a mangled enumeration cursor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 5343fdda switched S3 blob enumeration from an opaque continuation token to a hash cursor (StartAfter), per the port contract. Its fallback for a page containing no canonical blob was wrong: it stored the full key (`0a/junk.tmp`), stripped it to a basename (`junk.tmp`), and the next call fed that to `object_key()` — producing `ju/junk.tmp.blob`. Wrong shard and a doubled extension, so the resume jumped to an arbitrary position: skipped objects, or backwards into a loop. The cursor can only ever be a real hash, because `object_key()` is applied to it. So instead of synthesising one, keep listing internally until the page holds at least one blob or the bucket is exhausted. The continuation token is used only inside the call and never escapes. Two pathological cases cannot produce a cursor at all — `is_truncated` with no token (protocol violation), and a run of foreign keys long enough to buffer the bucket. Both now fail loudly. A visible job failure beats a sweep reporting "no missing blobs" having read a fraction of them. Extract `hash_from_object_key` as the paired inverse of `object_key`, with the round-trip and the rejection set under test. It also now requires the shard to match the hash's own prefix, which the inline filter did not check. --- .../services/s3_blob_backend.rs | 265 ++++++++++++------ 1 file changed, 176 insertions(+), 89 deletions(-) diff --git a/src/infrastructure/services/s3_blob_backend.rs b/src/infrastructure/services/s3_blob_backend.rs index 9dfc2591..6f877f2c 100644 --- a/src/infrastructure/services/s3_blob_backend.rs +++ b/src/infrastructure/services/s3_blob_backend.rs @@ -65,6 +65,32 @@ impl S3BlobBackend { let prefix = &hash[0..2]; format!("{}/{}.blob", prefix, hash) } + + /// Inverse of [`Self::object_key`] — the hash a key names, or `None` + /// when the key is not one we wrote. + /// + /// Deliberately strict, and paired with `object_key` so the round-trip + /// stays honest. Enumeration passes no prefix to S3, so this filter is + /// the *only* thing separating our namespace from everything else in + /// the bucket; a lenient match would feed a non-hash into + /// `object_key`, which slices `[0..2]` and would produce a nonsense + /// resume position. + fn hash_from_object_key(key: &str) -> Option { + let (prefix, rest) = key.split_once('/')?; + if prefix.len() != 2 || !prefix.chars().all(|c| c.is_ascii_hexdigit()) { + return None; + } + let stem = rest.strip_suffix(".blob")?; + if stem.len() != 64 || !stem.chars().all(|c| c.is_ascii_hexdigit()) { + return None; + } + // The shard must be the hash's own first two characters, or + // `object_key(hash)` would not reproduce this key. + if !stem.starts_with(prefix) { + return None; + } + Some(stem.to_string()) + } } impl BlobStorageBackend for S3BlobBackend { @@ -464,14 +490,21 @@ impl BlobStorageBackend for S3BlobBackend { None // Remote backend — no local path } - /// Enumerate blobs via S3 `ListObjectsV2`. Cursor is the S3 - /// continuation token verbatim (opaque). Filter: keys must - /// match `/<64-hex>.blob` — matches how `blob_key` writes - /// them — so any future non-blob namespace living in the same - /// bucket (e.g. `thumbnails/.jpg`) is skipped - /// automatically. No prefix passed to S3 so we get everything - /// in one paginated scan; the client-side filter enforces - /// correctness. + /// Enumerate blobs via S3 `ListObjectsV2`, in ascending hash order. + /// + /// The cursor is a **hash**, per the port contract — resumed via + /// `StartAfter`, not a continuation token. That is what lets a caller + /// resume the backend side of a merge-join from a checkpoint it + /// already holds; a continuation token would force re-enumeration + /// from the start on every resume. + /// + /// No prefix is passed to S3, so the scan covers the whole bucket and + /// [`Self::hash_from_object_key`] does the filtering. Keys that are + /// not ours come back as `unknowns` rather than being dropped, so an + /// operator can see what is sharing the bucket. **On a bucket shared + /// with other workloads that means every foreign object is reported + /// as an unknown on every sweep** — give OxiCloud its own bucket, or + /// expect the noise. fn list_blob_hashes( &self, cursor: Option, @@ -492,99 +525,110 @@ impl BlobStorageBackend for S3BlobBackend { }; Box::pin(async move { - let mut req = self - .client - .list_objects_v2() - .bucket(&self.bucket) - .max_keys(limit.min(1000) as i32); - // Resume after a HASH, not a continuation token (port contract). - // ListObjectsV2 supports this natively via StartAfter, and it is - // what lets a caller resume the backend side of a merge-join from - // a checkpoint it holds — a continuation token would force a - // re-enumeration from the start on every resume. - if let Some(c) = cursor { - req = req.start_after(Self::object_key(&c)); - } + // A page's cursor can only be the last blob hash on it, because + // the contract says the cursor IS a hash and `StartAfter` needs + // `object_key()` applied to it. A page holding only foreign keys + // therefore yields no cursor — and returning `None` there would + // end enumeration while the bucket still has objects, making an + // audit job under-report. That is the worst failure shape for a + // check whose entire purpose is finding missing data. + // + // So keep listing until the accumulated page holds at least one + // blob, or the bucket is exhausted. The continuation token is + // used only INSIDE this call and never escapes as a cursor. + // Bounded on foreign keys accumulated rather than on requests + // made: the request count scales with the caller's `limit`, so a + // request cap would fire on a healthy bucket merely because the + // caller paged finely. + const MAX_UNKNOWNS: usize = 10_000; - let resp = req.send().await.map_err(|e| { - DomainError::new( - ErrorKind::InternalError, - "Blob", - format!("S3 ListObjectsV2 failed: {e}"), - ) - })?; - - let objects = resp.contents.unwrap_or_default(); - let mut blobs: Vec = Vec::with_capacity(objects.len()); + let mut blobs: Vec = Vec::new(); let mut unknowns: Vec = Vec::new(); - // Last key of the page regardless of classification — the resume - // fallback for an all-unknowns page (see next_cursor below). - let mut last_key: Option = None; + let mut continuation: Option = None; + let mut requests = 0usize; + // Assigned on every path through the loop body before any exit. + let mut truncated; - for obj in objects { - let Some(key) = obj.key else { continue }; - let mtime = obj.last_modified.and_then(|ts| { - let secs = ts.secs(); - let nsecs = ts.subsec_nanos(); - chrono::DateTime::::from_timestamp(secs, nsecs) - }); + loop { + let mut req = self + .client + .list_objects_v2() + .bucket(&self.bucket) + .max_keys(limit.min(1000) as i32); + match (&continuation, &cursor) { + // Mid-loop: continue exactly where the last inner + // request stopped. + (Some(token), _) => req = req.continuation_token(token), + // First request: resume after the caller's hash. + (None, Some(c)) => req = req.start_after(Self::object_key(c)), + (None, None) => {} + } - // Canonical S3 key shape: `/<64-hex>.blob`. - // Anything else is a sidecar or foreign namespace - // (e.g. future `thumbnails/.jpg` if Ed adds - // that) — surface as an unknown so operators know - // it's there. Recovery framework can decide per- - // pattern how to act. - let is_canonical = key.split_once('/').and_then(|(prefix, rest)| { - if prefix.len() != 2 || !prefix.chars().all(|c| c.is_ascii_hexdigit()) { - return None; - } - rest.strip_suffix(".blob") - .filter(|stem| { - stem.len() == 64 && stem.chars().all(|c| c.is_ascii_hexdigit()) - }) - .map(|s| s.to_string()) - }); + let resp = req.send().await.map_err(|e| { + DomainError::new( + ErrorKind::InternalError, + "Blob", + format!("S3 ListObjectsV2 failed: {e}"), + ) + })?; - match is_canonical { - Some(hash) => { - last_key = Some(hash.clone()); - blobs.push(BackendBlobEntry { hash, mtime }) - } - None => { - last_key = Some(key.clone()); - unknowns.push(BackendUnknownEntry { path: key, mtime }) + requests += 1; + truncated = resp.is_truncated.unwrap_or(false); + continuation = resp.next_continuation_token; + + for obj in resp.contents.unwrap_or_default() { + let Some(key) = obj.key else { continue }; + let mtime = obj.last_modified.and_then(|ts| { + chrono::DateTime::::from_timestamp( + ts.secs(), + ts.subsec_nanos(), + ) + }); + + match Self::hash_from_object_key(&key) { + Some(hash) => blobs.push(BackendBlobEntry { hash, mtime }), + // Not ours: a spool file, a sidecar, or another + // workload sharing the bucket. Surfaced rather than + // dropped so operators can see it; the recovery + // framework decides per pattern how to act. + None => unknowns.push(BackendUnknownEntry { path: key, mtime }), } } + + if !blobs.is_empty() || !truncated { + break; + } + + // `is_truncated` with no token is a protocol violation, and a + // huge run of foreign keys means we would buffer the bucket to + // find one blob. Neither can produce a valid cursor, so fail + // loudly: a visible job failure beats a sweep that silently + // reports "no missing blobs" having read a fraction of them. + if continuation.is_none() || unknowns.len() >= MAX_UNKNOWNS { + return Err(DomainError::new( + ErrorKind::InternalError, + "Blob", + format!( + "S3 enumeration stalled after {requests} request(s) and {} \ + non-blob key(s) without reaching a blob, so no resume cursor \ + can be produced. Bucket '{}' likely holds a large foreign \ + namespace — give OxiCloud a dedicated bucket.", + unknowns.len(), + self.bucket, + ), + )); + } } - // Resume point: the last hash of this page, not the continuation - // token — see the StartAfter note above. - // - // `blobs.last()` alone is NOT sufficient. A page can legitimately - // contain only non-canonical keys (`.tmp` spool files, `.corrupt` - // sidecars), which are filtered into `unknowns`; `blobs` is then - // empty and a naive `blobs.last()` yields None, silently ending - // enumeration while `is_truncated` says otherwise. A consistency - // sweep would under-report rather than fail — the worst shape of - // bug for an audit job. - // - // So fall back to the last KEY seen. StartAfter is a plain string - // comparison, so any key works as a resume point; it need not be - // a hash. The port contract's "cursor is a hash" is what CALLERS - // may synthesise, not a restriction on what backends may return. - let next_cursor = if resp.is_truncated.unwrap_or(false) { - match blobs.last() { - Some(entry) => Some(entry.hash.clone()), - None => last_key.map(|k| { - // Strip the `blobs//` prefix: object_key() re-adds - // it when this comes back as a cursor. - k.rsplit('/').next().unwrap_or(&k).to_string() - }), - } + // Always a real hash: the loop above only exits with an empty + // `blobs` when the listing is exhausted, and then there is + // nothing to resume from. + let next_cursor = if truncated { + blobs.last().map(|entry| entry.hash.clone()) } else { None }; + Ok(BlobListPage { blobs, unknowns, @@ -659,3 +703,46 @@ where _ => format!("unknown SDK error: {err:?}"), } } + +#[cfg(test)] +mod tests { + use super::*; + + const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + + /// The enumeration cursor is fed straight back into `object_key`, so a + /// key that does not round-trip would resume at the wrong position. + #[test] + fn object_key_round_trips_through_hash_from_object_key() { + let key = S3BlobBackend::object_key(H); + assert_eq!(key, format!("0a/{H}.blob")); + assert_eq!( + S3BlobBackend::hash_from_object_key(&key).as_deref(), + Some(H) + ); + } + + /// Each of these previously risked being treated as a hash and sliced + /// `[0..2]` to build a resume position. + #[test] + fn non_canonical_keys_are_rejected() { + let cases = [ + "0a/junk.tmp".to_string(), // spool file + "junk.tmp".to_string(), // no shard + "0a/junk".to_string(), // no suffix + "thumbnails/abc.jpg".to_string(), // foreign namespace + format!("0a/{H}.blob.corrupt"), // sidecar + format!("0a/{H}"), // suffix missing + format!("zz/{H}.blob"), // non-hex shard + format!("ff/{H}.blob"), // shard != hash prefix + format!("0a/{}.blob", &H[..63]), // wrong length + ]; + for key in &cases { + assert_eq!( + S3BlobBackend::hash_from_object_key(key), + None, + "must not be read as a blob: {key}" + ); + } + } +} From 9f8ec141f3543a0597aae50270929500d8924881 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 23:04:53 +0200 Subject: [PATCH 09/66] feat(storage): single-source the copy fan-out via copy_file_satellites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was written twice — the copy_file CTE and storage.copy_folder_tree — and had already drifted: the tree path bumped storage.blobs only, missing manifests, which was silent data loss on any multi-chunk file. Fixing it meant writing the same logic a second time. Step 9 adds a file-keyed satellite table, which would mean a third and fourth. Two SQL functions: storage.add_blob_references(TEXT[]) — the manifest-first reference contract for SQL callers, returning hashes that matched no registry row. Set-based so the tree path keeps its single-statement cost; a per-row helper would have made a 10k-file copy 10k calls. storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus the blob reference. The body is the copy-semantics declaration: what is absent (comments, favorites, content-keyed derived rows) is listed with its reason, so the taxonomy is executable rather than documented elsewhere and drifting. Both copy paths now call it. The single-file path becomes a real transaction, which also fixes the reference being best-effort: a failed add_reference used to log a warning and leave a copy holding no reference at all — the exact shape that gets its content reaped. It cannot be a CTE arm, because data-modifying CTEs share one snapshot and the function must read the row the INSERT just wrote. Verified against a scratch PG with all migrations applied: multi-chunk manifest 1→2, single-chunk alias bumped at manifest level only (the NOT EXISTS guard), chunks behind a manifest untouched, dead properties duplicated, length mismatch rejected, repeats counted. tests/api/derived_blob_copy.hurl covers it end-to-end and answers the question the copy raises: content_derived_blobs is NOT copied. A copy carries the same blob_hash, so it resolves the same derived row — the test asserts byte-identical thumbnails from both copy paths, then deletes the original, runs GC, and requires both copies to still serve. That last step only passes if the references are real. --- .../20261019000000_copy_file_satellites.sql | 332 ++++++++++++++++++ .../pg/file_blob_write_repository.rs | 68 ++-- tests/api/derived_blob_copy.hurl | 326 +++++++++++++++++ tests/api/run.sh | 1 + 4 files changed, 700 insertions(+), 27 deletions(-) create mode 100644 migrations/20261019000000_copy_file_satellites.sql create mode 100644 tests/api/derived_blob_copy.hurl diff --git a/migrations/20261019000000_copy_file_satellites.sql b/migrations/20261019000000_copy_file_satellites.sql new file mode 100644 index 00000000..030812bd --- /dev/null +++ b/migrations/20261019000000_copy_file_satellites.sql @@ -0,0 +1,332 @@ +-- Step 8 of `docs/plan/derived-blobs.md` — single-source the copy fan-out. +-- +-- "What follows a file when the file is copied" was written twice: once in +-- the `copy_file` CTE (Rust, `file_blob_write_repository.rs`) and once in +-- `storage.copy_folder_tree`. They had already drifted — the tree path +-- bumped `storage.blobs` only, missing manifests entirely, which was silent +-- data loss on a multi-chunk file (fixed in `20261016000000`, and the fix +-- had to be written a second time rather than in one place). +-- +-- The plan adds file-keyed satellite tables (`file_attached_blobs`, step 9). +-- Adding them against two copy sites means writing the same cascade a third +-- and fourth time, into sites that have already proven they drift. So the +-- fan-out gets exactly one home first. +-- +-- Two functions land here: +-- +-- * `storage.add_blob_references(TEXT[])` — the manifest-first reference +-- contract, expressed once for SQL callers. `DedupService::add_reference` +-- is the Rust twin; they must change together, which is why the shared +-- contract is spelled out in both doc comments. +-- +-- * `storage.copy_file_satellites(UUID[], UUID[])` — everything that +-- follows a file on copy. The body IS the copy-semantics declaration: +-- what is absent is a documented decision (see the trailing comments), +-- not an omission someone has to notice. +-- +-- Set-based rather than per-row on purpose. A per-row helper would have made +-- a 10k-file folder copy 10k function calls; taking arrays keeps the tree +-- path's single-statement cost while still having one implementation. The +-- single-file path passes one-element arrays. + +-- ── The reference contract, for SQL callers ────────────────────────────── +-- +-- Increment the reference count for each hash in `p_hashes`, counting +-- repeats (pass the hash once per referencing row). Returns the hashes that +-- matched NEITHER table, so callers can decide how loud to be — a copy +-- inherits a pre-existing breakage and should warn, whereas an ingest +-- referencing a nonexistent blob is a hard error. +-- +-- MANIFEST FIRST, `storage.blobs` only as fallback. The order is the whole +-- point: a CDC file's `blob_hash` names a manifest +-- (`chunk_manifests.file_hash`), not a chunk, so bumping `storage.blobs` +-- first would match nothing for a multi-chunk file and take no reference at +-- all. +-- +-- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing. For a +-- SINGLE-chunk file the whole-file hash EQUALS its lone chunk's hash (both +-- are BLAKE3 over the same bytes), so without the guard one reference would +-- be counted at both levels — turning an under-count into an over-count. +-- +-- Mirrors `DedupService::add_reference`, including the asymmetry on +-- `orphaned_at`: only `storage.blobs` carries that column, so only the blobs +-- branch clears it. A chunk resurrected inside its GC grace window must lose +-- its orphan stamp or `dedup_gc` reaps live content. +CREATE OR REPLACE FUNCTION storage.add_blob_references(p_hashes TEXT[]) +RETURNS TEXT[] AS $$ +DECLARE + v_unmatched TEXT[]; +BEGIN + IF p_hashes IS NULL OR cardinality(p_hashes) = 0 THEN + RETURN ARRAY[]::TEXT[]; + END IF; + + WITH hc AS ( + SELECT h AS blob_hash, COUNT(*)::int AS cnt + FROM unnest(p_hashes) AS h + WHERE h IS NOT NULL + GROUP BY h + ), + bumped_manifests AS ( + UPDATE storage.chunk_manifests m + SET ref_count = m.ref_count + hc.cnt + FROM hc + WHERE m.file_hash = hc.blob_hash + RETURNING m.file_hash + ), + bumped_blobs AS ( + UPDATE storage.blobs b + SET ref_count = b.ref_count + hc.cnt, + orphaned_at = NULL + FROM hc + WHERE b.hash = hc.blob_hash + AND NOT EXISTS ( + SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash + ) + RETURNING b.hash + ) + SELECT COALESCE(array_agg(hc.blob_hash), ARRAY[]::TEXT[]) + INTO v_unmatched + FROM hc + WHERE NOT EXISTS (SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash) + AND NOT EXISTS (SELECT 1 FROM bumped_blobs WHERE hash = hc.blob_hash); + + RETURN v_unmatched; +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION storage.add_blob_references(TEXT[]) IS + 'Manifest-first blob reference increment for SQL callers. Returns hashes ' + 'that matched no registry row. Rust twin: DedupService::add_reference — ' + 'change both together.'; + +-- ── What follows a file on copy ────────────────────────────────────────── +-- +-- `p_old_ids[i]` is copied to `p_new_ids[i]`; the new `storage.files` rows +-- must already be inserted and visible (both callers insert in an earlier +-- statement of the same transaction). +-- +-- Every satellite of a copied file belongs in this body. What is NOT here is +-- listed at the bottom, with the reason — the taxonomy is executable rather +-- than living in a document that drifts from the code. +CREATE OR REPLACE FUNCTION storage.copy_file_satellites( + p_old_ids UUID[], + p_new_ids UUID[] +) RETURNS void AS $$ +DECLARE + v_unmatched TEXT[]; +BEGIN + IF p_old_ids IS NULL OR cardinality(p_old_ids) = 0 THEN + RETURN; + END IF; + + IF p_new_ids IS NULL OR cardinality(p_old_ids) <> cardinality(p_new_ids) THEN + -- Positional correspondence is the whole interface; a length + -- mismatch would silently attach satellites to the wrong file. + RAISE EXCEPTION + 'copy_file_satellites: id arrays must correspond positionally (% old vs % new)', + cardinality(p_old_ids), COALESCE(cardinality(p_new_ids), 0); + END IF; + + -- 1. WebDAV dead properties. RFC 4918 §8.8 requires COPY to duplicate + -- them: properties describe the resource, and the copy is a resource. + INSERT INTO storage.webdav_dead_properties + (file_id, namespace, local_name, value) + SELECT m.new_id, dp.namespace, dp.local_name, dp.value + FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id) + JOIN storage.webdav_dead_properties dp ON dp.file_id = m.old_id; + + -- 2. A reference on the copied content, so deleting the original cannot + -- reap bytes the copy still needs. Read from the NEW rows rather than + -- the old ones: that is what makes an unreferenceable copy impossible + -- to create, since a row that failed to insert contributes nothing. + SELECT storage.add_blob_references(array_agg(f.blob_hash)) + INTO v_unmatched + FROM unnest(p_new_ids) AS n(id) + JOIN storage.files f ON f.id = n.id + WHERE NOT f.is_trashed; + + IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN + -- Warn, do not abort. A missing registry row means the SOURCE file + -- was already broken; the copy merely inherits it. Failing here + -- would abort an entire folder copy over one pre-existing fault, + -- which is worse than completing it and reporting. The blob-level + -- audit jobs are what surface the underlying breakage. + RAISE WARNING + 'copy_file_satellites: % copied file(s) reference a blob with no registry row (first: %); source was already broken', + cardinality(v_unmatched), v_unmatched[1]; + END IF; + + -- ── Deliberately absent ────────────────────────────────────────────── + -- + -- storage.comments (future): NOT copied. A copy is a new artifact; the + -- discussion belongs to the original. + -- + -- storage.file_attached_blobs (step 9): WILL be copied here, with a + -- reference taken per attached blob_hash via add_blob_references. + -- + -- content_derived_blobs, blob_extracted_text, faces.faces: content-keyed. + -- The copy shares the source's hash, so it already sees them — copying + -- would duplicate rows that are keyed on the very thing being shared. + -- + -- storage.favorites, recent_items, shares: properties of the ORIGINAL's + -- relationship to users, not of its content. +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION storage.copy_file_satellites(UUID[], UUID[]) IS + 'Single source of truth for what follows a file on copy. Both copy paths ' + '(single-file and copy_folder_tree) call it. Adding a file-keyed satellite ' + 'table means editing this function, and only this function.'; + +-- ── Route copy_folder_tree through it ──────────────────────────────────── +-- +-- Only two blocks change versus `20261016000000`: the inline reference bump +-- and the per-file dead-property INSERT are both replaced by one +-- `copy_file_satellites` call. The folder dead-property INSERT stays inline +-- — folders are not files and have no satellite fan-out to share. +CREATE OR REPLACE FUNCTION storage.copy_folder_tree( + p_source_id UUID, + p_target_parent_id UUID, -- NULL = copy to root (keeps source drive) + p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name +) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$ +DECLARE + v_root_lpath ltree; + v_root_depth INT; + v_max_depth INT; + v_level INT; + v_folders BIGINT := 0; + v_files BIGINT := 0; + v_inserted BIGINT; + v_new_root UUID; + v_dest_drive_id UUID; +BEGIN + -- Validate source exists. + SELECT fo.lpath, nlevel(fo.lpath) + INTO v_root_lpath, v_root_depth + FROM storage.folders fo + WHERE fo.id = p_source_id AND NOT fo.is_trashed; + + IF v_root_lpath IS NULL THEN + RAISE EXCEPTION 'Source folder not found: %', p_source_id + USING ERRCODE = 'P0002'; -- no_data_found + END IF; + + -- Resolve destination drive_id once up front (cross-drive copy path). + IF p_target_parent_id IS NULL THEN + SELECT fo.drive_id INTO v_dest_drive_id + FROM storage.folders fo + WHERE fo.id = p_source_id; + ELSE + SELECT fo.drive_id INTO v_dest_drive_id + FROM storage.folders fo + WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed; + IF v_dest_drive_id IS NULL THEN + RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id + USING ERRCODE = 'P0002'; + END IF; + END IF; + + -- Temp mapping: every folder in the subtree → new UUID. + CREATE TEMP TABLE IF NOT EXISTS _copy_map( + old_id UUID PRIMARY KEY, + new_id UUID NOT NULL DEFAULT gen_random_uuid() + ) ON COMMIT DROP; + TRUNCATE _copy_map; + + INSERT INTO _copy_map(old_id) + SELECT fo.id + FROM storage.folders fo + WHERE NOT fo.is_trashed + AND fo.lpath <@ v_root_lpath; + + SELECT cm.new_id INTO v_new_root + FROM _copy_map cm WHERE cm.old_id = p_source_id; + + SELECT MAX(nlevel(fo.lpath)) + INTO v_max_depth + FROM storage.folders fo + JOIN _copy_map cm ON fo.id = cm.old_id; + + -- ── Insert folders level by level ── + -- Post-D7: `user_id` intentionally omitted from the column list so + -- copied rows leave the (now-nullable) column NULL. Provenance is + -- carried by `created_by` / `updated_by` (§14 columns) — preserved + -- from source so authorship survives the copy. + FOR v_level IN v_root_depth .. v_max_depth LOOP + INSERT INTO storage.folders( + id, name, parent_id, + drive_id, created_by, updated_by + ) + SELECT cm.new_id, + CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL + THEN p_dest_name ELSE fo.name END, + CASE WHEN fo.id = p_source_id THEN p_target_parent_id + ELSE pm.new_id END, + v_dest_drive_id, + fo.created_by, + fo.updated_by + FROM storage.folders fo + JOIN _copy_map cm ON fo.id = cm.old_id + LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id + WHERE NOT fo.is_trashed + AND nlevel(fo.lpath) = v_level; + + GET DIAGNOSTICS v_inserted = ROW_COUNT; + v_folders := v_folders + v_inserted; + END LOOP; + + -- Temp mapping for files src→dst (dst ids pre-allocated so we can hand + -- both sides to copy_file_satellites below). + CREATE TEMP TABLE IF NOT EXISTS _copy_file_map( + old_id UUID PRIMARY KEY, + new_id UUID NOT NULL DEFAULT gen_random_uuid() + ) ON COMMIT DROP; + TRUNCATE _copy_file_map; + + INSERT INTO _copy_file_map(old_id) + SELECT f.id + FROM storage.files f + JOIN _copy_map cm ON f.folder_id = cm.old_id + WHERE NOT f.is_trashed; + + -- ── Batch copy all files (zero-copy: same blob_hash) ── + -- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`. + INSERT INTO storage.files( + id, name, folder_id, blob_hash, size, mime_type, + media_sort_date, drive_id, created_by, updated_by + ) + SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size, + f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by, + f.updated_by + FROM storage.files f + JOIN _copy_map cm ON f.folder_id = cm.old_id + JOIN _copy_file_map fm ON fm.old_id = f.id + WHERE NOT f.is_trashed; + + GET DIAGNOSTICS v_files = ROW_COUNT; + + -- Everything that follows a file on copy — blob references and dead + -- properties — in one call, shared with the single-file copy path. + -- + -- Both aggregates order by `old_id`, which is what makes the two arrays + -- correspond positionally; `array_agg` without a matching ORDER BY would + -- be free to pair a file with another file's satellites. + IF v_files > 0 THEN + PERFORM storage.copy_file_satellites( + (SELECT array_agg(old_id ORDER BY old_id) FROM _copy_file_map), + (SELECT array_agg(new_id ORDER BY old_id) FROM _copy_file_map) + ); + END IF; + + -- Folder dead properties. Files are handled inside copy_file_satellites; + -- folders have no other satellites, so this stays here. + INSERT INTO storage.webdav_dead_properties + (folder_id, namespace, local_name, value) + SELECT cm.new_id, dp.namespace, dp.local_name, dp.value + FROM storage.webdav_dead_properties dp + JOIN _copy_map cm ON dp.folder_id = cm.old_id; + + RETURN QUERY SELECT v_new_root::text, v_folders, v_files; +END; +$$ LANGUAGE plpgsql; diff --git a/src/infrastructure/repositories/pg/file_blob_write_repository.rs b/src/infrastructure/repositories/pg/file_blob_write_repository.rs index 0e57777c..f9cb56af 100644 --- a/src/infrastructure/repositories/pg/file_blob_write_repository.rs +++ b/src/infrastructure/repositories/pg/file_blob_write_repository.rs @@ -596,8 +596,25 @@ impl FileWritePort for FileBlobWriteRepository { new_name: Option<&str>, caller_id: Uuid, ) -> Result { - // Atomic CTE: read source file → insert new row with same blob_hash → increment ref_count. - // Single round-trip; blob content is NOT copied (dedup makes this zero-copy). + // Two statements in one transaction: insert the new row (same + // blob_hash — blob content is never copied, dedup makes this + // zero-copy), then run the shared satellite fan-out. + // + // `storage.copy_file_satellites` is the single home for everything + // that follows a file on copy — dead properties and the + // manifest-aware blob reference — shared with + // `storage.copy_folder_tree`. Two sites implementing that + // separately is what let the tree path ship a version that missed + // manifests entirely (migration `20261019000000`). + // + // It cannot be a CTE arm: data-modifying CTEs all observe the same + // snapshot, so a function called alongside the INSERT would not see + // the new `storage.files` row it needs to read `blob_hash` from, + // and the dead-property INSERT would fail its foreign key. Hence a + // real transaction — which also fixes the reference being + // best-effort before: a failed `add_reference` used to log a + // warning and leave a copy holding no reference at all, the exact + // shape that gets its content reaped. // // §14: `created_by = $4 = updated_by = caller_id` — the caller // authored this copy. The previous binding used @@ -607,8 +624,10 @@ impl FileWritePort for FileBlobWriteRepository { let target_fid = target_folder_id.clone(); let rename_to = new_name.map(|s| s.to_string()); - let row = retry_on_deadlock("files.copy", || { - sqlx::query_as::< + let row = retry_on_deadlock("files.copy", || async { + let mut tx = self.pool.begin().await?; + + let row = sqlx::query_as::< _, ( String, @@ -662,20 +681,6 @@ impl FileWritePort for FileBlobWriteRepository { blob_hash, created_by, updated_by - ), - -- RFC 4918 §8.8 — dead properties MUST be duplicated on - -- COPY. With the id-keyed store (migration - -- 20260830000001) this is a single batch INSERT keyed on - -- the new file's id. Runs in the same query as the file - -- INSERT so either both land or neither does — atomic - -- by virtue of being one statement. - dead_prop_copy AS ( - INSERT INTO storage.webdav_dead_properties - (file_id, namespace, local_name, value) - SELECT (SELECT id FROM new_file), - dp.namespace, dp.local_name, dp.value - FROM storage.webdav_dead_properties dp - WHERE dp.file_id = $1::uuid ) SELECT id_text, name, folder_id, size, mime_type, created_at, updated_at, @@ -687,7 +692,22 @@ impl FileWritePort for FileBlobWriteRepository { .bind(&target_fid) .bind(&rename_to) .bind(caller_id) - .fetch_optional(self.pool.as_ref()) + .fetch_optional(&mut *tx) + .await?; + + if let Some(ref new_row) = row { + // `new_row.0` is the new file's id as text; PG casts it. + sqlx::query( + "SELECT storage.copy_file_satellites(ARRAY[$1::uuid], ARRAY[$2::uuid])", + ) + .bind(file_id) + .bind(&new_row.0) + .execute(&mut *tx) + .await?; + } + + tx.commit().await?; + Ok(row) }) .await .map_err(|e| { @@ -705,14 +725,8 @@ impl FileWritePort for FileBlobWriteRepository { let blob_hash = &row.7; - // Increment blob reference count (best-effort; INSERT already succeeded) - if let Err(e) = self.dedup.add_reference(blob_hash).await { - tracing::warn!( - "Failed to increment blob ref for copy {}: {}", - &blob_hash[..12], - e - ); - } + // No `add_reference` here: `copy_file_satellites` took it inside the + // transaction above, so a copy that exists always holds a reference. tracing::info!( "📋 BLOB COPY: {} (hash: {}, zero-copy via dedup)", diff --git a/tests/api/derived_blob_copy.hurl b/tests/api/derived_blob_copy.hurl new file mode 100644 index 00000000..20e5987b --- /dev/null +++ b/tests/api/derived_blob_copy.hurl @@ -0,0 +1,326 @@ +# ============================================================= +# OxiCloud – Derived blobs survive a copy, and are SHARED not duplicated +# ============================================================= +# Guards two properties of `docs/plan/derived-blobs.md` that are easy to +# break and silent when broken. +# +# 1. **Derived content is content-keyed, so a copy gets it for free.** +# `storage.content_derived_blobs` is keyed on `source_hash`, and a copy +# carries the SAME `blob_hash` as its original. So the copy resolves to +# the very same thumbnail row — nothing is duplicated, and nothing is +# re-rendered. A regression that made copy duplicate those rows would +# still return 200 here; the byte-identity assertions are what catch it, +# because a re-render produces different bytes than a cache hit only if +# the pipeline is non-deterministic — so we also assert the ref_count, +# which a duplicated row would inflate. +# +# 2. **A copy takes a real blob reference, via BOTH copy paths.** +# `storage.copy_file_satellites` (migration `20261019000000`) is now the +# single home for that, called by the single-file path and by +# `storage.copy_folder_tree`. The tree path previously bumped +# `storage.blobs` only — which matched nothing for a manifest-backed +# file, so a folder copy took NO reference and deleting the original +# reaped bytes the copy still needed. Steps 6 and 9 are what would fail. +# +# The strongest assertion is step 11: after the ORIGINAL is permanently +# deleted and GC has run, both copies must still serve their thumbnail. +# That only holds if the references were real. +# +# Coverage note: `dedup-test.jpg` is single-chunk, so `file_hash` equals its +# lone chunk's hash — the aliasing case whose `NOT EXISTS` guard stops one +# reference being counted at both levels. The multi-chunk fan-out (where +# file_hash names a manifest that is NOT a chunk) differs only in that the +# hashes differ; it has no thumbnail-capable fixture at this size, so it is +# covered at the SQL level rather than here. +# +# Prerequisites: setup.hurl must have run (admin user exists). +# +# Run: +# hurl --variables-file tests/api/test.env --file-root tests \ +# --test tests/api/derived_blob_copy.hurl +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Source and destination folders +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-derived-src" +} + +HTTP 201 +[Captures] +src_folder_id: jsonpath "$.id" + + +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-derived-dst" +} + +HTTP 201 +[Captures] +dst_folder_id: jsonpath "$.id" + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Upload the source image +# +# `content_hash` is captured rather than hardcoded so the test does not +# break if the fixture is ever regenerated. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +folder_id: {{src_folder_id}} +file: file,fixtures/dedup-test.jpg; image/jpeg + +HTTP 201 +[Captures] +orig_file_id: jsonpath "$.id" +orig_file_name: jsonpath "$.name" +blob_hash: jsonpath "$.content_hash" +[Asserts] +jsonpath "$.content_hash" isString + + +# One file holds the blob. +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 1 + + +# ───────────────────────────────────────────────────────────── +# Step 4 – Render the thumbnail. THIS is what creates the derived blob: +# `content_derived_blobs(source_hash = blob_hash, 'thumbnail', …)`. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +thumb_bytes: bytes + + +# ───────────────────────────────────────────────────────────── +# Step 5 – Single-file copy into the destination folder +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/files/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "file_ids": ["{{orig_file_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +file_copy_id: jsonpath "$.successful[0].id" +[Asserts] +jsonpath "$.successful[0].id" != "{{orig_file_id}}" + + +# ───────────────────────────────────────────────────────────── +# Step 6 – The copy took a reference. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 2 + + +# ───────────────────────────────────────────────────────────── +# Step 7 – The copy serves the SAME thumbnail bytes. +# +# It shares the original's `blob_hash`, so it resolves the same +# `content_derived_blobs` row. Nothing was copied to make this work — +# that is the content-keying payoff. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 8 – Folder copy — the OTHER copy path, through +# `storage.copy_folder_tree` → `copy_file_satellites`. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/folders/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "folder_ids": ["{{src_folder_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +tree_root_id: jsonpath "$.successful[0].new_root_folder_id" +[Asserts] +jsonpath "$.stats.failed" == 0 + + +GET {{base_url}}/api/files?folder_id={{tree_root_id}} +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +tree_copy_id: jsonpath "$[0].id" +[Asserts] +jsonpath "$" count == 1 +jsonpath "$[0].name" == "{{orig_file_name}}" +jsonpath "$[0].id" != "{{orig_file_id}}" +jsonpath "$[0].content_hash" == "{{blob_hash}}" + + +# ───────────────────────────────────────────────────────────── +# Step 9 – Three references now. Before `copy_file_satellites` the tree +# path contributed nothing here and this stayed at 2. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.ref_count" == 3 + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 10 – Permanently delete the ORIGINAL. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/files/{{orig_file_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_orig_id: jsonpath "$.items[?(@.resource.id == '{{orig_file_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_orig_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +# Two copies remain, so the content must too. +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 2 + + +# ───────────────────────────────────────────────────────────── +# Step 11 – Run GC, then prove both copies still work. +# +# This is the assertion the whole file exists for. If either copy had +# failed to take a reference, the original's deletion would have walked +# the count to 0 and GC would have reaped the content AND its derived +# thumbnail — leaving these 5xx. That was a real, shipped bug on the +# folder-copy path. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/dedup_gc/trigger +Authorization: Bearer {{token}} +[Options] +delay: 500ms + +HTTP 200 + + +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 12 – Teardown. Hurl files share one database within run.sh, so +# everything created here must go, including from trash. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/folders/{{src_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +DELETE {{base_url}}/api/folders/{{dst_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_src_id: jsonpath "$.items[?(@.resource.id == '{{src_folder_id}}')].resource.id" +trash_dst_id: jsonpath "$.items[?(@.resource.id == '{{dst_folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_src_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +DELETE {{base_url}}/api/trash/{{trash_dst_id}} +Authorization: Bearer {{token}} + +HTTP 200 diff --git a/tests/api/run.sh b/tests/api/run.sh index 3ff37afb..67619aec 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -166,6 +166,7 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/recent.hurl" \ "$API_DIR/batch_folder_copy.hurl" \ "$API_DIR/dedup_blob_cleanup.hurl" \ + "$API_DIR/derived_blob_copy.hurl" \ "$API_DIR/dedup_admin_gate.hurl" \ "$API_DIR/admin_jobs.hurl" \ "$API_DIR/recoverable_jobs.hurl" \ From 46dc25a9a8ccea1132f850c6016ebf7ff4fe52c7 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 23:07:37 +0200 Subject: [PATCH 10/66] test(api): scope derived_blob_copy claims to what it can observe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The byte-identity assertions were documented as proving that a copy shares the original's content_derived_blobs row. They prove no such thing: rendering is deterministic in the source bytes and the variant, so a copy that re-rendered from scratch returns identical bytes. The copy is in fact a moka hit — that cache is keyed on (source_hash, size, format), which the copy shares — so it never reaches the derived tier here at all. Nor is there an assertion that would fix it. Duplication is impossible by construction: the PK is (source_hash, kind, variant), a copy carries the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING. The schema enforces the property, so no runtime behaviour can violate it and there is nothing to catch. Same limitation narrows step 11: it proves the SOURCE content survived GC, not the derived blob — a reaped derived blob is re-rendered transparently from the live source. What the file does prove is unchanged and is the part that was broken: both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and purging the original does not destroy the copies. No assertions changed. --- tests/api/derived_blob_copy.hurl | 74 ++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 28 deletions(-) diff --git a/tests/api/derived_blob_copy.hurl b/tests/api/derived_blob_copy.hurl index 20e5987b..00e96bc4 100644 --- a/tests/api/derived_blob_copy.hurl +++ b/tests/api/derived_blob_copy.hurl @@ -1,30 +1,46 @@ # ============================================================= # OxiCloud – Derived blobs survive a copy, and are SHARED not duplicated # ============================================================= -# Guards two properties of `docs/plan/derived-blobs.md` that are easy to -# break and silent when broken. +# Guards ONE property, the one that was actually broken: # -# 1. **Derived content is content-keyed, so a copy gets it for free.** -# `storage.content_derived_blobs` is keyed on `source_hash`, and a copy -# carries the SAME `blob_hash` as its original. So the copy resolves to -# the very same thumbnail row — nothing is duplicated, and nothing is -# re-rendered. A regression that made copy duplicate those rows would -# still return 200 here; the byte-identity assertions are what catch it, -# because a re-render produces different bytes than a cache hit only if -# the pipeline is non-deterministic — so we also assert the ref_count, -# which a duplicated row would inflate. +# **A copy takes a real blob reference, via BOTH copy paths.** # -# 2. **A copy takes a real blob reference, via BOTH copy paths.** -# `storage.copy_file_satellites` (migration `20261019000000`) is now the -# single home for that, called by the single-file path and by -# `storage.copy_folder_tree`. The tree path previously bumped -# `storage.blobs` only — which matched nothing for a manifest-backed -# file, so a folder copy took NO reference and deleting the original -# reaped bytes the copy still needed. Steps 6 and 9 are what would fail. +# `storage.copy_file_satellites` (migration `20261019000000`) is the single +# home for that, called by the single-file path and by +# `storage.copy_folder_tree`. The tree path previously bumped +# `storage.blobs` only — which matched nothing for a manifest-backed file, +# so a folder copy took NO reference, and deleting the original reaped +# bytes the copy still needed. Steps 6 and 9 assert the ref_count; step 11 +# purges the original, runs GC, and requires both copies to still serve. # -# The strongest assertion is step 11: after the ORIGINAL is permanently -# deleted and GC has run, both copies must still serve their thumbnail. -# That only holds if the references were real. +# ── What this file does NOT prove, and why it cannot ───────────────────── +# +# It does not prove the copy SHARES the original's `content_derived_blobs` +# row rather than getting its own. Two reasons, and neither is fixable by +# adding assertions here: +# +# 1. Duplication is impossible by construction, so there is nothing to +# catch. The PK is `(source_hash, kind, variant)` and a copy carries the +# SAME `source_hash`, so a second INSERT conflicts — and +# `store_derived_blob` is already `ON CONFLICT DO NOTHING`. The schema +# enforces the property; no runtime behaviour can violate it. +# +# 2. Which tier served a thumbnail is invisible over HTTP. Stored derived +# blob, moka RAM cache, and a fresh re-render all return identical bytes +# with identical status — rendering is deterministic in the source bytes +# and the variant. The copy is in fact a moka hit (that cache is keyed on +# `(source_hash, size, format)`, which the copy shares), so it never +# reaches the derived tier at all in this test. +# +# The `bytes ==` assertions below therefore establish that the pipeline is +# deterministic and that the copies are readable — NOT that the derived +# tier was consulted. Read-path tier selection is observable only from +# inside the process, so it belongs in a Rust unit test over +# `ThumbnailService::get_cached_thumbnail`, not here. +# +# By the same limitation, step 11 proves the SOURCE content survived GC. It +# does not prove the derived blob survived: had GC reaped it, the server +# would re-render from the still-alive source and still answer 200. # # Coverage note: `dedup-test.jpg` is single-chunk, so `file_hash` equals its # lone chunk's hash — the aliasing case whose `NOT EXISTS` guard stops one @@ -157,11 +173,10 @@ jsonpath "$.ref_count" == 2 # ───────────────────────────────────────────────────────────── -# Step 7 – The copy serves the SAME thumbnail bytes. +# Step 7 – The copy is readable and renders the same bytes. # -# It shares the original's `blob_hash`, so it resolves the same -# `content_derived_blobs` row. Nothing was copied to make this work — -# that is the content-keying payoff. +# NOT a proof of derived-blob sharing — see the header. This catches the +# copy being unreadable or resolving to different content. # ───────────────────────────────────────────────────────────── GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview Authorization: Bearer {{token}} @@ -261,9 +276,12 @@ jsonpath "$.ref_count" == 2 # # This is the assertion the whole file exists for. If either copy had # failed to take a reference, the original's deletion would have walked -# the count to 0 and GC would have reaped the content AND its derived -# thumbnail — leaving these 5xx. That was a real, shipped bug on the -# folder-copy path. +# the count to 0 and GC would have reaped the SOURCE CONTENT — leaving +# these 5xx. That was a real, shipped bug on the folder-copy path. +# +# Scope: this proves the source content survived. It says nothing about +# whether the derived blob survived, because a reaped derived blob is +# re-rendered transparently from the live source. See the header. # ───────────────────────────────────────────────────────────── POST {{base_url}}/api/admin/jobs/dedup_gc/trigger Authorization: Bearer {{token}} From a3a93b90ec92d471816ba6aceac8bc9e8cf58a79 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 23:47:57 +0200 Subject: [PATCH 11/66] fix(thumbnails): key the ETag on content hash, not file id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with Cache-Control: public, max-age=31536000, immutable. Replacing a file's content preserves its id — file_upload_service rebuilds the entity with parts.id and a new hash, then fires on_file_updated, which deletes and regenerates the thumbnails — so the server produced a new thumbnail while still advertising the old ETag. Because `immutable` tells a conforming browser not to revalidate at all inside the freshness window, clients kept rendering the previous image for up to a year, unfixably. Keyed on the content hash the directive becomes honest: a thumbnail is a pure function of (source bytes, size, format), so that triple identifies the response. New content yields a new ETag. The same change fixes the opposite direction. A copy, or any dedup twin, had a different id and therefore a different ETag, so clients refetched bytes they already held even though both are served from the same derived blob. Now identical content agrees on an ETag and revalidates to 304 across files, users and copies. Both thumbnail endpoints were affected: the REST handler and the NextCloud preview handler. Cost is one PK lookup ahead of the 304 decision, where the id-keyed version needed none — paid for by no longer serving stale images. It is partly recovered: both handlers already resolved the same hash further down for the render path, and that second lookup is now gone, so the cache-miss path is unchanged and only the 304 path pays. The resolved hash is also handed to get_cached_thumbnail instead of None, saving the service its own lookup. No new disclosure: content_hash is already on FileDto and returned by GET /api/files/{id}. Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite in place via WebDAV PUT, assert the ETag changed, assert a client holding the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the sharing direction: a copy answers with the SAME ETag and revalidates to 304, which is the one externally observable consequence of content-keying and was not previously testable. --- src/interfaces/api/handlers/file_handler.rs | 86 +++++++---- src/interfaces/nextcloud/preview_handler.rs | 57 ++++--- tests/api/derived_blob_copy.hurl | 21 ++- tests/api/run.sh | 1 + tests/api/thumbnail_etag_content_keyed.hurl | 161 ++++++++++++++++++++ 5 files changed, 267 insertions(+), 59 deletions(-) create mode 100644 tests/api/thumbnail_etag_content_keyed.hurl diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 58b576f0..7d2e7f71 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -395,19 +395,24 @@ impl FileHandler { /// Get a thumbnail for a file (image or video). /// - /// **Cache-first**: if the thumbnail already exists in the moka in-memory - /// cache or on disk, serve it immediately — **zero DB queries**. The - /// ownership check was already performed when the thumbnail was first - /// generated (at upload) or uploaded (PUT by the owner). UUIDv4 file IDs - /// have 122 bits of entropy, making enumeration infeasible. + /// **Cache-first**: once past the hash lookup below, a thumbnail already + /// in the moka in-memory cache or on disk is served without further DB + /// work. The ownership check was already performed when the thumbnail + /// was first generated (at upload) or uploaded (PUT by the owner). + /// UUIDv4 file IDs have 122 bits of entropy, making enumeration + /// infeasible. /// - /// **ETag / 304**: responses carry an immutable ETag. If the browser - /// sends `If-None-Match` matching the ETag, we return 304 Not Modified - /// without touching cache or DB — pure header round-trip. + /// **ETag / 304**: responses carry an immutable ETag keyed on the + /// **content hash**, so it identifies the bytes rather than the file. + /// Replacing a file's content changes it (correct invalidation), and two + /// files with identical content share it (a copy revalidates to 304 + /// instead of refetching). Costs one PK lookup on the 304 path, which an + /// id-keyed ETag avoided at the price of never invalidating — see the + /// comment at the ETag construction. /// - /// The DB path is only taken on a **cache miss for images** where the - /// thumbnail hasn't been generated yet (first access after upload if - /// background generation hasn't finished). + /// Beyond that, the DB path is only taken on a **cache miss for images** + /// where the thumbnail hasn't been generated yet (first access after + /// upload if background generation hasn't finished). pub(super) async fn get_thumbnail_impl( State(state): State, auth_user: AuthUser, @@ -449,16 +454,44 @@ impl FileHandler { let format = ThumbnailFormat::from_accept(headers.get(header::ACCEPT).and_then(|v| v.to_str().ok())); - // ── ETag short-circuit (Solution C) ────────────────────────── - // Thumbnails are immutable — the ETag never changes for a given - // (file_id, size, format) triple. If the browser already has it, return - // 304 with zero I/O or DB work. Format is in the ETag so a client that - // switched codecs doesn't get a stale 304. + // ── ETag short-circuit ─────────────────────────────────────── + // Keyed on the CONTENT hash, not the file id. A thumbnail is a pure + // function of (source bytes, size, format), so that triple genuinely + // identifies the response — which is what makes the `immutable` + // directive below an honest claim. + // + // Keying on `file_id` was wrong in both directions. Replacing a + // file's content preserves its id (`file_upload_service` rebuilds the + // entity with `parts.id` and a new hash, then fires + // `on_file_updated`, which regenerates the thumbnails), so the ETag + // never changed — and since `immutable` tells a browser not to + // revalidate at all inside the freshness window, clients kept the old + // preview for up to a year. Conversely a copy, or any dedup twin, got + // a *different* id and so refetched bytes it already held, even + // though the server serves both from the same derived blob. + // + // Cost: one PK lookup, where the id-keyed version needed none. It + // buys correct invalidation plus 304s shared across every file with + // the same content. The lookup runs after the authz check above, + // which has already hit the database. + // + // No new disclosure: `content_hash` is already on `FileDto` and + // returned by `GET /api/files/{id}`, so any caller who reaches here + // could read it anyway. + let blob_hash = match state + .repositories + .file_read_repository + .get_blob_hash(&id) + .await + { + Ok(h) => h, + Err(err) => return AppError::from(err).into_response(), + }; let etag = { let (s, f) = (thumb_size.as_str(), format.as_str()); - let mut e = String::with_capacity(9 + id.len() + s.len() + f.len()); + let mut e = String::with_capacity(9 + blob_hash.len() + s.len() + f.len()); e.push_str("\"thumb-"); - e.push_str(&id); + e.push_str(&blob_hash); e.push('-'); e.push_str(s); e.push('-'); @@ -486,7 +519,9 @@ impl FileHandler { if let Some(data) = thumbnail_service .get_cached_thumbnail( &id, - None, + // Already resolved for the ETag above — hand it over rather + // than let the service look it up a second time. + Some(&blob_hash), thumb_size.into(), format, Some(&state.core.dedup_service), @@ -534,18 +569,7 @@ impl FileHandler { .into_response(); } - // Resolve the blob hash (content-addressable storage). - let blob_hash = match state - .repositories - .file_read_repository - .get_blob_hash(&id) - .await - { - Ok(hash) => hash, - Err(_) => { - return AppError::internal_error("File blob not found").into_response(); - } - }; + // `blob_hash` was resolved above to build the ETag — no second lookup. if let Some(data) = thumbnail_service .get_cached_thumbnail( &id, diff --git a/src/interfaces/nextcloud/preview_handler.rs b/src/interfaces/nextcloud/preview_handler.rs index 8a32a0d6..385501b8 100644 --- a/src/interfaces/nextcloud/preview_handler.rs +++ b/src/interfaces/nextcloud/preview_handler.rs @@ -137,19 +137,40 @@ pub async fn handle_preview( } }; - // Conditional revalidation — the ETag is derived from (object id, size) - // only, so it is computable right here, BEFORE the blob-hash query and - // the thumbnail cache/disk read. NC clients revalidate gallery previews - // constantly; the REST thumbnail endpoint has honoured `If-None-Match` - // since PHOTOS-ETAG — this endpoint set an immutable ETag but never - // compared it, so every revalidation re-ran the whole pipeline and - // re-shipped the body (ROUND10). Authz already passed above; a 304 - // must never skip the Read check. + // Conditional revalidation. NC clients revalidate gallery previews + // constantly; this endpoint set an immutable ETag but never compared it, + // so every revalidation re-ran the whole pipeline and re-shipped the body + // (ROUND10). Authz already passed above; a 304 must never skip the Read + // check. + // + // Keyed on the CONTENT hash, matching the REST thumbnail endpoint. A + // thumbnail is a pure function of (source bytes, size), so that pair + // identifies the response and `immutable` below is honest. Keying on the + // object id meant replacing a file's content — which preserves the id — + // left every client showing the old preview for up to a year, since + // `immutable` suppresses revalidation entirely. + // + // This moves the blob-hash query ahead of the 304 rather than adding one: + // the same lookup used to sit just below, on the path that renders. + let blob_hash = match state + .repositories + .file_read_repository + .get_blob_hash(&object_id) + .await + { + Ok(hash) => hash, + Err(_) => { + return Response::builder() + .status(StatusCode::NOT_FOUND) + .body(Body::from("File blob not found")) + .unwrap(); + } + }; let etag = { let s = thumb_size.as_str(); - let mut e = String::with_capacity(9 + object_id.len() + s.len()); + let mut e = String::with_capacity(9 + blob_hash.len() + s.len()); e.push_str("\"thumb-"); - e.push_str(&object_id); + e.push_str(&blob_hash); e.push('-'); e.push_str(s); e.push('"'); @@ -179,21 +200,7 @@ pub async fn handle_preview( .unwrap(); } - // Resolve the blob hash (content-addressable storage) - let blob_hash = match state - .repositories - .file_read_repository - .get_blob_hash(&object_id) - .await - { - Ok(hash) => hash, - Err(_) => { - return Response::builder() - .status(StatusCode::NOT_FOUND) - .body(Body::from("File blob not found")) - .unwrap(); - } - }; + // `blob_hash` was resolved above to build the ETag. if let Some(data) = state .core .thumbnail_service diff --git a/tests/api/derived_blob_copy.hurl b/tests/api/derived_blob_copy.hurl index 00e96bc4..488c9fba 100644 --- a/tests/api/derived_blob_copy.hurl +++ b/tests/api/derived_blob_copy.hurl @@ -140,6 +140,7 @@ Authorization: Bearer {{token}} HTTP 200 [Captures] thumb_bytes: bytes +thumb_etag: header "ETag" # ───────────────────────────────────────────────────────────── @@ -173,10 +174,14 @@ jsonpath "$.ref_count" == 2 # ───────────────────────────────────────────────────────────── -# Step 7 – The copy is readable and renders the same bytes. +# Step 7 – The copy is readable, renders the same bytes, and carries the +# SAME ETag as the original. # -# NOT a proof of derived-blob sharing — see the header. This catches the -# copy being unreadable or resolving to different content. +# The ETag is keyed on the content hash, which the copy shares. Two +# different files agreeing on an ETag is the one externally visible +# consequence of content-keying — a file-id-keyed ETag could not produce +# it. The 304 below is the payoff: a client that already holds the +# original's thumbnail does not refetch it for the copy. # ───────────────────────────────────────────────────────────── GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview Authorization: Bearer {{token}} @@ -184,6 +189,16 @@ Authorization: Bearer {{token}} HTTP 200 [Asserts] bytes == {{thumb_bytes}} +header "ETag" == "{{thumb_etag}}" + + +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{thumb_etag}} + +HTTP 304 +[Asserts] +header "ETag" == "{{thumb_etag}}" # ───────────────────────────────────────────────────────────── diff --git a/tests/api/run.sh b/tests/api/run.sh index 67619aec..d644513b 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -167,6 +167,7 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/batch_folder_copy.hurl" \ "$API_DIR/dedup_blob_cleanup.hurl" \ "$API_DIR/derived_blob_copy.hurl" \ + "$API_DIR/thumbnail_etag_content_keyed.hurl" \ "$API_DIR/dedup_admin_gate.hurl" \ "$API_DIR/admin_jobs.hurl" \ "$API_DIR/recoverable_jobs.hurl" \ diff --git a/tests/api/thumbnail_etag_content_keyed.hurl b/tests/api/thumbnail_etag_content_keyed.hurl new file mode 100644 index 00000000..b37b6d26 --- /dev/null +++ b/tests/api/thumbnail_etag_content_keyed.hurl @@ -0,0 +1,161 @@ +# ============================================================= +# OxiCloud – Thumbnail ETag is keyed on CONTENT, not on file id +# ============================================================= +# Regression guard for a stale-cache bug. +# +# The thumbnail ETag used to be `"thumb-{file_id}-{size}-{format}"`, sent +# with `Cache-Control: public, max-age=31536000, immutable`. Replacing a +# file's content preserves its id — the upload service rebuilds the entity +# with `parts.id` and a new hash, then fires `on_file_updated`, which +# regenerates the thumbnails — so the server produced a NEW thumbnail while +# advertising the OLD ETag. And `immutable` tells a conforming browser not +# to revalidate at all inside the freshness window, so clients kept showing +# the previous image for up to a year with no way to invalidate it. +# +# Keying on the content hash fixes it: new bytes → new hash → new ETag. +# +# This file asserts the invalidation direction. The sharing direction (two +# distinct files with identical content answering with the SAME ETag, so a +# copy revalidates to 304) is covered in `derived_blob_copy.hurl`. +# +# Overwrite goes through WebDAV PUT because that is the path that replaces +# content in place; the REST upload endpoint creates a new file instead. +# +# Prerequisites: setup.hurl must have run (admin user exists). +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Upload the first image +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +file: file,fixtures/red-image.png; image/png + +HTTP 201 +[Captures] +file_id: jsonpath "$.id" +file_name: jsonpath "$.name" +hash_before: jsonpath "$.content_hash" + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Its thumbnail, and the ETag that goes with it +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +etag_before: header "ETag" +thumb_before: bytes +[Asserts] +header "Cache-Control" contains "immutable" + + +# Unchanged content revalidates to 304 — the caching path works. +GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{etag_before}} + +HTTP 304 + + +# ───────────────────────────────────────────────────────────── +# Step 4 – Replace the content in place, keeping the same file id. +# ───────────────────────────────────────────────────────────── +PUT {{base_url}}/webdav/{{file_name}} +Authorization: Bearer {{token}} +Content-Type: image/png +file,fixtures/green-image.png; + +HTTP * +[Asserts] +status >= 200 +status < 300 + + +# Same file row, different content. +GET {{base_url}}/api/files/{{file_id}} +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +hash_after: jsonpath "$.content_hash" +[Asserts] +jsonpath "$.id" == "{{file_id}}" +jsonpath "$.content_hash" != "{{hash_before}}" + + +# ───────────────────────────────────────────────────────────── +# Step 5 – The ETag must have changed with the content. +# +# This is the assertion the file exists for. With the id-keyed ETag it was +# byte-identical to `etag_before`, and the next request would have been +# answered 304 from cache — serving the OLD image indefinitely. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +etag_after: header "ETag" +[Asserts] +header "ETag" != "{{etag_before}}" + + +# A client holding the stale ETag must be told to refetch, not given a 304. +GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{etag_before}} + +HTTP 200 +[Asserts] +header "ETag" == "{{etag_after}}" + + +# ...and the new ETag revalidates normally. +GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{etag_after}} + +HTTP 304 + + +# ───────────────────────────────────────────────────────────── +# Step 6 – Teardown. Hurl files share one database within run.sh. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/files/{{file_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_id: jsonpath "$.items[?(@.resource.id == '{{file_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_id}} +Authorization: Bearer {{token}} + +HTTP 200 From ec61ce77f8bc9fbd8325242a8ea2635d86e56165 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 23:52:36 +0200 Subject: [PATCH 12/66] docs(plan): ETag moves to the derived hash at the read-order flip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format), which is one term short: a thumbnail is a function of those PLUS the renderer. Change the encoder or a quality setting and identical inputs produce different bytes under an unchanged ETag — the same staleness class the commit fixed, one level down. It bites when an already-cached thumbnail is re-rendered after a renderer change. Keying on the derived blob's own hash removes the term entirely: the ETag IS the hash of the bytes, so any output change invalidates by construction. It is self-consistent for free, because store_derived_blob is ON CONFLICT DO NOTHING — a re-render never displaces the stored row, so the ETag always equals what the derived tier will serve. No renderer version constant to remember to bump. Records why it cannot land yet. The derived tier is read LAST by design, so an ETag naming the derived hash would describe a tier the response probably did not come from; sidecar and derived agree at creation but diverge if a sidecar is re-rendered while the derived row stays pinned by DO NOTHING. An ETag that lies about the body is worse than one that is merely coarse. Also the tier is WebP-only (variant is the size, with no format term) and empty for anything predating this work until derived_import backfills. So it lands at step 10 with the flip, keeping today's form as the fallback for ungenerated variants and formats the tier does not hold. The LEFT JOIN already planned for the read path returns the derived hash in the same query, so it costs no extra round-trip. --- docs/plan/derived-blobs.md | 48 +++++++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 149915b9..17c0b65c 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -949,6 +949,48 @@ the request. Read order: backend stack, which is where the disk cache lives. 4. Generate only if step 2 found no row. +### HTTP ETag — move to the derived hash when the order flips + +Shipped ahead of this plan (2026-08-24): the thumbnail ETag is +`"thumb-{source_hash}-{size}-{format}"` on both the REST and the +NextCloud preview endpoint. It replaced a `file_id`-keyed ETag that, +combined with `Cache-Control: immutable`, meant replacing a file's +content never invalidated the client's copy — `file_id` survives the +replacement, so the ETag did too, for a year. + +**That key is still one term short.** A thumbnail is a function of +`(source bytes, size, format, RENDERER)`. Change the encoder, a quality +setting, or EXIF-rotation handling, and identical inputs produce +different bytes under an unchanged ETag — the same staleness class, one +level down. It bites when an already-cached thumbnail is re-rendered +after a renderer change (sidecar evicted, regenerated on miss). + +**The fix is to key on the derived blob's own hash** — the ETag then +*is* the hash of the bytes served, so any change in output invalidates +by construction, with no version constant to remember to bump. It is +self-consistent for free: `store_derived_blob` is +`ON CONFLICT DO NOTHING`, so a re-render never displaces the stored +row, and the ETag therefore always equals what the derived tier will +serve. + +**It must land with the read-order flip, not before.** Today the +derived tier is deliberately read LAST, so an ETag naming the derived +hash would describe a tier the response probably did not come from. +The two agree at creation — `render_and_persist_all_webp` writes both +from the same bytes — but diverge if a sidecar is re-rendered while the +derived row stays pinned by `DO NOTHING`. Sidecar is served, ETag +describes the other one: an ETag that lies about the body is worse than +one that is merely coarse. Two further reasons it has to wait: the +derived tier is WebP-only (`variant = size.dir_name()`, no format in +the key), so non-WebP clients have no row to key on; and nothing +predating this work has a row until `derived_import` backfills. + +So at step 10, alongside the flip: ETag becomes the derived hash, with +the current `source_hash` form kept as the fallback for a variant not +yet generated and for formats the derived tier does not hold. The +`LEFT JOIN` in step 2 above already returns the derived hash in the +same query, so the ETag costs no extra round-trip. + **The disk cache is `CachedBlobBackend`, reused unchanged.** No thumbnail-specific cache, no second root path. Routing derived blobs through the same stack gets, for free: @@ -1171,7 +1213,11 @@ hardcoded SQL). New sources bolt on independently. declare its version semantics. 10. **`derived_import` job + the dual-read fallback** — see the migration section. Phase 3 (deleting the fallback and the sidecar - dirs) is a separate later release, gated on an empty tail. + dirs) is a separate later release, gated on an empty tail. **The + HTTP ETag moves to the derived hash here**, with the read-order + flip and not before — see "HTTP ETag" under *Read path and + caching* for why keying it earlier would make the ETag describe a + tier the response did not come from. 11. **`DedupService` → `BlobHandler` rename** — decided, mechanical, 34 files. Standalone commit, `src/AGENTS.md` updated with it. Can land at any point; last is easiest, since every earlier slice From 19a8186c666ea2acd591bce217ff3ac7eeed08da Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 00:01:19 +0200 Subject: [PATCH 13/66] test(api): upload into an explicit folder in the ETag test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The upload omitted folder_id, which the handler needs to resolve the file's owner — it answers 500, not a root upload. Every other upload in the suite passes it; this was the only one that did not, which is why nothing caught it earlier. The folder also gives the WebDAV overwrite a deterministic path (/webdav/hurl-etag-src/) instead of depending on where a folder-less upload would have landed. Teardown now removes it and purges it from trash, keeping the shared database clean for the files that run after. --- tests/api/thumbnail_etag_content_keyed.hurl | 44 +++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/tests/api/thumbnail_etag_content_keyed.hurl b/tests/api/thumbnail_etag_content_keyed.hurl index b37b6d26..cf8908e9 100644 --- a/tests/api/thumbnail_etag_content_keyed.hurl +++ b/tests/api/thumbnail_etag_content_keyed.hurl @@ -41,11 +41,28 @@ token: jsonpath "$.access_token" # ───────────────────────────────────────────────────────────── -# Step 2 – Upload the first image +# Step 2 – Upload the first image into a folder of its own. +# +# `folder_id` is required — the upload path resolves the owner from the +# destination folder, so omitting it is a 500, not a root upload. The +# folder also gives the WebDAV overwrite in step 4 a deterministic path. # ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-etag-src" +} + +HTTP 201 +[Captures] +folder_id: jsonpath "$.id" + + POST {{base_url}}/api/files/upload Authorization: Bearer {{token}} [MultipartFormData] +folder_id: {{folder_id}} file: file,fixtures/red-image.png; image/png HTTP 201 @@ -80,7 +97,7 @@ HTTP 304 # ───────────────────────────────────────────────────────────── # Step 4 – Replace the content in place, keeping the same file id. # ───────────────────────────────────────────────────────────── -PUT {{base_url}}/webdav/{{file_name}} +PUT {{base_url}}/webdav/hurl-etag-src/{{file_name}} Authorization: Bearer {{token}} Content-Type: image/png file,fixtures/green-image.png; @@ -139,7 +156,8 @@ HTTP 304 # ───────────────────────────────────────────────────────────── -# Step 6 – Teardown. Hurl files share one database within run.sh. +# Step 6 – Teardown. Hurl files share one database within run.sh, so the +# folder goes too, and both leave trash empty behind them. # ───────────────────────────────────────────────────────────── DELETE {{base_url}}/api/files/{{file_id}} Authorization: Bearer {{token}} @@ -159,3 +177,23 @@ DELETE {{base_url}}/api/trash/{{trash_id}} Authorization: Bearer {{token}} HTTP 200 + + +DELETE {{base_url}}/api/folders/{{folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_folder_id: jsonpath "$.items[?(@.resource.id == '{{folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_folder_id}} +Authorization: Bearer {{token}} + +HTTP 200 From 2c0ba37290d5578db209925038b07c769bc349d5 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 00:07:09 +0200 Subject: [PATCH 14/66] test(api): read the DTO from the folder listing, not the download route MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GET /api/files/{id} is the download route — it returned the PNG bytes, so the jsonpath capture failed on a UTF-8 decode. /{id}/metadata is the EXIF endpoint and carries no FileDto either. Listing the folder gives the DTO, and since the folder holds exactly this one file, count == 1 also proves the WebDAV PUT overwrote in place rather than creating a second file beside it. Also drops an unused bytes capture and records why the body is not asserted after the overwrite: the moka tier is keyed on file_id and invalidated from the spawned task in on_file_updated, so a request landing first sees the previous bytes under the new ETag. Asserting on bytes would be a race. --- tests/api/thumbnail_etag_content_keyed.hurl | 22 ++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/tests/api/thumbnail_etag_content_keyed.hurl b/tests/api/thumbnail_etag_content_keyed.hurl index cf8908e9..805b1438 100644 --- a/tests/api/thumbnail_etag_content_keyed.hurl +++ b/tests/api/thumbnail_etag_content_keyed.hurl @@ -81,7 +81,6 @@ Authorization: Bearer {{token}} HTTP 200 [Captures] etag_before: header "ETag" -thumb_before: bytes [Asserts] header "Cache-Control" contains "immutable" @@ -109,15 +108,22 @@ status < 300 # Same file row, different content. -GET {{base_url}}/api/files/{{file_id}} +# +# Listed rather than fetched by id: `/api/files/{id}` is the DOWNLOAD +# route (it returns the image bytes) and `/{id}/metadata` is the EXIF +# endpoint — neither carries the FileDto. The folder holds exactly this +# one file, so `count == 1` also proves the PUT overwrote in place +# instead of creating a second file beside it. +GET {{base_url}}/api/files?folder_id={{folder_id}} Authorization: Bearer {{token}} HTTP 200 [Captures] -hash_after: jsonpath "$.content_hash" +hash_after: jsonpath "$[0].content_hash" [Asserts] -jsonpath "$.id" == "{{file_id}}" -jsonpath "$.content_hash" != "{{hash_before}}" +jsonpath "$" count == 1 +jsonpath "$[0].id" == "{{file_id}}" +jsonpath "$[0].content_hash" != "{{hash_before}}" # ───────────────────────────────────────────────────────────── @@ -126,6 +132,12 @@ jsonpath "$.content_hash" != "{{hash_before}}" # This is the assertion the file exists for. With the id-keyed ETag it was # byte-identical to `etag_before`, and the next request would have been # answered 304 from cache — serving the OLD image indefinitely. +# +# Deliberately asserts the ETag only, not that the BODY changed. The moka +# tier is still keyed on file_id and is invalidated from the spawned task +# in `on_file_updated`, so a request landing before that task runs gets the +# previous bytes under the new ETag. Asserting on bytes here would be a +# race; the incoherence itself is tracked separately. # ───────────────────────────────────────────────────────────── GET {{base_url}}/api/files/{{file_id}}/thumbnail/preview Authorization: Bearer {{token}} From cf21ee2f8fd960b0f19807ba8daabe6688491e90 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 05:56:21 +0200 Subject: [PATCH 15/66] fix(thumbnails): key the moka tier on content, closing the ETag race MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fe9c4f49 made the ETag content-keyed, but the RAM tier was still keyed on file_id, so the two disagreed about what identifies a thumbnail. Replacing a file's content preserves its id, so the moka entry stayed reachable while the ETag had already changed — and invalidation runs from the spawned task in on_file_updated. A request landing in that window got the NEW ETag over the OLD bytes, and because the response is immutable with a one-year max-age, the client cached those stale bytes permanently. The bug fe9c4f49 set out to fix, arriving through a different door. Keying on the hash removes the window rather than narrowing it: new content is a different key, so the old entry cannot be hit. Correctness no longer depends on the invalidation task winning a race against the next request. This also aligns the RAM tier with what disk already did — sidecars have always been written to get_thumbnail_path(blob_hash, ...). The tier that had the bug was the one keyed differently from every other. Two further consequences: N copies of one photo now share a single entry instead of occupying N for identical bytes, and delete_thumbnails shrinks to the external entries, which are the only genuinely per-file artifacts. Video frames stay file-keyed under an `ext-{file_id}` id — they are per-file by nature. The namespaces cannot collide: hashes are 64 hex characters. get_cached_thumbnail takes blob_hash as an Option, and a caller without one now skips the RAM tier and falls through to disk rather than consulting a file-id key. That is correct, not merely tolerable — a file-id key is the stale entry this change exists to prevent. Both HTTP handlers resolve the hash to build the ETag, so only internal callers that never had one are affected. --- .../services/thumbnail_service.rs | 189 ++++++++++-------- 1 file changed, 110 insertions(+), 79 deletions(-) diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 52fee4af..3da76cbe 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -69,15 +69,60 @@ impl ThumbnailSize { } } -/// Cache key for thumbnails. Includes `format` so WebP and the JPEG fallback for -/// the same (file_id, size) are distinct entries (no cross-format collision). +/// Cache key for the in-RAM thumbnail tier (moka). Includes `format` so WebP +/// and the JPEG fallback for the same (content, size) are distinct entries +/// (no cross-format collision). +/// +/// Not to be confused with the two other caches on this path: the sidecar +/// files under `thumbnails_root` (already blob-hash keyed), and +/// `CachedBlobBackend`, the on-disk LRU in front of a remote blob backend +/// that only comes into play when a derived blob is read through the dedup +/// stack. #[derive(Debug, Clone, PartialEq, Eq, Hash)] struct ThumbnailCacheKey { - file_id: String, + /// Content hash for rendered thumbnails; `ext-{file_id}` for + /// client-uploaded video frames, which really are per-file. + /// + /// Keying on the hash rather than the file id is what makes the tier + /// coherent with the HTTP ETag. When a file's content is replaced its + /// id survives, so a file-keyed entry stayed valid-looking and had to + /// be invalidated explicitly — which `on_file_updated` does from a + /// spawned task, leaving a window where the response carried the NEW + /// ETag over the OLD bytes. Since the response is `immutable` with a + /// one-year max-age, a client landing in that window cached stale + /// bytes permanently. Content keying removes the window rather than + /// narrowing it: new content is a different key, so it cannot hit. + /// + /// It also stops N copies of one photo occupying N entries for + /// identical bytes. + /// + /// The two namespaces cannot collide: hashes are 64 hex characters, + /// and the external form carries an `ext-` prefix and a UUID. + id: String, size: ThumbnailSize, format: ThumbnailFormat, } +impl ThumbnailCacheKey { + /// Rendered thumbnail — keyed by the source content hash. + fn content(hash: &str, size: ThumbnailSize, format: ThumbnailFormat) -> Self { + Self { + id: hash.to_string(), + size, + format, + } + } + + /// Client-uploaded video frame — genuinely per-file, always JPEG. + fn external(file_id: &str, size: ThumbnailSize) -> Self { + Self { + id: format!("ext-{file_id}"), + size, + format: ThumbnailFormat::Jpeg, + } + } +} + /// Maximum pixel count before rejecting decode (50 megapixels → ~200 MB RGBA). /// Images above this are silently skipped — protects against single-image OOM. const MAX_DECODE_PIXELS: u64 = 50_000_000; @@ -234,11 +279,7 @@ impl ThumbnailService { format: ThumbnailFormat, original_path: &Path, ) -> Result { - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format, - }; + let cache_key = ThumbnailCacheKey::content(blob_hash, size, format); let thumb_path = self.get_thumbnail_path(blob_hash, size, format); let original_owned = original_path.to_path_buf(); @@ -312,11 +353,7 @@ impl ThumbnailService { format: ThumbnailFormat, original_data: Bytes, ) -> Result { - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format, - }; + let cache_key = ThumbnailCacheKey::content(blob_hash, size, format); let thumb_path = self.get_thumbnail_path(blob_hash, size, format); let file_id_owned = file_id.to_string(); @@ -369,11 +406,7 @@ impl ThumbnailService { format: ThumbnailFormat, dedup: Arc, ) -> Result { - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format, - }; + let cache_key = ThumbnailCacheKey::content(blob_hash, size, format); let thumb_path = self.get_thumbnail_path(blob_hash, size, format); let file_id_owned = file_id.to_string(); @@ -484,13 +517,19 @@ impl ThumbnailService { // today's behaviour, which is what the port impl wants. dedup: Option<&DedupService>, ) -> Option { - // 1. Check in-memory cache - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format, - }; - if let Some(bytes) = self.cache.get(&cache_key).await + // 1. Check in-memory cache. + // + // Keyed by content, so a caller that did not resolve the hash cannot + // consult this tier — it falls through to disk, which is correct + // rather than merely acceptable: a file-id key would be the stale + // entry the content key exists to avoid. Every HTTP path passes the + // hash (both handlers resolve it to build the ETag), so the fall + // through is confined to internal callers that never had one. + if let Some(hash) = blob_hash + && let Some(bytes) = self + .cache + .get(&ThumbnailCacheKey::content(hash, size, format)) + .await && !bytes.is_empty() { return Some(bytes); @@ -509,11 +548,7 @@ impl ThumbnailService { // key's format must describe them. Inserting under `cache_key` (whose // format is the *requested* format, possibly Webp) would store JPEG // bytes behind a Webp key — a latent cross-format invariant violation. - let ext_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format: ThumbnailFormat::Jpeg, - }; + let ext_key = ThumbnailCacheKey::external(file_id, size); self.cache.insert(ext_key, bytes.clone()).await; return Some(bytes); } @@ -524,7 +559,12 @@ impl ThumbnailService { if let Ok(data) = fs::read(&thumb_path).await { let bytes = Bytes::from(data); // Populate in-memory cache for next hit - self.cache.insert(cache_key, bytes.clone()).await; + self.cache + .insert( + ThumbnailCacheKey::content(hash, size, format), + bytes.clone(), + ) + .await; return Some(bytes); } @@ -559,7 +599,12 @@ impl ThumbnailService { } } let bytes = Bytes::from(buf); - self.cache.insert(cache_key, bytes.clone()).await; + self.cache + .insert( + ThumbnailCacheKey::content(hash, size, format), + bytes.clone(), + ) + .await; Some(bytes) } @@ -643,11 +688,7 @@ impl ThumbnailService { .map_err(|e| ThumbnailError::IoError(e.to_string()))?; // Populate in-memory cache (external thumbnails are JPEG) - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format: ThumbnailFormat::Jpeg, - }; + let cache_key = ThumbnailCacheKey::external(file_id, size); self.cache.insert(cache_key, bytes.clone()).await; tracing::info!("✅ Stored external thumbnail: {} {:?}", file_id, size); @@ -989,11 +1030,8 @@ impl ThumbnailService { let thumb_path = self.get_thumbnail_path(&blob_hash, *size, ThumbnailFormat::Webp); if let Ok(data) = fs::read(&thumb_path).await { - let cache_key = ThumbnailCacheKey { - file_id: file_id.clone(), - size: *size, - format: ThumbnailFormat::Webp, - }; + let cache_key = + ThumbnailCacheKey::content(&blob_hash, *size, ThumbnailFormat::Webp); self.cache.insert(cache_key, Bytes::from(data)).await; } } @@ -1043,8 +1081,8 @@ impl ThumbnailService { } }; - // Save each size to disk (keyed by blob_hash for dedup) - // AND populate moka (keyed by file_id for fast serving). + // Save each size to disk and populate moka — both keyed by + // blob_hash, so the two tiers agree and a copy shares them. for (size, bytes) in thumbnails { let thumb_path = self.get_thumbnail_path(&blob_hash, size, ThumbnailFormat::Webp); if let Some(parent) = thumb_path.parent() { @@ -1054,11 +1092,8 @@ impl ThumbnailService { tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e); } else { // Populate in-memory cache for instant first-hit serving - let cache_key = ThumbnailCacheKey { - file_id: file_id.clone(), - size, - format: ThumbnailFormat::Webp, - }; + let cache_key = + ThumbnailCacheKey::content(&blob_hash, size, ThumbnailFormat::Webp); self.cache.insert(cache_key, bytes).await; tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size); } @@ -1111,11 +1146,8 @@ impl ThumbnailService { let thumb_path = self.get_thumbnail_path(&blob_hash, *size, ThumbnailFormat::Webp); if let Ok(data) = fs::read(&thumb_path).await { - let cache_key = ThumbnailCacheKey { - file_id: file_id.clone(), - size: *size, - format: ThumbnailFormat::Webp, - }; + let cache_key = + ThumbnailCacheKey::content(&blob_hash, *size, ThumbnailFormat::Webp); self.cache.insert(cache_key, Bytes::from(data)).await; } } @@ -1228,11 +1260,7 @@ impl ThumbnailService { ); } - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size, - format: ThumbnailFormat::Webp, - }; + let cache_key = ThumbnailCacheKey::content(blob_hash, size, ThumbnailFormat::Webp); self.cache.insert(cache_key, bytes).await; tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size); } @@ -1276,11 +1304,8 @@ impl ThumbnailService { for size in ThumbnailSize::all() { let p = self.get_thumbnail_path(&blob_hash, *size, ThumbnailFormat::Webp); if let Ok(data) = fs::read(&p).await { - let key = ThumbnailCacheKey { - file_id: file_id.clone(), - size: *size, - format: ThumbnailFormat::Webp, - }; + let key = + ThumbnailCacheKey::content(&blob_hash, *size, ThumbnailFormat::Webp); self.cache.insert(key, Bytes::from(data)).await; } } @@ -1379,24 +1404,30 @@ impl ThumbnailService { Ok(tmp) } - /// Delete thumbnails for a file. + /// Delete the per-file thumbnail artifacts for a file. /// - /// Only invalidates the in-memory moka cache (keyed by file_id). - /// Disk thumbnails are keyed by blob_hash and may be shared by - /// other files with the same content — they are cleaned up via - /// `delete_blob_thumbnails` when the blob is garbage-collected. - /// Also removes any external (video-frame) thumbnails stored by file_id. + /// Only the external (video-frame) entries are file-keyed, so only those + /// are removed — from moka and from disk. Rendered thumbnails, in both + /// tiers, are keyed by blob_hash and may be shared with any other file + /// holding the same content; they are reclaimed by + /// `delete_blob_thumbnails` when the blob itself is garbage-collected. + /// + /// Content keying is also why this no longer has to win a race. When a + /// file's content is replaced the rendered entries are unreachable by + /// construction — a new hash is a new key — rather than needing explicit + /// invalidation before the next request arrives. pub async fn delete_thumbnails(&self, file_id: &str) -> Result<(), ThumbnailError> { for size in ThumbnailSize::all() { - // Remove from moka cache (lock-free invalidation) — both codecs. - for format in [ThumbnailFormat::Webp, ThumbnailFormat::Jpeg] { - let cache_key = ThumbnailCacheKey { - file_id: file_id.to_string(), - size: *size, - format, - }; - self.cache.invalidate(&cache_key).await; - } + // Only the external (per-file) entry needs invalidating. Rendered + // thumbnails are keyed by content hash, so replacing a file's + // content yields a different key and the old entry simply cannot + // be hit again — which is the point: correctness no longer depends + // on this call winning a race against the next request. And on + // deletion the entry stays valid for any other file sharing that + // content, so dropping it would only cost a re-read. + self.cache + .invalidate(&ThumbnailCacheKey::external(file_id, *size)) + .await; // Remove external (video-frame) thumbnails stored by file_id (JPEG-only) let ext_path = self From fac82fea2353b288c4fd364c169c058c5e29321c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 07:21:32 +0200 Subject: [PATCH 16/66] feat(storage): add storage.file_attached_blobs, the file-keyed half MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 9 of docs/plan/derived-blobs.md. content_derived_blobs holds bytes that are a pure function of a file's content, so they are keyed by that content and shared by every file holding it. This table holds the opposite: bytes a user supplied or chose, which must never be shared across files. The key is what enforces it. That difference is a security boundary, not a modelling preference. A content-keyed client preview would let user A upload a file plus a preview that misrepresents it; when user B later uploads the same bytes, dedup matches and B is served A's preview. Content-keying is only safe when the server can derive the bytes — there is nothing to poison, because the same input yields the same output for everyone. Required now rather than deferred: the SPA already generates and PUTs previews for PDFs, and there is no server-side regeneration path for them, so the sidecar migration has nowhere else to put those bytes. uploaded_by is NOT NULL with no foreign key, per the provenance convention rather than the plan's sketch. A FK with ON DELETE SET NULL discards the audit trail exactly when it matters, and without an ON DELETE clause it would block deleting a user outright. Deleting the uploader must not rewrite history. FileAttachedReferenceSource is registered in built_in_registry before anything writes to the table, so dedup_gc's reap predicate already knows it exists — otherwise the first sweep after the first attachment would delete it. Manifest level only, like the derived source: these blobs are almost always single-chunk, so contributing at chunk level would double-count against the aliased hash. copy_file_satellites gains one arm: attachments are DUPLICATED, since the key is file_id and the copy is a different file, with uploaded_by carried over — the person who supplied the bytes did not change because someone copied the file. Each duplicate takes its own reference, so the bytes stay deduplicated while the mapping does not. Both golden SQL tests updated: the new fragment lands inside the reap predicate's NOT(...) group and as a summed term in the manifest recompute. Verified on a scratch PG with every migration applied — the attachment duplicates to 2 rows holding 2 references with provenance intact, while the content-keyed thumbnail stays 1 row reachable from both files. --- .../20261020000000_file_attached_blobs.sql | 157 ++++++++++++++++++ .../repositories/pg/blob_reference_sources.rs | 119 ++++++++++++- src/infrastructure/services/dedup_service.rs | 3 +- .../services/manifests_consistency_service.rs | 3 +- 4 files changed, 279 insertions(+), 3 deletions(-) create mode 100644 migrations/20261020000000_file_attached_blobs.sql diff --git a/migrations/20261020000000_file_attached_blobs.sql b/migrations/20261020000000_file_attached_blobs.sql new file mode 100644 index 00000000..d8d1b346 --- /dev/null +++ b/migrations/20261020000000_file_attached_blobs.sql @@ -0,0 +1,157 @@ +-- Step 9 of `docs/plan/derived-blobs.md` — the file-keyed half of the pair. +-- +-- `content_derived_blobs` holds bytes that are a pure deterministic function +-- of a file's content, so they are keyed by that content and shared across +-- every file holding it. This table holds the opposite: bytes a USER supplied +-- or chose. Those must never be shared across files, and the key is what +-- enforces it. +-- +-- The distinction is a security boundary, not a modelling preference. If a +-- client-uploaded preview were content-keyed, user A could upload a file plus +-- a preview that misrepresents it; when user B later uploads the same bytes, +-- dedup would match and B would be served A's preview. Content-keying is only +-- safe when the bytes are derivable from the content by the server — nothing +-- to poison, because anyone with the same input gets the same output. +-- +-- Required now rather than deferred: the SPA already generates and PUTs +-- previews for PDFs, and there is no server-side regeneration path for them, +-- so the sidecar migration has nowhere else to put those bytes. + +CREATE TABLE IF NOT EXISTS storage.file_attached_blobs ( + file_id UUID NOT NULL REFERENCES storage.files(id) ON DELETE CASCADE, + kind TEXT NOT NULL CHECK (kind IN ('preview', 'subtitle', 'cover_art')), + variant TEXT NOT NULL, + blob_hash VARCHAR(64) NOT NULL, + content_type TEXT NOT NULL, + -- Provenance convention: NOT NULL and NO foreign key. A FK with + -- ON DELETE SET NULL loses the audit trail exactly when it matters most, + -- and without an ON DELETE clause it would block deleting a user + -- outright. Deleting the uploader must not rewrite history, so the id is + -- retained even once it no longer resolves. Rows imported with no known + -- uploader carry the all-zeros sentinel. + uploaded_by UUID NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (file_id, kind, variant) +); + +-- Reverse lookup for the reference recompute and for dedup_gc's reap +-- predicate: both ask "does any row reference this blob?". +CREATE INDEX IF NOT EXISTS idx_file_attached_blobs_blob_hash + ON storage.file_attached_blobs (blob_hash); + +-- ── The routing rule, recorded on both tables ──────────────────────────── +-- Choosing the wrong table is a silent poisoning bug rather than a compile +-- error, so the rule lives where an implementor will actually meet it. + +COMMENT ON TABLE storage.file_attached_blobs IS + 'User-supplied or user-chosen artifacts (client previews, subtitles, cover art) keyed by FILE. File-keyed on purpose: these bytes are not derivable from the file''s content, so sharing them across files with identical content would let one user''s upload be served for another user''s file. Server-derived bytes must NOT be stored here — see docs/plan/derived-blobs.md.'; + +COMMENT ON COLUMN storage.file_attached_blobs.file_id IS + 'The file these bytes are attached to. ON DELETE CASCADE: the attachment has no meaning without it. Deleting the row does NOT release the blob reference — the owning service does that in its on_file_deleted hook.'; + +COMMENT ON COLUMN storage.file_attached_blobs.blob_hash IS + 'The attached Blob. Reference HOLDER — bumps chunk_manifests.ref_count via DedupService::add_reference. Dedup still applies to the bytes themselves; what is forbidden is sharing the MAPPING across files.'; + +COMMENT ON COLUMN storage.file_attached_blobs.variant IS + 'Opaque discriminator within a kind (preview | en | fr | cover...). New axes go inside this string, never into new columns.'; + +COMMENT ON COLUMN storage.file_attached_blobs.uploaded_by IS + 'Who supplied these bytes. Retained after the user is deleted — deleting a user must not rewrite provenance. The only trace that an Editor on a shared file replaced the owner''s preview.'; + +COMMENT ON TABLE storage.content_derived_blobs IS + 'Server-derived artifacts (thumbnails, transcodes) keyed by the BLAKE3 of their SOURCE content. Content-keyed on purpose: identical content shares one derivation. ROUTING RULE — bytes that are a pure deterministic function of the file''s content belong here; bytes that are user-supplied or user-chosen belong in storage.file_attached_blobs, which is file-keyed and never shared. See docs/plan/derived-blobs.md.'; + +-- ── Teach the copy fan-out about it ────────────────────────────────────── +-- +-- Only the attached-blobs arm is new versus `20261019000000`; everything +-- else is that definition verbatim. Adding a file-keyed table is now one +-- edit in one function, which is the whole point of having consolidated the +-- two copy paths first. +CREATE OR REPLACE FUNCTION storage.copy_file_satellites( + p_old_ids UUID[], + p_new_ids UUID[] +) RETURNS void AS $$ +DECLARE + v_unmatched TEXT[]; +BEGIN + IF p_old_ids IS NULL OR cardinality(p_old_ids) = 0 THEN + RETURN; + END IF; + + IF p_new_ids IS NULL OR cardinality(p_old_ids) <> cardinality(p_new_ids) THEN + -- Positional correspondence is the whole interface; a length + -- mismatch would silently attach satellites to the wrong file. + RAISE EXCEPTION + 'copy_file_satellites: id arrays must correspond positionally (% old vs % new)', + cardinality(p_old_ids), COALESCE(cardinality(p_new_ids), 0); + END IF; + + -- 1. WebDAV dead properties. RFC 4918 §8.8 requires COPY to duplicate + -- them: properties describe the resource, and the copy is a resource. + INSERT INTO storage.webdav_dead_properties + (file_id, namespace, local_name, value) + SELECT m.new_id, dp.namespace, dp.local_name, dp.value + FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id) + JOIN storage.webdav_dead_properties dp ON dp.file_id = m.old_id; + + -- 2. A reference on the copied content, so deleting the original cannot + -- reap bytes the copy still needs. Read from the NEW rows rather than + -- the old ones: that is what makes an unreferenceable copy impossible + -- to create, since a row that failed to insert contributes nothing. + SELECT storage.add_blob_references(array_agg(f.blob_hash)) + INTO v_unmatched + FROM unnest(p_new_ids) AS n(id) + JOIN storage.files f ON f.id = n.id + WHERE NOT f.is_trashed; + + IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN + -- Warn, do not abort. A missing registry row means the SOURCE file + -- was already broken; the copy merely inherits it. Failing here + -- would abort an entire folder copy over one pre-existing fault, + -- which is worse than completing it and reporting. The blob-level + -- audit jobs are what surface the underlying breakage. + RAISE WARNING + 'copy_file_satellites: % copied file(s) reference a blob with no registry row (first: %); source was already broken', + cardinality(v_unmatched), v_unmatched[1]; + END IF; + + -- 3. File-keyed attachments — client previews, subtitles, cover art. + -- DUPLICATED rather than shared, because the key is `file_id` and the + -- copy is a different file. `uploaded_by` carries over: the person + -- who supplied the bytes did not change because someone copied the + -- file, and rewriting it to the copier would forge provenance. + -- + -- Each duplicated row is a new reference on the same blob, so the + -- bytes are still deduplicated — it is the MAPPING that must not be + -- shared, not the content. + WITH copied AS ( + INSERT INTO storage.file_attached_blobs + (file_id, kind, variant, blob_hash, content_type, uploaded_by) + SELECT m.new_id, a.kind, a.variant, a.blob_hash, a.content_type, a.uploaded_by + FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id) + JOIN storage.file_attached_blobs a ON a.file_id = m.old_id + RETURNING blob_hash + ) + SELECT storage.add_blob_references(array_agg(blob_hash)) + INTO v_unmatched + FROM copied; + + IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN + RAISE WARNING + 'copy_file_satellites: % attached blob(s) reference no registry row (first: %); source was already broken', + cardinality(v_unmatched), v_unmatched[1]; + END IF; + + -- ── Deliberately absent ────────────────────────────────────────────── + -- + -- storage.comments (future): NOT copied. A copy is a new artifact; the + -- discussion belongs to the original. + -- + -- content_derived_blobs, blob_extracted_text, faces.faces: content-keyed. + -- The copy shares the source's hash, so it already sees them — copying + -- would duplicate rows that are keyed on the very thing being shared. + -- + -- storage.favorites, recent_items, shares: properties of the ORIGINAL's + -- relationship to users, not of its content. +END; +$$ LANGUAGE plpgsql; diff --git a/src/infrastructure/repositories/pg/blob_reference_sources.rs b/src/infrastructure/repositories/pg/blob_reference_sources.rs index ea3c1ac1..d273d2b2 100644 --- a/src/infrastructure/repositories/pg/blob_reference_sources.rs +++ b/src/infrastructure/repositories/pg/blob_reference_sources.rs @@ -29,6 +29,7 @@ use crate::domain::errors::DomainError; const FILES_ALIAS: &str = "cnt_f"; const MANIFEST_ALIAS: &str = "cnt_m"; const DERIVED_ALIAS: &str = "cnt_d"; +const ATTACHED_ALIAS: &str = "cnt_a"; /// Fragment for [`FilesReferenceSource`], as a free function so the SQL /// shape can be tested without constructing a pool — it is a property of @@ -117,6 +118,33 @@ fn content_derived_exists_sql(level: RefLevel, outer_hash_expr: &str) -> Option< } } +/// Fragment for [`FileAttachedReferenceSource`]. +/// +/// **Manifest level only**, for the same reason as the derived source: an +/// attached artifact's `blob_hash` names a Blob, never a chunk, and these are +/// almost always single-chunk — so contributing at the chunk level would +/// double-count against the aliased hash. +fn file_attached_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option { + match level { + RefLevel::Chunk => None, + RefLevel::Manifest => Some(format!( + "(SELECT COUNT(*) FROM storage.file_attached_blobs {ATTACHED_ALIAS} \ + WHERE {ATTACHED_ALIAS}.blob_hash = {outer_hash_expr})" + )), + } +} + +/// Short-circuiting existence form, used by `dedup_gc`'s reap predicate. +fn file_attached_exists_sql(level: RefLevel, outer_hash_expr: &str) -> Option { + match level { + RefLevel::Chunk => None, + RefLevel::Manifest => Some(format!( + "EXISTS (SELECT 1 FROM storage.file_attached_blobs {ATTACHED_ALIAS} \ + WHERE {ATTACHED_ALIAS}.blob_hash = {outer_hash_expr})" + )), + } +} + /// Every built-in blob-reference source, in one place. /// /// THE definition of "what references a blob". `DedupService::new` uses it @@ -131,7 +159,10 @@ pub fn built_in_registry(pool: Arc) -> BlobReferenceRegistry { // Registered before anything writes a derived blob: dedup_gc's reap // predicate must already know this table exists, or the first sweep // after the first thumbnail deletes it. - registry.register(Arc::new(ContentDerivedReferenceSource::new(pool))); + registry.register(Arc::new(ContentDerivedReferenceSource::new(pool.clone()))); + // Same rule as above: registered before the first attachment is written, + // so dedup_gc's reap predicate already knows the table exists. + registry.register(Arc::new(FileAttachedReferenceSource::new(pool))); registry } @@ -388,6 +419,92 @@ impl BlobReferenceSource for ContentDerivedReferenceSource { } } +// ─── storage.file_attached_blobs ───────────────────────────────────────── + +/// References held by `storage.file_attached_blobs.blob_hash` — bytes a user +/// supplied for one specific file. +/// +/// Structurally the twin of [`ContentDerivedReferenceSource`]: same level, +/// same shape, different table. The difference that matters is upstream — the +/// row is keyed by `file_id` rather than by content, so the same bytes +/// attached to two files are two rows and therefore two references. Dedup +/// still applies to the bytes; what must not be shared is the mapping. +/// +/// `file_id` is deliberately not a reference at this layer: it is an +/// `ON DELETE CASCADE` foreign key, so the row disappears with the file, and +/// the blob reference it held is released by the owning service's +/// `on_file_deleted` hook. +pub struct FileAttachedReferenceSource { + pool: Arc, +} + +impl FileAttachedReferenceSource { + pub fn new(pool: Arc) -> Self { + Self { pool } + } +} + +#[async_trait] +impl BlobReferenceSource for FileAttachedReferenceSource { + fn source_name(&self) -> &'static str { + "file_attached" + } + + fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option { + file_attached_ref_sql(level, outer_hash_expr) + } + + fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option { + file_attached_exists_sql(level, outer_hash_expr) + } + + async fn count_references(&self, blob_hash: &str) -> Result { + let n: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM storage.file_attached_blobs WHERE blob_hash = $1", + ) + .bind(blob_hash) + .fetch_one(self.pool.as_ref()) + .await + .map_err(|e| { + DomainError::internal_error("BlobRefSource", format!("attached count: {e}")) + })?; + Ok(n.max(0) as u64) + } + + async fn list_referenced_blobs( + &self, + cursor: Option>, + limit: usize, + ) -> Result<(Vec, Option>), DomainError> { + // Paged by `blob_hash`, same as the derived source: it IS the value + // returned, and DISTINCT collapses one Blob attached to several files. + let after: Option = match cursor { + Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| { + DomainError::internal_error("BlobRefSource", format!("bad attached cursor: {e}")) + })?), + None => None, + }; + + let rows: Vec<(String,)> = sqlx::query_as( + "SELECT DISTINCT blob_hash FROM storage.file_attached_blobs + WHERE ($1::text IS NULL OR blob_hash > $1) + ORDER BY blob_hash + LIMIT $2", + ) + .bind(after) + .bind(limit as i64) + .fetch_all(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("BlobRefSource", format!("attached page: {e}")))?; + + let next = rows + .last() + .map(|(h,)| h.clone().into_bytes()) + .filter(|_| rows.len() == limit); + Ok((rows.into_iter().map(|(h,)| h).collect(), next)) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index 41bda424..c4826648 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -3550,7 +3550,8 @@ mod tests { FROM storage.chunk_manifests m WHERE m.ref_count <= 0 OR NOT (EXISTS (SELECT 1 FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash) - OR EXISTS (SELECT 1 FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash)) + OR EXISTS (SELECT 1 FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash) + OR EXISTS (SELECT 1 FROM storage.file_attached_blobs cnt_a WHERE cnt_a.blob_hash = m.file_hash)) LIMIT $1 ) RETURNING file_hash, chunk_hashes, total_size"#; diff --git a/src/infrastructure/services/manifests_consistency_service.rs b/src/infrastructure/services/manifests_consistency_service.rs index 3c2d1248..a149c372 100644 --- a/src/infrastructure/services/manifests_consistency_service.rs +++ b/src/infrastructure/services/manifests_consistency_service.rs @@ -432,7 +432,8 @@ mod tests { m.chunk_count AS chunk_count, ((SELECT COUNT(*) FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash) - + (SELECT COUNT(*) FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash))::bigint AS actual_ref_count + + (SELECT COUNT(*) FROM storage.content_derived_blobs cnt_d WHERE cnt_d.blob_hash = m.file_hash) + + (SELECT COUNT(*) FROM storage.file_attached_blobs cnt_a WHERE cnt_a.blob_hash = m.file_hash))::bigint AS actual_ref_count FROM storage.chunk_manifests m WHERE ($1::text IS NULL OR m.file_hash > $1) ORDER BY m.file_hash From 64ff98257127fb9596a42a98bcdc41680b7109f7 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 07:39:57 +0200 Subject: [PATCH 17/66] feat(thumbnails): uploaded previews survive a copy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else — keyed by file id, on local disk. No copy path duplicates it and no other instance can see it, so a copied file lost the preview its owner uploaded. Silently: the server falls back to rendering one from the source, or to 204 for a PDF, which has no render path at all. A user-supplied preview is not derivable from the content, so once lost it is gone. The PUT now also records a storage.file_attached_blobs row, which copy_file_satellites already duplicates, so both copy paths carry it. Best-effort: the sidecar has already succeeded by then and the user can see their thumbnail, so failing the request would report an error for an operation that visibly worked. Read path consults attachments ahead of every content-derived tier: an uploaded preview is an explicit choice about THIS file and must beat anything rendered from its content. Cached under the per-file key — a content key would leak those bytes to every other file sharing the content, which is the poisoning the file-keyed table exists to prevent. store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin: re-uploading a preview is a deliberate replacement, where a re-derived thumbnail is the same bytes again. The superseded blob's reference is released, or it would be pinned forever with nothing pointing at it. Deletion goes through a trigger, not a hook. file_id is ON DELETE CASCADE, and on_file_deleted fires AFTER delete_file — by then the cascade has run and there is nothing left to enumerate. This matters most for folder deletion, where PG cascades folders to files to attachments and Rust never sees the rows at all. storage.decrement_blob_ref keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused verbatim rather than transcribed into a second trigger that can drift. DELETE only: a replacement updates in place and is handled in Rust, so adding UPDATE would double-decrement. Extracted read_blob_to_bytes, shared by the attached and derived tiers — the only difference between them is which table produced the hash. tests/api/attached_thumbnail_copy.hurl guards it. The file is red and the uploaded thumbnail is green, so a render could never produce the uploaded bytes; the pre-upload render is captured first and required to change, which stops three identical renders from satisfying the byte-equality. Then both copy paths must serve the upload, and after the original is purged and GC runs, both copies must still serve it — each holds its own reference, because the rows are duplicated rather than shared. --- ..._file_attached_blobs_decrement_trigger.sql | 28 ++ src/infrastructure/services/dedup_service.rs | 100 +++++++ .../services/thumbnail_service.rs | 79 +++-- src/interfaces/api/handlers/file_handler.rs | 52 +++- tests/api/attached_thumbnail_copy.hurl | 275 ++++++++++++++++++ tests/api/run.sh | 1 + 6 files changed, 511 insertions(+), 24 deletions(-) create mode 100644 migrations/20261021000000_file_attached_blobs_decrement_trigger.sql create mode 100644 tests/api/attached_thumbnail_copy.hurl diff --git a/migrations/20261021000000_file_attached_blobs_decrement_trigger.sql b/migrations/20261021000000_file_attached_blobs_decrement_trigger.sql new file mode 100644 index 00000000..a1d8c368 --- /dev/null +++ b/migrations/20261021000000_file_attached_blobs_decrement_trigger.sql @@ -0,0 +1,28 @@ +-- Release the blob reference when an attachment row goes away. +-- +-- `storage.file_attached_blobs.file_id` is `ON DELETE CASCADE`, so deleting a +-- file removes its attachment rows inside the database — invisible to Rust. +-- The lifecycle hook cannot cover this: `on_file_deleted` fires AFTER +-- `delete_file`, by which point the cascade has already run and there is +-- nothing left to read. The references would survive with no row behind them, +-- and `dedup_gc` would see a positive count forever — bytes pinned for good. +-- +-- `storage.decrement_blob_ref()` already exists for exactly this, on +-- `storage.files`. It keys off `OLD.blob_hash` and is otherwise +-- table-agnostic, so it applies verbatim — and reusing it keeps the +-- manifest-first decrement contract defined in one place rather than +-- transcribed into a second trigger that can drift. +-- +-- Only DELETE. Replacing a preview updates `blob_hash` in place +-- (`store_attached_blob` is ON CONFLICT DO UPDATE), and the reference to the +-- superseded blob is released there, in Rust. Adding UPDATE here would +-- double-decrement it. + +CREATE OR REPLACE TRIGGER trg_file_attached_blobs_decrement_blob_ref + AFTER DELETE ON storage.file_attached_blobs + FOR EACH ROW + EXECUTE FUNCTION storage.decrement_blob_ref(); + +COMMENT ON TRIGGER trg_file_attached_blobs_decrement_blob_ref + ON storage.file_attached_blobs IS + 'Releases the blob reference held by an attachment row. Needed because file_id is ON DELETE CASCADE, so rows vanish inside the DB where the Rust lifecycle hooks cannot see them.'; diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index c4826648..c0d1d15f 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -576,6 +576,106 @@ impl DedupService { /// `docs/plan/derived-blobs.md`. /// /// Returns the derived blob hash. + /// Attach user-supplied bytes to a FILE — the file-keyed twin of + /// [`Self::store_derived_blob`]. + /// + /// Same storage path (the bytes are still content-addressed and still + /// deduplicated), different mapping: the row is keyed by `file_id`, so + /// two files holding identical attached bytes get two rows and two + /// references. Sharing the mapping is what must not happen — a + /// content-keyed client preview would let one user's upload be served + /// for another user's file. + /// + /// `ON CONFLICT … DO UPDATE`, unlike the derived twin: re-uploading a + /// preview for the same `(file_id, kind, variant)` is a deliberate + /// replacement, whereas a re-derived thumbnail is the same bytes again. + /// The reference held by the row being replaced is released. + pub async fn store_attached_blob( + &self, + file_id: &str, + kind: &str, + variant: &str, + content_type: &str, + bytes: Bytes, + uploaded_by: uuid::Uuid, + ) -> Result { + let stored = self + .store_from_stream( + stream::once(async move { Ok::(bytes) }), + Some(content_type.to_string()), + ) + .await?; + let attached_hash = stored.hash().to_string(); + + // `previous` is the hash this row pointed at before, when it existed + // and differed — the reference to release once the row no longer + // holds it. + let previous: Option<(Option,)> = sqlx::query_as( + "INSERT INTO storage.file_attached_blobs + (file_id, kind, variant, blob_hash, content_type, uploaded_by) + VALUES ($1::uuid, $2, $3, $4, $5, $6) + ON CONFLICT (file_id, kind, variant) DO UPDATE + SET blob_hash = EXCLUDED.blob_hash, + content_type = EXCLUDED.content_type, + uploaded_by = EXCLUDED.uploaded_by, + created_at = now() + RETURNING NULLIF(storage.file_attached_blobs.blob_hash, EXCLUDED.blob_hash)", + ) + .bind(file_id) + .bind(kind) + .bind(variant) + .bind(&attached_hash) + .bind(content_type) + .bind(uploaded_by) + .fetch_optional(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?; + + // A replaced row's old blob loses its only reference from here. Not + // releasing it would pin those bytes forever — nothing else points at + // a superseded preview. + if let Some((Some(old_hash),)) = previous + && old_hash != attached_hash + && let Err(e) = self.remove_reference(&old_hash).await + { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to release replaced attached-blob reference for {}", + &old_hash[..old_hash.len().min(12)], + ); + } + + Ok(attached_hash) + } + + /// Look up bytes attached to a file. File-keyed counterpart of + /// [`Self::find_derived_blob`]. + pub async fn find_attached_blob( + &self, + file_id: &str, + kind: &str, + variant: &str, + ) -> Option { + sqlx::query_as::<_, (String, String)>( + "SELECT blob_hash, content_type FROM storage.file_attached_blobs + WHERE file_id = $1::uuid AND kind = $2 AND variant = $3", + ) + .bind(file_id) + .bind(kind) + .bind(variant) + .fetch_optional(self.pool.as_ref()) + .await + .ok() + .flatten() + .map(|(blob_hash, content_type)| { + crate::application::ports::dedup_ports::DerivedBlobRef { + blob_hash, + content_type, + } + }) + } + pub async fn store_derived_blob( &self, source_hash: &str, diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 3da76cbe..92f91c07 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -505,6 +505,39 @@ impl ThumbnailService { /// `blob_hash` is used to locate the file on disk (dedup-aware). /// If `None`, only the in-memory cache is checked (used for video /// thumbnails where blob_hash is not yet resolved). + /// Drain a blob through the dedup stack into memory. + /// + /// Shared by the attached and derived tiers — the only difference between + /// them is which table produced the hash, so the read itself belongs in + /// one place. Returns `None` on a read fault rather than propagating: a + /// missing satellite must degrade to the next tier, never break a gallery. + async fn read_blob_to_bytes( + dedup: &DedupService, + blob_hash: &str, + file_id: &str, + size: ThumbnailSize, + ) -> Option { + use futures::StreamExt; + let mut stream = dedup.read_blob_stream(blob_hash).await.ok()?; + let mut buf = Vec::new(); + while let Some(chunk) = stream.next().await { + match chunk { + Ok(part) => buf.extend_from_slice(&part), + Err(e) => { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "thumbnail blob read failed for {} {:?}", + file_id, + size, + ); + return None; + } + } + } + Some(Bytes::from(buf)) + } + pub async fn get_cached_thumbnail( &self, file_id: &str, @@ -553,6 +586,32 @@ impl ThumbnailService { return Some(bytes); } + // 2b. Bytes the USER attached to this file, if any. + // + // Ahead of every content-derived tier below on purpose: an uploaded + // preview is an explicit choice about THIS file and must beat + // anything the server would render from its content. It is also the + // only tier a copy can inherit — the `ext-` sidecar above is keyed by + // file_id and is not copied, so without this branch a copied file + // silently falls back to a rendered thumbnail, or to none at all for + // a PDF that has no server-side render path. + if let Some(dedup) = dedup + && let Some(attached) = dedup + .find_attached_blob(file_id, "preview", size.dir_name()) + .await + && let Some(bytes) = + Self::read_blob_to_bytes(dedup, &attached.blob_hash, file_id, size).await + { + // Cached under the per-file key: these bytes belong to this file, + // not to its content, so a content key would leak them to every + // other file sharing that content — the poisoning the file-keyed + // table exists to prevent. + self.cache + .insert(ThumbnailCacheKey::external(file_id, size), bytes.clone()) + .await; + return Some(bytes); + } + // 3. Check disk for blob-hash thumbnails (needs blob_hash to locate) let hash = blob_hash?; let thumb_path = self.get_thumbnail_path(hash, size, format); @@ -580,25 +639,7 @@ impl ThumbnailService { let derived = dedup .find_derived_blob(hash, "thumbnail", size.dir_name()) .await?; - use futures::StreamExt; - let mut stream = dedup.read_blob_stream(&derived.blob_hash).await.ok()?; - let mut buf = Vec::new(); - while let Some(chunk) = stream.next().await { - match chunk { - Ok(part) => buf.extend_from_slice(&part), - Err(e) => { - tracing::warn!( - target: "oxicloud::dedup", - error = %e, - "derived thumbnail read failed for {} {:?}", - file_id, - size, - ); - return None; - } - } - } - let bytes = Bytes::from(buf); + let bytes = Self::read_blob_to_bytes(dedup, &derived.blob_hash, file_id, size).await?; self.cache .insert( ThumbnailCacheKey::content(hash, size, format), diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 7d2e7f71..7fdf95be 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -726,15 +726,57 @@ impl FileHandler { return AppError::from(err).into_response(); } - // Validate, re-encode to WebP, and store - match thumbnail_service + // Validate, re-encode, and store the per-file sidecar. + let stored = match thumbnail_service .store_external_thumbnail(&id, thumb_size.into(), body) .await { - Ok(_) => StatusCode::CREATED.into_response(), - Err(err) => AppError::internal_error(format!("Failed to store thumbnail: {}", err)) - .into_response(), + Ok(bytes) => bytes, + Err(err) => { + return AppError::internal_error(format!("Failed to store thumbnail: {}", err)) + .into_response(); + } + }; + + // Also record it as a file-keyed attachment. + // + // The sidecar above is `ext-{file_id}.jpg` on local disk, which no + // copy path duplicates and no other instance can see. Without this + // row a copied file loses the preview its owner uploaded — falling + // back to a rendered thumbnail, or to nothing at all for a PDF, which + // has no server-side render path. `copy_file_satellites` duplicates + // the row, so the copy inherits the bytes. + // + // File-keyed, never content-keyed: these bytes are the uploader's + // claim about THIS file, and sharing them across files with identical + // content is the poisoning vector `storage.file_attached_blobs` + // exists to prevent. + // + // Best-effort: the sidecar already succeeded, so the user has their + // thumbnail. Failing the request here would report an error for an + // operation that visibly worked. + if let Err(e) = state + .core + .dedup_service + .store_attached_blob( + &id, + "preview", + thumb_size.dir_name(), + "image/jpeg", + stored, + auth_user.id, + ) + .await + { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + file_id = %id, + "failed to record attached thumbnail; sidecar written, copies will not inherit it" + ); } + + StatusCode::CREATED.into_response() } // ═══════════════════════════════════════════════════════════════════════ diff --git a/tests/api/attached_thumbnail_copy.hurl b/tests/api/attached_thumbnail_copy.hurl new file mode 100644 index 00000000..bf4831b1 --- /dev/null +++ b/tests/api/attached_thumbnail_copy.hurl @@ -0,0 +1,275 @@ +# ============================================================= +# OxiCloud – An UPLOADED thumbnail survives both copy paths +# ============================================================= +# A user-supplied preview is not derivable from the file's content, so +# nothing can regenerate it. If a copy loses it, it is gone — and the loss +# is silent, because the server quietly falls back to rendering one from +# the source (or to 204 for a PDF, which has no render path at all). +# +# That was the behaviour before `storage.file_attached_blobs`: the PUT +# wrote `ext-{file_id}.jpg`, keyed by file id, which no copy path +# duplicates and no other instance can see. +# +# The test distinguishes "preserved" from "re-rendered" by making the two +# visibly different: the FILE is red-image.png, the uploaded thumbnail is +# derived from green-image.png. A server-side render of the file could +# only ever produce the red one. So byte-equality with the post-upload +# bytes proves the copy served the ATTACHMENT, not a fresh render. +# +# Step 4 is what makes that airtight — it captures the rendered thumbnail +# BEFORE the upload and requires the upload to change it. Without that, +# byte-equality across copies could be satisfied by three identical +# renders. +# +# Prerequisites: setup.hurl must have run (admin user exists). +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Source and destination folders +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-attach-src" +} + +HTTP 201 +[Captures] +src_folder_id: jsonpath "$.id" + + +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-attach-dst" +} + +HTTP 201 +[Captures] +dst_folder_id: jsonpath "$.id" + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Upload the file (RED) +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +folder_id: {{src_folder_id}} +file: file,fixtures/red-image.png; image/png + +HTTP 201 +[Captures] +orig_file_id: jsonpath "$.id" +orig_file_name: jsonpath "$.name" + + +# ───────────────────────────────────────────────────────────── +# Step 4 – The server-rendered thumbnail, before any upload. +# Captured so the upload can be shown to have replaced it. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +rendered_thumb: bytes + + +# ───────────────────────────────────────────────────────────── +# Step 5 – Upload a custom thumbnail (GREEN) for that file +# ───────────────────────────────────────────────────────────── +PUT {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +Content-Type: image/png +file,fixtures/green-image.png; + +HTTP 201 + + +# It must now serve the upload, not the render. +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +uploaded_thumb: bytes +[Asserts] +bytes != {{rendered_thumb}} + + +# ───────────────────────────────────────────────────────────── +# Step 6 – Single-file copy → the attachment comes with it. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/files/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "file_ids": ["{{orig_file_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +file_copy_id: jsonpath "$.successful[0].id" +[Asserts] +jsonpath "$.successful[0].id" != "{{orig_file_id}}" + + +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{uploaded_thumb}} +bytes != {{rendered_thumb}} + + +# ───────────────────────────────────────────────────────────── +# Step 7 – Folder copy → same, through storage.copy_folder_tree. +# +# The other copy path. It reaches the attachment through the same +# `copy_file_satellites` call, and this is the leg that would break if +# the tree path ever grew its own fan-out again. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/folders/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "folder_ids": ["{{src_folder_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +tree_root_id: jsonpath "$.successful[0].new_root_folder_id" +[Asserts] +jsonpath "$.stats.failed" == 0 + + +GET {{base_url}}/api/files?folder_id={{tree_root_id}} +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +tree_copy_id: jsonpath "$[0].id" +[Asserts] +jsonpath "$" count == 1 +jsonpath "$[0].name" == "{{orig_file_name}}" +jsonpath "$[0].id" != "{{orig_file_id}}" + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{uploaded_thumb}} +bytes != {{rendered_thumb}} + + +# ───────────────────────────────────────────────────────────── +# Step 8 – Delete the ORIGINAL, run GC, and require both copies to keep +# serving the upload. +# +# Each copy holds its own reference on the attached blob — the rows are +# duplicated, not shared, because the table is file-keyed. If the copy +# had failed to take one, deleting the original would walk the count to +# zero and GC would reap bytes that cannot be regenerated. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/files/{{orig_file_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_orig_id: jsonpath "$.items[?(@.resource.id == '{{orig_file_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_orig_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +POST {{base_url}}/api/admin/jobs/dedup_gc/trigger +Authorization: Bearer {{token}} +[Options] +delay: 500ms + +HTTP 200 + + +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{uploaded_thumb}} + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{uploaded_thumb}} + + +# ───────────────────────────────────────────────────────────── +# Step 9 – Teardown. Hurl files share one database within run.sh. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/folders/{{src_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +DELETE {{base_url}}/api/folders/{{dst_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_src_id: jsonpath "$.items[?(@.resource.id == '{{src_folder_id}}')].resource.id" +trash_dst_id: jsonpath "$.items[?(@.resource.id == '{{dst_folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_src_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +DELETE {{base_url}}/api/trash/{{trash_dst_id}} +Authorization: Bearer {{token}} + +HTTP 200 diff --git a/tests/api/run.sh b/tests/api/run.sh index d644513b..3d17c303 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -168,6 +168,7 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/dedup_blob_cleanup.hurl" \ "$API_DIR/derived_blob_copy.hurl" \ "$API_DIR/thumbnail_etag_content_keyed.hurl" \ + "$API_DIR/attached_thumbnail_copy.hurl" \ "$API_DIR/dedup_admin_gate.hurl" \ "$API_DIR/admin_jobs.hurl" \ "$API_DIR/recoverable_jobs.hurl" \ From 88819797618e8cc212b8d7b8486d5c9f41611161 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 07:45:02 +0200 Subject: [PATCH 18/66] test(api): assert the consistency jobs are clean after the whole suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The disk checks above prove nothing leaked. These prove the bookkeeping behind them is honest: every refcount matches what the reference sources hold, and no row points at bytes that are gone. End of suite is the only place this is cheap. One database serves every hurl file, so by here the counters have absorbed every upload, copy, move, share, trash and purge the suite performed — across both copy paths, the derived tier and the attached tier. Drift that no individual test would notice, because each only inspects its own file, surfaces as a mismatch. Runs after the GC drain deliberately: mid-sweep state is legitimately inconsistent — a manifest can sit at zero waiting for the next pass — so checking earlier would report normal in-flight state as drift. Zero findings is the assertion. These four tenants are read-only, so anything they report is a real invariant violation rather than a repair opportunity. A job missing from the build is skipped with a warning instead of failing, so this does not break on a feature-gated build. Unknown job names and unwrapped-vs-wrapped response shapes both degrade to a visible warning rather than a silent pass: list_job_runs currently returns a bare array, and the .runs/.items fallbacks exist so a future wrapping does not quietly turn the whole check into a no-op. --- tests/api/storage_cleanup_check.sh | 72 ++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index a8c6745e..aea335db 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -404,3 +404,75 @@ if [[ -n "$UPLOAD_FILES" ]]; then fi log "OK — no blobs, thumbnails, or chunked-upload leftovers remain on disk." + +# ── 5. Whole-suite consistency sweep ────────────────────────────────────────── +# +# The disk checks above prove nothing LEAKED. These prove the bookkeeping +# behind it is honest — that every refcount matches what the reference +# sources actually hold, and that no row points at bytes that are gone. +# +# End of suite is the right place, and the only place it is cheap. One +# database serves every hurl file (which is why each must tear down after +# itself), so by the time we get here the counters have absorbed every +# upload, copy, move, share, trash and purge the suite performed — across +# both copy paths, the derived tier and the attached tier. A drift that no +# single test would notice, because each only inspects its own file, shows +# up here as a mismatch. +# +# It runs AFTER the GC drain deliberately: mid-sweep state is legitimately +# inconsistent (a manifest can sit at zero waiting for the next pass), so +# checking before the drain would report normal in-flight state as drift. +# +# Zero findings is the assertion. These jobs are read-only, so a finding +# here is a real invariant violation, not a repair opportunity. + +CONSISTENCY_JOBS=( + files_consistency + blobs_consistency + manifests_consistency + backend_consistency +) + +CONSISTENCY_FAILED=0 +for job in "${CONSISTENCY_JOBS[@]}"; do + TRIGGER=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/$job/trigger") \ + || { log "WARNING: $job not registered in this build — skipped"; continue; } + [[ -z "$TRIGGER" ]] && { log "WARNING: $job returned an empty body — skipped"; continue; } + + # The trigger is synchronous for these tenants, but the run row is what + # carries the findings, so read it back rather than trusting the + # trigger's own summary. + # `list_job_runs` returns a bare JSON array, newest first. The `.runs` / + # `.items` fallbacks are there so a future wrapping of the response does + # not silently turn this check into a no-op. + RUN_ID=$(curl -sf -H "$AUTH" "$base_url/api/admin/jobs/$job/runs?limit=1" \ + | jq -r 'if type == "array" then .[0].id + else ((.runs // .items // [])[0].id) end // empty') + if [[ -z "$RUN_ID" ]]; then + log "WARNING: could not resolve a run id for $job — skipped" + continue + fi + + FINDINGS=$(curl -sf -H "$AUTH" \ + "$base_url/api/admin/jobs/$job/runs/$RUN_ID/findings?limit=100") + COUNT=$(echo "$FINDINGS" | jq -r \ + 'if type == "array" then length + else ((.findings // .items // []) | length) end') + + if [[ "$COUNT" -gt 0 ]]; then + log "$job reported $COUNT finding(s):" + echo "$FINDINGS" | jq -r \ + 'if type == "array" then .[] else (.findings // .items // [])[] end + | " \(.severity // "?") \(.kind // .finding_kind // "?") \(.details // {} | tostring)"' \ + 2>/dev/null | head -20 + CONSISTENCY_FAILED=1 + else + log "$job: clean." + fi +done + +if [[ "$CONSISTENCY_FAILED" -eq 1 ]]; then + fail "consistency jobs reported findings after the full suite — see above" +fi + +log "OK — all consistency jobs clean after the full suite." From 6c5e53fee46b8427c38cba34fd29de82b0e0f518 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 07:46:26 +0200 Subject: [PATCH 19/66] test(api): assert the blob registry is empty, not just the disk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The disk check proves no BYTES are left. This proves no ROWS are, which fails differently and worse: a stale storage.blobs row with nothing behind it means a reference was never released, and dedup_gc will skip it forever because its count never reaches zero. Silent, permanent, and invisible to a check that only looks at the filesystem. Zero is the right assertion, not "fewer than before". By this point the suite has deleted its users, their drives and everything cascading beneath, and the disk check has already insisted the blob store is empty. A non-zero registry beside an empty disk is exactly the divergence the consistency jobs report — caught here first because one number is easier to read than a findings list. Degrades to a warning if the endpoint is unavailable rather than failing, so a build without the admin dedup surface still runs the rest. --- tests/api/storage_cleanup_check.sh | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index aea335db..56ee60b4 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -405,6 +405,34 @@ fi log "OK — no blobs, thumbnails, or chunked-upload leftovers remain on disk." +# ── 4b. …and the registry agrees the store is empty ─────────────────────────── +# +# The disk check above proves no BYTES are left. This proves no ROWS are, +# which is the other direction and fails differently: a stale +# `storage.blobs` row with nothing behind it means a reference was never +# released, and the next `dedup_gc` will keep skipping it forever because +# its count never reaches zero. +# +# Zero is the right assertion here, not "fewer than before". Everything the +# suite created has been deleted by this point — the users, their drives, +# and the cascade beneath them — and the disk check has already insisted the +# blob store is empty. A non-zero registry alongside an empty disk is +# precisely the divergence the consistency jobs below would report, caught +# here first because a single number is easier to read than a findings list. +STATS=$(curl -sf -H "$AUTH" "$base_url/api/admin/dedup/stats" || true) +if [[ -z "$STATS" ]]; then + log "WARNING: /api/admin/dedup/stats unavailable — registry emptiness not checked" +else + REMAINING_BLOBS=$(echo "$STATS" | jq -r '.unique_blobs // 0') + REMAINING_BYTES=$(echo "$STATS" | jq -r '.total_physical_bytes // 0') + if [[ "$REMAINING_BLOBS" -ne 0 ]]; then + log "Registry still reports $REMAINING_BLOBS blob(s), $REMAINING_BYTES physical byte(s):" + echo "$STATS" | jq -r 'to_entries[] | " \(.key): \(.value)"' 2>/dev/null | head + fail "$REMAINING_BLOBS blob row(s) remain in the registry while the disk is empty" + fi + log "Registry clean: 0 blobs, 0 physical bytes." +fi + # ── 5. Whole-suite consistency sweep ────────────────────────────────────────── # # The disk checks above prove nothing LEAKED. These prove the bookkeeping From d71dd973e7b834ecf25b3139c47fb8deee879069 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 08:13:51 +0200 Subject: [PATCH 20/66] fix(thumbnails): two defects the copy test exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Per-file overrides must beat the content tier in RAM as well as on disk. The content-keyed lookup ran first, so a thumbnail already rendered from the file's content sat in RAM under content(blob_hash) and shadowed a preview uploaded afterwards — permanently. Invisible before the moka rekey, because both lived under one file-id key and the upload simply overwrote the render. Order is now uniform: per-file RAM, per-file disk (ext-), per-file DB, then content RAM, blob-hash disk, derived blob. 2. store_attached_blob never wrote a row. Its RETURNING clause compared the stored hash against EXCLUDED, and PostgreSQL only permits EXCLUDED in the SET and WHERE of DO UPDATE — a runtime syntax error on every call. The superseded hash now comes from a SELECT taken before the upsert; losing that race leaves one stale reference, which the manifest recompute reports, rather than anything being lost. The second hid behind the first for a whole cycle, and behind `ext-{file_id}.jpg`: the ORIGINAL kept serving its uploaded preview from local disk, so the feature looked healthy. Only a copy, which has a different file_id and therefore no ext- file, depends on the row — and the row was never there. The handler's best-effort warn! completed the disguise, so it is now error!: a failure there means copies silently lose the preview, and nothing else signals it. --- src/infrastructure/services/dedup_service.rs | 34 ++++++++++---- .../services/thumbnail_service.rs | 47 ++++++++++++++----- src/interfaces/api/handlers/file_handler.rs | 8 +++- 3 files changed, 66 insertions(+), 23 deletions(-) diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index c0d1d15f..62684f30 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -607,10 +607,29 @@ impl DedupService { .await?; let attached_hash = stored.hash().to_string(); - // `previous` is the hash this row pointed at before, when it existed - // and differed — the reference to release once the row no longer - // holds it. - let previous: Option<(Option,)> = sqlx::query_as( + // Read the hash being superseded BEFORE upserting. + // + // It cannot come from `RETURNING`: PostgreSQL only permits `EXCLUDED` + // in the `SET` and `WHERE` of `DO UPDATE`, so a RETURNING clause + // comparing old against new is a syntax error — and one that surfaces + // only at runtime, where this method's best-effort caller swallows it + // into a warning while the sidecar keeps the feature looking healthy. + // + // The gap between this SELECT and the upsert is benign: losing the + // race leaves one stale reference, which the manifest recompute + // reports rather than anything being lost or served wrongly. + let previous: Option<(String,)> = sqlx::query_as( + "SELECT blob_hash FROM storage.file_attached_blobs + WHERE file_id = $1::uuid AND kind = $2 AND variant = $3", + ) + .bind(file_id) + .bind(kind) + .bind(variant) + .fetch_optional(self.pool.as_ref()) + .await + .map_err(|e| DomainError::internal_error("Dedup", format!("read attached blob: {e}")))?; + + sqlx::query( "INSERT INTO storage.file_attached_blobs (file_id, kind, variant, blob_hash, content_type, uploaded_by) VALUES ($1::uuid, $2, $3, $4, $5, $6) @@ -618,8 +637,7 @@ impl DedupService { SET blob_hash = EXCLUDED.blob_hash, content_type = EXCLUDED.content_type, uploaded_by = EXCLUDED.uploaded_by, - created_at = now() - RETURNING NULLIF(storage.file_attached_blobs.blob_hash, EXCLUDED.blob_hash)", + created_at = now()", ) .bind(file_id) .bind(kind) @@ -627,14 +645,14 @@ impl DedupService { .bind(&attached_hash) .bind(content_type) .bind(uploaded_by) - .fetch_optional(self.pool.as_ref()) + .execute(self.pool.as_ref()) .await .map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?; // A replaced row's old blob loses its only reference from here. Not // releasing it would pin those bytes forever — nothing else points at // a superseded preview. - if let Some((Some(old_hash),)) = previous + if let Some((old_hash,)) = previous && old_hash != attached_hash && let Err(e) = self.remove_reference(&old_hash).await { diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 92f91c07..4e76938a 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -550,19 +550,22 @@ impl ThumbnailService { // today's behaviour, which is what the port impl wants. dedup: Option<&DedupService>, ) -> Option { - // 1. Check in-memory cache. + // A file-specific override beats anything derived from the content, + // and that has to hold at EVERY tier — including RAM. Checking the + // content-keyed entry first would let a previously-rendered + // thumbnail shadow a preview the user has since uploaded: the render + // is cached under `content(hash)`, the upload lands under + // `external(file_id)`, and the content key would win forever. // - // Keyed by content, so a caller that did not resolve the hash cannot - // consult this tier — it falls through to disk, which is correct - // rather than merely acceptable: a file-id key would be the stale - // entry the content key exists to avoid. Every HTTP path passes the - // hash (both handlers resolve it to build the ETag), so the fall - // through is confined to internal callers that never had one. - if let Some(hash) = blob_hash - && let Some(bytes) = self - .cache - .get(&ThumbnailCacheKey::content(hash, size, format)) - .await + // So the order is: per-file RAM, per-file disk, per-file DB, then the + // content-keyed tiers. Same precedence as the disk tiers below, just + // applied one level up. + + // 1. Per-file override in RAM (uploaded preview / video frame). + if let Some(bytes) = self + .cache + .get(&ThumbnailCacheKey::external(file_id, size)) + .await && !bytes.is_empty() { return Some(bytes); @@ -612,8 +615,26 @@ impl ThumbnailService { return Some(bytes); } - // 3. Check disk for blob-hash thumbnails (needs blob_hash to locate) + // 3. Content-keyed RAM tier. Below the per-file tiers by the rule + // above; still ahead of every disk read. + // + // A caller that did not resolve the hash cannot consult it and + // falls through to disk. That is correct rather than merely + // acceptable: a file-id key here would be the stale entry content + // keying exists to avoid. Both HTTP handlers resolve the hash to + // build the ETag, so the fall-through is confined to internal + // callers that never had one. let hash = blob_hash?; + if let Some(bytes) = self + .cache + .get(&ThumbnailCacheKey::content(hash, size, format)) + .await + && !bytes.is_empty() + { + return Some(bytes); + } + + // 4. Check disk for blob-hash thumbnails (needs blob_hash to locate) let thumb_path = self.get_thumbnail_path(hash, size, format); if let Ok(data) = fs::read(&thumb_path).await { let bytes = Bytes::from(data); diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 7fdf95be..6fbc0845 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -768,11 +768,15 @@ impl FileHandler { ) .await { - tracing::warn!( + // ERROR, not WARN: the sidecar keeps the feature looking healthy + // on this box, so nothing else signals that copies are silently + // losing the preview. A syntax error in the upsert hid behind a + // warning for an entire test cycle exactly this way. + tracing::error!( target: "oxicloud::dedup", error = %e, file_id = %id, - "failed to record attached thumbnail; sidecar written, copies will not inherit it" + "failed to record attached thumbnail; sidecar written, copies will NOT inherit it" ); } From 7d9418f63cb87371cfe2b9cf8cb6b6ec7a39a4cf Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 21:28:34 +0200 Subject: [PATCH 21/66] fix(thumbnails): ETag names the blob actually served MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fe9c4f49 keyed the ETag on the SOURCE file's content hash. That is wrong whenever the response comes from a satellite table, and for attachments it is wrong in two ways. Uploading a preview does not change the file's content, so a source-keyed ETag does not change either — and with `immutable` set, clients never revalidate and keep the previous render for up to a year. The exact staleness fe9c4f49 set out to fix, re-entering through the attachment path. Worse: a copy inherits the source hash, so an original and a copy have identical ETags. Give either one a different uploaded preview and they serve different bytes under one validator, which a shared cache may hand to either request. That is a collision, not just staleness. thumbnail_content_id resolves the identity through the same tier precedence the read path uses: an attached blob's own hash, else a derived blob's own hash, else the source-keyed form. An ETag naming a different tier than the one answering is worse than a coarse one, so the two orders must not drift. Derived-hash keying is strictly better than source-keying and never worse. The sidecar and the derived row are written from the same bytes; where they can diverge — a sidecar re-rendered while the derived row stays pinned by ON CONFLICT DO NOTHING — source-keying is wrong too, because the renderer is not part of that key. This is the step 10 change arriving early, forced by the attachment case; the plan note stands for the read-order flip itself. Known gap: a legacy ext-{file_id}.jpg with no file_attached_blobs row yet falls through to the source-keyed form. No worse than today, and it resolves when the import backfills. attached_thumbnail_copy.hurl now asserts ETags, which is why this went unnoticed: it compared bytes only, and thumbnail_etag_content_keyed covers content replacement rather than preview upload. A fresh GET returned the right bytes throughout — the same "healthy locally, broken for anyone caching" shape as the two bugs before it. --- .../services/thumbnail_service.rs | 56 +++++++++++++++++++ src/interfaces/api/handlers/file_handler.rs | 28 ++++++---- src/interfaces/nextcloud/preview_handler.rs | 27 +++++---- tests/api/attached_thumbnail_copy.hurl | 33 +++++++++++ 4 files changed, 122 insertions(+), 22 deletions(-) diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 4e76938a..dee210e0 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -505,6 +505,62 @@ impl ThumbnailService { /// `blob_hash` is used to locate the file on disk (dedup-aware). /// If `None`, only the in-memory cache is checked (used for video /// thumbnails where blob_hash is not yet resolved). + /// Identity of the bytes a thumbnail request will serve — the body of its + /// HTTP ETag. + /// + /// Mirrors the tier precedence in [`Self::get_cached_thumbnail`], because + /// an ETag that names a different tier than the one answering is worse + /// than a coarse one: it lets two resources serving different bytes share + /// a validator, and a shared cache may then hand either to either. + /// + /// * An **attached** blob wins, and its own hash is the identity. Nothing + /// else works: uploading a preview does not change the file's content, + /// so a source-keyed ETag would not change either — and with + /// `immutable` set, clients would never revalidate. Worse, a copy + /// inherits the source hash, so an original and a copy carrying + /// *different* uploaded previews would collide on one ETag. + /// * Otherwise a **derived** blob's own hash. Strictly better than + /// source-keying, never worse: the pair is written from the same bytes, + /// and where they can diverge — a sidecar re-rendered while the derived + /// row stays pinned by `ON CONFLICT DO NOTHING` — a source-keyed ETag + /// is wrong too, because the renderer is not part of the key. + /// * Otherwise the source-keyed form, which still identifies a render of + /// known content at a known size and format. + /// + /// Known gap: a legacy `ext-{file_id}.jpg` with no `file_attached_blobs` + /// row yet falls through to the source-keyed form, so those bytes keep + /// today's coarse validator until the import backfills the row. No worse + /// than current behaviour, and it disappears with the migration. + pub async fn thumbnail_content_id( + &self, + file_id: &str, + blob_hash: &str, + size: ThumbnailSize, + format: ThumbnailFormat, + dedup: Option<&DedupService>, + ) -> String { + if let Some(dedup) = dedup { + if let Some(attached) = dedup + .find_attached_blob(file_id, "preview", size.dir_name()) + .await + { + return attached.blob_hash; + } + if let Some(derived) = dedup + .find_derived_blob(blob_hash, "thumbnail", size.dir_name()) + .await + { + return derived.blob_hash; + } + } + format!( + "thumb-{}-{}-{}", + blob_hash, + size.dir_name(), + format.as_str() + ) + } + /// Drain a blob through the dedup stack into memory. /// /// Shared by the attached and derived tiers — the only difference between diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 6fbc0845..9413fe19 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -487,18 +487,22 @@ impl FileHandler { Ok(h) => h, Err(err) => return AppError::from(err).into_response(), }; - let etag = { - let (s, f) = (thumb_size.as_str(), format.as_str()); - let mut e = String::with_capacity(9 + blob_hash.len() + s.len() + f.len()); - e.push_str("\"thumb-"); - e.push_str(&blob_hash); - e.push('-'); - e.push_str(s); - e.push('-'); - e.push_str(f); - e.push('"'); - e - }; + // The identity of the bytes about to be served, resolved through the + // same tier precedence the read path uses — an uploaded preview's own + // hash, else a derived thumbnail's own hash, else the source-keyed + // form. See `ThumbnailService::thumbnail_content_id`. + let etag = format!( + "\"{}\"", + thumbnail_service + .thumbnail_content_id( + &id, + &blob_hash, + thumb_size.into(), + format, + Some(&state.core.dedup_service), + ) + .await + ); if let Some(if_none_match) = headers.get(header::IF_NONE_MATCH) && let Ok(val) = if_none_match.to_str() && (val == etag || val == "*") diff --git a/src/interfaces/nextcloud/preview_handler.rs b/src/interfaces/nextcloud/preview_handler.rs index 385501b8..6c169fe7 100644 --- a/src/interfaces/nextcloud/preview_handler.rs +++ b/src/interfaces/nextcloud/preview_handler.rs @@ -166,16 +166,23 @@ pub async fn handle_preview( .unwrap(); } }; - let etag = { - let s = thumb_size.as_str(); - let mut e = String::with_capacity(9 + blob_hash.len() + s.len()); - e.push_str("\"thumb-"); - e.push_str(&blob_hash); - e.push('-'); - e.push_str(s); - e.push('"'); - e - }; + // Same tier-precedence resolution as the REST endpoint: an uploaded + // preview's own hash, else a derived thumbnail's own hash, else the + // source-keyed form. NC pins JPEG, so that is the format asked for. + let etag = format!( + "\"{}\"", + state + .core + .thumbnail_service + .thumbnail_content_id( + &object_id, + &blob_hash, + thumb_size.into(), + ThumbnailFormat::Jpeg, + Some(&state.core.dedup_service), + ) + .await + ); if let Some(inm) = req.headers().get(header::IF_NONE_MATCH) && let Ok(client_etag) = inm.to_str() && (client_etag == etag || client_etag == "*") diff --git a/tests/api/attached_thumbnail_copy.hurl b/tests/api/attached_thumbnail_copy.hurl index bf4831b1..a14683e5 100644 --- a/tests/api/attached_thumbnail_copy.hurl +++ b/tests/api/attached_thumbnail_copy.hurl @@ -92,6 +92,7 @@ Authorization: Bearer {{token}} HTTP 200 [Captures] rendered_thumb: bytes +rendered_etag: header "ETag" # ───────────────────────────────────────────────────────────── @@ -112,8 +113,33 @@ Authorization: Bearer {{token}} HTTP 200 [Captures] uploaded_thumb: bytes +uploaded_etag: header "ETag" [Asserts] bytes != {{rendered_thumb}} +# The ETag must move with the bytes. It is keyed on the ATTACHED blob's own +# hash, because uploading a preview leaves the file's content — and so a +# source-keyed ETag — unchanged. With `immutable` set, an unchanged +# validator means clients never revalidate and keep the old render for a +# year. +header "ETag" != "{{rendered_etag}}" + + +# A client holding the pre-upload validator must be told to refetch. +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{rendered_etag}} + +HTTP 200 +[Asserts] +header "ETag" == "{{uploaded_etag}}" + + +# ...and the new one revalidates. +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +If-None-Match: {{uploaded_etag}} + +HTTP 304 # ───────────────────────────────────────────────────────────── @@ -141,6 +167,12 @@ HTTP 200 [Asserts] bytes == {{uploaded_thumb}} bytes != {{rendered_thumb}} +# Same bytes, so the same validator — the copy's attachment row points at +# the same blob. This is also what stops the collision a source-keyed ETag +# would allow: the copy inherits the source hash, so if either side later +# gets a DIFFERENT preview the two would serve different bytes under one +# ETag, and a shared cache could hand either to either. +header "ETag" == "{{uploaded_etag}}" # ───────────────────────────────────────────────────────────── @@ -184,6 +216,7 @@ HTTP 200 [Asserts] bytes == {{uploaded_thumb}} bytes != {{rendered_thumb}} +header "ETag" == "{{uploaded_etag}}" # ───────────────────────────────────────────────────────────── From 95648f2fa3b04089941184c9aa8d1ee1de992baa Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 23:17:49 +0200 Subject: [PATCH 22/66] =?UTF-8?q?fix(thumbnails):=20private,=20no-cache=20?= =?UTF-8?q?=E2=80=94=20the=20URL=20is=20gated=20and=20mutable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thumbnails were served `public, max-age=31536000, immutable`. Two problems, and the first is a security one. `public` on a Permission::Read gated resource lets any shared cache — a corporate proxy, a CDN — store one user's thumbnail and serve it to another. `Vary: Accept` was no defence: it does not vary on Authorization. Now `private`. `immutable` was a promise this URL cannot keep. It is keyed by file id, and its bytes change when a preview is uploaded, when content is replaced, or when an attachment is removed. `immutable` tells a client not to revalidate at all during the freshness lifetime, so with a one-year max-age a browser that fetched once would never see a new preview — which also made the content-keyed ETag unobservable in practice. A correct validator is worthless if nothing asks. Now `no-cache`, which still stores the body and only requires revalidation, answered by the ETag with a body-less 304. The hurl tests could not have caught this: hurl always sends the request, so If-None-Match was exercised and passed while a browser obeying `immutable` never got that far. Same "correct on the wire, wrong in practice" shape as the bugs before it, so the test now asserts the directives themselves rather than only the 304 behaviour. One definition, shared by the REST and NextCloud endpoints, which are gated identically and must not drift. /_app/immutable is untouched: those are hash-named static assets, genuinely content-addressed and public, where the directive is honest. Cost is a conditional request per thumbnail per page load. Recovering it needs a content-addressed URL — where `immutable` would be true — but that puts the hash in the URL of an authorized resource, so it stays `private` regardless, and it touches the SPA and the file DTO. Separate change. --- src/interfaces/api/handlers/file_handler.rs | 70 ++++++++++++++------- src/interfaces/nextcloud/preview_handler.rs | 22 ++++--- tests/api/thumbnail_etag_content_keyed.hurl | 10 ++- 3 files changed, 71 insertions(+), 31 deletions(-) diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 9413fe19..07400fc2 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -393,6 +393,35 @@ impl FileHandler { // THUMBNAILS // ═══════════════════════════════════════════════════════════════════════ + /// Cache policy for every thumbnail response. + /// + /// **`private`**, because a thumbnail is authorization-gated: the handler + /// runs a `Permission::Read` check before serving it. `public` let any + /// shared cache — a corporate proxy, a CDN — store one user's thumbnail + /// and hand it to another. `Vary: Accept` did not help, because it does + /// not vary on `Authorization`. + /// + /// **`no-cache`**, not `immutable`, because this URL is keyed by file id + /// and its bytes are mutable: uploading a preview, replacing the file's + /// content, or removing an attachment all change what it serves. + /// `immutable` promises the opposite, so a client that fetched once would + /// not revalidate — for a year, under the previous `max-age` — and would + /// never see a new preview. That also made the content-keyed ETag + /// unobservable in a browser: a correct validator is worthless if nothing + /// asks. + /// + /// `no-cache` still stores the body; it only requires revalidation before + /// reuse, which the ETag answers with a body-less 304. + /// + /// The cost is a conditional request per thumbnail per page load. Buying + /// that back needs a content-addressed URL, where `immutable` would be + /// honest — but the hash would then be in the URL of an authorized + /// resource, so it stays `private` regardless. Separate change; it + /// touches the SPA and the file DTO. + /// Shared with the NextCloud preview endpoint, which is gated the same + /// way and must not drift from this policy. + pub(crate) const THUMBNAIL_CACHE_CONTROL: &'static str = "private, no-cache"; + /// Get a thumbnail for a file (image or video). /// /// **Cache-first**: once past the hash lookup below, a thumbnail already @@ -402,13 +431,15 @@ impl FileHandler { /// UUIDv4 file IDs have 122 bits of entropy, making enumeration /// infeasible. /// - /// **ETag / 304**: responses carry an immutable ETag keyed on the - /// **content hash**, so it identifies the bytes rather than the file. - /// Replacing a file's content changes it (correct invalidation), and two - /// files with identical content share it (a copy revalidates to 304 - /// instead of refetching). Costs one PK lookup on the 304 path, which an - /// id-keyed ETag avoided at the price of never invalidating — see the - /// comment at the ETag construction. + /// **ETag / 304**: the ETag names the **blob actually served** — an + /// uploaded preview's hash, else a derived thumbnail's, else the + /// source-keyed form (see `ThumbnailService::thumbnail_content_id`). So + /// replacing content or uploading a preview invalidates correctly, and + /// two files serving identical bytes share a validator. Costs one or two + /// indexed lookups on the 304 path, which an id-keyed ETag avoided at the + /// price of never invalidating. Cache policy is + /// [`Self::THUMBNAIL_CACHE_CONTROL`] — `private, no-cache`, since this + /// URL is authorization-gated and its bytes are mutable. /// /// Beyond that, the DB path is only taken on a **cache miss for images** /// where the thumbnail hasn't been generated yet (first access after @@ -455,20 +486,17 @@ impl FileHandler { ThumbnailFormat::from_accept(headers.get(header::ACCEPT).and_then(|v| v.to_str().ok())); // ── ETag short-circuit ─────────────────────────────────────── - // Keyed on the CONTENT hash, not the file id. A thumbnail is a pure - // function of (source bytes, size, format), so that triple genuinely - // identifies the response — which is what makes the `immutable` - // directive below an honest claim. + // Keyed on the CONTENT served, not the file id. // // Keying on `file_id` was wrong in both directions. Replacing a // file's content preserves its id (`file_upload_service` rebuilds the // entity with `parts.id` and a new hash, then fires // `on_file_updated`, which regenerates the thumbnails), so the ETag - // never changed — and since `immutable` tells a browser not to - // revalidate at all inside the freshness window, clients kept the old - // preview for up to a year. Conversely a copy, or any dedup twin, got - // a *different* id and so refetched bytes it already held, even - // though the server serves both from the same derived blob. + // never changed — and the response was `immutable` with a one-year + // max-age, so clients never revalidated and kept the old preview. + // Conversely a copy, or any dedup twin, got a *different* id and so + // refetched bytes it already held, even though the server serves both + // from the same derived blob. // // Cost: one PK lookup, where the id-keyed version needed none. It // buys correct invalidation plus 304s shared across every file with @@ -511,7 +539,7 @@ impl FileHandler { .status(StatusCode::NOT_MODIFIED) .header(header::ETAG, &etag) .header(header::VARY, header::ACCEPT.as_str()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, Self::THUMBNAIL_CACHE_CONTROL) .body(Body::empty()) .unwrap() .into_response(); @@ -539,7 +567,7 @@ impl FileHandler { crate::common::mime_detect::thumbnail_content_type(&data), ) .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, Self::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, &etag) .header(header::VARY, header::ACCEPT.as_str()) .body(Body::from(data)) @@ -591,7 +619,7 @@ impl FileHandler { crate::common::mime_detect::thumbnail_content_type(&data), ) .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, Self::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, &etag) .header(header::VARY, header::ACCEPT.as_str()) .body(Body::from(data)) @@ -623,7 +651,7 @@ impl FileHandler { crate::common::mime_detect::thumbnail_content_type(&data), ) .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, Self::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, &etag) .header(header::VARY, header::ACCEPT.as_str()) .body(Body::from(data)) @@ -655,7 +683,7 @@ impl FileHandler { crate::common::mime_detect::thumbnail_content_type(&data), ) .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, Self::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, &etag) .header(header::VARY, header::ACCEPT.as_str()) .body(Body::from(data)) diff --git a/src/interfaces/nextcloud/preview_handler.rs b/src/interfaces/nextcloud/preview_handler.rs index 6c169fe7..6f8e1c0b 100644 --- a/src/interfaces/nextcloud/preview_handler.rs +++ b/src/interfaces/nextcloud/preview_handler.rs @@ -17,6 +17,9 @@ use crate::application::ports::storage_ports::FileReadPort; use crate::application::ports::thumbnail_ports::{ThumbnailFormat, ThumbnailPort, ThumbnailSize}; use crate::common::di::AppState; use crate::domain::services::authorization::{Permission, Resource, Subject}; +// One definition of the thumbnail cache policy, shared with the REST +// endpoint: both are Permission::Read gated, so both must stay `private`. +use crate::interfaces::api::handlers::file_handler::FileHandler; use crate::interfaces::middleware::auth::AuthUser; use uuid::Uuid; @@ -143,12 +146,13 @@ pub async fn handle_preview( // (ROUND10). Authz already passed above; a 304 must never skip the Read // check. // - // Keyed on the CONTENT hash, matching the REST thumbnail endpoint. A - // thumbnail is a pure function of (source bytes, size), so that pair - // identifies the response and `immutable` below is honest. Keying on the - // object id meant replacing a file's content — which preserves the id — - // left every client showing the old preview for up to a year, since - // `immutable` suppresses revalidation entirely. + // Keyed on the CONTENT of the bytes served, matching the REST thumbnail + // endpoint. Keying on the object id meant replacing a file's content — + // which preserves the id — left the validator unchanged, and the response + // was `immutable` with a one-year max-age, so clients never revalidated + // and showed the old preview indefinitely. Both halves are fixed: the + // ETag names what is served (see `thumbnail_content_id`) and the policy + // is `private, no-cache` (see `FileHandler::THUMBNAIL_CACHE_CONTROL`). // // This moves the blob-hash query ahead of the 304 rather than adding one: // the same lookup used to sit just below, on the path that renders. @@ -189,7 +193,7 @@ pub async fn handle_preview( { return Response::builder() .status(StatusCode::NOT_MODIFIED) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, FileHandler::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, etag) .body(Body::empty()) .unwrap(); @@ -226,7 +230,7 @@ pub async fn handle_preview( .status(StatusCode::OK) .header(header::CONTENT_TYPE, "image/jpeg") .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, FileHandler::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, etag) .body(Body::from(data)) .unwrap(); @@ -251,7 +255,7 @@ pub async fn handle_preview( .status(StatusCode::OK) .header(header::CONTENT_TYPE, "image/jpeg") .header(header::CONTENT_LENGTH, data.len()) - .header(header::CACHE_CONTROL, "public, max-age=31536000, immutable") + .header(header::CACHE_CONTROL, FileHandler::THUMBNAIL_CACHE_CONTROL) .header(header::ETAG, etag) .body(Body::from(data)) .unwrap(), diff --git a/tests/api/thumbnail_etag_content_keyed.hurl b/tests/api/thumbnail_etag_content_keyed.hurl index 805b1438..919463ac 100644 --- a/tests/api/thumbnail_etag_content_keyed.hurl +++ b/tests/api/thumbnail_etag_content_keyed.hurl @@ -82,7 +82,15 @@ HTTP 200 [Captures] etag_before: header "ETag" [Asserts] -header "Cache-Control" contains "immutable" +# `private`, because a thumbnail is Permission::Read gated — `public` let a +# shared proxy hand one user's thumbnail to another. `no-cache` rather than +# `immutable`, because this URL is keyed by file id and its bytes change +# when content is replaced or a preview uploaded; `immutable` suppressed +# revalidation entirely, which made the ETag below unobservable in a real +# client. +header "Cache-Control" contains "private" +header "Cache-Control" contains "no-cache" +header "Cache-Control" not contains "immutable" # Unchanged content revalidates to 304 — the caching path works. From f80a28763e6232b1055f965b40c29db0723e221c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Tue, 25 Aug 2026 23:57:16 +0200 Subject: [PATCH 23/66] =?UTF-8?q?feat(storage):=20thumb=5Fderived=5Fimport?= =?UTF-8?q?=20=E2=80=94=20backfill=20the=20derived=20tier=20from=20sidecar?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First half of step 10. Every server-rendered thumbnail written before content_derived_blobs existed lives only as {thumbnails_root}/{size}/{hash}.webp — local-disk state that another instance cannot see, a backend migration does not carry, and no consistency job covers. This walks those files into the blob store and records the mapping, so the derived tier can become authoritative and the sidecar can be deleted. A registered JobRegistry tenant rather than a script: the volume is unbounded, so it needs a cursor, resume, cooperative cancel and run history, and an operator needs somewhere to watch it. Cursor is {size_dir}/{filename} over a sorted walk, which totally orders the traversal. Idempotent by construction — each file is skipped when a row already exists, and store_derived_blob is ON CONFLICT DO NOTHING with release-on-conflict beneath it, so re-runs cannot inflate refcounts. Re-running is the expected operator behaviour, since Phase 3 (deleting the sidecars) is gated on a run reporting zero imported. hash_from_sidecar_name deliberately rejects ext-{file_id}.jpg. Those bytes are user-supplied and file-keyed; importing them here would content-key them and share one user's uploaded preview onto every file with identical content. They belong to thumb_attached_import. Both the accept and the reject set are under test. Unreadable files and store failures record a finding and continue: a sidecar removed by a concurrent GC unlink between listing and read is expected, not fatal, and the file is left in place for the next run. Registered unconditionally rather than behind a flag — a migration nobody can find is a migration nobody runs. --- src/common/di.rs | 18 + src/infrastructure/services/mod.rs | 1 + .../services/thumb_derived_import_service.rs | 313 ++++++++++++++++++ 3 files changed, 332 insertions(+) create mode 100644 src/infrastructure/services/thumb_derived_import_service.rs diff --git a/src/common/di.rs b/src/common/di.rs index 4630da69..a31c0980 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -1476,6 +1476,24 @@ impl AppServiceFactory { .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) .await; + // Step 10 migration tenant: backfills `content_derived_blobs` from + // the on-disk thumbnail sidecars that predate it. Idempotent, so it + // is safe to trigger repeatedly — Phase 3 (deleting the sidecars) is + // gated on a run reporting zero imported. Registered unconditionally + // rather than behind a flag: a migration nobody can find is a + // migration nobody runs. + // + // `.thumbnails` lives under the storage path, matching + // `ThumbnailService::new(&self.storage_path, …)` above. + let _ = Arc::new( + crate::infrastructure::services::thumb_derived_import_service::ThumbDerivedImport::new( + std::path::Path::new(&self.storage_path).join(".thumbnails"), + core.dedup_service.clone(), + ), + ) + .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) + .await; + // Third recoverable-run tenant. Iterates `storage.files` // and reports parent-folder-trashed cascade misses, // `missing_blob` (data-loss indicator — file references diff --git a/src/infrastructure/services/mod.rs b/src/infrastructure/services/mod.rs index 2fbb00ae..8ff128d6 100644 --- a/src/infrastructure/services/mod.rs +++ b/src/infrastructure/services/mod.rs @@ -57,6 +57,7 @@ pub mod session_liveness_gauges; pub mod share_unlock_cookie; pub mod smtp_email_sender; pub mod swappable_blob_backend; +pub mod thumb_derived_import_service; pub mod thumbnail_service; #[cfg(test)] mod thumbnail_service_test; diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs new file mode 100644 index 00000000..c01f7e31 --- /dev/null +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -0,0 +1,313 @@ +//! `thumb_derived_import` — backfill `storage.content_derived_blobs` from the +//! on-disk thumbnail sidecars that predate it. +//! +//! Step 10 of `docs/plan/derived-blobs.md`. Every server-rendered thumbnail +//! written before `content_derived_blobs` existed lives only as +//! `{thumbnails_root}/{size}/{hash}.webp`. That is local-disk state: another +//! instance cannot see it, a backend migration does not carry it, and no +//! consistency job covers it. This job moves those bytes into the blob store +//! and records the mapping, after which the derived tier can become +//! authoritative and the sidecar can be deleted. +//! +//! **Thumbnails only, and that is permanent.** The table also holds +//! `kind = 'transcode'`, but transcoding lands *after* this migration, so +//! transcodes are born into the table and never pass through a sidecar era. +//! This job will not grow a transcode arm. +//! +//! ### Idempotent by construction +//! +//! Each file is skipped when a row already exists for its +//! `(source_hash, 'thumbnail', variant)`, and `store_derived_blob` is +//! `ON CONFLICT DO NOTHING` with a release-on-conflict underneath, so a +//! re-run cannot inflate refcounts. Re-running is the expected operator +//! behaviour — Phase 3 (deleting the sidecars) is gated on a run reporting +//! zero imported. +//! +//! ### Multi-instance caveat +//! +//! Sidecars are local. Running this on one instance migrates only that +//! instance's files, so Phase 3 must be gated on *every* instance reporting +//! an empty tail. The run history does not aggregate across instances; that +//! remains an operator responsibility. + +use std::path::PathBuf; +use std::sync::Arc; + +use async_trait::async_trait; +use bytes::Bytes; +use tokio::fs; + +use crate::application::ports::thumbnail_ports::ThumbnailSize; +use crate::infrastructure::scheduler::{ + JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, + RunStatus, record_or_log, +}; +use crate::infrastructure::services::dedup_service::DedupService; + +pub const THUMB_DERIVED_IMPORT_JOB_NAME: &str = "thumb_derived_import"; + +/// Files handled between checkpoints. Each one is a read plus (at most) a +/// blob write, so this is deliberately smaller than a pure-DB sweep's page. +const BATCH_SIZE: usize = 100; + +pub struct ThumbDerivedImport { + thumbnails_root: PathBuf, + dedup: Arc, +} + +impl ThumbDerivedImport { + pub fn new(thumbnails_root: PathBuf, dedup: Arc) -> Self { + Self { + thumbnails_root, + dedup, + } + } + + pub async fn register_recoverable_job( + self: Arc, + registry: &JobRegistry, + provider: &Arc, + ) -> Arc { + registry + .register_recoverable_job(self.clone(), provider.clone(), None) + .await; + self + } + + /// The hash a sidecar filename names, or `None` when the file is not one. + /// + /// Strict, and deliberately rejects `ext-{file_id}.jpg`: those are + /// user-supplied, file-keyed bytes. Importing them here would content-key + /// them and share one user's uploaded preview onto every file with + /// identical content — the poisoning `file_attached_blobs` exists to + /// prevent. They belong to `thumb_attached_import`. + fn hash_from_sidecar_name(name: &str) -> Option<&str> { + let stem = name.strip_suffix(".webp")?; + if stem.len() != 64 || !stem.chars().all(|c| c.is_ascii_hexdigit()) { + return None; + } + Some(stem) + } + + /// Sorted sidecar filenames for one size directory. + /// + /// Sorted so the cursor is meaningful: resume skips everything at or + /// before it, which only works over a stable order. + async fn sidecar_names(&self, size: ThumbnailSize) -> Vec { + let dir = self.thumbnails_root.join(size.dir_name()); + let Ok(mut entries) = fs::read_dir(&dir).await else { + return Vec::new(); + }; + let mut names = Vec::new(); + while let Ok(Some(entry)) = entries.next_entry().await { + if let Some(name) = entry.file_name().to_str() + && Self::hash_from_sidecar_name(name).is_some() + { + names.push(name.to_string()); + } + } + names.sort(); + names + } +} + +#[async_trait] +impl RecoverableJobHandler for ThumbDerivedImport { + fn name(&self) -> &str { + THUMB_DERIVED_IMPORT_JOB_NAME + } + + async fn count_total(&self) -> Option { + let mut total = 0u64; + for size in ThumbnailSize::all() { + total += self.sidecar_names(*size).await.len() as u64; + } + Some(total) + } + + async fn run_resumable( + &self, + store: &dyn JobStore, + _args: &JobRunArgs, + resume_cursor: Option>, + ) -> RunOutcome { + // Cursor is `{size_dir}/{filename}` — the last file completed. Sizes + // are walked in `ThumbnailSize::all()` order, and names are sorted + // within each, so the pair totally orders the walk. + let cursor: Option = match resume_cursor { + None => None, + Some(b) if b.is_empty() => None, + Some(b) => match String::from_utf8(b) { + Ok(s) => Some(s), + Err(e) => { + return RunOutcome::Failed { + message: format!("invalid cursor: not valid UTF-8: {e}"), + }; + } + }, + }; + + let mut imported = 0u64; + let mut already = 0u64; + let mut failed = 0u64; + let mut since_checkpoint = 0usize; + let variant_of = |s: ThumbnailSize| s.dir_name().to_string(); + + for size in ThumbnailSize::all() { + let dir_name = variant_of(*size); + for name in self.sidecar_names(*size).await { + let position = format!("{dir_name}/{name}"); + + // Resume: everything at or before the cursor is done. + if let Some(c) = &cursor + && position.as_str() <= c.as_str() + { + continue; + } + + match store.status().await { + Ok(RunStatus::CancelRequested) => { + return RunOutcome::Paused { + cursor: position.into_bytes(), + }; + } + Ok(_) => {} + Err(e) => { + return RunOutcome::Failed { + message: format!("status poll: {e}"), + }; + } + } + + let Some(hash) = Self::hash_from_sidecar_name(&name) else { + continue; + }; + + // Already mapped — the common case on a re-run, and the + // reason this job is safe to trigger repeatedly. + if self + .dedup + .find_derived_blob(hash, "thumbnail", &dir_name) + .await + .is_some() + { + already += 1; + } else { + let path = self.thumbnails_root.join(&dir_name).join(&name); + match fs::read(&path).await { + Ok(data) => { + match self + .dedup + .store_derived_blob( + hash, + "thumbnail", + &dir_name, + "image/webp", + Bytes::from(data), + ) + .await + { + Ok(_) => imported += 1, + Err(e) => { + failed += 1; + record_or_log( + store, + THUMB_DERIVED_IMPORT_JOB_NAME, + "thumbnail_import_failed", + "anomaly", + None, + serde_json::json!({ + "path": position, + "hash": hash, + "error": format!("{e}"), + "note": "sidecar left in place; safe to re-run", + }), + ) + .await; + } + } + } + Err(e) => { + // Unreadable, or removed between listing and read + // (a concurrent GC unlink). Neither is fatal. + failed += 1; + record_or_log( + store, + THUMB_DERIVED_IMPORT_JOB_NAME, + "thumbnail_unreadable", + "anomaly", + None, + serde_json::json!({ + "path": position, + "error": format!("{e}"), + }), + ) + .await; + } + } + } + + since_checkpoint += 1; + if since_checkpoint >= BATCH_SIZE { + if let Err(e) = store + .checkpoint(position.clone().into_bytes(), since_checkpoint as u64) + .await + { + return RunOutcome::Failed { + message: format!("checkpoint: {e}"), + }; + } + since_checkpoint = 0; + } + } + } + + tracing::info!( + target: "oxicloud::dedup", + event = "thumb_derived_import.completed", + run_id = %store.run_id(), + imported = imported, + already_present = already, + failed = failed, + "thumb_derived_import: {imported} imported, {already} already present, {failed} failed" + ); + + RunOutcome::completed() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + + #[test] + fn accepts_a_canonical_sidecar_name() { + assert_eq!( + ThumbDerivedImport::hash_from_sidecar_name(&format!("{H}.webp")), + Some(H) + ); + } + + /// `ext-` files are user-supplied and file-keyed. Importing one here + /// would content-key it and share it across every file with identical + /// content — the exact poisoning the table split prevents. + #[test] + fn rejects_external_and_malformed_names() { + for name in [ + format!("ext-{H}.jpg"), + "ext-3f2b1c00-0000-0000-0000-000000000000.jpg".to_string(), + format!("{H}.jpg"), + format!("{}.webp", &H[..63]), + H.to_string(), + "junk.webp".to_string(), + ] { + assert_eq!( + ThumbDerivedImport::hash_from_sidecar_name(&name), + None, + "must not be imported as a derived thumbnail: {name}" + ); + } + } +} From 7a2ebe0fdc28652f88e8ef6678cdca0497c365e2 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 00:20:14 +0200 Subject: [PATCH 24/66] =?UTF-8?q?feat(storage):=20thumb=5Fattached=5Fimpor?= =?UTF-8?q?t=20=E2=80=94=20backfill=20uploaded=20previews?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twin of thumb_derived_import, for the other sidecar shape: {thumbnails_root}/{size}/ext-{file_id}.jpg, the previews a user uploaded — notably the SPA's client-side PDF generator, which has no server-side render path at all. Until a row exists, a copy of the file LOSES the preview: the sidecar is keyed by file_id, no copy path duplicates it, and the server silently falls back to rendering from the source, or to nothing for a PDF. That is the bug file_attached_blobs closed for new uploads; this closes it for everything already on disk. Separate job rather than an arm of the derived import, because the keying differs and that difference is the security boundary. These bytes are not derivable from the file's content, so content-keying them would share one user's uploaded preview onto every file with identical content. Each job's name filter rejects the other's shape, and both directions are under test. Idempotence needs more care here than in the derived twin. store_attached_blob is ON CONFLICT DO UPDATE, so calling it for an existing row releases the previous reference and takes a new one — harmless once, but a job doing it every run would churn refcounts. The row is therefore checked first and the store reached only on a genuine insert. uploaded_by is the nil sentinel: disk records no uploader, and inventing one — the file's created_by, say — would fabricate provenance that could later read as evidence an Editor replaced someone's preview. The column is NOT NULL with no FK precisely so provenance survives, and a sentinel says "unknown" honestly. Orphaned sidecars (no storage.files row) are counted and reported, not deleted. This job imports; it does not reclaim. Existence is checked explicitly rather than letting the foreign key reject the insert, so an orphan is counted as one instead of surfacing as an opaque constraint error. --- src/common/di.rs | 14 + src/infrastructure/services/mod.rs | 1 + .../services/thumb_attached_import_service.rs | 375 ++++++++++++++++++ 3 files changed, 390 insertions(+) create mode 100644 src/infrastructure/services/thumb_attached_import_service.rs diff --git a/src/common/di.rs b/src/common/di.rs index a31c0980..11fa4777 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -1494,6 +1494,20 @@ impl AppServiceFactory { .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) .await; + // Its file-keyed twin: `ext-{file_id}.jpg` previews the user uploaded, + // which no copy path duplicates today. Separate job, separate keying — + // routing these into the content-keyed table would share one user's + // preview onto every file with identical content. + let _ = Arc::new( + crate::infrastructure::services::thumb_attached_import_service::ThumbAttachedImport::new( + std::path::Path::new(&self.storage_path).join(".thumbnails"), + core.dedup_service.clone(), + maintenance_pool.clone(), + ), + ) + .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) + .await; + // Third recoverable-run tenant. Iterates `storage.files` // and reports parent-folder-trashed cascade misses, // `missing_blob` (data-loss indicator — file references diff --git a/src/infrastructure/services/mod.rs b/src/infrastructure/services/mod.rs index 8ff128d6..9ffaa836 100644 --- a/src/infrastructure/services/mod.rs +++ b/src/infrastructure/services/mod.rs @@ -57,6 +57,7 @@ pub mod session_liveness_gauges; pub mod share_unlock_cookie; pub mod smtp_email_sender; pub mod swappable_blob_backend; +pub mod thumb_attached_import_service; pub mod thumb_derived_import_service; pub mod thumbnail_service; #[cfg(test)] diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs new file mode 100644 index 00000000..cb7235f7 --- /dev/null +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -0,0 +1,375 @@ +//! `thumb_attached_import` — backfill `storage.file_attached_blobs` from the +//! `ext-{file_id}.jpg` sidecars that predate it. +//! +//! Second half of step 10's migration, and the twin of +//! `thumb_derived_import`. These are the thumbnails a *user* supplied — the +//! SPA's client-side generator, notably for PDFs, which have no server-side +//! render path at all. They live only as +//! `{thumbnails_root}/{size}/ext-{file_id}.jpg` on local disk. +//! +//! Until a row exists, a **copy of the file loses the preview**: the sidecar +//! is keyed by `file_id`, no copy path duplicates it, and the server silently +//! falls back to rendering from the source (or to nothing, for a PDF). That +//! is the bug `file_attached_blobs` closed for new uploads; this job closes +//! it for everything already on disk. +//! +//! ### File-keyed, and that is the whole point +//! +//! These bytes are **not** derivable from the file's content, so they must +//! never be content-keyed. Sharing one user's uploaded preview across every +//! file with identical content is the poisoning vector the table split +//! exists to prevent — see `docs/plan/derived-blobs.md`. `thumb_derived_import` +//! deliberately rejects `ext-` names for the same reason, and the two jobs +//! are separate so neither can drift into the other's keying. +//! +//! ### Idempotence needs care here +//! +//! Unlike the derived twin, `store_attached_blob` is `ON CONFLICT DO UPDATE`: +//! calling it for a row that already exists releases the previous reference +//! and takes a new one. Harmless once, but a job that did it on every run +//! would churn refcounts. So each file is skipped when a row is already +//! present, and the store is only reached on a genuine insert. +//! +//! ### Multi-instance caveat +//! +//! Sidecars are local, so this migrates only the instance it runs on. Phase 3 +//! must be gated on every instance reporting an empty tail. + +use std::path::PathBuf; +use std::sync::Arc; + +use async_trait::async_trait; +use bytes::Bytes; +use sqlx::PgPool; +use tokio::fs; +use uuid::Uuid; + +use crate::application::ports::thumbnail_ports::ThumbnailSize; +use crate::infrastructure::scheduler::{ + JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, + RunStatus, record_or_log, +}; +use crate::infrastructure::services::dedup_service::DedupService; + +pub const THUMB_ATTACHED_IMPORT_JOB_NAME: &str = "thumb_attached_import"; + +/// Files handled between checkpoints — a read plus at most a blob write each. +const BATCH_SIZE: usize = 100; + +/// `uploaded_by` for imported rows. +/// +/// Disk records no uploader, and the column is deliberately `NOT NULL` with no +/// FK so provenance survives a user deletion. A sentinel says "imported, real +/// uploader unknown" honestly; inventing an owner — the file's `created_by`, +/// say — would fabricate provenance that could later be read as evidence an +/// Editor replaced someone's preview. +const IMPORTED_UPLOADER: Uuid = Uuid::nil(); + +pub struct ThumbAttachedImport { + thumbnails_root: PathBuf, + dedup: Arc, + pool: Arc, +} + +impl ThumbAttachedImport { + pub fn new(thumbnails_root: PathBuf, dedup: Arc, pool: Arc) -> Self { + Self { + thumbnails_root, + dedup, + pool, + } + } + + pub async fn register_recoverable_job( + self: Arc, + registry: &JobRegistry, + provider: &Arc, + ) -> Arc { + registry + .register_recoverable_job(self.clone(), provider.clone(), None) + .await; + self + } + + /// The file id an external sidecar names, or `None` when the file is not + /// one of ours. + /// + /// Requires a parseable UUID: the name is about to be used as a foreign + /// key, and a malformed one should be reported rather than fed to the + /// database. + fn file_id_from_sidecar_name(name: &str) -> Option { + let stem = name.strip_prefix("ext-")?.strip_suffix(".jpg")?; + Uuid::parse_str(stem).ok() + } + + /// Sorted external-sidecar filenames for one size directory. + /// + /// Sorted because the cursor resumes by skipping everything at or before + /// it, which only works over a stable order. + async fn sidecar_names(&self, size: ThumbnailSize) -> Vec { + let dir = self.thumbnails_root.join(size.dir_name()); + let Ok(mut entries) = fs::read_dir(&dir).await else { + return Vec::new(); + }; + let mut names = Vec::new(); + while let Ok(Some(entry)) = entries.next_entry().await { + if let Some(name) = entry.file_name().to_str() + && Self::file_id_from_sidecar_name(name).is_some() + { + names.push(name.to_string()); + } + } + names.sort(); + names + } + + /// Does the file still exist? Checked explicitly rather than letting the + /// foreign key reject the insert, so an orphaned sidecar is *counted* as + /// an orphan instead of surfacing as an opaque constraint error. + async fn file_exists(&self, file_id: Uuid) -> bool { + sqlx::query_scalar::<_, i64>("SELECT 1 FROM storage.files WHERE id = $1") + .bind(file_id) + .fetch_optional(self.pool.as_ref()) + .await + .ok() + .flatten() + .is_some() + } +} + +#[async_trait] +impl RecoverableJobHandler for ThumbAttachedImport { + fn name(&self) -> &str { + THUMB_ATTACHED_IMPORT_JOB_NAME + } + + async fn count_total(&self) -> Option { + let mut total = 0u64; + for size in ThumbnailSize::all() { + total += self.sidecar_names(*size).await.len() as u64; + } + Some(total) + } + + async fn run_resumable( + &self, + store: &dyn JobStore, + _args: &JobRunArgs, + resume_cursor: Option>, + ) -> RunOutcome { + // Cursor is `{size_dir}/{filename}`, matching thumb_derived_import: + // sizes walk in `ThumbnailSize::all()` order and names are sorted + // within each, so the pair totally orders the traversal. + let cursor: Option = match resume_cursor { + None => None, + Some(b) if b.is_empty() => None, + Some(b) => match String::from_utf8(b) { + Ok(s) => Some(s), + Err(e) => { + return RunOutcome::Failed { + message: format!("invalid cursor: not valid UTF-8: {e}"), + }; + } + }, + }; + + let mut imported = 0u64; + let mut already = 0u64; + let mut orphaned = 0u64; + let mut failed = 0u64; + let mut since_checkpoint = 0usize; + + for size in ThumbnailSize::all() { + let dir_name = size.dir_name().to_string(); + for name in self.sidecar_names(*size).await { + let position = format!("{dir_name}/{name}"); + + if let Some(c) = &cursor + && position.as_str() <= c.as_str() + { + continue; + } + + match store.status().await { + Ok(RunStatus::CancelRequested) => { + return RunOutcome::Paused { + cursor: position.into_bytes(), + }; + } + Ok(_) => {} + Err(e) => { + return RunOutcome::Failed { + message: format!("status poll: {e}"), + }; + } + } + + let Some(file_id) = Self::file_id_from_sidecar_name(&name) else { + continue; + }; + let file_id_str = file_id.to_string(); + + // Already mapped. Checked BEFORE storing, because + // `store_attached_blob` is ON CONFLICT DO UPDATE and would + // release then retake the reference on every run. + if self + .dedup + .find_attached_blob(&file_id_str, "preview", &dir_name) + .await + .is_some() + { + already += 1; + } else if !self.file_exists(file_id).await { + // The file is gone; the sidecar outlived it. Reported + // rather than deleted — this job imports, it does not + // reclaim, and a destructive default on a migration is + // exactly what `no silent auto-repair` forbids. + orphaned += 1; + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "attached_sidecar_orphan", + "anomaly", + None, + serde_json::json!({ + "path": position, + "file_id": file_id_str, + "note": "no storage.files row; sidecar left in place for the operator", + }), + ) + .await; + } else { + let path = self.thumbnails_root.join(&dir_name).join(&name); + match fs::read(&path).await { + Ok(data) => { + match self + .dedup + .store_attached_blob( + &file_id_str, + "preview", + &dir_name, + // store_external_thumbnail re-encodes to + // JPEG before writing, so the extension + // is authoritative here. + "image/jpeg", + Bytes::from(data), + IMPORTED_UPLOADER, + ) + .await + { + Ok(_) => imported += 1, + Err(e) => { + failed += 1; + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "attached_import_failed", + "anomaly", + None, + serde_json::json!({ + "path": position, + "file_id": file_id_str, + "error": format!("{e}"), + "note": "sidecar left in place; safe to re-run", + }), + ) + .await; + } + } + } + Err(e) => { + failed += 1; + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "attached_sidecar_unreadable", + "anomaly", + None, + serde_json::json!({ + "path": position, + "error": format!("{e}"), + }), + ) + .await; + } + } + } + + since_checkpoint += 1; + if since_checkpoint >= BATCH_SIZE { + if let Err(e) = store + .checkpoint(position.clone().into_bytes(), since_checkpoint as u64) + .await + { + return RunOutcome::Failed { + message: format!("checkpoint: {e}"), + }; + } + since_checkpoint = 0; + } + } + } + + tracing::info!( + target: "oxicloud::dedup", + event = "thumb_attached_import.completed", + run_id = %store.run_id(), + imported = imported, + already_present = already, + orphaned = orphaned, + failed = failed, + "thumb_attached_import: {imported} imported, {already} already present, \ + {orphaned} orphaned, {failed} failed" + ); + + RunOutcome::completed() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const UUID: &str = "3f2b1c00-1111-2222-3333-444455556666"; + const HASH: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + + #[test] + fn accepts_an_external_sidecar_name() { + assert_eq!( + ThumbAttachedImport::file_id_from_sidecar_name(&format!("ext-{UUID}.jpg")), + Some(Uuid::parse_str(UUID).unwrap()) + ); + } + + /// The content-keyed sidecars belong to `thumb_derived_import`. Importing + /// one here would file-key bytes that are shared across every file with + /// the same content, so each such file would take its own reference to + /// content it does not own. + #[test] + fn rejects_content_keyed_and_malformed_names() { + for name in [ + format!("{HASH}.webp"), + format!("{HASH}.jpg"), + format!("ext-{UUID}.webp"), + format!("ext-{UUID}"), + "ext-not-a-uuid.jpg".to_string(), + format!("{UUID}.jpg"), + ] { + assert_eq!( + ThumbAttachedImport::file_id_from_sidecar_name(&name), + None, + "must not be imported as an attached preview: {name}" + ); + } + } + + /// The sentinel must be stable: rows carrying it are how an operator + /// tells an imported preview from one with real provenance. + #[test] + fn imported_uploader_is_the_nil_sentinel() { + assert_eq!( + IMPORTED_UPLOADER.to_string(), + "00000000-0000-0000-0000-000000000000" + ); + } +} From 49e7bb15a6ecf45bb832cdf1cfdafaffdee7d107 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 00:39:17 +0200 Subject: [PATCH 25/66] test(storage): cover the sidecar walk for both import jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both imports run over ONE directory, where the two legacy shapes sit side by side, so the property worth asserting spans them: together they must claim every real sidecar exactly once, and neither may take the other's. A job that drifted into the other's shape would content-key user-supplied bytes — sharing one user's uploaded preview onto every file with identical content — and no per-job test in isolation would notice. So the fixture is shared. `legacy_tree` builds a directory holding a content-keyed .webp pair, an ext- upload, and a stray README, and both test modules walk it: derived claims exactly the two hashes in sorted order, attached claims exactly the ext- file, the two sets are disjoint, and between them they account for all three real sidecars. `sidecar_names` became an associated function taking the root instead of reading `self`, which is what makes this testable at all — the walk is the half that decides which files a job claims, and it needed no pool to verify. Sorting is asserted rather than assumed, since the cursor resumes by skipping everything at or before it and a stable order is the only thing that makes that correct. A missing size directory is covered too: normal on a fresh install, and it must yield no work rather than abort the walk. Not covered here, and it needs a pooled fixture that does not exist: the round trip itself — store the blob, write the row, and confirm a COPY inherits the preview. That belongs in the API-level harness, where the legacy state can be manufactured through the real write path and then stripped. --- .../services/thumb_attached_import_service.rs | 51 ++++++++++- .../services/thumb_derived_import_service.rs | 89 +++++++++++++++++-- 2 files changed, 131 insertions(+), 9 deletions(-) diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index cb7235f7..2733a2db 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -106,8 +106,12 @@ impl ThumbAttachedImport { /// /// Sorted because the cursor resumes by skipping everything at or before /// it, which only works over a stable order. - async fn sidecar_names(&self, size: ThumbnailSize) -> Vec { - let dir = self.thumbnails_root.join(size.dir_name()); + /// + /// Takes the root rather than reading `self`, so the walk — the half that + /// decides which files this job claims, and therefore which keying they + /// get — is testable against a temp directory with no database in sight. + async fn sidecar_names(root: &std::path::Path, size: ThumbnailSize) -> Vec { + let dir = root.join(size.dir_name()); let Ok(mut entries) = fs::read_dir(&dir).await else { return Vec::new(); }; @@ -146,7 +150,9 @@ impl RecoverableJobHandler for ThumbAttachedImport { async fn count_total(&self) -> Option { let mut total = 0u64; for size in ThumbnailSize::all() { - total += self.sidecar_names(*size).await.len() as u64; + total += Self::sidecar_names(&self.thumbnails_root, *size) + .await + .len() as u64; } Some(total) } @@ -181,7 +187,7 @@ impl RecoverableJobHandler for ThumbAttachedImport { for size in ThumbnailSize::all() { let dir_name = size.dir_name().to_string(); - for name in self.sidecar_names(*size).await { + for name in Self::sidecar_names(&self.thumbnails_root, *size).await { let position = format!("{dir_name}/{name}"); if let Some(c) = &cursor @@ -341,6 +347,43 @@ mod tests { ); } + /// The other half of the partition. Reuses the same legacy tree as + /// `thumb_derived_import`'s test on purpose: the two jobs run over one + /// directory, so the property that matters is that together they claim + /// every real sidecar exactly once, and neither takes the other's. + #[tokio::test] + async fn walk_claims_only_uploaded_previews() { + use crate::infrastructure::services::thumb_derived_import_service::ThumbDerivedImport; + + let tmp = + crate::infrastructure::services::thumb_derived_import_service::tests::legacy_tree() + .await; + + let attached = ThumbAttachedImport::sidecar_names(tmp.path(), ThumbnailSize::Preview).await; + let derived = ThumbDerivedImport::sidecar_names(tmp.path(), ThumbnailSize::Preview).await; + + assert_eq!( + attached, + vec!["ext-3f2b1c00-1111-2222-3333-444455556666.jpg".to_string()], + "must claim the uploaded preview and nothing else" + ); + + // Disjoint: no file is imported under both keyings, which would take + // two references and — worse — content-key user-supplied bytes. + for a in &attached { + assert!( + !derived.contains(a), + "both jobs claimed {a}; keying would be ambiguous" + ); + } + // And nothing real is dropped: README.txt is the only unclaimed file. + assert_eq!( + attached.len() + derived.len(), + 3, + "the three real sidecars must be claimed exactly once between them" + ); + } + /// The content-keyed sidecars belong to `thumb_derived_import`. Importing /// one here would file-key bytes that are shared across every file with /// the same content, so each such file would take its own reference to diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index c01f7e31..6393b5a4 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -93,8 +93,12 @@ impl ThumbDerivedImport { /// /// Sorted so the cursor is meaningful: resume skips everything at or /// before it, which only works over a stable order. - async fn sidecar_names(&self, size: ThumbnailSize) -> Vec { - let dir = self.thumbnails_root.join(size.dir_name()); + /// + /// Takes the root rather than reading `self`, so the walk — the half that + /// decides which files this job claims, and therefore which keying they + /// get — is testable against a temp directory with no database in sight. + pub(crate) async fn sidecar_names(root: &std::path::Path, size: ThumbnailSize) -> Vec { + let dir = root.join(size.dir_name()); let Ok(mut entries) = fs::read_dir(&dir).await else { return Vec::new(); }; @@ -120,7 +124,9 @@ impl RecoverableJobHandler for ThumbDerivedImport { async fn count_total(&self) -> Option { let mut total = 0u64; for size in ThumbnailSize::all() { - total += self.sidecar_names(*size).await.len() as u64; + total += Self::sidecar_names(&self.thumbnails_root, *size) + .await + .len() as u64; } Some(total) } @@ -155,7 +161,7 @@ impl RecoverableJobHandler for ThumbDerivedImport { for size in ThumbnailSize::all() { let dir_name = variant_of(*size); - for name in self.sidecar_names(*size).await { + for name in Self::sidecar_names(&self.thumbnails_root, *size).await { let position = format!("{dir_name}/{name}"); // Resume: everything at or before the cursor is done. @@ -277,7 +283,11 @@ impl RecoverableJobHandler for ThumbDerivedImport { } #[cfg(test)] -mod tests { +// `pub(crate)` so the attached import's test can reuse `legacy_tree`. Both +// jobs walk ONE directory, so the property worth asserting spans them — that +// together they claim every sidecar exactly once — and that needs a shared +// fixture rather than two that can drift apart. +pub(crate) mod tests { use super::*; const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; @@ -290,6 +300,75 @@ mod tests { ); } + /// A legacy `.thumbnails` tree as it exists before the migration: both + /// sidecar shapes side by side in the same size directory, which is + /// exactly how they are written today. + /// + /// Returns the temp dir — the caller must hold it, or the directory is + /// removed while the test is still reading it. + pub(crate) async fn legacy_tree() -> tempfile::TempDir { + let tmp = tempfile::tempdir().expect("create temp dir"); + for size in ThumbnailSize::all() { + let dir = tmp.path().join(size.dir_name()); + tokio::fs::create_dir_all(&dir).await.unwrap(); + // Server-rendered, content-keyed. `b` sorts after `0a…`, so the + // pair also proves the listing is ordered rather than incidental. + tokio::fs::write(dir.join(format!("{H}.webp")), b"webp") + .await + .unwrap(); + tokio::fs::write( + dir.join("b111111111111111111111111111111111111111111111111111111111111111.webp"), + b"webp2", + ) + .await + .unwrap(); + // User-uploaded, file-keyed. + tokio::fs::write( + dir.join("ext-3f2b1c00-1111-2222-3333-444455556666.jpg"), + b"jpeg", + ) + .await + .unwrap(); + // Neither: a stray file that must be claimed by no one. + tokio::fs::write(dir.join("README.txt"), b"nope") + .await + .unwrap(); + } + tmp + } + + /// The migration's core invariant: this job claims the content-keyed + /// sidecars and *only* those, leaving the uploaded previews for + /// `thumb_attached_import`. Getting this wrong content-keys user-supplied + /// bytes, which shares one user's preview onto every file with identical + /// content. + #[tokio::test] + async fn walk_claims_only_content_keyed_sidecars_in_sorted_order() { + let tmp = legacy_tree().await; + let names = ThumbDerivedImport::sidecar_names(tmp.path(), ThumbnailSize::Preview).await; + + assert_eq!( + names, + vec![ + format!("{H}.webp"), + "b111111111111111111111111111111111111111111111111111111111111111.webp".to_string(), + ], + "must claim both content-keyed sidecars, sorted, and nothing else" + ); + } + + /// A missing size directory is normal on a fresh install and must not + /// abort the walk — the job simply has nothing to import. + #[tokio::test] + async fn missing_size_directory_yields_no_work() { + let tmp = tempfile::tempdir().expect("create temp dir"); + assert!( + ThumbDerivedImport::sidecar_names(tmp.path(), ThumbnailSize::Icon) + .await + .is_empty() + ); + } + /// `ext-` files are user-supplied and file-keyed. Importing one here /// would content-key it and share it across every file with identical /// content — the exact poisoning the table split prevents. From 671e6ac0e78e4e614f4484fdd26abbebdbb27904 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 00:49:49 +0200 Subject: [PATCH 26/66] test(api): end-to-end check of both sidecar import jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing exercised these jobs. Their unit tests cover the directory walk — which files each claims — but neither had ever executed a run. The test environment always starts fresh, so there is no pre-migration data to import. This creates it, and the reconstruction is EXACT rather than an imitation: the on-disk layout did not change in this work. A rendered thumbnail has always been written to {size}/{hash}.webp and an uploaded preview to {size}/ext-{id}.jpg; the only new thing is the row. So upload through the real API, then delete the row, and what remains on disk is byte-for-byte what a pre-migration install has. Deleting the row must also release the reference it held, or the manufactured state would carry a reference no legacy install ever had and storage_cleanup_check.sh would report a leak this script caused. file_attached_blobs has an ON DELETE trigger for that; content_derived_blobs does not — its Rust purge path releases explicitly — so the strip decrements it directly. Three assertions, in increasing order of what they catch: 1. Both rows come back, and the imported attached row carries the nil uploader sentinel rather than a fabricated one. 2. A COPY inherits the imported preview. This is the user-visible point and was impossible before the row existed: the ext- sidecar is keyed by file_id, no copy path duplicates it, so the copy silently fell back to a render. 3. Re-running imports nothing and changes no refcount. The likeliest silent defect — store_attached_blob is ON CONFLICT DO UPDATE, so an import that skipped its existence check would release and retake a reference every run, invisible except as drift. psql runs inside the compose container rather than depending on a host binary, matching how spawn-db.sh probes readiness. Ordered before storage_cleanup_check.sh, which deletes everything it needs. --- tests/api/run.sh | 4 + tests/api/thumb_import_check.sh | 213 ++++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100755 tests/api/thumb_import_check.sh diff --git a/tests/api/run.sh b/tests/api/run.sh index 3d17c303..75d89c34 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -243,6 +243,10 @@ if ! hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" -- exit 1 fi +# Migration check runs BEFORE the cleanup sweep, which deletes everything +# it would otherwise need. +bash "$API_DIR/thumb_import_check.sh" + bash "$API_DIR/storage_cleanup_check.sh" # ── 5. OPAQUE crypto handshake — the parts Hurl can't drive ───────────── diff --git a/tests/api/thumb_import_check.sh b/tests/api/thumb_import_check.sh new file mode 100755 index 00000000..6b42420f --- /dev/null +++ b/tests/api/thumb_import_check.sh @@ -0,0 +1,213 @@ +#!/usr/bin/env bash +# ============================================================= +# OxiCloud – legacy sidecar → blob migration (both import jobs) +# ============================================================= +# Exercises `thumb_derived_import` and `thumb_attached_import` end to end, +# which nothing else does: their unit tests cover only the directory walk, +# never a run. +# +# ── How legacy state is manufactured ───────────────────────────────────── +# +# The test environment always starts fresh, so there is no pre-migration +# data to import. We create it, and the reconstruction is EXACT rather than +# an imitation: the on-disk layout did not change in this work. A +# server-rendered thumbnail has always been written to +# `{size}/{hash}.webp`, and an uploaded preview to `{size}/ext-{id}.jpg`. +# The only thing that is new is the DB row. +# +# So: upload through the real API (which writes both the file and the row), +# then delete the row. What remains on disk is byte-for-byte what a +# pre-migration install has. +# +# Deleting the row must also release the reference it held, or the +# manufactured state would carry a reference no legacy install ever had and +# the end-of-suite registry check would report a leak that this script +# caused. `file_attached_blobs` has an ON DELETE trigger that does it; +# `content_derived_blobs` does not, so we decrement explicitly. +# +# ── What is asserted ───────────────────────────────────────────────────── +# +# 1. Both rows come back after the import. +# 2. The uploaded preview survives a COPY — the user-visible point of +# `file_attached_blobs`, and impossible before the row existed. +# 3. Re-running imports nothing and changes no refcount. This is the +# defect most likely to be silent: `store_attached_blob` is +# ON CONFLICT DO UPDATE, so an import that skipped its existence +# check would release and retake a reference on every run. +# +# Runs BEFORE storage_cleanup_check.sh, which deletes everything. +# +# Prerequisites: setup.hurl has run (admin exists); docker compose db up. +# ============================================================= + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +COMPOSE_FILE="$REPO_ROOT/tests/common/docker-compose.test.yml" + +# shellcheck source=test.env +source "$SCRIPT_DIR/test.env" + +log() { echo "[thumb-import] $*"; } +fail() { echo $'\e[31m'"[thumb-import] FAIL: $*"$'\e[0m' >&2; exit 1; } + +# psql inside the compose container — no host psql dependency, matching +# how spawn-db.sh probes readiness. +sql() { + docker compose -f "$COMPOSE_FILE" exec -T postgres-test \ + psql -U oxicloud_test -d oxicloud_test -tAqc "$1" +} + +TOKEN=$(curl -sf -X POST "$base_url/api/auth/login" \ + -H "Content-Type: application/json" \ + -d "{\"username\":\"$username\",\"password\":\"$password\"}" \ + | jq -r '.access_token') +[[ -n "$TOKEN" && "$TOKEN" != "null" ]] || fail "login failed" +AUTH="Authorization: Bearer $TOKEN" + +# ── 1. Create a file with BOTH sidecar shapes ──────────────────────────── + +SRC_FOLDER=$(curl -sf -X POST "$base_url/api/folders" -H "$AUTH" \ + -H "Content-Type: application/json" \ + -d '{"name":"hurl-import-src"}' | jq -r '.id') +DST_FOLDER=$(curl -sf -X POST "$base_url/api/folders" -H "$AUTH" \ + -H "Content-Type: application/json" \ + -d '{"name":"hurl-import-dst"}' | jq -r '.id') +[[ -n "$SRC_FOLDER" && "$SRC_FOLDER" != "null" ]] || fail "folder create failed" + +UPLOAD=$(curl -sf -X POST "$base_url/api/files/upload" -H "$AUTH" \ + -F "folder_id=$SRC_FOLDER" \ + -F "file=@$REPO_ROOT/tests/fixtures/red-image.png;type=image/png") +FILE_ID=$(echo "$UPLOAD" | jq -r '.id') +BLOB_HASH=$(echo "$UPLOAD" | jq -r '.content_hash') +[[ -n "$FILE_ID" && "$FILE_ID" != "null" ]] || fail "upload failed: $UPLOAD" +log "uploaded file=$FILE_ID hash=${BLOB_HASH:0:12}" + +# Render → writes {size}/{hash}.webp AND the content_derived_blobs row. +curl -sf -H "$AUTH" "$base_url/api/files/$FILE_ID/thumbnail/preview" -o /dev/null \ + || fail "render thumbnail failed" + +# Upload → writes ext-{file_id}.jpg AND the file_attached_blobs row. +curl -sf -X PUT -H "$AUTH" -H "Content-Type: image/png" \ + --data-binary "@$REPO_ROOT/tests/fixtures/green-image.png" \ + "$base_url/api/files/$FILE_ID/thumbnail/preview" -o /dev/null \ + || fail "upload thumbnail failed" + +UPLOADED_THUMB=$(mktemp) +curl -sf -H "$AUTH" "$base_url/api/files/$FILE_ID/thumbnail/preview" -o "$UPLOADED_THUMB" + +DERIVED_BEFORE=$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';") +ATTACHED_BEFORE=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") +[[ "$DERIVED_BEFORE" -ge 1 ]] || fail "expected a content_derived_blobs row before stripping" +[[ "$ATTACHED_BEFORE" -ge 1 ]] || fail "expected a file_attached_blobs row before stripping" +log "rows present before stripping: derived=$DERIVED_BEFORE attached=$ATTACHED_BEFORE" + +# ── 2. Strip the rows → this IS the legacy state ───────────────────────── +# +# Release each reference as the row goes, so the manufactured state matches +# a pre-migration install rather than carrying references it never had. +# file_attached_blobs does this via its ON DELETE trigger; the derived table +# has no trigger (its Rust purge path releases explicitly), so do it here. + +sql "WITH gone AS ( + DELETE FROM storage.content_derived_blobs + WHERE source_hash='$BLOB_HASH' + RETURNING blob_hash + ) + UPDATE storage.chunk_manifests m + SET ref_count = GREATEST(m.ref_count - 1, 0) + FROM gone WHERE m.file_hash = gone.blob_hash;" >/dev/null + +sql "DELETE FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';" >/dev/null + +[[ "$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';")" == "0" ]] \ + || fail "derived row survived the strip" +[[ "$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';")" == "0" ]] \ + || fail "attached row survived the strip" +log "legacy state manufactured: files on disk, no rows." + +# ── 3. Run the imports ─────────────────────────────────────────────────── + +for job in thumb_derived_import thumb_attached_import; do + curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/$job/trigger" >/dev/null \ + || fail "$job trigger failed" + log "$job triggered." +done + +DERIVED_AFTER=$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';") +ATTACHED_AFTER=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") +[[ "$DERIVED_AFTER" -ge 1 ]] || fail "thumb_derived_import did not restore the row" +[[ "$ATTACHED_AFTER" -ge 1 ]] || fail "thumb_attached_import did not restore the row" +log "rows restored: derived=$DERIVED_AFTER attached=$ATTACHED_AFTER" + +# Provenance: imported rows carry the sentinel, which is how an operator +# tells them from previews with a real uploader. +UPLOADER=$(sql "SELECT uploaded_by FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") +[[ "$UPLOADER" == "00000000-0000-0000-0000-000000000000" ]] \ + || fail "imported row should carry the nil uploader sentinel, got '$UPLOADER'" + +# ── 4. The user-visible point: a COPY inherits the preview ─────────────── +# +# Impossible before the row existed — the ext- sidecar is keyed by file_id +# and no copy path duplicates it, so the copy fell back to a render. + +COPY_ID=$(curl -sf -X POST "$base_url/api/batch/files/copy" -H "$AUTH" \ + -H "Content-Type: application/json" \ + -d "{\"file_ids\":[\"$FILE_ID\"],\"target_folder_id\":\"$DST_FOLDER\"}" \ + | jq -r '.successful[0].id') +[[ -n "$COPY_ID" && "$COPY_ID" != "null" ]] || fail "copy failed" + +COPY_THUMB=$(mktemp) +curl -sf -H "$AUTH" "$base_url/api/files/$COPY_ID/thumbnail/preview" -o "$COPY_THUMB" +cmp -s "$UPLOADED_THUMB" "$COPY_THUMB" \ + || fail "copy did not inherit the imported preview" +log "copy inherits the imported preview." + +# ── 5. Idempotence: a second run imports nothing and churns nothing ────── +# +# The likely silent defect. store_attached_blob is ON CONFLICT DO UPDATE, so +# an import that skipped its existence check would release and retake a +# reference every run — invisible except as refcount drift. + +ATTACHED_HASH=$(sql "SELECT blob_hash FROM storage.file_attached_blobs WHERE file_id='$FILE_ID' LIMIT 1;") +[[ -n "$ATTACHED_HASH" ]] || fail "no attached blob_hash to check refcounts against" +# A single-chunk blob has a manifest whose file_hash equals its own hash, so +# this is the counter add_reference actually touches. `-` rather than empty +# keeps the later comparison meaningful if the manifest is unexpectedly absent. +REFS_BEFORE=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$ATTACHED_HASH';") +REFS_BEFORE=${REFS_BEFORE:--} + +for job in thumb_derived_import thumb_attached_import; do + curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/$job/trigger" >/dev/null \ + || fail "$job re-trigger failed" +done + +REFS_AFTER=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$ATTACHED_HASH';") +REFS_AFTER=${REFS_AFTER:--} +DERIVED_2=$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';") +ATTACHED_2=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") + +[[ "$REFS_AFTER" == "$REFS_BEFORE" ]] \ + || fail "re-run changed the attached blob refcount: $REFS_BEFORE → $REFS_AFTER" +[[ "$DERIVED_2" == "$DERIVED_AFTER" ]] || fail "re-run duplicated derived rows" +[[ "$ATTACHED_2" == "$ATTACHED_AFTER" ]] || fail "re-run duplicated attached rows" +log "re-run is a no-op: rows and refcounts unchanged." + +# ── 6. Teardown ────────────────────────────────────────────────────────── +# Everything created here must go — one database serves the whole suite, +# and storage_cleanup_check.sh afterwards asserts the registry drains to +# zero. + +rm -f "$UPLOADED_THUMB" "$COPY_THUMB" + +for folder in "$SRC_FOLDER" "$DST_FOLDER"; do + curl -sf -X DELETE -H "$AUTH" "$base_url/api/folders/$folder" -o /dev/null || true +done +TRASH=$(curl -sf -H "$AUTH" "$base_url/api/trash/resources" || echo '{}') +for folder in "$SRC_FOLDER" "$DST_FOLDER"; do + tid=$(echo "$TRASH" | jq -r --arg id "$folder" '.items[]? | select(.resource.id == $id) | .resource.id') + [[ -n "$tid" ]] && curl -sf -X DELETE -H "$AUTH" "$base_url/api/trash/$tid" -o /dev/null || true +done + +log "OK — both imports restore their rows, the copy inherits the preview, and re-running is a no-op." From 4ae1531286c56c0a2e9a449939b12ba1131d8c74 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 09:07:45 +0200 Subject: [PATCH 27/66] docs(plan): job-driven sidecar deletion, and the persist-consolidation blocker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two revisions from working through step 10. **Deletion moves into the import jobs, not a release.** Sidecars are local disk, so a release cannot know whether every instance has drained — gating on "an empty tail" asks an operator to coordinate a fact nothing reports, and there is no telling when or whether they trigger the jobs at all. Each job unlinking what it has imported makes every instance drain itself. Constrained three ways: verify the derived blob reads back before unlinking (a store that reported success but landed unreadable would otherwise take the last copy), only after the read-order flip (or the derived tier takes its first production traffic by accident), and opt-in, since a migration that deletes by default is surprising. Scheduled tick rather than boot trigger — idempotent and resumable, so periodic is safe, while walking .thumbnails/ at startup delays readiness for nothing. **Found while checking the dual-write assumption: it does not hold.** store_derived_blob has ONE call site; fs::write(&thumb_path, …) has five. get_thumbnail, generate_and_persist and generate_all_sizes_background all persist sidecar-only. That breaks the migration's premise rather than being untidy — on-demand renders keep producing un-migrated state after the import runs, so the tail never empties and the deletion gate never opens. One persist_thumbnail owning sidecar + derived + moka is therefore a prerequisite, and it makes "stop writing sidecars" a later one-line change instead of four edits. Noted that ThumbnailService holds no DedupService, so it must be threaded through. Also corrects a claim I put in thumb_derived_import's own docs: transcoding is NOT a later step. ImageTranscodeService exists and caches .transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it must re-key file→content — legitimate only because a transcode is derivable. Its .skip markers remain an open question. --- docs/plan/derived-blobs.md | 104 ++++++++++++++++-- .../services/thumb_derived_import_service.rs | 10 +- 2 files changed, 101 insertions(+), 13 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 17c0b65c..e3f31686 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1151,8 +1151,72 @@ filesystem and a remote backend for hours). It sweeps the cold tail: reference forever. - Reports imported / skipped-orphan / skipped-jpeg / failed counts. -**Phase 3 (release N+1): delete** the fallback module and the sidecar -directories, gated on the job reporting an empty tail. +**Phase 3: the job deletes, not a release.** *(revised 2026-08-26 — +supersedes "delete in release N+1, gated on an empty tail")* + +Sidecars are **local disk**. A release cannot know whether every +instance has drained, so gating deletion on "the tail is empty" asks an +operator to coordinate a fact nothing reports — and there is no way to +know when, or whether, they will trigger the jobs at all. Instead the +import unlinks each sidecar it has successfully imported, so **each +instance drains itself** and the directory becomes removable once +genuinely empty. + +Three constraints on that: + +- **Verify readback before unlinking.** Import → read the derived blob + back through the normal stack → *then* delete. A store that reported + success but landed unreadable would otherwise take the last copy with + it. Cheap next to the decode already performed, and it is the + difference between a migration and a data-loss bug. +- **Only after the read-order flip.** Deleting while the sidecar is + still read *first* sends reads to the derived tier as a side effect of + the migration — its first production traffic arriving by accident + rather than by decision. +- **Opt-in** (`?delete_imported=true`). A migration that deletes on its + default setting is surprising, and it is the same instinct as + no-silent-auto-repair: early runs import only, so an operator can + inspect before committing. + +Register it as a **scheduled tick**, not a boot-time trigger: it is +idempotent and resumable, so periodic is safe, whereas walking a large +`.thumbnails/` during startup delays readiness for nothing. + +The only remaining *release* is removing the fallback read path once the +directories are empty — by which point no data is at stake. + +### Prerequisite: one persist function (found 2026-08-26) + +**Four render paths write a sidecar; only one also writes the derived +row.** `store_derived_blob` has a single call site — in +`render_and_persist_all_webp` — while `fs::write(&thumb_path, …)` has +five. `get_thumbnail`, `generate_and_persist` and +`generate_all_sizes_background` all persist sidecar-only. + +That breaks the migration's premise rather than merely being untidy: an +on-demand render (cache miss, a size never generated, an evicted +sidecar) keeps producing un-migrated state *after* the import runs, so +the tail never empties and the deletion gate never opens. + +So before the imports can converge, all render paths must go through +**one** `persist_thumbnail` that writes the sidecar, the derived blob +and the moka entry together — the same single-source move as +`storage.copy_file_satellites`. What it writes then becomes a policy in +one place, so "stop writing sidecars" is later a one-line change rather +than four edits. + +Interim setting is **dual-write**, for two reasons: it is what makes the +backlog finite, and it leaves reads untouched while the derived tier is +still unproven. Cost is one extra local `fs::write` per render, +negligible beside the decode. Note the sidecar *read* path must survive +until the directories are empty regardless, so stopping the write early +buys nothing. + +Cost to be aware of: `ThumbnailService` holds no `DedupService` — it is +a per-call parameter (`dedup: Option<&DedupService>`) — so the +consolidation threads it through those paths, and +`generate_and_persist` takes a `thumb_path` where it will need the +`blob_hash` instead. ### The "just delete it" opt-out is no longer universally safe @@ -1211,13 +1275,35 @@ hardcoded SQL). New sources bolt on independently. table. Lands with step 5. File-keyed, never in `content_derived_blobs`. Register it in `copy_file_satellites` and declare its version semantics. -10. **`derived_import` job + the dual-read fallback** — see the - migration section. Phase 3 (deleting the fallback and the sidecar - dirs) is a separate later release, gated on an empty tail. **The - HTTP ETag moves to the derived hash here**, with the read-order - flip and not before — see "HTTP ETag" under *Read path and - caching* for why keying it earlier would make the ETag describe a - tier the response did not come from. +10. **Import jobs + the dual-read fallback** — see the migration + section. Revised ordering as of 2026-08-26: + + a. **Consolidate onto one `persist_thumbnail`** (dual-write). The + blocker: today only one of four render paths writes the derived + row, so the import can never converge. See *Prerequisite: one + persist function*. + b. **`thumb_derived_import`** (shipped) and **`thumb_attached_import`** + (shipped) — two jobs, not one, because the keying differs and that + difference is the security boundary. A third, `transcode_import`, + is still needed: `ImageTranscodeService` **already exists** and + caches `.transcoded/{ext}/{file_id}.{ext}`, so those must be + **re-keyed** file→content on import (legitimate only because a + transcode is derivable). Its `.skip` markers — a cached negative + verdict with no bytes — remain an open question. + c. **Flip the read order**, derived first. The HTTP ETag's + source-keyed fallback becomes unreachable here; the attached and + derived halves already landed early, forced by the attachment + case (see *HTTP ETag*). + d. **Enable deletion** in the import jobs (opt-in, readback-verified). + e. **Remove the fallback read path** once the directory no longer + *exists* — not merely once it is empty. Two reasons. Empty is a + momentary property an on-demand render can undo, whereas absence + is one-way and observable, so the job removes the directory after + draining it and that absence is the proof. And it is far cheaper + to test: existence is a single `stat`, while emptiness costs an + `opendir`/`readdir`/`closedir` — which matters if the fallback + ever gates on it per read rather than once at boot. The only + remaining release, and no data is at stake by then. 11. **`DedupService` → `BlobHandler` rename** — decided, mechanical, 34 files. Standalone commit, `src/AGENTS.md` updated with it. Can land at any point; last is easiest, since every earlier slice diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 6393b5a4..cf0d95dd 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -9,10 +9,12 @@ //! and records the mapping, after which the derived tier can become //! authoritative and the sidecar can be deleted. //! -//! **Thumbnails only, and that is permanent.** The table also holds -//! `kind = 'transcode'`, but transcoding lands *after* this migration, so -//! transcodes are born into the table and never pass through a sidecar era. -//! This job will not grow a transcode arm. +//! **Thumbnails only.** The table also holds `kind = 'transcode'`, and those +//! need their own import — `ImageTranscodeService` already exists and caches +//! to `.transcoded/{ext}/{file_id}.{ext}`, a different tree with a different +//! key. Importing them means **re-keying** file→content, which is legitimate +//! only because a transcode is derivable from the source bytes. Separate job; +//! this one will not grow a transcode arm. //! //! ### Idempotent by construction //! From 395296a7e74096300df02aa9f0ab8e6b0b1c36e5 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 10:03:10 +0200 Subject: [PATCH 28/66] fix(storage): file_exists misreported every file as missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `SELECT 1 FROM storage.files WHERE id = $1` decoded as i64. PostgreSQL types a bare `1` as int4, so the decode always failed — and since `.ok().flatten()` turns a decode error into the same None as "no row", file_exists reported false for every file. thumb_attached_import therefore classified every sidecar as an orphan and imported nothing. Caught by thumb_import_check.sh on its first run: the derived import restored its rows, the attached one restored none. Now `SELECT EXISTS(...)`, which yields a real bool and always returns exactly one row, so absence means absence. A query error still degrades to false — the safe direction, leaving the file on disk as a reported orphan rather than importing it against a row that may not exist. The failure mode is the point, and it is the third of this shape in two days: an error converted into an innocuous-looking outcome. So the check script now dumps a job's findings when an assertion fails. The jobs already recorded exactly why they skipped each file — the attached_sidecar_orphan findings naming the cause were sitting in the run while the script reported only "did not restore the row", which is indistinguishable from the job never having run. --- .../services/thumb_attached_import_service.rs | 19 +++++++++---- tests/api/thumb_import_check.sh | 27 ++++++++++++++++++- 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index 2733a2db..0486f660 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -130,14 +130,23 @@ impl ThumbAttachedImport { /// Does the file still exist? Checked explicitly rather than letting the /// foreign key reject the insert, so an orphaned sidecar is *counted* as /// an orphan instead of surfacing as an opaque constraint error. + /// `SELECT EXISTS(...)`, deliberately, rather than `SELECT 1 … LIMIT 1`. + /// + /// PostgreSQL types a bare `1` as `int4`, so decoding it as `i64` fails — + /// and because a decode error is indistinguishable from "no row" once + /// swallowed, every sidecar would be misreported as an orphan and nothing + /// would import. `EXISTS` yields a real `bool` and always returns exactly + /// one row, so absence means absence. + /// + /// A query error still degrades to `false`, which is the safe direction: + /// the file is reported as an orphan and left on disk for the operator, + /// rather than imported against a row that may not exist. async fn file_exists(&self, file_id: Uuid) -> bool { - sqlx::query_scalar::<_, i64>("SELECT 1 FROM storage.files WHERE id = $1") + sqlx::query_scalar::<_, bool>("SELECT EXISTS(SELECT 1 FROM storage.files WHERE id = $1)") .bind(file_id) - .fetch_optional(self.pool.as_ref()) + .fetch_one(self.pool.as_ref()) .await - .ok() - .flatten() - .is_some() + .unwrap_or(false) } } diff --git a/tests/api/thumb_import_check.sh b/tests/api/thumb_import_check.sh index 6b42420f..f7978972 100755 --- a/tests/api/thumb_import_check.sh +++ b/tests/api/thumb_import_check.sh @@ -50,7 +50,32 @@ COMPOSE_FILE="$REPO_ROOT/tests/common/docker-compose.test.yml" source "$SCRIPT_DIR/test.env" log() { echo "[thumb-import] $*"; } -fail() { echo $'\e[31m'"[thumb-import] FAIL: $*"$'\e[0m' >&2; exit 1; } + +# Dump a job's findings before dying. Without this, an import that ran but +# imported nothing looks identical to one that never ran — and the jobs +# record precisely why they skipped a file (orphan, unreadable, store +# failed). The first failure of this script was a misreported orphan, and +# the finding naming it was sitting in the run the whole time. +dump_findings() { + local job="$1" run_id findings + run_id=$(curl -sf -H "$AUTH" "$base_url/api/admin/jobs/$job/runs?limit=1" 2>/dev/null \ + | jq -r 'if type == "array" then .[0].id else ((.runs // .items // [])[0].id) end // empty') + [[ -z "$run_id" ]] && { echo " ($job: no run found)" >&2; return; } + findings=$(curl -sf -H "$AUTH" \ + "$base_url/api/admin/jobs/$job/runs/$run_id/findings?limit=20" 2>/dev/null || echo '[]') + echo " $job findings:" >&2 + echo "$findings" | jq -r \ + 'if type == "array" then .[] else (.findings // .items // [])[] end + | " \(.kind // .finding_kind // "?") \(.details // {} | tostring)"' 2>/dev/null >&2 \ + || echo " (unparseable)" >&2 +} + +fail() { + echo $'\e[31m'"[thumb-import] FAIL: $*"$'\e[0m' >&2 + dump_findings thumb_derived_import + dump_findings thumb_attached_import + exit 1 +} # psql inside the compose container — no host psql dependency, matching # how spawn-db.sh probes readiness. From 7ec387003daddb78ce98280757eb98c3b017eec3 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 21:29:25 +0200 Subject: [PATCH 29/66] refactor(thumbnails): one persist_rendered for every render path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 10(a), the blocker. Four render paths each wrote the sidecar and exactly one also recorded the content_derived_blobs row, so an on-demand render — a cache miss, a size never generated, an evicted sidecar — produced state the migration could never see. That breaks the migration's premise rather than being untidy: thumb_derived_import would never reach an empty tail, so the gate for deleting the sidecar would never open. Now every rendered thumbnail goes through persist_rendered, which owns what persisting means. Raw `fs::write(&thumb_path, …)` drops from five sites to two: the one inside persist_rendered, and store_external_thumbnail's `ext-{file_id}.jpg`, which is file-keyed and legitimately a different thing. The path that matters most already had what it needed: get_thumbnail_from_blob — the REST handler's fallthrough on a cache miss — holds `dedup` and simply never used it for persistence. It now dual-writes at no cost. render_and_persist_all_webp had its own copy of the dual-write logic; that copy is gone, so retiring the interim dual-write later is one edit here rather than a hunt. Two paths still pass `None` and remain sidecar-only: `get_thumbnail` (renders from an on-disk original) and `generate_all_sizes_background` (the path variant; the _from_blob sibling has dedup). Closing those means threading a DedupService in from their callers. Left visible as an explicit `None` at the call site rather than an absent write — the gap is now something a reader trips over instead of something they have to notice is missing. Adds ThumbnailFormat::mime() beside ext(), since the derived row needs a media type and an extension without a matching one is how a WebP ends up labelled JPEG. --- src/application/ports/thumbnail_ports.rs | 12 ++ .../services/thumbnail_service.rs | 166 ++++++++++++------ 2 files changed, 126 insertions(+), 52 deletions(-) diff --git a/src/application/ports/thumbnail_ports.rs b/src/application/ports/thumbnail_ports.rs index 2bd20c9b..195f384b 100644 --- a/src/application/ports/thumbnail_ports.rs +++ b/src/application/ports/thumbnail_ports.rs @@ -94,6 +94,18 @@ impl ThumbnailFormat { } } + /// Media type, for `content_derived_blobs.content_type` and for any + /// response serving these bytes. + /// + /// Beside `ext` deliberately: the two must agree, and an extension + /// without a matching media type is how a WebP ends up labelled JPEG. + pub fn mime(self) -> &'static str { + match self { + ThumbnailFormat::Webp => "image/webp", + ThumbnailFormat::Jpeg => "image/jpeg", + } + } + /// Pick the output format from a request `Accept` header: WebP when the /// client advertises `image/webp`, JPEG otherwise. A plain substring check /// is sufficient — no client sends `image/webp;q=0`, and every WebP-capable diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index dee210e0..3cfbff57 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -261,6 +261,71 @@ impl ThumbnailService { .join(format!("{}.{}", blob_hash, format.ext())) } + /// Persist a freshly rendered thumbnail to every durable tier. + /// + /// **One place that knows what persisting a thumbnail means.** Before + /// this, four render paths each wrote the sidecar and exactly one also + /// recorded the `content_derived_blobs` row, so an on-demand render — a + /// cache miss, a size never generated, an evicted sidecar — produced + /// state the migration could never see. That is not untidy, it breaks + /// the migration's premise: `thumb_derived_import` would never reach an + /// empty tail, and the gate for deleting the sidecar would never open. + /// + /// Scope is deliberately *durable* tiers only. The moka entry is left to + /// callers because several persist through `cache.entry().or_insert_with`, + /// which already owns the insert; doing it here too would write twice. + /// + /// Dual-write is the interim setting, not the destination. Once the + /// derived tier is authoritative and the imports have drained, dropping + /// the sidecar becomes a one-line change *here* rather than four edits + /// spread across the file — which is the point of consolidating first. + /// + /// Both writes are best-effort and logged: the bytes are already rendered + /// and about to be served, so a persistence failure must cost a + /// re-render later, never the response now. `dedup: None` means + /// sidecar-only — a caller that could not supply one, which is visible at + /// the call site rather than hidden as a missing line. + async fn persist_rendered( + &self, + blob_hash: &str, + size: ThumbnailSize, + format: ThumbnailFormat, + bytes: &Bytes, + dedup: Option<&DedupService>, + ) { + let thumb_path = self.get_thumbnail_path(blob_hash, size, format); + if let Some(parent) = thumb_path.parent() { + let _ = fs::create_dir_all(parent).await; + } + if let Err(e) = fs::write(&thumb_path, bytes).await { + tracing::warn!( + "Failed to save thumbnail sidecar {} {:?}: {e}", + &blob_hash[..blob_hash.len().min(12)], + size + ); + } + + if let Some(dedup) = dedup + && let Err(e) = dedup + .store_derived_blob( + blob_hash, + "thumbnail", + size.dir_name(), + format.mime(), + bytes.clone(), + ) + .await + { + tracing::warn!( + target: "oxicloud::dedup", + error = %e, + "failed to record derived blob for {} {:?}", + &blob_hash[..blob_hash.len().min(12)], + size, + ); + } + } + /// Get a thumbnail, generating it if needed. /// /// # Arguments @@ -284,6 +349,7 @@ impl ThumbnailService { let thumb_path = self.get_thumbnail_path(blob_hash, size, format); let original_owned = original_path.to_path_buf(); let file_id_owned = file_id.to_string(); + let blob_hash_owned = blob_hash.to_string(); // Moka's entry().or_insert_with() guarantees that for the same key // only ONE init closure runs; concurrent callers await the same @@ -306,11 +372,11 @@ impl ThumbnailService { tracing::info!("🎨 Generating thumbnail: {} {:?}", file_id_owned, size); match self.generate_thumbnail(&original_owned, size, format).await { Ok(bytes) => { - // Save to disk (best-effort — don't fail the request) - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - let _ = fs::write(&thumb_path, &bytes).await; + // `None`: renders from an on-disk original and holds + // no DedupService, so sidecar-only. Visible here + // rather than absent. + self.persist_rendered(&blob_hash_owned, size, format, &bytes, None) + .await; bytes } Err(e) => { @@ -357,6 +423,7 @@ impl ThumbnailService { let thumb_path = self.get_thumbnail_path(blob_hash, size, format); let file_id_owned = file_id.to_string(); + let blob_hash_owned = blob_hash.to_string(); let entry = self .cache @@ -375,8 +442,19 @@ impl ThumbnailService { tracing::warn!("Decode semaphore closed, skipping {}", file_id_owned); return Bytes::new(); }; - self.generate_and_persist(&file_id_owned, &thumb_path, size, format, original_data) - .await + // `None`: this entry point takes the original bytes directly + // and has no DedupService, so it persists sidecar-only. The + // gap is visible here rather than hidden as a missing write, + // and closing it means threading dedup in from its callers. + self.generate_and_persist( + &file_id_owned, + &blob_hash_owned, + size, + format, + original_data, + None, + ) + .await }) .await; @@ -440,8 +518,19 @@ impl ThumbnailService { return Bytes::new(); } }; - self.generate_and_persist(&file_id_owned, &thumb_path, size, format, original_data) - .await + // The on-demand render the REST handler falls through to on a + // cache miss — the busiest path that previously wrote a + // sidecar and no row. `dedup` is already in scope here, so + // dual-writing costs nothing. + self.generate_and_persist( + &file_id_owned, + &blob_hash_owned, + size, + format, + original_data, + Some(dedup.as_ref()), + ) + .await }) .await; @@ -464,10 +553,11 @@ impl ThumbnailService { async fn generate_and_persist( &self, file_id: &str, - thumb_path: &Path, + blob_hash: &str, size: ThumbnailSize, format: ThumbnailFormat, original_data: Bytes, + dedup: Option<&DedupService>, ) -> Bytes { tracing::info!("🎨 Generating thumbnail: {} {:?}", file_id, size); match Self::generate_thumbnail_from_data( @@ -479,10 +569,8 @@ impl ThumbnailService { .await { Ok(bytes) => { - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - let _ = fs::write(&thumb_path, &bytes).await; + self.persist_rendered(blob_hash, size, format, &bytes, dedup) + .await; bytes } Err(e) => { @@ -1202,13 +1290,12 @@ impl ThumbnailService { // Save each size to disk and populate moka — both keyed by // blob_hash, so the two tiers agree and a copy shares them. for (size, bytes) in thumbnails { - let thumb_path = self.get_thumbnail_path(&blob_hash, size, ThumbnailFormat::Webp); - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - if let Err(e) = fs::write(&thumb_path, &bytes).await { - tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e); - } else { + // `None`: this variant renders from a path and holds no + // DedupService — `generate_all_sizes_background_from_blob` is + // the one that does. Sidecar-only, visibly so. + self.persist_rendered(&blob_hash, size, ThumbnailFormat::Webp, &bytes, None) + .await; + { // Populate in-memory cache for instant first-hit serving let cache_key = ThumbnailCacheKey::content(&blob_hash, size, ThumbnailFormat::Webp); @@ -1346,37 +1433,12 @@ impl ThumbnailService { }; for (size, bytes) in thumbnails { - let thumb_path = self.get_thumbnail_path(blob_hash, size, ThumbnailFormat::Webp); - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - if let Err(e) = fs::write(&thumb_path, &bytes).await { - tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e); - continue; - } - - // Tier-3 copy. Best-effort and logged: a failure here must not - // cost the user their thumbnail, which is already on disk and in - // the cache. `derived_import` sweeps anything missed. - if let Some(dedup) = dedup - && let Err(e) = dedup - .store_derived_blob( - blob_hash, - "thumbnail", - size.dir_name(), - "image/webp", - bytes.clone(), - ) - .await - { - tracing::warn!( - target: "oxicloud::dedup", - error = %e, - "failed to record derived blob for {} {:?}", - file_id, - size, - ); - } + // Was the only path that wrote both tiers, with its own copy of + // the logic. Now the same `persist_rendered` every other render + // path uses, so there is one definition of what persisting means + // and the interim dual-write can be retired in one place. + self.persist_rendered(blob_hash, size, ThumbnailFormat::Webp, &bytes, dedup) + .await; let cache_key = ThumbnailCacheKey::content(blob_hash, size, ThumbnailFormat::Webp); self.cache.insert(cache_key, bytes).await; From 18649eb7b96f63a1d130d113fc3aa205d1289b6b Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 21:42:39 +0200 Subject: [PATCH 30/66] fix(thumbnails): drop derived-hash ETag until the read-order flip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ETag is computed BEFORE the body. On a cache miss no content_derived_blobs row exists, so thumbnail_content_id returned the source-keyed form — then rendering created that row, and the next request resolved to the derived hash instead. The validator changed as a side effect of producing the body, making every first render immediately stale. Latent until aaf08532: before the consolidation, the on-demand render path never wrote a derived row, so the flip had nothing to trigger it. Fixing one gap exposed the other. Caught by thumbnail_etag_content_keyed.hurl — two consecutive GETs of an unchanged file stopped revalidating to 304. The plan already said derived-hash keying must land WITH the read-order flip and not before; I brought it forward anyway when the attachment case forced the attached half. This is the evidence for the constraint, so the plan now records the attempt and why it failed rather than leaving the note as untested caution. The attached lookup stays — it has no such window, since an upload writes its row synchronously before any read can observe it, and it fixes a real collision: a copy inherits the source hash, so an original and a copy carrying different uploaded previews would otherwise share one validator while serving different bytes. The flip removes the hazard for the derived half too: once that tier is authoritative it is populated before it is consulted, so no row can appear between two reads. --- docs/plan/derived-blobs.md | 16 ++++++++ .../services/thumbnail_service.rs | 39 +++++++++++-------- 2 files changed, 38 insertions(+), 17 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index e3f31686..db386840 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -991,6 +991,22 @@ yet generated and for formats the derived tier does not hold. The `LEFT JOIN` in step 2 above already returns the derived hash in the same query, so the ETag costs no extra round-trip. +**Attempted early (2026-08-26) and reverted — the constraint above is +load-bearing.** The attached half shipped and is correct, because an +upload writes its row synchronously before any read can observe it. The +*derived* half was brought forward at the same time and had to be backed +out: the ETag is computed **before** the body, so on a cache miss no row +exists and the handler emits the source-keyed form — then rendering +creates the row, and the very next request resolves to the derived hash. +The validator changed as a side effect of producing the body, so every +first render was immediately stale. Caught by +`thumbnail_etag_content_keyed.hurl`, where two consecutive GETs of an +unchanged file stopped revalidating to 304. + +The flip is what removes the hazard: once the derived tier is +authoritative it is populated before it is consulted, so there is no +window in which the row appears between two reads. + **The disk cache is `CachedBlobBackend`, reused unchanged.** No thumbnail-specific cache, no second root path. Routing derived blobs through the same stack gets, for free: diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 3cfbff57..6c2a9270 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -607,14 +607,26 @@ impl ThumbnailService { /// `immutable` set, clients would never revalidate. Worse, a copy /// inherits the source hash, so an original and a copy carrying /// *different* uploaded previews would collide on one ETag. - /// * Otherwise a **derived** blob's own hash. Strictly better than - /// source-keying, never worse: the pair is written from the same bytes, - /// and where they can diverge — a sidecar re-rendered while the derived - /// row stays pinned by `ON CONFLICT DO NOTHING` — a source-keyed ETag - /// is wrong too, because the renderer is not part of the key. - /// * Otherwise the source-keyed form, which still identifies a render of + /// * Otherwise the **source-keyed** form, which identifies a render of /// known content at a known size and format. /// + /// # Why the derived blob's own hash is NOT used yet + /// + /// It would be a better key — the hash *is* the bytes, so any change in + /// output invalidates by construction. But it cannot be resolved here + /// without flipping on the first render: the ETag is computed *before* + /// the body, so on a cache miss no `content_derived_blobs` row exists yet + /// and this returns the source-keyed form — then rendering *creates* that + /// row, and the next request resolves to the derived hash instead. The + /// validator would change as a side effect of producing the body, making + /// every first render immediately stale. + /// + /// It belongs with the read-order flip, when the derived tier becomes + /// authoritative and is populated before it is consulted. See + /// `docs/plan/derived-blobs.md`. The attached lookup above has no such + /// problem: an upload writes its row synchronously, before any read that + /// could observe it. + /// /// Known gap: a legacy `ext-{file_id}.jpg` with no `file_attached_blobs` /// row yet falls through to the source-keyed form, so those bytes keep /// today's coarse validator until the import backfills the row. No worse @@ -627,19 +639,12 @@ impl ThumbnailService { format: ThumbnailFormat, dedup: Option<&DedupService>, ) -> String { - if let Some(dedup) = dedup { - if let Some(attached) = dedup + if let Some(dedup) = dedup + && let Some(attached) = dedup .find_attached_blob(file_id, "preview", size.dir_name()) .await - { - return attached.blob_hash; - } - if let Some(derived) = dedup - .find_derived_blob(blob_hash, "thumbnail", size.dir_name()) - .await - { - return derived.blob_hash; - } + { + return attached.blob_hash; } format!( "thumb-{}-{}-{}", From 2775e6d5673f85f312bf937b0f1de04a665ea0c7 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 21:58:21 +0200 Subject: [PATCH 31/66] docs: the two sidecar-only paths are production-unreachable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closing out step 10(a). The remaining `None` call sites in persist_rendered looked like an open gap; they are not reachable in production. Both `get_thumbnail` and the path variant of `generate_all_sizes_background` are called only from the `ThumbnailPort` impl, and nothing holds a `dyn ThumbnailPort` — which the existing note in get_cached_thumbnail already recorded and a grep confirms. Live renders go through get_thumbnail_from_blob and generate_all_sizes_background_from_blob, both of which carry a DedupService and dual-write. So threading a DedupService through them would be work with no runtime effect. Recorded at each call site instead, with the condition that matters: gaining a real caller means taking a DedupService first, or the gap persist_rendered exists to close reopens — sidecar-only output the import can never see, so the tail never empties and the deletion gate never opens. Marks 10(a) done in the plan with that caveat stated rather than implied. --- docs/plan/derived-blobs.md | 12 ++++++---- .../services/thumbnail_service.rs | 24 ++++++++++++++----- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index db386840..6e460d74 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1294,10 +1294,14 @@ hardcoded SQL). New sources bolt on independently. 10. **Import jobs + the dual-read fallback** — see the migration section. Revised ordering as of 2026-08-26: - a. **Consolidate onto one `persist_thumbnail`** (dual-write). The - blocker: today only one of four render paths writes the derived - row, so the import can never converge. See *Prerequisite: one - persist function*. + a. **Consolidate onto one `persist_rendered`** (dual-write) — **done + 2026-08-26**. Was the blocker: only one of four render paths wrote + the derived row, so the import could never converge. Every live + render now dual-writes. Two paths still pass `None` and stay + sidecar-only, which is safe *only* because both are reachable + solely through the `ThumbnailPort` impl and nothing holds a + `dyn ThumbnailPort` — if either gains a real caller it must take a + `DedupService` first. See *Prerequisite: one persist function*. b. **`thumb_derived_import`** (shipped) and **`thumb_attached_import`** (shipped) — two jobs, not one, because the keying differs and that difference is the security boundary. A third, `transcode_import`, diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 6c2a9270..a22e48a9 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -372,9 +372,18 @@ impl ThumbnailService { tracing::info!("🎨 Generating thumbnail: {} {:?}", file_id_owned, size); match self.generate_thumbnail(&original_owned, size, format).await { Ok(bytes) => { - // `None`: renders from an on-disk original and holds - // no DedupService, so sidecar-only. Visible here - // rather than absent. + // `None` — sidecar-only, and that is acceptable here + // ONLY because this path is production-unreachable: + // its sole caller is the `ThumbnailPort` impl, and + // nothing holds a `dyn ThumbnailPort` (checked). Live + // renders go through `get_thumbnail_from_blob`, which + // dual-writes. + // + // If this ever gains a real caller it must take a + // `DedupService` first, or it reopens the gap + // `persist_rendered` exists to close: sidecar-only + // output the import can never see, so the tail never + // empties. self.persist_rendered(&blob_hash_owned, size, format, &bytes, None) .await; bytes @@ -1295,9 +1304,12 @@ impl ThumbnailService { // Save each size to disk and populate moka — both keyed by // blob_hash, so the two tiers agree and a copy shares them. for (size, bytes) in thumbnails { - // `None`: this variant renders from a path and holds no - // DedupService — `generate_all_sizes_background_from_blob` is - // the one that does. Sidecar-only, visibly so. + // `None` — sidecar-only, acceptable for the same reason as + // `get_thumbnail`: the path variant is reached only through + // the unused `ThumbnailPort` impl. The live upload path is + // `generate_all_sizes_background_from_blob`, which carries a + // `DedupService` and dual-writes. Give this one a real caller + // and it needs one too. self.persist_rendered(&blob_hash, size, ThumbnailFormat::Webp, &bytes, None) .await; { From 48d9e164d4872963150c4d6058301aa3b61bf756 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 22:27:54 +0200 Subject: [PATCH 32/66] test(thumbnails): pin get_cached_thumbnail's tier precedence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function produced four bugs in two days, every one an ordering mistake rather than a logic error, and every one caught only by an end-to-end run comparing bytes against something independent: the content-keyed RAM entry shadowing an uploaded preview so a PUT appeared to do nothing; that precedence being right on disk but wrong in RAM; a validator flipping because a tier was populated as a side effect of producing the body; a decode error reading as "absent". They all violate one sentence — a file-specific override beats anything derived from the content, at every tier — so that is what these pin. Step 10c is entirely a precedence change (derived ahead of sidecar), and it should not be another end-to-end guess. No database needed. With `dedup: None` the two DB tiers are skipped, and what remains — per-file RAM, ext- disk, content RAM, blob-hash sidecar — is exactly where the bugs were. Seven cases: the two override rules, RAM over disk within the content tiers, the sidecar answering alone, the ext- read caching under the PER-FILE key (a content key would leak one user's preview to every file sharing the content), a hashless caller falling through instead of guessing, and moka's empty-bytes negative entry not being served as a thumbnail. Verified by mutation, not just by passing: restoring the old precedence fails exactly the two tests that encode the rule, and no others. Lives in thumbnail_service.rs rather than the sibling test file because seeding tiers needs the private `cache` field. --- .../services/thumbnail_service.rs | 222 ++++++++++++++++++ 1 file changed, 222 insertions(+) diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index a22e48a9..85a5a861 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -1998,6 +1998,228 @@ pub struct ThumbnailStats { pub max_cache_bytes: usize, } +#[cfg(test)] +mod tier_selection_tests { + //! Precedence in [`ThumbnailService::get_cached_thumbnail`]. + //! + //! This function produced four bugs in two days, every one of them an + //! ordering mistake rather than a logic error, and every one caught only + //! by an end-to-end run comparing bytes against something independent: + //! + //! * the content-keyed RAM entry shadowing an uploaded preview, so a PUT + //! appeared to do nothing; + //! * the same precedence being right on disk but wrong in RAM; + //! * a validator flipping because a tier was populated as a side effect + //! of producing the body; + //! * a decode error reading as "absent". + //! + //! The rule they all violate is one sentence: **a file-specific override + //! beats anything derived from the content, at every tier.** These tests + //! pin it, so the read-order flip (step 10c of + //! `docs/plan/derived-blobs.md`) is a change with a safety net rather + //! than another end-to-end guess. + //! + //! `dedup: None` throughout, which skips the two DB-backed tiers and + //! needs no database. What remains — per-file RAM, `ext-` disk, content + //! RAM, blob-hash sidecar — is exactly where the bugs were. + + use super::*; + use std::time::Duration; + + const HASH: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + const FILE_ID: &str = "3f2b1c00-1111-2222-3333-444455556666"; + const SIZE: ThumbnailSize = ThumbnailSize::Preview; + const FMT: ThumbnailFormat = ThumbnailFormat::Webp; + + fn service(root: &std::path::Path) -> ThumbnailService { + ThumbnailService::new(root, 100, 10 * 1024 * 1024, Some(Duration::from_secs(5))) + } + + /// Seed the per-file disk tier (`ext-{file_id}.jpg`). + async fn write_ext_sidecar(root: &std::path::Path, bytes: &[u8]) { + let dir = root.join(".thumbnails").join(SIZE.dir_name()); + tokio::fs::create_dir_all(&dir).await.unwrap(); + tokio::fs::write(dir.join(format!("ext-{FILE_ID}.jpg")), bytes) + .await + .unwrap(); + } + + /// Seed the content-keyed disk tier (`{hash}.webp`). + async fn write_blob_sidecar(root: &std::path::Path, bytes: &[u8]) { + let dir = root.join(".thumbnails").join(SIZE.dir_name()); + tokio::fs::create_dir_all(&dir).await.unwrap(); + tokio::fs::write(dir.join(format!("{HASH}.{}", FMT.ext())), bytes) + .await + .unwrap(); + } + + /// The bug from 2026-08-25: a render cached under the content key + /// shadowed a preview the user uploaded afterwards, permanently, because + /// the content tier was consulted first. The PUT looked like a no-op. + #[tokio::test] + async fn per_file_ram_beats_content_ram() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + svc.cache + .insert( + ThumbnailCacheKey::content(HASH, SIZE, FMT), + Bytes::from_static(b"rendered-from-content"), + ) + .await; + svc.cache + .insert( + ThumbnailCacheKey::external(FILE_ID, SIZE), + Bytes::from_static(b"uploaded-by-user"), + ) + .await; + + let got = svc + .get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + assert_eq!(got.as_deref(), Some(&b"uploaded-by-user"[..])); + } + + /// Same rule one tier down: the per-file file on disk must win over a + /// content-keyed entry still sitting in RAM. + #[tokio::test] + async fn ext_disk_beats_content_ram() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + svc.cache + .insert( + ThumbnailCacheKey::content(HASH, SIZE, FMT), + Bytes::from_static(b"rendered-from-content"), + ) + .await; + write_ext_sidecar(tmp.path(), b"uploaded-on-disk").await; + + let got = svc + .get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + assert_eq!(got.as_deref(), Some(&b"uploaded-on-disk"[..])); + } + + /// Within the content-keyed tiers, RAM still beats disk — the ordinary + /// cache property, asserted so the flip cannot invert it by accident. + #[tokio::test] + async fn content_ram_beats_blob_sidecar() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + write_blob_sidecar(tmp.path(), b"on-disk").await; + svc.cache + .insert( + ThumbnailCacheKey::content(HASH, SIZE, FMT), + Bytes::from_static(b"in-ram"), + ) + .await; + + let got = svc + .get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + assert_eq!(got.as_deref(), Some(&b"in-ram"[..])); + } + + /// The sidecar answers when nothing above it does. After step 10c this + /// becomes the *fallback* rather than the primary content tier, and this + /// test is what proves it still answers at all. + #[tokio::test] + async fn blob_sidecar_answers_when_nothing_else_does() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + write_blob_sidecar(tmp.path(), b"only-on-disk").await; + + let got = svc + .get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + assert_eq!(got.as_deref(), Some(&b"only-on-disk"[..])); + } + + /// Reading the `ext-` file must cache it under the PER-FILE key. + /// + /// Under a content key those bytes would be served for every other file + /// sharing the same content — one user's uploaded preview leaking across + /// files, which is the poisoning the keying split exists to prevent. + #[tokio::test] + async fn ext_disk_read_caches_under_the_per_file_key() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + write_ext_sidecar(tmp.path(), b"uploaded").await; + + svc.get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + + assert_eq!( + svc.cache + .get(&ThumbnailCacheKey::external(FILE_ID, SIZE)) + .await + .as_deref(), + Some(&b"uploaded"[..]), + "must populate the per-file key" + ); + assert!( + svc.cache + .get(&ThumbnailCacheKey::content(HASH, SIZE, FMT)) + .await + .is_none(), + "must NOT populate the content key — those bytes are not derived \ + from this content and would leak to every file sharing it" + ); + } + + /// A caller with no hash cannot consult the content-keyed tiers, and must + /// fall through rather than guess. Yesterday's alternative — keying RAM + /// on `file_id` — is precisely the stale entry content-keying removed. + #[tokio::test] + async fn missing_hash_skips_content_tiers_but_still_reads_ext_disk() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + write_blob_sidecar(tmp.path(), b"content-keyed").await; + assert!( + svc.get_cached_thumbnail(FILE_ID, None, SIZE, FMT, None) + .await + .is_none(), + "without a hash the content tiers are unreachable" + ); + + write_ext_sidecar(tmp.path(), b"per-file").await; + assert_eq!( + svc.get_cached_thumbnail(FILE_ID, None, SIZE, FMT, None) + .await + .as_deref(), + Some(&b"per-file"[..]), + "the per-file tier needs no hash and must still answer" + ); + } + + /// Empty bytes are moka's negative-entry convention (a previous render + /// failed). They must not be served as a thumbnail, or a failure gets + /// cached and returned as success. + #[tokio::test] + async fn empty_cache_entry_is_not_served() { + let tmp = tempfile::tempdir().unwrap(); + let svc = service(tmp.path()); + + svc.cache + .insert(ThumbnailCacheKey::content(HASH, SIZE, FMT), Bytes::new()) + .await; + write_blob_sidecar(tmp.path(), b"real-bytes").await; + + let got = svc + .get_cached_thumbnail(FILE_ID, Some(HASH), SIZE, FMT, None) + .await; + assert_eq!( + got.as_deref(), + Some(&b"real-bytes"[..]), + "a negative entry must fall through, not be served" + ); + } +} + #[cfg(test)] mod tests { use super::*; From 260e6bb74f5c67c11a46a8585aa9741e5d97b8a2 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 22:51:30 +0200 Subject: [PATCH 33/66] feat(thumbnails): read the derived tier ahead of the sidecar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 10c. The sidecar is local disk — invisible to other instances, uncarried by a backend migration, uncovered by any consistency job. Reading the derived tier first is what makes that state deletable. Not the cost it appears to be: CachedBlobBackend gives the blob read a local disk cache and moka absorbs the repeats above it. Not a two-line swap, for two reasons. A derived MISS must fall through to the sidecar; the old code terminated the lookup with `?` because it was last. While the imports drain, most content has a sidecar and no row — terminating there would report "no thumbnail" for nearly all of it. And the derived tier is WebP-only. store_derived_blob writes image/webp and keys `variant` on the size alone, with no format term, so a JPEG request matches the WebP row and would be served the wrong codec. The old ordering hid this because the .jpg sidecar won first. So the lookup is gated to WebP, JPEG clients stay on the sidecar — and the sidecar cannot be deleted for them until `variant` encodes format. That is a new prerequisite for step 10e, recorded in the plan rather than discovered later. Also corrects the plan: I had written that this flip removes the derived-hash ETag hazard. It does not. A first render still creates the row as a side effect of producing the body, whatever the read order, so two consecutive reads still straddle its appearance. The real fix is resolving the ETag after generation on the 200 path — a 304 only fires when the client already holds a validator, which implies the row exists. That is a handler restructure, not an ordering change. --- docs/plan/derived-blobs.md | 37 ++++++++-- .../services/thumbnail_service.rs | 69 ++++++++++++------- 2 files changed, 74 insertions(+), 32 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 6e460d74..55c201fc 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1003,9 +1003,19 @@ first render was immediately stale. Caught by `thumbnail_etag_content_keyed.hurl`, where two consecutive GETs of an unchanged file stopped revalidating to 304. -The flip is what removes the hazard: once the derived tier is -authoritative it is populated before it is consulted, so there is no -window in which the row appears between two reads. +**The flip alone does NOT remove the hazard** *(corrected 2026-08-26 — +an earlier revision of this paragraph claimed it did)*. A first render +still creates the row as a side effect of producing the body, whatever +the read order, so two consecutive reads would still straddle its +appearance. + +What actually removes it is resolving the ETag **after** generation on +the 200 path. A 304 can only fire when the client already holds a +validator, which means it has been served before, which means the row +exists — so the *conditional* path can safely consult the derived hash +up front, while the *generating* path computes it from bytes it now +holds. That is a handler restructure, not an ordering change, and it is +the actual prerequisite for the derived-hash ETag. **The disk cache is `CachedBlobBackend`, reused unchanged.** No thumbnail-specific cache, no second root path. Routing derived @@ -1310,10 +1320,23 @@ hardcoded SQL). New sources bolt on independently. **re-keyed** file→content on import (legitimate only because a transcode is derivable). Its `.skip` markers — a cached negative verdict with no bytes — remain an open question. - c. **Flip the read order**, derived first. The HTTP ETag's - source-keyed fallback becomes unreachable here; the attached and - derived halves already landed early, forced by the attachment - case (see *HTTP ETag*). + c. **Flip the read order**, derived first — **done 2026-08-26**. Two + things it is not: a two-line swap, and an ETag fix. + + A derived miss must **fall through** to the sidecar, where the old + code terminated the lookup — while the imports drain, most content + has a sidecar and no row, so terminating would report "no + thumbnail" for nearly everything. + + And it is **WebP-only**. `store_derived_blob` writes `image/webp` + with `variant` keyed on size alone, no format term, so a JPEG + request matches the WebP row and gets the wrong codec — a + regression the old ordering hid, because the `.jpg` sidecar won + first. JPEG clients therefore stay on the sidecar, **and the + sidecar cannot be deleted for them** until `variant` encodes + format. That is a new prerequisite for (e), not a detail: it means + a migration to `(kind, variant, format)` — or a format term inside + `variant` — has to land before the directories can go. d. **Enable deletion** in the import jobs (opt-in, readback-verified). e. **Remove the fallback read path** once the directory no longer *exists* — not merely once it is empty. Two reasons. Empty is a diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 85a5a861..6794f794 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -792,11 +792,35 @@ impl ThumbnailService { return Some(bytes); } - // 4. Check disk for blob-hash thumbnails (needs blob_hash to locate) - let thumb_path = self.get_thumbnail_path(hash, size, format); - if let Ok(data) = fs::read(&thumb_path).await { - let bytes = Bytes::from(data); - // Populate in-memory cache for next hit + // 4. Derived blob — the authoritative content tier (step 10c). + // + // Ahead of the sidecar now, rather than last. The sidecar is local + // disk: invisible to other instances, uncarried by a backend + // migration, uncovered by any consistency job. Reading the derived + // tier first is what lets that disk state become deletable, and it is + // not the cost it looks like — `CachedBlobBackend` gives the blob read + // a local disk cache, and moka absorbs the repeats above it. + // + // A miss FALLS THROUGH rather than ending the lookup. That is the + // whole reason this is not a two-line swap: while the imports are + // draining, most content has a sidecar and no row, and terminating + // here would return "no thumbnail" for all of it. + // + // **WebP only.** `store_derived_blob` writes `image/webp` and keys + // `variant` on the size alone, with no format term, so a JPEG request + // would match the WebP row and be served the wrong codec — a + // regression the old ordering hid, because the `.jpg` sidecar won + // first. Until `variant` encodes format, JPEG clients stay on the + // sidecar, and the sidecar therefore cannot be deleted for them. See + // docs/plan/derived-blobs.md. + if format == ThumbnailFormat::Webp + && let Some(dedup) = dedup + && let Some(derived) = dedup + .find_derived_blob(hash, "thumbnail", size.dir_name()) + .await + && let Some(bytes) = + Self::read_blob_to_bytes(dedup, &derived.blob_hash, file_id, size).await + { self.cache .insert( ThumbnailCacheKey::content(hash, size, format), @@ -806,26 +830,21 @@ impl ThumbnailService { return Some(bytes); } - // 4. Tier-3 derived blob. Deliberately LAST while the sidecar still - // exists: for every thumbnail already on disk this branch is never - // reached, so the DB stays off the hot path and a fault here cannot - // break a working gallery. It answers only what disk cannot — another - // instance's render, or a box whose sidecar was never populated. - // - // The order flips (derived blob first, sidecar as fallback) in the - // release that removes the sidecar; see docs/plan/derived-blobs.md. - let dedup = dedup?; - let derived = dedup - .find_derived_blob(hash, "thumbnail", size.dir_name()) - .await?; - let bytes = Self::read_blob_to_bytes(dedup, &derived.blob_hash, file_id, size).await?; - self.cache - .insert( - ThumbnailCacheKey::content(hash, size, format), - bytes.clone(), - ) - .await; - Some(bytes) + // 5. Blob-hash sidecar — fallback for content not yet imported, and + // the only content tier a non-WebP request can reach. + let thumb_path = self.get_thumbnail_path(hash, size, format); + if let Ok(data) = fs::read(&thumb_path).await { + let bytes = Bytes::from(data); + self.cache + .insert( + ThumbnailCacheKey::content(hash, size, format), + bytes.clone(), + ) + .await; + return Some(bytes); + } + + None } /// Store an externally-generated thumbnail (e.g. client-side video frame). From 4997eb4fb07ce86cc6d5b05a3933d0ec6bb0fcb5 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 22:54:44 +0200 Subject: [PATCH 34/66] docs(plan): sequence transcode_import behind its two prerequisites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answering "when is transcode_import planned": it is the remaining third of 10(b), but it must not be next, and both reasons were learned on the thumbnail side rather than predicted. Format has to move into `variant` first. Transcodes are inherently multi-format and `variant` is keyed on size alone — the same gap found in 10c that keeps JPEG thumbnails on the sidecar. Importing before that means migrating into a schema that cannot hold the data without collisions across formats. One migration unblocks both, which is the argument for doing it before either. And step 7 must precede the import. ImageTranscodeService writes only its file-keyed disk cache today, so the import would run against a cache that is still growing and never reach an empty tail — exactly the trap persist_rendered had to close for thumbnails, where one of four render paths recorded a row and the tail could never empty. Order: format-in-variant → step 7 → transcode_import. Also records why the .skip markers are still open: a cached negative verdict has no bytes, so it does not fit a table whose point is pointing at a blob. --- docs/plan/derived-blobs.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 55c201fc..91fd0558 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1319,7 +1319,24 @@ hardcoded SQL). New sources bolt on independently. caches `.transcoded/{ext}/{file_id}.{ext}`, so those must be **re-keyed** file→content on import (legitimate only because a transcode is derivable). Its `.skip` markers — a cached negative - verdict with no bytes — remain an open question. + verdict with no bytes — remain an open question, since they do not + fit a table whose point is pointing at a blob. + + **Not next, and deliberately so.** Two prerequisites, both learned + the hard way on the thumbnail side: + + * **Format must move into `variant` first.** Transcodes are + inherently multi-format, and `variant` is keyed on size alone — + the same gap that keeps JPEG thumbnails on the sidecar (see + 10c). Importing before that means migrating into a schema that + cannot hold the data without collisions. One migration unblocks + both. + * **Step 7 before the import.** `ImageTranscodeService` writes only + its file-keyed cache today, so an import would run against a + cache still growing and never reach an empty tail — precisely + the trap `persist_rendered` had to close for thumbnails. + + Order: format-in-`variant` → step 7 → `transcode_import`. c. **Flip the read order**, derived first — **done 2026-08-26**. Two things it is not: a two-line swap, and an ETag fix. From d7de1c41e7422381efe8a4e7784ffc3b0c23c014 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 23:07:58 +0200 Subject: [PATCH 35/66] fix(files): missing folder_id is 400, not 500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Uploading without folder_id answered `500 Internal Error: folder_id is required to determine file owner`. A missing required field is the caller's error; as an internal_error it produced `error_type: Internal Error`, which the SPA cannot distinguish from the server breaking — so a malformed request looked like an outage. Both sites become validation_error (ErrorKind::InvalidInput → 400), with messages that say WHY the field is needed rather than restating that it is: the destination folder determines the file's owner and drive. The OpenAPI request body described it as "optional folder_id field", which is how it came to be omitted — hit while writing thumbnail_etag_content_keyed.hurl, where the upload was written from the documented contract and 500'd. Now stated as required. Regression test asserts the status AND that error_type is not "Internal Error", since the contract the SPA switches on is error_type rather than the message. --- .../pg/file_blob_write_repository.rs | 20 ++++++++++++----- src/interfaces/api/handlers/file_handler.rs | 2 +- tests/api/files-folders.hurl | 22 +++++++++++++++++++ 3 files changed, 37 insertions(+), 7 deletions(-) diff --git a/src/infrastructure/repositories/pg/file_blob_write_repository.rs b/src/infrastructure/repositories/pg/file_blob_write_repository.rs index f9cb56af..dca03852 100644 --- a/src/infrastructure/repositories/pg/file_blob_write_repository.rs +++ b/src/infrastructure/repositories/pg/file_blob_write_repository.rs @@ -130,9 +130,11 @@ impl FileBlobWriteRepository { DomainError::internal_error("FileBlobWrite", format!("parent lookup: {e}")) })? .ok_or_else(|| DomainError::not_found("Folder", fid)), - None => Err(DomainError::internal_error( - "FileBlobWrite", - "folder_id is required to determine the target drive", + // Same reasoning as the owner lookup below: caller error, not + // server error. + None => Err(DomainError::validation_error( + "folder_id is required: the destination folder determines the \ + target drive", )), } } @@ -289,9 +291,15 @@ impl FileBlobWriteRepository { rollback_err ); } - return Err(DomainError::internal_error( - "FileBlobWrite", - "folder_id is required to determine file owner", + // A missing required field is the caller's error, not the + // server's. As `internal_error` this surfaced as 500 / + // `error_type: Internal Error`, which the SPA cannot tell apart + // from the server breaking — so a malformed upload looked like an + // outage. The OpenAPI body description called the field optional, + // which is how it came to be omitted in the first place. + return Err(DomainError::validation_error( + "folder_id is required: the destination folder determines the \ + file's owner and drive", )); }; diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 07400fc2..193d45df 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -1552,7 +1552,7 @@ pub async fn list_files_query( #[utoipa::path( post, path = "/api/files/upload", - request_body(content_type = "multipart/form-data", description = "File data + optional folder_id field"), + request_body(content_type = "multipart/form-data", description = "File data + folder_id (required: it determines the file's owner and drive)"), responses( (status = 201, description = "File uploaded", body = FileDto), (status = 400, description = "Invalid request"), diff --git a/tests/api/files-folders.hurl b/tests/api/files-folders.hurl index d1dcaf26..e23fd0ca 100644 --- a/tests/api/files-folders.hurl +++ b/tests/api/files-folders.hurl @@ -357,3 +357,25 @@ Authorization: Bearer {{token}} HTTP 200 [Asserts] header "Content-Type" startsWith "image/" + + +# ───────────────────────────────────────────────────────────── +# Step 23 – Upload without folder_id is a CLIENT error +# +# The destination folder determines the file's owner and drive, so the +# field is required. It used to answer 500 / `error_type: Internal +# Error`, which the SPA cannot distinguish from the server breaking — a +# malformed request looked like an outage. The OpenAPI body description +# called the field optional, which is how it came to be omitted. +# +# Asserts the status AND the error_type, because the contract the SPA +# switches on is `error_type`, not the message. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +file: file,fixtures/hello.txt; text/plain + +HTTP 400 +[Asserts] +jsonpath "$.error_type" != "Internal Error" From c656e684d4c5f62adbd4cd9f3700156ce753062f Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 26 Aug 2026 23:24:27 +0200 Subject: [PATCH 36/66] feat(consistency): merge-join backend_consistency, both directions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The job walked the backend and probed the DB with WHERE hash = ANY($1) over each page, so it could only ever see backend-only entries. A registry row whose bytes are gone never appears in a backend listing — it was invisible here by construction, and that half was left to blobs_consistency's per-row HEAD probe, which does not survive the row counts this plan produces. Both sides are now ordered by hash — the backend by contract since 5343fdda, the DB by ORDER BY hash — so one pass yields both deltas: orphan_blob for bytes with no row, and blob_missing_from_backend for a row with no bytes. The latter is severity data_loss rather than inconsistent: an orphan wastes space, this loses a file. Three things the merge needs that a probe did not. A horizon. The two pages cover different ranges, so only their overlap can be judged — beyond it, a hash missing from one side may simply be on the next page of the other, and emitting there would invent findings in both directions at once. When a side is exhausted its entries cannot be on a later page, so the other's tail becomes judgeable. One cursor for both sides. They share an ordering, so "resume after H" is start_after(H) on the backend and hash > H in the DB. The cursor advances to the horizon, not the backend's own next_cursor, which would skip the un-judged tail of whichever side reached further. Format is unchanged, so paused runs resume. And an ordering premise worth stating rather than assuming: hashes are lowercase BLAKE3 hex of fixed length, so collation and byte order rank identically over [0-9a-f]. A hash column admitting uppercase or variable length would break this silently. Recorded in the module docs and beside the query. blobs_consistency still emits its own blob_missing_from_backend; the two now overlap. Retiring that probe is the follow-up, not folded in here. --- .../services/backend_consistency_service.rs | 229 +++++++++++++----- 1 file changed, 173 insertions(+), 56 deletions(-) diff --git a/src/infrastructure/services/backend_consistency_service.rs b/src/infrastructure/services/backend_consistency_service.rs index ed3a9f70..a9c56d7f 100644 --- a/src/infrastructure/services/backend_consistency_service.rs +++ b/src/infrastructure/services/backend_consistency_service.rs @@ -1,18 +1,36 @@ //! Fifth tenant of Part 2 (recoverable-run engine). //! -//! Iterates the storage backend's blob-enumeration surface and -//! reports every blob physically present on the backend that has NO -//! matching row in `storage.blobs`. Complements -//! `blobs_consistency` (which walks the DB and probes the backend): -//! together they close the reference graph. +//! **Merge-joins** the backend's blob enumeration against +//! `storage.blobs`, both ordered by hash, so a single pass yields the +//! delta in *both* directions rather than one. //! -//! ### Per-row check +//! It previously walked the backend and probed the DB with +//! `WHERE hash = ANY($1)` over each page, which could only ever see +//! backend-only entries: a row whose bytes are gone never appears in a +//! backend listing, so it was invisible here by construction. That half +//! was left to `blobs_consistency`'s per-row HEAD probe, which does not +//! survive the row counts this plan produces — see +//! `docs/plan/derived-blobs.md`. +//! +//! ### Per-row checks //! //! * `orphan_blob` (severity `inconsistent`) — bytes on disk / S3 / //! Azure with no registry row. Not data-loss (nothing broken — //! just storage overhead), but points at dedup_gc or //! ingest-path drift. Recovery = register-registry-row (if the //! bytes are still needed) OR delete the file (if truly orphan). +//! * `blob_missing_from_backend` (severity `data_loss`) — a registry +//! row whose bytes are absent. The opposite direction and the more +//! serious one: an orphan wastes space, this loses a file. +//! +//! ### Why the two orderings agree +//! +//! The merge-join's premise is that the backend's byte order and the +//! database's `ORDER BY hash` rank identically. They do, because hashes +//! are lowercase BLAKE3 hex of fixed length: over `[0-9a-f]` digits +//! precede letters in both, and there is no case to fold. A hash column +//! that ever admitted uppercase or variable length would break this +//! silently and in both directions at once. //! //! ### Run-level check //! @@ -42,7 +60,6 @@ //! denominator (backend count ≈ blob count on a healthy install; //! deviation IS the finding). -use std::collections::HashSet; use std::sync::Arc; use async_trait::async_trait; @@ -376,60 +393,161 @@ impl RecoverableJobHandler for BackendConsistencyCheck { return RunOutcome::completed(); } - // Batch DB probe: which of these hashes have a - // `storage.blobs` row? One `WHERE hash = ANY($1)` per - // batch — indexed lookup, cheap even on millions of - // rows. - let batch_hashes: Vec = page.blobs.iter().map(|e| e.hash.clone()).collect(); - let db_present: HashSet = if batch_hashes.is_empty() { - HashSet::new() - } else { - match sqlx::query_as::<_, (String,)>( - r#"SELECT hash FROM storage.blobs WHERE hash = ANY($1)"#, - ) - .bind(&batch_hashes[..]) - .fetch_all(self.pool.as_ref()) - .await - { - Ok(rows) => rows.into_iter().map(|(h,)| h).collect(), - Err(e) => { - return RunOutcome::Failed { - message: format!("db probe: {e}"), - }; - } + // ── Merge-join, not a one-sided probe ──────────────── + // + // Both sides are ordered by hash ascending — the backend by + // contract (`BlobStorageBackend::list_blob_hashes`), the DB by + // `ORDER BY hash` — so one pass yields BOTH deltas instead of + // one: + // + // * present on the backend, absent from the DB → `orphan_blob` + // * present in the DB, absent from the backend → + // `blob_missing_from_backend` (data loss, not overhead) + // + // The old form probed `WHERE hash = ANY($1)` over the backend + // page, so it could only ever see the first kind: a row whose + // bytes are gone never appears in a backend listing and was + // invisible here by construction. + // + // Ordering is the whole premise, so it is worth being explicit + // about why the two agree. Hashes are lowercase BLAKE3 hex of + // fixed length, and over `[0-9a-f]` the database collation and + // byte order rank identically (digits before letters in both, + // no case folding to disagree about). A hash column that ever + // admitted uppercase or variable length would break this + // silently, in both directions. + let db_hashes: Vec = match sqlx::query_as::<_, (String,)>( + r#"SELECT hash FROM storage.blobs + WHERE ($1::text IS NULL OR hash > $1) + ORDER BY hash + LIMIT $2"#, + ) + .bind(cursor.as_deref()) + .bind(BATCH_SIZE as i64) + .fetch_all(self.pool.as_ref()) + .await + { + Ok(rows) => rows.into_iter().map(|(h,)| h).collect(), + Err(e) => { + return RunOutcome::Failed { + message: format!("db page: {e}"), + }; } }; - for entry in &page.blobs { - if db_present.contains(&entry.hash) { - continue; - } - if let Some(mtime) = entry.mtime - && mtime > grace_cutoff - { - continue; - } + // The two pages cover different ranges, so only the overlap can + // be judged. Beyond `horizon` a hash missing from one side may + // simply be on the next page of the other, and emitting there + // would invent findings in both directions. When a side is + // exhausted its entries cannot be "on a later page", so the + // other side's tail becomes judgeable. + let backend_last = page.blobs.last().map(|e| e.hash.as_str()); + let db_last = db_hashes.last().map(|s| s.as_str()); + let backend_done = page.next_cursor.is_none(); + let db_done = db_hashes.len() < BATCH_SIZE; - finding_count += 1; - record_or_log( - store, - BACKEND_CONSISTENCY_JOB_NAME, - "orphan_blob", - "inconsistent", - None, - serde_json::json!({ - "hash": entry.hash, - "mtime": entry.mtime.map(|t| t.to_rfc3339()), - "backend": backend.backend_type(), - }), - ) - .await; + let horizon: Option<&str> = match (backend_last, db_last) { + _ if backend_done && db_done => None, // judge everything + (Some(b), Some(d)) if backend_done => Some(b.max(d)), + (Some(b), Some(d)) if db_done => Some(b.max(d)), + (Some(b), Some(d)) => Some(b.min(d)), + (Some(b), None) => Some(b), + (None, Some(d)) => Some(d), + (None, None) => None, + }; + let in_range = |h: &str| horizon.is_none_or(|limit| h <= limit); + + let mut bi = page.blobs.iter().peekable(); + let mut di = db_hashes.iter().peekable(); + loop { + match (bi.peek(), di.peek()) { + (Some(b), Some(d)) if b.hash == **d => { + bi.next(); + di.next(); + } + // Backend-only: bytes with no registry row. + (Some(b), d_opt) + if d_opt.is_none_or(|d| b.hash.as_str() < d.as_str()) + && in_range(&b.hash) => + { + // Grace window: the write path is + // durability-before-visibility, so bytes exist + // briefly before their row does. Without this every + // in-flight upload reads as an orphan. + if !matches!(b.mtime, Some(m) if m > grace_cutoff) { + finding_count += 1; + record_or_log( + store, + BACKEND_CONSISTENCY_JOB_NAME, + "orphan_blob", + "inconsistent", + None, + serde_json::json!({ + "hash": b.hash, + "mtime": b.mtime.map(|t| t.to_rfc3339()), + "backend": backend.backend_type(), + }), + ) + .await; + } + bi.next(); + } + // DB-only: a row whose bytes are gone. Severity is + // `data_loss`, not `inconsistent` — an orphan wastes + // space, this loses a file. + (b_opt, Some(d)) + if b_opt.is_none_or(|b| d.as_str() < b.hash.as_str()) && in_range(d) => + { + finding_count += 1; + record_or_log( + store, + BACKEND_CONSISTENCY_JOB_NAME, + "blob_missing_from_backend", + "data_loss", + None, + serde_json::json!({ + "hash": d, + "backend": backend.backend_type(), + "note": "registry row with no bytes on the backend", + }), + ) + .await; + di.next(); + } + // Past the horizon on both sides, or both exhausted. + _ => break, + } } - // Advance cursor + checkpoint. Scanned count tracks - // both blobs and unknowns since we walked both. + // Advance to the horizon, not the backend's own cursor. + // + // One hash serves both sides: they share an ordering, so "resume + // after H" means `start_after(H)` on the backend and + // `WHERE hash > H` in the DB. Advancing past the horizon would + // skip the un-judged tail of whichever side reached further. + // + // Scanned count covers blobs and unknowns, since both were + // walked. let batch_len = (page.blobs.len() + page.unknowns.len()) as u64; - cursor = page.next_cursor; + let exhausted = backend_done && db_done; + cursor = if exhausted { + None + } else { + horizon.map(|h| h.to_string()) + }; + + // Neither side exhausted yet no horizon means neither returned a + // row — nothing left to compare, and continuing would spin on the + // same empty pages forever. + if cursor.is_none() && !exhausted && horizon.is_none() { + tracing::debug!( + target: "oxicloud::consistency", + event = "backend_consistency.no_horizon", + run_id = %store.run_id(), + "both sides returned no rows before exhaustion; ending the sweep" + ); + } + let cursor_bytes = cursor .as_ref() .map(|s| s.as_bytes().to_vec()) @@ -440,8 +558,7 @@ impl RecoverableJobHandler for BackendConsistencyCheck { }; } - // Backend returned no next_cursor → enumeration - // complete. Emit the completion log and return. + // Both sides drained → the sweep is complete. if cursor.is_none() { tracing::info!( target: "oxicloud::consistency", From 86d0d655836d4c2117e168b8116b3e275235d380 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 07:21:35 +0200 Subject: [PATCH 37/66] feat(storage): derived variant encodes the output format MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `content_derived_blobs.variant` held the size alone, so one source could hold exactly one artifact per size regardless of codec. That surfaced when the read order flipped in 10c: a JPEG request matched the WebP row and would have been served the wrong codec — hidden previously because the .jpg sidecar won first. The flip had to be gated to WebP, which meant JPEG clients could never leave the sidecar, which meant the sidecar could never be deleted. It blocks transcodes harder: those are multi-format by nature, so two output codecs of one source collide on the primary key without a format term. The axis goes inside the string rather than into a fourth PK column, per the column's own rule — "new axes go inside this string, never into new columns". Shape is {size}.{ext}: preview.webp, icon.jpg, later 720p.webp. The backfill is deterministic, not a guess: store_derived_blob has only ever written "image/webp" for thumbnails. content_type is checked anyway rather than assumed — a row that fails the assumption is left alone and counted in a warning, because the read path then simply misses it and falls back to the sidecar, whereas guessing a codec would serve wrong bytes. Idempotent via NOT LIKE '%.%', so a re-apply cannot produce preview.webp.webp; verified on a scratch PG by applying it twice. One helper builds the string, because it is a primary-key component: a writer and reader that disagree do not fail loudly, they just never find each other's rows and the derived tier silently looks empty. It lives on the service's ThumbnailSize, not the port's — they are distinct types, which the compiler pointed out after I put it on the wrong one. The WebP gate on the read path is now removed: each codec has its own row, so JPEG can finally reach the derived tier — the prerequisite for deleting the sidecar for those clients. file_attached_blobs keeps a bare size: store_external_thumbnail re-encodes everything to JPEG, so it is single-format by construction and a format term would cost a migration for nothing. --- ...2000000_derived_variant_encodes_format.sql | 56 +++++++++++++++++++ .../services/thumb_derived_import_service.rs | 13 ++++- .../services/thumbnail_service.rs | 36 ++++++++---- 3 files changed, 93 insertions(+), 12 deletions(-) create mode 100644 migrations/20261022000000_derived_variant_encodes_format.sql diff --git a/migrations/20261022000000_derived_variant_encodes_format.sql b/migrations/20261022000000_derived_variant_encodes_format.sql new file mode 100644 index 00000000..f3efcd16 --- /dev/null +++ b/migrations/20261022000000_derived_variant_encodes_format.sql @@ -0,0 +1,56 @@ +-- Put the output format inside `variant`, where the plan says new axes go. +-- +-- `content_derived_blobs.variant` held the size alone (`icon` | `preview` | +-- `large`), so a size could hold exactly ONE stored artifact regardless of +-- codec. That surfaced when the read order flipped (step 10c): a JPEG request +-- matched the WebP row and would have been served the wrong codec, which the +-- old ordering hid because the `.jpg` sidecar won first. The flip had to be +-- gated to WebP, which in turn means JPEG clients can never leave the sidecar +-- — so the sidecar can never be deleted. +-- +-- It blocks transcodes harder still: those are multi-format by nature, so +-- without a format term two output codecs of one source collide on the +-- primary key. +-- +-- Per the column's own comment — "new axes go inside this string, never into +-- new columns" — the axis goes in the string rather than into a fourth PK +-- column. The PK stays `(source_hash, kind, variant)`. +-- +-- Shape: `{size}.{ext}` — `preview.webp`, `icon.jpg`, and later `720p.webp` +-- for transcodes. +-- +-- The backfill is deterministic rather than a guess: `store_derived_blob` has +-- only ever been called with `"image/webp"` for thumbnails, so every existing +-- thumbnail row is WebP. `content_type` is checked anyway rather than assumed +-- — if that assumption is ever wrong, the row is left alone for a human to +-- look at instead of being silently mislabelled. + +UPDATE storage.content_derived_blobs + SET variant = variant || '.webp' + WHERE kind = 'thumbnail' + AND content_type = 'image/webp' + -- Idempotent: skip anything already carrying a format suffix, so a + -- re-applied migration cannot produce `preview.webp.webp`. + AND variant NOT LIKE '%.%'; + +-- Anything left without a format suffix did not match the WebP assumption. +-- Surfaced as a warning rather than coerced: the read path will simply miss +-- those rows and fall back to the sidecar, which is safe, whereas guessing a +-- codec would serve the wrong bytes. +DO $$ +DECLARE + v_unsuffixed INT; +BEGIN + SELECT COUNT(*) INTO v_unsuffixed + FROM storage.content_derived_blobs + WHERE kind = 'thumbnail' AND variant NOT LIKE '%.%'; + + IF v_unsuffixed > 0 THEN + RAISE WARNING + 'derived_variant_encodes_format: % thumbnail row(s) have no format suffix (content_type was not image/webp). They will be ignored by the read path and re-derived on demand; inspect before deleting the sidecars.', + v_unsuffixed; + END IF; +END $$; + +COMMENT ON COLUMN storage.content_derived_blobs.variant IS + 'Opaque discriminator carrying every axis but the source and the kind: size AND output format, as {size}.{ext} (preview.webp | icon.jpg | 720p.webp). New axes go inside this string, never into new columns. A format term is required — without one, two codecs of the same source collide on the primary key, and the read path cannot tell which codec a row holds.'; diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index cf0d95dd..4a189f7a 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -159,7 +159,18 @@ impl RecoverableJobHandler for ThumbDerivedImport { let mut already = 0u64; let mut failed = 0u64; let mut since_checkpoint = 0usize; - let variant_of = |s: ThumbnailSize| s.dir_name().to_string(); + // Sidecars under `.thumbnails/` are `{hash}.webp` — the filter that + // built this list requires the extension — so the imported rows are + // WebP, and the variant must say so since migration + // `20261022000000`. Writing the bare size here would produce rows the + // read path can never match. + let variant_of = |s: ThumbnailSize| { + format!( + "{}.{}", + s.dir_name(), + crate::application::ports::thumbnail_ports::ThumbnailFormat::Webp.ext() + ) + }; for size in ThumbnailSize::all() { let dir_name = variant_of(*size); diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 6794f794..bb596d80 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -59,6 +59,22 @@ impl ThumbnailSize { } } + /// The `content_derived_blobs.variant` value for this size and format. + /// + /// One place builds the string, because it is a primary-key component: a + /// writer and a reader that disagree do not fail loudly, they simply + /// never find each other's rows — the read falls back to the sidecar and + /// the derived tier silently looks empty. + /// + /// The format term is what lets one source hold both codecs at a size. + /// Without it a JPEG request matched the WebP row and would be served the + /// wrong codec, which is why the step-10c read flip had to be gated to + /// WebP and why JPEG clients could never leave the sidecar. See migration + /// `20261022000000`. + pub fn derived_variant(&self, format: ThumbnailFormat) -> String { + format!("{}.{}", self.dir_name(), format.ext()) + } + /// Get all thumbnail sizes pub fn all() -> &'static [ThumbnailSize] { &[ @@ -310,7 +326,7 @@ impl ThumbnailService { .store_derived_blob( blob_hash, "thumbnail", - size.dir_name(), + &size.derived_variant(format), format.mime(), bytes.clone(), ) @@ -806,17 +822,15 @@ impl ThumbnailService { // draining, most content has a sidecar and no row, and terminating // here would return "no thumbnail" for all of it. // - // **WebP only.** `store_derived_blob` writes `image/webp` and keys - // `variant` on the size alone, with no format term, so a JPEG request - // would match the WebP row and be served the wrong codec — a - // regression the old ordering hid, because the `.jpg` sidecar won - // first. Until `variant` encodes format, JPEG clients stay on the - // sidecar, and the sidecar therefore cannot be deleted for them. See - // docs/plan/derived-blobs.md. - if format == ThumbnailFormat::Webp - && let Some(dedup) = dedup + // All formats, since migration `20261022000000` put the output format + // inside `variant`. Before that, `variant` was the size alone, so a + // JPEG request matched the WebP row and would have been served the + // wrong codec — the flip had to be gated to WebP, which meant JPEG + // clients could never leave the sidecar and the sidecar could never + // be deleted. Now each codec has its own row. + if let Some(dedup) = dedup && let Some(derived) = dedup - .find_derived_blob(hash, "thumbnail", size.dir_name()) + .find_derived_blob(hash, "thumbnail", &size.derived_variant(format)) .await && let Some(bytes) = Self::read_blob_to_bytes(dedup, &derived.blob_hash, file_id, size).await From d202b4b5ca2b01bb305a03cdf978c7cf3e6b0e38 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 13:18:51 +0200 Subject: [PATCH 38/66] fix(storage): the derived import conflated variant with directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 76590160 changed `variant_of` to return `{size}.{ext}`, but that value was also being used as the on-disk DIRECTORY. Reads became `.thumbnails/preview.webp/{hash}.webp`, which does not exist, so every sidecar counted as unreadable and thumb_derived_import restored nothing. Caught by thumb_import_check.sh on the run after the migration — the harness earning its keep twice now, since this is the second defect it has caught that no unit test could. They are genuinely two strings and are now named as such: `dir_name` for the path, `variant` for the row key. The cursor keeps using the directory, so a run paused before the migration resumes at the same position rather than restarting. Also stops podman's compose-provider banner from burying the script's output. Filtered rather than discarded, so genuine psql errors still surface — swallowing those would turn a broken query into a silently wrong assertion. Suppressing it at the source needs `[engine] compose_warning_logs = false` in containers.conf, which is per-developer config and cannot be relied on in CI. --- .../services/thumb_derived_import_service.rs | 24 ++++++++++++------- tests/api/thumb_import_check.sh | 11 ++++++++- 2 files changed, 25 insertions(+), 10 deletions(-) diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 4a189f7a..570305ed 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -159,11 +159,14 @@ impl RecoverableJobHandler for ThumbDerivedImport { let mut already = 0u64; let mut failed = 0u64; let mut since_checkpoint = 0usize; - // Sidecars under `.thumbnails/` are `{hash}.webp` — the filter that - // built this list requires the extension — so the imported rows are - // WebP, and the variant must say so since migration - // `20261022000000`. Writing the bare size here would produce rows the - // read path can never match. + // Two different strings, and conflating them is a real trap: the + // DIRECTORY is `{size}` on disk, while the VARIANT is + // `{size}.{ext}` since migration `20261022000000`. Using the variant + // as a path yields `.thumbnails/preview.webp/…`, which does not + // exist, so every file reads as unreadable and nothing imports. + // + // Sidecars here are always `{hash}.webp` — the name filter requires + // that extension — so the variant is unconditionally the WebP one. let variant_of = |s: ThumbnailSize| { format!( "{}.{}", @@ -173,8 +176,11 @@ impl RecoverableJobHandler for ThumbDerivedImport { }; for size in ThumbnailSize::all() { - let dir_name = variant_of(*size); + let dir_name = size.dir_name(); // on-disk directory + let variant = variant_of(*size); // content_derived_blobs.variant for name in Self::sidecar_names(&self.thumbnails_root, *size).await { + // Cursor position uses the DIRECTORY, so a run paused before + // this change resumes at the same place. let position = format!("{dir_name}/{name}"); // Resume: everything at or before the cursor is done. @@ -206,13 +212,13 @@ impl RecoverableJobHandler for ThumbDerivedImport { // reason this job is safe to trigger repeatedly. if self .dedup - .find_derived_blob(hash, "thumbnail", &dir_name) + .find_derived_blob(hash, "thumbnail", &variant) .await .is_some() { already += 1; } else { - let path = self.thumbnails_root.join(&dir_name).join(&name); + let path = self.thumbnails_root.join(dir_name).join(&name); match fs::read(&path).await { Ok(data) => { match self @@ -220,7 +226,7 @@ impl RecoverableJobHandler for ThumbDerivedImport { .store_derived_blob( hash, "thumbnail", - &dir_name, + &variant, "image/webp", Bytes::from(data), ) diff --git a/tests/api/thumb_import_check.sh b/tests/api/thumb_import_check.sh index f7978972..763bfd22 100755 --- a/tests/api/thumb_import_check.sh +++ b/tests/api/thumb_import_check.sh @@ -80,8 +80,17 @@ fail() { # psql inside the compose container — no host psql dependency, matching # how spawn-db.sh probes readiness. sql() { + # Podman's docker-compose shim prints a provider banner to stderr on every + # invocation, which buries this script's own output. Filtered rather than + # discarded (`2>/dev/null`) so genuine psql errors still surface — losing + # those would turn a broken query into a silently wrong assertion. + # + # Suppressing it at the source needs `[engine] compose_warning_logs = false` + # in containers.conf, which is per-developer config and cannot be relied on + # in CI. docker compose -f "$COMPOSE_FILE" exec -T postgres-test \ - psql -U oxicloud_test -d oxicloud_test -tAqc "$1" + psql -U oxicloud_test -d oxicloud_test -tAqc "$1" \ + 2> >(grep -v 'Executing external compose provider' >&2) } TOKEN=$(curl -sf -X POST "$base_url/api/auth/login" \ From c04f0c6824ceb436f8d5a21d5b64eb365fcea71c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 13:30:58 +0200 Subject: [PATCH 39/66] docs(plan): scope step 7, the transcode dual-write MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigated but deliberately not started — the remaining pieces span a service, its DI wiring and an unresolved design question, and a half-wired service is worse than none. ImageTranscodeService does not write the derived tier at all today, the same gap persist_rendered closed for thumbnails, and it must close before transcode_import can converge or the import chases a growing cache. Three pieces, and the first is smaller than it looks. get_transcoded takes file_id while the table is content-keyed, but the hash is already in scope one frame up — file_retrieval_service::try_transcode is called where dto.content_hash is live — so it is a parameter to thread, not a lookup to invent. Explicitly NOT by hashing original_content on the fly, which would be a BLAKE3 over the whole file per request. Second, the service has no BlobHandler field, so this touches the constructor and DI; ThumbnailService hit the same ordering problem and solved it with a per-call parameter, which is the cheaper precedent. Third, the .skip markers stay unresolved: a negative verdict has no bytes, so it does not fit a table whose row points at a blob. Either leave them local and recompute per instance, or model a sentinel. It is the only genuinely open design question left in step 10. --- docs/plan/derived-blobs.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 91fd0558..6e92b386 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1289,7 +1289,38 @@ hardcoded SQL). New sources bolt on independently. after, step 5 lands at scale; per-row HEADs do not survive a 4× row count. 7. **`ImageTranscodeService`** — same shape, `kind = 'transcode'`, - no new table. + no new table. **Scoped 2026-08-27, not started.** The service does + not currently write the derived tier at all, which is the same gap + `persist_rendered` closed for thumbnails, and it must be closed + before `transcode_import` can converge. + + Three concrete pieces, in order: + + * **Thread the source hash to the call site.** `get_transcoded` takes + `file_id` only, and the table is content-keyed. The hash IS + available one frame up — `file_retrieval_service::try_transcode` + is called from a scope holding `dto.content_hash` — so it is a + parameter to add, not a lookup to invent. Do **not** hash + `original_content` on the fly: that is a BLAKE3 over the whole + file on every request. + * **Give the service a `BlobHandler`.** It has no field for one + (`cache_dir`, `memory_cache`, `stats`), so this is a constructor + and DI change — mind the construction order, as + `ThumbnailService` hit the same thing and solved it with a + per-call parameter instead. + * **Decide the `.skip` markers.** `{file_id}.{ext}.skip` records a + negative verdict ("result was not smaller — serve the original") + and has no bytes, so it does not fit a table whose row points at a + blob. Options: leave them as a purely local cache and accept the + verdict being recomputed per instance, or model it with a sentinel + `blob_hash`. Unresolved; it is the only genuinely open design + question in step 10. + + Note the cache is keyed `{file_id}:{ext}` in memory and + `.transcoded/{ext}/{file_id}.{ext}` on disk, so it also carries the + file-vs-content keying mismatch that `transcode_import` has to + re-key. Fixing the write path first means the import only has to + handle history, not a moving target. 8. **`storage.copy_file_satellites` consolidation** — collapse the two copy paths onto one helper, with a manifest-aware reference bump. **Blocks step 9**: adding a file-keyed table before this means From 647929ed11559961679a28b8ab3e86d5e21b4d38 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 13:46:32 +0200 Subject: [PATCH 40/66] docs(plan): negative verdicts get a nullable blob_hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the .skip-marker question. The table can say "here is the artifact" but not "there is deliberately no artifact", and absence of a row is ambiguous — it means both never-attempted and attempted-and-not- worth-it, which destroys the only thing a negative cache holds. Two live cases, not one. Transcodes drop a .skip marker when WebP is not smaller. Thumbnails return empty Bytes for undecodable sources and ones over MAX_DECODE_PIXELS — RAM-only today, so after moka evicts, a 60-megapixel upload has its full decode attempted again, forever. Same gap, same table, so do both together. Nullable rather than a sentinel hash: a sentinel stops blob_hash naming a real blob and every future reader has to know the lie. Costs are one `AND blob_hash IS NOT NULL` in ContentDerivedReferenceSource, the same guard on the dangling-derived check, and dropping NOT NULL. Permanent vs transient is the load-bearing split, and today's code cannot tell them apart: generate_and_persist collapses every error into empty Bytes, timeouts and semaphore closures included. Survivable while the sentinel lives in moka, which evicts. Persist that same signal and a thumbnail that timed out once under load is unrenderable forever. So the renderer must return a typed outcome first, and only permanently-unrenderable earns a row. The asymmetry sets the default — a wrongly-cached transient is silent and permanent, a not-cached permanent only costs repeated work — so unclassified errors are treated as transient. No TTL and no renderer-version term. Time is the wrong axis: the verdict is deterministic in (content, encoder) and does not decay, so a TTL re-attempts an OOMing decode on a schedule while still leaving staleness for most of the window after a deploy. What removes the need for a mechanism is that negative rows are disposable — they hold no data, so a library upgrade invalidates them with a DELETE ... WHERE blob_hash IS NULL in the same migration as the dependency bump. Recorded explicitly so nobody later builds the expiry logic this replaces. --- docs/plan/derived-blobs.md | 109 +++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 10 deletions(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 6e92b386..5aa2ca82 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -339,6 +339,97 @@ Same trims as `content_derived_blobs` — no `size`, no `format`, no Generic naming rather than `file_previews` because the family is real, and each member would otherwise be a new table plus a new `BlobReferenceSource` plus a new term in the consistency recompute. + +### Negative verdicts — `blob_hash` must be nullable + +*(Resolved 2026-08-27. Supersedes the "`.skip` markers are an open +question" note.)* + +The table says "here is the artifact". It cannot say **"there is +deliberately no artifact"**, and absence of a row is ambiguous — it +means both *never attempted* and *attempted, not worth it*. Collapsing +those destroys the only information a negative cache exists to hold. + +Two live cases, not one: + +* **Transcode not beneficial.** `ImageTranscodeService` can only learn + whether WebP is smaller by doing the full decode + encode. When it is + not, it records a zero-byte `{file_id}.{ext}.skip` marker so the next + GET does not repeat the work. +* **Thumbnail not renderable.** `generate_and_persist` returns empty + `Bytes` — "moka's zero-weight negative-entry convention" — for sources + over `MAX_DECODE_PIXELS` (50 MP) or that fail to decode. This one is + **RAM-only**: after moka evicts, a 60-megapixel upload has its full + decode attempted again, forever. + +So this is not a transcode quirk. The rule generalises to every `kind`: + +> **Any derivation whose failure is deterministic in the source content +> is worth memoising negatively.** + +**The discriminator, or the table fills with noise.** Persist a negative +only when it is *both* expensive to compute *and* deterministic in the +content. `can_transcode(mime)` and "not an image" are cheap metadata +checks — recomputing is free and a row would be pure overhead. It is the +ones that cost a decode that earn a row. + +**Permanent vs transient is the load-bearing split.** "Deterministic" +above means *a property of the content*, not merely *a failure that +happened*: + +| Permanent — cache it | Transient — never cache it | +|---|---| +| decode failed (corrupt / unsupported) | generation timeout | +| over `MAX_DECODE_PIXELS` | decode semaphore closed | +| transcode result not smaller | blob read I/O error, OOM under load | + +**Today's code cannot tell them apart, and that must be fixed before any +of this is persisted.** `generate_and_persist` collapses *every* error +into empty `Bytes` — timeouts and semaphore closures included. That is +survivable while the sentinel lives only in moka, which evicts; write +the same signal to the database and a thumbnail that timed out once +under load is marked unrenderable **forever**. So the renderer must +return a typed outcome — rendered / permanently-unrenderable / transient +failure — and only the middle one earns a row. + +The asymmetry sets the default. A wrongly-cached transient is silent and +permanent; a not-cached permanent merely costs repeated work. **When in +doubt, do not cache** — treat unclassified errors as transient. + +**Representation: nullable `blob_hash`.** A sentinel hash was considered +and rejected — it stops `blob_hash` naming a real blob, and every future +reader has to know the lie. NULL says what is true. Costs: + +* `ContentDerivedReferenceSource` needs `AND blob_hash IS NOT NULL`; a + row holding no blob holds no reference. +* The dangling-derived check (row 9 of the coverage matrix) needs the + same guard, or every negative verdict reports as a broken row. +* The `NOT NULL` constraint is dropped. + +**No TTL, and no renderer-version term either.** Both were considered. +Time is the wrong axis: the verdict is deterministic in +`(content, encoder)` and does not decay, so a TTL re-attempts an OOMing +decode on a schedule — reintroducing the exact waste the negative +exists to prevent — while still leaving staleness for most of the window +after a deploy. + +What makes the mechanism unnecessary is that **negative rows are +disposable**: they hold no data, so discarding one costs only a +re-derivation. Upgrading the image library invalidates them with a line +in the same migration as the dependency bump — + +```sql +DELETE FROM storage.content_derived_blobs WHERE blob_hash IS NULL; +``` + +— which beats a version term that must be remembered and leaves dead +rows behind when bumped. Write this down, or someone later builds the +expiry logic this paragraph exists to prevent. + +Same shape, outside this table and not solved here: +`blob_extracted_text` (no extractable text) and `faces.faces` (no faces +detected) are both deterministic negatives currently indistinguishable +from "never processed". With `kind` it's a one-line `ALTER … CHECK`: | Kind | Why it lands here | @@ -1308,13 +1399,10 @@ hardcoded SQL). New sources bolt on independently. and DI change — mind the construction order, as `ThumbnailService` hit the same thing and solved it with a per-call parameter instead. - * **Decide the `.skip` markers.** `{file_id}.{ext}.skip` records a - negative verdict ("result was not smaller — serve the original") - and has no bytes, so it does not fit a table whose row points at a - blob. Options: leave them as a purely local cache and accept the - verdict being recomputed per instance, or model it with a sentinel - `blob_hash`. Unresolved; it is the only genuinely open design - question in step 10. + * **Write the `.skip` markers as negative rows** — resolved: nullable + `blob_hash`, see *Negative verdicts*. Thumbnails need the same + treatment for undecodable and over-`MAX_DECODE_PIXELS` sources, + which are RAM-only today, so do both together rather than twice. Note the cache is keyed `{file_id}:{ext}` in memory and `.transcoded/{ext}/{file_id}.{ext}` on disk, so it also carries the @@ -1349,9 +1437,10 @@ hardcoded SQL). New sources bolt on independently. is still needed: `ImageTranscodeService` **already exists** and caches `.transcoded/{ext}/{file_id}.{ext}`, so those must be **re-keyed** file→content on import (legitimate only because a - transcode is derivable). Its `.skip` markers — a cached negative - verdict with no bytes — remain an open question, since they do not - fit a table whose point is pointing at a blob. + transcode is derivable). Its `.skip` markers import as **negative + rows** with a NULL `blob_hash` — see *Negative verdicts* — so the + verdict survives the deletion of `.transcoded/`, which it + otherwise would not. **Not next, and deliberately so.** Two prerequisites, both learned the hard way on the thumbnail side: From f4e4b518a9997f54e0471865176cf1a6b29c6a3b Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 18:32:49 +0200 Subject: [PATCH 41/66] feat(thumbnails): classify render failures as permanent or transient MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prerequisite for persisting negative verdicts, and the reason step 7 cannot start with the storage change. generate_and_persist collapses EVERY error into empty Bytes — timeouts and closed semaphores included. That is survivable only because the sentinel lives in moka, which evicts. Write the same signal to content_derived_blobs and a thumbnail that timed out once under load is unrenderable forever. So the classification lands first, on its own, before anything can persist it. It maps onto the existing variants without restructuring, because timeouts already surface as TaskError: ImageError, UnsupportedFormat -> permanent. The decoder rejected these bytes, or they exceed MAX_DECODE_PIXELS. Facts about the image. TaskError, IoError -> transient. Timeout, closed decode semaphore, join failure, unreadable source. Facts about the moment. The asymmetry sets the default: a wrongly-persisted transient marks a good image unrenderable for good, while a wrongly-omitted permanent merely costs a repeated decode. Anything not clearly a content property is therefore transient. Tested by pinning the mapping rather than trusting variant names to stay put — the timeout case especially, since it is the one that turns a load spike into data loss. Nothing consumes this yet; it exists so the storage change cannot be written without it. --- .../services/thumbnail_service.rs | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index bb596d80..f852f761 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -2023,6 +2023,37 @@ pub enum ThumbnailError { UnsupportedFormat, } +impl ThumbnailError { + /// Is this failure a property of the CONTENT, rather than of the moment? + /// + /// Prerequisite for persisting negative verdicts to + /// `content_derived_blobs` (see `docs/plan/derived-blobs.md` §Negative + /// verdicts). Only a permanent failure may be recorded: it will give the + /// same answer forever, so remembering it saves a decode. A transient one + /// must never be recorded — the next attempt may well succeed, and a row + /// saying otherwise is silent, permanent data loss for that file. + /// + /// The asymmetry is why the default is `false`. A wrongly-persisted + /// transient marks a perfectly good image unrenderable for good; a + /// wrongly-omitted permanent merely costs a repeated decode. So anything + /// not clearly a content property is treated as transient. + /// + /// * [`Self::ImageError`] — the decoder rejected these bytes, or they + /// exceed `MAX_DECODE_PIXELS`. Both are facts about the image. + /// * [`Self::UnsupportedFormat`] — likewise. + /// * [`Self::TaskError`] — timeout, closed decode semaphore, join + /// failure. All say the machine was busy, not that the image is bad. A + /// timeout under load is the exact case that must not be cached. + /// * [`Self::IoError`] — the source could not be read. Says nothing about + /// whether it is renderable. + pub fn is_permanent(&self) -> bool { + match self { + ThumbnailError::ImageError(_) | ThumbnailError::UnsupportedFormat => true, + ThumbnailError::TaskError(_) | ThumbnailError::IoError(_) => false, + } + } +} + /// Statistics about the thumbnail cache #[derive(Debug, Clone)] pub struct ThumbnailStats { @@ -2229,6 +2260,27 @@ mod tier_selection_tests { ); } + /// A timeout must never be recorded as a permanent verdict. + /// + /// It is the case that turns a load spike into permanent data loss: + /// `generate_and_persist` collapses every error into empty `Bytes`, which + /// is survivable only while that sentinel lives in moka and evicts. + /// Before any of it reaches `content_derived_blobs`, timeouts must + /// classify as transient — so this pins the mapping rather than trusting + /// the variant names to stay put. + #[test] + fn only_content_failures_are_permanent() { + // Facts about the image — safe to remember. + assert!(ThumbnailError::ImageError("decode failed".into()).is_permanent()); + assert!(ThumbnailError::UnsupportedFormat.is_permanent()); + + // Facts about the moment — must never be remembered. `timeout(...)` + // and the decode semaphore both surface as TaskError. + assert!(!ThumbnailError::TaskError("thumbnail generation timed out".into()).is_permanent()); + assert!(!ThumbnailError::TaskError("Decode semaphore closed".into()).is_permanent()); + assert!(!ThumbnailError::IoError("blob read failed".into()).is_permanent()); + } + /// Empty bytes are moka's negative-entry convention (a previous render /// failed). They must not be served as a thumbnail, or a failure gets /// cached and returned as success. From 12158ccf592d327b89a432e65d590547ef666d50 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 19:06:09 +0200 Subject: [PATCH 42/66] fix(storage): thumb_derived_import claims JPEG sidecars too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The filter was strip_suffix(".webp"), but persist_rendered writes {hash}.{format.ext()} — so any client not advertising WebP leaves {hash}.jpg on disk. Correct only while the derived tier was WebP-only; once variant carried the format (20261022000000) a JPEG sidecar became ordinary content, and leaving it unclaimed would keep .thumbnails/ permanently non-empty — the very signal step 10e gates on. The migration could never finish. Both codecs are now claimed and the format comes from the file's own extension, so a .jpg imports AS JPEG. Deriving the variant and content_type from it rather than hardcoding WebP is the point: a mislabelled row would serve the wrong codec to whoever the read path then matched it for. ThumbnailFormat::ALL exists so the claim list and the write path cannot drift — adding a format without teaching the import about it would strand that codec silently. The `ext-` rejection now carries real weight. Previously .jpg was rejected wholesale, so the two jobs could not overlap by construction; now they share an extension and only the prefix separates them. Both directions stay under test. Caught by the cross-job assertion, which counts every real sidecar being claimed exactly once — the fixture gained a .jpg and the total moved 3 to 4, which is the test noticing rather than a test to update. --- src/application/ports/thumbnail_ports.rs | 6 ++ .../services/thumb_attached_import_service.rs | 7 +- .../services/thumb_derived_import_service.rs | 96 +++++++++++++------ 3 files changed, 78 insertions(+), 31 deletions(-) diff --git a/src/application/ports/thumbnail_ports.rs b/src/application/ports/thumbnail_ports.rs index 195f384b..1fa7cff7 100644 --- a/src/application/ports/thumbnail_ports.rs +++ b/src/application/ports/thumbnail_ports.rs @@ -77,6 +77,12 @@ pub enum ThumbnailFormat { } impl ThumbnailFormat { + /// Every format, for callers that must handle all of them — notably + /// `thumb_derived_import`, which claims one sidecar extension per format + /// and would silently strand a codec if this list and the write path + /// drifted apart. + pub const ALL: [ThumbnailFormat; 2] = [ThumbnailFormat::Webp, ThumbnailFormat::Jpeg]; + /// Stable name, byte-identical to the derived `Debug` output (see /// [`ThumbnailSize::as_str`] — same ETag-stability contract). pub fn as_str(self) -> &'static str { diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index 0486f660..63a19542 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -386,10 +386,13 @@ mod tests { ); } // And nothing real is dropped: README.txt is the only unclaimed file. + // Two content-keyed .webp, one content-keyed .jpg, one ext- upload. + // The .jpg pair is the interesting one: same extension, opposite + // keying, and only the `ext-` prefix separates them. assert_eq!( attached.len() + derived.len(), - 3, - "the three real sidecars must be claimed exactly once between them" + 4, + "every real sidecar must be claimed exactly once between the two jobs" ); } diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 570305ed..c52da984 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -39,7 +39,7 @@ use async_trait::async_trait; use bytes::Bytes; use tokio::fs; -use crate::application::ports::thumbnail_ports::ThumbnailSize; +use crate::application::ports::thumbnail_ports::{ThumbnailFormat, ThumbnailSize}; use crate::infrastructure::scheduler::{ JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, RunStatus, record_or_log, @@ -76,19 +76,35 @@ impl ThumbDerivedImport { self } - /// The hash a sidecar filename names, or `None` when the file is not one. + /// The hash and format a sidecar filename names, or `None` when the file + /// is not one of ours. /// /// Strict, and deliberately rejects `ext-{file_id}.jpg`: those are /// user-supplied, file-keyed bytes. Importing them here would content-key /// them and share one user's uploaded preview onto every file with /// identical content — the poisoning `file_attached_blobs` exists to - /// prevent. They belong to `thumb_attached_import`. - fn hash_from_sidecar_name(name: &str) -> Option<&str> { - let stem = name.strip_suffix(".webp")?; + /// prevent. They belong to `thumb_attached_import`. That rejection + /// carries the weight now that `.jpg` is otherwise claimed, since the two + /// jobs would otherwise both want it. + /// + /// Returns the format too, because the row + /// key needs both since migration `20261022000000`. + /// + /// Both codecs are claimed. `persist_rendered` writes + /// `{hash}.{format.ext()}`, so any client that does not advertise WebP + /// leaves `{hash}.jpg` on disk. While the derived tier was WebP-only + /// those were unmigratable by design; now that `variant` carries the + /// format they are ordinary content, and skipping them would leave + /// `.thumbnails/` permanently non-empty — which is the signal step 10e + /// gates the fallback removal on. + fn hash_from_sidecar_name(name: &str) -> Option<(&str, ThumbnailFormat)> { + let (stem, format) = ThumbnailFormat::ALL + .iter() + .find_map(|f| name.strip_suffix(&format!(".{}", f.ext())).map(|s| (s, *f)))?; if stem.len() != 64 || !stem.chars().all(|c| c.is_ascii_hexdigit()) { return None; } - Some(stem) + Some((stem, format)) } /// Sorted sidecar filenames for one size directory. @@ -159,25 +175,14 @@ impl RecoverableJobHandler for ThumbDerivedImport { let mut already = 0u64; let mut failed = 0u64; let mut since_checkpoint = 0usize; - // Two different strings, and conflating them is a real trap: the - // DIRECTORY is `{size}` on disk, while the VARIANT is - // `{size}.{ext}` since migration `20261022000000`. Using the variant - // as a path yields `.thumbnails/preview.webp/…`, which does not - // exist, so every file reads as unreadable and nothing imports. - // - // Sidecars here are always `{hash}.webp` — the name filter requires - // that extension — so the variant is unconditionally the WebP one. - let variant_of = |s: ThumbnailSize| { - format!( - "{}.{}", - s.dir_name(), - crate::application::ports::thumbnail_ports::ThumbnailFormat::Webp.ext() - ) - }; - + // The DIRECTORY is `{size}` on disk; the VARIANT is `{size}.{ext}` + // since migration `20261022000000`. Conflating them is a real trap: + // using the variant as a path yields `.thumbnails/preview.webp/…`, + // which does not exist, so every file reads as unreadable and nothing + // imports. The variant is therefore built per FILE, from the format + // its extension names, not once per size. for size in ThumbnailSize::all() { let dir_name = size.dir_name(); // on-disk directory - let variant = variant_of(*size); // content_derived_blobs.variant for name in Self::sidecar_names(&self.thumbnails_root, *size).await { // Cursor position uses the DIRECTORY, so a run paused before // this change resumes at the same place. @@ -204,9 +209,15 @@ impl RecoverableJobHandler for ThumbDerivedImport { } } - let Some(hash) = Self::hash_from_sidecar_name(&name) else { + let Some((hash, format)) = Self::hash_from_sidecar_name(&name) else { continue; }; + // Both derived from the file's OWN extension, so a `.jpg` + // sidecar becomes a JPEG row rather than being mislabelled + // WebP — which would serve the wrong codec to anyone the read + // path then matched it for. + let variant = format!("{dir_name}.{}", format.ext()); + let content_type = format.mime(); // Already mapped — the common case on a re-run, and the // reason this job is safe to trigger repeatedly. @@ -227,7 +238,7 @@ impl RecoverableJobHandler for ThumbDerivedImport { hash, "thumbnail", &variant, - "image/webp", + content_type, Bytes::from(data), ) .await @@ -310,12 +321,27 @@ pub(crate) mod tests { use super::*; const H: &str = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"; + /// A second hash, for the JPEG sidecar in `legacy_tree`. + const H2: &str = "c222222222222222222222222222222222222222222222222222222222222222"; + /// BOTH codecs are claimed, and the format comes from the extension. + /// + /// `.jpg` was previously rejected here, which was correct only while the + /// derived tier was WebP-only. Once `variant` carried the format + /// (migration `20261022000000`) a JPEG sidecar became ordinary content, + /// and leaving it unclaimed would keep `.thumbnails/` permanently + /// non-empty — the very signal step 10e gates on. #[test] - fn accepts_a_canonical_sidecar_name() { + fn accepts_both_codecs_and_reports_the_format() { assert_eq!( ThumbDerivedImport::hash_from_sidecar_name(&format!("{H}.webp")), - Some(H) + Some((H, ThumbnailFormat::Webp)) + ); + assert_eq!( + ThumbDerivedImport::hash_from_sidecar_name(&format!("{H}.jpg")), + Some((H, ThumbnailFormat::Jpeg)), + "a JPEG sidecar must import, and as JPEG — labelling it WebP \ + would serve the wrong codec" ); } @@ -349,6 +375,13 @@ pub(crate) mod tests { .await .unwrap(); // Neither: a stray file that must be claimed by no one. + // Server-rendered JPEG: what a client not advertising WebP + // leaves behind. Claimed by the derived import, and must not be + // confused with the `ext-` upload above despite sharing an + // extension. + tokio::fs::write(dir.join(format!("{H2}.jpg")), b"jpeg") + .await + .unwrap(); tokio::fs::write(dir.join("README.txt"), b"nope") .await .unwrap(); @@ -371,8 +404,10 @@ pub(crate) mod tests { vec![ format!("{H}.webp"), "b111111111111111111111111111111111111111111111111111111111111111.webp".to_string(), + format!("{H2}.jpg"), ], - "must claim both content-keyed sidecars, sorted, and nothing else" + "must claim every content-keyed sidecar of EITHER codec, sorted, \ + and nothing else" ); } @@ -394,12 +429,15 @@ pub(crate) mod tests { #[test] fn rejects_external_and_malformed_names() { for name in [ + // `ext-` prefixed: user-supplied and file-keyed, whatever the + // extension. Now that .jpg is otherwise claimed, this is the case + // that keeps the two jobs disjoint. format!("ext-{H}.jpg"), "ext-3f2b1c00-0000-0000-0000-000000000000.jpg".to_string(), - format!("{H}.jpg"), format!("{}.webp", &H[..63]), H.to_string(), "junk.webp".to_string(), + "junk.jpg".to_string(), ] { assert_eq!( ThumbDerivedImport::hash_from_sidecar_name(&name), From 80d53721313a5367b7616f232a79113d177807b3 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 22:22:07 +0200 Subject: [PATCH 43/66] test(api): cover WebP/JPEG negotiation, which nothing exercised MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit from_accept returns JPEG unless Accept contains image/webp, and no thumbnail test sent the header — curl defaults to */*, which does not match. So the whole suite ran on JPEG and the WebP path was never exercised over HTTP, despite being what background generation writes and what the derived tier was built around. The gap was invisible because the JPEG results were all correct. Three assertions, one property each. Content-Type proves negotiation happened: thumbnail_content_type sniffs the body with `infer` rather than echoing the request, so image/webp cannot be right by accident — serve JPEG bytes down the WebP path and it reads image/jpeg and fails. Differing bytes prove they are genuinely two artifacts rather than one served twice. Differing ETags prove the validators are separate. `variant` has carried the format only since 20261022000000; before that a JPEG request could match the WebP row and be served the wrong codec, and a shared validator is exactly how a cache would then hand either to either. A final conditional request confirms each codec revalidates against its own. Together these also cover the per-format variant keying that lets one source hold both codecs — the prerequisite for JPEG clients ever leaving the sidecar, and therefore for step 10e. Note on placement: thumb_etag stays in step 4's capture block, beside thumb_bytes. Every later request omits Accept and so negotiates JPEG, so the validator must be the JPEG one — captured after the new block it would describe a different codec than the bytes next to it, and the copy assertions compare against both. --- tests/api/derived_blob_copy.hurl | 59 ++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/tests/api/derived_blob_copy.hurl b/tests/api/derived_blob_copy.hurl index 488c9fba..7f6f0345 100644 --- a/tests/api/derived_blob_copy.hurl +++ b/tests/api/derived_blob_copy.hurl @@ -140,9 +140,68 @@ Authorization: Bearer {{token}} HTTP 200 [Captures] thumb_bytes: bytes +# The JPEG validator, since every later request omits `Accept` and so +# negotiates JPEG too. Captured here rather than after step 4b, or it +# would belong to a different codec than the bytes beside it. thumb_etag: header "ETag" +# ───────────────────────────────────────────────────────────── +# Step 4b – Codec negotiation: WebP and JPEG are separate artifacts. +# +# Every other request in the suite omits `Accept`, and curl defaults to +# `*/*`, which `ThumbnailFormat::from_accept` maps to JPEG — so without +# this case the WebP path is never exercised at all, despite being what +# background generation writes and what the derived tier was built +# around. +# +# The three assertions are one property each: +# +# Content-Type — negotiation actually happened (it is byte-sniffed +# from the body, so it cannot be right by accident). +# bytes — the two really are different artifacts. +# ETag — the validators are distinct. `variant` carries the +# format since migration 20261022000000; before that a +# JPEG request could match the WebP row and be served +# the wrong codec, and a shared validator is how a +# cache would then hand either to either. +# +# Together they also cover the per-format variant keying that lets one +# source hold both codecs — the prerequisite for JPEG clients ever +# leaving the sidecar. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +Accept: image/webp + +HTTP 200 +[Captures] +webp_bytes: bytes +webp_etag: header "ETag" +[Asserts] +header "Content-Type" == "image/webp" + + +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +Accept: image/jpeg + +HTTP 200 +[Asserts] +header "Content-Type" == "image/jpeg" +bytes != {{webp_bytes}} +header "ETag" != "{{webp_etag}}" + + +# Each codec revalidates against its OWN validator. +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} +Accept: image/webp +If-None-Match: {{webp_etag}} + +HTTP 304 + + # ───────────────────────────────────────────────────────────── # Step 5 – Single-file copy into the destination folder # ───────────────────────────────────────────────────────────── From ae9ff0d8b347d0ca725ff597f4c977b9a6585941 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 22:58:05 +0200 Subject: [PATCH 44/66] feat(storage): thumb_derived_import drains the sidecars it imports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 10d, for the content-keyed half. Opt-in via the existing `repair` flag rather than a new one — the house rule is that a job does not mutate on its default setting, so early runs import only and an operator can inspect before committing. Deleting from the job rather than a later release is what makes the migration self-draining. Sidecars are LOCAL disk, so no release can know whether every instance has finished; each instance draining itself needs no coordination at all. Verification before unlinking is the load-bearing part. store_derived_blob reporting success is not proof the bytes are retrievable — a backend that accepted a write it cannot serve would otherwise have the last copy deleted on top of it. The blob is read back and its length compared against the sidecar's; a failure keeps the file, records a finding, and the next run retries. That read is the difference between a migration and a data-loss bug. Deletion applies to already-imported files too, not just fresh ones. A run without `repair` leaves the sidecar behind, and a later run with it would otherwise classify the file as "already imported" and never drain it — and import-then-enable-deletion is the expected operator sequence, so that is the common path rather than an edge case. Directories are removed once genuinely empty, because ABSENCE is what step 10e gates on, not emptiness: empty is momentary and an on-demand render can repopulate it a second later, while absence is one-way and cheaper to test (one stat, versus opendir/readdir/closedir). remove_dir refuses a non-empty directory, so it needs no emptiness check and cannot race a concurrent write into deleting live files. thumb_attached_import still needs the same treatment; verify_and_unlink should move somewhere shared rather than being copied into it. --- .../services/thumb_derived_import_service.rs | 120 +++++++++++++++++- 1 file changed, 115 insertions(+), 5 deletions(-) diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index c52da984..900d3cfe 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -107,6 +107,35 @@ impl ThumbDerivedImport { Some((stem, format)) } + /// Delete a sidecar, but only after proving the blob that replaced it can + /// actually be read back. + /// + /// The verification is the whole point. `store_derived_blob` reporting + /// success is not proof the bytes are retrievable — a backend that + /// accepted a write it cannot serve would otherwise have the last copy + /// deleted on top of it. This is a migration, and the difference between + /// a migration and a data-loss bug is exactly this read. + /// + /// Length is compared rather than full bytes: it catches the realistic + /// failures (absent, empty, truncated) without a second full read of the + /// sidecar, which the already-imported path would otherwise need. + /// + /// Returns whether the file was removed. A failed verification leaves the + /// sidecar in place — the run reports it and the next one retries, which + /// is the safe direction. + async fn verify_and_unlink(&self, derived_hash: &str, path: &std::path::Path) -> bool { + let Ok(meta) = fs::metadata(path).await else { + return false; + }; + let Ok(stored) = self.dedup.read_blob_bytes(derived_hash).await else { + return false; + }; + if stored.is_empty() || stored.len() as u64 != meta.len() { + return false; + } + fs::remove_file(path).await.is_ok() + } + /// Sorted sidecar filenames for one size directory. /// /// Sorted so the cursor is meaningful: resume skips everything at or @@ -152,9 +181,19 @@ impl RecoverableJobHandler for ThumbDerivedImport { async fn run_resumable( &self, store: &dyn JobStore, - _args: &JobRunArgs, + args: &JobRunArgs, resume_cursor: Option>, ) -> RunOutcome { + // `?repair=true` opts into deleting each sidecar once it has been + // imported AND read back. Off by default, matching the house rule + // that a job does not mutate on its default setting — early runs + // import only, so an operator can inspect before committing. + // + // Deleting from the job rather than from a later release is what + // makes the migration self-draining: sidecars are LOCAL disk, so no + // release can know whether every instance has finished, whereas each + // instance draining itself needs no coordination at all. + let delete_imported = args.repair; // Cursor is `{size_dir}/{filename}` — the last file completed. Sizes // are walked in `ThumbnailSize::all()` order, and names are sorted // within each, so the pair totally orders the walk. @@ -174,6 +213,8 @@ impl RecoverableJobHandler for ThumbDerivedImport { let mut imported = 0u64; let mut already = 0u64; let mut failed = 0u64; + let mut deleted = 0u64; + let mut unverified = 0u64; let mut since_checkpoint = 0usize; // The DIRECTORY is `{size}` on disk; the VARIANT is `{size}.{ext}` // since migration `20261022000000`. Conflating them is a real trap: @@ -221,13 +262,40 @@ impl RecoverableJobHandler for ThumbDerivedImport { // Already mapped — the common case on a re-run, and the // reason this job is safe to trigger repeatedly. - if self + // + // Deletion applies here too, not just to fresh imports: a run + // without `repair` leaves the sidecar behind, and a later run + // with it would otherwise classify the file as "already + // imported" and never drain it. Import-then-enable-deletion + // is the expected operator sequence, so this is the common + // path, not an edge case. + if let Some(existing) = self .dedup .find_derived_blob(hash, "thumbnail", &variant) .await - .is_some() { already += 1; + if delete_imported { + let path = self.thumbnails_root.join(dir_name).join(&name); + if self.verify_and_unlink(&existing.blob_hash, &path).await { + deleted += 1; + } else { + unverified += 1; + record_or_log( + store, + THUMB_DERIVED_IMPORT_JOB_NAME, + "sidecar_delete_unverified", + "anomaly", + None, + serde_json::json!({ + "path": position, + "hash": hash, + "note": "derived blob did not read back; sidecar kept", + }), + ) + .await; + } + } } else { let path = self.thumbnails_root.join(dir_name).join(&name); match fs::read(&path).await { @@ -243,7 +311,29 @@ impl RecoverableJobHandler for ThumbDerivedImport { ) .await { - Ok(_) => imported += 1, + Ok(derived_hash) => { + imported += 1; + if delete_imported { + if self.verify_and_unlink(&derived_hash, &path).await { + deleted += 1; + } else { + unverified += 1; + record_or_log( + store, + THUMB_DERIVED_IMPORT_JOB_NAME, + "sidecar_delete_unverified", + "anomaly", + None, + serde_json::json!({ + "path": position, + "hash": hash, + "note": "derived blob did not read back; sidecar kept", + }), + ) + .await; + } + } + } Err(e) => { failed += 1; record_or_log( @@ -298,6 +388,23 @@ impl RecoverableJobHandler for ThumbDerivedImport { } } + // Remove the size directories once genuinely empty, because ABSENCE + // is what step 10e gates the fallback removal on — not emptiness. + // Empty is momentary: an on-demand render can repopulate it the next + // second. Absence is one-way, and far cheaper to test besides — one + // `stat` versus an opendir/readdir/closedir. + // + // `remove_dir` refuses a non-empty directory, so this needs no + // emptiness check of its own and cannot race a concurrent write into + // deleting live files. + if delete_imported { + for size in ThumbnailSize::all() { + let dir = self.thumbnails_root.join(size.dir_name()); + let _ = fs::remove_dir(&dir).await; + } + let _ = fs::remove_dir(&self.thumbnails_root).await; + } + tracing::info!( target: "oxicloud::dedup", event = "thumb_derived_import.completed", @@ -305,7 +412,10 @@ impl RecoverableJobHandler for ThumbDerivedImport { imported = imported, already_present = already, failed = failed, - "thumb_derived_import: {imported} imported, {already} already present, {failed} failed" + deleted = deleted, + unverified = unverified, + "thumb_derived_import: {imported} imported, {already} already present, \ + {failed} failed, {deleted} sidecar(s) deleted, {unverified} kept unverified" ); RunOutcome::completed() From df619a7ed909904de4e9e79f3084723438aa3b2f Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 23:31:31 +0200 Subject: [PATCH 45/66] feat(storage): thumb_attached_import drains its sidecars too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes step 10d. Same `?repair=true` opt-in and the same readback-before-unlink as the derived half, and the check matters more here: these sidecars hold the bytes that CANNOT be regenerated — a client-uploaded PDF preview has no server-side render path — so the read is the only thing between a migration and permanent loss, not belt-and-braces. verify_and_unlink is shared rather than copied. Two versions of "only delete after proving the replacement is readable" would be two chances to weaken one, and it is the rule the whole deletion step rests on. Deletion covers the already-imported branch as well as fresh imports, for the same reason as the derived job: a run without `repair` leaves the sidecar behind, and a later run with it would otherwise see "already imported" and never drain. Import first, enable deletion after, is the expected operator sequence, so that branch is the common path. Orphaned sidecars stay untouched — this job imports, it does not reclaim, and a destructive default on a migration is what no-silent- auto-repair forbids. Unverifiable ones are kept and reported, so the next run retries. With both halves draining, `.thumbnails/` can now actually empty and the directory removal in the derived job can succeed — though not stably until dual-write stops, since any render or upload recreates it. --- .../services/thumb_attached_import_service.rs | 85 +++++++++++++++++-- .../services/thumb_derived_import_service.rs | 21 ++++- 2 files changed, 97 insertions(+), 9 deletions(-) diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index 63a19542..e6304329 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -50,6 +50,10 @@ use crate::infrastructure::scheduler::{ RunStatus, record_or_log, }; use crate::infrastructure::services::dedup_service::DedupService; +// The readback-then-unlink rule is shared, not copied: two versions of it +// would be two chances to weaken one, and this is the check standing between +// a migration and permanent loss. +use crate::infrastructure::services::thumb_derived_import_service::ThumbDerivedImport; pub const THUMB_ATTACHED_IMPORT_JOB_NAME: &str = "thumb_attached_import"; @@ -169,7 +173,7 @@ impl RecoverableJobHandler for ThumbAttachedImport { async fn run_resumable( &self, store: &dyn JobStore, - _args: &JobRunArgs, + args: &JobRunArgs, resume_cursor: Option>, ) -> RunOutcome { // Cursor is `{size_dir}/{filename}`, matching thumb_derived_import: @@ -191,6 +195,16 @@ impl RecoverableJobHandler for ThumbAttachedImport { let mut imported = 0u64; let mut already = 0u64; let mut orphaned = 0u64; + let mut deleted = 0u64; + let mut unverified = 0u64; + // Same opt-in as thumb_derived_import: `?repair=true`. + // + // The readback before unlinking matters more here than there. These + // sidecars are the ones that CANNOT be regenerated — a client-uploaded + // PDF preview has no server-side render path — so it is not + // belt-and-braces, it is the only thing between a migration and + // permanent loss. + let delete_imported = args.repair; let mut failed = 0u64; let mut since_checkpoint = 0usize; @@ -227,13 +241,43 @@ impl RecoverableJobHandler for ThumbAttachedImport { // Already mapped. Checked BEFORE storing, because // `store_attached_blob` is ON CONFLICT DO UPDATE and would // release then retake the reference on every run. - if self + if let Some(existing) = self .dedup .find_attached_blob(&file_id_str, "preview", &dir_name) .await - .is_some() { already += 1; + // Drains on a later run too: importing first and enabling + // deletion afterwards is the expected operator sequence, + // so reaching here is the common path rather than an edge + // case. + if delete_imported { + let path = self.thumbnails_root.join(&dir_name).join(&name); + if ThumbDerivedImport::verify_and_unlink( + &self.dedup, + &existing.blob_hash, + &path, + ) + .await + { + deleted += 1; + } else { + unverified += 1; + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "sidecar_delete_unverified", + "anomaly", + None, + serde_json::json!({ + "path": position, + "file_id": file_id_str, + "note": "attached blob did not read back; sidecar kept", + }), + ) + .await; + } + } } else if !self.file_exists(file_id).await { // The file is gone; the sidecar outlived it. Reported // rather than deleted — this job imports, it does not @@ -272,7 +316,35 @@ impl RecoverableJobHandler for ThumbAttachedImport { ) .await { - Ok(_) => imported += 1, + Ok(attached_hash) => { + imported += 1; + if delete_imported { + if ThumbDerivedImport::verify_and_unlink( + &self.dedup, + &attached_hash, + &path, + ) + .await + { + deleted += 1; + } else { + unverified += 1; + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "sidecar_delete_unverified", + "anomaly", + None, + serde_json::json!({ + "path": position, + "file_id": file_id_str, + "note": "attached blob did not read back; sidecar kept", + }), + ) + .await; + } + } + } Err(e) => { failed += 1; record_or_log( @@ -333,8 +405,11 @@ impl RecoverableJobHandler for ThumbAttachedImport { already_present = already, orphaned = orphaned, failed = failed, + deleted = deleted, + unverified = unverified, "thumb_attached_import: {imported} imported, {already} already present, \ - {orphaned} orphaned, {failed} failed" + {orphaned} orphaned, {failed} failed, {deleted} sidecar(s) deleted, \ + {unverified} kept unverified" ); RunOutcome::completed() diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 900d3cfe..7bef4064 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -123,11 +123,18 @@ impl ThumbDerivedImport { /// Returns whether the file was removed. A failed verification leaves the /// sidecar in place — the run reports it and the next one retries, which /// is the safe direction. - async fn verify_and_unlink(&self, derived_hash: &str, path: &std::path::Path) -> bool { + /// Shared with `thumb_attached_import` rather than copied into it: both + /// jobs delete a sidecar only after proving its replacement is readable, + /// and two copies of that rule would be two chances to weaken one. + pub(crate) async fn verify_and_unlink( + dedup: &DedupService, + stored_hash: &str, + path: &std::path::Path, + ) -> bool { let Ok(meta) = fs::metadata(path).await else { return false; }; - let Ok(stored) = self.dedup.read_blob_bytes(derived_hash).await else { + let Ok(stored) = dedup.read_blob_bytes(stored_hash).await else { return false; }; if stored.is_empty() || stored.len() as u64 != meta.len() { @@ -277,7 +284,7 @@ impl RecoverableJobHandler for ThumbDerivedImport { already += 1; if delete_imported { let path = self.thumbnails_root.join(dir_name).join(&name); - if self.verify_and_unlink(&existing.blob_hash, &path).await { + if Self::verify_and_unlink(&self.dedup, &existing.blob_hash, &path).await { deleted += 1; } else { unverified += 1; @@ -314,7 +321,13 @@ impl RecoverableJobHandler for ThumbDerivedImport { Ok(derived_hash) => { imported += 1; if delete_imported { - if self.verify_and_unlink(&derived_hash, &path).await { + if Self::verify_and_unlink( + &self.dedup, + &derived_hash, + &path, + ) + .await + { deleted += 1; } else { unverified += 1; From 791e2da4f410bb149aabeda0ec11b5039a2f87f8 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 23:42:43 +0200 Subject: [PATCH 46/66] docs(plan): the sequence was missing "stop writing sidecars" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 10 went ...enable deletion, then remove the fallback "once the directory no longer exists". That gate is unreachable as written: while persist_rendered dual-writes and the PUT still writes ext-, any render or upload recreates the tree seconds after the job removes it, so the directory never stays absent and (e) can never fire. Adds it as d2, between deletion and fallback removal, with the split that only became visible while implementing 10d. Rendered sidecars can stop immediately — the read flip has landed, existing files are untouched so un-imported boxes keep their fallback, and a failed derived store costs a re-render rather than data, since that content is regenerable by definition. Uploaded ones cannot, yet. upload_thumbnail_impl logs and still returns 201 when store_attached_blob fails, which is safe only because the ext- sidecar catches it. Remove the sidecar while the store is best-effort and a user's preview vanishes silently behind a success response — and these are precisely the bytes with no server-side render path. So the PUT must become fatal first. Order recorded explicitly: make it fatal, then drop the sidecar. Reversed, it trades a silent data-loss window for an empty directory. --- docs/plan/derived-blobs.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 5aa2ca82..6ffe378e 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1474,7 +1474,38 @@ hardcoded SQL). New sources bolt on independently. format. That is a new prerequisite for (e), not a detail: it means a migration to `(kind, variant, format)` — or a format term inside `variant` — has to land before the directories can go. - d. **Enable deletion** in the import jobs (opt-in, readback-verified). + d. **Enable deletion** in the import jobs (opt-in, readback-verified) + — **done 2026-08-27**, both halves, sharing one + `verify_and_unlink`. + + d2. **Stop writing sidecars.** *(Added 2026-08-27 — the sequence + above was missing this, and (e)'s gate is unreachable without + it: dual-write means any render or upload recreates the + directory seconds after the job removes it, so "no longer + exists" can never hold.)* + + Two sides, and they differ in what a failed write costs: + + * **Rendered / content-keyed — safe now.** Delete the `fs::write` + in `persist_rendered`. No gate beyond the read flip, which has + landed. Existing sidecars are untouched, so a box that has not + imported yet keeps its fallback for old content; new content + goes only to the derived tier, which the read path already + prefers. If the derived store fails the bytes are still served + and simply not cached — regenerable by definition. + * **Uploaded / file-keyed — needs a change first.** + `upload_thumbnail_impl` logs and still returns 201 when + `store_attached_blob` fails, which is safe *only* because the + `ext-` sidecar catches it. Remove that sidecar while the store + is best-effort and a user's uploaded preview can vanish + silently behind a success response. These are the + non-regenerable bytes, so **the PUT must fail** before the + write is removed. + + Order matters: make the attached store fatal, *then* drop its + sidecar. Reversed, it trades a silent data-loss window for an + empty directory. + e. **Remove the fallback read path** once the directory no longer *exists* — not merely once it is empty. Two reasons. Empty is a momentary property an on-demand render can undo, whereas absence From c778b670065cc2da50632f0e916698ce32b75141 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Thu, 27 Aug 2026 23:53:50 +0200 Subject: [PATCH 47/66] feat(thumbnails): stop writing sidecars (step 10d2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The write paths now persist only to the blob tiers. Until this, dual-write meant any render or upload recreated .thumbnails/ seconds after the import job removed it, so step 10e's gate — "the directory no longer exists" — could never hold. Rendered thumbnails: the fs::write in persist_rendered is gone. Safe because the read flip landed first, so nothing depended on that write to be found, and a failed derived store now costs a re-render rather than data — regenerable by definition. Existing sidecars are untouched and stay readable through the fallback until the import drains them. Uploaded previews needed a change first, and the order was not optional. upload_thumbnail_impl logged and still returned 201 when store_attached_blob failed — safe only while ext-{file_id}.jpg was a second copy. These bytes have NO server-side render path, so removing the sidecar while the store stayed best-effort would lose a user's upload behind a success response. The PUT is now fatal, and drops the RAM entry too, or the cache would keep serving a preview that was never persisted and vanishes on eviction, contradicting the error the client just received. Only then does the ext- write go. thumb_import_check.sh had to change with it: its premise was "upload, then delete the row, and what remains on disk is legacy state", which no longer holds now that nothing writes sidecars. It lays them down itself, with the bytes the API just served, at the exact paths the pre-10d2 code used. The reconstruction stays faithful — same bytes, same paths — it just no longer depends on current code to produce a shape current code has stopped producing. Both sidecars get identical bytes, which is realistic rather than a shortcut: they dedup to one blob while keeping separate mappings, which is the property the keying split exists to preserve. --- .../services/thumbnail_service.rs | 50 +++++++++---------- src/interfaces/api/handlers/file_handler.rs | 20 ++++++-- tests/api/thumb_import_check.sh | 26 ++++++++++ 3 files changed, 66 insertions(+), 30 deletions(-) diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index f852f761..72b45e97 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -309,18 +309,18 @@ impl ThumbnailService { bytes: &Bytes, dedup: Option<&DedupService>, ) { - let thumb_path = self.get_thumbnail_path(blob_hash, size, format); - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - if let Err(e) = fs::write(&thumb_path, bytes).await { - tracing::warn!( - "Failed to save thumbnail sidecar {} {:?}: {e}", - &blob_hash[..blob_hash.len().min(12)], - size - ); - } - + // Step 10d2: the sidecar write is GONE. The derived tier is the only + // durable home for a rendered thumbnail now. + // + // Safe because the read flip landed first: reads already prefer the + // derived tier, so nothing depended on this write to be found. And a + // failure below costs a re-render rather than data — a rendered + // thumbnail is regenerable by definition, which is exactly why this + // side could stop before the uploaded one. + // + // Existing sidecars are untouched. They stay readable through the + // fallback tier until the import drains them, so a box that has not + // run the job yet loses nothing. if let Some(dedup) = dedup && let Err(e) = dedup .store_derived_blob( @@ -928,19 +928,19 @@ impl ThumbnailService { let bytes = Bytes::from(jpeg_bytes); - // External thumbnails are stored by file_id (not dedup-able) - let thumb_path = self - .thumbnails_root - .join(size.dir_name()) - .join(format!("ext-{}.jpg", file_id)); - if let Some(parent) = thumb_path.parent() { - let _ = fs::create_dir_all(parent).await; - } - fs::write(&thumb_path, &bytes) - .await - .map_err(|e| ThumbnailError::IoError(e.to_string()))?; - - // Populate in-memory cache (external thumbnails are JPEG) + // Step 10d2: the `ext-{file_id}.jpg` sidecar write is GONE. The + // durable store is now `file_attached_blobs`, written by the caller — + // which is why that write had to become fatal first, in the same + // change. These bytes have no server-side render path, so a + // best-effort store with no sidecar behind it would lose a user's + // upload silently. + // + // This function now re-encodes and caches; it does not persist. The + // RAM entry stays because it is what serves the request that follows, + // and the caller drops it if the durable write fails. + // + // Existing `ext-` files remain readable through the fallback tier + // until `thumb_attached_import` drains them. let cache_key = ThumbnailCacheKey::external(file_id, size); self.cache.insert(cache_key, bytes.clone()).await; diff --git a/src/interfaces/api/handlers/file_handler.rs b/src/interfaces/api/handlers/file_handler.rs index 193d45df..7fcf2194 100644 --- a/src/interfaces/api/handlers/file_handler.rs +++ b/src/interfaces/api/handlers/file_handler.rs @@ -800,16 +800,26 @@ impl FileHandler { ) .await { - // ERROR, not WARN: the sidecar keeps the feature looking healthy - // on this box, so nothing else signals that copies are silently - // losing the preview. A syntax error in the upsert hid behind a - // warning for an entire test cycle exactly this way. + // FATAL as of step 10d2, where it used to warn and return 201. + // + // That was safe only while `ext-{file_id}.jpg` existed as a + // second copy. With the sidecar gone this is the ONLY durable + // home for bytes that have no server-side render path — a + // client-generated PDF preview cannot be recreated — so + // succeeding here would lose a user's upload behind a success + // response. Silent, and unrecoverable. + // + // The RAM entry is dropped too, or the cache would keep serving a + // preview that was never persisted and vanishes on eviction, + // contradicting the error the client just received. + let _ = thumbnail_service.delete_thumbnails(&id).await; tracing::error!( target: "oxicloud::dedup", error = %e, file_id = %id, - "failed to record attached thumbnail; sidecar written, copies will NOT inherit it" + "failed to record attached thumbnail; upload rejected" ); + return AppError::internal_error("Failed to store thumbnail").into_response(); } StatusCode::CREATED.into_response() diff --git a/tests/api/thumb_import_check.sh b/tests/api/thumb_import_check.sh index 763bfd22..552f0895 100755 --- a/tests/api/thumb_import_check.sh +++ b/tests/api/thumb_import_check.sh @@ -45,6 +45,7 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" COMPOSE_FILE="$REPO_ROOT/tests/common/docker-compose.test.yml" +STORAGE_PATH="${OXICLOUD_STORAGE_PATH:-$REPO_ROOT/tests/api/storage}" # shellcheck source=test.env source "$SCRIPT_DIR/test.env" @@ -131,6 +132,31 @@ curl -sf -X PUT -H "$AUTH" -H "Content-Type: image/png" \ UPLOADED_THUMB=$(mktemp) curl -sf -H "$AUTH" "$base_url/api/files/$FILE_ID/thumbnail/preview" -o "$UPLOADED_THUMB" +# ── 1b. Lay down the sidecars the server no longer writes ──────────────── +# +# Since step 10d2 the write paths persist ONLY to the blob tiers, so an +# upload no longer leaves anything under .thumbnails/ — which is the point, +# but it removes the source this test used to manufacture legacy state from. +# +# So write them here, with the bytes the API just served, at the exact paths +# the pre-10d2 code used: `{size}/{blob_hash}.jpg` for the rendered +# thumbnail and `{size}/ext-{file_id}.jpg` for the upload. Requests omit +# `Accept`, so both negotiate JPEG. +# +# This keeps the reconstruction faithful rather than approximate: same +# bytes, same paths, same filenames a pre-migration install holds. What it +# no longer does is rely on the current code to produce them — which it +# cannot, and should not. +SIDECAR_DIR="$STORAGE_PATH/.thumbnails/preview" +mkdir -p "$SIDECAR_DIR" +cp "$UPLOADED_THUMB" "$SIDECAR_DIR/ext-$FILE_ID.jpg" +cp "$UPLOADED_THUMB" "$SIDECAR_DIR/$BLOB_HASH.jpg" +# Identical bytes in both, which is realistic rather than a shortcut: they +# dedup to one blob, so the derived and attached rows end up referencing the +# same content while keeping separate mappings — exactly the property the +# keying split exists to preserve. +log "legacy sidecars written to $SIDECAR_DIR" + DERIVED_BEFORE=$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';") ATTACHED_BEFORE=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") [[ "$DERIVED_BEFORE" -ge 1 ]] || fail "expected a content_derived_blobs row before stripping" From 1b68ee093edf4d3a9b8a3d4296d10caa64b7dd2b Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 00:03:54 +0200 Subject: [PATCH 48/66] feat(storage): both import jobs tick daily instead of manual-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Registered with interval None, so they ran only when someone remembered to trigger them — which was your objection to gating anything on operator timing. Now daily. Not boot-time: that would delay readiness for a filesystem walk, and both jobs are idempotent and resumable, so periodic is strictly better. The tick deliberately does NOT delete. `repair` defaults false, so scheduled runs import and stop; unlinking stays a deliberate operator action, per no-silent-auto-repair. That splits the two halves the way their risk differs — the backfill is safe to automate, removing files is not. Cost once drained is a read_dir over three directories returning nothing, and after the directory itself is removed, not even that. --- .../services/thumb_attached_import_service.rs | 9 ++++++++- .../services/thumb_derived_import_service.rs | 16 +++++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index e6304329..2f13ec43 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -89,8 +89,15 @@ impl ThumbAttachedImport { registry: &JobRegistry, provider: &Arc, ) -> Arc { + // Daily, matching `thumb_derived_import` — and it does not delete on + // the tick either, since `repair` defaults false. See that job for + // the reasoning. registry - .register_recoverable_job(self.clone(), provider.clone(), None) + .register_recoverable_job( + self.clone(), + provider.clone(), + Some(std::time::Duration::from_secs(24 * 3600)), + ) .await; self } diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 7bef4064..dcf0aa55 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -70,8 +70,22 @@ impl ThumbDerivedImport { registry: &JobRegistry, provider: &Arc, ) -> Arc { + // Daily tick rather than manual-only. Ops cannot be relied on to + // remember a migration, and boot-time would delay readiness for a + // filesystem walk — whereas this is idempotent and resumable, so + // periodic is safe and it drains on its own. + // + // The tick does NOT delete: `repair` defaults false, so scheduled + // runs import and stop. Deletion stays a deliberate operator action, + // per no-silent-auto-repair. Once drained, a run is a `read_dir` over + // three directories that returns nothing — and after the directory is + // removed, not even that. registry - .register_recoverable_job(self.clone(), provider.clone(), None) + .register_recoverable_job( + self.clone(), + provider.clone(), + Some(std::time::Duration::from_secs(24 * 3600)), + ) .await; self } From e5746a4f48e3bf5fce33ef0ab5c905de3cc6673c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 00:11:01 +0200 Subject: [PATCH 49/66] test(api): the probe must NOT leave a thumbnail sidecar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit storage_cleanup_check.sh asserted a sidecar exists on disk after fetching a thumbnail. Correct while `.thumbnails/` was the durable store; wrong since 10d2 removed that write. The check failed on exactly the behaviour it was meant to confirm. Inverted rather than deleted, because the inverse is the more useful guard: a sidecar reappearing means a write path regressed to the legacy shape, which would silently make `.thumbnails/` un-emptyable and strand step 10e forever — its gate is the directory being gone, and a single recreated file holds it open. The HTTP 200 above already proves the thumbnail works; this now proves it got there the new way. Both of the helper's streams are silenced at the call site. It reports absence loudly — red banner plus a `find` dump — because absence used to be the failure; here it is the expected result, and leaving that visible would cry wolf on every clean run. --- tests/api/storage_cleanup_check.sh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index 56ee60b4..7fd988d8 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -61,8 +61,25 @@ HTTP_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" -H "$AUTH" \ log "Thumbnail fetched (HTTP 200)." assert_local_blob_existsy "$FIXTURE" "$STORAGE_PATH" || fail "probe blob not found on disk" -assert_preview_existsy "$FIXTURE" "$STORAGE_PATH" || fail "probe thumbnail not found on disk" -log "Probe blob and thumbnail confirmed present on disk." + +# The thumbnail must NOT be on disk — inverted at step 10d2, when the sidecar +# write was removed. +# +# It used to assert the opposite, and that was right while `.thumbnails/` was +# the durable store. Now the durable home is `content_derived_blobs` plus the +# blob tier, and a sidecar reappearing here means a write path regressed to +# the legacy shape — which would silently make `.thumbnails/` un-emptyable and +# strand step 10e forever, since its gate is the directory being gone. +# +# The HTTP 200 above is what proves the thumbnail actually works; this proves +# it got there the new way. +# Both streams silenced: the helper reports absence loudly, with a red banner +# and a `find` dump, because absence used to be the failure. Here it is the +# expected result, so leaving that visible would cry wolf on every clean run. +if assert_preview_existsy "$FIXTURE" "$STORAGE_PATH" >/dev/null 2>&1; then + fail "a thumbnail sidecar was written — the legacy write path is back (step 10d2 removed it)" +fi +log "Probe blob on disk; thumbnail served from the derived tier, no sidecar written." # ── 1c. Delete every non-admin user created by earlier Hurl tests ───────────── # From fced39c798ef5440cf9d50fe03808d489de07114 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 00:16:51 +0200 Subject: [PATCH 50/66] test(api): drain GC on two zero passes, and diagnose leftovers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three blobs survived the sweep. Five seconds of async-unlink polling did not remove them, so they were never queued — GC had not judged them collectible, and the loop had already exited. It broke on the FIRST zero-reap pass. A single zero only says nothing was collectible at that instant: releases cascade, since reaping a source drops the references its derived and attached rows held and `on_blob_deleted` does that from spawned tasks, so a pass can land in the gap between "source reaped" and "dependents released" and report zero with work outstanding. The import jobs added a level to that chain, which is when it started biting. Now two consecutive zeros, with the bound raised to match — one extra trigger over an empty store is cheaper than a false pass reporting a clean disk. The rest is diagnosis, because a list of paths cannot tell the three causes apart and they need opposite fixes: a positive refcount means a release was missed, an orphan means the reap predicate has a gap, and a row without a manifest means the registry is inconsistent. Each leftover now reports its manifest and blob refcounts plus how many files, derived rows and attached rows point at it — so if this is a real leak rather than the race, the next run names it instead of costing another full pass through the suite. --- tests/api/storage_cleanup_check.sh | 45 ++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index 7fd988d8..9865655d 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -328,7 +328,7 @@ log "Reconciliation sweep triggered." GC_TOTAL_BLOBS=0 GC_TOTAL_BYTES=0 GC_DRAINED=0 -for gc_pass in 1 2 3; do +for gc_pass in 1 2 3 4; do GC_RESULT=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/dedup_gc/trigger?force=true") [[ -z "$GC_RESULT" ]] && fail "trigger-gc returned an empty body (pass $gc_pass)" GC_BLOBS=$(echo "$GC_RESULT" | jq -r '.outcome.count // 0') @@ -336,9 +336,26 @@ for gc_pass in 1 2 3; do GC_TOTAL_BLOBS=$((GC_TOTAL_BLOBS + GC_BLOBS)) GC_TOTAL_BYTES=$((GC_TOTAL_BYTES + GC_BYTES)) log "GC pass $gc_pass reaped $GC_BLOBS blob(s), $GC_BYTES byte(s) freed." + # Break on TWO consecutive zero passes, not one. + # + # A single zero only says nothing was collectible *at that instant*. + # Releases cascade — reaping a source blob drops the references its + # derived and attached rows held, and `on_blob_deleted` does that from + # spawned tasks — so a pass can land in the gap between "source reaped" + # and "dependents released" and report zero while work remains. The + # import jobs added a level to that chain, which is when this started + # biting. + # + # Cheap insurance: one extra trigger over an empty store, versus a + # false pass that reports a clean disk while blobs remain. if [[ "$GC_BLOBS" -eq 0 ]]; then - GC_DRAINED=1 - break + if [[ "${GC_ZERO_STREAK:-0}" -ge 1 ]]; then + GC_DRAINED=1 + break + fi + GC_ZERO_STREAK=1 + else + GC_ZERO_STREAK=0 fi # Breathe before the next trigger, for two reasons: # @@ -408,6 +425,28 @@ fi if [[ -n "$BLOB_FILES" ]]; then BLOB_COUNT=$(echo "$BLOB_FILES" | wc -l | tr -d ' ') + + # Say WHY each one survived, not just that it did. A path alone cannot + # distinguish the three causes, and they need opposite fixes: a positive + # refcount means something still references it (a release was missed), an + # orphan means GC never considered it (a reap predicate gap), and a row + # with no manifest means the registry itself is inconsistent. Diagnosing + # that by hand costs a round-trip through the whole suite. + log "Diagnosing leftovers (refcounts and referrers):" + while read -r f; do + [[ -z "$f" ]] && continue + h=$(basename "$f" .blob) + docker compose -f "$COMPOSE_FILE" exec -T postgres-test \ + psql -U oxicloud_test -d oxicloud_test -tAqc " + SELECT ' $h' + || ' manifest_refs=' || COALESCE((SELECT ref_count::text FROM storage.chunk_manifests WHERE file_hash='$h'), '-') + || ' blob_refs=' || COALESCE((SELECT ref_count::text FROM storage.blobs WHERE hash='$h'), '-') + || ' files=' || (SELECT count(*) FROM storage.files WHERE blob_hash='$h') + || ' derived=' || (SELECT count(*) FROM storage.content_derived_blobs WHERE blob_hash='$h') + || ' attached=' || (SELECT count(*) FROM storage.file_attached_blobs WHERE blob_hash='$h');" \ + 2> >(grep -v 'Executing external compose provider' >&2) || true + done <<< "$BLOB_FILES" + log "Leftover blob files ($BLOB_COUNT):" echo "$BLOB_FILES" fail "$BLOB_COUNT blob file(s) remain on disk after full cleanup" From 2b9505f344dda49377eb780378dbb7f9a93ff6dc Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 07:43:41 +0200 Subject: [PATCH 51/66] fix(api): define COMPOSE_FILE so the leftover diagnosis actually runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 52c31a68 added a per-leftover refcount dump to storage_cleanup_check.sh but referenced COMPOSE_FILE, which that script never defines — only thumb_import_check.sh does. It would have run `docker compose -f ""`, failed, and been swallowed by the `|| true` guarding the loop. A silent no-op: the diagnosis would print nothing and the failure would look exactly as uninformative as the one it was written to explain. The same shape as the three bugs this suite has already caught — an error dressed up as an unremarkable result — and I wrote it into the tool meant to find them. The `|| true` stays, so one unreadable blob cannot abort the loop before the others report. --- tests/api/storage_cleanup_check.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index 9865655d..2bee36e2 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -18,6 +18,10 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" STORAGE_PATH="${OXICLOUD_STORAGE_PATH:-$REPO_ROOT/tests/api/storage}" +# Needed by the leftover diagnosis below. Without it `docker compose -f ""` +# fails and the `|| true` there swallows it, so the diagnosis silently prints +# nothing and the failure looks exactly as uninformative as before. +COMPOSE_FILE="$REPO_ROOT/tests/common/docker-compose.test.yml" # shellcheck source=test.env source "$SCRIPT_DIR/test.env" From 01191103452100b294d90e8924f35ba7ed079b7c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 08:12:25 +0200 Subject: [PATCH 52/66] test(api): diagnose all leftovers, and name the pinning source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The run gave the decisive fact: `derived=1`. A content_derived_blobs row still points at the leftover blob, so GC is CORRECT to keep it — the leak is the row, not the bytes. purge_derived_blobs only runs when the SOURCE is reaped, so the question is why that never happened. So the dump now prints the source hash and what still holds it: src_files, src_manifest, src_blob. If the source has a live file the answer is "not deleted"; if it has none but a positive refcount, a release was missed upstream; if it has no row at all, the source was reaped WITHOUT purging, which would be a real ordering bug in reap_blob. Also fixes the dump reporting only one of three blobs. `docker compose exec -T` reads stdin, so it consumed the rest of the here-string feeding the loop — the other two were never queried and vanished silently. The same silent-truncation shape the diagnosis exists to expose, in the diagnosis. `< /dev/null` closes it. --- tests/api/storage_cleanup_check.sh | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index 2bee36e2..66117a5d 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -447,8 +447,21 @@ if [[ -n "$BLOB_FILES" ]]; then || ' blob_refs=' || COALESCE((SELECT ref_count::text FROM storage.blobs WHERE hash='$h'), '-') || ' files=' || (SELECT count(*) FROM storage.files WHERE blob_hash='$h') || ' derived=' || (SELECT count(*) FROM storage.content_derived_blobs WHERE blob_hash='$h') - || ' attached=' || (SELECT count(*) FROM storage.file_attached_blobs WHERE blob_hash='$h');" \ + || ' attached=' || (SELECT count(*) FROM storage.file_attached_blobs WHERE blob_hash='$h') + -- When a derived row is what pins the blob, the question is + -- why its SOURCE was never reaped — purge_derived_blobs only + -- runs from the source's reap. Print the source and whether + -- anything still holds it. + || COALESCE((SELECT ' src=' || d.source_hash + || ' src_files=' || (SELECT count(*) FROM storage.files WHERE blob_hash = d.source_hash) + || ' src_manifest=' || COALESCE((SELECT ref_count::text FROM storage.chunk_manifests WHERE file_hash = d.source_hash), '-') + || ' src_blob=' || COALESCE((SELECT ref_count::text FROM storage.blobs WHERE hash = d.source_hash), '-') + FROM storage.content_derived_blobs d WHERE d.blob_hash='$h' LIMIT 1), '');" \ + < /dev/null \ 2> >(grep -v 'Executing external compose provider' >&2) || true + # `< /dev/null`: `docker compose exec -T` reads stdin, and without this it + # consumes the rest of the here-string — so only the FIRST leftover was + # ever diagnosed and the others vanished silently. done <<< "$BLOB_FILES" log "Leftover blob files ($BLOB_COUNT):" From ea5d3003e07b686327e625942be143beef7ff5a7 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 08:58:11 +0200 Subject: [PATCH 53/66] fix(dedup): bulk manifest reap orphaned every derived row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real leak, found by storage_cleanup_check.sh: three blobs surviving a full teardown, all `derived=1`, all naming one `src` whose manifest, blob row and files were already gone. The source had been reaped without its derived rows being purged. `reap_blob` purges correctly for the single-blob path. The BULK manifest reap did not — it iterated the deleted batch only to invalidate the manifest cache, so every manifest reaped that way left its content_derived_blobs rows behind. The predicate is not at fault. It protects a manifest that IS a derived artifact (content_derived_blobs.blob_hash) and deliberately not one that is the SOURCE of them, because counting source_hash as a reference would pin every original for as long as a thumbnail existed. The source is therefore reaped correctly and the purge simply has to follow it. The consequence is permanent, not cosmetic: the orphaned row holds chunk_manifests.ref_count at 1 on the thumbnail's own blob, so GC is thereafter CORRECT to refuse it — which is why three passes with force=true reclaimed nothing. Every deleted image left three behind, one per size, growing forever. Fixed at the reap rather than in any deletion path, which is where all of them converge: folder cascade, drive deletion, user deletion and single-file delete all reach it through the decrement trigger, so one call covers every route. --- src/infrastructure/services/dedup_service.rs | 22 ++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index 62684f30..6f189f15 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -2971,6 +2971,28 @@ impl DedupService { // and accounting remain below and run only after refcounts succeed. for (file_hash, _, _) in &batch { self.manifest_cache.invalidate(file_hash).await; + + // Drop everything derived FROM this Blob, exactly as + // `reap_blob` does for the single-blob path. + // + // Without this, bulk manifest reaping orphans the rows: the + // reap predicate protects a manifest that IS a derived + // artifact (`content_derived_blobs.blob_hash`), but + // deliberately not one that is the SOURCE of them — counting + // `source_hash` as a reference would pin every original for + // as long as a thumbnail existed. So the source is reaped + // correctly, and the purge has to follow it. + // + // It did not, and the leak is permanent rather than cosmetic: + // the orphaned row holds `chunk_manifests.ref_count` at 1 on + // the thumbnail's own blob, so GC is thereafter *correct* to + // refuse it and those bytes are never reclaimed. Every + // deleted image left three of them behind — one per size. + // + // Found by storage_cleanup_check.sh: three leftover blobs, + // all `derived=1`, all naming one `src` whose manifest, blob + // row and files were already gone. + self.purge_derived_blobs(file_hash).await; } if batch.len() == 1 { From f6bb677d91e3c00f99c21d804f96fd3ce46d71df Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 09:04:05 +0200 Subject: [PATCH 54/66] test(api): exercise the deletion path, which nothing did MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No test passed repair=true, so verify_and_unlink, the sidecar_delete_unverified finding and the directory removal had never executed. That left the one destructive part of the migration as its least-tested code: everything else is additive and recoverable, this unlinks files after a readback check, and a defect costs bytes. Four assertions, because "the files are gone" cannot by itself tell a correct drain from a destructive one: sidecars gone — the drain happened rows still present — it deleted the COPY, not the record. Removing the row would strand the blob in exactly the way the bulk-reap bug just did: a live reference with nothing behind it, which GC is then correct to refuse forever. unverified == 0 — every unlink passed its readback rather than being skipped, which is the property that makes deleting safe at all directory absent — the signal step 10e gates on, and why remove_dir is used: it refuses a non-empty directory, so success proves emptiness rather than asserting it Preconditions assert the sidecars exist first, or a no-op run would pass all four by doing nothing. The directory check logs rather than fails, since a concurrent render could legitimately repopulate it. --- tests/api/thumb_import_check.sh | 63 +++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/tests/api/thumb_import_check.sh b/tests/api/thumb_import_check.sh index 552f0895..fc399956 100755 --- a/tests/api/thumb_import_check.sh +++ b/tests/api/thumb_import_check.sh @@ -254,6 +254,69 @@ ATTACHED_2=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id [[ "$ATTACHED_2" == "$ATTACHED_AFTER" ]] || fail "re-run duplicated attached rows" log "re-run is a no-op: rows and refcounts unchanged." +# ── 5b. Deletion: the destructive half, and the only one that can lose data +# +# Everything above is additive and recoverable. This unlinks files after a +# readback check, so a defect here costs bytes — and until now it had never +# executed under test at all: no test passed `repair=true`, so +# verify_and_unlink, the sidecar_delete_unverified finding and the directory +# removal were entirely unexercised. +# +# Four assertions, because "the files are gone" alone cannot tell a correct +# drain from a destructive one: +# +# sidecars gone — the drain happened +# rows still present — it deleted the COPY, not the record. Removing the +# row would strand the blob exactly as the bulk-reap +# bug did. +# unverified == 0 — every unlink passed its readback rather than being +# skipped, which is what makes the deletion safe +# directory absent — the signal step 10e gates on, and the reason +# `remove_dir` is used: it refuses a non-empty +# directory, so success proves emptiness + +[[ -f "$SIDECAR_DIR/$BLOB_HASH.jpg" ]] || fail "precondition: derived sidecar already gone before the repair run" +[[ -f "$SIDECAR_DIR/ext-$FILE_ID.jpg" ]] || fail "precondition: attached sidecar already gone before the repair run" + +for job in thumb_derived_import thumb_attached_import; do + curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/$job/trigger?repair=true" >/dev/null \ + || fail "$job repair-trigger failed" + log "$job triggered with repair=true." +done + +[[ ! -f "$SIDECAR_DIR/$BLOB_HASH.jpg" ]] || fail "derived sidecar survived a repair run" +[[ ! -f "$SIDECAR_DIR/ext-$FILE_ID.jpg" ]] || fail "attached sidecar survived a repair run" + +DERIVED_KEPT=$(sql "SELECT count(*) FROM storage.content_derived_blobs WHERE source_hash='$BLOB_HASH';") +ATTACHED_KEPT=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id='$FILE_ID';") +[[ "$DERIVED_KEPT" -ge 1 ]] || fail "deletion removed the derived row; it must delete only the sidecar" +[[ "$ATTACHED_KEPT" -ge 1 ]] || fail "deletion removed the attached row; it must delete only the sidecar" + +for job in thumb_derived_import thumb_attached_import; do + run_id=$(curl -sf -H "$AUTH" "$base_url/api/admin/jobs/$job/runs?limit=1" \ + | jq -r 'if type == "array" then .[0].id else ((.runs // .items // [])[0].id) end // empty') + if [[ -n "$run_id" ]]; then + unverified=$(curl -sf -H "$AUTH" \ + "$base_url/api/admin/jobs/$job/runs/$run_id/findings?limit=50" \ + | jq -r '[ (if type == "array" then .[] else (.findings // .items // [])[] end) + | select((.kind // .finding_kind) == "sidecar_delete_unverified") ] | length') + [[ "${unverified:-0}" -eq 0 ]] \ + || fail "$job reported $unverified unverified unlink(s) — a blob did not read back" + fi +done + +# The directory removal is best-effort in the job (another test's render could +# repopulate it), so treat its absence as confirmation rather than a hard +# requirement. +if [[ -d "$STORAGE_PATH/.thumbnails" ]]; then + log "NOTE: .thumbnails/ still present — non-empty when the job ran (find below)" + find "$STORAGE_PATH/.thumbnails" -type f | head -5 +else + log ".thumbnails/ removed — the absence step 10e gates on." +fi + +log "deletion verified: sidecars drained, rows kept, every unlink read back." + # ── 6. Teardown ────────────────────────────────────────────────────────── # Everything created here must go — one database serves the whole suite, # and storage_cleanup_check.sh afterwards asserts the registry drains to From de0f625d4c48c49644c9b285da0033b797352261 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 12:43:05 +0200 Subject: [PATCH 55/66] fix(dedup): refuse a derived mapping whose source is already gone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Permanent blob leak, three rows per image. Confirmed green after this. The leftovers named their source, and it had no manifest, no blob row and no files. Nothing will ever reap that hash again, so purge_derived_blobs can never fire for it — meaning the rows were written AFTER the source died, not left behind by a reap that skipped them. Two earlier attempts assumed the latter and fixed the wrong thing. Background thumbnail generation is spawned and unawaited, so an upload deleted promptly — constant in a test suite, occasional for real users — has its render finish after GC reaped the blob and then record three mappings to a corpse. Each pins its own thumbnail blob at ref_count 1, which GC is thereafter CORRECT to refuse: that is why three passes with force=true reclaimed nothing and why the leak was invisible, a healthy system declining to delete referenced data. store_derived_blob now inserts only WHERE the source still exists, checking both tables since source_hash names a Blob — a manifest for CDC content, a bare blob row for legacy whole-file content. A refused insert falls into the existing `inserted == 0` branch and releases the reference, so the thumbnail blob becomes collectible rather than stranded. Closed in both directions: if the source dies before the statement's snapshot the row is refused; if after, that reap's purge finds the row. 034f1050 stays — the bulk manifest reap genuinely lacked the purge that reap_blob had, and two manifest reap paths with only one purging is its own defect. It just was not this one. Still missing, and now clearly worth building: the orphan-mapping check the plan's coverage matrix already lists (content_derived_blobs. source_hash with no Blob behind it). This stops new ones; nothing yet finds the ones already on disk. --- src/infrastructure/services/dedup_service.rs | 60 ++++++++++++++++++-- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs index 6f189f15..2d6b04ca 100644 --- a/src/infrastructure/services/dedup_service.rs +++ b/src/infrastructure/services/dedup_service.rs @@ -711,9 +711,31 @@ impl DedupService { let derived_hash = stored.hash().to_string(); let inserted = sqlx::query( + // The source must still EXIST, or this row can never be cleaned + // up. `purge_derived_blobs` runs from the source's reap, so a + // mapping written after that reap is unreachable forever: nothing + // will reap that hash a second time, and the orphaned row holds + // its derived blob's ref_count at 1, which GC is then correct to + // refuse. Permanent leak, three rows per image. + // + // It is not hypothetical. Background thumbnail generation is + // spawned and unawaited, so an upload deleted promptly — which a + // test suite does constantly, and users do occasionally — has its + // render finish AFTER the blob was reaped and then record a + // mapping to a corpse. + // + // Checking both tables because `source_hash` names a Blob: + // a manifest for CDC content, a bare blob row for legacy + // whole-file content. + // + // Zero rows here is indistinguishable from the ON CONFLICT case, + // and both want the same handling — release the reference the + // blob write just took — which the caller already does. "INSERT INTO storage.content_derived_blobs (source_hash, kind, variant, blob_hash, content_type) - VALUES ($1, $2, $3, $4, $5) + SELECT $1, $2, $3, $4, $5 + WHERE EXISTS (SELECT 1 FROM storage.chunk_manifests WHERE file_hash = $1) + OR EXISTS (SELECT 1 FROM storage.blobs WHERE hash = $1) ON CONFLICT (source_hash, kind, variant) DO NOTHING", ) .bind(source_hash) @@ -727,9 +749,14 @@ impl DedupService { .rows_affected(); if inserted == 0 { - // Someone else already mapped this variant. Our reference has no - // row behind it; leaving it would inflate ref_count on every - // re-render and pin the blob forever. + // Two causes, one correct response. + // + // Either someone else already mapped this variant (ON CONFLICT), + // or the source Blob no longer exists so the WHERE EXISTS above + // refused the row. Both leave our blob write with no mapping + // behind it, and in both cases keeping the reference would pin + // the blob forever — inflating ref_count on every re-render in + // the first case, stranding an unreachable blob in the second. if let Err(e) = self.remove_reference(&derived_hash).await { tracing::warn!( target: "oxicloud::dedup", @@ -838,6 +865,31 @@ impl DedupService { } }; + // Silent on success until now, which made three distinct outcomes + // indistinguishable from the outside: never called, called and found + // nothing, or found rows whose release then failed. Chasing an + // orphaned-derived-row leak cost several full suite runs for exactly + // that reason, so the call announces itself. + // + // `info` when it actually deleted something — that is rare (only when + // a source Blob dies) and it is the line that proves the reap path + // reached here. `debug` for the common no-op. + if derived.is_empty() { + tracing::debug!( + target: "oxicloud::dedup", + "purge_derived_blobs: no rows for {}", + &source_hash[..source_hash.len().min(12)], + ); + } else { + tracing::info!( + target: "oxicloud::dedup", + rows = derived.len(), + "purge_derived_blobs: releasing {} derived row(s) for {}", + derived.len(), + &source_hash[..source_hash.len().min(12)], + ); + } + for (blob_hash,) in derived { if let Err(e) = self.remove_reference(&blob_hash).await { tracing::warn!( From 4fef34b2303ca8b6aadfab101b2110e8a10a5a46 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 19:33:02 +0200 Subject: [PATCH 56/66] =?UTF-8?q?feat(consistency):=20derived=5Fconsistenc?= =?UTF-8?q?y=20=E2=80=94=20the=20last=20coverage-matrix=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finds derived mappings whose Blob is gone on either side. Nothing else can, and that is the point rather than an oversight: every other job reasons from a Blob outwards, so a row whose SOURCE was reaped breaks none of their invariants — valid reference, exactly correct refcount, bytes present on the backend. Every check agrees the system is healthy while the artifact is pinned forever. A leak that looks like correctness, which is why it took four suite runs to name. Two findings: derived_orphan_mapping (inconsistent) — source_hash has neither a manifest nor a blob row, so purge_derived_blobs can never fire for it. Storage that grows and never reclaims. derived_dangling_blob (data_loss) — blob_hash has no Blob behind it. The mapping promises an artifact that is gone, so a read finds the row and then fails. Existence means EITHER table on both sides, since source_hash and blob_hash each name a Blob: a manifest for CDC content, a bare blob row for legacy whole-file content. Checking one would report every legacy blob as missing. Paged on the full primary key with a row-value comparison rather than source_hash alone — a source has several variants, so a page boundary can fall inside one and advancing by source would skip the rest. Both existence probes fold into the page query, so a page is one round-trip rather than 2xN. Cursor round-trip is tested, including that a malformed one fails loudly: silently restarting would make a paged audit under-report, which is the worst failure available to a job whose purpose is finding what is missing. e4c78ae0 stops new orphans at the write side; this finds the ones already on disk, which that fix cannot reach. Added to the end-of-suite sweep so it runs against real state every time. --- src/common/di.rs | 13 + .../services/derived_consistency_service.rs | 328 ++++++++++++++++++ src/infrastructure/services/mod.rs | 1 + tests/api/storage_cleanup_check.sh | 6 + 4 files changed, 348 insertions(+) create mode 100644 src/infrastructure/services/derived_consistency_service.rs diff --git a/src/common/di.rs b/src/common/di.rs index 11fa4777..bae22651 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -1494,6 +1494,19 @@ impl AppServiceFactory { .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) .await; + // Finds derived mappings whose Blob is gone on either side. Nothing + // else can: a row whose SOURCE was reaped still holds a valid + // reference to a real artifact with a correct refcount, so every + // other check agrees the system is healthy while the artifact is + // pinned forever. Read-only. + let _ = Arc::new( + crate::infrastructure::services::derived_consistency_service::DerivedConsistencyCheck::new( + maintenance_pool.clone(), + ), + ) + .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) + .await; + // Its file-keyed twin: `ext-{file_id}.jpg` previews the user uploaded, // which no copy path duplicates today. Separate job, separate keying — // routing these into the content-keyed table would share one user's diff --git a/src/infrastructure/services/derived_consistency_service.rs b/src/infrastructure/services/derived_consistency_service.rs new file mode 100644 index 00000000..f22380f4 --- /dev/null +++ b/src/infrastructure/services/derived_consistency_service.rs @@ -0,0 +1,328 @@ +//! `derived_consistency` — the last unbuilt row of the coverage matrix. +//! +//! Walks `storage.content_derived_blobs` and reports mappings that point at +//! Blobs which no longer exist, in either direction. +//! +//! ### Why nothing else finds these +//! +//! Every other job reasons from a Blob outwards: `blobs_consistency` and +//! `manifests_consistency` recompute refcounts for rows that exist, +//! `backend_consistency` merge-joins the registry against the backend. A +//! derived row whose SOURCE is gone breaks none of those invariants — the +//! row holds a perfectly valid reference to a real artifact, the refcount is +//! exactly right, and the bytes are present on the backend. Every check +//! agrees the system is healthy. +//! +//! It is only wrong one level up: nothing will ever reap that source again, +//! so `purge_derived_blobs` can never fire, so the mapping is unreachable and +//! its artifact is pinned forever. A leak that looks like correctness. +//! +//! That is not hypothetical — it shipped. Background thumbnail generation is +//! spawned and unawaited, so an upload deleted promptly had its render +//! complete after GC reaped the blob and then record three mappings to a +//! corpse (fixed at the write side in `store_derived_blob`, which now +//! refuses a mapping whose source is gone). This job finds the ones already +//! on disk, which that fix cannot reach. +//! +//! ### Per-row checks +//! +//! * `derived_orphan_mapping` (severity `inconsistent`) — `source_hash` has +//! neither a manifest nor a blob row. Storage overhead that grows and never +//! reclaims. Recovery = delete the row, which releases the artifact. +//! * `derived_dangling_blob` (severity `data_loss`) — `blob_hash` has no Blob +//! behind it. The opposite and the more serious one: the mapping promises +//! an artifact that is gone, so a read finds a row and then fails. +//! +//! Read-only, per the house default. Both findings name a row rather than a +//! range, so recovery can act on them individually. + +use std::sync::Arc; + +use async_trait::async_trait; +use sqlx::PgPool; + +use crate::infrastructure::scheduler::{ + JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, + RunStatus, record_or_log, +}; + +pub const DERIVED_CONSISTENCY_JOB_NAME: &str = "derived_consistency"; + +/// Rows per page. Each is two indexed existence probes folded into the page +/// query, so this can be larger than a job doing per-row I/O. +const BATCH_SIZE: i64 = 500; + +pub struct DerivedConsistencyCheck { + pool: Arc, +} + +/// One row plus the two existence answers, resolved server-side so a page +/// costs one round-trip rather than `2 × rows`. +#[derive(Debug, sqlx::FromRow)] +struct DerivedRow { + source_hash: String, + kind: String, + variant: String, + blob_hash: String, + source_exists: bool, + artifact_exists: bool, +} + +impl DerivedConsistencyCheck { + pub fn new(pool: Arc) -> Self { + Self { pool } + } + + pub async fn register_recoverable_job( + self: Arc, + registry: &JobRegistry, + provider: &Arc, + ) -> Arc { + registry + .register_recoverable_job(self.clone(), provider.clone(), None) + .await; + self + } + + /// Page query, keyed on the full primary key. + /// + /// Row-value comparison (`(a,b,c) > ($1,$2,$3)`) rather than + /// `source_hash > $1`: a source has several variants, so a page boundary + /// can fall inside one, and advancing by source alone would skip its + /// remaining rows. The tuple form is also index-friendly — it matches the + /// primary key's own ordering. + /// + /// "Exists" means EITHER table, because `source_hash` and `blob_hash` both + /// name a Blob: a manifest for CDC content, a bare `storage.blobs` row for + /// legacy whole-file content. Checking only one would report every legacy + /// blob as missing. + const PAGE_SQL: &'static str = r#" + SELECT d.source_hash, + d.kind, + d.variant, + d.blob_hash, + (EXISTS (SELECT 1 FROM storage.chunk_manifests m WHERE m.file_hash = d.source_hash) + OR EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.source_hash)) + AS source_exists, + (EXISTS (SELECT 1 FROM storage.chunk_manifests m WHERE m.file_hash = d.blob_hash) + OR EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.blob_hash)) + AS artifact_exists + FROM storage.content_derived_blobs d + WHERE ($1::text IS NULL + OR (d.source_hash, d.kind, d.variant) > ($1::text, $2::text, $3::text)) + ORDER BY d.source_hash, d.kind, d.variant + LIMIT $4"#; +} + +/// Cursor is the primary-key triple, newline-joined. +/// +/// Safe as a delimiter: `source_hash` is hex, `kind` comes from a CHECK +/// constraint, and `variant` is a size/format token — none can contain a +/// newline. +fn encode_cursor(r: &DerivedRow) -> Vec { + format!("{}\n{}\n{}", r.source_hash, r.kind, r.variant).into_bytes() +} + +fn decode_cursor(bytes: Vec) -> Result, String> { + if bytes.is_empty() { + return Ok(None); + } + let s = String::from_utf8(bytes).map_err(|e| format!("not valid UTF-8: {e}"))?; + let mut parts = s.splitn(3, '\n'); + match (parts.next(), parts.next(), parts.next()) { + (Some(a), Some(b), Some(c)) => Ok(Some((a.into(), b.into(), c.into()))), + _ => Err(format!( + "expected three newline-separated fields, got {s:?}" + )), + } +} + +#[async_trait] +impl RecoverableJobHandler for DerivedConsistencyCheck { + fn name(&self) -> &str { + DERIVED_CONSISTENCY_JOB_NAME + } + + async fn count_total(&self) -> Option { + sqlx::query_as::<_, (i64,)>("SELECT COUNT(*) FROM storage.content_derived_blobs") + .fetch_one(self.pool.as_ref()) + .await + .ok() + .map(|(n,)| n.max(0) as u64) + } + + async fn run_resumable( + &self, + store: &dyn JobStore, + _args: &JobRunArgs, + resume_cursor: Option>, + ) -> RunOutcome { + let mut cursor = match resume_cursor.map(decode_cursor).transpose() { + Ok(c) => c.flatten(), + Err(message) => return RunOutcome::Failed { message }, + }; + + let mut finding_count = 0u64; + + loop { + match store.status().await { + Ok(RunStatus::CancelRequested) => { + return RunOutcome::Paused { + cursor: cursor + .as_ref() + .map(|(a, b, c)| format!("{a}\n{b}\n{c}").into_bytes()) + .unwrap_or_default(), + }; + } + Ok(_) => {} + Err(e) => { + return RunOutcome::Failed { + message: format!("status poll: {e}"), + }; + } + } + + let (ch, ck, cv) = match &cursor { + Some((a, b, c)) => (Some(a.as_str()), Some(b.as_str()), Some(c.as_str())), + None => (None, None, None), + }; + + let rows: Vec = match sqlx::query_as(Self::PAGE_SQL) + .bind(ch) + .bind(ck) + .bind(cv) + .bind(BATCH_SIZE) + .fetch_all(self.pool.as_ref()) + .await + { + Ok(r) => r, + Err(e) => { + return RunOutcome::Failed { + message: format!("derived page: {e}"), + }; + } + }; + + if rows.is_empty() { + break; + } + + for row in &rows { + if !row.source_exists { + finding_count += 1; + record_or_log( + store, + DERIVED_CONSISTENCY_JOB_NAME, + "derived_orphan_mapping", + "inconsistent", + None, + serde_json::json!({ + "source_hash": row.source_hash, + "kind": row.kind, + "variant": row.variant, + "blob_hash": row.blob_hash, + "note": "source Blob is gone, so purge_derived_blobs can never fire; \ + this row pins its artifact forever", + }), + ) + .await; + } + + if !row.artifact_exists { + finding_count += 1; + record_or_log( + store, + DERIVED_CONSISTENCY_JOB_NAME, + "derived_dangling_blob", + "data_loss", + None, + serde_json::json!({ + "source_hash": row.source_hash, + "kind": row.kind, + "variant": row.variant, + "blob_hash": row.blob_hash, + "note": "mapping promises an artifact with no Blob behind it; \ + a read finds the row and then fails", + }), + ) + .await; + } + } + + let scanned = rows.len() as u64; + cursor = rows + .last() + .map(|r| (r.source_hash.clone(), r.kind.clone(), r.variant.clone())); + + let checkpoint = rows.last().map(encode_cursor).unwrap_or_default(); + if let Err(e) = store.checkpoint(checkpoint, scanned).await { + return RunOutcome::Failed { + message: format!("checkpoint: {e}"), + }; + } + + if scanned < BATCH_SIZE as u64 { + break; + } + } + + tracing::info!( + target: "oxicloud::consistency", + event = "derived_consistency.completed", + run_id = %store.run_id(), + finding_count = finding_count, + "derived_consistency completed with {} finding(s)", + finding_count + ); + + RunOutcome::completed() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn row(source: &str, kind: &str, variant: &str) -> DerivedRow { + DerivedRow { + source_hash: source.into(), + kind: kind.into(), + variant: variant.into(), + blob_hash: "b".into(), + source_exists: true, + artifact_exists: true, + } + } + + /// The cursor must survive the round trip, or a resumed run silently + /// restarts or skips — the failure mode a paged audit job can least + /// afford, since it would under-report rather than error. + #[test] + fn cursor_round_trips() { + let r = row("0a1b", "thumbnail", "preview.webp"); + let decoded = decode_cursor(encode_cursor(&r)).unwrap(); + assert_eq!( + decoded, + Some(( + "0a1b".to_string(), + "thumbnail".to_string(), + "preview.webp".to_string() + )) + ); + } + + /// An empty cursor means "from the beginning", not a parse error — the + /// scheduler hands one back for a fresh run. + #[test] + fn empty_cursor_starts_from_the_beginning() { + assert_eq!(decode_cursor(Vec::new()).unwrap(), None); + } + + /// A malformed cursor must fail loudly. Silently treating it as "start + /// over" would turn a corrupt checkpoint into a job that never finishes + /// and never says why. + #[test] + fn malformed_cursor_is_an_error() { + assert!(decode_cursor(b"only-one-field".to_vec()).is_err()); + } +} diff --git a/src/infrastructure/services/mod.rs b/src/infrastructure/services/mod.rs index 9ffaa836..1e6d9f83 100644 --- a/src/infrastructure/services/mod.rs +++ b/src/infrastructure/services/mod.rs @@ -10,6 +10,7 @@ pub mod compression_service; pub mod consistency_batch_service; pub mod db_pool_monitor; pub mod dedup_service; +pub mod derived_consistency_service; pub mod dpop_nonce_service; pub mod dpop_replay_cache; pub mod dpop_verifier; diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index 66117a5d..f9b30a8e 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -532,6 +532,12 @@ CONSISTENCY_JOBS=( blobs_consistency manifests_consistency backend_consistency + # Catches what the others structurally cannot: a derived mapping whose + # source Blob is gone looks healthy to every refcount-based check — valid + # reference, correct count, bytes present — while pinning its artifact + # forever. That leak reached this suite as three unreclaimable blobs and + # took four runs to identify. + derived_consistency ) CONSISTENCY_FAILED=0 From b3221e265d9d58fce0869539a29760dfcc6558b4 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 20:05:22 +0200 Subject: [PATCH 57/66] feat(consistency): satellites_consistency covers both tables, and the sweep covers every job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends the derived check to `file_attached_blobs` and renames it, since the two tables are one concept — the content-keyed and file-keyed halves of "things attached to a Blob" — and `storage.copy_file_satellites` already established the vocabulary. The attached half is the one that cannot be recovered. `attached_dangling_blob` is data_loss with `recoverable: false`: those bytes were user-supplied and have no server-side render path, so nothing can regenerate them. Its derived twin carries `recoverable: true`, because a derived artifact is a pure function of its source and re-rendering restores it. Same finding shape, materially different stakes, and the detail says which. No orphan-mapping check on the attached side, deliberately: `file_id` is ON DELETE CASCADE, so a row cannot outlive its file. The database enforces what the derived table cannot, since a content hash has no row to point a foreign key at — which is exactly why only that half could rot. One job walking two tables needs a phase in the cursor, or an attached checkpoint would be replayed against the derived table and silently re-scan or skip. Two things the sweep was missing, found while checking whether every consistency job is actually exercised: drives_consistency and folders_consistency were registered but never run by any test. Now included; the list is exhaustive by intent. An unknown job was a warning-and-skip. That protected feature-gated builds at the cost of something worse: this list said `derived_consistency` for one commit after the rename and would have dropped that coverage without a word, leaving the suite green over a check that no longer ran. It fails now. --- src/common/di.rs | 6 +- .../services/derived_consistency_service.rs | 328 ------------ src/infrastructure/services/mod.rs | 2 +- .../satellites_consistency_service.rs | 506 ++++++++++++++++++ tests/api/admin_jobs.hurl | 32 +- tests/api/storage_cleanup_check.sh | 21 +- 6 files changed, 550 insertions(+), 345 deletions(-) delete mode 100644 src/infrastructure/services/derived_consistency_service.rs create mode 100644 src/infrastructure/services/satellites_consistency_service.rs diff --git a/src/common/di.rs b/src/common/di.rs index bae22651..92b1b572 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -1494,13 +1494,13 @@ impl AppServiceFactory { .register_recoverable_job(&core.job_registry, &job_store_provider_dyn) .await; - // Finds derived mappings whose Blob is gone on either side. Nothing - // else can: a row whose SOURCE was reaped still holds a valid + // Both satellite tables, checked for mappings whose Blob is gone. + // Nothing else can: a row whose SOURCE was reaped still holds a valid // reference to a real artifact with a correct refcount, so every // other check agrees the system is healthy while the artifact is // pinned forever. Read-only. let _ = Arc::new( - crate::infrastructure::services::derived_consistency_service::DerivedConsistencyCheck::new( + crate::infrastructure::services::satellites_consistency_service::SatellitesConsistencyCheck::new( maintenance_pool.clone(), ), ) diff --git a/src/infrastructure/services/derived_consistency_service.rs b/src/infrastructure/services/derived_consistency_service.rs deleted file mode 100644 index f22380f4..00000000 --- a/src/infrastructure/services/derived_consistency_service.rs +++ /dev/null @@ -1,328 +0,0 @@ -//! `derived_consistency` — the last unbuilt row of the coverage matrix. -//! -//! Walks `storage.content_derived_blobs` and reports mappings that point at -//! Blobs which no longer exist, in either direction. -//! -//! ### Why nothing else finds these -//! -//! Every other job reasons from a Blob outwards: `blobs_consistency` and -//! `manifests_consistency` recompute refcounts for rows that exist, -//! `backend_consistency` merge-joins the registry against the backend. A -//! derived row whose SOURCE is gone breaks none of those invariants — the -//! row holds a perfectly valid reference to a real artifact, the refcount is -//! exactly right, and the bytes are present on the backend. Every check -//! agrees the system is healthy. -//! -//! It is only wrong one level up: nothing will ever reap that source again, -//! so `purge_derived_blobs` can never fire, so the mapping is unreachable and -//! its artifact is pinned forever. A leak that looks like correctness. -//! -//! That is not hypothetical — it shipped. Background thumbnail generation is -//! spawned and unawaited, so an upload deleted promptly had its render -//! complete after GC reaped the blob and then record three mappings to a -//! corpse (fixed at the write side in `store_derived_blob`, which now -//! refuses a mapping whose source is gone). This job finds the ones already -//! on disk, which that fix cannot reach. -//! -//! ### Per-row checks -//! -//! * `derived_orphan_mapping` (severity `inconsistent`) — `source_hash` has -//! neither a manifest nor a blob row. Storage overhead that grows and never -//! reclaims. Recovery = delete the row, which releases the artifact. -//! * `derived_dangling_blob` (severity `data_loss`) — `blob_hash` has no Blob -//! behind it. The opposite and the more serious one: the mapping promises -//! an artifact that is gone, so a read finds a row and then fails. -//! -//! Read-only, per the house default. Both findings name a row rather than a -//! range, so recovery can act on them individually. - -use std::sync::Arc; - -use async_trait::async_trait; -use sqlx::PgPool; - -use crate::infrastructure::scheduler::{ - JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, - RunStatus, record_or_log, -}; - -pub const DERIVED_CONSISTENCY_JOB_NAME: &str = "derived_consistency"; - -/// Rows per page. Each is two indexed existence probes folded into the page -/// query, so this can be larger than a job doing per-row I/O. -const BATCH_SIZE: i64 = 500; - -pub struct DerivedConsistencyCheck { - pool: Arc, -} - -/// One row plus the two existence answers, resolved server-side so a page -/// costs one round-trip rather than `2 × rows`. -#[derive(Debug, sqlx::FromRow)] -struct DerivedRow { - source_hash: String, - kind: String, - variant: String, - blob_hash: String, - source_exists: bool, - artifact_exists: bool, -} - -impl DerivedConsistencyCheck { - pub fn new(pool: Arc) -> Self { - Self { pool } - } - - pub async fn register_recoverable_job( - self: Arc, - registry: &JobRegistry, - provider: &Arc, - ) -> Arc { - registry - .register_recoverable_job(self.clone(), provider.clone(), None) - .await; - self - } - - /// Page query, keyed on the full primary key. - /// - /// Row-value comparison (`(a,b,c) > ($1,$2,$3)`) rather than - /// `source_hash > $1`: a source has several variants, so a page boundary - /// can fall inside one, and advancing by source alone would skip its - /// remaining rows. The tuple form is also index-friendly — it matches the - /// primary key's own ordering. - /// - /// "Exists" means EITHER table, because `source_hash` and `blob_hash` both - /// name a Blob: a manifest for CDC content, a bare `storage.blobs` row for - /// legacy whole-file content. Checking only one would report every legacy - /// blob as missing. - const PAGE_SQL: &'static str = r#" - SELECT d.source_hash, - d.kind, - d.variant, - d.blob_hash, - (EXISTS (SELECT 1 FROM storage.chunk_manifests m WHERE m.file_hash = d.source_hash) - OR EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.source_hash)) - AS source_exists, - (EXISTS (SELECT 1 FROM storage.chunk_manifests m WHERE m.file_hash = d.blob_hash) - OR EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.blob_hash)) - AS artifact_exists - FROM storage.content_derived_blobs d - WHERE ($1::text IS NULL - OR (d.source_hash, d.kind, d.variant) > ($1::text, $2::text, $3::text)) - ORDER BY d.source_hash, d.kind, d.variant - LIMIT $4"#; -} - -/// Cursor is the primary-key triple, newline-joined. -/// -/// Safe as a delimiter: `source_hash` is hex, `kind` comes from a CHECK -/// constraint, and `variant` is a size/format token — none can contain a -/// newline. -fn encode_cursor(r: &DerivedRow) -> Vec { - format!("{}\n{}\n{}", r.source_hash, r.kind, r.variant).into_bytes() -} - -fn decode_cursor(bytes: Vec) -> Result, String> { - if bytes.is_empty() { - return Ok(None); - } - let s = String::from_utf8(bytes).map_err(|e| format!("not valid UTF-8: {e}"))?; - let mut parts = s.splitn(3, '\n'); - match (parts.next(), parts.next(), parts.next()) { - (Some(a), Some(b), Some(c)) => Ok(Some((a.into(), b.into(), c.into()))), - _ => Err(format!( - "expected three newline-separated fields, got {s:?}" - )), - } -} - -#[async_trait] -impl RecoverableJobHandler for DerivedConsistencyCheck { - fn name(&self) -> &str { - DERIVED_CONSISTENCY_JOB_NAME - } - - async fn count_total(&self) -> Option { - sqlx::query_as::<_, (i64,)>("SELECT COUNT(*) FROM storage.content_derived_blobs") - .fetch_one(self.pool.as_ref()) - .await - .ok() - .map(|(n,)| n.max(0) as u64) - } - - async fn run_resumable( - &self, - store: &dyn JobStore, - _args: &JobRunArgs, - resume_cursor: Option>, - ) -> RunOutcome { - let mut cursor = match resume_cursor.map(decode_cursor).transpose() { - Ok(c) => c.flatten(), - Err(message) => return RunOutcome::Failed { message }, - }; - - let mut finding_count = 0u64; - - loop { - match store.status().await { - Ok(RunStatus::CancelRequested) => { - return RunOutcome::Paused { - cursor: cursor - .as_ref() - .map(|(a, b, c)| format!("{a}\n{b}\n{c}").into_bytes()) - .unwrap_or_default(), - }; - } - Ok(_) => {} - Err(e) => { - return RunOutcome::Failed { - message: format!("status poll: {e}"), - }; - } - } - - let (ch, ck, cv) = match &cursor { - Some((a, b, c)) => (Some(a.as_str()), Some(b.as_str()), Some(c.as_str())), - None => (None, None, None), - }; - - let rows: Vec = match sqlx::query_as(Self::PAGE_SQL) - .bind(ch) - .bind(ck) - .bind(cv) - .bind(BATCH_SIZE) - .fetch_all(self.pool.as_ref()) - .await - { - Ok(r) => r, - Err(e) => { - return RunOutcome::Failed { - message: format!("derived page: {e}"), - }; - } - }; - - if rows.is_empty() { - break; - } - - for row in &rows { - if !row.source_exists { - finding_count += 1; - record_or_log( - store, - DERIVED_CONSISTENCY_JOB_NAME, - "derived_orphan_mapping", - "inconsistent", - None, - serde_json::json!({ - "source_hash": row.source_hash, - "kind": row.kind, - "variant": row.variant, - "blob_hash": row.blob_hash, - "note": "source Blob is gone, so purge_derived_blobs can never fire; \ - this row pins its artifact forever", - }), - ) - .await; - } - - if !row.artifact_exists { - finding_count += 1; - record_or_log( - store, - DERIVED_CONSISTENCY_JOB_NAME, - "derived_dangling_blob", - "data_loss", - None, - serde_json::json!({ - "source_hash": row.source_hash, - "kind": row.kind, - "variant": row.variant, - "blob_hash": row.blob_hash, - "note": "mapping promises an artifact with no Blob behind it; \ - a read finds the row and then fails", - }), - ) - .await; - } - } - - let scanned = rows.len() as u64; - cursor = rows - .last() - .map(|r| (r.source_hash.clone(), r.kind.clone(), r.variant.clone())); - - let checkpoint = rows.last().map(encode_cursor).unwrap_or_default(); - if let Err(e) = store.checkpoint(checkpoint, scanned).await { - return RunOutcome::Failed { - message: format!("checkpoint: {e}"), - }; - } - - if scanned < BATCH_SIZE as u64 { - break; - } - } - - tracing::info!( - target: "oxicloud::consistency", - event = "derived_consistency.completed", - run_id = %store.run_id(), - finding_count = finding_count, - "derived_consistency completed with {} finding(s)", - finding_count - ); - - RunOutcome::completed() - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn row(source: &str, kind: &str, variant: &str) -> DerivedRow { - DerivedRow { - source_hash: source.into(), - kind: kind.into(), - variant: variant.into(), - blob_hash: "b".into(), - source_exists: true, - artifact_exists: true, - } - } - - /// The cursor must survive the round trip, or a resumed run silently - /// restarts or skips — the failure mode a paged audit job can least - /// afford, since it would under-report rather than error. - #[test] - fn cursor_round_trips() { - let r = row("0a1b", "thumbnail", "preview.webp"); - let decoded = decode_cursor(encode_cursor(&r)).unwrap(); - assert_eq!( - decoded, - Some(( - "0a1b".to_string(), - "thumbnail".to_string(), - "preview.webp".to_string() - )) - ); - } - - /// An empty cursor means "from the beginning", not a parse error — the - /// scheduler hands one back for a fresh run. - #[test] - fn empty_cursor_starts_from_the_beginning() { - assert_eq!(decode_cursor(Vec::new()).unwrap(), None); - } - - /// A malformed cursor must fail loudly. Silently treating it as "start - /// over" would turn a corrupt checkpoint into a job that never finishes - /// and never says why. - #[test] - fn malformed_cursor_is_an_error() { - assert!(decode_cursor(b"only-one-field".to_vec()).is_err()); - } -} diff --git a/src/infrastructure/services/mod.rs b/src/infrastructure/services/mod.rs index 1e6d9f83..5ad26293 100644 --- a/src/infrastructure/services/mod.rs +++ b/src/infrastructure/services/mod.rs @@ -10,7 +10,6 @@ pub mod compression_service; pub mod consistency_batch_service; pub mod db_pool_monitor; pub mod dedup_service; -pub mod derived_consistency_service; pub mod dpop_nonce_service; pub mod dpop_replay_cache; pub mod dpop_verifier; @@ -52,6 +51,7 @@ pub mod plugins; pub mod recent_recording_hook; pub mod retry_blob_backend; pub mod s3_blob_backend; +pub mod satellites_consistency_service; pub mod search_index; pub mod session_cleanup_service; pub mod session_liveness_gauges; diff --git a/src/infrastructure/services/satellites_consistency_service.rs b/src/infrastructure/services/satellites_consistency_service.rs new file mode 100644 index 00000000..dbdea998 --- /dev/null +++ b/src/infrastructure/services/satellites_consistency_service.rs @@ -0,0 +1,506 @@ +//! `satellites_consistency` — the last unbuilt row of the coverage matrix. +//! +//! Walks both satellite tables and reports mappings pointing at Blobs that no +//! longer exist. One job rather than two, because the tables are one concept +//! — the content-keyed and file-keyed halves of "things attached to a Blob" — +//! and the vocabulary already exists in `storage.copy_file_satellites`. +//! +//! ### Why nothing else finds these +//! +//! Every other job reasons from a Blob outwards: `blobs_consistency` and +//! `manifests_consistency` recompute refcounts for rows that exist, +//! `backend_consistency` merge-joins the registry against the backend. A +//! satellite row whose SOURCE is gone breaks none of those invariants — the +//! row holds a valid reference to a real artifact, the refcount is exactly +//! right, and the bytes are present on the backend. Every check agrees the +//! system is healthy. +//! +//! It is only wrong one level up: nothing will ever reap that source again, +//! so `purge_derived_blobs` can never fire, so the mapping is unreachable and +//! its artifact is pinned forever. A leak that looks like correctness, which +//! is why it survived four full suite runs before being named. +//! +//! That is not hypothetical — it shipped. Background thumbnail generation is +//! spawned and unawaited, so an upload deleted promptly had its render +//! complete after GC reaped the blob and then record three mappings to a +//! corpse. Fixed at the write side in `store_derived_blob`, which now refuses +//! a mapping whose source is gone; this job finds the ones already on disk, +//! which that fix cannot reach. +//! +//! ### Per-row checks +//! +//! * `derived_orphan_mapping` (`inconsistent`) — a `content_derived_blobs` +//! row whose `source_hash` has neither a manifest nor a blob row. Storage +//! that grows and never reclaims. +//! * `derived_dangling_blob` (`data_loss`) — its `blob_hash` has no Blob. +//! The mapping promises an artifact that is gone, so a read finds the row +//! and then fails. Recoverable in practice: a derived artifact is a pure +//! function of its source, so re-rendering restores it. +//! * `attached_dangling_blob` (`data_loss`) — the same for +//! `file_attached_blobs`, and **the one that cannot be recovered**. These +//! bytes are user-supplied — a client-generated PDF preview has no +//! server-side render path — so there is nothing to regenerate from. Same +//! finding shape as the derived case, materially higher stakes. +//! +//! There is deliberately no orphan-mapping check for the attached table: +//! `file_id` is `REFERENCES storage.files(id) ON DELETE CASCADE`, so a row +//! cannot outlive its file. The database enforces what the derived table +//! cannot, since a content hash has no row to point a foreign key at — which +//! is precisely why only that half could rot. +//! +//! Read-only, per the house default. Findings name a row rather than a range, +//! so recovery can act on them individually. + +use std::sync::Arc; + +use async_trait::async_trait; +use sqlx::PgPool; +use uuid::Uuid; + +use crate::infrastructure::scheduler::{ + JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome, + RunStatus, record_or_log, +}; + +pub const SATELLITES_CONSISTENCY_JOB_NAME: &str = "satellites_consistency"; + +/// Rows per page. Existence probes fold into the page query, so a page costs +/// one round-trip rather than `2 × rows`. +const BATCH_SIZE: i64 = 500; + +/// "Does this hash name a Blob?" — either table, because a Blob is a manifest +/// for CDC content and a bare `storage.blobs` row for legacy whole-file +/// content. Checking one would report every legacy blob as missing. +macro_rules! blob_exists { + ($col:literal) => { + concat!( + "(EXISTS (SELECT 1 FROM storage.chunk_manifests m WHERE m.file_hash = ", + $col, + ") OR EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = ", + $col, + "))" + ) + }; +} + +pub struct SatellitesConsistencyCheck { + pool: Arc, +} + +#[derive(Debug, sqlx::FromRow)] +struct DerivedRow { + source_hash: String, + kind: String, + variant: String, + blob_hash: String, + source_exists: bool, + artifact_exists: bool, +} + +#[derive(Debug, sqlx::FromRow)] +struct AttachedRow { + file_id: Uuid, + kind: String, + variant: String, + blob_hash: String, + uploaded_by: Uuid, + artifact_exists: bool, +} + +impl SatellitesConsistencyCheck { + pub fn new(pool: Arc) -> Self { + Self { pool } + } + + pub async fn register_recoverable_job( + self: Arc, + registry: &JobRegistry, + provider: &Arc, + ) -> Arc { + registry + .register_recoverable_job(self.clone(), provider.clone(), None) + .await; + self + } + + /// Both page queries key on the full primary key with a row-value + /// comparison, not on the first column: a source (or file) has several + /// variants, so a page boundary can fall inside one and advancing by the + /// first column alone would skip the rest. The tuple form also matches + /// the primary key's own ordering, so it stays index-friendly. + const DERIVED_PAGE_SQL: &'static str = concat!( + "SELECT d.source_hash, d.kind, d.variant, d.blob_hash, ", + blob_exists!("d.source_hash"), + " AS source_exists, ", + blob_exists!("d.blob_hash"), + " AS artifact_exists + FROM storage.content_derived_blobs d + WHERE ($1::text IS NULL + OR (d.source_hash, d.kind, d.variant) > ($1::text, $2::text, $3::text)) + ORDER BY d.source_hash, d.kind, d.variant + LIMIT $4" + ); + + const ATTACHED_PAGE_SQL: &'static str = concat!( + "SELECT a.file_id, a.kind, a.variant, a.blob_hash, a.uploaded_by, ", + blob_exists!("a.blob_hash"), + " AS artifact_exists + FROM storage.file_attached_blobs a + WHERE ($1::uuid IS NULL + OR (a.file_id, a.kind, a.variant) > ($1::uuid, $2::text, $3::text)) + ORDER BY a.file_id, a.kind, a.variant + LIMIT $4" + ); +} + +/// Cursor is `{phase}\n{a}\n{b}\n{c}`. +/// +/// The phase is what lets one job walk two tables and still resume exactly: +/// without it, a cursor from the attached pass would be replayed against the +/// derived table and silently re-scan or skip. Newline is a safe delimiter — +/// hashes are hex, uuids are uuids, `kind` comes from a CHECK constraint, and +/// `variant` is a size/format token. +#[derive(Debug, PartialEq, Clone, Copy)] +enum Phase { + Derived, + Attached, +} + +impl Phase { + fn as_str(self) -> &'static str { + match self { + Phase::Derived => "derived", + Phase::Attached => "attached", + } + } +} + +fn encode_cursor(phase: Phase, a: &str, b: &str, c: &str) -> Vec { + format!("{}\n{a}\n{b}\n{c}", phase.as_str()).into_bytes() +} + +type Cursor = Option<(Phase, String, String, String)>; + +fn decode_cursor(bytes: Vec) -> Result { + if bytes.is_empty() { + return Ok(None); + } + let s = String::from_utf8(bytes).map_err(|e| format!("not valid UTF-8: {e}"))?; + let mut parts = s.splitn(4, '\n'); + match (parts.next(), parts.next(), parts.next(), parts.next()) { + (Some("derived"), Some(a), Some(b), Some(c)) => { + Ok(Some((Phase::Derived, a.into(), b.into(), c.into()))) + } + (Some("attached"), Some(a), Some(b), Some(c)) => { + Ok(Some((Phase::Attached, a.into(), b.into(), c.into()))) + } + _ => Err(format!("malformed cursor: {s:?}")), + } +} + +#[async_trait] +impl RecoverableJobHandler for SatellitesConsistencyCheck { + fn name(&self) -> &str { + SATELLITES_CONSISTENCY_JOB_NAME + } + + async fn count_total(&self) -> Option { + sqlx::query_as::<_, (i64,)>( + "SELECT (SELECT COUNT(*) FROM storage.content_derived_blobs) + + (SELECT COUNT(*) FROM storage.file_attached_blobs)", + ) + .fetch_one(self.pool.as_ref()) + .await + .ok() + .map(|(n,)| n.max(0) as u64) + } + + async fn run_resumable( + &self, + store: &dyn JobStore, + _args: &JobRunArgs, + resume_cursor: Option>, + ) -> RunOutcome { + let start = match resume_cursor.map(decode_cursor).transpose() { + Ok(c) => c.flatten(), + Err(message) => return RunOutcome::Failed { message }, + }; + + let mut finding_count = 0u64; + + // ── Phase 1: content-keyed ─────────────────────────────────────── + // Skipped entirely when resuming mid-attached, since that phase runs + // strictly after this one. + let mut derived_cursor = match &start { + Some((Phase::Attached, ..)) => None, + Some((Phase::Derived, a, b, c)) => Some((a.clone(), b.clone(), c.clone())), + None => None, + }; + let skip_derived = matches!(&start, Some((Phase::Attached, ..))); + + if !skip_derived { + loop { + if let Some(outcome) = poll_cancel( + store, + derived_cursor + .as_ref() + .map(|(a, b, c)| encode_cursor(Phase::Derived, a, b, c)), + ) + .await + { + return outcome; + } + + let (ch, ck, cv) = match &derived_cursor { + Some((a, b, c)) => (Some(a.as_str()), Some(b.as_str()), Some(c.as_str())), + None => (None, None, None), + }; + + let rows: Vec = match sqlx::query_as(Self::DERIVED_PAGE_SQL) + .bind(ch) + .bind(ck) + .bind(cv) + .bind(BATCH_SIZE) + .fetch_all(self.pool.as_ref()) + .await + { + Ok(r) => r, + Err(e) => { + return RunOutcome::Failed { + message: format!("derived page: {e}"), + }; + } + }; + if rows.is_empty() { + break; + } + + for row in &rows { + if !row.source_exists { + finding_count += 1; + record_or_log( + store, + SATELLITES_CONSISTENCY_JOB_NAME, + "derived_orphan_mapping", + "inconsistent", + None, + serde_json::json!({ + "source_hash": row.source_hash, + "kind": row.kind, + "variant": row.variant, + "blob_hash": row.blob_hash, + "note": "source Blob is gone, so purge_derived_blobs can never \ + fire; this row pins its artifact forever", + }), + ) + .await; + } + if !row.artifact_exists { + finding_count += 1; + record_or_log( + store, + SATELLITES_CONSISTENCY_JOB_NAME, + "derived_dangling_blob", + "data_loss", + None, + serde_json::json!({ + "source_hash": row.source_hash, + "kind": row.kind, + "variant": row.variant, + "blob_hash": row.blob_hash, + "recoverable": true, + "note": "artifact missing; derived content is a pure function of \ + its source, so re-rendering restores it", + }), + ) + .await; + } + } + + let scanned = rows.len() as u64; + let last = rows.last().unwrap(); + derived_cursor = Some(( + last.source_hash.clone(), + last.kind.clone(), + last.variant.clone(), + )); + if let Err(e) = store + .checkpoint( + encode_cursor(Phase::Derived, &last.source_hash, &last.kind, &last.variant), + scanned, + ) + .await + { + return RunOutcome::Failed { + message: format!("checkpoint: {e}"), + }; + } + if scanned < BATCH_SIZE as u64 { + break; + } + } + } + + // ── Phase 2: file-keyed ────────────────────────────────────────── + // No orphan-mapping check here: `file_id` is ON DELETE CASCADE, so a + // row cannot outlive its file. Only the artifact side can rot. + let mut attached_cursor: Option<(Uuid, String, String)> = match &start { + Some((Phase::Attached, a, b, c)) => match Uuid::parse_str(a) { + Ok(id) => Some((id, b.clone(), c.clone())), + Err(e) => { + return RunOutcome::Failed { + message: format!("attached cursor is not a uuid: {e}"), + }; + } + }, + _ => None, + }; + + loop { + if let Some(outcome) = poll_cancel( + store, + attached_cursor + .as_ref() + .map(|(a, b, c)| encode_cursor(Phase::Attached, &a.to_string(), b, c)), + ) + .await + { + return outcome; + } + + let (ch, ck, cv) = match &attached_cursor { + Some((a, b, c)) => (Some(*a), Some(b.as_str()), Some(c.as_str())), + None => (None, None, None), + }; + + let rows: Vec = match sqlx::query_as(Self::ATTACHED_PAGE_SQL) + .bind(ch) + .bind(ck) + .bind(cv) + .bind(BATCH_SIZE) + .fetch_all(self.pool.as_ref()) + .await + { + Ok(r) => r, + Err(e) => { + return RunOutcome::Failed { + message: format!("attached page: {e}"), + }; + } + }; + if rows.is_empty() { + break; + } + + for row in &rows { + if !row.artifact_exists { + finding_count += 1; + record_or_log( + store, + SATELLITES_CONSISTENCY_JOB_NAME, + "attached_dangling_blob", + "data_loss", + None, + serde_json::json!({ + "file_id": row.file_id, + "kind": row.kind, + "variant": row.variant, + "blob_hash": row.blob_hash, + "uploaded_by": row.uploaded_by, + "recoverable": false, + "note": "UNRECOVERABLE: these bytes were user-supplied and have no \ + server-side render path, so nothing can regenerate them", + }), + ) + .await; + } + } + + let scanned = rows.len() as u64; + let last = rows.last().unwrap(); + attached_cursor = Some((last.file_id, last.kind.clone(), last.variant.clone())); + if let Err(e) = store + .checkpoint( + encode_cursor( + Phase::Attached, + &last.file_id.to_string(), + &last.kind, + &last.variant, + ), + scanned, + ) + .await + { + return RunOutcome::Failed { + message: format!("checkpoint: {e}"), + }; + } + if scanned < BATCH_SIZE as u64 { + break; + } + } + + tracing::info!( + target: "oxicloud::consistency", + event = "satellites_consistency.completed", + run_id = %store.run_id(), + finding_count = finding_count, + "satellites_consistency completed with {} finding(s)", + finding_count + ); + + RunOutcome::completed() + } +} + +/// Cooperative cancel, shared by both phases so neither can forget it. +async fn poll_cancel(store: &dyn JobStore, cursor: Option>) -> Option { + match store.status().await { + Ok(RunStatus::CancelRequested) => Some(RunOutcome::Paused { + cursor: cursor.unwrap_or_default(), + }), + Ok(_) => None, + Err(e) => Some(RunOutcome::Failed { + message: format!("status poll: {e}"), + }), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The phase is what lets one job walk two tables and resume exactly. + /// Without it an attached cursor would be replayed against the derived + /// table, silently re-scanning or skipping — an audit job under-reporting + /// is the worst failure available to it. + #[test] + fn cursor_round_trips_and_keeps_its_phase() { + for phase in [Phase::Derived, Phase::Attached] { + let encoded = encode_cursor(phase, "0a1b", "thumbnail", "preview.webp"); + assert_eq!( + decode_cursor(encoded).unwrap(), + Some(( + phase, + "0a1b".to_string(), + "thumbnail".to_string(), + "preview.webp".to_string() + )) + ); + } + } + + #[test] + fn empty_cursor_starts_from_the_beginning() { + assert_eq!(decode_cursor(Vec::new()).unwrap(), None); + } + + /// Loudly, rather than silently restarting: a corrupt checkpoint that + /// reads as "start over" gives a job that never finishes and never says + /// why. + #[test] + fn malformed_cursor_is_an_error() { + assert!(decode_cursor(b"only-one-field".to_vec()).is_err()); + assert!(decode_cursor(b"bogus\na\nb\nc".to_vec()).is_err()); + } +} diff --git a/tests/api/admin_jobs.hurl b/tests/api/admin_jobs.hurl index 92464c9a..82e4121a 100644 --- a/tests/api/admin_jobs.hurl +++ b/tests/api/admin_jobs.hurl @@ -225,13 +225,22 @@ jsonpath "$.outcome.count" exists # Step 4c — Trigger `consistency_batch`. Coordinator (plain # JobHandler) — snapshots the registry, filters names # ending `_consistency`, sequentially triggers each. -# `outcome.count` = number of children dispatched (6 as -# of the refcount_cascade fix: drives + folders + -# files + blobs + manifests + backend). `extra.per_check` -# carries a per-child outcome -# map. Batch itself always returns ok — child failures -# live inside per_check. `?deep=true` propagates as -# `extra.deep`. +# `extra.per_check` carries a per-child outcome map. Batch +# itself always returns ok — child failures live inside +# per_check. `?deep=true` propagates as `extra.deep`. +# +# NO assertion on `outcome.count`. The batch auto-discovers +# tenants via `.ends_with("_consistency")`, so a hardcoded +# total breaks every time one is added — it broke on +# `manifests_consistency` and again on +# `satellites_consistency`, each time asserting arithmetic +# rather than behaviour. Per the house rule: `contains` per +# item, never a total. +# +# What matters is that every child SUCCEEDED, which +# `err == 0` states directly and without a magic number, plus +# a named check per tenant below so a job silently dropping +# out of the batch is still caught. # ───────────────────────────────────────────────────────────── POST {{base_url}}/api/admin/jobs/consistency_batch/trigger?deep=true Authorization: Bearer {{admin_token}} @@ -240,9 +249,9 @@ HTTP 200 [Asserts] jsonpath "$.ok" == true jsonpath "$.outcome.outcome" == "ok" -jsonpath "$.outcome.count" == 6 jsonpath "$.outcome.extra.deep" == true -jsonpath "$.outcome.extra.ok" == 6 +# Zero failures, whatever the tenant count happens to be. `ok` is not +# asserted against a number for the same reason `count` is not. jsonpath "$.outcome.extra.err" == 0 # per_check is keyed by child job name. `manifests_consistency` was added # by the refcount_cascade fix — see docs/plan/derived-blobs.md and @@ -255,6 +264,11 @@ jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok" jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok" jsonpath "$.outcome.extra.per_check.manifests_consistency.outcome" == "ok" jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok" +# Finds satellite mappings whose Blob is gone — the one class every +# refcount-based check above reports as healthy, because the row holds a +# valid reference with an exactly correct count while pinning an artifact +# that can never be reclaimed. +jsonpath "$.outcome.extra.per_check.satellites_consistency.outcome" == "ok" # ───────────────────────────────────────────────────────────── diff --git a/tests/api/storage_cleanup_check.sh b/tests/api/storage_cleanup_check.sh index f9b30a8e..74f2fef4 100755 --- a/tests/api/storage_cleanup_check.sh +++ b/tests/api/storage_cleanup_check.sh @@ -527,24 +527,37 @@ fi # Zero findings is the assertion. These jobs are read-only, so a finding # here is a real invariant violation, not a repair opportunity. +# EVERY registered consistency tenant. Keep this list exhaustive: two of +# these (drives, folders) were missing until 2026-08-28 and had never run +# under test at all. CONSISTENCY_JOBS=( files_consistency + folders_consistency + drives_consistency blobs_consistency manifests_consistency backend_consistency - # Catches what the others structurally cannot: a derived mapping whose + # Catches what the others structurally cannot: a satellite mapping whose # source Blob is gone looks healthy to every refcount-based check — valid # reference, correct count, bytes present — while pinning its artifact # forever. That leak reached this suite as three unreclaimable blobs and # took four runs to identify. - derived_consistency + satellites_consistency ) CONSISTENCY_FAILED=0 for job in "${CONSISTENCY_JOBS[@]}"; do + # FAIL on an unknown job rather than warn-and-skip. + # + # The warning was there so a feature-gated build would not break, but the + # cost is worse than the case it protects: renaming a job (or a typo) + # silently removes it from the sweep, and the suite goes on reporting + # green over a check that no longer runs. This list said + # `derived_consistency` for exactly one commit after the rename and would + # have skipped it without comment. TRIGGER=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/$job/trigger") \ - || { log "WARNING: $job not registered in this build — skipped"; continue; } - [[ -z "$TRIGGER" ]] && { log "WARNING: $job returned an empty body — skipped"; continue; } + || fail "$job could not be triggered — renamed, unregistered, or a typo in CONSISTENCY_JOBS" + [[ -z "$TRIGGER" ]] && fail "$job returned an empty body" # The trigger is synchronous for these tenants, but the run row is what # carries the findings, so read it back rather than trusting the From 1a3d7d201a8f47c7a33bf288d35812e505c0f80c Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 22:52:21 +0200 Subject: [PATCH 58/66] fix(storage): skip sidecars whose source is gone, before writing anything MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the import on a real install produced a store-then-discard loop: NEW BLOB (CDC) immediately followed by MANIFEST DELETED, once per sidecar. store_derived_blob wrote the bytes, the source-exists guard refused the row, and `inserted == 0` released the reference again. The refusal is right — `.thumbnails/` outlives years of deleted files, and importing those would recreate exactly the orphan rows e4c78ae0 eliminated. The mistake was deciding it AFTER the write. Now checked before the read and the store, via blob_exists (manifest first, blob as fallback). Two costs it removes: a blob write plus a manifest delete per dead sidecar on EVERY run, and a tail that never empties — unimportable files are rediscovered forever, so the job never reports zero and step 10e's gate never opens. Reported as `sidecar_source_gone` so the scale is visible before anything is removed, and deleted under `repair`. That is the one unlink in this job needing no readback: there is nothing to read back and nothing to regenerate from. Counted separately in the completion log, because "skipped, source gone" and "already present" mean different things to an operator deciding whether the migration has converged. Worth noting for anyone reading the raw logs: NEW BLOB names the hash of the STORED BYTES, while the sidecar filename is the SOURCE hash. They are different values, so grepping the log hash against .thumbnails finds nothing. The new finding carries both. --- .../services/thumb_derived_import_service.rs | 49 ++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index dcf0aa55..f9417e18 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -236,6 +236,7 @@ impl RecoverableJobHandler for ThumbDerivedImport { let mut failed = 0u64; let mut deleted = 0u64; let mut unverified = 0u64; + let mut dead_source = 0u64; let mut since_checkpoint = 0usize; // The DIRECTORY is `{size}` on disk; the VARIANT is `{size}.{ext}` // since migration `20261022000000`. Conflating them is a real trap: @@ -317,6 +318,50 @@ impl RecoverableJobHandler for ThumbDerivedImport { .await; } } + } else if !self.dedup.blob_exists(hash).await { + // The source is gone, so this sidecar cannot be imported: + // a mapping to a dead source is precisely the orphan row + // `store_derived_blob` now refuses, because nothing would + // ever reap that hash again and the row would pin its + // artifact forever. + // + // Checked BEFORE the read and the blob write, not after. + // Without this the refusal still happens, but only once + // the bytes have been stored — so every run writes a blob + // and immediately deletes its manifest again, per dead + // sidecar, forever. On a real install where `.thumbnails/` + // has outlived years of deleted files, that is most of + // them. + // + // It also matters for the tail: these files are + // unimportable by definition, so a run that keeps + // rediscovering them never reports zero and step 10e's + // gate never opens. Under `repair` they are deleted — + // safe, and the only unlink here that needs no readback, + // since there is nothing to read back and nothing to + // regenerate from. + dead_source += 1; + if delete_imported { + let path = self.thumbnails_root.join(dir_name).join(&name); + if fs::remove_file(&path).await.is_ok() { + deleted += 1; + } + } else { + record_or_log( + store, + THUMB_DERIVED_IMPORT_JOB_NAME, + "sidecar_source_gone", + "anomaly", + None, + serde_json::json!({ + "path": position, + "source_hash": hash, + "note": "source Blob no longer exists; the thumbnail is \ + unimportable and is deleted on a repair run", + }), + ) + .await; + } } else { let path = self.thumbnails_root.join(dir_name).join(&name); match fs::read(&path).await { @@ -441,8 +486,10 @@ impl RecoverableJobHandler for ThumbDerivedImport { failed = failed, deleted = deleted, unverified = unverified, + dead_source = dead_source, "thumb_derived_import: {imported} imported, {already} already present, \ - {failed} failed, {deleted} sidecar(s) deleted, {unverified} kept unverified" + {failed} failed, {deleted} sidecar(s) deleted, {unverified} kept unverified, \ + {dead_source} skipped (source gone)" ); RunOutcome::completed() From b485db46fa82fc1e8c7f3e4b924b83103f967cc6 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Fri, 28 Aug 2026 23:03:43 +0200 Subject: [PATCH 59/66] feat(storage): audit every sidecar deletion, and reclaim orphaned uploads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes to the import jobs' destructive path. thumb_attached_import now deletes orphaned sidecars under `repair`, matching the dead-source case on the derived side. An `ext-` file whose owner is gone is unimportable — the FK on file_id would reject the row — so leaving it means it is rediscovered every run, the tail never empties and step 10e's gate never opens. Safe despite these being the non-regenerable bytes: the preview is keyed to a file_id that no longer exists, so nothing can reference it again. Unrecoverable and unreachable are different things, and this is both. And every deletion is now audited. A one-way migration removing user-visible files should leave a trail that outlives the run history: findings are per-run and get purged, whereas target: "audit" is separable and retained. If a preview later turns out to be missing, this is the only record saying the migration removed it and when. `owner` carries the id the file belonged to — source_hash for content-keyed, file_id for uploaded — because that is where an investigation starts, and the raw logs cannot supply it: NEW BLOB names the hash of the STORED BYTES, a different value from the sidecar's own name, which is why grepping one against the other finds nothing. reason is a stable key: `imported` (replaced by a verified blob), `source_gone`, `orphaned`. The first lives inside verify_and_unlink so a verified deletion cannot be logged inconsistently; the other two are explicit, since those paths have nothing to verify against. --- .../services/thumb_attached_import_service.rs | 73 ++++++++++++++----- .../services/thumb_derived_import_service.rs | 65 ++++++++++++++++- 2 files changed, 119 insertions(+), 19 deletions(-) diff --git a/src/infrastructure/services/thumb_attached_import_service.rs b/src/infrastructure/services/thumb_attached_import_service.rs index 2f13ec43..2176bfa9 100644 --- a/src/infrastructure/services/thumb_attached_import_service.rs +++ b/src/infrastructure/services/thumb_attached_import_service.rs @@ -262,6 +262,8 @@ impl RecoverableJobHandler for ThumbAttachedImport { let path = self.thumbnails_root.join(&dir_name).join(&name); if ThumbDerivedImport::verify_and_unlink( &self.dedup, + THUMB_ATTACHED_IMPORT_JOB_NAME, + &file_id_str, &existing.blob_hash, &path, ) @@ -286,24 +288,59 @@ impl RecoverableJobHandler for ThumbAttachedImport { } } } else if !self.file_exists(file_id).await { - // The file is gone; the sidecar outlived it. Reported - // rather than deleted — this job imports, it does not - // reclaim, and a destructive default on a migration is - // exactly what `no silent auto-repair` forbids. + // The file is gone, so this sidecar is unimportable: the + // FK on `file_id` would reject the row. Mirrors the + // dead-source case in thumb_derived_import. + // + // Reported by default — a destructive default on a + // migration is what no-silent-auto-repair forbids — and + // deleted under `repair`, because otherwise it is + // rediscovered on every run, the tail never empties, and + // step 10e's gate never opens. + // + // Safe to delete despite these being the non-regenerable + // bytes: the preview is keyed to a `file_id` that no + // longer exists, so nothing can ever reference it again. + // Unrecoverable and unreachable are different things, and + // this is both. + // + // No readback before unlinking, unlike the imported path: + // there is no row and no blob to read back, and nothing to + // regenerate from either. orphaned += 1; - record_or_log( - store, - THUMB_ATTACHED_IMPORT_JOB_NAME, - "attached_sidecar_orphan", - "anomaly", - None, - serde_json::json!({ - "path": position, - "file_id": file_id_str, - "note": "no storage.files row; sidecar left in place for the operator", - }), - ) - .await; + if delete_imported { + let path = self.thumbnails_root.join(&dir_name).join(&name); + if fs::remove_file(&path).await.is_ok() { + deleted += 1; + // Explicit: nothing to verify against, so this + // bypasses verify_and_unlink. Worth auditing + // loudest of all — these bytes were + // user-supplied and cannot be regenerated, even + // though the file that owned them is gone. + crate::infrastructure::services::thumb_derived_import_service::audit_sidecar_deleted( + THUMB_ATTACHED_IMPORT_JOB_NAME, + "orphaned", + &file_id_str, + "-", + &path, + ); + } + } else { + record_or_log( + store, + THUMB_ATTACHED_IMPORT_JOB_NAME, + "attached_sidecar_orphan", + "anomaly", + None, + serde_json::json!({ + "path": position, + "file_id": file_id_str, + "note": "no storage.files row; unimportable, and deleted on a \ + repair run since nothing can reference it again", + }), + ) + .await; + } } else { let path = self.thumbnails_root.join(&dir_name).join(&name); match fs::read(&path).await { @@ -328,6 +365,8 @@ impl RecoverableJobHandler for ThumbAttachedImport { if delete_imported { if ThumbDerivedImport::verify_and_unlink( &self.dedup, + THUMB_ATTACHED_IMPORT_JOB_NAME, + &file_id_str, &attached_hash, &path, ) diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index f9417e18..77010f8c 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -48,6 +48,41 @@ use crate::infrastructure::services::dedup_service::DedupService; pub const THUMB_DERIVED_IMPORT_JOB_NAME: &str = "thumb_derived_import"; +/// Record a sidecar deletion on the audit channel. +/// +/// Both import jobs delete user-visible files during a one-way migration, so +/// the trail has to survive the run history: findings are per-run and get +/// purged, whereas `target: "audit"` is separable and retained. If a preview +/// later turns out to be missing, this is the only record that says the +/// migration removed it, when, and on whose behalf. +/// +/// `owner` is the id the file belonged to — a `source_hash` for content-keyed +/// sidecars, a `file_id` for uploaded ones. That is the field an +/// investigation starts from, and the raw `NEW BLOB` logs cannot supply it: +/// they name the hash of the stored bytes, which is a different value from +/// the sidecar's own name. +/// +/// `reason` is a stable machine-readable key, per the convention: `imported` +/// (replaced by a verified blob), `source_gone`, `orphaned`. +pub(crate) fn audit_sidecar_deleted( + job: &str, + reason: &str, + owner: &str, + blob_hash: &str, + path: &std::path::Path, +) { + tracing::info!( + target: "audit", + event = "thumbnail.sidecar_deleted", + reason = reason, + job = job, + owner = owner, + blob_hash = blob_hash, + path = %path.display(), + "👮🏻‍♂️ migration deleted a thumbnail sidecar ({reason})", + ); +} + /// Files handled between checkpoints. Each one is a read plus (at most) a /// blob write, so this is deliberately smaller than a pure-DB sweep's page. const BATCH_SIZE: usize = 100; @@ -142,6 +177,8 @@ impl ThumbDerivedImport { /// and two copies of that rule would be two chances to weaken one. pub(crate) async fn verify_and_unlink( dedup: &DedupService, + job: &str, + owner: &str, stored_hash: &str, path: &std::path::Path, ) -> bool { @@ -154,7 +191,11 @@ impl ThumbDerivedImport { if stored.is_empty() || stored.len() as u64 != meta.len() { return false; } - fs::remove_file(path).await.is_ok() + if fs::remove_file(path).await.is_err() { + return false; + } + audit_sidecar_deleted(job, "imported", owner, stored_hash, path); + true } /// Sorted sidecar filenames for one size directory. @@ -299,7 +340,15 @@ impl RecoverableJobHandler for ThumbDerivedImport { already += 1; if delete_imported { let path = self.thumbnails_root.join(dir_name).join(&name); - if Self::verify_and_unlink(&self.dedup, &existing.blob_hash, &path).await { + if Self::verify_and_unlink( + &self.dedup, + THUMB_DERIVED_IMPORT_JOB_NAME, + hash, + &existing.blob_hash, + &path, + ) + .await + { deleted += 1; } else { unverified += 1; @@ -345,6 +394,16 @@ impl RecoverableJobHandler for ThumbDerivedImport { let path = self.thumbnails_root.join(dir_name).join(&name); if fs::remove_file(&path).await.is_ok() { deleted += 1; + // Audited explicitly: this unlink bypasses + // verify_and_unlink, which has nothing to verify + // against here. + audit_sidecar_deleted( + THUMB_DERIVED_IMPORT_JOB_NAME, + "source_gone", + hash, + "-", + &path, + ); } } else { record_or_log( @@ -382,6 +441,8 @@ impl RecoverableJobHandler for ThumbDerivedImport { if delete_imported { if Self::verify_and_unlink( &self.dedup, + THUMB_DERIVED_IMPORT_JOB_NAME, + hash, &derived_hash, &path, ) From 1ea382666032b8daa02e365f1eec10d9cbffa6cf Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sat, 29 Aug 2026 00:04:12 +0200 Subject: [PATCH 60/66] =?UTF-8?q?feat(jobs):=20jobs=20describe=20themselve?= =?UTF-8?q?s=20=E2=80=94=20description,=20mutates,=20repair=5Fdescription?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The admin panel had no repair toggle wired to anything but a hardcoded name list naming the two refcount tenants, so `thumb_derived_import` and `thumb_attached_import` could not be run in repair mode from the UI at all despite supporting it. And nothing in the job list said what any given job does or whether clicking Run on production writes anything. Three defaulted methods on `JobHandler` and `RecoverableJobHandler`: fn description(&self) -> &'static str fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly fn repair_description(&self) -> Option<&'static str> `RecoverableAdapter` forwards them — the registry only holds `dyn JobHandler`, so a tenant's metadata is invisible otherwise, and falling back to the defaults would report every recoverable job as read-only, including the ones that delete files. Three values rather than a boolean because a job can be read-only by default and destructive under `?repair=true`; a boolean answers wrongly for one of its two modes, and `false` on something that unlinks files is the dangerous direction to be wrong in. `repair_description` returning `Option` collapses "does it repair" and "what does repair do" into one method: presence gates the toggle, content is the confirmation text — which the frontend cannot invent, since correcting a counter and deleting sidecars are not the same warning. `OnRepairOnly` with no `repair_description` is rejected at registration: it claims to mutate only under a flag it does not support. All 17 registered jobs declare all three. The panel now renders the description under each name, badges read-only jobs, confirms before a plain run of a mutating one, and offers the repair variant off the backend flag instead of the name list. Descriptions are English in the trait, next to the behaviour: one in `locales/*.json` rots invisibly the moment a job changes, and a translator cannot know what `manifests_consistency` reconciles. i18n can layer on later keyed by job name with these as the fallback. Co-Authored-By: Claude Opus 5 (1M context) --- docs/plan/job-registry.md | 52 ++++ frontend/src/lib/api/types.ts | 24 ++ .../src/lib/components/AdminJobsPanel.svelte | 242 ++++++++++++------ frontend/static/locales/en.json | 8 +- .../services/storage_usage_service.rs | 15 +- src/infrastructure/scheduler/handler.rs | 38 ++- src/infrastructure/scheduler/mod.rs | 2 +- src/infrastructure/scheduler/recoverable.rs | 109 +++++++- src/infrastructure/scheduler/registry.rs | 100 +++++++- src/infrastructure/scheduler/types.rs | 42 +++ .../services/backend_consistency_service.rs | 8 + .../services/backend_migration_service.rs | 18 +- .../services/backend_rotate_service.rs | 18 +- .../services/blobs_consistency_service.rs | 27 +- .../services/consistency_batch_service.rs | 24 +- src/infrastructure/services/dedup_service.rs | 14 + .../services/drives_consistency_service.rs | 7 + .../services/files_consistency_service.rs | 7 + .../services/folders_consistency_service.rs | 8 + .../services/grant_cleanup_service.rs | 14 +- .../services/manifests_consistency_service.rs | 24 +- .../satellites_consistency_service.rs | 9 + .../services/session_cleanup_service.rs | 13 +- .../services/thumb_attached_import_service.rs | 82 ++++-- .../services/thumb_derived_import_service.rs | 88 +++++-- .../services/trash_cleanup_service.rs | 13 +- tests/api/admin_jobs.hurl | 34 +++ 27 files changed, 906 insertions(+), 134 deletions(-) diff --git a/docs/plan/job-registry.md b/docs/plan/job-registry.md index ebc30e6e..98c40617 100644 --- a/docs/plan/job-registry.md +++ b/docs/plan/job-registry.md @@ -171,6 +171,58 @@ Native services implement this trait on an existing service type (no new wrapper) and register a single `Arc` with the scheduler. +### Self-description — `description` / `mutates` / `repair_description` + +Three defaulted methods on both `JobHandler` and `RecoverableJobHandler` +let a job tell the admin UI what it is. `RecoverableAdapter` forwards +them, since the registry only ever holds `dyn JobHandler`. + +```rust +fn description(&self) -> &'static str { "" } +fn mutates(&self) -> Mutates { Mutates::Never } +fn repair_description(&self) -> Option<&'static str> { None } + +pub enum Mutates { Never, Always, OnRepairOnly } +``` + +They surface on `JobSummary` (`GET /api/admin/jobs`) and drive the +panel: `Never` earns a read-only badge and triggers straight through, +`Always` confirms first, `OnRepairOnly` is safe to run and confirms only +when the repair variant is picked. `repair_description.is_some()` is +what renders the repair toggle at all, and its text is the confirmation +copy. + +**Why three values and not a boolean.** A job can be read-only by +default and destructive under `?repair=true`; a boolean has to answer +wrongly for one of those two modes, and `false` on something that +deletes files is the dangerous direction to be wrong in. It is also +where the recovery framework is heading — discovery-only default, +mutation behind an opt-in — so a tenant that later grows a repair arm +changes this one value and nothing else. + +**Why `Option<&str>` and not `supports_repair: bool` + prose.** +Presence gates the toggle, content supplies the wording. Split across +two methods they can disagree; and the frontend cannot invent the +wording itself, because correcting a counter and unlinking files off +disk are not the same warning. The two are independent, not derived +from each other: the thumbnail imports are `Always` *and* +repair-capable. + +`OnRepairOnly` with no `repair_description` is rejected at registration +— it claims to mutate only under a flag it does not support, and would +render as safe with no reachable mutating path. + +**Why English in the trait, not `locales/*.json`.** A description that +lives away from the behaviour rots the moment a job changes, invisibly, +and a translator cannot know what `manifests_consistency` reconciles. +i18n can layer on later keyed by job name with these as the fallback, +matching the frontend's `t(key, params, fallback)` — a missing +translation then degrades to English from code rather than to a blank +panel. No rework needed to get there. + +Defaults exist so the methods could be added without touching every +job at once; every registered job declares all three today. + ### `JobOutcome` ```rust diff --git a/frontend/src/lib/api/types.ts b/frontend/src/lib/api/types.ts index 9c967495..e3757566 100644 --- a/frontend/src/lib/api/types.ts +++ b/frontend/src/lib/api/types.ts @@ -618,8 +618,32 @@ export interface PausedRunBrief { total?: number; } +/** + * When a job changes state — `RecoverableJobHandler::mutates()` on the + * backend. Three values rather than a boolean because the interesting + * case is conditional: a job can be read-only by default and destructive + * under `?repair=true`. + * + * - `never` — read-only under every flag. Render a read-only badge; no + * confirmation needed to trigger. + * - `always` — changes state on a plain run. Confirm before triggering. + * - `on_repair_only` — safe to trigger; confirm only when the repair + * toggle is on. + */ +export type Mutates = 'never' | 'always' | 'on_repair_only'; + export interface JobSummary { name: string; + /** One or two sentences on what the job does, in English, authored + * next to the handler. Absent for jobs that haven't declared one — + * omit the line rather than rendering an empty block. */ + description?: string; + mutates: Mutates; + /** Present iff `?repair=true` does something beyond a default run; + * describes what it ADDS. Presence is what gates the repair toggle; + * the text is the confirmation copy. Independent of `mutates` — the + * thumbnail import jobs are `always` AND repair-capable. */ + repair_description?: string; interval_ms?: number; next_run_at?: string; last_run_at?: string; diff --git a/frontend/src/lib/components/AdminJobsPanel.svelte b/frontend/src/lib/components/AdminJobsPanel.svelte index d247ba58..e7e0a778 100644 --- a/frontend/src/lib/components/AdminJobsPanel.svelte +++ b/frontend/src/lib/components/AdminJobsPanel.svelte @@ -167,7 +167,7 @@ .slice() // `consistency_batch` is served by the top-bar // action buttons; hiding it here removes the - // duplicate table row. `hasBatch` still checks the + // duplicate table row. `batchJob` still reads from the // full fetched list so the top buttons only render // when the coordinator is actually registered. .filter((j) => j.name !== 'consistency_batch') @@ -180,7 +180,7 @@ // Track whether the coordinator is registered so the // top-bar buttons can gate on it without checking `jobs` // (which now filters it out). - hasBatch = fetched.some((j) => j.name === 'consistency_batch'); + batchJob = fetched.find((j) => j.name === 'consistency_batch') ?? null; loadError = null; } catch (e) { loadError = errorMessage(e); @@ -250,13 +250,15 @@ // ─── Expansion toggles ───────────────────────────────────────────── - function toggleJob(name: string) { - if (expandedJob === name) { + function toggleJob(job: JobSummary) { + if (expandedJob === job.name) { expandedJob = null; } else { - expandedJob = name; - // Lazy-load on first open, refresh on subsequent opens. - void loadRuns(name); + expandedJob = job.name; + // Lazy-load on first open, refresh on subsequent opens. Only + // recoverable jobs have runs to load — the others expand purely + // to show their description. + if (isRecoverable(job)) void loadRuns(job.name); } } @@ -458,8 +460,9 @@ * Per-severity finding counts from `last_outcome.extra.severity_counts` * (a JSON object populated by `run_or_resume`). Missing / older * runs return an empty record — callers should tolerate absent keys. - * The three severity values are the ones consistency tenants emit - * today: `data_loss`, `inconsistent`, `anomaly`. + * Severity values emitted today: `data_loss`, `inconsistent`, + * `anomaly`. The set is open (the column is TEXT), so unknown keys + * must degrade rather than throw. */ function lastSeverityCounts(job: JobSummary): Record { if (!job.last_outcome || job.last_outcome.outcome !== 'ok') return {}; @@ -480,6 +483,13 @@ return (s.data_loss ?? 0) + (s.inconsistent ?? 0); } + /** + * Informational findings. `anomaly` is the wire value; "notice" is + * what the panel calls it — there is no separate `notice` severity. + * A job that acted on what it found (a repair run deleting an + * orphaned sidecar) records the same severity and says so in the + * finding's `detail`. + */ function anomalyFindingCount(job: JobSummary): number { return lastSeverityCounts(job).anomaly ?? 0; } @@ -636,29 +646,65 @@ return name === 'consistency_batch' || name === 'blobs_consistency'; } - // Jobs whose handler consults `args.repair` and applies a - // corrective UPDATE against the finding it just emitted. Only the - // two ref_count tenants today; `consistency_batch` also accepts - // the flag (fans out to both) and is surfaced separately as the - // top-bar "Repair ref_counts" button. Keep this list narrow — - // adding a job here without a matching backend handler produces a - // silently no-op button that confuses operators. - function supportsRepair(name: string): boolean { - return name === 'blobs_consistency' || name === 'manifests_consistency'; + // Whether `?repair=true` does anything for this job — declared by the + // handler itself via `repair_description()`, not by a name allowlist + // here. The allowlist this replaces named only the two ref_count + // tenants and silently omitted every repair-capable job added since, + // so the thumbnail imports could not be run in repair mode from the + // panel at all despite supporting it. + function supportsRepair(job: JobSummary): boolean { + return !!job.repair_description; } - async function onTriggerWithRepairConfirm(name: string) { + // What the repair adds, in the handler's own words. The backend owns + // this string precisely because the wording differs per job: correcting + // a counter and unlinking files off disk are not the same warning, and + // the frontend has no way to tell them apart. + async function onTriggerWithRepairConfirm(job: JobSummary) { const ok = await confirmDialog({ - title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'), - message: t( - 'admin.jobs.run_repair_confirm_body_scoped', - { name }, - 'Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.' + title: t( + 'admin.jobs.run_repair_confirm_title_scoped', + { name: job.name }, + 'Run {{name}} in repair mode?' ), + message: job.repair_description ?? '', confirmText: t('admin.jobs.run_repair_confirm', 'Repair'), danger: true }); - if (ok) await onTrigger(name, { repair: true }); + if (ok) await onTrigger(job.name, { repair: true }); + } + + // Confirmation before a plain run of a job that writes. `never` jobs + // trigger straight through — that is the point of the flag — and + // `on_repair_only` jobs are read-only until the repair variant is + // picked, which carries its own confirm. + async function onTriggerGuarded(job: JobSummary) { + if (job.mutates === 'always') { + const ok = await confirmDialog({ + title: t('admin.jobs.run_mutating_confirm_title', { name: job.name }, 'Run {{name}}?'), + message: + job.description || + t('admin.jobs.run_mutating_confirm_body', 'This job changes stored state when it runs.'), + confirmText: t('admin.jobs.run', 'Run'), + danger: true + }); + if (!ok) return; + } + await onTrigger(job.name); + } + + // Row badge. `never` is the one worth stating outright — it is the + // answer to "is it safe to click this on production?", and it is the + // question an operator asks before every trigger. + function mutatesLabel(job: JobSummary): string | null { + switch (job.mutates) { + case 'never': + return t('admin.jobs.mutates_never', 'read-only'); + case 'on_repair_only': + return t('admin.jobs.mutates_on_repair_only', 'read-only unless repaired'); + default: + return null; + } } function isRunning(job: JobSummary): boolean { @@ -679,11 +725,13 @@ // coordinator is registered (should always be true post-Slice 5, // but check defensively so the button doesn't appear on an old // deployment before this component is upgraded). - // Coordinator registration flag — set imperatively in - // `loadJobs` because `jobs` no longer contains the - // `consistency_batch` row (filtered out to avoid duplicating the - // top-bar action buttons). - let hasBatch = $state(false); + // Held as the whole summary rather than a boolean because the + // top-bar buttons need its `repair_description` — the coordinator + // describes its own repair semantics, same as every table row. + // Set imperatively in `loadJobs` because `jobs` no longer contains + // the `consistency_batch` row (filtered out to avoid duplicating + // the top-bar action buttons). + let batchJob = $state(null);
@@ -697,10 +745,12 @@

- {#if hasBatch} + {#if batchJob} + {@const batch = batchJob} - - + + {#if batch.repair_description} + + {/if} {/if} {/if} - {#if supportsRepair(job.name)} + {#if supportsRepair(job)}