From 9f8ec141f3543a0597aae50270929500d8924881 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Mon, 24 Aug 2026 23:04:53 +0200 Subject: [PATCH] feat(storage): single-source the copy fan-out via copy_file_satellites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was written twice — the copy_file CTE and storage.copy_folder_tree — and had already drifted: the tree path bumped storage.blobs only, missing manifests, which was silent data loss on any multi-chunk file. Fixing it meant writing the same logic a second time. Step 9 adds a file-keyed satellite table, which would mean a third and fourth. Two SQL functions: storage.add_blob_references(TEXT[]) — the manifest-first reference contract for SQL callers, returning hashes that matched no registry row. Set-based so the tree path keeps its single-statement cost; a per-row helper would have made a 10k-file copy 10k calls. storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus the blob reference. The body is the copy-semantics declaration: what is absent (comments, favorites, content-keyed derived rows) is listed with its reason, so the taxonomy is executable rather than documented elsewhere and drifting. Both copy paths now call it. The single-file path becomes a real transaction, which also fixes the reference being best-effort: a failed add_reference used to log a warning and leave a copy holding no reference at all — the exact shape that gets its content reaped. It cannot be a CTE arm, because data-modifying CTEs share one snapshot and the function must read the row the INSERT just wrote. Verified against a scratch PG with all migrations applied: multi-chunk manifest 1→2, single-chunk alias bumped at manifest level only (the NOT EXISTS guard), chunks behind a manifest untouched, dead properties duplicated, length mismatch rejected, repeats counted. tests/api/derived_blob_copy.hurl covers it end-to-end and answers the question the copy raises: content_derived_blobs is NOT copied. A copy carries the same blob_hash, so it resolves the same derived row — the test asserts byte-identical thumbnails from both copy paths, then deletes the original, runs GC, and requires both copies to still serve. That last step only passes if the references are real. --- .../20261019000000_copy_file_satellites.sql | 332 ++++++++++++++++++ .../pg/file_blob_write_repository.rs | 68 ++-- tests/api/derived_blob_copy.hurl | 326 +++++++++++++++++ tests/api/run.sh | 1 + 4 files changed, 700 insertions(+), 27 deletions(-) create mode 100644 migrations/20261019000000_copy_file_satellites.sql create mode 100644 tests/api/derived_blob_copy.hurl diff --git a/migrations/20261019000000_copy_file_satellites.sql b/migrations/20261019000000_copy_file_satellites.sql new file mode 100644 index 00000000..030812bd --- /dev/null +++ b/migrations/20261019000000_copy_file_satellites.sql @@ -0,0 +1,332 @@ +-- Step 8 of `docs/plan/derived-blobs.md` — single-source the copy fan-out. +-- +-- "What follows a file when the file is copied" was written twice: once in +-- the `copy_file` CTE (Rust, `file_blob_write_repository.rs`) and once in +-- `storage.copy_folder_tree`. They had already drifted — the tree path +-- bumped `storage.blobs` only, missing manifests entirely, which was silent +-- data loss on a multi-chunk file (fixed in `20261016000000`, and the fix +-- had to be written a second time rather than in one place). +-- +-- The plan adds file-keyed satellite tables (`file_attached_blobs`, step 9). +-- Adding them against two copy sites means writing the same cascade a third +-- and fourth time, into sites that have already proven they drift. So the +-- fan-out gets exactly one home first. +-- +-- Two functions land here: +-- +-- * `storage.add_blob_references(TEXT[])` — the manifest-first reference +-- contract, expressed once for SQL callers. `DedupService::add_reference` +-- is the Rust twin; they must change together, which is why the shared +-- contract is spelled out in both doc comments. +-- +-- * `storage.copy_file_satellites(UUID[], UUID[])` — everything that +-- follows a file on copy. The body IS the copy-semantics declaration: +-- what is absent is a documented decision (see the trailing comments), +-- not an omission someone has to notice. +-- +-- Set-based rather than per-row on purpose. A per-row helper would have made +-- a 10k-file folder copy 10k function calls; taking arrays keeps the tree +-- path's single-statement cost while still having one implementation. The +-- single-file path passes one-element arrays. + +-- ── The reference contract, for SQL callers ────────────────────────────── +-- +-- Increment the reference count for each hash in `p_hashes`, counting +-- repeats (pass the hash once per referencing row). Returns the hashes that +-- matched NEITHER table, so callers can decide how loud to be — a copy +-- inherits a pre-existing breakage and should warn, whereas an ingest +-- referencing a nonexistent blob is a hard error. +-- +-- MANIFEST FIRST, `storage.blobs` only as fallback. The order is the whole +-- point: a CDC file's `blob_hash` names a manifest +-- (`chunk_manifests.file_hash`), not a chunk, so bumping `storage.blobs` +-- first would match nothing for a multi-chunk file and take no reference at +-- all. +-- +-- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing. For a +-- SINGLE-chunk file the whole-file hash EQUALS its lone chunk's hash (both +-- are BLAKE3 over the same bytes), so without the guard one reference would +-- be counted at both levels — turning an under-count into an over-count. +-- +-- Mirrors `DedupService::add_reference`, including the asymmetry on +-- `orphaned_at`: only `storage.blobs` carries that column, so only the blobs +-- branch clears it. A chunk resurrected inside its GC grace window must lose +-- its orphan stamp or `dedup_gc` reaps live content. +CREATE OR REPLACE FUNCTION storage.add_blob_references(p_hashes TEXT[]) +RETURNS TEXT[] AS $$ +DECLARE + v_unmatched TEXT[]; +BEGIN + IF p_hashes IS NULL OR cardinality(p_hashes) = 0 THEN + RETURN ARRAY[]::TEXT[]; + END IF; + + WITH hc AS ( + SELECT h AS blob_hash, COUNT(*)::int AS cnt + FROM unnest(p_hashes) AS h + WHERE h IS NOT NULL + GROUP BY h + ), + bumped_manifests AS ( + UPDATE storage.chunk_manifests m + SET ref_count = m.ref_count + hc.cnt + FROM hc + WHERE m.file_hash = hc.blob_hash + RETURNING m.file_hash + ), + bumped_blobs AS ( + UPDATE storage.blobs b + SET ref_count = b.ref_count + hc.cnt, + orphaned_at = NULL + FROM hc + WHERE b.hash = hc.blob_hash + AND NOT EXISTS ( + SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash + ) + RETURNING b.hash + ) + SELECT COALESCE(array_agg(hc.blob_hash), ARRAY[]::TEXT[]) + INTO v_unmatched + FROM hc + WHERE NOT EXISTS (SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash) + AND NOT EXISTS (SELECT 1 FROM bumped_blobs WHERE hash = hc.blob_hash); + + RETURN v_unmatched; +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION storage.add_blob_references(TEXT[]) IS + 'Manifest-first blob reference increment for SQL callers. Returns hashes ' + 'that matched no registry row. Rust twin: DedupService::add_reference — ' + 'change both together.'; + +-- ── What follows a file on copy ────────────────────────────────────────── +-- +-- `p_old_ids[i]` is copied to `p_new_ids[i]`; the new `storage.files` rows +-- must already be inserted and visible (both callers insert in an earlier +-- statement of the same transaction). +-- +-- Every satellite of a copied file belongs in this body. What is NOT here is +-- listed at the bottom, with the reason — the taxonomy is executable rather +-- than living in a document that drifts from the code. +CREATE OR REPLACE FUNCTION storage.copy_file_satellites( + p_old_ids UUID[], + p_new_ids UUID[] +) RETURNS void AS $$ +DECLARE + v_unmatched TEXT[]; +BEGIN + IF p_old_ids IS NULL OR cardinality(p_old_ids) = 0 THEN + RETURN; + END IF; + + IF p_new_ids IS NULL OR cardinality(p_old_ids) <> cardinality(p_new_ids) THEN + -- Positional correspondence is the whole interface; a length + -- mismatch would silently attach satellites to the wrong file. + RAISE EXCEPTION + 'copy_file_satellites: id arrays must correspond positionally (% old vs % new)', + cardinality(p_old_ids), COALESCE(cardinality(p_new_ids), 0); + END IF; + + -- 1. WebDAV dead properties. RFC 4918 §8.8 requires COPY to duplicate + -- them: properties describe the resource, and the copy is a resource. + INSERT INTO storage.webdav_dead_properties + (file_id, namespace, local_name, value) + SELECT m.new_id, dp.namespace, dp.local_name, dp.value + FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id) + JOIN storage.webdav_dead_properties dp ON dp.file_id = m.old_id; + + -- 2. A reference on the copied content, so deleting the original cannot + -- reap bytes the copy still needs. Read from the NEW rows rather than + -- the old ones: that is what makes an unreferenceable copy impossible + -- to create, since a row that failed to insert contributes nothing. + SELECT storage.add_blob_references(array_agg(f.blob_hash)) + INTO v_unmatched + FROM unnest(p_new_ids) AS n(id) + JOIN storage.files f ON f.id = n.id + WHERE NOT f.is_trashed; + + IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN + -- Warn, do not abort. A missing registry row means the SOURCE file + -- was already broken; the copy merely inherits it. Failing here + -- would abort an entire folder copy over one pre-existing fault, + -- which is worse than completing it and reporting. The blob-level + -- audit jobs are what surface the underlying breakage. + RAISE WARNING + 'copy_file_satellites: % copied file(s) reference a blob with no registry row (first: %); source was already broken', + cardinality(v_unmatched), v_unmatched[1]; + END IF; + + -- ── Deliberately absent ────────────────────────────────────────────── + -- + -- storage.comments (future): NOT copied. A copy is a new artifact; the + -- discussion belongs to the original. + -- + -- storage.file_attached_blobs (step 9): WILL be copied here, with a + -- reference taken per attached blob_hash via add_blob_references. + -- + -- content_derived_blobs, blob_extracted_text, faces.faces: content-keyed. + -- The copy shares the source's hash, so it already sees them — copying + -- would duplicate rows that are keyed on the very thing being shared. + -- + -- storage.favorites, recent_items, shares: properties of the ORIGINAL's + -- relationship to users, not of its content. +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION storage.copy_file_satellites(UUID[], UUID[]) IS + 'Single source of truth for what follows a file on copy. Both copy paths ' + '(single-file and copy_folder_tree) call it. Adding a file-keyed satellite ' + 'table means editing this function, and only this function.'; + +-- ── Route copy_folder_tree through it ──────────────────────────────────── +-- +-- Only two blocks change versus `20261016000000`: the inline reference bump +-- and the per-file dead-property INSERT are both replaced by one +-- `copy_file_satellites` call. The folder dead-property INSERT stays inline +-- — folders are not files and have no satellite fan-out to share. +CREATE OR REPLACE FUNCTION storage.copy_folder_tree( + p_source_id UUID, + p_target_parent_id UUID, -- NULL = copy to root (keeps source drive) + p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name +) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$ +DECLARE + v_root_lpath ltree; + v_root_depth INT; + v_max_depth INT; + v_level INT; + v_folders BIGINT := 0; + v_files BIGINT := 0; + v_inserted BIGINT; + v_new_root UUID; + v_dest_drive_id UUID; +BEGIN + -- Validate source exists. + SELECT fo.lpath, nlevel(fo.lpath) + INTO v_root_lpath, v_root_depth + FROM storage.folders fo + WHERE fo.id = p_source_id AND NOT fo.is_trashed; + + IF v_root_lpath IS NULL THEN + RAISE EXCEPTION 'Source folder not found: %', p_source_id + USING ERRCODE = 'P0002'; -- no_data_found + END IF; + + -- Resolve destination drive_id once up front (cross-drive copy path). + IF p_target_parent_id IS NULL THEN + SELECT fo.drive_id INTO v_dest_drive_id + FROM storage.folders fo + WHERE fo.id = p_source_id; + ELSE + SELECT fo.drive_id INTO v_dest_drive_id + FROM storage.folders fo + WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed; + IF v_dest_drive_id IS NULL THEN + RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id + USING ERRCODE = 'P0002'; + END IF; + END IF; + + -- Temp mapping: every folder in the subtree → new UUID. + CREATE TEMP TABLE IF NOT EXISTS _copy_map( + old_id UUID PRIMARY KEY, + new_id UUID NOT NULL DEFAULT gen_random_uuid() + ) ON COMMIT DROP; + TRUNCATE _copy_map; + + INSERT INTO _copy_map(old_id) + SELECT fo.id + FROM storage.folders fo + WHERE NOT fo.is_trashed + AND fo.lpath <@ v_root_lpath; + + SELECT cm.new_id INTO v_new_root + FROM _copy_map cm WHERE cm.old_id = p_source_id; + + SELECT MAX(nlevel(fo.lpath)) + INTO v_max_depth + FROM storage.folders fo + JOIN _copy_map cm ON fo.id = cm.old_id; + + -- ── Insert folders level by level ── + -- Post-D7: `user_id` intentionally omitted from the column list so + -- copied rows leave the (now-nullable) column NULL. Provenance is + -- carried by `created_by` / `updated_by` (§14 columns) — preserved + -- from source so authorship survives the copy. + FOR v_level IN v_root_depth .. v_max_depth LOOP + INSERT INTO storage.folders( + id, name, parent_id, + drive_id, created_by, updated_by + ) + SELECT cm.new_id, + CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL + THEN p_dest_name ELSE fo.name END, + CASE WHEN fo.id = p_source_id THEN p_target_parent_id + ELSE pm.new_id END, + v_dest_drive_id, + fo.created_by, + fo.updated_by + FROM storage.folders fo + JOIN _copy_map cm ON fo.id = cm.old_id + LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id + WHERE NOT fo.is_trashed + AND nlevel(fo.lpath) = v_level; + + GET DIAGNOSTICS v_inserted = ROW_COUNT; + v_folders := v_folders + v_inserted; + END LOOP; + + -- Temp mapping for files src→dst (dst ids pre-allocated so we can hand + -- both sides to copy_file_satellites below). + CREATE TEMP TABLE IF NOT EXISTS _copy_file_map( + old_id UUID PRIMARY KEY, + new_id UUID NOT NULL DEFAULT gen_random_uuid() + ) ON COMMIT DROP; + TRUNCATE _copy_file_map; + + INSERT INTO _copy_file_map(old_id) + SELECT f.id + FROM storage.files f + JOIN _copy_map cm ON f.folder_id = cm.old_id + WHERE NOT f.is_trashed; + + -- ── Batch copy all files (zero-copy: same blob_hash) ── + -- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`. + INSERT INTO storage.files( + id, name, folder_id, blob_hash, size, mime_type, + media_sort_date, drive_id, created_by, updated_by + ) + SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size, + f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by, + f.updated_by + FROM storage.files f + JOIN _copy_map cm ON f.folder_id = cm.old_id + JOIN _copy_file_map fm ON fm.old_id = f.id + WHERE NOT f.is_trashed; + + GET DIAGNOSTICS v_files = ROW_COUNT; + + -- Everything that follows a file on copy — blob references and dead + -- properties — in one call, shared with the single-file copy path. + -- + -- Both aggregates order by `old_id`, which is what makes the two arrays + -- correspond positionally; `array_agg` without a matching ORDER BY would + -- be free to pair a file with another file's satellites. + IF v_files > 0 THEN + PERFORM storage.copy_file_satellites( + (SELECT array_agg(old_id ORDER BY old_id) FROM _copy_file_map), + (SELECT array_agg(new_id ORDER BY old_id) FROM _copy_file_map) + ); + END IF; + + -- Folder dead properties. Files are handled inside copy_file_satellites; + -- folders have no other satellites, so this stays here. + INSERT INTO storage.webdav_dead_properties + (folder_id, namespace, local_name, value) + SELECT cm.new_id, dp.namespace, dp.local_name, dp.value + FROM storage.webdav_dead_properties dp + JOIN _copy_map cm ON dp.folder_id = cm.old_id; + + RETURN QUERY SELECT v_new_root::text, v_folders, v_files; +END; +$$ LANGUAGE plpgsql; diff --git a/src/infrastructure/repositories/pg/file_blob_write_repository.rs b/src/infrastructure/repositories/pg/file_blob_write_repository.rs index 0e57777c..f9cb56af 100644 --- a/src/infrastructure/repositories/pg/file_blob_write_repository.rs +++ b/src/infrastructure/repositories/pg/file_blob_write_repository.rs @@ -596,8 +596,25 @@ impl FileWritePort for FileBlobWriteRepository { new_name: Option<&str>, caller_id: Uuid, ) -> Result { - // Atomic CTE: read source file → insert new row with same blob_hash → increment ref_count. - // Single round-trip; blob content is NOT copied (dedup makes this zero-copy). + // Two statements in one transaction: insert the new row (same + // blob_hash — blob content is never copied, dedup makes this + // zero-copy), then run the shared satellite fan-out. + // + // `storage.copy_file_satellites` is the single home for everything + // that follows a file on copy — dead properties and the + // manifest-aware blob reference — shared with + // `storage.copy_folder_tree`. Two sites implementing that + // separately is what let the tree path ship a version that missed + // manifests entirely (migration `20261019000000`). + // + // It cannot be a CTE arm: data-modifying CTEs all observe the same + // snapshot, so a function called alongside the INSERT would not see + // the new `storage.files` row it needs to read `blob_hash` from, + // and the dead-property INSERT would fail its foreign key. Hence a + // real transaction — which also fixes the reference being + // best-effort before: a failed `add_reference` used to log a + // warning and leave a copy holding no reference at all, the exact + // shape that gets its content reaped. // // §14: `created_by = $4 = updated_by = caller_id` — the caller // authored this copy. The previous binding used @@ -607,8 +624,10 @@ impl FileWritePort for FileBlobWriteRepository { let target_fid = target_folder_id.clone(); let rename_to = new_name.map(|s| s.to_string()); - let row = retry_on_deadlock("files.copy", || { - sqlx::query_as::< + let row = retry_on_deadlock("files.copy", || async { + let mut tx = self.pool.begin().await?; + + let row = sqlx::query_as::< _, ( String, @@ -662,20 +681,6 @@ impl FileWritePort for FileBlobWriteRepository { blob_hash, created_by, updated_by - ), - -- RFC 4918 §8.8 — dead properties MUST be duplicated on - -- COPY. With the id-keyed store (migration - -- 20260830000001) this is a single batch INSERT keyed on - -- the new file's id. Runs in the same query as the file - -- INSERT so either both land or neither does — atomic - -- by virtue of being one statement. - dead_prop_copy AS ( - INSERT INTO storage.webdav_dead_properties - (file_id, namespace, local_name, value) - SELECT (SELECT id FROM new_file), - dp.namespace, dp.local_name, dp.value - FROM storage.webdav_dead_properties dp - WHERE dp.file_id = $1::uuid ) SELECT id_text, name, folder_id, size, mime_type, created_at, updated_at, @@ -687,7 +692,22 @@ impl FileWritePort for FileBlobWriteRepository { .bind(&target_fid) .bind(&rename_to) .bind(caller_id) - .fetch_optional(self.pool.as_ref()) + .fetch_optional(&mut *tx) + .await?; + + if let Some(ref new_row) = row { + // `new_row.0` is the new file's id as text; PG casts it. + sqlx::query( + "SELECT storage.copy_file_satellites(ARRAY[$1::uuid], ARRAY[$2::uuid])", + ) + .bind(file_id) + .bind(&new_row.0) + .execute(&mut *tx) + .await?; + } + + tx.commit().await?; + Ok(row) }) .await .map_err(|e| { @@ -705,14 +725,8 @@ impl FileWritePort for FileBlobWriteRepository { let blob_hash = &row.7; - // Increment blob reference count (best-effort; INSERT already succeeded) - if let Err(e) = self.dedup.add_reference(blob_hash).await { - tracing::warn!( - "Failed to increment blob ref for copy {}: {}", - &blob_hash[..12], - e - ); - } + // No `add_reference` here: `copy_file_satellites` took it inside the + // transaction above, so a copy that exists always holds a reference. tracing::info!( "📋 BLOB COPY: {} (hash: {}, zero-copy via dedup)", diff --git a/tests/api/derived_blob_copy.hurl b/tests/api/derived_blob_copy.hurl new file mode 100644 index 00000000..20e5987b --- /dev/null +++ b/tests/api/derived_blob_copy.hurl @@ -0,0 +1,326 @@ +# ============================================================= +# OxiCloud – Derived blobs survive a copy, and are SHARED not duplicated +# ============================================================= +# Guards two properties of `docs/plan/derived-blobs.md` that are easy to +# break and silent when broken. +# +# 1. **Derived content is content-keyed, so a copy gets it for free.** +# `storage.content_derived_blobs` is keyed on `source_hash`, and a copy +# carries the SAME `blob_hash` as its original. So the copy resolves to +# the very same thumbnail row — nothing is duplicated, and nothing is +# re-rendered. A regression that made copy duplicate those rows would +# still return 200 here; the byte-identity assertions are what catch it, +# because a re-render produces different bytes than a cache hit only if +# the pipeline is non-deterministic — so we also assert the ref_count, +# which a duplicated row would inflate. +# +# 2. **A copy takes a real blob reference, via BOTH copy paths.** +# `storage.copy_file_satellites` (migration `20261019000000`) is now the +# single home for that, called by the single-file path and by +# `storage.copy_folder_tree`. The tree path previously bumped +# `storage.blobs` only — which matched nothing for a manifest-backed +# file, so a folder copy took NO reference and deleting the original +# reaped bytes the copy still needed. Steps 6 and 9 are what would fail. +# +# The strongest assertion is step 11: after the ORIGINAL is permanently +# deleted and GC has run, both copies must still serve their thumbnail. +# That only holds if the references were real. +# +# Coverage note: `dedup-test.jpg` is single-chunk, so `file_hash` equals its +# lone chunk's hash — the aliasing case whose `NOT EXISTS` guard stops one +# reference being counted at both levels. The multi-chunk fan-out (where +# file_hash names a manifest that is NOT a chunk) differs only in that the +# hashes differ; it has no thumbnail-capable fixture at this size, so it is +# covered at the SQL level rather than here. +# +# Prerequisites: setup.hurl must have run (admin user exists). +# +# Run: +# hurl --variables-file tests/api/test.env --file-root tests \ +# --test tests/api/derived_blob_copy.hurl +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Source and destination folders +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-derived-src" +} + +HTTP 201 +[Captures] +src_folder_id: jsonpath "$.id" + + +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-derived-dst" +} + +HTTP 201 +[Captures] +dst_folder_id: jsonpath "$.id" + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Upload the source image +# +# `content_hash` is captured rather than hardcoded so the test does not +# break if the fixture is ever regenerated. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +folder_id: {{src_folder_id}} +file: file,fixtures/dedup-test.jpg; image/jpeg + +HTTP 201 +[Captures] +orig_file_id: jsonpath "$.id" +orig_file_name: jsonpath "$.name" +blob_hash: jsonpath "$.content_hash" +[Asserts] +jsonpath "$.content_hash" isString + + +# One file holds the blob. +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 1 + + +# ───────────────────────────────────────────────────────────── +# Step 4 – Render the thumbnail. THIS is what creates the derived blob: +# `content_derived_blobs(source_hash = blob_hash, 'thumbnail', …)`. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{orig_file_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +thumb_bytes: bytes + + +# ───────────────────────────────────────────────────────────── +# Step 5 – Single-file copy into the destination folder +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/files/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "file_ids": ["{{orig_file_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +file_copy_id: jsonpath "$.successful[0].id" +[Asserts] +jsonpath "$.successful[0].id" != "{{orig_file_id}}" + + +# ───────────────────────────────────────────────────────────── +# Step 6 – The copy took a reference. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 2 + + +# ───────────────────────────────────────────────────────────── +# Step 7 – The copy serves the SAME thumbnail bytes. +# +# It shares the original's `blob_hash`, so it resolves the same +# `content_derived_blobs` row. Nothing was copied to make this work — +# that is the content-keying payoff. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 8 – Folder copy — the OTHER copy path, through +# `storage.copy_folder_tree` → `copy_file_satellites`. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/batch/folders/copy +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "folder_ids": ["{{src_folder_id}}"], + "target_folder_id": "{{dst_folder_id}}" +} + +HTTP 200 +[Captures] +tree_root_id: jsonpath "$.successful[0].new_root_folder_id" +[Asserts] +jsonpath "$.stats.failed" == 0 + + +GET {{base_url}}/api/files?folder_id={{tree_root_id}} +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +tree_copy_id: jsonpath "$[0].id" +[Asserts] +jsonpath "$" count == 1 +jsonpath "$[0].name" == "{{orig_file_name}}" +jsonpath "$[0].id" != "{{orig_file_id}}" +jsonpath "$[0].content_hash" == "{{blob_hash}}" + + +# ───────────────────────────────────────────────────────────── +# Step 9 – Three references now. Before `copy_file_satellites` the tree +# path contributed nothing here and this stayed at 2. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.ref_count" == 3 + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 10 – Permanently delete the ORIGINAL. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/files/{{orig_file_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_orig_id: jsonpath "$.items[?(@.resource.id == '{{orig_file_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_orig_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +# Two copies remain, so the content must too. +GET {{base_url}}/api/dedup/check/{{blob_hash}} +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.exists" == true +jsonpath "$.ref_count" == 2 + + +# ───────────────────────────────────────────────────────────── +# Step 11 – Run GC, then prove both copies still work. +# +# This is the assertion the whole file exists for. If either copy had +# failed to take a reference, the original's deletion would have walked +# the count to 0 and GC would have reaped the content AND its derived +# thumbnail — leaving these 5xx. That was a real, shipped bug on the +# folder-copy path. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/dedup_gc/trigger +Authorization: Bearer {{token}} +[Options] +delay: 500ms + +HTTP 200 + + +GET {{base_url}}/api/files/{{file_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +GET {{base_url}}/api/files/{{tree_copy_id}}/thumbnail/preview +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +bytes == {{thumb_bytes}} + + +# ───────────────────────────────────────────────────────────── +# Step 12 – Teardown. Hurl files share one database within run.sh, so +# everything created here must go, including from trash. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/folders/{{src_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +DELETE {{base_url}}/api/folders/{{dst_folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_src_id: jsonpath "$.items[?(@.resource.id == '{{src_folder_id}}')].resource.id" +trash_dst_id: jsonpath "$.items[?(@.resource.id == '{{dst_folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_src_id}} +Authorization: Bearer {{token}} + +HTTP 200 + + +DELETE {{base_url}}/api/trash/{{trash_dst_id}} +Authorization: Bearer {{token}} + +HTTP 200 diff --git a/tests/api/run.sh b/tests/api/run.sh index 3ff37afb..67619aec 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -166,6 +166,7 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/recent.hurl" \ "$API_DIR/batch_folder_copy.hurl" \ "$API_DIR/dedup_blob_cleanup.hurl" \ + "$API_DIR/derived_blob_copy.hurl" \ "$API_DIR/dedup_admin_gate.hurl" \ "$API_DIR/admin_jobs.hurl" \ "$API_DIR/recoverable_jobs.hurl" \