12158ccf592d327b89a432e65d590547ef666d50
2051 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Edouard Vanbelle
|
12158ccf59 |
fix(storage): thumb_derived_import claims JPEG sidecars too
The filter was strip_suffix(".webp"), but persist_rendered writes
{hash}.{format.ext()} — so any client not advertising WebP leaves
{hash}.jpg on disk. Correct only while the derived tier was WebP-only;
once variant carried the format (20261022000000) a JPEG sidecar became
ordinary content, and leaving it unclaimed would keep .thumbnails/
permanently non-empty — the very signal step 10e gates on. The migration
could never finish.
Both codecs are now claimed and the format comes from the file's own
extension, so a .jpg imports AS JPEG. Deriving the variant and
content_type from it rather than hardcoding WebP is the point: a
mislabelled row would serve the wrong codec to whoever the read path
then matched it for.
ThumbnailFormat::ALL exists so the claim list and the write path cannot
drift — adding a format without teaching the import about it would
strand that codec silently.
The `ext-` rejection now carries real weight. Previously .jpg was
rejected wholesale, so the two jobs could not overlap by construction;
now they share an extension and only the prefix separates them. Both
directions stay under test.
Caught by the cross-job assertion, which counts every real sidecar being
claimed exactly once — the fixture gained a .jpg and the total moved 3
to 4, which is the test noticing rather than a test to update.
|
||
|
Edouard Vanbelle
|
f4e4b518a9 |
feat(thumbnails): classify render failures as permanent or transient
Prerequisite for persisting negative verdicts, and the reason step 7 cannot start with the storage change. generate_and_persist collapses EVERY error into empty Bytes — timeouts and closed semaphores included. That is survivable only because the sentinel lives in moka, which evicts. Write the same signal to content_derived_blobs and a thumbnail that timed out once under load is unrenderable forever. So the classification lands first, on its own, before anything can persist it. It maps onto the existing variants without restructuring, because timeouts already surface as TaskError: ImageError, UnsupportedFormat -> permanent. The decoder rejected these bytes, or they exceed MAX_DECODE_PIXELS. Facts about the image. TaskError, IoError -> transient. Timeout, closed decode semaphore, join failure, unreadable source. Facts about the moment. The asymmetry sets the default: a wrongly-persisted transient marks a good image unrenderable for good, while a wrongly-omitted permanent merely costs a repeated decode. Anything not clearly a content property is therefore transient. Tested by pinning the mapping rather than trusting variant names to stay put — the timeout case especially, since it is the one that turns a load spike into data loss. Nothing consumes this yet; it exists so the storage change cannot be written without it. |
||
|
Edouard Vanbelle
|
647929ed11 |
docs(plan): negative verdicts get a nullable blob_hash
Resolves the .skip-marker question. The table can say "here is the artifact" but not "there is deliberately no artifact", and absence of a row is ambiguous — it means both never-attempted and attempted-and-not- worth-it, which destroys the only thing a negative cache holds. Two live cases, not one. Transcodes drop a .skip marker when WebP is not smaller. Thumbnails return empty Bytes for undecodable sources and ones over MAX_DECODE_PIXELS — RAM-only today, so after moka evicts, a 60-megapixel upload has its full decode attempted again, forever. Same gap, same table, so do both together. Nullable rather than a sentinel hash: a sentinel stops blob_hash naming a real blob and every future reader has to know the lie. Costs are one `AND blob_hash IS NOT NULL` in ContentDerivedReferenceSource, the same guard on the dangling-derived check, and dropping NOT NULL. Permanent vs transient is the load-bearing split, and today's code cannot tell them apart: generate_and_persist collapses every error into empty Bytes, timeouts and semaphore closures included. Survivable while the sentinel lives in moka, which evicts. Persist that same signal and a thumbnail that timed out once under load is unrenderable forever. So the renderer must return a typed outcome first, and only permanently-unrenderable earns a row. The asymmetry sets the default — a wrongly-cached transient is silent and permanent, a not-cached permanent only costs repeated work — so unclassified errors are treated as transient. No TTL and no renderer-version term. Time is the wrong axis: the verdict is deterministic in (content, encoder) and does not decay, so a TTL re-attempts an OOMing decode on a schedule while still leaving staleness for most of the window after a deploy. What removes the need for a mechanism is that negative rows are disposable — they hold no data, so a library upgrade invalidates them with a DELETE ... WHERE blob_hash IS NULL in the same migration as the dependency bump. Recorded explicitly so nobody later builds the expiry logic this replaces. |
||
|
Edouard Vanbelle
|
c04f0c6824 |
docs(plan): scope step 7, the transcode dual-write
Investigated but deliberately not started — the remaining pieces span a service, its DI wiring and an unresolved design question, and a half-wired service is worse than none. ImageTranscodeService does not write the derived tier at all today, the same gap persist_rendered closed for thumbnails, and it must close before transcode_import can converge or the import chases a growing cache. Three pieces, and the first is smaller than it looks. get_transcoded takes file_id while the table is content-keyed, but the hash is already in scope one frame up — file_retrieval_service::try_transcode is called where dto.content_hash is live — so it is a parameter to thread, not a lookup to invent. Explicitly NOT by hashing original_content on the fly, which would be a BLAKE3 over the whole file per request. Second, the service has no BlobHandler field, so this touches the constructor and DI; ThumbnailService hit the same ordering problem and solved it with a per-call parameter, which is the cheaper precedent. Third, the .skip markers stay unresolved: a negative verdict has no bytes, so it does not fit a table whose row points at a blob. Either leave them local and recompute per instance, or model a sentinel. It is the only genuinely open design question left in step 10. |
||
|
Edouard Vanbelle
|
d202b4b5ca |
fix(storage): the derived import conflated variant with directory
76590160 changed `variant_of` to return `{size}.{ext}`, but that value
was also being used as the on-disk DIRECTORY. Reads became
`.thumbnails/preview.webp/{hash}.webp`, which does not exist, so every
sidecar counted as unreadable and thumb_derived_import restored nothing.
Caught by thumb_import_check.sh on the run after the migration — the
harness earning its keep twice now, since this is the second defect it
has caught that no unit test could.
They are genuinely two strings and are now named as such: `dir_name` for
the path, `variant` for the row key. The cursor keeps using the
directory, so a run paused before the migration resumes at the same
position rather than restarting.
Also stops podman's compose-provider banner from burying the script's
output. Filtered rather than discarded, so genuine psql errors still
surface — swallowing those would turn a broken query into a silently
wrong assertion. Suppressing it at the source needs
`[engine] compose_warning_logs = false` in containers.conf, which is
per-developer config and cannot be relied on in CI.
|
||
|
Edouard Vanbelle
|
86d0d65583 |
feat(storage): derived variant encodes the output format
`content_derived_blobs.variant` held the size alone, so one source could
hold exactly one artifact per size regardless of codec. That surfaced
when the read order flipped in 10c: a JPEG request matched the WebP row
and would have been served the wrong codec — hidden previously because
the .jpg sidecar won first. The flip had to be gated to WebP, which
meant JPEG clients could never leave the sidecar, which meant the
sidecar could never be deleted.
It blocks transcodes harder: those are multi-format by nature, so two
output codecs of one source collide on the primary key without a format
term.
The axis goes inside the string rather than into a fourth PK column,
per the column's own rule — "new axes go inside this string, never into
new columns". Shape is {size}.{ext}: preview.webp, icon.jpg, later
720p.webp.
The backfill is deterministic, not a guess: store_derived_blob has only
ever written "image/webp" for thumbnails. content_type is checked anyway
rather than assumed — a row that fails the assumption is left alone and
counted in a warning, because the read path then simply misses it and
falls back to the sidecar, whereas guessing a codec would serve wrong
bytes. Idempotent via NOT LIKE '%.%', so a re-apply cannot produce
preview.webp.webp; verified on a scratch PG by applying it twice.
One helper builds the string, because it is a primary-key component:
a writer and reader that disagree do not fail loudly, they just never
find each other's rows and the derived tier silently looks empty. It
lives on the service's ThumbnailSize, not the port's — they are distinct
types, which the compiler pointed out after I put it on the wrong one.
The WebP gate on the read path is now removed: each codec has its own
row, so JPEG can finally reach the derived tier — the prerequisite for
deleting the sidecar for those clients.
file_attached_blobs keeps a bare size: store_external_thumbnail
re-encodes everything to JPEG, so it is single-format by construction
and a format term would cost a migration for nothing.
|
||
|
Edouard Vanbelle
|
c656e684d4 |
feat(consistency): merge-join backend_consistency, both directions
The job walked the backend and probed the DB with WHERE hash = ANY($1) over each page, so it could only ever see backend-only entries. A registry row whose bytes are gone never appears in a backend listing — it was invisible here by construction, and that half was left to blobs_consistency's per-row HEAD probe, which does not survive the row counts this plan produces. Both sides are now ordered by hash — the backend by contract since 5343fdda, the DB by ORDER BY hash — so one pass yields both deltas: orphan_blob for bytes with no row, and blob_missing_from_backend for a row with no bytes. The latter is severity data_loss rather than inconsistent: an orphan wastes space, this loses a file. Three things the merge needs that a probe did not. A horizon. The two pages cover different ranges, so only their overlap can be judged — beyond it, a hash missing from one side may simply be on the next page of the other, and emitting there would invent findings in both directions at once. When a side is exhausted its entries cannot be on a later page, so the other's tail becomes judgeable. One cursor for both sides. They share an ordering, so "resume after H" is start_after(H) on the backend and hash > H in the DB. The cursor advances to the horizon, not the backend's own next_cursor, which would skip the un-judged tail of whichever side reached further. Format is unchanged, so paused runs resume. And an ordering premise worth stating rather than assuming: hashes are lowercase BLAKE3 hex of fixed length, so collation and byte order rank identically over [0-9a-f]. A hash column admitting uppercase or variable length would break this silently. Recorded in the module docs and beside the query. blobs_consistency still emits its own blob_missing_from_backend; the two now overlap. Retiring that probe is the follow-up, not folded in here. |
||
|
Edouard Vanbelle
|
d7de1c41e7 |
fix(files): missing folder_id is 400, not 500
Uploading without folder_id answered `500 Internal Error: folder_id is required to determine file owner`. A missing required field is the caller's error; as an internal_error it produced `error_type: Internal Error`, which the SPA cannot distinguish from the server breaking — so a malformed request looked like an outage. Both sites become validation_error (ErrorKind::InvalidInput → 400), with messages that say WHY the field is needed rather than restating that it is: the destination folder determines the file's owner and drive. The OpenAPI request body described it as "optional folder_id field", which is how it came to be omitted — hit while writing thumbnail_etag_content_keyed.hurl, where the upload was written from the documented contract and 500'd. Now stated as required. Regression test asserts the status AND that error_type is not "Internal Error", since the contract the SPA switches on is error_type rather than the message. |
||
|
Edouard Vanbelle
|
4997eb4fb0 |
docs(plan): sequence transcode_import behind its two prerequisites
Answering "when is transcode_import planned": it is the remaining third of 10(b), but it must not be next, and both reasons were learned on the thumbnail side rather than predicted. Format has to move into `variant` first. Transcodes are inherently multi-format and `variant` is keyed on size alone — the same gap found in 10c that keeps JPEG thumbnails on the sidecar. Importing before that means migrating into a schema that cannot hold the data without collisions across formats. One migration unblocks both, which is the argument for doing it before either. And step 7 must precede the import. ImageTranscodeService writes only its file-keyed disk cache today, so the import would run against a cache that is still growing and never reach an empty tail — exactly the trap persist_rendered had to close for thumbnails, where one of four render paths recorded a row and the tail could never empty. Order: format-in-variant → step 7 → transcode_import. Also records why the .skip markers are still open: a cached negative verdict has no bytes, so it does not fit a table whose point is pointing at a blob. |
||
|
Edouard Vanbelle
|
260e6bb74f |
feat(thumbnails): read the derived tier ahead of the sidecar
Step 10c. The sidecar is local disk — invisible to other instances, uncarried by a backend migration, uncovered by any consistency job. Reading the derived tier first is what makes that state deletable. Not the cost it appears to be: CachedBlobBackend gives the blob read a local disk cache and moka absorbs the repeats above it. Not a two-line swap, for two reasons. A derived MISS must fall through to the sidecar; the old code terminated the lookup with `?` because it was last. While the imports drain, most content has a sidecar and no row — terminating there would report "no thumbnail" for nearly all of it. And the derived tier is WebP-only. store_derived_blob writes image/webp and keys `variant` on the size alone, with no format term, so a JPEG request matches the WebP row and would be served the wrong codec. The old ordering hid this because the .jpg sidecar won first. So the lookup is gated to WebP, JPEG clients stay on the sidecar — and the sidecar cannot be deleted for them until `variant` encodes format. That is a new prerequisite for step 10e, recorded in the plan rather than discovered later. Also corrects the plan: I had written that this flip removes the derived-hash ETag hazard. It does not. A first render still creates the row as a side effect of producing the body, whatever the read order, so two consecutive reads still straddle its appearance. The real fix is resolving the ETag after generation on the 200 path — a 304 only fires when the client already holds a validator, which implies the row exists. That is a handler restructure, not an ordering change. |
||
|
Edouard Vanbelle
|
48d9e164d4 |
test(thumbnails): pin get_cached_thumbnail's tier precedence
This function produced four bugs in two days, every one an ordering mistake rather than a logic error, and every one caught only by an end-to-end run comparing bytes against something independent: the content-keyed RAM entry shadowing an uploaded preview so a PUT appeared to do nothing; that precedence being right on disk but wrong in RAM; a validator flipping because a tier was populated as a side effect of producing the body; a decode error reading as "absent". They all violate one sentence — a file-specific override beats anything derived from the content, at every tier — so that is what these pin. Step 10c is entirely a precedence change (derived ahead of sidecar), and it should not be another end-to-end guess. No database needed. With `dedup: None` the two DB tiers are skipped, and what remains — per-file RAM, ext- disk, content RAM, blob-hash sidecar — is exactly where the bugs were. Seven cases: the two override rules, RAM over disk within the content tiers, the sidecar answering alone, the ext- read caching under the PER-FILE key (a content key would leak one user's preview to every file sharing the content), a hashless caller falling through instead of guessing, and moka's empty-bytes negative entry not being served as a thumbnail. Verified by mutation, not just by passing: restoring the old precedence fails exactly the two tests that encode the rule, and no others. Lives in thumbnail_service.rs rather than the sibling test file because seeding tiers needs the private `cache` field. |
||
|
Edouard Vanbelle
|
2775e6d567 |
docs: the two sidecar-only paths are production-unreachable
Closing out step 10(a). The remaining `None` call sites in persist_rendered looked like an open gap; they are not reachable in production. Both `get_thumbnail` and the path variant of `generate_all_sizes_background` are called only from the `ThumbnailPort` impl, and nothing holds a `dyn ThumbnailPort` — which the existing note in get_cached_thumbnail already recorded and a grep confirms. Live renders go through get_thumbnail_from_blob and generate_all_sizes_background_from_blob, both of which carry a DedupService and dual-write. So threading a DedupService through them would be work with no runtime effect. Recorded at each call site instead, with the condition that matters: gaining a real caller means taking a DedupService first, or the gap persist_rendered exists to close reopens — sidecar-only output the import can never see, so the tail never empties and the deletion gate never opens. Marks 10(a) done in the plan with that caveat stated rather than implied. |
||
|
Edouard Vanbelle
|
18649eb7b9 |
fix(thumbnails): drop derived-hash ETag until the read-order flip
The ETag is computed BEFORE the body. On a cache miss no content_derived_blobs row exists, so thumbnail_content_id returned the source-keyed form — then rendering created that row, and the next request resolved to the derived hash instead. The validator changed as a side effect of producing the body, making every first render immediately stale. Latent until aaf08532: before the consolidation, the on-demand render path never wrote a derived row, so the flip had nothing to trigger it. Fixing one gap exposed the other. Caught by thumbnail_etag_content_keyed.hurl — two consecutive GETs of an unchanged file stopped revalidating to 304. The plan already said derived-hash keying must land WITH the read-order flip and not before; I brought it forward anyway when the attachment case forced the attached half. This is the evidence for the constraint, so the plan now records the attempt and why it failed rather than leaving the note as untested caution. The attached lookup stays — it has no such window, since an upload writes its row synchronously before any read can observe it, and it fixes a real collision: a copy inherits the source hash, so an original and a copy carrying different uploaded previews would otherwise share one validator while serving different bytes. The flip removes the hazard for the derived half too: once that tier is authoritative it is populated before it is consulted, so no row can appear between two reads. |
||
|
Edouard Vanbelle
|
7ec387003d |
refactor(thumbnails): one persist_rendered for every render path
Step 10(a), the blocker. Four render paths each wrote the sidecar and
exactly one also recorded the content_derived_blobs row, so an on-demand
render — a cache miss, a size never generated, an evicted sidecar —
produced state the migration could never see. That breaks the
migration's premise rather than being untidy: thumb_derived_import would
never reach an empty tail, so the gate for deleting the sidecar would
never open.
Now every rendered thumbnail goes through persist_rendered, which owns
what persisting means. Raw `fs::write(&thumb_path, …)` drops from five
sites to two: the one inside persist_rendered, and
store_external_thumbnail's `ext-{file_id}.jpg`, which is file-keyed and
legitimately a different thing.
The path that matters most already had what it needed:
get_thumbnail_from_blob — the REST handler's fallthrough on a cache miss
— holds `dedup` and simply never used it for persistence. It now
dual-writes at no cost.
render_and_persist_all_webp had its own copy of the dual-write logic;
that copy is gone, so retiring the interim dual-write later is one edit
here rather than a hunt.
Two paths still pass `None` and remain sidecar-only: `get_thumbnail`
(renders from an on-disk original) and `generate_all_sizes_background`
(the path variant; the _from_blob sibling has dedup). Closing those
means threading a DedupService in from their callers. Left visible as an
explicit `None` at the call site rather than an absent write — the gap
is now something a reader trips over instead of something they have to
notice is missing.
Adds ThumbnailFormat::mime() beside ext(), since the derived row needs a
media type and an extension without a matching one is how a WebP ends up
labelled JPEG.
|
||
|
Edouard Vanbelle
|
395296a7e7 |
fix(storage): file_exists misreported every file as missing
`SELECT 1 FROM storage.files WHERE id = $1` decoded as i64. PostgreSQL types a bare `1` as int4, so the decode always failed — and since `.ok().flatten()` turns a decode error into the same None as "no row", file_exists reported false for every file. thumb_attached_import therefore classified every sidecar as an orphan and imported nothing. Caught by thumb_import_check.sh on its first run: the derived import restored its rows, the attached one restored none. Now `SELECT EXISTS(...)`, which yields a real bool and always returns exactly one row, so absence means absence. A query error still degrades to false — the safe direction, leaving the file on disk as a reported orphan rather than importing it against a row that may not exist. The failure mode is the point, and it is the third of this shape in two days: an error converted into an innocuous-looking outcome. So the check script now dumps a job's findings when an assertion fails. The jobs already recorded exactly why they skipped each file — the attached_sidecar_orphan findings naming the cause were sitting in the run while the script reported only "did not restore the row", which is indistinguishable from the job never having run. |
||
|
Edouard Vanbelle
|
4ae1531286 |
docs(plan): job-driven sidecar deletion, and the persist-consolidation blocker
Two revisions from working through step 10.
**Deletion moves into the import jobs, not a release.** Sidecars are
local disk, so a release cannot know whether every instance has drained
— gating on "an empty tail" asks an operator to coordinate a fact
nothing reports, and there is no telling when or whether they trigger
the jobs at all. Each job unlinking what it has imported makes every
instance drain itself. Constrained three ways: verify the derived blob
reads back before unlinking (a store that reported success but landed
unreadable would otherwise take the last copy), only after the
read-order flip (or the derived tier takes its first production traffic
by accident), and opt-in, since a migration that deletes by default is
surprising. Scheduled tick rather than boot trigger — idempotent and
resumable, so periodic is safe, while walking .thumbnails/ at startup
delays readiness for nothing.
**Found while checking the dual-write assumption: it does not hold.**
store_derived_blob has ONE call site; fs::write(&thumb_path, …) has
five. get_thumbnail, generate_and_persist and
generate_all_sizes_background all persist sidecar-only. That breaks the
migration's premise rather than being untidy — on-demand renders keep
producing un-migrated state after the import runs, so the tail never
empties and the deletion gate never opens. One persist_thumbnail owning
sidecar + derived + moka is therefore a prerequisite, and it makes "stop
writing sidecars" a later one-line change instead of four edits. Noted
that ThumbnailService holds no DedupService, so it must be threaded
through.
Also corrects a claim I put in thumb_derived_import's own docs:
transcoding is NOT a later step. ImageTranscodeService exists and caches
.transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it
must re-key file→content — legitimate only because a transcode is
derivable. Its .skip markers remain an open question.
|
||
|
Edouard Vanbelle
|
671e6ac0e7 |
test(api): end-to-end check of both sidecar import jobs
Nothing exercised these jobs. Their unit tests cover the directory walk
— which files each claims — but neither had ever executed a run.
The test environment always starts fresh, so there is no pre-migration
data to import. This creates it, and the reconstruction is EXACT rather
than an imitation: the on-disk layout did not change in this work. A
rendered thumbnail has always been written to {size}/{hash}.webp and an
uploaded preview to {size}/ext-{id}.jpg; the only new thing is the row.
So upload through the real API, then delete the row, and what remains on
disk is byte-for-byte what a pre-migration install has.
Deleting the row must also release the reference it held, or the
manufactured state would carry a reference no legacy install ever had
and storage_cleanup_check.sh would report a leak this script caused.
file_attached_blobs has an ON DELETE trigger for that;
content_derived_blobs does not — its Rust purge path releases explicitly
— so the strip decrements it directly.
Three assertions, in increasing order of what they catch:
1. Both rows come back, and the imported attached row carries the nil
uploader sentinel rather than a fabricated one.
2. A COPY inherits the imported preview. This is the user-visible
point and was impossible before the row existed: the ext- sidecar
is keyed by file_id, no copy path duplicates it, so the copy
silently fell back to a render.
3. Re-running imports nothing and changes no refcount. The likeliest
silent defect — store_attached_blob is ON CONFLICT DO UPDATE, so an
import that skipped its existence check would release and retake a
reference every run, invisible except as drift.
psql runs inside the compose container rather than depending on a host
binary, matching how spawn-db.sh probes readiness. Ordered before
storage_cleanup_check.sh, which deletes everything it needs.
|
||
|
Edouard Vanbelle
|
49e7bb15a6 |
test(storage): cover the sidecar walk for both import jobs
Both imports run over ONE directory, where the two legacy shapes sit side by side, so the property worth asserting spans them: together they must claim every real sidecar exactly once, and neither may take the other's. A job that drifted into the other's shape would content-key user-supplied bytes — sharing one user's uploaded preview onto every file with identical content — and no per-job test in isolation would notice. So the fixture is shared. `legacy_tree` builds a directory holding a content-keyed .webp pair, an ext- upload, and a stray README, and both test modules walk it: derived claims exactly the two hashes in sorted order, attached claims exactly the ext- file, the two sets are disjoint, and between them they account for all three real sidecars. `sidecar_names` became an associated function taking the root instead of reading `self`, which is what makes this testable at all — the walk is the half that decides which files a job claims, and it needed no pool to verify. Sorting is asserted rather than assumed, since the cursor resumes by skipping everything at or before it and a stable order is the only thing that makes that correct. A missing size directory is covered too: normal on a fresh install, and it must yield no work rather than abort the walk. Not covered here, and it needs a pooled fixture that does not exist: the round trip itself — store the blob, write the row, and confirm a COPY inherits the preview. That belongs in the API-level harness, where the legacy state can be manufactured through the real write path and then stripped. |
||
|
Edouard Vanbelle
|
7a2ebe0fdc |
feat(storage): thumb_attached_import — backfill uploaded previews
Twin of thumb_derived_import, for the other sidecar shape:
{thumbnails_root}/{size}/ext-{file_id}.jpg, the previews a user
uploaded — notably the SPA's client-side PDF generator, which has no
server-side render path at all.
Until a row exists, a copy of the file LOSES the preview: the sidecar is
keyed by file_id, no copy path duplicates it, and the server silently
falls back to rendering from the source, or to nothing for a PDF. That
is the bug file_attached_blobs closed for new uploads; this closes it
for everything already on disk.
Separate job rather than an arm of the derived import, because the
keying differs and that difference is the security boundary. These bytes
are not derivable from the file's content, so content-keying them would
share one user's uploaded preview onto every file with identical
content. Each job's name filter rejects the other's shape, and both
directions are under test.
Idempotence needs more care here than in the derived twin.
store_attached_blob is ON CONFLICT DO UPDATE, so calling it for an
existing row releases the previous reference and takes a new one —
harmless once, but a job doing it every run would churn refcounts. The
row is therefore checked first and the store reached only on a genuine
insert.
uploaded_by is the nil sentinel: disk records no uploader, and inventing
one — the file's created_by, say — would fabricate provenance that could
later read as evidence an Editor replaced someone's preview. The column
is NOT NULL with no FK precisely so provenance survives, and a sentinel
says "unknown" honestly.
Orphaned sidecars (no storage.files row) are counted and reported, not
deleted. This job imports; it does not reclaim. Existence is checked
explicitly rather than letting the foreign key reject the insert, so an
orphan is counted as one instead of surfacing as an opaque constraint
error.
|
||
|
Edouard Vanbelle
|
f80a28763e |
feat(storage): thumb_derived_import — backfill the derived tier from sidecars
First half of step 10. Every server-rendered thumbnail written before
content_derived_blobs existed lives only as
{thumbnails_root}/{size}/{hash}.webp — local-disk state that another
instance cannot see, a backend migration does not carry, and no
consistency job covers. This walks those files into the blob store and
records the mapping, so the derived tier can become authoritative and
the sidecar can be deleted.
A registered JobRegistry tenant rather than a script: the volume is
unbounded, so it needs a cursor, resume, cooperative cancel and run
history, and an operator needs somewhere to watch it. Cursor is
{size_dir}/{filename} over a sorted walk, which totally orders the
traversal.
Idempotent by construction — each file is skipped when a row already
exists, and store_derived_blob is ON CONFLICT DO NOTHING with
release-on-conflict beneath it, so re-runs cannot inflate refcounts.
Re-running is the expected operator behaviour, since Phase 3 (deleting
the sidecars) is gated on a run reporting zero imported.
hash_from_sidecar_name deliberately rejects ext-{file_id}.jpg. Those
bytes are user-supplied and file-keyed; importing them here would
content-key them and share one user's uploaded preview onto every file
with identical content. They belong to thumb_attached_import. Both the
accept and the reject set are under test.
Unreadable files and store failures record a finding and continue: a
sidecar removed by a concurrent GC unlink between listing and read is
expected, not fatal, and the file is left in place for the next run.
Registered unconditionally rather than behind a flag — a migration
nobody can find is a migration nobody runs.
|
||
|
Edouard Vanbelle
|
95648f2fa3 |
fix(thumbnails): private, no-cache — the URL is gated and mutable
Thumbnails were served `public, max-age=31536000, immutable`. Two problems, and the first is a security one. `public` on a Permission::Read gated resource lets any shared cache — a corporate proxy, a CDN — store one user's thumbnail and serve it to another. `Vary: Accept` was no defence: it does not vary on Authorization. Now `private`. `immutable` was a promise this URL cannot keep. It is keyed by file id, and its bytes change when a preview is uploaded, when content is replaced, or when an attachment is removed. `immutable` tells a client not to revalidate at all during the freshness lifetime, so with a one-year max-age a browser that fetched once would never see a new preview — which also made the content-keyed ETag unobservable in practice. A correct validator is worthless if nothing asks. Now `no-cache`, which still stores the body and only requires revalidation, answered by the ETag with a body-less 304. The hurl tests could not have caught this: hurl always sends the request, so If-None-Match was exercised and passed while a browser obeying `immutable` never got that far. Same "correct on the wire, wrong in practice" shape as the bugs before it, so the test now asserts the directives themselves rather than only the 304 behaviour. One definition, shared by the REST and NextCloud endpoints, which are gated identically and must not drift. /_app/immutable is untouched: those are hash-named static assets, genuinely content-addressed and public, where the directive is honest. Cost is a conditional request per thumbnail per page load. Recovering it needs a content-addressed URL — where `immutable` would be true — but that puts the hash in the URL of an authorized resource, so it stays `private` regardless, and it touches the SPA and the file DTO. Separate change. |
||
|
Edouard Vanbelle
|
7d9418f63c |
fix(thumbnails): ETag names the blob actually served
fe9c4f49 keyed the ETag on the SOURCE file's content hash. That is wrong
whenever the response comes from a satellite table, and for attachments
it is wrong in two ways.
Uploading a preview does not change the file's content, so a
source-keyed ETag does not change either — and with `immutable` set,
clients never revalidate and keep the previous render for up to a year.
The exact staleness fe9c4f49 set out to fix, re-entering through the
attachment path.
Worse: a copy inherits the source hash, so an original and a copy have
identical ETags. Give either one a different uploaded preview and they
serve different bytes under one validator, which a shared cache may hand
to either request. That is a collision, not just staleness.
thumbnail_content_id resolves the identity through the same tier
precedence the read path uses: an attached blob's own hash, else a
derived blob's own hash, else the source-keyed form. An ETag naming a
different tier than the one answering is worse than a coarse one, so the
two orders must not drift.
Derived-hash keying is strictly better than source-keying and never
worse. The sidecar and the derived row are written from the same bytes;
where they can diverge — a sidecar re-rendered while the derived row
stays pinned by ON CONFLICT DO NOTHING — source-keying is wrong too,
because the renderer is not part of that key. This is the step 10 change
arriving early, forced by the attachment case; the plan note stands for
the read-order flip itself.
Known gap: a legacy ext-{file_id}.jpg with no file_attached_blobs row
yet falls through to the source-keyed form. No worse than today, and it
resolves when the import backfills.
attached_thumbnail_copy.hurl now asserts ETags, which is why this went
unnoticed: it compared bytes only, and thumbnail_etag_content_keyed
covers content replacement rather than preview upload. A fresh GET
returned the right bytes throughout — the same "healthy locally, broken
for anyone caching" shape as the two bugs before it.
|
||
|
Edouard Vanbelle
|
d71dd973e7 |
fix(thumbnails): two defects the copy test exposed
1. Per-file overrides must beat the content tier in RAM as well as on
disk. The content-keyed lookup ran first, so a thumbnail already
rendered from the file's content sat in RAM under content(blob_hash)
and shadowed a preview uploaded afterwards — permanently. Invisible
before the moka rekey, because both lived under one file-id key and
the upload simply overwrote the render. Order is now uniform:
per-file RAM, per-file disk (ext-), per-file DB, then content RAM,
blob-hash disk, derived blob.
2. store_attached_blob never wrote a row. Its RETURNING clause compared
the stored hash against EXCLUDED, and PostgreSQL only permits
EXCLUDED in the SET and WHERE of DO UPDATE — a runtime syntax error
on every call. The superseded hash now comes from a SELECT taken
before the upsert; losing that race leaves one stale reference, which
the manifest recompute reports, rather than anything being lost.
The second hid behind the first for a whole cycle, and behind
`ext-{file_id}.jpg`: the ORIGINAL kept serving its uploaded preview from
local disk, so the feature looked healthy. Only a copy, which has a
different file_id and therefore no ext- file, depends on the row — and
the row was never there. The handler's best-effort warn! completed the
disguise, so it is now error!: a failure there means copies silently
lose the preview, and nothing else signals it.
|
||
|
Edouard Vanbelle
|
6c5e53fee4 |
test(api): assert the blob registry is empty, not just the disk
The disk check proves no BYTES are left. This proves no ROWS are, which fails differently and worse: a stale storage.blobs row with nothing behind it means a reference was never released, and dedup_gc will skip it forever because its count never reaches zero. Silent, permanent, and invisible to a check that only looks at the filesystem. Zero is the right assertion, not "fewer than before". By this point the suite has deleted its users, their drives and everything cascading beneath, and the disk check has already insisted the blob store is empty. A non-zero registry beside an empty disk is exactly the divergence the consistency jobs report — caught here first because one number is easier to read than a findings list. Degrades to a warning if the endpoint is unavailable rather than failing, so a build without the admin dedup surface still runs the rest. |
||
|
Edouard Vanbelle
|
8881979761 |
test(api): assert the consistency jobs are clean after the whole suite
The disk checks above prove nothing leaked. These prove the bookkeeping behind them is honest: every refcount matches what the reference sources hold, and no row points at bytes that are gone. End of suite is the only place this is cheap. One database serves every hurl file, so by here the counters have absorbed every upload, copy, move, share, trash and purge the suite performed — across both copy paths, the derived tier and the attached tier. Drift that no individual test would notice, because each only inspects its own file, surfaces as a mismatch. Runs after the GC drain deliberately: mid-sweep state is legitimately inconsistent — a manifest can sit at zero waiting for the next pass — so checking earlier would report normal in-flight state as drift. Zero findings is the assertion. These four tenants are read-only, so anything they report is a real invariant violation rather than a repair opportunity. A job missing from the build is skipped with a warning instead of failing, so this does not break on a feature-gated build. Unknown job names and unwrapped-vs-wrapped response shapes both degrade to a visible warning rather than a silent pass: list_job_runs currently returns a bare array, and the .runs/.items fallbacks exist so a future wrapping does not quietly turn the whole check into a no-op. |
||
|
Edouard Vanbelle
|
64ff982571 |
feat(thumbnails): uploaded previews survive a copy
Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else —
keyed by file id, on local disk. No copy path duplicates it and no other
instance can see it, so a copied file lost the preview its owner
uploaded. Silently: the server falls back to rendering one from the
source, or to 204 for a PDF, which has no render path at all. A
user-supplied preview is not derivable from the content, so once lost it
is gone.
The PUT now also records a storage.file_attached_blobs row, which
copy_file_satellites already duplicates, so both copy paths carry it.
Best-effort: the sidecar has already succeeded by then and the user can
see their thumbnail, so failing the request would report an error for an
operation that visibly worked.
Read path consults attachments ahead of every content-derived tier: an
uploaded preview is an explicit choice about THIS file and must beat
anything rendered from its content. Cached under the per-file key — a
content key would leak those bytes to every other file sharing the
content, which is the poisoning the file-keyed table exists to prevent.
store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin:
re-uploading a preview is a deliberate replacement, where a re-derived
thumbnail is the same bytes again. The superseded blob's reference is
released, or it would be pinned forever with nothing pointing at it.
Deletion goes through a trigger, not a hook. file_id is ON DELETE
CASCADE, and on_file_deleted fires AFTER delete_file — by then the
cascade has run and there is nothing left to enumerate. This matters
most for folder deletion, where PG cascades folders to files to
attachments and Rust never sees the rows at all. storage.decrement_blob_ref
keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused
verbatim rather than transcribed into a second trigger that can drift.
DELETE only: a replacement updates in place and is handled in Rust, so
adding UPDATE would double-decrement.
Extracted read_blob_to_bytes, shared by the attached and derived tiers —
the only difference between them is which table produced the hash.
tests/api/attached_thumbnail_copy.hurl guards it. The file is red and
the uploaded thumbnail is green, so a render could never produce the
uploaded bytes; the pre-upload render is captured first and required to
change, which stops three identical renders from satisfying the
byte-equality. Then both copy paths must serve the upload, and after the
original is purged and GC runs, both copies must still serve it — each
holds its own reference, because the rows are duplicated rather than
shared.
|
||
|
Edouard Vanbelle
|
fac82fea23 |
feat(storage): add storage.file_attached_blobs, the file-keyed half
Step 9 of docs/plan/derived-blobs.md. content_derived_blobs holds bytes that are a pure function of a file's content, so they are keyed by that content and shared by every file holding it. This table holds the opposite: bytes a user supplied or chose, which must never be shared across files. The key is what enforces it. That difference is a security boundary, not a modelling preference. A content-keyed client preview would let user A upload a file plus a preview that misrepresents it; when user B later uploads the same bytes, dedup matches and B is served A's preview. Content-keying is only safe when the server can derive the bytes — there is nothing to poison, because the same input yields the same output for everyone. Required now rather than deferred: the SPA already generates and PUTs previews for PDFs, and there is no server-side regeneration path for them, so the sidecar migration has nowhere else to put those bytes. uploaded_by is NOT NULL with no foreign key, per the provenance convention rather than the plan's sketch. A FK with ON DELETE SET NULL discards the audit trail exactly when it matters, and without an ON DELETE clause it would block deleting a user outright. Deleting the uploader must not rewrite history. FileAttachedReferenceSource is registered in built_in_registry before anything writes to the table, so dedup_gc's reap predicate already knows it exists — otherwise the first sweep after the first attachment would delete it. Manifest level only, like the derived source: these blobs are almost always single-chunk, so contributing at chunk level would double-count against the aliased hash. copy_file_satellites gains one arm: attachments are DUPLICATED, since the key is file_id and the copy is a different file, with uploaded_by carried over — the person who supplied the bytes did not change because someone copied the file. Each duplicate takes its own reference, so the bytes stay deduplicated while the mapping does not. Both golden SQL tests updated: the new fragment lands inside the reap predicate's NOT(...) group and as a summed term in the manifest recompute. Verified on a scratch PG with every migration applied — the attachment duplicates to 2 rows holding 2 references with provenance intact, while the content-keyed thumbnail stays 1 row reachable from both files. |
||
|
Edouard Vanbelle
|
cf21ee2f8f |
fix(thumbnails): key the moka tier on content, closing the ETag race
fe9c4f49 made the ETag content-keyed, but the RAM tier was still keyed on
file_id, so the two disagreed about what identifies a thumbnail. Replacing
a file's content preserves its id, so the moka entry stayed reachable while
the ETag had already changed — and invalidation runs from the spawned task
in on_file_updated. A request landing in that window got the NEW ETag over
the OLD bytes, and because the response is immutable with a one-year
max-age, the client cached those stale bytes permanently. The bug fe9c4f49
set out to fix, arriving through a different door.
Keying on the hash removes the window rather than narrowing it: new content
is a different key, so the old entry cannot be hit. Correctness no longer
depends on the invalidation task winning a race against the next request.
This also aligns the RAM tier with what disk already did — sidecars have
always been written to get_thumbnail_path(blob_hash, ...). The tier that
had the bug was the one keyed differently from every other. Two further
consequences: N copies of one photo now share a single entry instead of
occupying N for identical bytes, and delete_thumbnails shrinks to the
external entries, which are the only genuinely per-file artifacts.
Video frames stay file-keyed under an `ext-{file_id}` id — they are
per-file by nature. The namespaces cannot collide: hashes are 64 hex
characters.
get_cached_thumbnail takes blob_hash as an Option, and a caller without one
now skips the RAM tier and falls through to disk rather than consulting a
file-id key. That is correct, not merely tolerable — a file-id key is the
stale entry this change exists to prevent. Both HTTP handlers resolve the
hash to build the ETag, so only internal callers that never had one are
affected.
|
||
|
Edouard Vanbelle
|
2c0ba37290 |
test(api): read the DTO from the folder listing, not the download route
GET /api/files/{id} is the download route — it returned the PNG bytes,
so the jsonpath capture failed on a UTF-8 decode. /{id}/metadata is the
EXIF endpoint and carries no FileDto either. Listing the folder gives
the DTO, and since the folder holds exactly this one file, count == 1
also proves the WebDAV PUT overwrote in place rather than creating a
second file beside it.
Also drops an unused bytes capture and records why the body is not
asserted after the overwrite: the moka tier is keyed on file_id and
invalidated from the spawned task in on_file_updated, so a request
landing first sees the previous bytes under the new ETag. Asserting on
bytes would be a race.
|
||
|
Edouard Vanbelle
|
19a8186c66 |
test(api): upload into an explicit folder in the ETag test
The upload omitted folder_id, which the handler needs to resolve the file's owner — it answers 500, not a root upload. Every other upload in the suite passes it; this was the only one that did not, which is why nothing caught it earlier. The folder also gives the WebDAV overwrite a deterministic path (/webdav/hurl-etag-src/<name>) instead of depending on where a folder-less upload would have landed. Teardown now removes it and purges it from trash, keeping the shared database clean for the files that run after. |
||
|
Edouard Vanbelle
|
ec61ce77f8 |
docs(plan): ETag moves to the derived hash at the read-order flip
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format), which is one term short: a thumbnail is a function of those PLUS the renderer. Change the encoder or a quality setting and identical inputs produce different bytes under an unchanged ETag — the same staleness class the commit fixed, one level down. It bites when an already-cached thumbnail is re-rendered after a renderer change. Keying on the derived blob's own hash removes the term entirely: the ETag IS the hash of the bytes, so any output change invalidates by construction. It is self-consistent for free, because store_derived_blob is ON CONFLICT DO NOTHING — a re-render never displaces the stored row, so the ETag always equals what the derived tier will serve. No renderer version constant to remember to bump. Records why it cannot land yet. The derived tier is read LAST by design, so an ETag naming the derived hash would describe a tier the response probably did not come from; sidecar and derived agree at creation but diverge if a sidecar is re-rendered while the derived row stays pinned by DO NOTHING. An ETag that lies about the body is worse than one that is merely coarse. Also the tier is WebP-only (variant is the size, with no format term) and empty for anything predating this work until derived_import backfills. So it lands at step 10 with the flip, keeping today's form as the fallback for ungenerated variants and formats the tier does not hold. The LEFT JOIN already planned for the read path returns the derived hash in the same query, so it costs no extra round-trip. |
||
|
Edouard Vanbelle
|
a3a93b90ec |
fix(thumbnails): key the ETag on content hash, not file id
The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with
Cache-Control: public, max-age=31536000, immutable. Replacing a file's
content preserves its id — file_upload_service rebuilds the entity with
parts.id and a new hash, then fires on_file_updated, which deletes and
regenerates the thumbnails — so the server produced a new thumbnail while
still advertising the old ETag. Because `immutable` tells a conforming
browser not to revalidate at all inside the freshness window, clients kept
rendering the previous image for up to a year, unfixably.
Keyed on the content hash the directive becomes honest: a thumbnail is a
pure function of (source bytes, size, format), so that triple identifies
the response. New content yields a new ETag.
The same change fixes the opposite direction. A copy, or any dedup twin,
had a different id and therefore a different ETag, so clients refetched
bytes they already held even though both are served from the same derived
blob. Now identical content agrees on an ETag and revalidates to 304
across files, users and copies.
Both thumbnail endpoints were affected: the REST handler and the
NextCloud preview handler.
Cost is one PK lookup ahead of the 304 decision, where the id-keyed
version needed none — paid for by no longer serving stale images. It is
partly recovered: both handlers already resolved the same hash further
down for the render path, and that second lookup is now gone, so the
cache-miss path is unchanged and only the 304 path pays. The resolved
hash is also handed to get_cached_thumbnail instead of None, saving the
service its own lookup.
No new disclosure: content_hash is already on FileDto and returned by
GET /api/files/{id}.
Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite
in place via WebDAV PUT, assert the ETag changed, assert a client holding
the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the
sharing direction: a copy answers with the SAME ETag and revalidates to
304, which is the one externally observable consequence of content-keying
and was not previously testable.
|
||
|
Edouard Vanbelle
|
46dc25a9a8 |
test(api): scope derived_blob_copy claims to what it can observe
The byte-identity assertions were documented as proving that a copy shares the original's content_derived_blobs row. They prove no such thing: rendering is deterministic in the source bytes and the variant, so a copy that re-rendered from scratch returns identical bytes. The copy is in fact a moka hit — that cache is keyed on (source_hash, size, format), which the copy shares — so it never reaches the derived tier here at all. Nor is there an assertion that would fix it. Duplication is impossible by construction: the PK is (source_hash, kind, variant), a copy carries the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING. The schema enforces the property, so no runtime behaviour can violate it and there is nothing to catch. Same limitation narrows step 11: it proves the SOURCE content survived GC, not the derived blob — a reaped derived blob is re-rendered transparently from the live source. What the file does prove is unchanged and is the part that was broken: both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and purging the original does not destroy the copies. No assertions changed. |
||
|
Edouard Vanbelle
|
9f8ec141f3 |
feat(storage): single-source the copy fan-out via copy_file_satellites
Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was written twice — the copy_file CTE and storage.copy_folder_tree — and had already drifted: the tree path bumped storage.blobs only, missing manifests, which was silent data loss on any multi-chunk file. Fixing it meant writing the same logic a second time. Step 9 adds a file-keyed satellite table, which would mean a third and fourth. Two SQL functions: storage.add_blob_references(TEXT[]) — the manifest-first reference contract for SQL callers, returning hashes that matched no registry row. Set-based so the tree path keeps its single-statement cost; a per-row helper would have made a 10k-file copy 10k calls. storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus the blob reference. The body is the copy-semantics declaration: what is absent (comments, favorites, content-keyed derived rows) is listed with its reason, so the taxonomy is executable rather than documented elsewhere and drifting. Both copy paths now call it. The single-file path becomes a real transaction, which also fixes the reference being best-effort: a failed add_reference used to log a warning and leave a copy holding no reference at all — the exact shape that gets its content reaped. It cannot be a CTE arm, because data-modifying CTEs share one snapshot and the function must read the row the INSERT just wrote. Verified against a scratch PG with all migrations applied: multi-chunk manifest 1→2, single-chunk alias bumped at manifest level only (the NOT EXISTS guard), chunks behind a manifest untouched, dead properties duplicated, length mismatch rejected, repeats counted. tests/api/derived_blob_copy.hurl covers it end-to-end and answers the question the copy raises: content_derived_blobs is NOT copied. A copy carries the same blob_hash, so it resolves the same derived row — the test asserts byte-identical thumbnails from both copy paths, then deletes the original, runs GC, and requires both copies to still serve. That last step only passes if the references are real. |
||
|
Edouard Vanbelle
|
7261b5b175 |
fix(s3): never return a mangled enumeration cursor
5343fdda switched S3 blob enumeration from an opaque continuation token to a hash cursor (StartAfter), per the port contract. Its fallback for a page containing no canonical blob was wrong: it stored the full key (`0a/junk.tmp`), stripped it to a basename (`junk.tmp`), and the next call fed that to `object_key()` — producing `ju/junk.tmp.blob`. Wrong shard and a doubled extension, so the resume jumped to an arbitrary position: skipped objects, or backwards into a loop. The cursor can only ever be a real hash, because `object_key()` is applied to it. So instead of synthesising one, keep listing internally until the page holds at least one blob or the bucket is exhausted. The continuation token is used only inside the call and never escapes. Two pathological cases cannot produce a cursor at all — `is_truncated` with no token (protocol violation), and a run of foreign keys long enough to buffer the bucket. Both now fail loudly. A visible job failure beats a sweep reporting "no missing blobs" having read a fraction of them. Extract `hash_from_object_key` as the paired inverse of `object_key`, with the round-trip and the rejection set under test. It also now requires the shard to match the hash's own prefix, which the inline filter did not check. |
||
|
Edouard Vanbelle
|
7f5ee7401f |
refactor(storage): make blob enumeration ordered and hash-cursored
Precondition for the merge-join in backend_consistency (step 6 / option A of docs/plan/derived-blobs.md), landed separately because it is independently useful and carries the risk. Two contract changes on BlobStorageBackend::list_blob_hashes: 1. Entries MUST be in ascending hash order. Every shipped backend already did this — local sorts within each shard and walks 00..ff, and since the shard IS the hash prefix that is globally sorted; S3 and Azure list lexicographically by key and blobs/<xx>/<hash> sorts identically to <hash>. It was accidental, and a future backend enumerating in any other order would have silently made the merge-join emit bogus blob_missing_from_backend findings at data_loss severity. 2. The cursor is the last hash returned, not an opaque backend token. This is what lets a caller resume from a checkpoint it already holds — the merge-join keeps one cursor for both the DB walk and the backend walk instead of a compound one, which in turn means blobs_consistency's existing cursor format survives and no paused run is stranded. Local already derived its position from a hash; it now emits the bare hash instead of "<shard>/<hash>", and still accepts both legacy forms so a run paused across this deploy resumes. The bare-shard form works through the same path unchanged, since "3f" sorts before every 64-char hash beginning "3f". S3 moves from continuation_token to StartAfter, which supports this natively. One non-obvious case handled: a page can contain only non-canonical keys (.tmp spool files, .corrupt sidecars), which are filtered into `unknowns`, leaving `blobs` empty — a naive blobs.last() would return no cursor and silently end enumeration while is_truncated said otherwise, making an audit job under-report. It now falls back to the last key seen; StartAfter is a string comparison, so a non-hash resume point is fine. "Cursor is a hash" constrains what callers may synthesise, not what backends may return. Azure is unaffected — it does not implement list_blob_hashes (TODO, inherits the NotSupported default). Adds the first test for enumeration at all: ordering across shards with deliberately out-of-order inserts, complete paged traversal, and resume from a caller-synthesised cursor. NOT verified against real S3 — no bucket available here. The local path is covered by the new test; the StartAfter change is reasoned from the API contract and needs exercising against a real bucket before it is relied on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Edouard Vanbelle
|
a8223cab65 |
test(storage-check): drain the GC cascade instead of one pass
One dedup_gc pass cannot fully drain now that thumbnails are derived blobs. Reaping a source releases the references its derived artifacts hold (each content_derived_blobs row pins a manifest), and those releases happen mid-sweep — the derived chunks are stamped orphaned as the pass is already walking past them, because remove_manifest_reference deliberately does not unlink, to avoid racing a concurrent upload re-referencing the same chunk. They are collectible only on the NEXT sweep, which is why the check saw 15 leftover blobs. Loops until a pass reclaims nothing rather than hardcoding two. Two is correct only while the derivation graph is one level deep — a thumbnail is derived from a file, nothing is derived from a thumbnail. That is a property of the data, not an invariant the code enforces, so a fixed count would silently under-drain the day transcodes-of-thumbnails or E2E-wrapped derivatives exist, and the failure would surface as a confusing leftover-file assertion rather than the design change it is. Bounded at 3 with a warning if it does not settle. Sleeps between passes. The JobRegistry serialises runs of the same job, so a back-to-back trigger risks rejection as already-running — which returns 0 reaped and would exit the loop early, declaring success with blobs still on disk. A false pass is worse than a slow one. It also gives the previous pass's detached unlink tasks (spawned by on_blob_deleted, awaited by nothing) time to land. Deliberately NOT fixed in production code: derived chunks land inside the 1-hour orphan grace, so a second immediate sweep would collect nothing there and the next scheduled run picks them up. A fixpoint loop in garbage_collect would be dead code outside force=true, which is only this test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Edouard Vanbelle
|
c276d3861e |
fix(thumbnails): release derived blobs when their source is reaped
Caught by the api-test storage check: 15 blob files left on disk after a full cleanup. Since 3736b577 thumbnails are stored as derived blobs, and each content_derived_blobs row holds a manifest reference — but nothing ever deleted those rows, so the reference outlived the source and GC could never reclaim the bytes. The plan specifies this cascade; I implemented the write and read paths and missed it. Adds `purge_derived_blobs`, the delete counterpart of `store_derived_blob`: deletes every row derived from a source hash and releases the reference each held. It lives on DedupService alongside its store/find siblings because ThumbnailService cannot hold a DedupService — it implements BlobLifecycleHook, and holding one would close the DedupService -> BlobLifecycleService -> hook -> DedupService cycle the existing comment warns about. All five reap sites now go through `reap_blob`, which purges then fires the lifecycle hooks, so no path can drop a blob without first releasing what was derived from it. Previously each site called fire_blob_hooks directly, which only cleaned the sidecar files ThumbnailService owns. `reap_blob` is boxed because it is mutually recursive with `remove_reference`: releasing a thumbnail's reference can reap the thumbnail's own blob, which re-enters here. It terminates after one level — nothing is derived from a thumbnail, so the inner purge finds no rows. That bound is a property of the data, not an invariant the code enforces, so it is stated at the definition. fmt, clippy --all-features --all-targets, unit tests clean. The api-test storage check is the real verdict — it is what found this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Edouard Vanbelle
|
60b94e1183 |
feat(thumbnails): serve derived blobs when the sidecar cannot
Step 5, read path — Option 2 of the two shapes discussed: the derived
blob is consulted LAST, after the sidecar, not first.
Read order is now
moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob
For every thumbnail already on disk the new branch is never reached, so
the database stays off the hot path and a fault in it cannot break a
working gallery. It answers only what disk cannot: a thumbnail rendered
by another instance, or a box whose sidecar was never populated. Legacy
content keeps serving from disk until `derived_import` migrates it.
That inverts the plan's stated order deliberately. Derived-blob-first is
right for the END state, because it is what lets the sidecar be deleted;
sidecar-first is right transitionally, because the risky reordering
should happen after the table has been seen serving real reads. The flip
belongs in the release that removes the sidecar, and the comment at the
branch says so.
The existing precedence is preserved and now documented: the file-keyed
client upload (ext-) is checked BEFORE the content-keyed server render.
That ordering is a security property, not a preference — content-keyed
artifacts are shared across every file with that content, so checking
the file-keyed one first is what keeps one user's uploaded preview from
ever being served for another user's identical file.
Shape notes:
* `find_derived_blob` lands on DedupPort/DedupService as the read
counterpart of `store_derived_blob`, so ThumbnailService needs no pool
field — and therefore ThumbnailService::new, DI and three tests are
untouched.
* It carries `content_type`, which is what will retire the byte-sniffing
in the handlers once reads are table-primary.
* The parameter is `Option<&DedupService>`, concrete rather than
`&dyn DedupPort`: DedupPort uses native `async fn` and so is not
dyn-compatible, and ThumbnailPort is never used as a trait object
(checked) — both handlers hold the concrete Arc. `None` means
sidecar-only, which is exactly today's behaviour and what the abstract
port impl passes.
fmt, clippy --all-features --all-targets, 35 unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Edouard Vanbelle
|
1c488b7df5 |
feat(thumbnails): also store derived thumbnails as blobs
Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.
That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.
DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:
* writes the bytes through the normal CDC path, so derived blobs
inherit the backend, encryption, migration and key rotation that
source content already gets;
* records (source_hash, kind, variant) -> blob_hash;
* releases the reference store_from_stream took IF the mapping
already existed. Two instances racing to render the same thumbnail
must leave ref_count at 1, not 2 — otherwise every re-render
inflates it and pins the blob forever.
ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.
The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.
Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.
fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Edouard Vanbelle
|
a7938344dd |
feat(storage): add content_derived_blobs table + reference source
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Edouard Vanbelle
|
a7b7045ed2 |
docs(plan): record the blobs_consistency -> chunks_consistency rename
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Dionisio Pozo
|
76e5a8b458 |
Merge pull request #695 from EdouardVanbelle/ci/bundled-binaries
ci: bundled binaries |
||
|
Edouard Vanbelle
|
724e6bec36 |
fix(release-binaries): drop x86_64-apple-darwin target
macos-13 runner tier is being phased out by GitHub — queues persistently exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to 'cargo install --features bundled-assets' from source, Docker --platform linux/amd64, or a Linux VM. |
||
|
Edouard Vanbelle
|
c5dfb389f2 |
fix(release-binaries): drop alpine container, use musl-tools natively
JS-based GH Actions (checkout, artifact steps, setup-node) can't run inside Alpine on ARM64 — Node.js binary shipped by the actions requires glibc, and the x64-Alpine workaround doesn't extend to arm64. Cross-compile natively via 'rustup target add' + musl-tools instead. |
||
|
Edouard Vanbelle
|
d69c985921 |
test(bundled-binary): surface actual response on locale-JSON failure
The 'broken pipe' error under set -euo pipefail was masking the real answer. Capture status + content-type + first-char in one curl, dump first 200 bytes of body on failure so we can see what the server actually served. |
||
|
Edouard Vanbelle
|
807a0efd27 |
docs: OXICLOUD_ENABLE_VIDEO_THUMBNAILS + bundled-binary design record
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in example.env and docs/config/env.md — closes the discoverability gap where the env var was only visible in Rust docstrings. Also lands docs/plan/bundled-binary.md — the design record referenced from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs, and the Dockerfile. |
||
|
Edouard Vanbelle
|
fac4214856 |
feat(bundled-binary): release-binaries.yml + install docs + binstall metadata
Adds the tag-triggered workflow that builds 4 musl-linux + macOS tarballs and attaches them to the tag's GitHub Release. Ships a matching install guide (docs/install/binary.md) with SHA256SUMS verify, systemd unit, upgrade flow, and hardware notes. Adds [package.metadata.binstall] so 'cargo binstall oxicloud' works automatically once the first release lands. Also re-enables incremental compilation in the dev profile — the 'modest single-crate savings' rationale from when the crate was small has been outgrown; full rebuild ~10 min is now the dev-loop bottleneck. |
||
|
Edouard Vanbelle
|
5c956c9bde | feat(bundled-binary): add a test | ||
|
Edouard Vanbelle
|
e562a3de30 | feat(bundled-binary): include /static-dist into binary |