The public landing page's inline media preview (added in 6ee26e46)
requests /api/s/{token}/file/{item_id}, but assert_file_in_share went
through resolve_folder_share, which hard-rejects non-folder shares —
so for a single-file share the video src got a 400 and the player
rendered empty: the preview box appeared but nothing would play.
The AuthZ gate now branches on item_type instead: a file share only
accepts file_id == share.item_id, a folder share still requires the
file to live in the shared subtree, and anything else is NotFound
(same shape as "file doesn't exist", preserving anti-enumeration).
Password/expiry checks still happen inside
get_shared_link_with_unlock, unchanged.
Also extend public_shares.hurl section 8b: the file-share token must
stream its own item (200 + inline disposition) and reject an outsider
file id with 404.
NOTE: fmt/clippy/api-test could not run on the authoring machine (no
Rust toolchain or Docker) — run `just check` + `just api-test` before
pushing.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- normalize username into lowercase (this is already ASCII only)
- permit users to login with their username with insensitive case
- if a disabled account is reactivated and got a collision, it will normalize it too
- server will stop on collision (ex: 2 entries with `Alice` and `alice`)
in a such case admin can run:
```
oxicloud migrate lowercase-usernames --dry-run
```
then
```
oxicloud migrate lowercase-usernames
```
- do not increment ref_count if new attachement to a file with same data
- add audit log to help identifying other future issue in ref_count
- prevent race condition while attaching a blob
- server now provide it's config via /api/config (possibility to feature flag)
- client use /api/config to enable / disable some features
- capability to disable the message bus, somme OPS may not want this feature and
consume persistent connections from server (websocket):
OXICLOUD_MESSAGEBUS_ENABLE (true by default)
Two fixes from Ed's run.
## The assertion I called load-bearing was measuring nothing
`backend_migration` is a DETACHED job: the trigger spawns the handler
and returns 202 in milliseconds, carrying no outcome and no run_id.
I had modelled it on `admin_jobs.hurl`, where the jobs are synchronous
and the response IS the outcome.
So `duration < 120000` on the trigger would have passed against the
ORIGINAL unbounded behaviour — it timed the dispatch, not the
migration. The one assert the file existed for proved nothing.
The bound is now a polling budget: `/runs?limit=1` with `retry: 60`,
`retry-interval: 2000`. 120s, then hurl fails on the last assert.
Against a 15-minute hang the row sits in `Running` and the budget
exhausts, which is the failure this file is for. `run_id` comes from
`$[0].id` (runs are `ORDER BY started_at DESC`), since the 202 body
has none.
## A count assert on a registry, again
`storage_multi_entry.hurl` asserted `$.entries count == 3` and
`s3_blackhole` made it 4. The failure reads "expected 3, got 4",
naming neither the entry that appeared nor whether it belonged.
Replaced with per-name `contains`, which is what a registry wants:
membership asserted per item, so declaring a new entry does not break
an unrelated file. Positional asserts stay — entry ORDER is a separate
property and a real one, since the boot fallback picks `[0]` when no
active pointer exists.
Its comment also said "Two entries declared" while asserting three:
the drift a count invites, visible in the same three lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regression test `docs/plan/jobs-handling-recoverable-error.md`
§Testing asks for: assert the run reaches Paused, that `error_message`
names the cause, and that it does so in bounded time rather than
hanging.
## The endpoint has to HANG, not refuse
`s3_stub` already existed and points at `127.0.0.1:9999`, where nothing
listens. That connection is REFUSED — ECONNREFUSED, immediately — and
that path was never broken. A test built on it would pass with no
timeout configured anywhere, which is worse than no test: it would read
as coverage of exactly the failure it cannot see.
So `s3_blackhole` points at `192.0.2.1`, TEST-NET-1 (RFC 5737),
reserved for documentation and guaranteed unrouted. A SYN goes
unanswered — no RST, no ICMP — which is the failure that used to hang
until the OS abandoned TCP retransmission ~15 minutes later, with the
job neither running nor failed the whole time.
Ed's suggestion, and it is the right fixture: a server that never
answers is reproducible in a way that unplugging a cable is not.
## The load-bearing assertion is `duration`
Every other assert in the file would also pass against the old hanging
behaviour, given fifteen minutes. `duration < 120000` is the only one
that fails if the bound is ever removed. The threshold is deliberately
loose — three orders of magnitude from the failure it guards, so a slow
runner cannot make it flaky.
## Why it is safe in the shared suite
The run fails at `target.initialize()`, which is BEFORE
`migration_readonly` is engaged, so this file cannot leave the server
read-only for whatever runs next. A mid-copy failure would have held
the freeze — that is why this shape was chosen.
Teardown is mandatory rather than tidy: `open_or_start` picks up the
latest non-terminal row, so a Paused row left behind would be RESUMED
by the next `backend_migration` trigger in the suite, silently
retargeting an unrelated test at the black hole. The file cancels its
own run and asserts the row reached Cancelled.
Placed second-to-last. It is the slowest file in the suite by design —
it waits out an unreachable endpoint to prove the wait is bounded — so
that cost lands after everything else has reported. Azurite stays last
for the reason its own comment gives.
Not yet executed: the suite tears down containers and Ed usually has a
run in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An e2e run emitted 81 × 429 in 763 log lines. The env already set
LOGIN/REGISTER/REFRESH to 36000/hour, and that changed nothing, because
those three are the only rate limiters with env vars — and they are the
wrong ones. They key on the client IP and guard the unauthenticated
front door. The limiters that fired key on the CALLER ID.
The log distinguishes them: all 81 landed on target `http::api`, never
`http::api::auth`, where login/register/refresh live.
The likely culprit is `user_profile_rate_limiter`, 60 lookups/min/caller,
guarding the visibility query behind GET /api/users/{id}. The whole
suite runs as a single `admin`, so every test shares one bucket; admin
views resolve an owner name per row and the run creates 34 users, so a
minute of tests clears 60 easily. Nothing failed, because the SPA
degrades to an unresolved name — which is exactly the problem, since
that noise would hide a real rate-limit regression.
Adds OXICLOUD_RATE_LIMIT_USER_PROFILE_MAX / _WINDOW_SECS and
OXICLOUD_RATE_LIMIT_DELTA_UPLOAD_MAX / _WINDOW_SECS, following the
existing three exactly. Defaults are the literals they replaced (60/60
and 240/60), so an operator who sets nothing sees no change; a unit test
pins that, because the failure is silent in both directions — too low
and real users get 429s on listings, too high and the `access_grants`
query loses the guard that stops an attacker exhausting it with random
UUIDs.
`tests/common/server.env` (shared by the e2e AND hurl suites) sets both
to a 1-hour budget, matching the posture already used for the other
three rather than a raised per-minute rate that would still burst-trip.
The docs now state the IP-vs-caller split, since that is what decides
which knob to reach for — and note that several actors sharing one
identity (CI, a bot, a kiosk) share one caller bucket.
Left alone: the four narrower env files (OIDC, webdav-drive-root) keep
their existing MAX=3600 with default windows. No evidence they trip the
per-caller limits, and adding config on speculation is how these files
drift.
Not fixed here: rate-limit rejections emit NO audit line, which is why
the attribution above reads "likely" rather than "confirmed" — nothing
in the log names the limiter. AGENTS.md requires one for every
rejection; that is a separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`apiAdminCreateUser` intermittently failed with
`401 {"error":"DPoP nonce required","error_type":"DpopVerificationFailed"}`,
most visibly in admin.spec.ts's pagination test.
Not a nonce-rotation race — the nonce pool keeps a 3-minute overlap
window precisely so in-flight requests survive rotation. It is a
service-worker-control race.
`browserFetch` issues a raw page-context `fetch`, and the DPoP proof is
attached by the Service Worker intercepting it — the helper's own doc
comment says so. The SPA's proof-and-retry logic lives in `client.ts`'s
`dpopFetch`, which this helper deliberately bypasses. So when the SW is
not yet controlling the page, the request goes out unsigned, the
middleware sees a bound session with no proof (`dpop.rs`, the
`expected_jkt` match), and answers with a nonce challenge. Nothing
retries it: the SW that would have signed it is what is missing, and
`dpopFetch` was never in the path.
The window is real on every fresh browser context. `service-worker.ts`
does `skipWaiting()` + `clients.claim()`, which is correct, but claiming
is asynchronous — the first navigation loads uncontrolled, then
install → activate → claim. `waitForLoadState('networkidle')` says
nothing about SW control, so a helper called a few lines after
`apiLogin` can land inside it. Slower CI widens it, which is why this
showed up there and not locally.
The fix waits inside the same `page.evaluate` as the fetch, so it costs
one property check once the page is controlled and needs no per-page
bookkeeping. It is bounded at 10s: if the SW never claims, the request
goes out as before and the resulting 401 stays the clear signal it is
today rather than becoming an unexplained Playwright timeout.
Worth stating because it is easy to get wrong: **`ready` is not
`controlling`.** `navigator.serviceWorker.ready` resolves once a
registration is active, while `controller` stays null until that worker
has claimed THIS page. Awaiting only `ready` looks correct and still
flakes.
This is a test bug, not a product one. Real paths go through `apiFetch`,
which signs in page JS; the SW is the safety net for requests that
bypass it (`<img src>`, downloads). This helper is the only caller
relying on the SW as its primary signer.
Not verified by running the suite: a flake reproduces on CI timing, so
one green local run would prove nothing. The mechanism is confirmed from
the code path — SW-signed proof, no fallback retry, asynchronous claim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an Azurite service and a scenario that audits the Azure backend
through `?storage=azurite`. It is the only coverage of that code path in
the tree: `AzureBlobBackend` has unit tests for its name parser and
ordering, but nothing else speaks the protocol, and a paid account is
not an option for CI. Azurite implements the real Blob REST API, so this
exercises SharedKey signing, prefix/marker paging, and the 256-way shard
walk with its termination.
## Harness
`docker-compose.test.yml` gains an azurite service on 10000 (tmpfs, so
it dies with the stack). `spawn-db.sh` provisions the container itself,
because `AzureBlobBackend::initialize` verifies rather than creates —
signed by hand with curl + openssl rather than pulling a ~700 MB `az`
image for one PUT. Two traps are commented there: the account key is
base64 but HMAC wants raw bytes, and the canonicalized resource repeats
the account name (`/{acc}/{acc}/{container}`) because the emulator puts
in the path what real Azure puts in the host. Getting that wrong yields
403, not a hint.
The `azurite` entry is declared in `server.env` but never activated, so
the suite's active backend stays local and only this file reaches Azure.
## What it asserts, and what it cannot
A failure surfaces as `ok: false`, because an enumeration error now
fails the run rather than degrading to a per-row probe.
It deliberately asserts no finding count. The container starts empty and
the job's grace window is an hour, so a freshly-uploaded blob is skipped
in both directions by design — an audit here can only report zero, and
"zero findings" would pass whether enumeration worked or returned
nothing. The one positive assert, `scanned_count != 0`, therefore sits
on the local control, which does hold blobs; `scanned_count` accumulates
via `checkpoint`, which the empty-page early return skips.
## No cutover, deliberately
Putting real bytes in the container means `backend_migration
?storage=azurite`, which hangs on the first blob: `head_check` issues a
~40-byte ranged GET, `azure_core` 0.21 attaches
`x-ms-range-get-content-crc64` to anything under 4 MiB, Azurite 500s,
and the deterministic error is retried forever while
`migration_readonly` refuses writes app-wide. The full chain and the
rejected workaround are in the file header. The scenario is still
ordered last in `run.sh` — it is the only one needing a second service,
and the cutover comes back there once the official SDK lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The durable tier has been content-keyed since it was introduced —
`content_derived_blobs(source_hash, kind, variant)` — but the moka cache
in front of it was still `{file_id}:{ext}`, so the layer closest to the
request used the wrong axis while the layer behind it used the right
one. That was legacy shape, and I had defended it in a comment as
"deliberate: per-request-path and short-lived", which was a
rationalisation rather than a reason. Ed asked why, and there is no why.
Transcoding is a pure function of the source bytes. Under file keying,
two files with identical content held two RAM entries for identical
bytes, and the second file was a guaranteed miss that fell through to a
DB lookup plus a blob read to fetch what was already in memory under
another key.
Now keyed by content hash when the caller has one, by file id only when
it does not — the same `content` / `external` split `ThumbnailCacheKey`
already makes, and for the same reason: hash-less callers (external
mounts) have no content identity to key on. Prefixed `c:` / `f:` so the
namespaces stay disjoint; a hash and a UUID cannot collide in practice,
but "in practice" is how a file ends up served another file's bytes.
`invalidate` now clears only the file-keyed entry. Dropping content
entries there would be wrong, not merely wasteful: one file's content
changing says nothing about the other files sharing the old bytes, and
evicting theirs would make one user's edit cost everyone else a
re-transcode. Content entries need no eviction — new content is a new
hash, so the old key is never consulted again.
transcode_cache.hurl updated to match, and its header corrected: the
second file is now a RAM hit rather than a derived-tier read, so that
scenario can no longer isolate the durable tier. It says so, and points
at satellites_consistency and a restart as what covers it instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the job's surface: that it is registered, declares the metadata
the admin panel switches on (`mutates: always`, recoverable, a repair
description), and that a run against a drained tree completes cleanly
with zeroed counters.
It deliberately does NOT cover the re-keying, which is the part that
matters most. That needs `.transcoded/webp/` entries on disk before the
run, and nothing reachable over HTTP can create them — since the write
path moved to the derived tier, only hash-less callers still write
there, and hurl cannot place files in the server's storage directory.
The migration is validated by a snapshot restore instead, the way the
thumbnail one was; the file says so rather than implying coverage it
does not have.
The empty-tree assertions still earn their place. A drained tree is
what every run after the first sees, so it is the overwhelming
majority of this job's lifetime, and "does nothing, quietly" is a real
property: the thumbnail teardown warned `could not be removed / No
such file or directory` on every boot after its migration finished —
warning about success forever — and that was caught by eye, not by a
test. Counters are asserted as exact zeros, since finding work in a
directory the API cannot populate is the shape a re-keying bug would
take.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writing the hurl scenario surfaced a gap: a transcode that comes out
larger than the original runs a full decode + encode and increments no
counter at all. `transcodes` is bumped only on the success path, beside
`bytes_saved`, so the most expensive failure mode was invisible — a
multi-megapixel image decoded and re-encoded on every request, for
every file sharing that content, producing nothing.
That is precisely the cost the persisted negative verdict exists to
stop paying, and it could not be measured before or after. `not_beneficial`
counts it, kept separate from `transcodes` because conflating "work
done" with "work that paid off" would hide exactly what an operator
needs to see.
It is also what lets the hurl scenario assert the negative half: the
first fetch increments it, the second — a distinct file with identical
content — leaves it untouched, which is the negative row being read
rather than the verdict recomputed.
Assertions are exact equality against captured values throughout, no
`>` or `<`. A "greater than" would pass if a counter moved for the
wrong reason; equality against the prior reading catches any transcode
from any source, including one this scenario did not intend to cause.
Also fixes two URLs the first runs caught: file download is
`GET /api/files/{id}`, not `/content`, and the trash listing is
`/api/trash/resources`. And the duplicate uploads go to a second
folder — re-uploading the same filename into the same folder returns
the EXISTING file id, which would have made both halves of every
"two files, one content" pair the same row and left the scenario
asserting nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `GET /api/admin/transcode/stats` and a hurl scenario that uses it
to assert both halves of the caching contract.
The endpoint exists because the property was previously unobservable.
`derived_blob_copy.hurl` records the same limitation for thumbnails:
stored blob, RAM cache and a fresh re-render return identical bytes
with identical status, so no HTTP-level assertion can tell them apart.
Counters can. `transcodes` is work done; `cache_hits` and `disk_hits`
are work avoided, and a rising `transcodes` against a flat `disk_hits`
is exactly what a broken derived tier looks like from outside.
Each case uploads the same bytes as TWO distinct files. Re-fetching one
file would only prove moka works — that cache is keyed `{file_id}:{ext}`.
A second file with identical content is a guaranteed memory miss but the
same content hash, so avoiding a transcode there can only be the
content-keyed tier answering. That is the whole point of keying
derivations by content rather than by file, and this is the first test
that can see it.
The negative half is the one the row exists for: without it the server
re-runs a full decode + encode of a half-megabyte screenshot for every
file sharing that content, on every request, to discard the result each
time.
Assertions capture-then-compare rather than computing deltas — hurl has
no arithmetic in predicates, and pinning the exact prior value is
stricter anyway, since a transcode triggered from anywhere shows up.
Absolute values are never asserted: other scenarios in the same run
transcode too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The transcode negative path — "the result came out larger, serve the
original and remember that" — had no test because no synthetic image
reaches it. Measured against the real encoder: flat colour goes
4780 → 186 bytes, a diagonal gradient 24852 → 102, and uniform RGBA
noise still loses by ~242 bytes at every size, a margin constant in
absolute terms and so one that never flips. Grayscale does not help
either; WebP's subtract-green transform handles R=G=B.
Two things have to be true at once and only real content does both.
The encoder is the `image` crate's own minimal VP8L writer, not
libwebp, so it wins only where redundancy is extreme enough for any
encoder to find it. And the original has to be near PNG-optimal, which
a screenshot from a real capture tool is: a 2x Retina UI is long
identical runs, flat panels and sharp edges — precisely what PNG's
scanline filters plus zlib were built for.
So the fixture is a real OxiCloud screenshot (emails masked by
overtyping rather than block-filling, which would have added back the
flat redundancy the property depends on; re-verified negative after
masking, 556180 -> 511124 bytes).
`fixture_premise` pins both halves of what tests/api/transcode_cache.hurl
will assume — this one negative, red-image.png positive. Without the
guard a future encoder bump would silently turn the negative half of
that scenario into a second positive test: still passing, no longer
checking what it was written to check.
Worth recording for whenever libwebp replaces this encoder: most of
these screenshots would likely flip to positive, which leaves every
stored negative row a stale verdict. An encoder change has to purge
them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the derived check to `file_attached_blobs` and renames it, since
the two tables are one concept — the content-keyed and file-keyed halves
of "things attached to a Blob" — and `storage.copy_file_satellites`
already established the vocabulary.
The attached half is the one that cannot be recovered.
`attached_dangling_blob` is data_loss with `recoverable: false`: those
bytes were user-supplied and have no server-side render path, so nothing
can regenerate them. Its derived twin carries `recoverable: true`,
because a derived artifact is a pure function of its source and
re-rendering restores it. Same finding shape, materially different
stakes, and the detail says which.
No orphan-mapping check on the attached side, deliberately: `file_id` is
ON DELETE CASCADE, so a row cannot outlive its file. The database
enforces what the derived table cannot, since a content hash has no row
to point a foreign key at — which is exactly why only that half could
rot.
One job walking two tables needs a phase in the cursor, or an attached
checkpoint would be replayed against the derived table and silently
re-scan or skip.
Two things the sweep was missing, found while checking whether every
consistency job is actually exercised:
drives_consistency and folders_consistency were registered but never
run by any test. Now included; the list is exhaustive by intent.
An unknown job was a warning-and-skip. That protected feature-gated
builds at the cost of something worse: this list said
`derived_consistency` for one commit after the rename and would have
dropped that coverage without a word, leaving the suite green over a
check that no longer ran. It fails now.
Finds derived mappings whose Blob is gone on either side. Nothing else
can, and that is the point rather than an oversight: every other job
reasons from a Blob outwards, so a row whose SOURCE was reaped breaks
none of their invariants — valid reference, exactly correct refcount,
bytes present on the backend. Every check agrees the system is healthy
while the artifact is pinned forever. A leak that looks like
correctness, which is why it took four suite runs to name.
Two findings:
derived_orphan_mapping (inconsistent) — source_hash has neither a
manifest nor a blob row, so purge_derived_blobs can never fire for it.
Storage that grows and never reclaims.
derived_dangling_blob (data_loss) — blob_hash has no Blob behind it.
The mapping promises an artifact that is gone, so a read finds the row
and then fails.
Existence means EITHER table on both sides, since source_hash and
blob_hash each name a Blob: a manifest for CDC content, a bare blob row
for legacy whole-file content. Checking one would report every legacy
blob as missing.
Paged on the full primary key with a row-value comparison rather than
source_hash alone — a source has several variants, so a page boundary
can fall inside one and advancing by source would skip the rest. Both
existence probes fold into the page query, so a page is one round-trip
rather than 2xN. Cursor round-trip is tested, including that a malformed
one fails loudly: silently restarting would make a paged audit
under-report, which is the worst failure available to a job whose
purpose is finding what is missing.
e4c78ae0 stops new orphans at the write side; this finds the ones
already on disk, which that fix cannot reach. Added to the end-of-suite
sweep so it runs against real state every time.
No test passed repair=true, so verify_and_unlink, the
sidecar_delete_unverified finding and the directory removal had never
executed. That left the one destructive part of the migration as its
least-tested code: everything else is additive and recoverable, this
unlinks files after a readback check, and a defect costs bytes.
Four assertions, because "the files are gone" cannot by itself tell a
correct drain from a destructive one:
sidecars gone — the drain happened
rows still present — it deleted the COPY, not the record. Removing
the row would strand the blob in exactly the way
the bulk-reap bug just did: a live reference
with nothing behind it, which GC is then correct
to refuse forever.
unverified == 0 — every unlink passed its readback rather than
being skipped, which is the property that makes
deleting safe at all
directory absent — the signal step 10e gates on, and why remove_dir
is used: it refuses a non-empty directory, so
success proves emptiness rather than asserting it
Preconditions assert the sidecars exist first, or a no-op run would pass
all four by doing nothing. The directory check logs rather than fails,
since a concurrent render could legitimately repopulate it.
The run gave the decisive fact: `derived=1`. A content_derived_blobs row
still points at the leftover blob, so GC is CORRECT to keep it — the
leak is the row, not the bytes. purge_derived_blobs only runs when the
SOURCE is reaped, so the question is why that never happened.
So the dump now prints the source hash and what still holds it:
src_files, src_manifest, src_blob. If the source has a live file the
answer is "not deleted"; if it has none but a positive refcount, a
release was missed upstream; if it has no row at all, the source was
reaped WITHOUT purging, which would be a real ordering bug in reap_blob.
Also fixes the dump reporting only one of three blobs. `docker compose
exec -T` reads stdin, so it consumed the rest of the here-string feeding
the loop — the other two were never queried and vanished silently. The
same silent-truncation shape the diagnosis exists to expose, in the
diagnosis. `< /dev/null` closes it.
52c31a68 added a per-leftover refcount dump to storage_cleanup_check.sh
but referenced COMPOSE_FILE, which that script never defines — only
thumb_import_check.sh does. It would have run `docker compose -f ""`,
failed, and been swallowed by the `|| true` guarding the loop.
A silent no-op: the diagnosis would print nothing and the failure would
look exactly as uninformative as the one it was written to explain. The
same shape as the three bugs this suite has already caught — an error
dressed up as an unremarkable result — and I wrote it into the tool
meant to find them.
The `|| true` stays, so one unreadable blob cannot abort the loop before
the others report.
Three blobs survived the sweep. Five seconds of async-unlink polling did
not remove them, so they were never queued — GC had not judged them
collectible, and the loop had already exited.
It broke on the FIRST zero-reap pass. A single zero only says nothing
was collectible at that instant: releases cascade, since reaping a
source drops the references its derived and attached rows held and
`on_blob_deleted` does that from spawned tasks, so a pass can land in
the gap between "source reaped" and "dependents released" and report
zero with work outstanding. The import jobs added a level to that chain,
which is when it started biting. Now two consecutive zeros, with the
bound raised to match — one extra trigger over an empty store is
cheaper than a false pass reporting a clean disk.
The rest is diagnosis, because a list of paths cannot tell the three
causes apart and they need opposite fixes: a positive refcount means a
release was missed, an orphan means the reap predicate has a gap, and a
row without a manifest means the registry is inconsistent. Each leftover
now reports its manifest and blob refcounts plus how many files, derived
rows and attached rows point at it — so if this is a real leak rather
than the race, the next run names it instead of costing another full
pass through the suite.
storage_cleanup_check.sh asserted a sidecar exists on disk after
fetching a thumbnail. Correct while `.thumbnails/` was the durable
store; wrong since 10d2 removed that write. The check failed on exactly
the behaviour it was meant to confirm.
Inverted rather than deleted, because the inverse is the more useful
guard: a sidecar reappearing means a write path regressed to the legacy
shape, which would silently make `.thumbnails/` un-emptyable and strand
step 10e forever — its gate is the directory being gone, and a single
recreated file holds it open.
The HTTP 200 above already proves the thumbnail works; this now proves
it got there the new way.
Both of the helper's streams are silenced at the call site. It reports
absence loudly — red banner plus a `find` dump — because absence used to
be the failure; here it is the expected result, and leaving that visible
would cry wolf on every clean run.
The write paths now persist only to the blob tiers. Until this, dual-write
meant any render or upload recreated .thumbnails/ seconds after the import
job removed it, so step 10e's gate — "the directory no longer exists" —
could never hold.
Rendered thumbnails: the fs::write in persist_rendered is gone. Safe
because the read flip landed first, so nothing depended on that write to
be found, and a failed derived store now costs a re-render rather than
data — regenerable by definition. Existing sidecars are untouched and stay
readable through the fallback until the import drains them.
Uploaded previews needed a change first, and the order was not optional.
upload_thumbnail_impl logged and still returned 201 when store_attached_blob
failed — safe only while ext-{file_id}.jpg was a second copy. These bytes
have NO server-side render path, so removing the sidecar while the store
stayed best-effort would lose a user's upload behind a success response.
The PUT is now fatal, and drops the RAM entry too, or the cache would keep
serving a preview that was never persisted and vanishes on eviction,
contradicting the error the client just received. Only then does the ext-
write go.
thumb_import_check.sh had to change with it: its premise was "upload, then
delete the row, and what remains on disk is legacy state", which no longer
holds now that nothing writes sidecars. It lays them down itself, with the
bytes the API just served, at the exact paths the pre-10d2 code used. The
reconstruction stays faithful — same bytes, same paths — it just no longer
depends on current code to produce a shape current code has stopped
producing. Both sidecars get identical bytes, which is realistic rather
than a shortcut: they dedup to one blob while keeping separate mappings,
which is the property the keying split exists to preserve.
from_accept returns JPEG unless Accept contains image/webp, and no
thumbnail test sent the header — curl defaults to */*, which does not
match. So the whole suite ran on JPEG and the WebP path was never
exercised over HTTP, despite being what background generation writes and
what the derived tier was built around. The gap was invisible because
the JPEG results were all correct.
Three assertions, one property each.
Content-Type proves negotiation happened: thumbnail_content_type sniffs
the body with `infer` rather than echoing the request, so image/webp
cannot be right by accident — serve JPEG bytes down the WebP path and it
reads image/jpeg and fails.
Differing bytes prove they are genuinely two artifacts rather than one
served twice.
Differing ETags prove the validators are separate. `variant` has carried
the format only since 20261022000000; before that a JPEG request could
match the WebP row and be served the wrong codec, and a shared validator
is exactly how a cache would then hand either to either. A final
conditional request confirms each codec revalidates against its own.
Together these also cover the per-format variant keying that lets one
source hold both codecs — the prerequisite for JPEG clients ever leaving
the sidecar, and therefore for step 10e.
Note on placement: thumb_etag stays in step 4's capture block, beside
thumb_bytes. Every later request omits Accept and so negotiates JPEG, so
the validator must be the JPEG one — captured after the new block it
would describe a different codec than the bytes next to it, and the
copy assertions compare against both.
76590160 changed `variant_of` to return `{size}.{ext}`, but that value
was also being used as the on-disk DIRECTORY. Reads became
`.thumbnails/preview.webp/{hash}.webp`, which does not exist, so every
sidecar counted as unreadable and thumb_derived_import restored nothing.
Caught by thumb_import_check.sh on the run after the migration — the
harness earning its keep twice now, since this is the second defect it
has caught that no unit test could.
They are genuinely two strings and are now named as such: `dir_name` for
the path, `variant` for the row key. The cursor keeps using the
directory, so a run paused before the migration resumes at the same
position rather than restarting.
Also stops podman's compose-provider banner from burying the script's
output. Filtered rather than discarded, so genuine psql errors still
surface — swallowing those would turn a broken query into a silently
wrong assertion. Suppressing it at the source needs
`[engine] compose_warning_logs = false` in containers.conf, which is
per-developer config and cannot be relied on in CI.
Uploading without folder_id answered `500 Internal Error: folder_id is
required to determine file owner`. A missing required field is the
caller's error; as an internal_error it produced `error_type: Internal
Error`, which the SPA cannot distinguish from the server breaking — so a
malformed request looked like an outage.
Both sites become validation_error (ErrorKind::InvalidInput → 400), with
messages that say WHY the field is needed rather than restating that it
is: the destination folder determines the file's owner and drive.
The OpenAPI request body described it as "optional folder_id field",
which is how it came to be omitted — hit while writing
thumbnail_etag_content_keyed.hurl, where the upload was written from the
documented contract and 500'd. Now stated as required.
Regression test asserts the status AND that error_type is not "Internal
Error", since the contract the SPA switches on is error_type rather than
the message.
`SELECT 1 FROM storage.files WHERE id = $1` decoded as i64. PostgreSQL
types a bare `1` as int4, so the decode always failed — and since
`.ok().flatten()` turns a decode error into the same None as "no row",
file_exists reported false for every file. thumb_attached_import
therefore classified every sidecar as an orphan and imported nothing.
Caught by thumb_import_check.sh on its first run: the derived import
restored its rows, the attached one restored none.
Now `SELECT EXISTS(...)`, which yields a real bool and always returns
exactly one row, so absence means absence. A query error still degrades
to false — the safe direction, leaving the file on disk as a reported
orphan rather than importing it against a row that may not exist.
The failure mode is the point, and it is the third of this shape in two
days: an error converted into an innocuous-looking outcome. So the check
script now dumps a job's findings when an assertion fails. The jobs
already recorded exactly why they skipped each file — the
attached_sidecar_orphan findings naming the cause were sitting in the run
while the script reported only "did not restore the row", which is
indistinguishable from the job never having run.
Nothing exercised these jobs. Their unit tests cover the directory walk
— which files each claims — but neither had ever executed a run.
The test environment always starts fresh, so there is no pre-migration
data to import. This creates it, and the reconstruction is EXACT rather
than an imitation: the on-disk layout did not change in this work. A
rendered thumbnail has always been written to {size}/{hash}.webp and an
uploaded preview to {size}/ext-{id}.jpg; the only new thing is the row.
So upload through the real API, then delete the row, and what remains on
disk is byte-for-byte what a pre-migration install has.
Deleting the row must also release the reference it held, or the
manufactured state would carry a reference no legacy install ever had
and storage_cleanup_check.sh would report a leak this script caused.
file_attached_blobs has an ON DELETE trigger for that;
content_derived_blobs does not — its Rust purge path releases explicitly
— so the strip decrements it directly.
Three assertions, in increasing order of what they catch:
1. Both rows come back, and the imported attached row carries the nil
uploader sentinel rather than a fabricated one.
2. A COPY inherits the imported preview. This is the user-visible
point and was impossible before the row existed: the ext- sidecar
is keyed by file_id, no copy path duplicates it, so the copy
silently fell back to a render.
3. Re-running imports nothing and changes no refcount. The likeliest
silent defect — store_attached_blob is ON CONFLICT DO UPDATE, so an
import that skipped its existence check would release and retake a
reference every run, invisible except as drift.
psql runs inside the compose container rather than depending on a host
binary, matching how spawn-db.sh probes readiness. Ordered before
storage_cleanup_check.sh, which deletes everything it needs.
Thumbnails were served `public, max-age=31536000, immutable`. Two
problems, and the first is a security one.
`public` on a Permission::Read gated resource lets any shared cache — a
corporate proxy, a CDN — store one user's thumbnail and serve it to
another. `Vary: Accept` was no defence: it does not vary on
Authorization. Now `private`.
`immutable` was a promise this URL cannot keep. It is keyed by file id,
and its bytes change when a preview is uploaded, when content is
replaced, or when an attachment is removed. `immutable` tells a client
not to revalidate at all during the freshness lifetime, so with a
one-year max-age a browser that fetched once would never see a new
preview — which also made the content-keyed ETag unobservable in
practice. A correct validator is worthless if nothing asks. Now
`no-cache`, which still stores the body and only requires revalidation,
answered by the ETag with a body-less 304.
The hurl tests could not have caught this: hurl always sends the
request, so If-None-Match was exercised and passed while a browser
obeying `immutable` never got that far. Same "correct on the wire, wrong
in practice" shape as the bugs before it, so the test now asserts the
directives themselves rather than only the 304 behaviour.
One definition, shared by the REST and NextCloud endpoints, which are
gated identically and must not drift. /_app/immutable is untouched:
those are hash-named static assets, genuinely content-addressed and
public, where the directive is honest.
Cost is a conditional request per thumbnail per page load. Recovering it
needs a content-addressed URL — where `immutable` would be true — but
that puts the hash in the URL of an authorized resource, so it stays
`private` regardless, and it touches the SPA and the file DTO. Separate
change.
fe9c4f49 keyed the ETag on the SOURCE file's content hash. That is wrong
whenever the response comes from a satellite table, and for attachments
it is wrong in two ways.
Uploading a preview does not change the file's content, so a
source-keyed ETag does not change either — and with `immutable` set,
clients never revalidate and keep the previous render for up to a year.
The exact staleness fe9c4f49 set out to fix, re-entering through the
attachment path.
Worse: a copy inherits the source hash, so an original and a copy have
identical ETags. Give either one a different uploaded preview and they
serve different bytes under one validator, which a shared cache may hand
to either request. That is a collision, not just staleness.
thumbnail_content_id resolves the identity through the same tier
precedence the read path uses: an attached blob's own hash, else a
derived blob's own hash, else the source-keyed form. An ETag naming a
different tier than the one answering is worse than a coarse one, so the
two orders must not drift.
Derived-hash keying is strictly better than source-keying and never
worse. The sidecar and the derived row are written from the same bytes;
where they can diverge — a sidecar re-rendered while the derived row
stays pinned by ON CONFLICT DO NOTHING — source-keying is wrong too,
because the renderer is not part of that key. This is the step 10 change
arriving early, forced by the attachment case; the plan note stands for
the read-order flip itself.
Known gap: a legacy ext-{file_id}.jpg with no file_attached_blobs row
yet falls through to the source-keyed form. No worse than today, and it
resolves when the import backfills.
attached_thumbnail_copy.hurl now asserts ETags, which is why this went
unnoticed: it compared bytes only, and thumbnail_etag_content_keyed
covers content replacement rather than preview upload. A fresh GET
returned the right bytes throughout — the same "healthy locally, broken
for anyone caching" shape as the two bugs before it.
The disk check proves no BYTES are left. This proves no ROWS are, which
fails differently and worse: a stale storage.blobs row with nothing
behind it means a reference was never released, and dedup_gc will skip
it forever because its count never reaches zero. Silent, permanent, and
invisible to a check that only looks at the filesystem.
Zero is the right assertion, not "fewer than before". By this point the
suite has deleted its users, their drives and everything cascading
beneath, and the disk check has already insisted the blob store is
empty. A non-zero registry beside an empty disk is exactly the
divergence the consistency jobs report — caught here first because one
number is easier to read than a findings list.
Degrades to a warning if the endpoint is unavailable rather than
failing, so a build without the admin dedup surface still runs the rest.
The disk checks above prove nothing leaked. These prove the bookkeeping
behind them is honest: every refcount matches what the reference sources
hold, and no row points at bytes that are gone.
End of suite is the only place this is cheap. One database serves every
hurl file, so by here the counters have absorbed every upload, copy,
move, share, trash and purge the suite performed — across both copy
paths, the derived tier and the attached tier. Drift that no individual
test would notice, because each only inspects its own file, surfaces as a
mismatch.
Runs after the GC drain deliberately: mid-sweep state is legitimately
inconsistent — a manifest can sit at zero waiting for the next pass — so
checking earlier would report normal in-flight state as drift.
Zero findings is the assertion. These four tenants are read-only, so
anything they report is a real invariant violation rather than a repair
opportunity. A job missing from the build is skipped with a warning
instead of failing, so this does not break on a feature-gated build.
Unknown job names and unwrapped-vs-wrapped response shapes both degrade
to a visible warning rather than a silent pass: list_job_runs currently
returns a bare array, and the .runs/.items fallbacks exist so a future
wrapping does not quietly turn the whole check into a no-op.
Completes step 9. The PUT wrote `ext-{file_id}.jpg` and nothing else —
keyed by file id, on local disk. No copy path duplicates it and no other
instance can see it, so a copied file lost the preview its owner
uploaded. Silently: the server falls back to rendering one from the
source, or to 204 for a PDF, which has no render path at all. A
user-supplied preview is not derivable from the content, so once lost it
is gone.
The PUT now also records a storage.file_attached_blobs row, which
copy_file_satellites already duplicates, so both copy paths carry it.
Best-effort: the sidecar has already succeeded by then and the user can
see their thumbnail, so failing the request would report an error for an
operation that visibly worked.
Read path consults attachments ahead of every content-derived tier: an
uploaded preview is an explicit choice about THIS file and must beat
anything rendered from its content. Cached under the per-file key — a
content key would leak those bytes to every other file sharing the
content, which is the poisoning the file-keyed table exists to prevent.
store_attached_blob is ON CONFLICT DO UPDATE, unlike its derived twin:
re-uploading a preview is a deliberate replacement, where a re-derived
thumbnail is the same bytes again. The superseded blob's reference is
released, or it would be pinned forever with nothing pointing at it.
Deletion goes through a trigger, not a hook. file_id is ON DELETE
CASCADE, and on_file_deleted fires AFTER delete_file — by then the
cascade has run and there is nothing left to enumerate. This matters
most for folder deletion, where PG cascades folders to files to
attachments and Rust never sees the rows at all. storage.decrement_blob_ref
keys off OLD.blob_hash and is otherwise table-agnostic, so it is reused
verbatim rather than transcribed into a second trigger that can drift.
DELETE only: a replacement updates in place and is handled in Rust, so
adding UPDATE would double-decrement.
Extracted read_blob_to_bytes, shared by the attached and derived tiers —
the only difference between them is which table produced the hash.
tests/api/attached_thumbnail_copy.hurl guards it. The file is red and
the uploaded thumbnail is green, so a render could never produce the
uploaded bytes; the pre-upload render is captured first and required to
change, which stops three identical renders from satisfying the
byte-equality. Then both copy paths must serve the upload, and after the
original is purged and GC runs, both copies must still serve it — each
holds its own reference, because the rows are duplicated rather than
shared.
GET /api/files/{id} is the download route — it returned the PNG bytes,
so the jsonpath capture failed on a UTF-8 decode. /{id}/metadata is the
EXIF endpoint and carries no FileDto either. Listing the folder gives
the DTO, and since the folder holds exactly this one file, count == 1
also proves the WebDAV PUT overwrote in place rather than creating a
second file beside it.
Also drops an unused bytes capture and records why the body is not
asserted after the overwrite: the moka tier is keyed on file_id and
invalidated from the spawned task in on_file_updated, so a request
landing first sees the previous bytes under the new ETag. Asserting on
bytes would be a race.
The upload omitted folder_id, which the handler needs to resolve the
file's owner — it answers 500, not a root upload. Every other upload in
the suite passes it; this was the only one that did not, which is why
nothing caught it earlier.
The folder also gives the WebDAV overwrite a deterministic path
(/webdav/hurl-etag-src/<name>) instead of depending on where a
folder-less upload would have landed. Teardown now removes it and
purges it from trash, keeping the shared database clean for the files
that run after.
cjw
Edouard Vanbelle