The filter was strip_suffix(".webp"), but persist_rendered writes
{hash}.{format.ext()} — so any client not advertising WebP leaves
{hash}.jpg on disk. Correct only while the derived tier was WebP-only;
once variant carried the format (20261022000000) a JPEG sidecar became
ordinary content, and leaving it unclaimed would keep .thumbnails/
permanently non-empty — the very signal step 10e gates on. The migration
could never finish.
Both codecs are now claimed and the format comes from the file's own
extension, so a .jpg imports AS JPEG. Deriving the variant and
content_type from it rather than hardcoding WebP is the point: a
mislabelled row would serve the wrong codec to whoever the read path
then matched it for.
ThumbnailFormat::ALL exists so the claim list and the write path cannot
drift — adding a format without teaching the import about it would
strand that codec silently.
The `ext-` rejection now carries real weight. Previously .jpg was
rejected wholesale, so the two jobs could not overlap by construction;
now they share an extension and only the prefix separates them. Both
directions stay under test.
Caught by the cross-job assertion, which counts every real sidecar being
claimed exactly once — the fixture gained a .jpg and the total moved 3
to 4, which is the test noticing rather than a test to update.
`SELECT 1 FROM storage.files WHERE id = $1` decoded as i64. PostgreSQL
types a bare `1` as int4, so the decode always failed — and since
`.ok().flatten()` turns a decode error into the same None as "no row",
file_exists reported false for every file. thumb_attached_import
therefore classified every sidecar as an orphan and imported nothing.
Caught by thumb_import_check.sh on its first run: the derived import
restored its rows, the attached one restored none.
Now `SELECT EXISTS(...)`, which yields a real bool and always returns
exactly one row, so absence means absence. A query error still degrades
to false — the safe direction, leaving the file on disk as a reported
orphan rather than importing it against a row that may not exist.
The failure mode is the point, and it is the third of this shape in two
days: an error converted into an innocuous-looking outcome. So the check
script now dumps a job's findings when an assertion fails. The jobs
already recorded exactly why they skipped each file — the
attached_sidecar_orphan findings naming the cause were sitting in the run
while the script reported only "did not restore the row", which is
indistinguishable from the job never having run.
Both imports run over ONE directory, where the two legacy shapes sit
side by side, so the property worth asserting spans them: together they
must claim every real sidecar exactly once, and neither may take the
other's. A job that drifted into the other's shape would content-key
user-supplied bytes — sharing one user's uploaded preview onto every
file with identical content — and no per-job test in isolation would
notice.
So the fixture is shared. `legacy_tree` builds a directory holding a
content-keyed .webp pair, an ext- upload, and a stray README, and both
test modules walk it: derived claims exactly the two hashes in sorted
order, attached claims exactly the ext- file, the two sets are disjoint,
and between them they account for all three real sidecars.
`sidecar_names` became an associated function taking the root instead of
reading `self`, which is what makes this testable at all — the walk is
the half that decides which files a job claims, and it needed no pool to
verify. Sorting is asserted rather than assumed, since the cursor
resumes by skipping everything at or before it and a stable order is the
only thing that makes that correct.
A missing size directory is covered too: normal on a fresh install, and
it must yield no work rather than abort the walk.
Not covered here, and it needs a pooled fixture that does not exist: the
round trip itself — store the blob, write the row, and confirm a COPY
inherits the preview. That belongs in the API-level harness, where the
legacy state can be manufactured through the real write path and then
stripped.
Twin of thumb_derived_import, for the other sidecar shape:
{thumbnails_root}/{size}/ext-{file_id}.jpg, the previews a user
uploaded — notably the SPA's client-side PDF generator, which has no
server-side render path at all.
Until a row exists, a copy of the file LOSES the preview: the sidecar is
keyed by file_id, no copy path duplicates it, and the server silently
falls back to rendering from the source, or to nothing for a PDF. That
is the bug file_attached_blobs closed for new uploads; this closes it
for everything already on disk.
Separate job rather than an arm of the derived import, because the
keying differs and that difference is the security boundary. These bytes
are not derivable from the file's content, so content-keying them would
share one user's uploaded preview onto every file with identical
content. Each job's name filter rejects the other's shape, and both
directions are under test.
Idempotence needs more care here than in the derived twin.
store_attached_blob is ON CONFLICT DO UPDATE, so calling it for an
existing row releases the previous reference and takes a new one —
harmless once, but a job doing it every run would churn refcounts. The
row is therefore checked first and the store reached only on a genuine
insert.
uploaded_by is the nil sentinel: disk records no uploader, and inventing
one — the file's created_by, say — would fabricate provenance that could
later read as evidence an Editor replaced someone's preview. The column
is NOT NULL with no FK precisely so provenance survives, and a sentinel
says "unknown" honestly.
Orphaned sidecars (no storage.files row) are counted and reported, not
deleted. This job imports; it does not reclaim. Existence is checked
explicitly rather than letting the foreign key reject the insert, so an
orphan is counted as one instead of surfacing as an opaque constraint
error.
Edouard Vanbelle