`blobs_consistency` probed `blob_exists` once per row and, under
`?deep=true`, read and re-hashed every blob. `backend_consistency`
already reports the same `blob_missing_from_backend` from its
merge-join — so the probe was duplicated work that found strictly less
(a DB walk cannot see backend-only orphans by construction) at N round
-trips instead of one enumeration. Every scheduled sweep paid for it.
All three physical checks move to `backend_consistency`:
* `blob_missing_from_backend` was already there; the duplicate is gone.
* `blob_corrupted` / `blob_unreadable` hook the matched arm of the
merge-join, which holds exactly the key pairs worth reading. Guarded
by `in_range` so a pair past the horizon is not read twice, and
`params.deep` is persisted on a fresh run and read back on resume so
a paused deep scan does not silently continue shallow.
Deep mode belongs there because it is backend work end to end: the
only DB input is the hash. Keeping it in `blobs_consistency` forced
that tenant to carry a backend for one flag.
What remains is the half that needs no backend: `refcount_mismatch`
and its repair. The constructor drops from five parameters to two —
no backend, no storage_entries, no storage_path_fallback — and
`?storage=<name>` / `?deep=true` are now inert there, which the
job description says outright.
`affected_files` is needed by both tenants, so it moves to a shared
`blob_diagnostics` module rather than being copied.
`PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined
in `blobs_consistency` and re-exported, which is backwards once the
DB-only tenant has no entry to scope. The create-grace window goes
with the probe — it existed to avoid flagging a blob whose bytes had
landed before its row, and the refcount comparison reads one
consistent snapshot.
Known cost: `backend_consistency` returns `backend_unenumerable` on
Azure and mid-migration, so on those configs missing bytes now go
unreported where the per-row probe caught them. That argues for the
Azure enumeration impl, not for keeping the probe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 10 went ...enable deletion, then remove the fallback "once the
directory no longer exists". That gate is unreachable as written: while
persist_rendered dual-writes and the PUT still writes ext-, any render
or upload recreates the tree seconds after the job removes it, so the
directory never stays absent and (e) can never fire.
Adds it as d2, between deletion and fallback removal, with the split
that only became visible while implementing 10d.
Rendered sidecars can stop immediately — the read flip has landed,
existing files are untouched so un-imported boxes keep their fallback,
and a failed derived store costs a re-render rather than data, since
that content is regenerable by definition.
Uploaded ones cannot, yet. upload_thumbnail_impl logs and still returns
201 when store_attached_blob fails, which is safe only because the ext-
sidecar catches it. Remove the sidecar while the store is best-effort
and a user's preview vanishes silently behind a success response — and
these are precisely the bytes with no server-side render path. So the
PUT must become fatal first.
Order recorded explicitly: make it fatal, then drop the sidecar.
Reversed, it trades a silent data-loss window for an empty directory.
Resolves the .skip-marker question. The table can say "here is the
artifact" but not "there is deliberately no artifact", and absence of a
row is ambiguous — it means both never-attempted and attempted-and-not-
worth-it, which destroys the only thing a negative cache holds.
Two live cases, not one. Transcodes drop a .skip marker when WebP is not
smaller. Thumbnails return empty Bytes for undecodable sources and ones
over MAX_DECODE_PIXELS — RAM-only today, so after moka evicts, a
60-megapixel upload has its full decode attempted again, forever. Same
gap, same table, so do both together.
Nullable rather than a sentinel hash: a sentinel stops blob_hash naming
a real blob and every future reader has to know the lie. Costs are one
`AND blob_hash IS NOT NULL` in ContentDerivedReferenceSource, the same
guard on the dangling-derived check, and dropping NOT NULL.
Permanent vs transient is the load-bearing split, and today's code
cannot tell them apart: generate_and_persist collapses every error into
empty Bytes, timeouts and semaphore closures included. Survivable while
the sentinel lives in moka, which evicts. Persist that same signal and a
thumbnail that timed out once under load is unrenderable forever. So the
renderer must return a typed outcome first, and only
permanently-unrenderable earns a row. The asymmetry sets the default —
a wrongly-cached transient is silent and permanent, a not-cached
permanent only costs repeated work — so unclassified errors are treated
as transient.
No TTL and no renderer-version term. Time is the wrong axis: the verdict
is deterministic in (content, encoder) and does not decay, so a TTL
re-attempts an OOMing decode on a schedule while still leaving staleness
for most of the window after a deploy. What removes the need for a
mechanism is that negative rows are disposable — they hold no data, so
a library upgrade invalidates them with a DELETE ... WHERE blob_hash IS
NULL in the same migration as the dependency bump. Recorded explicitly
so nobody later builds the expiry logic this replaces.
Investigated but deliberately not started — the remaining pieces span a
service, its DI wiring and an unresolved design question, and a
half-wired service is worse than none.
ImageTranscodeService does not write the derived tier at all today, the
same gap persist_rendered closed for thumbnails, and it must close
before transcode_import can converge or the import chases a growing
cache.
Three pieces, and the first is smaller than it looks. get_transcoded
takes file_id while the table is content-keyed, but the hash is already
in scope one frame up — file_retrieval_service::try_transcode is called
where dto.content_hash is live — so it is a parameter to thread, not a
lookup to invent. Explicitly NOT by hashing original_content on the
fly, which would be a BLAKE3 over the whole file per request.
Second, the service has no BlobHandler field, so this touches the
constructor and DI; ThumbnailService hit the same ordering problem and
solved it with a per-call parameter, which is the cheaper precedent.
Third, the .skip markers stay unresolved: a negative verdict has no
bytes, so it does not fit a table whose row points at a blob. Either
leave them local and recompute per instance, or model a sentinel. It is
the only genuinely open design question left in step 10.
Answering "when is transcode_import planned": it is the remaining third
of 10(b), but it must not be next, and both reasons were learned on the
thumbnail side rather than predicted.
Format has to move into `variant` first. Transcodes are inherently
multi-format and `variant` is keyed on size alone — the same gap found
in 10c that keeps JPEG thumbnails on the sidecar. Importing before that
means migrating into a schema that cannot hold the data without
collisions across formats. One migration unblocks both, which is the
argument for doing it before either.
And step 7 must precede the import. ImageTranscodeService writes only
its file-keyed disk cache today, so the import would run against a cache
that is still growing and never reach an empty tail — exactly the trap
persist_rendered had to close for thumbnails, where one of four render
paths recorded a row and the tail could never empty.
Order: format-in-variant → step 7 → transcode_import.
Also records why the .skip markers are still open: a cached negative
verdict has no bytes, so it does not fit a table whose point is pointing
at a blob.
Step 10c. The sidecar is local disk — invisible to other instances,
uncarried by a backend migration, uncovered by any consistency job.
Reading the derived tier first is what makes that state deletable. Not
the cost it appears to be: CachedBlobBackend gives the blob read a local
disk cache and moka absorbs the repeats above it.
Not a two-line swap, for two reasons.
A derived MISS must fall through to the sidecar; the old code
terminated the lookup with `?` because it was last. While the imports
drain, most content has a sidecar and no row — terminating there would
report "no thumbnail" for nearly all of it.
And the derived tier is WebP-only. store_derived_blob writes image/webp
and keys `variant` on the size alone, with no format term, so a JPEG
request matches the WebP row and would be served the wrong codec. The
old ordering hid this because the .jpg sidecar won first. So the lookup
is gated to WebP, JPEG clients stay on the sidecar — and the sidecar
cannot be deleted for them until `variant` encodes format. That is a new
prerequisite for step 10e, recorded in the plan rather than discovered
later.
Also corrects the plan: I had written that this flip removes the
derived-hash ETag hazard. It does not. A first render still creates the
row as a side effect of producing the body, whatever the read order, so
two consecutive reads still straddle its appearance. The real fix is
resolving the ETag after generation on the 200 path — a 304 only fires
when the client already holds a validator, which implies the row exists.
That is a handler restructure, not an ordering change.
Closing out step 10(a). The remaining `None` call sites in
persist_rendered looked like an open gap; they are not reachable in
production. Both `get_thumbnail` and the path variant of
`generate_all_sizes_background` are called only from the `ThumbnailPort`
impl, and nothing holds a `dyn ThumbnailPort` — which the existing note
in get_cached_thumbnail already recorded and a grep confirms. Live
renders go through get_thumbnail_from_blob and
generate_all_sizes_background_from_blob, both of which carry a
DedupService and dual-write.
So threading a DedupService through them would be work with no runtime
effect. Recorded at each call site instead, with the condition that
matters: gaining a real caller means taking a DedupService first, or the
gap persist_rendered exists to close reopens — sidecar-only output the
import can never see, so the tail never empties and the deletion gate
never opens.
Marks 10(a) done in the plan with that caveat stated rather than
implied.
The ETag is computed BEFORE the body. On a cache miss no
content_derived_blobs row exists, so thumbnail_content_id returned the
source-keyed form — then rendering created that row, and the next
request resolved to the derived hash instead. The validator changed as a
side effect of producing the body, making every first render
immediately stale.
Latent until aaf08532: before the consolidation, the on-demand render
path never wrote a derived row, so the flip had nothing to trigger it.
Fixing one gap exposed the other.
Caught by thumbnail_etag_content_keyed.hurl — two consecutive GETs of an
unchanged file stopped revalidating to 304.
The plan already said derived-hash keying must land WITH the read-order
flip and not before; I brought it forward anyway when the attachment
case forced the attached half. This is the evidence for the constraint,
so the plan now records the attempt and why it failed rather than
leaving the note as untested caution.
The attached lookup stays — it has no such window, since an upload
writes its row synchronously before any read can observe it, and it
fixes a real collision: a copy inherits the source hash, so an original
and a copy carrying different uploaded previews would otherwise share
one validator while serving different bytes.
The flip removes the hazard for the derived half too: once that tier is
authoritative it is populated before it is consulted, so no row can
appear between two reads.
Two revisions from working through step 10.
**Deletion moves into the import jobs, not a release.** Sidecars are
local disk, so a release cannot know whether every instance has drained
— gating on "an empty tail" asks an operator to coordinate a fact
nothing reports, and there is no telling when or whether they trigger
the jobs at all. Each job unlinking what it has imported makes every
instance drain itself. Constrained three ways: verify the derived blob
reads back before unlinking (a store that reported success but landed
unreadable would otherwise take the last copy), only after the
read-order flip (or the derived tier takes its first production traffic
by accident), and opt-in, since a migration that deletes by default is
surprising. Scheduled tick rather than boot trigger — idempotent and
resumable, so periodic is safe, while walking .thumbnails/ at startup
delays readiness for nothing.
**Found while checking the dual-write assumption: it does not hold.**
store_derived_blob has ONE call site; fs::write(&thumb_path, …) has
five. get_thumbnail, generate_and_persist and
generate_all_sizes_background all persist sidecar-only. That breaks the
migration's premise rather than being untidy — on-demand renders keep
producing un-migrated state after the import runs, so the tail never
empties and the deletion gate never opens. One persist_thumbnail owning
sidecar + derived + moka is therefore a prerequisite, and it makes "stop
writing sidecars" a later one-line change instead of four edits. Noted
that ThumbnailService holds no DedupService, so it must be threaded
through.
Also corrects a claim I put in thumb_derived_import's own docs:
transcoding is NOT a later step. ImageTranscodeService exists and caches
.transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it
must re-key file→content — legitimate only because a transcode is
derivable. Its .skip markers remain an open question.
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format),
which is one term short: a thumbnail is a function of those PLUS the
renderer. Change the encoder or a quality setting and identical inputs
produce different bytes under an unchanged ETag — the same staleness
class the commit fixed, one level down. It bites when an already-cached
thumbnail is re-rendered after a renderer change.
Keying on the derived blob's own hash removes the term entirely: the
ETag IS the hash of the bytes, so any output change invalidates by
construction. It is self-consistent for free, because store_derived_blob
is ON CONFLICT DO NOTHING — a re-render never displaces the stored row,
so the ETag always equals what the derived tier will serve. No renderer
version constant to remember to bump.
Records why it cannot land yet. The derived tier is read LAST by design,
so an ETag naming the derived hash would describe a tier the response
probably did not come from; sidecar and derived agree at creation but
diverge if a sidecar is re-rendered while the derived row stays pinned
by DO NOTHING. An ETag that lies about the body is worse than one that
is merely coarse. Also the tier is WebP-only (variant is the size, with
no format term) and empty for anything predating this work until
derived_import backfills.
So it lands at step 10 with the flip, keeping today's form as the
fallback for ungenerated variants and formats the tier does not hold.
The LEFT JOIN already planned for the read path returns the derived
hash in the same query, so it costs no extra round-trip.
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-13 runner tier is being phased out by GitHub — queues persistently
exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to
'cargo install --features bundled-assets' from source, Docker
--platform linux/amd64, or a Linux VM.
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in
example.env and docs/config/env.md — closes the discoverability gap
where the env var was only visible in Rust docstrings.
Also lands docs/plan/bundled-binary.md — the design record referenced
from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs,
and the Dockerfile.
Adds the tag-triggered workflow that builds 4 musl-linux + macOS
tarballs and attaches them to the tag's GitHub Release. Ships a
matching install guide (docs/install/binary.md) with SHA256SUMS
verify, systemd unit, upgrade flow, and hardware notes. Adds
[package.metadata.binstall] so 'cargo binstall oxicloud' works
automatically once the first release lands.
Also re-enables incremental compilation in the dev profile — the
'modest single-crate savings' rationale from when the crate was small
has been outgrown; full rebuild ~10 min is now the dev-loop bottleneck.
derived-blobs.md — consolidates several review rounds.
BLOCKER found while building the coverage matrix: the zero-ref manifest
sweep in dedup_gc (dedup_service.rs:2574) deletes a manifest when
`ref_count <= 0 OR NOT EXISTS (SELECT 1 FROM storage.files ...)`. That
OR hardcodes "storage.files is the only thing that can reference a
manifest", so a thumbnail manifest held only by content_derived_blobs
is deleted on the next GC run, its chunks dereferenced and the bytes
reaped. Promoted to prerequisite 0 and delivery step 2. It is also the
missing half of the unreconciled chunk_manifests.ref_count: that OR is
the hack that made the drift survivable.
Adds the 13-edge consistency coverage matrix (rows 1-6 and 13 covered,
7-11 not), and records that backend_consistency needs NO change — the
backend holds chunks, which neither new table references.
BlobReferenceSource correction: `ref_level()` was wrong because
FilesReferenceSource spans both levels (chunk for legacy manifest-less
rows, Blob for CDC rows). Replaced with
`ref_count_sql(level, alias) -> Option<String>`.
Also: migration of existing sidecar content; schema trim to the columns
nothing else owns (no size/format/codec/renderer; content_type kept as
non-key since it removes today's byte-sniffing); ext-{file_id}.jpg
corrected — the client generator ships and covers PDF, which has no
server-side rasteriser, so file_attached_blobs is required rather than
deferred; uploaded_by on the shares NOT NULL/no-FK convention; the
mermaid relation map; copy and version semantics with the
copy_file_satellites consolidation; the storage.files vs file_metadata
table-identity fix; DedupService -> BlobHandler recorded as decided.
NEW hidden-system.md — retires auth.users.image TEXT (inline base64
avatar, up to 512 KiB, already worked around with a narrow projection
after it was measured detoasting M avatars per group fan-out) in favour
of *_file_id pointers at ordinary storage.files rows in one shared
hidden system drive. Because storage.files is already a
BlobReferenceSource, a file pointer costs zero new reference sources
and zero new consistency edges. Records why the alternatives lose,
the drive's required properties (hidden at enumeration, trash off,
quota exempt, boot fail-fast), per-kind visibility in code, the
secrets exclusion rule, the avatar migration, and the future object
catalogue.
Docs only; no code or schema changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enrich OxiCloud to maximise the use of `dedup` Engine
2 cases will be covered:
- blobs issues from other blobs (thumbnail automatic generation from blob)
- by filename (ex: thumbnail uploaded from users)
A local cache will be added when blobs are remote (S3 or similar)
link are checking that email matches, +email alias are normalize into email
if email is already used on another account, link is not possible
not usurpation risk as the IDP is choosen by the admin
this make OIDC compliant with the invariant binding (issuer and subject)
admin can now rename their provider without breaking
clarifing federation_kind: report the kind of federation wired not the allowed login method
hybryd login method are still allowed
OPAQUE (RFC 9807) implementation (using `opaque-ke` crate)
with opaque authentfication, server will never receive the password (in the auth=password mode)
this is a must have to create trust with users to permit end to end encryption in the future
(we cannot know if user use the same password/passphrase for his asymetric key or his oxicloud auth,
this is why server must never have the password)
pass1: prepare server
add `auto_redirect_if_standalone_oidc` in `OXICLOUD_AUTH_POLICIES`
let admin decide to redirect immediately to IdP if OIDC is the only auth method enabled
permit an admin to specify `oidc` only as the only method to login/register
note that if OIDC is enabled, the engine always append oidc in OXICLOUD_AUTH_METHODS
we could move to an explicit declaration in a major release
Prevent services accessing directly to localstorage and prefer using an astraction layer
to expose full blob. The abstraction layer (dedup services) will cover backend storage
election (local, s3, ...), encryption, etc
This change permit audio_metadata_service, media_metadaa_service, face_indexing_service to handle
blobs without worring of the backend.
note: prefered way to handle blob is the streamed way. Some services may not have this possibility
Two chronic problems fall out:
1. **Split-brain config.** Admin edits DB via the panel; app boot ignores DB.
Migration completes; live backend hasn't moved. Admin has to remember to
copy env vars into `.env` and restart. Two sources of truth for the same
setting. Cutover is a manual multi-step flow; users routinely get it wrong.
2. **Migration data-loss window on concurrent writes.** The copy walks
`storage.blobs` in hash order. A blob whose hash is lex-lower than the
current cursor, written to source AFTER migration passed it, is never
copied to target. `passed=true, findings=0` completion does NOT guarantee
target has every blob. Silent.
3. **Migration target selection is fragile.** DTO passes the whole S3 config
at trigger time; secrets sit plaintext in `admin_settings`. Any future
pluggable-storage story compounds this (Azure, GCS, WebDAV-as-source, …).
This plan replaces the split-brain model with a single-source-of-truth
architecture:
- `.env` declares **N named storage entries** (immutable per-deploy).
- `admin_settings.storage.active_backend_name` holds ONE row — which named
entry the app currently runs on. That's the whole runtime config.
- Migration is the atomic transition from one active entry to another. Server
is put in read-only mode for the copy window; on completion, the active
pointer flips; a restart cuts over.
Edouard Vanbelle