Investigated but deliberately not started — the remaining pieces span a
service, its DI wiring and an unresolved design question, and a
half-wired service is worse than none.
ImageTranscodeService does not write the derived tier at all today, the
same gap persist_rendered closed for thumbnails, and it must close
before transcode_import can converge or the import chases a growing
cache.
Three pieces, and the first is smaller than it looks. get_transcoded
takes file_id while the table is content-keyed, but the hash is already
in scope one frame up — file_retrieval_service::try_transcode is called
where dto.content_hash is live — so it is a parameter to thread, not a
lookup to invent. Explicitly NOT by hashing original_content on the
fly, which would be a BLAKE3 over the whole file per request.
Second, the service has no BlobHandler field, so this touches the
constructor and DI; ThumbnailService hit the same ordering problem and
solved it with a per-call parameter, which is the cheaper precedent.
Third, the .skip markers stay unresolved: a negative verdict has no
bytes, so it does not fit a table whose row points at a blob. Either
leave them local and recompute per instance, or model a sentinel. It is
the only genuinely open design question left in step 10.
Answering "when is transcode_import planned": it is the remaining third
of 10(b), but it must not be next, and both reasons were learned on the
thumbnail side rather than predicted.
Format has to move into `variant` first. Transcodes are inherently
multi-format and `variant` is keyed on size alone — the same gap found
in 10c that keeps JPEG thumbnails on the sidecar. Importing before that
means migrating into a schema that cannot hold the data without
collisions across formats. One migration unblocks both, which is the
argument for doing it before either.
And step 7 must precede the import. ImageTranscodeService writes only
its file-keyed disk cache today, so the import would run against a cache
that is still growing and never reach an empty tail — exactly the trap
persist_rendered had to close for thumbnails, where one of four render
paths recorded a row and the tail could never empty.
Order: format-in-variant → step 7 → transcode_import.
Also records why the .skip markers are still open: a cached negative
verdict has no bytes, so it does not fit a table whose point is pointing
at a blob.
Step 10c. The sidecar is local disk — invisible to other instances,
uncarried by a backend migration, uncovered by any consistency job.
Reading the derived tier first is what makes that state deletable. Not
the cost it appears to be: CachedBlobBackend gives the blob read a local
disk cache and moka absorbs the repeats above it.
Not a two-line swap, for two reasons.
A derived MISS must fall through to the sidecar; the old code
terminated the lookup with `?` because it was last. While the imports
drain, most content has a sidecar and no row — terminating there would
report "no thumbnail" for nearly all of it.
And the derived tier is WebP-only. store_derived_blob writes image/webp
and keys `variant` on the size alone, with no format term, so a JPEG
request matches the WebP row and would be served the wrong codec. The
old ordering hid this because the .jpg sidecar won first. So the lookup
is gated to WebP, JPEG clients stay on the sidecar — and the sidecar
cannot be deleted for them until `variant` encodes format. That is a new
prerequisite for step 10e, recorded in the plan rather than discovered
later.
Also corrects the plan: I had written that this flip removes the
derived-hash ETag hazard. It does not. A first render still creates the
row as a side effect of producing the body, whatever the read order, so
two consecutive reads still straddle its appearance. The real fix is
resolving the ETag after generation on the 200 path — a 304 only fires
when the client already holds a validator, which implies the row exists.
That is a handler restructure, not an ordering change.
Closing out step 10(a). The remaining `None` call sites in
persist_rendered looked like an open gap; they are not reachable in
production. Both `get_thumbnail` and the path variant of
`generate_all_sizes_background` are called only from the `ThumbnailPort`
impl, and nothing holds a `dyn ThumbnailPort` — which the existing note
in get_cached_thumbnail already recorded and a grep confirms. Live
renders go through get_thumbnail_from_blob and
generate_all_sizes_background_from_blob, both of which carry a
DedupService and dual-write.
So threading a DedupService through them would be work with no runtime
effect. Recorded at each call site instead, with the condition that
matters: gaining a real caller means taking a DedupService first, or the
gap persist_rendered exists to close reopens — sidecar-only output the
import can never see, so the tail never empties and the deletion gate
never opens.
Marks 10(a) done in the plan with that caveat stated rather than
implied.
The ETag is computed BEFORE the body. On a cache miss no
content_derived_blobs row exists, so thumbnail_content_id returned the
source-keyed form — then rendering created that row, and the next
request resolved to the derived hash instead. The validator changed as a
side effect of producing the body, making every first render
immediately stale.
Latent until aaf08532: before the consolidation, the on-demand render
path never wrote a derived row, so the flip had nothing to trigger it.
Fixing one gap exposed the other.
Caught by thumbnail_etag_content_keyed.hurl — two consecutive GETs of an
unchanged file stopped revalidating to 304.
The plan already said derived-hash keying must land WITH the read-order
flip and not before; I brought it forward anyway when the attachment
case forced the attached half. This is the evidence for the constraint,
so the plan now records the attempt and why it failed rather than
leaving the note as untested caution.
The attached lookup stays — it has no such window, since an upload
writes its row synchronously before any read can observe it, and it
fixes a real collision: a copy inherits the source hash, so an original
and a copy carrying different uploaded previews would otherwise share
one validator while serving different bytes.
The flip removes the hazard for the derived half too: once that tier is
authoritative it is populated before it is consulted, so no row can
appear between two reads.
Two revisions from working through step 10.
**Deletion moves into the import jobs, not a release.** Sidecars are
local disk, so a release cannot know whether every instance has drained
— gating on "an empty tail" asks an operator to coordinate a fact
nothing reports, and there is no telling when or whether they trigger
the jobs at all. Each job unlinking what it has imported makes every
instance drain itself. Constrained three ways: verify the derived blob
reads back before unlinking (a store that reported success but landed
unreadable would otherwise take the last copy), only after the
read-order flip (or the derived tier takes its first production traffic
by accident), and opt-in, since a migration that deletes by default is
surprising. Scheduled tick rather than boot trigger — idempotent and
resumable, so periodic is safe, while walking .thumbnails/ at startup
delays readiness for nothing.
**Found while checking the dual-write assumption: it does not hold.**
store_derived_blob has ONE call site; fs::write(&thumb_path, …) has
five. get_thumbnail, generate_and_persist and
generate_all_sizes_background all persist sidecar-only. That breaks the
migration's premise rather than being untidy — on-demand renders keep
producing un-migrated state after the import runs, so the tail never
empties and the deletion gate never opens. One persist_thumbnail owning
sidecar + derived + moka is therefore a prerequisite, and it makes "stop
writing sidecars" a later one-line change instead of four edits. Noted
that ThumbnailService holds no DedupService, so it must be threaded
through.
Also corrects a claim I put in thumb_derived_import's own docs:
transcoding is NOT a later step. ImageTranscodeService exists and caches
.transcoded/{ext}/{file_id}.{ext}, so a third import is needed and it
must re-key file→content — legitimate only because a transcode is
derivable. Its .skip markers remain an open question.
The ETag shipped in fe9c4f49 is keyed on (source_hash, size, format),
which is one term short: a thumbnail is a function of those PLUS the
renderer. Change the encoder or a quality setting and identical inputs
produce different bytes under an unchanged ETag — the same staleness
class the commit fixed, one level down. It bites when an already-cached
thumbnail is re-rendered after a renderer change.
Keying on the derived blob's own hash removes the term entirely: the
ETag IS the hash of the bytes, so any output change invalidates by
construction. It is self-consistent for free, because store_derived_blob
is ON CONFLICT DO NOTHING — a re-render never displaces the stored row,
so the ETag always equals what the derived tier will serve. No renderer
version constant to remember to bump.
Records why it cannot land yet. The derived tier is read LAST by design,
so an ETag naming the derived hash would describe a tier the response
probably did not come from; sidecar and derived agree at creation but
diverge if a sidecar is re-rendered while the derived row stays pinned
by DO NOTHING. An ETag that lies about the body is worse than one that
is merely coarse. Also the tier is WebP-only (variant is the size, with
no format term) and empty for anything predating this work until
derived_import backfills.
So it lands at step 10 with the flip, keeping today's form as the
fallback for ungenerated variants and formats the tier does not hold.
The LEFT JOIN already planned for the read path returns the derived
hash in the same query, so it costs no extra round-trip.
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-13 runner tier is being phased out by GitHub — queues persistently
exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to
'cargo install --features bundled-assets' from source, Docker
--platform linux/amd64, or a Linux VM.
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in
example.env and docs/config/env.md — closes the discoverability gap
where the env var was only visible in Rust docstrings.
Also lands docs/plan/bundled-binary.md — the design record referenced
from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs,
and the Dockerfile.
derived-blobs.md — consolidates several review rounds.
BLOCKER found while building the coverage matrix: the zero-ref manifest
sweep in dedup_gc (dedup_service.rs:2574) deletes a manifest when
`ref_count <= 0 OR NOT EXISTS (SELECT 1 FROM storage.files ...)`. That
OR hardcodes "storage.files is the only thing that can reference a
manifest", so a thumbnail manifest held only by content_derived_blobs
is deleted on the next GC run, its chunks dereferenced and the bytes
reaped. Promoted to prerequisite 0 and delivery step 2. It is also the
missing half of the unreconciled chunk_manifests.ref_count: that OR is
the hack that made the drift survivable.
Adds the 13-edge consistency coverage matrix (rows 1-6 and 13 covered,
7-11 not), and records that backend_consistency needs NO change — the
backend holds chunks, which neither new table references.
BlobReferenceSource correction: `ref_level()` was wrong because
FilesReferenceSource spans both levels (chunk for legacy manifest-less
rows, Blob for CDC rows). Replaced with
`ref_count_sql(level, alias) -> Option<String>`.
Also: migration of existing sidecar content; schema trim to the columns
nothing else owns (no size/format/codec/renderer; content_type kept as
non-key since it removes today's byte-sniffing); ext-{file_id}.jpg
corrected — the client generator ships and covers PDF, which has no
server-side rasteriser, so file_attached_blobs is required rather than
deferred; uploaded_by on the shares NOT NULL/no-FK convention; the
mermaid relation map; copy and version semantics with the
copy_file_satellites consolidation; the storage.files vs file_metadata
table-identity fix; DedupService -> BlobHandler recorded as decided.
NEW hidden-system.md — retires auth.users.image TEXT (inline base64
avatar, up to 512 KiB, already worked around with a narrow projection
after it was measured detoasting M avatars per group fan-out) in favour
of *_file_id pointers at ordinary storage.files rows in one shared
hidden system drive. Because storage.files is already a
BlobReferenceSource, a file pointer costs zero new reference sources
and zero new consistency edges. Records why the alternatives lose,
the drive's required properties (hidden at enumeration, trash off,
quota exempt, boot fail-fast), per-kind visibility in code, the
secrets exclusion rule, the avatar migration, and the future object
catalogue.
Docs only; no code or schema changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enrich OxiCloud to maximise the use of `dedup` Engine
2 cases will be covered:
- blobs issues from other blobs (thumbnail automatic generation from blob)
- by filename (ex: thumbnail uploaded from users)
A local cache will be added when blobs are remote (S3 or similar)
link are checking that email matches, +email alias are normalize into email
if email is already used on another account, link is not possible
not usurpation risk as the IDP is choosen by the admin
this make OIDC compliant with the invariant binding (issuer and subject)
admin can now rename their provider without breaking
clarifing federation_kind: report the kind of federation wired not the allowed login method
hybryd login method are still allowed
Prevent services accessing directly to localstorage and prefer using an astraction layer
to expose full blob. The abstraction layer (dedup services) will cover backend storage
election (local, s3, ...), encryption, etc
This change permit audio_metadata_service, media_metadaa_service, face_indexing_service to handle
blobs without worring of the backend.
note: prefered way to handle blob is the streamed way. Some services may not have this possibility
Two chronic problems fall out:
1. **Split-brain config.** Admin edits DB via the panel; app boot ignores DB.
Migration completes; live backend hasn't moved. Admin has to remember to
copy env vars into `.env` and restart. Two sources of truth for the same
setting. Cutover is a manual multi-step flow; users routinely get it wrong.
2. **Migration data-loss window on concurrent writes.** The copy walks
`storage.blobs` in hash order. A blob whose hash is lex-lower than the
current cursor, written to source AFTER migration passed it, is never
copied to target. `passed=true, findings=0` completion does NOT guarantee
target has every blob. Silent.
3. **Migration target selection is fragile.** DTO passes the whole S3 config
at trigger time; secrets sit plaintext in `admin_settings`. Any future
pluggable-storage story compounds this (Azure, GCS, WebDAV-as-source, …).
This plan replaces the split-brain model with a single-source-of-truth
architecture:
- `.env` declares **N named storage entries** (immutable per-deploy).
- `admin_settings.storage.active_backend_name` holds ONE row — which named
entry the app currently runs on. That's the whole runtime config.
- Migration is the atomic transition from one active entry to another. Server
is put in read-only mode for the copy window; on completion, the active
pointer flips; a restart cuts over.
purpose is to design a job registry with a scheduler
thi aim to drive in the same way any services requiring execution of periodic background tasks
plugins could benefeciate it
purpose is mostly to add a normative way to implement consistency check per services
this is to edutcate implementors adding any new services
goal is to ensure data quality with oxicloud and resumable jobs by default
add env variable `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX`
which is by default:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX="@drive"`
so `/webdav/` -> points to user's personal drive (**backward compatibilit**y)
`/web/dav/@drive/{uuid|drive name}/` points to the respective drive
if admins want directly `/webdav/` pointing to list of drives they need to:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX=""`
+ ensure lock is per user (RFC 4918 §9.11)
fix: #554
what about exposing dead props into rest API
let user to store preferences, labels on resources
| Use case | What it looks like | Why dead-props help |
|---|---|---|
| Photo annotations | captions, ratings (1-5), notes per photo | already keyed by `file_id`; round-trips via WebDAV without re-implementing |
| Web-UI tags / labels | `oxi:user:tag/project=alpha`, color flags, "archived" markers | per-resource user metadata without new tables |
| Folder UI preferences | default sort, default view mode, "favourite" flag | persistent per-folder, shared across users on shared drives |
| Cross-protocol bridge | Thunderbird sets `oxi:lastsync=...` via PROPPATCH → web UI reads it via REST | one store, two surfaces — visibility goes both ways |
| Workflow / approval state | `reviewed_by=alice`, `due=2026-09-15` | ad-hoc state per resource without schema sprawl |
| Third-party integrations | external apps store scratch space per resource | lower barrier than implementing WebDAV |
Edouard Vanbelle