plan:
```
What step 5 is actually doing
Goal: stop writing to storage.files.user_id / storage.folders.user_id on new INSERTs. This is a prerequisite for step 6 (dropping the column entirely). The column has existed since the pre-D0 single-owner era; every read that used to key on it has already been migrated to drive-membership grants over the last N days.
What has to change together for this to be safe:
┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Piece │ What changes │ Why │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Swap storage.files uniqueness indexes from (folder_id, name, user_id) to (drive_id, folder_id, name) │ Otherwise, two new rows with user_id = NULL would both be allowed (PG treats NULLs as distinct) — │
│ (a) │ │ uniqueness silently breaks │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ ALTER user_id DROP NOT NULL on both tables │ Otherwise, dropping the INSERT bind violates NOT NULL and every write 500s │
│ (b) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Drop dead user_id-leading indexes │ Cheap cleanup — nothing scans them anymore │
│ (c) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ PL/pgSQL (d) │ Rewrite storage.copy_folder_tree without user_id in the INSERT column list │ Cross-drive copy runs entirely in SQL, needs the same treatment │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (e) │ ~12 INSERT column-list drops across folder_db_repository, file_blob_write_repository, drive_pg_repository, dedup_service, │ Actual write path │
│ │ folder_service, load-seed.rs │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (f) │ Simplify folder_db_repository::create_folder's parent lookup to only fetch drive_id (was fetching (user_id, drive_id)) │ It's fetching a value it no longer needs │
└──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
Stamp `orphaned_at = now()` on the ref-0 row so it sits inside the
GC grace window for the duration of this test. Without it,
`orphaned_at IS NULL` is treated by `garbage_collect` as
"pre-migration, immediately reapable" — and any sibling test in
the shared pool that calls `garbage_collect()` (e.g.
`garbage_collect_respects_grace_and_cross_checks`) would race
with the pin below and delete the row first.
call fire_blob_hooks to respect lifecycle
dedup_service::garbage_collect_with_grac must call fire_blob_hooks() once blob are dropped
so other services like thumbnail can proceed to their cleanup
renable thumbnail test, ensure that blob lifecycle correctly
trigger thumbnail cleanup on blob deletion
need to call `/api/admin/internal/trigger-gc?force=true`
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
This reverts ee51b32. The create_new change showed no measurable throughput
benefit — three 9-rep interleaved runs on the same ext4 device swung −12%..+21%
at the 256 KiB CDC size (a negative stat on a warm dentry cache is ~µs, below
the shared-disk noise floor). Applying the same "no change without a measured
win" bar used for the pool-sizing revert: the idiom/TOCTOU angle is real but the
race is already prevented upstream by the PG pin-or-classify serialisation, so
it's defence-in-depth only — not enough to keep an unmeasured change. Reverts
the production change, the bench, and its doc together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the try_exists(stat) + File::create(O_CREAT|O_TRUNC) pair with a single
OpenOptions::create_new (O_CREAT|O_EXCL), treating AlreadyExists as the existing
idempotent skip. One metadata syscall per new chunk instead of two (each a
spawn_blocking round-trip), and O_EXCL closes the check-then-create TOCTOU the
old pair left open (a racing writer could be truncated).
Honest measurement caveat (benches/BLOB-WRITE.md): the wall-clock throughput
effect is BELOW the noise floor of the test environment — three 9-rep
interleaved runs on the same ext4 device swing −12%..+21% at the 256 KiB CDC
size, because a negative stat on a warm dentry cache is ~µs, dwarfed by the
chunk's create+write+flush. So this is justified as a code-quality / correctness
change (canonical idiom, strictly fewer syscalls, closes a TOCTOU, zero
downside), NOT as a benchmarked perf win.
The sibling idea — reusing the File handle for the fsync sweep — is deliberately
NOT done: sync_blobs is a single end-of-stream sweep over all the upload's new
hashes, so retaining handles would hold thousands of FDs open (>ulimit) on a
large upload. The re-open sweep is a deliberate FD-frugal design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
This reverts the image-pool migration (commit 5629ba6). The bench
(bench_pool_concurrency / POOL-CONCURRENCY.md) measured the one pool it could
isolate — the thumbnail decode semaphore — and found flat throughput, p99 AND
peak RSS (137 MiB) from K=1..16: shrink-on-load already makes each decode
RAM-cheap, so sizing it to the CFS quota gains nothing measurable. Adding code
without a measured benefit isn't worth it.
Kept: the effective_parallelism() helper (it has a *measured* win in the Tokio
runtime — benches/RUNTIME.md) and the benchmark itself (reusable). The ffmpeg
video fan-out has a plausible a-priori case (one OS process per permit) but is
left as a future, deliberately-measured change rather than shipped on
speculation. Doc updated to record the decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The thumbnail decode semaphore, the transcode rayon pool, and the ffmpeg
video-thumbnail fan-out all sized from std::thread::available_parallelism(),
which honours CPU affinity but ignores the CFS bandwidth quota (--cpus /
cgroup cpu.max). Under a container quota they therefore permit one CPU-heavy
task per *host* core onto cores the scheduler can't grant — the same
over-subscription the runtime worker pool had.
Switch all three to common::runtime::effective_parallelism() (= min(affinity,
CFS quota)), and fix the doc comments that wrongly claimed available_parallelism
respects cgroup quotas. No change off-quota (effective == available there); the
env overrides (OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY,
OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The local backend inherited the trait's conservative read_prefetch() = 1
(strictly sequential chunk reassembly), while S3/Azure already use 8. The
prior rationale was that concurrent opens over scattered content-addressed
chunk files turn one sequential read into competing random I/O ("slower
cold"). Benchmarked that assumption with examples/bench_blob_prefetch
(sweeps the buffered(N) depth over a real LocalBlobBackend under disk-bound
vs network-bound consumers and warm vs cold page cache).
Result on SSD-class storage (median MB/s vs N=1):
warm disk-bound N=2 +11.8% N=8 +3.9% N=16 -4.4%
cold disk-bound N=2 +7.2% (no cold regression on SSD)
network-bound (throttled) ~0% at any N — the socket, not the disk, caps it
So N=8 is wrong for local (leaves gain on the table, risks HDD seek thrash)
and the network-bound win the analysis assumed doesn't materialize: buffered()
here overlaps the per-chunk File::open (cheap on local disk), not the data
read. N=2 captures most of the disk-bound gain — which covers localhost/LAN
downloads AND the internal blob reads that drain as fast as the disk delivers
(thumbnail render, transcode, ZIP export, content extraction) — at the lowest
fan-out. Env-tunable via OXICLOUD_LOCAL_READ_PREFETCH (set 1 on seek-bound
HDDs to restore the old behaviour; raise on fast NVMe). Signature unchanged,
so all ~16 LocalBlobBackend::new call sites are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
read_blob_bytes read the same storage.chunk_manifests PK row twice per
full-blob read — blob_size (SELECT total_size) then read_blob_stream
(SELECT chunk_hashes) — even though both columns live in one row. Fold
them into a single `SELECT chunk_hashes, total_size` and share the chunk
stream builder via a new stream_chunks helper. The legacy (no-manifest)
path is unchanged. Output is identical; the read just costs one fewer DB
round-trip.
Benchmark (examples/bench_blob_manifest.rs, isolates the manifest lookup
against the real Postgres): ~1.9x throughput and p50/p99 roughly halved on
that sub-step; the win is the removed round-trip under pool pressure during
upload bursts. Note this is the manifest sub-step only — end-to-end
read_blob_bytes is dominated by the actual chunk reads, and it is a
background path (thumbnail generation / EXIF / indexing), not normal gallery
serving. Methodology + honest framing in benches/BLOB-MANIFEST.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thumbnails are now generated eagerly as lossy WebP (the primary codec) and
served to clients that advertise `Accept: image/webp`; JPEG is kept as a lazy
fallback for older clients and NextCloud, generated on first request and then
cached like WebP.
- ThumbnailFormat{Webp,Jpeg} enum threaded through encode/render/generate, the
on-disk path ({hash}.webp / {hash}.jpg), the moka cache key
(file_id, size, format), and cleanup (both formats removed).
- file_handler: parse Accept -> format, format-keyed ETag, `Vary: Accept` on
every response (incl. 304) so shared caches never serve the wrong codec;
Content-Type is byte-sniffed (infer) so it always matches the bytes.
- preview_handler (NextCloud) pins JPEG.
- webp = "0.3" (vendored libwebp via cc, no system dependency).
WEBP_QUALITY=82, chosen via a quality sweep (bench Table E1): SSIM within
~0.005 of JPEG q80 (imperceptible at thumbnail scale) for ~62% fewer bytes. On
the photo-realistic bench corpus the full set (3 sizes x 3 photos) drops 65.6%
(213->73 KB); real photos with edges/text land nearer ~25-40%. Encode is +5ms,
paid once in the eager background generator (off the request path).
The bench corpus is now photo-realistic (per-channel sums of low-frequency
sinusoids) instead of white noise, which had distorted codec byte ratios.
Methodology + numbers in benches/WEBP.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The owner short-circuit in PgAclEngine::check ran a PK query
(SELECT user_id FROM storage.folders/files WHERE id=$1) on every authorization
check of a folder/file — the common case, since users mostly act on their own
resources. Memoise it in an owner_cache (moka, TTL 300s, 100k cap). The owner
column is immutable, so this is safe: the cache maps resource -> real owner and
can never grant a non-owner access (a different caller's owner==uid test fails
against the cached owner and falls through to grants); a hard-deleted resource
that briefly resolves to its former owner simply fails later at execution with
NotFound. The per-check sql_queries counter now increments only on a miss.
Removes 1 DB query + 1 pool-connection acquisition per owner check. Magnitude is
deployment-specific (query latency x whether the pool is contended); see
benches/ACL-OWNER-CACHE.md.
Also adds two DB perf-investigation harnesses, gated behind the `bench` feature
(need the dev Postgres; zero prod impact):
- examples/bench_db_pool.rs + benches/DB-POOL.md — pool size vs tail latency
- examples/bench_owner_cache.rs + benches/ACL-OWNER-CACHE.md — owner query vs cache
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the image crate's scalar resampler with fast_image_resize
(AVX2/SSE4.1/NEON) in a shared encode_thumbnail() helper. render_all now
converts to RGB8 once and SIMD-resizes the shared buffer per size. Lanczos3 for
downscaling, CatmullRom when upscaling (Lanczos rings on enlargement).
Also folds the duplicated path-variant generate_all_sizes_background into the
shared render path -- it had missed BOTH shrink-on-load and SIMD resizing -- so
every thumbnail path now goes through one optimised routine (no duplication).
Measured on 14 cores vs the post-1.5 state (benches/BASELINE.md):
- PNG 2.60x faster (33.6->12.9ms), GIF/WebP 1.25-1.6x: full-resolution decode
paths where the resize dominates, so SIMD helps most
- JPEG only ~7% (shrink-on-load already shrank the bitmap) but peak heap fell
another ~2.5x (17.6->7.1MB): tight RGB buffers, RGB conversion once
- quality SSIM 0.986-0.994 at identical dims (>=0.98 gate)
Thumbnails are now exactly max_dim on the long side (e.g. 400x266) vs the old
fit-within 399x266 -- a <=1px change, invisible under object-fit: cover.
Bench example gains an exact-dims quality reference + semaphore throughput table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shrink-on-load decoupled peak heap from source resolution (~18-25 MB per decode
regardless of MP), so the RAM ceiling that justified halving decode concurrency
is gone. max_concurrent_decodes() now defaults to all cores, with an
OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY override for ops tuning.
Measured on the real service path (Semaphore + spawn_blocking), 14 cores:
- 12MP: 92.7 -> 133.7 photos/s (1.44x)
- 24MP: 49.5 -> 69.9 photos/s (1.41x)
peak heap unchanged; cpus*2 yields nothing, confirming cpus is the right ceiling
for CPU-bound work (the gap to 2x is rayon oversubscription -- Task 1.7).
Adds a semaphore-bounded throughput harness (Table D) to the bench example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decode JPEGs at the smallest DCT scale (1/8·1/4·1/2·1/1) whose long axis is
still ≥ the largest needed thumbnail (800px), via jpeg-decoder, instead of a
full-resolution decode through the image crate. The full-res bitmap — the
dominant time and RAM cost — is never materialised. PNG/GIF/WebP and unusual
JPEG colour spaces (CMYK / 16-bit grey) fall back to a full decode.
Extracts the shared decode + EXIF-orientation logic into decode_oriented(),
removing the duplication that existed between render_thumbnail_from_data and
render_all_thumbnails_from_data.
Measured on 14 cores (see benches/BASELINE.md):
- render_all 1.8-2.0× faster (12MP 111->61ms, 48MP 398->203ms)
- peak heap 5.5-14.8× lower, now decoupled from source MP (~18-25MB regardless)
- saturated throughput 3-3.6× (parallel efficiency 4.9×->8.5×)
- quality SSIM 0.987-0.999 (>=0.98 gate), PSNR 47-55dB
Also adds the Phase 0 benchmark harness (gated behind the `bench` feature, zero
prod impact): deterministic image corpus (src/bench_support.rs), criterion
latency bench (benches/thumbnails.rs), and a peak-RAM/throughput/SSIM harness
(examples/bench_thumbnails_mem.rs). Baseline + before/after in benches/BASELINE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geotagged photos never appeared in Photos → Places because their GPS was
silently dropped. Capture-metadata extraction read GPS/camera/dimensions
only from kamadak-exif, which rejects many real-world EXIF blocks (phones,
photo editors, anything with a non-standard trailing IFD) with
`InvalidFormat("Unexpected next IFD")` and returns nothing. nom-exif —
already used to upgrade the capture date and far more lenient — parses those
same files fine, so only the date survived and latitude/longitude stayed
NULL.
nom-exif now also reads GPS (and the date) and `merge_image_metadata`
combines both sources:
- nom-exif's tz-correct date wins when present (unchanged behaviour);
- nom-exif's GPS only fills gaps kamadak left, so kamadak still wins when it
parsed the file (no regression);
- when kamadak fails outright, a record is still produced from nom-exif's
date/GPS instead of being discarded.
Existing photos can be backfilled via POST /api/admin/photos/metadata/reextract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.
Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.
- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
decode, NMS, the closed-form (complex-number) similarity transform,
bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
and the crate builds without it). Inference runs on spawn_blocking; each
session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
compiled in and runtime+models are configured; any missing piece or load
failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.
Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 5:
- FaceIndexingService: a FileLifecycleHook that, on image upload, detects +
embeds faces in a background task and stores them. Dedup-aware (clones an
identical blob's faces instead of re-running inference), reindexes on
overwrite, and relies on the DB cascade for deletes. Completely inert when
no model is ready.
- DI: registers the hook in the FileLifecycleService chain and exposes
PeopleService in AppState — both gated on OXICLOUD_ENABLE_FACES, both
using the default no-op analyzer until the operator wires a real ONNX
model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 2:
- domain: Face, Person, BoundingBox, DetectedFace (512-d embeddings).
- ports: FaceAnalyzerPort (detect + embed from raw bytes; decodes internally
so the application layer stays image/ML-crate agnostic) and FaceRepository
(user-scoped face/person persistence).
- DTOs: PersonDto, FaceBoxDto.
- NoopFaceAnalyzer — reports is_ready()==false and returns no faces, so the
whole People pipeline compiles and runs inert until the operator wires a
real ONNX-backed analyzer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
remove_manifest_reference unlinked a chunk's backing file right after the
row-delete committed — the same TOCTOU the GC grace window was added to
close: a concurrent upload of identical content can re-reference (pin) the
chunk in the gap between commit and unlink, after which the deferred
unlink strands a referenced chunk with no bytes.
Route physical chunk reclamation through the single grace-protected path:
on last reference, delete the manifest and decrement its chunks (stamping
orphaned_at on the ones that reach 0), but leave the chunk rows and files
for garbage_collect() to reclaim once orphaned past the grace window. The
manifest deletion and its blob-keyed thumbnail hook stay eager.
remove_legacy_reference and cleanup_if_orphaned's legacy path are left as
eager deletes on purpose: a legacy whole-file hash can never be re-created
by an ingest (uploads are always CDC now), so there is no writer to race —
the existing "row gone ⇒ no resurrection" reasoning holds for them.
Adds an integration test asserting a CDC manifest dereference leaves
chunks orphaned-but-present, then reclaimed by a post-grace GC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
Deduplication GC (garbage_collect, Phase 2):
- Add an orphan grace period before a ref_count=0 blob's backing file is
physically deleted, mirroring git's gc.pruneExpire. New
storage.blobs.orphaned_at records when a blob last reached ref_count 0;
the delete trigger and every decrement / 0-ref insert path stamp it,
every re-reference clears it.
- Cross-check that no manifest lists the chunk and no file points at the
blob before deleting it (mirrors Phase 1's file check), so a stale
ref_count can only delay collection, never delete live content.
- Unlink the backing files with bounded parallel fan-out.
Together these close a TOCTOU where a concurrent upload of identical
content could re-reference a chunk in the window between the GC row
delete committing and the backing file being unlinked. Individual file
deletes still reclaim eagerly; only bulk empty-trash and the periodic
sweep observe the grace window.
Trash: match ErrorKind::NotFound instead of substring-matching the error
message when treating an already-deleted item as success.
Content-index worker: supervise the drain loop and restart it with
backoff after a panic, instead of letting a panic silently freeze the
search index while the dirty queue grows unbounded.
Adds migration 20260802000000_blob_gc_grace.sql and an integration test
covering the grace window and reference cross-checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
prepare migration of permission to roles
this simplify drastically database (permission are now simply defined in code)
and will permit reuse of the same ReBAC engine to define owners of drives
mapping:
```
Role::Viewer => &[Permission::Read],
Role::Commenter => &[Permission::Read, Permission::Comment],
Role::Contributor => &[Permission::Read, Permission::Create],
Role::Editor => &[
Permission::Read,
Permission::Comment,
Permission::Create,
Permission::Update,
],
Role::Owner => &[
Permission::Read,
Permission::Comment,
Permission::Create,
Permission::Update,
Permission::Share,
Permission::Delete,
Permission::Manage,
],
```
Edouard Vanbelle