OIDC SSO login intermittently ended on a 403 "Invalid or expired OIDC state
— possible CSRF attack" even though the login had already succeeded
server-side.
Root cause: the (now-removed) legacy vanilla-JS frontend registered a
`/sw.js` service worker that, with navigation preload enabled, double-fetched
the top-level navigation to `/api/auth/oidc/callback`. The OIDC `state` is
single-use, so the first callback consumed it and logged the user in while
the duplicate (~0.4s later) found the state gone and returned the 403 the
browser rendered.
Backend — idempotent callback: after a successful web login, remember
`state -> exchange_code` in a short-lived (120s) cache. A duplicate callback
whose state was already consumed now replays that same redirect instead of
403-ing, returning the cached result directly without re-running the IdP code
exchange (the authorization `code` is single-use too). Keyed by the
unguessable 32-byte state, so it adds no new attack surface and fixes the 403
for everyone — including browsers still running a stale legacy service worker.
Frontend — evict the stale worker: the current SvelteKit app registers no
service worker, so fresh clients can't double-fire. But a browser that
previously loaded the legacy frontend still has `/sw.js` registered and
controlling pages (and `/sw.js` now 404s, so vendor self-cleanup is
inconsistent). killLegacyServiceWorker() runs first in the root layout's
onMount: it surgically unregisters only `/sw.js` workers, drops only the
legacy `oxicloud-cache-*` caches, and reloads once (guarded).
Fixes#510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The owner short-circuit in PgAclEngine::check ran a PK query
(SELECT user_id FROM storage.folders/files WHERE id=$1) on every authorization
check of a folder/file — the common case, since users mostly act on their own
resources. Memoise it in an owner_cache (moka, TTL 300s, 100k cap). The owner
column is immutable, so this is safe: the cache maps resource -> real owner and
can never grant a non-owner access (a different caller's owner==uid test fails
against the cached owner and falls through to grants); a hard-deleted resource
that briefly resolves to its former owner simply fails later at execution with
NotFound. The per-check sql_queries counter now increments only on a miss.
Removes 1 DB query + 1 pool-connection acquisition per owner check. Magnitude is
deployment-specific (query latency x whether the pool is contended); see
benches/ACL-OWNER-CACHE.md.
Also adds two DB perf-investigation harnesses, gated behind the `bench` feature
(need the dev Postgres; zero prod impact):
- examples/bench_db_pool.rs + benches/DB-POOL.md — pool size vs tail latency
- examples/bench_owner_cache.rs + benches/ACL-OWNER-CACHE.md — owner query vs cache
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the image crate's scalar resampler with fast_image_resize
(AVX2/SSE4.1/NEON) in a shared encode_thumbnail() helper. render_all now
converts to RGB8 once and SIMD-resizes the shared buffer per size. Lanczos3 for
downscaling, CatmullRom when upscaling (Lanczos rings on enlargement).
Also folds the duplicated path-variant generate_all_sizes_background into the
shared render path -- it had missed BOTH shrink-on-load and SIMD resizing -- so
every thumbnail path now goes through one optimised routine (no duplication).
Measured on 14 cores vs the post-1.5 state (benches/BASELINE.md):
- PNG 2.60x faster (33.6->12.9ms), GIF/WebP 1.25-1.6x: full-resolution decode
paths where the resize dominates, so SIMD helps most
- JPEG only ~7% (shrink-on-load already shrank the bitmap) but peak heap fell
another ~2.5x (17.6->7.1MB): tight RGB buffers, RGB conversion once
- quality SSIM 0.986-0.994 at identical dims (>=0.98 gate)
Thumbnails are now exactly max_dim on the long side (e.g. 400x266) vs the old
fit-within 399x266 -- a <=1px change, invisible under object-fit: cover.
Bench example gains an exact-dims quality reference + semaphore throughput table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shrink-on-load decoupled peak heap from source resolution (~18-25 MB per decode
regardless of MP), so the RAM ceiling that justified halving decode concurrency
is gone. max_concurrent_decodes() now defaults to all cores, with an
OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY override for ops tuning.
Measured on the real service path (Semaphore + spawn_blocking), 14 cores:
- 12MP: 92.7 -> 133.7 photos/s (1.44x)
- 24MP: 49.5 -> 69.9 photos/s (1.41x)
peak heap unchanged; cpus*2 yields nothing, confirming cpus is the right ceiling
for CPU-bound work (the gap to 2x is rayon oversubscription -- Task 1.7).
Adds a semaphore-bounded throughput harness (Table D) to the bench example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decode JPEGs at the smallest DCT scale (1/8·1/4·1/2·1/1) whose long axis is
still ≥ the largest needed thumbnail (800px), via jpeg-decoder, instead of a
full-resolution decode through the image crate. The full-res bitmap — the
dominant time and RAM cost — is never materialised. PNG/GIF/WebP and unusual
JPEG colour spaces (CMYK / 16-bit grey) fall back to a full decode.
Extracts the shared decode + EXIF-orientation logic into decode_oriented(),
removing the duplication that existed between render_thumbnail_from_data and
render_all_thumbnails_from_data.
Measured on 14 cores (see benches/BASELINE.md):
- render_all 1.8-2.0× faster (12MP 111->61ms, 48MP 398->203ms)
- peak heap 5.5-14.8× lower, now decoupled from source MP (~18-25MB regardless)
- saturated throughput 3-3.6× (parallel efficiency 4.9×->8.5×)
- quality SSIM 0.987-0.999 (>=0.98 gate), PSNR 47-55dB
Also adds the Phase 0 benchmark harness (gated behind the `bench` feature, zero
prod impact): deterministic image corpus (src/bench_support.rs), criterion
latency bench (benches/thumbnails.rs), and a peak-RAM/throughput/SSIM harness
(examples/bench_thumbnails_mem.rs). Baseline + before/after in benches/BASELINE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:
- GET /api/folders/paginated no-op duplicate of GET /api/folders
(discards the page arg); superseded by
the cursor-paginated /{id}/resources.
- POST /api/dedup/upload superseded by /api/files/upload, which
does the identical CDC dedup ingest.
- POST /api/people/{id}/hide the hide-person toggle was never built
into the UI (is_hidden never read).
- GET /api/admin/settings/general never called anywhere.
- GET /api/admin/settings/registration only the PUT is used; the GET had no
caller (PUT kept).
Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.
Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.
Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.
cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).
Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
to frontend/static/ so they ship with the SPA. This also fixes the
favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
to the SvelteKit /nextcloud/error route.
Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
fallback so `just dev` works without a prior build.
Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).
Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
and drop check-contrast/check-headings (coupled to the old token
taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.
Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
(superseded by /resources).
Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of folder uploads "freezing at ~95%": the global Content-Security-
Policy `script-src` was `'self'` + inline-script hashes with NO
`'wasm-unsafe-eval'`. Chromium therefore blocked `WebAssembly.instantiate`
("Wasm code generation disallowed by embedder"), so the vendored BLAKE3/FastCDC
WASM threw on instantiation — both on the main thread (instant by-hash uploads
and the batch dedup check) and inside the delta-upload worker. Every file then
fell back to a plain byte upload, and the backend logs showed 0 check-batch /
0 negotiate calls. Large files (32 MB service logs) compounded it and the
session token expired mid-upload, so the last handful failed.
- web/mod.rs: add `'wasm-unsafe-eval'` to `script-src`. WASM-only, safe variant
— does NOT enable `eval()`/`new Function()`. Restores instant uploads, delta
(sub-file dedup), and the client hashing the idempotent re-upload relies on.
- deltaUpload.ts: liveness watchdog on the delta worker. A healthy worker posts
progress sub-second; if it goes silent for 20 s it is wedged (WASM init or
chunking hung without throwing) — disable delta for this file AND every later
one so they fall straight through to a plain upload instead of each burning
the full 120 s+ delta timeout. Defense-in-depth so a broken WASM path can
never again freeze an upload for minutes.
cargo test: pass. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-uploading a partially-uploaded folder used to surface hundreds of spurious
"already exists" failures, and a file the watchdog aborted (or one the server
committed just before the client gave up) was lost.
Backend — save_file_with_blob_impl (the shared write path for both plain and
by-hash uploads): on a name conflict (23505), if the existing non-trashed file
holds byte-identical content (same folder, same name, same blob hash), return
that file as success instead of erroring. A different-content clash still
conflicts. Re-upload / re-sync becomes a clean no-op for everything already
stored — only the genuinely missing files transfer.
Frontend — uploadWithRetry: each file gets one automatic retry on a transient
failure (quota is never retried). With backend idempotency, retrying an
already-stored file is an instant no-op and a stalled/aborted file gets a real
second chance, so a folder upload self-completes instead of leaving gaps.
cargo test: 448 passed. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The normalized cursor-paginated /resources API is live and the bundled
frontend already uses it for favorites, recent, trash and grants. These five
deprecated old-format endpoints had no remaining frontend or protocol
consumers (the Nextcloud handlers call the service layer directly, not these
HTTP routes), so remove them for a uniform API and less duplicate listing
logic:
- GET /api/folders/{id}/contents → use /api/folders/{id}/resources
- GET /api/folders/{id}/contents/paginated → use /api/folders/{id}/resources
- GET /api/favorites → use /api/favorites/resources
- GET /api/recent → use /api/recent/resources
- GET /api/trash → use /api/trash/resources
Removes the HTTP handlers, their routes, OpenAPI path registrations, and the
now-dead list_folder_contents{,_paginated}_impl helpers + unused imports. The
underlying service methods (favorites_service.get_favorites,
trash_service.get_trash_items, etc.) are KEPT — the Nextcloud OCS/trashbin
handlers depend on them.
Deliberately NOT removed: GET /api/folders/{id}/listing. It is still the Files
view's primary data path and offers ETag/304 conditional caching plus
one-shot favorite/share badge sets that /resources does not yet provide;
migrating it needs a separate parity pass on /resources first.
Updates the OpenAPI structure test and the two docs that referenced the
removed paths. `cargo clippy -D warnings` clean; 443 lib tests pass; OpenAPI
regenerates with the 5 paths gone and the /resources replacements present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geotagged photos never appeared in Photos → Places because their GPS was
silently dropped. Capture-metadata extraction read GPS/camera/dimensions
only from kamadak-exif, which rejects many real-world EXIF blocks (phones,
photo editors, anything with a non-standard trailing IFD) with
`InvalidFormat("Unexpected next IFD")` and returns nothing. nom-exif —
already used to upgrade the capture date and far more lenient — parses those
same files fine, so only the date survived and latitude/longitude stayed
NULL.
nom-exif now also reads GPS (and the date) and `merge_image_metadata`
combines both sources:
- nom-exif's tz-correct date wins when present (unchanged behaviour);
- nom-exif's GPS only fills gaps kamadak left, so kamadak still wins when it
parsed the file (no regression);
- when kamadak fails outright, a record is still produced from nom-exif's
date/GPS instead of being discarded.
Existing photos can be backfilled via POST /api/admin/photos/metadata/reextract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two issues kept the Photos → Places map blank once the SPA could boot:
- CSP `worker-src 'self'` blocked MapLibre GL, which spawns its web worker
from a blob: URL, so the map never constructed. Allow `worker-src
'self' blob:` ('self' still covers same-origin workers like delta-upload).
- `checkBasemap()` trusted `res.ok`, but the SPA fallback serves index.html
(HTTP 200, text/html) for any missing path — so a missing basemap.pmtiles
read as "present", and pmtiles.js then choked on HTML ("Wrong magic
number for PMTiles archive"). Reject text/html responses so an absent
basemap falls back cleanly to the themed blank style.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global `script-src 'self'` CSP header blocked every inline <script>,
so the SvelteKit hydration bootstrap never ran and the SPA never mounted
(blank page behind the splash spinner). The anti-FOUC theme init in
app.html was blocked too.
Instead of weakening the policy with 'unsafe-inline', the backend now
builds `script-src` at startup from 'self' plus a SHA-256 hash of every
inline <script> in the served HTML shells. The policy stays strict (no
'unsafe-inline' for scripts) and the hashes are recomputed from the built
assets on each startup, so a frontend rebuild needs no header edit — even
though SvelteKit's bootstrap hash changes every build.
- web::content_security_policy builds the header; web::resolve_static_path
is extracted so serving and hashing read the exact same bytes.
- byte-exact inline-script extraction (skips src= externals), unit-tested
against a known SHA-256 vector and extraction edge cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit migration gated build.rs's static-dist/OUT_DIR generation
behind OXICLOUD_RUST_ASSETS=1 (early return), but login_v2_handler.rs
still embedded the page with
`include_str!(concat!(env!("OUT_DIR"), "/nextcloud-login.html"))`. With
OXICLOUD_RUST_ASSETS unset (the default), that file is never written to
OUT_DIR, so a clean `cargo build` failed to compile. (#489)
Migrate the page off include_str! to an askama template
(templates/nextcloud/login.html), mirroring the existing
DrivePickerTemplate in the same handler. This drops the only
compile-time dependency on the legacy build.rs pipeline, so the
OXICLOUD_RUST_ASSETS=1 CI workaround is no longer needed and is removed
from ci.yml, load-smoke.yml and load-nightly.yml.
Also fix the second failure on #489: with OXICLOUD_RUST_ASSETS=1 the
release pipeline panicked in copy_dir_recursive because `static/locales`
is now a symlink to frontend/static/locales. entry.file_type() reports
the link itself (not its target), so the symlinked directory was routed
to fs::copy and failed with "the source path is neither a regular file
nor a symlink to a regular file". Classify entries with fs::metadata,
which follows symlinks, so symlinked directories are traversed.
static/nextcloud-login.html is removed (its content moved into the
template; no other consumer) and dropped from build.rs HTML_INCLUDE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNCEMN6fC2xSmxqCVbstkd
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
Bring chroot to nextcloud capability: login on Nextcloud via username="{user}~{folder-uuid}
Doing a such login will chroot the folder folder-uuid
If user has several folder as root (parent=None), the login flow will request which
folder user want to chroot
now Drive is purely a metadata
each drive has always a root folder
this model minimize Oxicloud changes, and simplify
the Drive name is simply the folder's root's name
note: owner of Drive has more permission that an owner of the root folder
Bring the feature-rich main branch into the frontend Svelte rewrite
(PR #478, base bcn/frontend-svelte-rewrite). main moved well ahead of the
PR's branch point (b8a0018): it added the Places photo-map and People
(faces) backends, photos enhancements, the ReBAC→role-grants migration,
load tests, and more.
Conflicts resolved (4 files):
- Dockerfile: combine the explicit --bin allowlist (defence-in-depth from
main) with the SPA copy from the frontend build stage (PR).
- .github/workflows/ci.yml: keep the PR's Svelte frontend job
(svelte-check + eslint + stylelint + prettier + vitest); the legacy
static/-targeted tsc/locale/icon advisory steps don't fit the new
working-directory: frontend job and svelte-check supersedes them.
- justfile: keep both the new fe-* / dev recipes (PR) and the load-* k6
recipes (main).
- static/locales: keep the PR's symlink (-> ../frontend/static/locales);
main's new photos/people locale keys are folded into the Svelte locale
files alongside the ported views.
Backend (people/places/faces handlers, routes, DI, migrations) merged
cleanly. `cargo check --bins` passes. The new Places/People UI is not yet
in the Svelte app; that is ported in follow-up commits.
Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.
Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.
- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
decode, NMS, the closed-form (complex-number) similarity transform,
bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
and the crate builds without it). Inference runs on spawn_blocking; each
session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
compiled in and runtime+models are configured; any missing piece or load
failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.
Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 6 — the People HTTP API (mounted only when
OXICLOUD_ENABLE_FACES is on; every handler is caller-scoped):
- GET /api/people list identity clusters
- GET /api/people/{id}/photos a person's photo file ids
- PATCH /api/people/{id} name / rename a person
- POST /api/people/{id}/hide hide / unhide
- POST /api/people/merge merge two clusters
- POST /api/people/recluster re-run clustering
- DELETE /api/people/data erase all face data (opt-out)
- GET /api/people/faces/{file_id} face boxes for lightbox tagging
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 5:
- FaceIndexingService: a FileLifecycleHook that, on image upload, detects +
embeds faces in a background task and stores them. Dedup-aware (clones an
identical blob's faces instead of re-running inference), reindexes on
overwrite, and relies on the DB cascade for deletes. Completely inert when
no model is ready.
- DI: registers the hook in the FileLifecycleService chain and exposes
PeopleService in AppState — both gated on OXICLOUD_ENABLE_FACES, both
using the default no-op analyzer until the operator wires a real ONNX
model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 4:
- recluster(user): union-find connected-components over the user's face
embeddings (cosine threshold); groups of >= min_faces become persons, and
an existing person's name is preserved across reclusters. O(n^2) — fine
for moderate libraries; an ANN index is the documented scale-up.
- caller_id-scoped use cases for the HTTP layer: list_people (non-empty
clusters, cover thumbnail, most-photographed first), person_photos,
faces_for_file, rename, hide, merge, and delete_all (right to erasure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 3: FacePgRepository implements FaceRepository.
Embeddings stored/read as BYTEA (512 little-endian f32), bbox as REAL[].
Every query is user-scoped. Covers face CRUD, person CRUD (create / rename
/ cover / hide), files-for-person, and a transactional delete-all-for-user
(right to erasure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 2:
- domain: Face, Person, BoundingBox, DetectedFace (512-d embeddings).
- ports: FaceAnalyzerPort (detect + embed from raw bytes; decodes internally
so the application layer stays image/ML-crate agnostic) and FaceRepository
(user-scoped face/person persistence).
- DTOs: PersonDto, FaceBoxDto.
- NoopFaceAnalyzer — reports is_ready()==false and returns no faces, so the
whole People pipeline compiles and runs inert until the operator wires a
real ONNX-backed analyzer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
First slice of Phase 2 (People / faces):
- migration: `faces` schema with `faces.persons` and `faces.faces`.
Embeddings are stored as BYTEA (512 x f32) — no pgvector extension
dependency; similarity is computed in-app (pgvector/VectorChord is the
documented scale-up). Cascade deletes (by user and by source file)
satisfy the right to erasure.
- OXICLOUD_ENABLE_FACES feature flag, OFF by default (biometric data,
opt-in per deployment).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Three list endpoints resolved each resource with one query per id:
- GET /api/grants/incoming and /api/grants/outgoing used
join_all(ids.map(get_file)) + join_all(ids.map(get_folder)), so a single
page (limit ≤ 200) could demand ~200 concurrent connections from the
20-connection primary pool, causing acquire-timeouts and head-of-line
blocking under load.
- The NextCloud favorites REPORT (oc:filter-files) fetched get_file/
get_folder once per favorite — up to N serial round-trips per sync.
Add by-ids batch reads that mirror the existing get_file/get_folder column
mapping and NOT is_trashed filter:
- FileBlobReadRepository::get_files_by_ids / FolderDbRepository::get_folders_by_ids
(one SELECT ... WHERE id = ANY($1)), exposed as FileRetrievalService::
get_files_by_ids / FolderService::get_folders_by_ids returning DTOs.
- Both grant handlers and the favorites REPORT now issue two batch queries
total and look results up by id, preserving original order. Missing ids
(stale grants whose resource was deleted, or trashed/removed favorites)
drop out exactly as before. No auth-semantics change: these paths already
resolved ids vetted by the authorization engine / favorites table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
Adds a Places tab to the Photos section: a self-hosted MapLibre GL map of
the user's geotagged photos.
- places.js: lazy-loads vendored MapLibre + pmtiles.js on first open; draws
the server-aggregated clusters (GET /api/photos/geo) as HTML thumbnail
markers (no glyphs/sprites, no client-side clustering); pan/zoom refetches
for the new viewport; click a cluster to zoom in, or a single photo to
open it in the lightbox.
- Optional vector basemap read directly from /basemaps/basemap.pmtiles over
HTTP Range (label-light Protomaps style, light/dark aware); falls back to
a themed background when no basemap is present. ODbL attribution shown.
- "Moments | Places" sub-nav wired into the Photos section.
- enable_places now defaults on, since the route + UI are ready.
No new backend serving code: tower-http ServeDir already serves static/
with Range, so the .pmtiles basemap is just a static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
JWT access tokens freeze role/identity at login (access 1h, refresh 7d)
and validated tokens are cached for 30s. The Bearer and cookie auth paths
trusted claims.role and never re-checked the account, so demoting an admin,
or disabling/deleting an account, did not revoke access until the token
expired. The app-password path already re-read role/active from the DB;
only the JWT/cookie path had the gap.
Re-validate the caller against the live user record on the token path via
the already-cached get_user_flags (role / is_external / active), bounded by
USER_FLAGS_CACHE_TTL and invalidated eagerly on set_user_active /
change_user_role / delete_user_admin:
- middleware/user.rs: new resolve_live_role helper (+ pure decide_live_role
core) — returns the *current* role, rejects deleted (NotFound) and
deactivated accounts, and fails open on transient lookup errors (mirrors
require_internal_user). Login/refresh remain the canonical active gate.
- middleware/auth.rs: auth_middleware (Bearer + cookie) now populates
CurrentUser with the live role and rejects revoked accounts (Bearer ->
401 AccountInactive; cookie -> fall through to 401/login redirect).
require_admin emits an audit line on denial.
- middleware/admin.rs: require_admin / require_authenticated re-check the
live record, return the live role, and audit admin denials.
Downstream admin gates (dedup_handler, subject_group_handler, OCS) inherit
the live role automatically via CurrentUser / require_authenticated.
Tests: decide_live_role policy (active / demoted / deactivated / deleted /
transient fail-open) and AccountInactive -> 401.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
Phase 1 server side, gated on OXICLOUD_ENABLE_PLACES (off by default):
- migration: partial index on storage.file_metadata(longitude, latitude).
- FileBlobReadRepository::list_geo_clusters — plain-SQL grid aggregation
(no PostGIS) scoped to the caller's own non-trashed photos, returning a
centroid, count and a representative file id per non-empty cell.
- PlacesService (caller_id-scoped; user-scoped data needs no authz check,
mirroring RecentService) with a zoom→cell-size mapping.
- GET /api/photos/geo?bbox=w,s,e,n&zoom=N returning GeoCluster[]. The route
is mounted only when the Places service is present, and is registered in
the OpenAPI path list.
The map frontend (PMTiles serving + MapLibre module) is deferred pending
the basemap-sourcing and vendoring decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
list_media_files now LEFT JOINs storage.file_metadata and returns each
photo's pixel dimensions next to the sort date. The endpoint wraps FileDto
in a flattened PhotoDto carrying width/height, so the gallery can lay tiles
out at their true aspect ratio (justified layout) without a second per-file
metadata round-trip and without layout shift. FileItem gains optional
width/height. No change to FileDto or its other construction sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
remove_manifest_reference unlinked a chunk's backing file right after the
row-delete committed — the same TOCTOU the GC grace window was added to
close: a concurrent upload of identical content can re-reference (pin) the
chunk in the gap between commit and unlink, after which the deferred
unlink strands a referenced chunk with no bytes.
Route physical chunk reclamation through the single grace-protected path:
on last reference, delete the manifest and decrement its chunks (stamping
orphaned_at on the ones that reach 0), but leave the chunk rows and files
for garbage_collect() to reclaim once orphaned past the grace window. The
manifest deletion and its blob-keyed thumbnail hook stay eager.
remove_legacy_reference and cleanup_if_orphaned's legacy path are left as
eager deletes on purpose: a legacy whole-file hash can never be re-created
by an ingest (uploads are always CDC now), so there is no writer to race —
the existing "row gone ⇒ no resurrection" reasoning holds for them.
Adds an integration test asserting a CDC manifest dereference leaves
chunks orphaned-but-present, then reclaimed by a post-grace GC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
Deduplication GC (garbage_collect, Phase 2):
- Add an orphan grace period before a ref_count=0 blob's backing file is
physically deleted, mirroring git's gc.pruneExpire. New
storage.blobs.orphaned_at records when a blob last reached ref_count 0;
the delete trigger and every decrement / 0-ref insert path stamp it,
every re-reference clears it.
- Cross-check that no manifest lists the chunk and no file points at the
blob before deleting it (mirrors Phase 1's file check), so a stale
ref_count can only delay collection, never delete live content.
- Unlink the backing files with bounded parallel fan-out.
Together these close a TOCTOU where a concurrent upload of identical
content could re-reference a chunk in the window between the GC row
delete committing and the backing file being unlinked. Individual file
deletes still reclaim eagerly; only bulk empty-trash and the periodic
sweep observe the grace window.
Trash: match ErrorKind::NotFound instead of substring-matching the error
message when treating an already-deleted item as success.
Content-index worker: supervise the drain loop and restart it with
backoff after a panic, instead of letting a panic silently freeze the
search index while the dirty queue grows unbounded.
Adds migration 20260802000000_blob_gc_grace.sql and an integration test
covering the grace window and reference cross-checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
Standard CalDAV/CardDAV clients (Thunderbird, DAVx5, Apple
Calendar/Contacts) failed to connect, mounted collections read-only, or
could not discover address books, even though curl worked. Three
protocol-compliance gaps caused this:
1. Missing Basic-auth challenge on /caldav and /carddav.
The 401 returned for these surfaces carried no `WWW-Authenticate`
header (only /webdav did). Spec-compliant clients never send
credentials preemptively the way `curl -u` does — they wait for the
challenge — so Thunderbird never authenticated and failed with
"discovery failed" / 401. Extend the challenge to all DAV surfaces via
shared `is_dav_path` / `dav_basic_auth_challenge` helpers.
2. Calendars always advertised read-only.
The `current-user-privilege-set` write gate compared `owner_id`
against the literal string "current_user_id", which never matched a
real UUID, so `<D:write/>` was never emitted and clients mounted every
calendar read-only. Thread the caller's id through the CalDAV adapter
and grant write when the caller owns the calendar.
3. CardDAV discovery was incomplete.
There was no `/.well-known/carddav` route and the root PROPFIND
exposed neither `current-user-principal` nor `addressbook-home-set`,
so clients could not locate address books. Add the well-known redirect
and root/principal discovery responses mirroring the CalDAV adapter.
Adds unit tests for the auth challenge predicate, the calendar
owner/non-owner privilege split, and the CardDAV root/principal discovery
responses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016cVV9nRQjP6G6a8zbNUWMw
D-Prep (migration 20260801000002_drop_access_grants) drops
storage.access_grants and replaces it with storage.role_grants — one
row per role assignment instead of N rows per permission bundle. The
load seeder still spoke the old per-permission shape and failed every
nightly with `relation "storage.access_grants" does not exist`.
Both seeder call sites already grant the read bundle (single
permission), which maps cleanly to the `viewer` role; switching them
to the new schema is a one-row INSERT with the role name. The
conflict key drops `permission` since uniqueness is now per
(subject, resource).
prepare migration of permission to roles
this simplify drastically database (permission are now simply defined in code)
and will permit reuse of the same ReBAC engine to define owners of drives
mapping:
```
Role::Viewer => &[Permission::Read],
Role::Commenter => &[Permission::Read, Permission::Comment],
Role::Contributor => &[Permission::Read, Permission::Create],
Role::Editor => &[
Permission::Read,
Permission::Comment,
Permission::Create,
Permission::Update,
],
Role::Owner => &[
Permission::Read,
Permission::Comment,
Permission::Create,
Permission::Update,
Permission::Share,
Permission::Delete,
Permission::Manage,
],
```
Paul Meier