add:
- OXICLOUD_REGISTRATION_ALLOWED_EMAIL_DOMAINS to specify list of domains allowing a self registration
- OXICLOUD_REQUIRE_VERIFIED_EMAIL=true|false
- OXICLOUD_AUTH_METHODS=password,magic_link (login methods, OIDC is on top of this)
- OXICLOUD_AUTH_POLICIES=permit_magic_link_for_password_users (OIDC is on top)
Drops the boolean is_personal() wrapper in favor of matching
DriveKind::Personal/Shared at the two call sites, matching the
exhaustive-match convention already used for DriveKind elsewhere
(as_str, parse, DriveKindDto::from).
resolve_quota only ever reported the caller's personal envelope,
ignoring the drive_id already resolved at every PROPFIND call site —
shared drives with their own quota showed the wrong numbers. Adds
AppState::resolve_webdav_quota, shared by both WebDAV surfaces:
nil drive_id or personal drive -> account envelope, shared drive ->
its own storage.drives quota/used_bytes.
Also adds quota-used-bytes/quota-available-bytes to the NextCloud-
compatible surface, which previously had no RFC 4331 support at all.
Registers webdav_quota_properties.hurl and the new
nc_webdav_quota_properties.hurl in tests/api/run.sh — neither was
wired into the suite before this change.
add env variable `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX`
which is by default:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX="@drive"`
so `/webdav/` -> points to user's personal drive (**backward compatibilit**y)
`/web/dav/@drive/{uuid|drive name}/` points to the respective drive
if admins want directly `/webdav/` pointing to list of drives they need to:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX=""`
+ ensure lock is per user (RFC 4918 §9.11)
fix: #554
add `include_in_photo_index` and `include_in_music_index` policies
both true for default personal drive
photo is implemented
music is not yet implemented
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
This reverts the image-pool migration (commit 5629ba6). The bench
(bench_pool_concurrency / POOL-CONCURRENCY.md) measured the one pool it could
isolate — the thumbnail decode semaphore — and found flat throughput, p99 AND
peak RSS (137 MiB) from K=1..16: shrink-on-load already makes each decode
RAM-cheap, so sizing it to the CFS quota gains nothing measurable. Adding code
without a measured benefit isn't worth it.
Kept: the effective_parallelism() helper (it has a *measured* win in the Tokio
runtime — benches/RUNTIME.md) and the benchmark itself (reusable). The ffmpeg
video fan-out has a plausible a-priori case (one OS process per permit) but is
left as a future, deliberately-measured change rather than shipped on
speculation. Doc updated to record the decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The thumbnail decode semaphore, the transcode rayon pool, and the ffmpeg
video-thumbnail fan-out all sized from std::thread::available_parallelism(),
which honours CPU affinity but ignores the CFS bandwidth quota (--cpus /
cgroup cpu.max). Under a container quota they therefore permit one CPU-heavy
task per *host* core onto cores the scheduler can't grant — the same
over-subscription the runtime worker pool had.
Switch all three to common::runtime::effective_parallelism() (= min(affinity,
CFS quota)), and fix the doc comments that wrongly claimed available_parallelism
respects cgroup quotas. No change off-quota (effective == available there); the
env overrides (OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY,
OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the bare #[tokio::main] with an explicit runtime::Builder so both
pools are sized, logged at startup, and operator-tunable.
#[tokio::main] hides two defaults that misbehave under container limits:
- worker_threads = available_parallelism(), which honours CPU affinity
(sched_getaffinity: cpuset, taskset) but IGNORES the CFS bandwidth quota
(docker --cpus, cgroup v2 cpu.max, v1 cpu.cfs_quota_us). On a 2-core-quota
container on a 64-core host it spawns 64 workers time-slicing 2 cores.
- max_blocking_threads = 512, a multi-GB RSS blast radius for this heavy
spawn_blocking user (thumbnails, transcode, zip, PDF/text extraction,
Argon2 ~19 MB/hash).
New common::runtime module folds the CFS quota back in: effective_parallelism()
= min(available_parallelism, cgroup quota). runtime_pool_sizes() defaults
workers to that and caps the blocking pool at max(32, 8*workers), both
overridable via OXICLOUD_WORKER_THREADS (or TOKIO_WORKER_THREADS) and
OXICLOUD_MAX_BLOCKING_THREADS. The cgroup v1/v2 parsers are pure + unit-tested.
Note: this corrects the premise that "rayon respects the quota, tokio doesn't"
— the image/rayon pools also use available_parallelism(), so they over-spawn
under a CFS quota too; effective_parallelism() is the reusable fix. Unset env
on an uncontended host reproduces the previous worker count exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:
- GET /api/folders/paginated no-op duplicate of GET /api/folders
(discards the page arg); superseded by
the cursor-paginated /{id}/resources.
- POST /api/dedup/upload superseded by /api/files/upload, which
does the identical CDC dedup ingest.
- POST /api/people/{id}/hide the hide-person toggle was never built
into the UI (is_hidden never read).
- GET /api/admin/settings/general never called anywhere.
- GET /api/admin/settings/registration only the PUT is used; the GET had no
caller (PUT kept).
Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.
Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.
Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.
cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).
Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
to frontend/static/ so they ship with the SPA. This also fixes the
favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
to the SvelteKit /nextcloud/error route.
Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
fallback so `just dev` works without a prior build.
Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).
Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
and drop check-contrast/check-headings (coupled to the old token
taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.
Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
(superseded by /resources).
Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
now Drive is purely a metadata
each drive has always a root folder
this model minimize Oxicloud changes, and simplify
the Drive name is simply the folder's root's name
note: owner of Drive has more permission that an owner of the root folder
Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.
Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.
- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
decode, NMS, the closed-form (complex-number) similarity transform,
bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
and the crate builds without it). Inference runs on spawn_blocking; each
session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
compiled in and runtime+models are configured; any missing piece or load
failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.
Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 5:
- FaceIndexingService: a FileLifecycleHook that, on image upload, detects +
embeds faces in a background task and stores them. Dedup-aware (clones an
identical blob's faces instead of re-running inference), reindexes on
overwrite, and relies on the DB cascade for deletes. Completely inert when
no model is ready.
- DI: registers the hook in the FileLifecycleService chain and exposes
PeopleService in AppState — both gated on OXICLOUD_ENABLE_FACES, both
using the default no-op analyzer until the operator wires a real ONNX
model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
First slice of Phase 2 (People / faces):
- migration: `faces` schema with `faces.persons` and `faces.faces`.
Embeddings are stored as BYTEA (512 x f32) — no pgvector extension
dependency; similarity is computed in-app (pgvector/VectorChord is the
documented scale-up). Cascade deletes (by user and by source file)
satisfy the right to erasure.
- OXICLOUD_ENABLE_FACES feature flag, OFF by default (biometric data,
opt-in per deployment).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Adds a Places tab to the Photos section: a self-hosted MapLibre GL map of
the user's geotagged photos.
- places.js: lazy-loads vendored MapLibre + pmtiles.js on first open; draws
the server-aggregated clusters (GET /api/photos/geo) as HTML thumbnail
markers (no glyphs/sprites, no client-side clustering); pan/zoom refetches
for the new viewport; click a cluster to zoom in, or a single photo to
open it in the lightbox.
- Optional vector basemap read directly from /basemaps/basemap.pmtiles over
HTTP Range (label-light Protomaps style, light/dark aware); falls back to
a themed background when no basemap is present. ODbL attribution shown.
- "Moments | Places" sub-nav wired into the Photos section.
- enable_places now defaults on, since the route + UI are ready.
No new backend serving code: tower-http ServeDir already serves static/
with Range, so the .pmtiles basemap is just a static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 1 server side, gated on OXICLOUD_ENABLE_PLACES (off by default):
- migration: partial index on storage.file_metadata(longitude, latitude).
- FileBlobReadRepository::list_geo_clusters — plain-SQL grid aggregation
(no PostGIS) scoped to the caller's own non-trashed photos, returning a
centroid, count and a representative file id per non-empty cell.
- PlacesService (caller_id-scoped; user-scoped data needs no authz check,
mirroring RecentService) with a zoom→cell-size mapping.
- GET /api/photos/geo?bbox=w,s,e,n&zoom=N returning GeoCluster[]. The route
is mounted only when the Places service is present, and is registered in
the OpenAPI path list.
The map frontend (PMTiles serving + MapLibre module) is deferred pending
the basemap-sourcing and vendoring decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Dionisio Pozo