Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.
Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.
- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
decode, NMS, the closed-form (complex-number) similarity transform,
bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
and the crate builds without it). Inference runs on spawn_blocking; each
session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
compiled in and runtime+models are configured; any missing piece or load
failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.
Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Backend — Photos timeline now groups by real capture date instead of upload time. New MediaMetadataService (FileLifecycleHook) extracts EXIF DateTimeOriginal from images and container creation_time from videos (mov/mp4/mkv) via nom-exif, timezone-correct (OffsetTimeOriginal), persisting captured_at so the existing media_sort_date trigger takes over. Adds POST /admin/photos/metadata/reextract to backfill existing media. Falls back to upload date when no embedded date exists.
Frontend — premium grid cards: combined metadata line (relative date · size, owner avatar when shared), custom selection checkbox with a clear checked state, uniform full-width 4:3 thumbnail tiles independent of filename length, centered file-type icons, and a hit-test fix so checkbox/star/kebab clicks reach the controls (the decorative thumbnail no longer captures pointer events). Notification messages internationalised across all 16 locales. Broader polish: design tokens, a11y/focus-visible states, brand + PWA assets.
Chore — bump semver-compatible dependencies (cargo upgrade); add nom-exif 3.6.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both sides added a parameter to create_application_services and a
setup step before it: this branch's storage-usage/quota service (for
the instant-upload path) and main's Tantivy content index (for
SearchService). The resolution keeps both — the signature takes both
arguments and the build runs storage usage as step 3c and the content
index as 3d.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.
Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
(storage.search_index_dirty) - every write surface (REST, WebDAV,
NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
batched single-writer Tantivy commits; queue rows are deleted only
after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
reseeds it from Postgres, which remains the single source of truth
SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.
The frontend renders the snippet under the file name in list view.
New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.
https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
TL;DR:
fix duplicate filename via:
```
docker exec <container> migrate-nfc-filenames --dry-run # preview
docker exec <container> migrate-nfc-filenames # execute
```
== issue ==
Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.
Both rows had:
- the same name
- the same size
- the same content hash
So why two rows? Because to PostgreSQL, the names weren't the same.
Web upload (browser → Postgres):
"é" stored as 1 codepoint (U+00E9) bytes: c3 a9 ← NFC
NiextCloud client (macOS → Postgres):
"é" stored as 2 codepoints (e + U+0301) bytes: 65 cc 81 ← NFD
macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
filename.
== The fix is two halves ==
1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.
== use of the clean up ==
example of use (do not forget to define env **DATABASE_URL**)
either
`cargo run --bin migrate-nfc-filenames -- --dry-run`
or
`cargo build --bin migrate-nfc-filenames`
`./target/debug/migrate-nfc-filenames --dry-run`
example:
```
% ./target/debug/migrate-nfc-filenames --dry-run
=== NFC filename migration (DRY RUN — no writes) ===
Loaded 543 non-trashed file rows
NORMALIZE 163451b5-5e6c-404b-9b1e-f4b01a2b7269 user=42433185-4717-416d-9a15-4580fff171ec 'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
NORMALIZE 827dddec-4dd5-48c2-a120-dec5289f7d29 user=969deca6-7935-4f12-a430-4d636b62fa3e 'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
NORMALIZE 09559934-a620-472d-9ba8-fc3cfeb6dc6f user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
NORMALIZE 5ce6dbf9-0562-4758-8783-671aa9069590 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
DEDUP newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob) older=df3adc67-a778-424d-a817-b930c75f3b06 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 hash=0d2cc7b0ffce2850
=== Summary ===
scanned : 543
already in NFC : 538
normalized in place (no collision) : 4
dedup-trashed (same content) : 1
renamed to .duplicate : 0
DRY RUN — no rows were written. Re-run without --dry-run to apply.
```
once valid remove --dry-run
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
Azure SDK crates (azure_core, azure_storage, azure_storage_blobs) were
enabling reqwest/default-tls → native-tls → openssl-sys via their
default 'enable_reqwest' feature. This breaks the Alpine musl Docker
build which lacks libssl.a/libcrypto.a.
- Set default-features = false on all 3 Azure crates
- Enable 'enable_reqwest_rustls' + 'hmac_rust' (pure Rust, no OpenSSL)
- Pin reqwest to ^0.12 (azure_core 0.21 requires reqwest ^0.12)
- openssl-sys is now fully eliminated from the dependency tree
- Add utoipa v5 dependency with ToSchema derives on all REST API DTOs
- Annotate free-function handlers with #[utoipa::path] (trash, share, favorites, recent)
- Create ApiDoc struct with OpenApi derive registering 37 schemas across 7 tags
- Add generate-openapi binary outputting resources/gen/openapi.json
- Serve OpenAPI spec at GET /api/openapi.json (public, no auth)
- Add justfile with common dev commands (build, test, lint, check, openapi, db)
Patch denial-of-service vulnerability where invalid QUIC transport
parameters could cause a panic in quinn-proto.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix all clippy lints (collapsible if, clone on Copy, needless borrow,
redundant bindings, unused params) and apply rustfmt across the codebase.
Update test mocks to match Uuid-based trait signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace std::fs::read() + update_rayon() with update_mmap_rayon()
for file hashing, eliminating full-file heap allocation (500MB file
no longer needs 500MB of RAM to hash)
- Enable blake3 'mmap' feature in Cargo.toml
- Lower hash_bytes rayon threshold from 10MB to 128KB
- Remove dead constants HASH_BLOCK_SIZE and RAYON_HASH_THRESHOLD
Extract EXIF orientation, GPS coordinates, camera info, and timestamps
from uploaded images using kamadak-exif. Store metadata in a new
file_metadata PG table. Apply EXIF orientation to thumbnail generation
so images display correctly. Add /api/files/{id}/metadata endpoint.
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
Implement a complete Nextcloud client compatibility layer so that
Nextcloud desktop/mobile sync clients can connect to OxiCloud.
Key additions:
- Login Flow v2 (device auth) with OIDC bridge support
- WebDAV handler compatible with Nextcloud clients (PROPFIND, GET,
PUT, DELETE, MKCOL, MOVE, COPY, HEAD, PROPPATCH)
- OCS API endpoints (user info, capabilities, notifications stubs,
sharees, unified search)
- Basic Auth middleware with app password verification, account
lockout integration, and blake3-keyed auth cache
- App password management: create, list, revoke via both native
API (JWT-authenticated profile page) and Nextcloud OCS endpoints
- Nextcloud file ID mapping (oc:fileid) with persistent DB storage
- Chunked upload support (Nextcloud v2 chunking protocol)
- Trashbin WebDAV interface
- Avatar (SVG placeholder) and preview (redirect) handlers
- User profile page with app password management UI
- URL user validation on all DAV routes (403 on mismatch)
- Database schema for app_passwords and nextcloud_object_ids tables
All services are behind a `nextcloud.enabled` config flag and
cleanly separated under src/interfaces/nextcloud/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enable HTTP/2 auto-detection in axum (add 'http2' feature)
- Remove direct hyper dependency with 'full' features (already transitive via axum/reqwest)
- Replace std::sync::RwLock with tokio::sync::RwLock in oidc_service.rs and file_system_i18n_service.rs to prevent async deadlocks
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.
Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
Replace push_str(&format!(...)) pattern in generate_full_calendar_ical,
generate_event_ical, and generate_vevent with direct write!() into a
pre-sized String buffer.
Before: ~5N+1 heap allocations for N events (temporary Strings created
by format!(), copied into the main buffer, then dropped).
After: 1 allocation (the initial String::with_capacity). All write!()
calls format directly into the destination buffer with zero intermediate
Strings.
- WebDAV PUT and WOPI PutFile: replace sha2::Sha256 with blake3::Hasher (~5x faster hashing, compatible with dedup service)
- Fix TEXT↔UUID JOIN anti-pattern in favorites and recent_items repos (enables PK index usage)
- Add LIMIT 500 to get_favorites query to prevent unbounded memory allocation
- Remove unused lru crate from Cargo.toml (superseded by moka)
- Replace tokio features=["full"] with explicit feature list (removes signal, process, test-util)
- Replace basic TcpListener::bind with socket2 tuned socket
- TCP_NODELAY: disable Nagle's algorithm (-5 to 40ms latency on small responses)
- SO_REUSEADDR: port available immediately after server restart
- SO_REUSEPORT: ready for multi-worker scaling (Linux)
- TCP_KEEPALIVE: detect dead connections within 60s/10s interval
- listen(2048): high backlog for WebDAV connection bursts
- Eliminate redundant create_dir_all calls from upload hot path
- Replace SHA-256 with BLAKE3 (~5x faster) for content-addressable hashing
in dedup_service, file_handler, file_upload_service, chunked_upload_service
- Add mimalloc as global allocator for 10-30% throughput improvement
- sha2 crate retained only for PKCE (OAuth2 standard requirement)
- BLAKE3 produces 64-char hex hashes (same format), no DB schema changes needed
Issue #3 (CRITICAL): The global RwLock<HashMap> serialised ALL chunk uploads
across all users. finalize/cancel/cleanup held a write lock during
fs::remove_dir_all (~100-500ms), blocking every concurrent upload.
Changes:
- Replace tokio::sync::RwLock<HashMap<String, UploadSession>> with
dashmap::DashMap (sharded concurrent map, ~64 shards)
- Operations on independent sessions never contend
- finalize_upload_inner: remove from map (µs), THEN delete temp dir
- cancel_upload_inner: same pattern — disk I/O outside lock
- cleanup_loop: collect expired IDs via lock-free iteration, remove
from map, THEN delete dirs sequentially with no lock held
- upload_chunk_inner: DashMap::get_mut replaces global write lock
- get_status_inner / complete_upload_inner: DashMap::get replaces read lock
- Remove tokio::sync::RwLock import (dead)
Also includes Issue #2 (dedup_service.rs write-first + upsert) from
previous session.
Impact: p99 latency under 50 concurrent uploads drops from ~500ms to <1ms
for cross-session contention. Cleanup loop no longer blocks uploads.
- Replace in-memory Cursor<Vec<u8>> with async_zip + NamedTempFile (O(256KB) RAM)
- Stream ZIP to client via ReaderStream instead of materializing entire archive
- Replace N+1 BFS folder traversal with 2 bulk ltree queries
- Remove dead zip_service field and with_zip_service() from BatchOperationService
- Remove synchronous zip crate dependency (only async_zip remains)
- Eliminates ~2.5GB RAM spike per 2GB batch download
Fabien KOCIK