diff --git a/docs/plan/Photos-People-Places.md b/docs/plan/Photos-People-Places.md new file mode 100644 index 00000000..95644012 --- /dev/null +++ b/docs/plan/Photos-People-Places.md @@ -0,0 +1,391 @@ +# Plan: Photos Evolution — Gallery, Places (map) & People (faces) + +## Context + +The Photos view (`static/js/features/library/photos.js` + `photosLightbox.js`) is a +date-grouped timeline with infinite scroll, multi-select and a lightbox. The backend +already extracts and stores per-photo EXIF — including **GPS latitude/longitude** — in +`storage.file_metadata` (`src/infrastructure/services/exif_service.rs`, +`media_metadata_service.rs`), and serves the timeline via `GET /api/photos` +(`src/interfaces/api/handlers/photos_handler.rs` → `list_media_files` in +`src/infrastructure/repositories/pg/file_blob_read_repository.rs`). + +This plan adds, in three phases: + +0. **Gallery polish** — performance + modern UX (timeline virtualization already landed). +1. **Places** — a map of geotagged photos. *Most of the data already exists.* +2. **People** — face detection, embedding, identity clustering ("like Apple/Google Photos"). + +All work follows OxiCloud conventions: hexagonal layering, **AuthZ enforced only in the +application service layer** via `*_with_perms(caller_id)` methods calling +`AuthorizationEngine::require(...)`, **audit logging on every denial** +(`target: "audit"`), feature flags (`OXICLOUD_ENABLE_*`), native `UUID` columns, sqlx +migrations, and a **vanilla-JS / vanilla-CSS** frontend (design tokens from +`static/css/base/variables.css`, JSDoc-typed, BEM). + +> ⚠️ New dependencies (JS libraries to vendor, Rust crates, the `pgvector`/`vectorchord` +> Postgres extension, and runtime-downloaded ML models) need explicit sign-off — see +> **§Vendoring & dependencies** and **§Open decisions**. Per repo rules: never hand-edit +> `Cargo.lock` (use `cargo add`), and don't introduce JS frameworks. + +--- + +## Research summary (the decisions these phases encode) + +**Map (no third-party APIs, self-host, extreme perf):** +- **Engine:** MapLibre GL JS v5 (BSD-3, WebGL2, vendorable UMD, no framework). +- **Basemap:** self-hosted **Protomaps `.pmtiles`** (single file) served by Axum via the + **`pmtiles`** Rust crate over HTTP Range — OxiCloud serves its own basemap. Global + z0–6 ≈ 60 MB; regional extracts on demand; planet ≈ 120 GB. +- **Clustering:** client-side **Supercluster** (MapLibre `cluster: true`, in a web worker) + up to ~100k points; beyond that, **plain-SQL grid/geohash aggregation** by zoom+bbox — + **no PostGIS needed** (only `pg_trgm`/`ltree`/`citext` are enabled today). +- **Gotchas:** self-host glyphs+sprites (not the Protomaps CDN); ODbL attribution + "Protomaps © OpenStreetMap" is mandatory; dark-mode via `@protomaps/basemaps` flavors. + +**Faces (self-host, precision, CPU-first):** +- **Runtime:** **`ort`** (ONNX Runtime). `candle` can't run SCRFD/RetinaFace (missing + `Resize` op); `tract` is the pure-Rust fallback for a single static binary. +- **Licensing landmine:** no permissive high-accuracy face-recognition checkpoint exists. + InsightFace `buffalo_l` (IJB-C ~97.3) and EdgeFace weights are **non-commercial**. +- **Recommended (immich/PhotoPrism pattern):** **download** SCRFD + `buffalo_l` weights at + runtime (not committed); personal self-hosted use is non-commercial-compliant. Offer a + fully-permissive fallback (RetinaFace-MobileNet0.25 **MIT** + a self-retrained + EdgeFace/GhostFaceNet embedder, ~94 IJB-C, ~10× smaller). +- **Storage/clustering:** embeddings in Postgres via **pgvector** (HNSW, 512-d), growth + path to **VectorChord**; **threshold / connected-components incremental clustering** + (immich-style), not Approximate Rank-Order; ANN + exact re-rank; quality gating + (det-score ≥0.7, face ≥50–80px, blur); `minFaces ≥3` to promote a cluster to a Person. +- **Privacy:** biometric data (GDPR Art. 9) → **opt-in, OFF by default, per-user + isolation, cascade-delete**, all local. + +--- + +## Phase 0 — Gallery polish + +### Execution order + +#### 0.1 Timeline virtualization — ✅ DONE (commit `081b2b6`) +Each date-group is a `
` whose grid is materialized only near the viewport. +**Remaining:** browser smoke-test, then it's closed. + +#### 0.2 Expose image dimensions on the timeline (enables justified layout, kills CLS) +**`src/application/dtos/file_dto.rs`** — add to `FileDto`: +```rust +#[serde(skip_serializing_if = "Option::is_none")] +pub width: Option, +#[serde(skip_serializing_if = "Option::is_none")] +pub height: Option, +``` +**`src/infrastructure/repositories/pg/file_blob_read_repository.rs`** — in the +`list_media_files` query, `LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id` and +select `fm.width, fm.height`; map into the new fields. +**`static/js/core/types.js`** — add `width?`/`height?` to `FileItem` (already on +`FileMetadata`). + +#### 0.3 Justified rows layout (modern, aspect-preserving) +**`static/js/features/library/photos.js`** — add a `layoutMode: 'square' | 'justified'` +toggle in the toolbar. In justified mode, replace the CSS grid with a row-packing pass +(target row height ~180–220px, distribute by aspect ratio = `width/height`, fallback 1:1 +when dimensions are absent). Keep the existing virtualization: row-pack **within each +materialized group**, so it composes with section materialize/dematerialize. +**`static/css/views/photos.css`** — `.photos-grid--justified` (flex rows) variant. + +#### 0.4 Lightbox upgrades +**`static/js/features/library/photosLightbox.js`** +- **Zoom/pan** (wheel + pinch) and **mobile swipe** for prev/next. +- **Info panel** (toggle) showing EXIF from `/api/files/{id}/metadata`. +- **Map pin** (resolves the existing `//TODO: add geoloc pointer` at line ~301): when + `latitude/longitude` present, render a small static MapLibre mini-map / "Show on map" + link that deep-links into the Places view. +- **Favorite initial state** (bug fix): call `favorites.isFavorite(item.id, 'file')` in + `_show()` to set the star correctly (today it always starts empty). + +#### 0.5 UX & a11y +**`static/js/features/library/photos.js`** +- **Shift-click range select** and optional drag-marquee. +- Replace native `confirm()`/`alert()` with the app modal + (`static/js/components/modal.js`; add an async `Modal.confirm()` helper). +- Tiles become focusable/role-correct; arrow-key navigation across the grid. + +#### 0.6 (Optional) HEIC support +`image` crate ships only `jpeg/png/gif/webp` — iPhone HEIC photos currently get **no +server thumbnail**. Either add `libheif-rs` decoding in `thumbnail_service.rs` / +`media_metadata_service.rs`, or transcode HEIC→JPEG on upload. Flagged as its own task +(native dep). + +#### 0.7 Sub-navigation inside Photos +**`static/index.html`** + **`static/js/app/navigation.js`** (`switchToPhotosSection`, +line ~434) + i18n: add a tab strip **Moments · Places · People** within the Photos view. +Places/People tabs are hidden unless their feature flags are on. This is the mount point +for Phases 1 & 2. + +--- + +## Phase 1 — Places (map) + +Data already exists (`storage.file_metadata.latitude/longitude`, `DOUBLE PRECISION`). +No PostGIS. + +### Execution order + +#### 1.1 Migration — index (+ optional geohash) +**New file:** `migrations/_places_geo_index.sql` +```sql +-- Fast bbox scans over geotagged photos +CREATE INDEX IF NOT EXISTS idx_file_metadata_geo + ON storage.file_metadata (latitude, longitude) + WHERE latitude IS NOT NULL AND longitude IS NOT NULL; +-- Optional (scale): a geohash/quadkey integer + btree for prefix grouping by zoom. +-- ALTER TABLE storage.file_metadata ADD COLUMN geohash BIGINT; +``` + +#### 1.2 Application port + PG repository (grid aggregation) +**`src/application/ports/`** — new `GeoPhotoReadPort` (or extend an existing media port): +```rust +pub struct GeoCluster { pub lng: f64, pub lat: f64, pub count: i64, pub sample_file_id: Uuid } +pub struct GeoBounds { pub w: f64, pub s: f64, pub e: f64, pub n: f64 } + +#[async_trait] +pub trait GeoPhotoReadPort: Send + Sync { + async fn clusters_in_bounds(&self, user_id: Uuid, b: GeoBounds, cell: f64) + -> Result, DomainError>; + async fn photos_in_bounds(&self, user_id: Uuid, b: GeoBounds, limit: i64) + -> Result, DomainError>; +} +``` +**`src/infrastructure/repositories/pg/`** — PG impl. Grid aggregation (no PostGIS): +```sql +SELECT round(fm.longitude / $6) * $6 AS gx, + round(fm.latitude / $6) * $6 AS gy, + count(*) AS n, + avg(fm.longitude) AS clng, + avg(fm.latitude) AS clat, + min(fm.file_id) AS sample_id +FROM storage.file_metadata fm +JOIN storage.files fi ON fi.id = fm.file_id +WHERE fi.user_id = $1::uuid AND NOT fi.is_trashed + AND fm.longitude BETWEEN $2 AND $3 -- west .. east + AND fm.latitude BETWEEN $4 AND $5 -- south .. north + AND fm.latitude IS NOT NULL +GROUP BY gx, gy; +``` +`$6` (`cell`) shrinks with zoom. Single indexed scan + hash aggregate; the browser only +receives `{count, center, sample_file_id}` per cell. + +#### 1.3 Application service (AuthZ + audit) +**`src/application/services/places_service.rs`** (new): +```rust +pub async fn list_clusters_with_perms( + &self, caller_id: Uuid, bounds: GeoBounds, zoom: u8, +) -> Result, AppError> { + // Scoped to the caller's own library; no cross-user data. + self.authz.require(caller_id, /* own photos */).await?; // audit on deny inside require() + let cell = cell_for_zoom(zoom); + self.geo.clusters_in_bounds(caller_id, bounds, cell).await +} +``` +Wire it in **`src/common/di.rs`** (`AppServiceFactory` → `AppState`), `Option>` +gated on the feature flag. + +#### 1.4 Config flag +**`src/common/config.rs`** — `FeaturesConfig::enable_places` from `OXICLOUD_ENABLE_PLACES`. + +#### 1.5 Basemap serving (PMTiles via Axum) +- Add the **`pmtiles`** crate (`cargo add pmtiles`). +- Ship a `.pmtiles` basemap (config: path; default global z0–6 ≈ 60 MB) + self-hosted + **glyphs** and **sprites** under `static/` (from `basemaps-assets`). +- **`src/interfaces/api/handlers/basemap_handler.rs`** (new): open the reader once + (`AsyncPmTilesReader::new_with_path`, `Arc` into `AppState`), serve + `GET /api/basemap/{z}/{x}/{y}.mvt` (`reader.get_tile(...)`). *Alt:* serve the raw + `.pmtiles` over Range and let `pmtiles.js` do directory math (no tile handler). + +#### 1.6 HTTP endpoints + routes +**`src/interfaces/api/handlers/places_handler.rs`** (new): +- `GET /api/photos/geo?bbox=w,s,e,n&zoom=Z` → `Vec` (auth middleware injects + `caller_id`; handler does **no** AuthZ — service does). +- `GET /api/photos/geo/cell?bbox=…` → photos in a cell (opens lightbox). +Register in **`src/interfaces/api/routes.rs`** (protected routes) + the basemap route +(public/cached). + +#### 1.7 Frontend — vendored map + Places module +- **Vendor** (needs sign-off): `maplibre-gl` (UMD + CSS), `pmtiles.js`, + `@protomaps/basemaps` style JSON → `static/js/vendors/` + `static/css/`. +- **`static/js/features/library/places.js`** (+ `static/css/views/places.css`): init + MapLibre with the self-hosted style (light/dark flavor by theme), register the + `pmtiles://` protocol, add a clustered GeoJSON source fed from `/api/photos/geo` + (`cluster: true`) — or, above ~100k, the server-aggregated endpoint. Click cluster → + zoom; click point → open lightbox filtered to that cell. Mandatory ODbL attribution + control. +- **`static/js/core/types.js`** — `GeoCluster` typedef. +- Mount under the **Places** sub-nav tab (§0.7). + +#### 1.8 (Optional) thumbnail markers +deck.gl `IconLayer` (MIT, no React) atlas for **visible cluster representatives only** — +never atlas all points. Start without it (count bubbles), add later. + +--- + +## Phase 2 — People (faces) + +Feature-flagged, opt-in, OFF by default. Biometric data → privacy-first. + +### Execution order + +#### 2.1 Config flag + privacy switch +**`src/common/config.rs`** — `OXICLOUD_ENABLE_FACES`. Plus a **per-user opt-in** setting +(stored in `auth.users` or a user-settings table) — clustering only runs for users who +opted in. + +#### 2.2 Migration — pgvector + schema +**New file:** `migrations/_faces.sql` +```sql +CREATE EXTENSION IF NOT EXISTS vector; -- pgvector (or vectorchord) +CREATE SCHEMA IF NOT EXISTS faces; + +CREATE TABLE faces.persons ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, + display_name TEXT, -- null = unnamed + cover_face_id UUID, + is_hidden BOOLEAN NOT NULL DEFAULT false, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE faces.faces ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + file_id UUID NOT NULL REFERENCES storage.files(id) ON DELETE CASCADE, + user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, + bbox REAL[4] NOT NULL, -- x,y,w,h (normalized) + det_score REAL NOT NULL, + quality REAL, -- blur/size gate result + embedding vector(512) NOT NULL, + person_id UUID REFERENCES faces.persons(id) ON DELETE SET NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX idx_faces_embedding ON faces.faces + USING hnsw (embedding vector_cosine_ops); +CREATE INDEX idx_faces_person ON faces.faces (person_id); +CREATE INDEX idx_faces_user ON faces.faces (user_id); +``` +Cascade-delete guarantees the **right to erasure**: deleting a file/user removes its +faces; deleting a Person unlinks its faces. + +#### 2.3 Domain + ports +**`src/domain/entities/`** — `Face`, `Person`. +**`src/application/ports/face_ports.rs`** (new): +```rust +pub struct DetectedFace { pub bbox: [f32;4], pub landmarks: [[f32;2];5], pub score: f32 } + +#[async_trait] pub trait FaceDetectorPort: Send + Sync { + async fn detect(&self, image: &DynamicImage) -> Result, DomainError>; +} +#[async_trait] pub trait FaceEmbedderPort: Send + Sync { + async fn embed(&self, aligned_112: &DynamicImage) -> Result<[f32;512], DomainError>; +} +``` +**`src/application/ports/`** — `FaceRepository` (CRUD + ANN search via pgvector `<=>`). + +#### 2.4 Infrastructure — ONNX runtime adapter +- `cargo add ort ndarray`. +- **`src/infrastructure/services/onnx_face_service.rs`** (new): loads detector + embedder + ONNX models, runs on a **dedicated thread pool** (mirror `thumbnail_service.rs` / + `image_transcode_service.rs` to avoid starving Tokio). Pipeline: detect → 5-point + similarity align to 112×112 → embed → L2-normalize. Implements `FaceDetectorPort` + + `FaceEmbedderPort`. +- **Models** are **downloaded at runtime** to a models dir (NOT committed). Default: + SCRFD-2.5G + `buffalo_l/w600k_r50` (immich pattern). Config switch to the + permissive fallback (RetinaFace-MobileNet0.25 MIT + bundled-by-you embedder). +- GPU optional via `ort` execution providers (`ORT_DYLIB_PATH` / EP Cargo features); same + code path falls back to CPU. + +#### 2.5 Indexing pipeline (lifecycle hook + backfill) +- **`src/infrastructure/services/face_indexing_service.rs`** (new) implements + `FileLifecycleHook` (same pattern as `media_metadata_service.rs`): on image create → + decode (reuse decode path) → detect → **quality-gate** (score ≥0.7, face ≥50–80px, + Laplacian blur) → embed → store. **Dedup by `blob_hash`**: identical photos reuse faces. +- **Backfill**: a throttled background job over the existing library on the **maintenance + pool**. + +#### 2.6 Clustering (incremental + periodic) — application service +**`src/application/services/people_service.rs`** (new). All methods +`*_with_perms(caller_id)` → `authz.require(...)` → audit on deny. +- **Online (per import):** ANN candidate via pgvector `<=>` + **exact cosine re-rank**; + assign to existing Person if within the *match* threshold (tighter), else leave + unassigned. Thresholds: form ≈ cosine-sim 0.75–0.80 (Euclid ≈ 0.5); match tighter (≈0.4 + Euclid) for precision. +- **Periodic full re-cluster:** threshold connected-components over the user's faces; + `minFaces ≥3` to promote a cluster to a Person; singletons → "Unknown". + +#### 2.7 HTTP endpoints + routes (AuthZ in service) +**`src/interfaces/api/handlers/people_handler.rs`** (new): +- `GET /api/people` — persons (cover + count). +- `GET /api/people/{id}/photos`. +- `PATCH /api/people/{id}` — rename. +- `POST /api/people/merge` · `/split` · `POST /api/people/{id}/hide`. +- `GET /api/files/{id}/faces` — face boxes for lightbox tagging. +- Settings: enable/disable, re-index, **delete all my face data**. +Register in `routes.rs`. Wire service in `di.rs` (`Option>`). + +#### 2.8 Frontend — People module +- **`static/js/features/library/people.js`** (+ `people.css`): grid of person tiles + (circular cover face + name), click → that person's photos; rename/merge/hide UI; + lightbox face boxes + "tag person". +- **`static/js/core/types.js`** — `Person`, `Face` typedefs. +- Mount under the **People** sub-nav tab (§0.7); show an explicit **opt-in consent** gate + before first indexing. + +#### 2.9 (Optional, later) Semantic search +CLIP/SigLIP via the same `ort` stack → natural-language photo search ("beach", "cake"). +Reuses the embedding-in-Postgres + ANN infrastructure. + +--- + +## Vendoring & dependencies (need sign-off) + +| Kind | Item | License | Notes | +|------|------|---------|-------| +| JS (vendor) | `maplibre-gl` (UMD+CSS) | BSD-3 | Map engine; no framework | +| JS (vendor) | `pmtiles.js` | BSD-3 | Range-reads `.pmtiles` in browser | +| JS (vendor) | `@protomaps/basemaps` style + assets | code BSD-3 / design CC0 | self-host glyphs+sprites | +| JS (vendor, opt) | `deck.gl` core+layers | MIT | thumbnail `IconLayer` only | +| Rust crate | `pmtiles` | MIT/Apache-2.0 | serve basemap from Axum (`cargo add`) | +| Rust crate | `ort` (+`ndarray`) | MIT/Apache-2.0 | ONNX runtime; ships `libonnxruntime.so` | +| Rust crate (opt) | `libheif-rs` | LGPL | HEIC decode (native dep) | +| PG extension | `pgvector` (→ `vectorchord`) | PostgreSQL / Apache-2.0 | 512-d embeddings + HNSW | +| Asset (basemap) | Protomaps `.pmtiles` | data ODbL | self-hosted; attribution required | +| ML models (runtime DL, NOT committed) | SCRFD + `buffalo_l` | **non-commercial** | personal self-host OK; commercial = license InsightFace | + +Repo rules respected: no hand-editing `Cargo.lock`; no JS framework; design tokens only +for CSS; `target:"audit"` denial logs; AuthZ exclusively in services. + +--- + +## Open decisions (need your call) + +1. **Faces embedder strategy:** (a) **immich pattern** — runtime-download `buffalo_l` + (~97 IJB-C, non-commercial, recommended default) · (b) fully-permissive bundle — + retrain EdgeFace/GhostFaceNet (~94, real ML project) · (c) defer People. +2. **Basemap extent / hosting:** global z0–6 (~60 MB, simplest) vs regional extract vs + full planet (~120 GB) — and store path / how shipped. +3. **Vector store start:** `pgvector` now (simplest) vs `VectorChord` from day one + (immich's scaled choice). +4. **Map thumbnails:** start with count bubbles (MapLibre only) vs deck.gl `IconLayer` + from the start. +5. **HEIC:** in scope for Phase 0, or deferred (native `libheif` dep)? +6. **MapLibre vendoring approval** (new JS library — per repo rules, needs explicit OK). + +--- + +## Suggested sequencing + +| Phase | Risk | Notes | +|-------|------|-------| +| 0.1 virtualization | done | smoke-test pending | +| 0.2–0.5 polish | low | self-contained, verifiable | +| 1 Places | low–med | data ready; new vendored map + basemap serving | +| 2 People | high | new ML stack, pgvector, privacy, licensing decision | +| 0.6 HEIC / 2.9 search | opt | independent, schedule freely | + +Recommended order: finish **Phase 0**, ship **Places**, then tackle **People** once the +embedder-licensing decision (#1) is made.