Commit Graph

159 Commits

Author SHA1 Message Date
Edouard Vanbelle 302d2ff80f feat(recoverable-job): add core engine 2026-07-28 22:39:07 +02:00
Edouard Vanbelle 0e8b1fbbeb refactor(job-registry): simplify the job registering* 2026-07-28 21:13:09 +02:00
Edouard Vanbelle f66f7fa31f feat(job-registry): remplace /api/admin/internal/trigger-*
remplace /api/admin/internal/trigger-* to /api/admin/jobs/{...}/trigger
remove OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS
2026-07-27 23:54:43 +02:00
Edouard Vanbelle dfedde54a4 feat(job-registry): wire /api/admin/jobs/* 2026-07-27 23:38:12 +02:00
Edouard Vanbelle 72b99f5b9a refactor: apply clippy 2026-07-27 23:06:02 +02:00
Edouard Vanbelle 1190dfef87 feat(job-registry): wire grant cleaner 2026-07-27 23:03:22 +02:00
Edouard Vanbelle f740de3c10 feat(job-registry): wire dedub_gc 2026-07-27 22:56:02 +02:00
Edouard Vanbelle f5f794fde5 feat(job-registry): handle job without periodicity but with trigger 2026-07-27 22:53:13 +02:00
Edouard Vanbelle e29f3ba067 feat(job-registry): wire storage usage update 2026-07-27 22:33:04 +02:00
Edouard Vanbelle d0f4e1ef5a feat(job-registry): wire engine + wire trash cleaner 2026-07-27 22:24:15 +02:00
Edouard Vanbelle 99e6de6fce feat(search): invalidate user's cache on share or favorite change 2026-07-26 18:18:15 +02:00
Bradley Nelson 9f4bc54113 Merge upstream/main into feat/external-file-mounts
Resolve conflicts between the external-file-mounts feature and upstream's
D5/D7 refactor (per-file provenance, keyset pagination, cross-drive move
gates, resource-access hook, folder-cascade lifecycle hook).

Key resolutions:
- FolderService::new now takes (repo, authz, file_lifecycle, mount_router);
  all callers + DI updated.
- FileRetrievalService / FileManagementService keep both the mount_router
  and the new resource_access_hook / drive_repo / storage_usage wiring.
- list_files_batch_with_perms: adapt the mount branch from offset- to
  keyset (after_name) pagination, mirroring paginate_mount_entries.
- download_file_impl: keep upstream's &HeaderMap + `impl IntoResponse + use<>`
  signature, retain the mount-download branch.
- Mount DTOs: the retired `owner_id` field maps onto created_by/updated_by
  (the mount owner) — the fields the frontend now uses for owner display.
- admin/+page.svelte: keep upstream's user-delete modal + the 'mounts' tab.
- Bump memmap2 0.9.10 -> 0.9.11 (RUSTSEC critical advisory fix) and
  regenerate Cargo.lock against the merged Cargo.toml.
2026-07-21 17:09:36 -06:00
Edouard Vanbelle 190a2e32e9 refactor: apply rust formatter suggestion 2026-07-17 20:34:12 +02:00
Edouard Vanbelle fa0e4e1a89 fix(cache): invalidate drive used byte cache on explicit refresh from internal call
this fix https://github.com/AtalayaLabs/OxiCloud/issues/607
    which was introduced by commit 12dc648cff
    when a user does activity in a drive, admin can invalidate cache via the internal
    call /api/admin/internal/trigger-sweep
    this permit end 2 end test to validte immediately that used_bytes corresponds to the expected result
2026-07-17 20:20:09 +02:00
Claude 12dc648cff perf: round 4 — one-pass row paths, drive-selector cache, CalDAV single-parse, streamed Azure, batched hydration
Nine benchmark-gated changes (benches/ROUND4.md; every one ships with a
BEFORE/AFTER bench + equivalence gate, rollback rule as ROUND2/3):

- Row→entity path build: one-pass StoragePath::from_folder_and_name /
  from_joined + normalize_storage_name_owned + alloc-free Display —
  743→417 ns/file-row (1.78x), −5 allocs/row on every listing surface.
- WebDAV drive-selector: per-user readable_cache (single-flight, 30 s
  TTL, explicit invalidation incl. membership + group changes) replaces
  the grants join per request — 441 µs → 0.8 µs (~550x), 0 queries warm.
- CalDAV from_ical/update_ical_data: 8 full IcalParser runs per VEVENT
  → 1 (7.1x per PUT, 4.4x on 50-event imports); alloc-free split_vevents,
  chunk scan without the whole-body uppercase copy (1.4x), borrowed-key
  UID grouping (1.3x), REPORT props no longer cloned.
- PROPFIND emit: partition Vecs dropped (single-pass 404 list) + stack
  rendered RFC 3339/2822 dates, sizes, quoted etags (common::fmt,
  chrono-byte-identical, sweep-tested) on both DAV surfaces — 1.22x
  per page, 17.9→12.0 allocs/row.
- Grant-listing hydration: calendars/address books/playlists batch
  hydrate via = ANY($1) — 15 serial queries → 1 (~13x per sync poll).
- user-flags cache: get→insert → try_get_with single-flight (32→1
  queries per cold herd).
- Azure downloads: whole-blob Vec buffering → streamed SDK pages —
  TTFB 349→4 ms (87x), peak heap 480→1.9 MiB (254x) on 256 MiB blobs;
  new OXICLOUD_AZURE_ENDPOINT_URL override (Azurite/bench hook).
- Face indexing: unbounded per-image tokio::spawn → core-count
  semaphore, permit before blob read — peak heap 1175→176 MiB (6.7x).

Checks: cargo fmt, clippy --all-features --all-targets -D warnings,
cargo test --workspace (523 passed) + --features test_utils. hurl API
suite and dockerized integration DB not runnable in this environment —
left to CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-17 13:48:37 +00:00
Claude cd4c62042a perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):

DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
  folders/files UNION-ALL branches as sargable row-value comparisons with
  per-branch ORDER/LIMIT + two partial expression indexes
  (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
  (19.5x); other sort modes at parity or better. New migration
  20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
  on the timeline index, joins moved above the top-N. 50k-photo library:
  97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
  comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
  idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
  (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).

Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
  connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
  now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
  concurrent cold readers = 16 full remote downloads racing truncating
  writes on ONE deterministic .tmp (corruptible cache); now 1 download
  (16x less egress, 2.8x wall on a shared link) and torn files can never
  be renamed into the cache.

I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
  chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
  pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
  every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
  common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
  clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
  O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
  9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
  (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
  ~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
  nodes, three SDK stacks gone from every build). tokio "process" is now
  an explicit feature (was enabled transitively by aws-config).

Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
  4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
  asserts output identity across locales and a 3x floor.

Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
2026-07-17 11:10:27 +00:00
Claude aba89c4f5d perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
2026-07-16 14:20:20 +00:00
Edouard Vanbelle 54b5b3bf4f feat(caldav+carddav): auto create default cal & card
automatically create default Calendar and default addressbook per user
    (no creation if user already have a such resource)

    default name are "Personal"

    this is using the user's life cycle like does the drives

    answers to issue #545
2026-07-14 14:30:05 +02:00
M.Schmidt c62f97d8f2 readd removed utility, cleanup/shorten enum usage 2026-07-13 22:22:12 +02:00
M.Schmidt 6701ddfc17 Merge branch 'main' into rfc-4331-quota-properties
# Conflicts:
#	src/interfaces/nextcloud/report_handler.rs
#	src/interfaces/nextcloud/webdav_handler.rs
#	tests/api/run.sh
2026-07-13 20:32:01 +02:00
M.Schmidt 8a405af5e1 refactor(drive): match DriveKind directly instead of Drive::is_personal()
Drops the boolean is_personal() wrapper in favor of matching
DriveKind::Personal/Shared at the two call sites, matching the
exhaustive-match convention already used for DriveKind elsewhere
(as_str, parse, DriveKindDto::from).
2026-07-13 19:51:40 +02:00
M.Schmidt c07aeabd85 feat(webdav): drive-aware RFC 4331 quota properties
resolve_quota only ever reported the caller's personal envelope,
ignoring the drive_id already resolved at every PROPFIND call site —
shared drives with their own quota showed the wrong numbers. Adds
AppState::resolve_webdav_quota, shared by both WebDAV surfaces:
nil drive_id or personal drive -> account envelope, shared drive ->
its own storage.drives quota/used_bytes.

Also adds quota-used-bytes/quota-available-bytes to the NextCloud-
compatible surface, which previously had no RFC 4331 support at all.

Registers webdav_quota_properties.hurl and the new
nc_webdav_quota_properties.hurl in tests/api/run.sh — neither was
wired into the suite before this change.
2026-07-13 00:34:17 +02:00
Edouard Vanbelle ba620166ee feat(grant): clean up expired grants 2026-07-12 18:37:13 +02:00
Edouard Vanbelle 0fcd617fd1 feat(calendar,addressbook): migrate share to authz engine
migrate DB entries to authz engine
    and wire authz engine to caldav and carddav
2026-07-08 01:03:25 +02:00
Edouard Vanbelle 79f4792709 fix(quota): pre-check quota for COPY/MOVE
check qouta for a cross drive MOVE
    check quota for a COPY
2026-07-06 22:14:50 +02:00
Edouard Vanbelle b95e740b2f security(music): ensure read permission via authz 2026-07-05 22:53:05 +02:00
Edouard Vanbelle 2cda8e7e22 security(favorite,recent): ensure read permission 2026-07-05 22:53:05 +02:00
Edouard Vanbelle 858139ef3b feat(drive): prepare removal of user_id
this commit changes GET /api/<resources> to return resource caller has access to
    this is not anymmore resources users is owner of
2026-07-02 00:01:20 +02:00
Edouard Vanbelle 01ff7dab0b feat(drive): impl policy photo + music policies
add `include_in_photo_index` and `include_in_music_index` policies
    both true for default personal drive

    photo is implemented
    music is not yet implemented
2026-07-01 21:57:37 +02:00
M.Schmidt b5afa1a855 feat(webdav): implement RFC 4918 dead property store
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
2026-06-26 21:19:03 +02:00
Bradley Nelson d8229650ef feat(mounts): P3 — WebDAV/NextCloud path resolution + browse/download
Mounts are now browsable and downloadable over both WebDAV surfaces
(/webdav/ and NextCloud /remote.php/dav) and NextCloud clients. The work
funnels through the path-based service methods all three surfaces already use,
so both WebDAV handlers gain mount support with no handler changes.

- get_folder_by_path / get_file_by_path resolve a path that descends past a
  mount root to a synthetic ext: DTO (the mount root itself stays a real row)
- list_folders_paginated_with_perms / list_files_batch_with_perms branch to the
  provider, so PROPFIND Depth:1 enumerates mount directory contents in both
  WebDAV handlers
- get_file_stream / get_file_range_stream branch ext: file ids to the provider,
  so WebDAV GET streams mount content (Pin<Box<dyn Stream>> re-boxed)
- router injected into FileRetrievalService; new MountRouter::find_path delegates
  to the registry's (drive_id, mount_path) index
- WebDAV mkdir/delete/move/rename by path work via path→ext:id→the P2 service
  methods (no extra wiring)

Fixes a leading-slash normalization bug in the registry path index: materialized
folder paths arrive both as `Personal/Media` and `/Personal/Media`; keys and
lookups now normalize the leading slash (would have broken real WebDAV paths).

Known follow-up: WebDAV PUT (upload/update by path) still ingests to the CAS;
streaming a WebDAV PUT straight to the provider needs a pre-ingest branch in the
two WebDAV PUT handlers (mirrors the REST upload branch).

Integration test: get_folder_by_path/get_file_by_path resolution, PROPFIND
Depth:1 folder+file listing, and content streaming on a real mount + Postgres.
2026-06-25 00:51:16 -06:00
Bradley Nelson 8e3e31da4d feat(mounts): P2 — read-write REST for external mounts
Adds full CRUD on mount contents, mirroring the P1 read pattern (handlers/
services classify; authorization stays in the service via the mount-root
folder grant; the provider does the I/O).

- mkdir / rename / delete / move-within branch inside FolderService and
  FileManagementService (router injected into both)
- streaming upload via a new ExternalUploadService: the upload handler detects a
  mount destination BEFORE the CAS ingest and streams the multipart body
  straight to the provider (no BLAKE3/dedup). `write_stream` now takes a
  lifetime-bound boxed stream so the borrowing multipart field can be passed
  without buffering.
- deletes on mounts are permanent (no trash): the trash-first folder handler
  routes `ext:` ids straight to the provider delete; file delete goes through the
  branched delete_and_cleanup
- cross-backend move/copy (mount ↔ native, or between mounts) is forbidden
  (UnsupportedOperation); the mount root itself cannot be renamed/moved/deleted
- every mutation emits a `target:"audit" event="external_mount.write"` line
- shared mount_dto builders synthesize FolderDto/FileDto from a provider MountStat

Tests: 529 unit + integration tests for mkdir/rename/delete, file rename/delete,
streaming upload, cross-boundary forbid, and stranger-denied — all against real
Postgres + a real provider (testcontainers).
2026-06-25 00:30:10 -06:00
Bradley Nelson 3c31695579 feat(mounts): external file mounts P1 — pluggable provider + read-only REST
Adds the foundation for external file mounts: admin-configured backends
(raw host filesystem in v1; sftp/webdav/… as future provider kinds) surfaced
as a folder inside a user's drive. Mount contents are virtual/live-passthrough
— read straight from the backend, never stored in storage.files — and are a
deliberately separate, limited storage type (no dedup/sharing/trash/search).
The feature is dark by default (OXICLOUD_ENABLE_EXTERNAL_MOUNTS=false).

P1 scope (this PR): data model, the pluggable provider abstraction, and the
read-only REST surface (mount listing + download). Read-write (P2),
WebDAV/NextCloud path resolution (P3), and the admin UI (P4) follow.

Core model
- Mount root = a real storage.folders row; authorization for everything inside
  collapses onto that folder UUID (ltree-ancestry grant cascade).
- Children are virtual, addressed by ext:<mount_id>:<base64url(node_id)> where
  node_id is provider-owned and opaque to the rest of the system.
- A lock-free (arc-swap) MountRegistry maps mount-root UUID -> provider; a thin
  MountRouter::classify() is the single cheap hook handlers call before parsing
  an id as a UUID. With no mounts configured it always returns Regular, so
  existing code paths are unchanged.

Added
- migrations/20260805000000_external_mounts.sql (storage.external_mounts, kind + config JSONB)
- domain/services/external_mount_id (id envelope + virtual etags)
- application/ports/external_mount_ports (ExternalMountProvider, MountProviderFactory, repo port)
- infrastructure local_fs_mount_provider (tokio::fs, symlink-escape-safe) + factory
- application MountRegistry + MountRouter, pg ExternalMountRepository
- DI wiring (AppState.mount_router), FeaturesConfig.enable_external_mounts
- listing branch (FolderService::list_mount_dir_with_perms + folder_handler) and
  download branch (FileRetrievalService stat/open mount methods + file_handler)

Authorization stays in the service layer (authz.require(Resource::Folder(mount_id)));
handlers only classify. Cross-backend operations are out of scope for P1.

Tests: 529 unit tests + 5 testcontainers integration tests (real Postgres 17),
including end-to-end authorization (owner allowed, stranger denied). Line
coverage of the new modules is 84–100% (cargo-llvm-cov). Known gap:
file_handler::download_mount_file (HTTP glue) needs a full-app test (P4).
2026-06-24 23:52:01 -06:00
Edouard Vanbelle a5b24a7453 feat(drive): add /api/drive
- permit shared drive creation from oxicloud admin (for now)
    - prepare other personal drive creation (Not implemented), need to validate
    quota policies and strategy first
    - add hurl test to verify permissions
2026-06-23 23:50:57 +02:00
Edouard Vanbelle 184520c17a feat(drive): plug trash to drives 2026-06-23 22:04:20 +02:00
Edouard Vanbelle a82bae4136 feat(drive): api to handle Drive grants 2026-06-22 23:53:00 +02:00
Edouard Vanbelle 66f2aaa250 feat(drive): add drive policie
- add policy forbid_external_sharing
    - add policy forbid_sharing
    - add polocy forbid_cross_drive_move
    - add policy forbid_owner_role_change
2026-06-26 14:04:36 +02:00
Edouard Vanbelle ddb131da8b feat(drive): add policy forbid_public_links 2026-06-26 13:59:02 +02:00
Dionisio Pozo 996ace8a1d Merge pull request #524 from EdouardVanbelle/fix/cascading-deletion-and-file-life-cycle
fix(fileLifeCycle): add missing on_file_deleted() with cascading deletion
2026-06-26 12:35:25 +02:00
Edouard Vanbelle f4032e9e23 feat(recent): update recent list server side
initially the recent was done client side
    recent files are now directly updated on serverside when accessing a file

    note: nextcloud and webdav voluntary not included
2026-06-26 01:44:19 +02:00
Edouard Vanbelle fefba83cfc fix(fileLifeCycle): add missing on_file_deleted with cascading del
on_file_deleted() was missing on folder deletion or on cascading deletion
2026-06-25 23:36:50 +02:00
Edouard Vanbelle 6b8e2ba49c test(drive): check quota calculus 2026-06-24 23:01:34 +02:00
DioCrafts 5722481c4a feat(thumbnails): server-side video thumbnails via ffmpeg
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).

- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
  (infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
  extract one representative frame as PNG, bounded by its own semaphore + a
  per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
  videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
  generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
  to a size- and time-bounded temp file on the data volume, extract a frame, and
  reuse the shared render_and_persist_all_webp helper — so video thumbnails are
  WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
  Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
  at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
  path; the server is now the source of truth.

Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.

Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 23:23:04 +02:00
DioCrafts 54639d466a chore(frontend): remove the legacy vanilla-JS frontend and its tooling
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).

Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
  to frontend/static/ so they ship with the SPA. This also fixes the
  favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
  to the SvelteKit /nextcloud/error route.

Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
  the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
  fallback so `just dev` works without a prior build.

Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
  pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).

Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
  scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
  and drop check-contrast/check-headings (coupled to the old token
  taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.

Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
  (superseded by /resources).

Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 03:20:10 +02:00
DioCrafts b14c4dc911 perf: cache-stampede coalescing + DB safeguards; ui/i18n fixes
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
  (try_get_with / get_or_load) so N concurrent misses for the same key
  collapse to one disk read / transcode / query instead of a thundering herd.
  Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
  before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
  (OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
  a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
  (OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
  exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
  return to the OS and RSS tracks the live working set; benchmarked on
  musl/aarch64 at ~400MB reclaimed vs 0MB with the default.

Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
  selection toolbar and storage line — the call sites passed param names that
  didn't match the locale placeholders; unify on `count` and pass the storage
  template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
  "Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
  true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 14:42:10 +02:00
Edouard Vanbelle e5b9a4a8db feat(drive): remove create_home_folder
remove create_home_folder() & ensure_home_folder()

    now: on_user_created() and on_user_login both() call provision_if_needed()
    which calls **create_personal_drive_atomic()**

    add a helper to find Personal drive for a user and also it's root directorry
2026-06-19 16:06:38 +02:00
Edouard Vanbelle 16ea08b093 feat(drive): improve Drive model
now Drive is purely a metadata
    each drive has always a root folder
    this model minimize Oxicloud changes, and simplify
    the Drive name is simply the folder's root's name
    note: owner of Drive has more permission that an owner of the root folder
2026-06-19 16:06:37 +02:00
Edouard Vanbelle eab7a609b9 feat(drive): start implementation of drive
- add storage.drives
    - prepare migration phase
    - add created_by and updated_by on storage.folders
2026-06-19 16:06:37 +02:00
Claude 12ede47b2c feat(faces): real ONNX face analyzer (SCRFD + ArcFace), opt-in
Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.

Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.

- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
  decode, NMS, the closed-form (complex-number) similarity transform,
  bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
  11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
  Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
  and the crate builds without it). Inference runs on spawn_blocking; each
  session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
  not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
  EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
  compiled in and runtime+models are configured; any missing piece or load
  failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.

Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
2026-06-19 12:28:49 +00:00
Claude 1dca9d7d05 feat(faces): indexing pipeline + DI wiring
Phase 2 increment 5:
- FaceIndexingService: a FileLifecycleHook that, on image upload, detects +
  embeds faces in a background task and stores them. Dedup-aware (clones an
  identical blob's faces instead of re-running inference), reindexes on
  overwrite, and relies on the DB cascade for deletes. Completely inert when
  no model is ready.
- DI: registers the hook in the FileLifecycleService chain and exposes
  PeopleService in AppState — both gated on OXICLOUD_ENABLE_FACES, both
  using the default no-op analyzer until the operator wires a real ONNX
  model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
2026-06-19 11:46:57 +00:00