Commit Graph

197 Commits

Author SHA1 Message Date
Edouard Vanbelle 526903e0d5 fix(db-migration): keep timestamp order 2026-08-04 07:03:08 +02:00
Edouard Vanbelle 168d94370a feat(opaque): legacy login refuses migrated OPAQUE users 2026-08-04 07:03:08 +02:00
Edouard Vanbelle 7d7621e387 feat(opaque): wire API
- POST /api/auth/opaque/login/ke1 (public) — takes {userIdentifier, startLoginRequest}, resolves the identifier via the same @-dispatch as legacy login (AuthApplicationService::lookup_user_for_login, factored out), fetches envelope, runs ServerLogin::start (real branch for known users, dummy branch for anti-enum on unknown/unregistered), stashes state under a random exchange_id in the moka cache, returns {exchangeId, loginResponse}.
- POST /api/auth/opaque/login/ke3 (public) — atomic take from the cache FIRST (anti-enum + anti-replay), then decodes the payload, runs ServerLogin::finish, stamps opaque_migrated_at (Phase 3 signal), and mints a session via the new AuthApplicationService::mint_session_for_authenticated_user helper — returns the same AuthResponseDto shape as legacy login so the SPA has one downstream handler.
- Session mint factored: mint_session_for_authenticated_user(User) extracted from login() so both the legacy password path and OPAQUE KE3 converge through one implementation.
- OpaqueRepositoryPort::mark_migrated with COALESCE-preserving idempotent stamp of opaque_migrated_at.
- opaque-setup CLI + Dockerfile wiring already shipped (Step 0 hygiene).
- Routing fix: sub-prefix split (/api/auth/opaque/register vs /api/auth/opaque/login) — axum composes middleware between sibling nests at the same prefix, which was cross-applying auth+CSRF to my public login routes. Distinct prefixes side-step that cleanly. Documented in both main.rs and the router builder doc.
- Rate-limit sharing: login KE1/KE3 layered with the SAME login_limiter instance as legacy POST /api/auth/login, so an attacker can't halve the per-IP budget by spraying both endpoints.

Anti-enum + anti-replay hardening in KE3: take runs BEFORE payload parse so:
- Unknown / expired / already-consumed exchange_id → 401 InvalidCredentials (same shape as wrong-passphrase, no payload-shape leak)
- Consumed handle can't be re-used to spam parse attempts
2026-08-04 07:03:08 +02:00
Edouard Vanbelle 033146a6c9 feat(opaque): add login opaque exchange 2026-08-04 07:03:08 +02:00
Edouard Vanbelle acd4420fe3 feat(oidc): impl back channel logout 2026-08-03 08:00:13 +02:00
Edouard Vanbelle 166b8c4891 feat(oidc): RP initiator logout
request token invalidation to IdP (OIDC) on logout
2026-08-03 07:59:14 +02:00
Edouard Vanbelle 015f2da0f7 refactor(backend): normalize naming convention to backend rather storage
no ambiguity with the backend rather storage
2026-08-02 14:56:29 +02:00
Edouard Vanbelle a10d3254bf fix(storage migration): overwrite target blob if mismatch
if target blob already exists, migration will check the blob
    if header is already with the targeted key (or no key if not ciphered)  no need write
    otherwise write the blob (that will convert any blob with no header into the correct version)
2026-08-02 14:56:29 +02:00
Edouard Vanbelle bc481bdd99 feat(job): fix key rotation on local storage (replace blob) 2026-08-02 02:40:33 +02:00
Edouard Vanbelle f409658c96 feat(storage): hot swap backend on migrate 2026-08-01 17:11:07 +02:00
Edouard Vanbelle 507bc2e98d feat(recoverable-job): add backend_consistency (storage) 2026-07-30 00:15:20 +02:00
Edouard Vanbelle e29f3ba067 feat(job-registry): wire storage usage update 2026-07-27 22:33:04 +02:00
Edouard Vanbelle c22741bc7f refactor(search): normalize answer to /resources format 2026-07-26 18:18:15 +02:00
Dionisio Pozo ef3e3f45b3 Merge pull request #521 from BCNelson/feat/external-file-mounts
feat(mounts): external file mounts — pluggable provider, read-write, WebDAV/NextCloud, admin UI
2026-07-22 06:52:43 +02:00
Dionisio Pozo 86268049ff Merge pull request #641 from EdouardVanbelle/refactor/front-resource-list
feat(fileDto, folderDto): add is_favorite + is_shared
2026-07-22 06:51:28 +02:00
DioCrafts d66956824c Optimize storage, GC, and upload hot paths 2026-07-22 02:06:29 +02:00
Edouard Vanbelle d9987782c4 feat(fileDto, folderDto): add is_favorite + is_shared
- provide is_shared and is_favorite information in DTO, information propagated as badge/buttons per items
      regarding performances I try to be minimalis on SQL to prevent any perf regression

    - remove old set of sharedids and favoriteids (was not functionnal anymore)
    - fix date picker (no past selection) in grant
    - fix contextmenu close on /files section
2026-07-22 01:55:10 +02:00
Bradley Nelson 9f4bc54113 Merge upstream/main into feat/external-file-mounts
Resolve conflicts between the external-file-mounts feature and upstream's
D5/D7 refactor (per-file provenance, keyset pagination, cross-drive move
gates, resource-access hook, folder-cascade lifecycle hook).

Key resolutions:
- FolderService::new now takes (repo, authz, file_lifecycle, mount_router);
  all callers + DI updated.
- FileRetrievalService / FileManagementService keep both the mount_router
  and the new resource_access_hook / drive_repo / storage_usage wiring.
- list_files_batch_with_perms: adapt the mount branch from offset- to
  keyset (after_name) pagination, mirroring paginate_mount_entries.
- download_file_impl: keep upstream's &HeaderMap + `impl IntoResponse + use<>`
  signature, retain the mount-download branch.
- Mount DTOs: the retired `owner_id` field maps onto created_by/updated_by
  (the mount owner) — the fields the frontend now uses for owner display.
- admin/+page.svelte: keep upstream's user-delete modal + the 'mounts' tab.
- Bump memmap2 0.9.10 -> 0.9.11 (RUSTSEC critical advisory fix) and
  regenerate Cargo.lock against the merged Cargo.toml.
2026-07-21 17:09:36 -06:00
Dionisio Pozo 67fe944c2a Merge pull request #596 from swissiety/rfc-5789-http-patch 2026-07-22 00:32:27 +02:00
Claude 8e55caa8a9 perf(round29): cache-serve borrow-probe, NC REPORT href buffer, auth per-req allocs, DB over-fetch
Seven behaviour-preserving allocation / copy / bandwidth cuts, each behind a
counting-allocator BEFORE/AFTER gate that exit(1)s unless AFTER allocates
strictly fewer than BEFORE (benches/ROUND29.md, examples/bench_round29_micro.rs).

- [B] Content-cache serve fast path (optimized_inner Tier 1 +
  get_file_range_preloaded — the video-scrub hot path): probe the cache with a
  borrow first and build the owned get_or_load args (quoted-etag / key / id
  Strings) only on a miss, instead of allocating them before every probe and
  discarding them on a hit. 6 -> 0 allocs per cache hit. Splits get_or_load into
  get + load_and_cache so the miss path is not re-probed and the hit/miss stat
  counters stay byte-identical. Also drops the unconditional content_hash/name
  clones that ran for the >=10 MB streaming tier that used neither.
- [A] NextCloud REPORT emit loops: per-row href String (and format! per folder
  row) -> one reused href_buf via nc_href_into / nc_collection_href_into with the
  URL-encoded user computed once per page. 1497 fewer allocs on a 500-row page.
- [C] read_full: a single-frame blob is returned zero-copy instead of a second
  whole-payload memcpy into a fresh BytesMut; multi-frame path unchanged.
- [D] login-lockout key: to_lowercase()+format! -> one pre-sized ASCII buffer
  (non-ASCII keeps str::to_lowercase). 3 -> 1 alloc/req, byte-identical key.
- [E] NC composite-username parse: owned clone/to_string -> &str borrow of the
  already-owned raw_username. 1 -> 0 alloc on the common no-marker path.
- [F] get_contacts_in_group: stop SELECTing the discarded multi-KB vcard column
  (the live method ROUND25 §Q2 missed; ContactDto has no vcard field).
- [G] count_admin_users: add count_users_by_role -> scalar COUNT(*) instead of
  hydrating every admin's full row (incl. up-to-512 KiB avatar + ui_preferences
  JSONB) only to .len() it, on a bootstrap-polled status endpoint.

All seven gates pass; cargo fmt --check and cargo clippy --all-features
--all-targets -D warnings clean. §F/§G additionally validated against a live
PostgreSQL 16 with the full migration set (query validity, result equivalence,
600000 -> 8 byte wire delta on the admin count).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhpDZxSQTAGnAqCHUdtG5N
2026-07-21 10:25:36 +00:00
Claude c930f865b0 perf: round 14 — faces narrow projection, auth per-request allocs, CalDAV emit buffers, frontend set churn
Benchmark-gated (benches/ROUND14.md); every change ships a BEFORE/AFTER
benchmark with an equivalence gate and is rolled back on regression (the
rule is encoded as a GATE FAIL exit / threshold expect).

Backend
- Q1 faces_for_file → narrow face_boxes_for_file(id, person_id, bbox) with the
  caller filter pushed into SQL: drops the 2 KiB embedding BYTEA + 6 unused
  columns per face. 15-face lightbox open 0.312→0.219 ms, 32 KB→840 B/req.
- A1 cookie auth uses the borrow-only extract_cookie_str (already backs CSRF)
  instead of extract_cookie_value's owned String: -1 alloc/cookie request.
- A2 compute_relevance ASCII case-fold fast path vs name.to_lowercase() per
  result row (Unicode fallback preserved): 1.40x, 12→3 allocs/page.
- A3 sub pre-parsed to Uuid at decode time (TokenClaims.sub_id) vs re-parsing
  the 36-char claim on every request incl. cache hits: 22.7→0.7 ns.
- A4 auth + NextCloud middlewares borrow request.headers() instead of taking
  axum's HeaderMap extractor (a full map clone): 2→0 allocs/authed request.
- A5 CalDAV getlastmodified via the stack rfc2822_utc (byte-identical to
  chrono) vs a per-event to_rfc2822() heap String: 5→0 allocs.
- A6 CalDAV per-event href + quoted etag written into reused page buffers vs a
  fresh format! pair per event: 3.48x, 240→6 allocs/40-event page.

Frontend
- F1 t() shares one frozen EMPTY_PARAMS for the no-interpolation call forms vs
  a throwaway {} per call: -1 alloc/call.
- F2 favorites favoriteIds is a persistent SvelteSet with per-page add (clear
  on reset) vs a brand-new set over the whole accumulated list each page:
  22.3x over a 40-page drain (O(N^2)→O(N)).

Verified: cargo check --all-targets, cargo clippy -D warnings, both bench
packs (GATE PASS), frontend npm run check + vitest (4/4). ROUND14.md also
records the investigated-but-deferred backlog (music N+1, contact vcard
over-fetch, CachedBlobBackend syscalls, ResourceList.sections builder, etc.).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PymgCdK78NzUF3oRAQCJfN
2026-07-19 10:22:12 +00:00
M.Schmidt af74c94028 fix(webdav): make PATCH's concurrency guard a real compare-and-swap
The app-level ETag re-check before the write still left a gap between the check and the actual UPDATE for a concurrent writer to land in.
Push the check into the write path itself: swap_blob_hash now takes an expected_hash and only applies the SET under the same FOR UPDATE row lock it already held, closing the race instead of just narrowing it. Adds ErrorKind::PreconditionFailed (412) for the CAS-miss path; PUT/WOPI/chunked-upload keep blind-overwrite semantics by passing None
2026-07-19 10:36:07 +02:00
Claude f58d72a780 perf: round 13 — grouped-view virtualization, notification/login query narrowing, HTTP dedup, locale precompute
Benchmark-gated (BEFORE/AFTER + equivalence/safety gate per change), same
discipline as rounds 2-12. Full write-up in benches/ROUND13.md.

Shipped:
- V1 Grouped views windowed (files route + ResourceList). The grid arm was
  the last unwindowed path (trash is grouped-by-default in grid): each
  swimlane now feeds its own VirtualList, outer container a flex stack.
  vitest gate: 800-item grouped grid mounts <120 .file-item (was 800).
- Q1 get_users_by_ids drops the <=512 KiB avatar image + ui_preferences
  JSONB (notification path never reads them). 30-member fan-out 8.60 ->
  0.25 ms (34.3x), ~7.7 MB off the wire.
- Q2 Login provisioning is_empty() -> SELECT EXISTS for calendar + address
  book (every login). 0.193 -> 0.170 ms, widens with owned-row count.
- Q3 Recent-access prunes only when the upsert inserted (RETURNING xmax=0)
  — a re-access can't grow the set. 0.567 -> 0.324 ms (1.75x).
- L1 Locale supported-codes precomputed once vs rebuilt per anonymous
  request. 616 -> 17.3 ns (35.7x), 18 -> 1 allocs.
- H1 Duplicate /api TraceLayer removed (global stack already wraps it).
  1.86 -> 1.42 us/request, -6 allocs.
- H2 client_ip span field: borrow-only ClientIpDisplay vs owned String.
  187 -> 173 ns, -1 alloc.

Not shipped (discipline): the "media hooks read the blob 3x" lead was a
correctness bug, not a perf dup — the raw-path metadata/faces readers
resolve only for local+unencrypted+single-chunk blobs and silently produce
nothing otherwise. Flagged for maintainers; routing through read_blob_bytes
is a correctness fix (perf-neutral-to-negative), not a benchmark-gated
perf change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfidAJD5AHw23jtvBUNamB
2026-07-19 08:17:48 +00:00
Claude 50eca0627f perf: round 12 — auth write-path narrowing, fused quota gate, moka blob-cache index, media single-read, sized listing JSON
Benchmark-gated round (benches/ROUND12.md; every change ships with a
BEFORE/AFTER harness + equivalence gates, one candidate rejected by its
own bench):

DB / query shapes (bench_round12_queries):
- NC sharee search: username-only projection instead of the 21-column row
  (incl. the <=512 KiB avatar) per match, + gin_trgm_ops indexes on
  auth.users for the leading-wildcard ILIKE (4.98x; 54.7x with index).
- Password login: delete the redundant full-row update_user — create_session
  already stamps last_login_at in its own txn (4.45x per login).
- Email-verified stamp: narrow conditional UPDATE (8.9x); OIDC repeat login
  now compares profile state in memory and issues ZERO queries when nothing
  changed (was: full 17-column rewrite per login).
- Refresh rotation: revoke+insert+stamp fused into one transaction via new
  rotate_session port method (1.18x).
- WOPI CheckFileInfo / authorize_wopi_access: require(Read) + get_file +
  check(Update) overlapped with tokio::join!, original result precedence
  (cold 1.34x).
- Upload quota gate: user-envelope + drive-cap checks fused into ONE
  round-trip (check_upload_quotas) — the NC chunked PUT pays this per
  chunk (1.81x, 2 -> 1 queries/chunk); shared verdict evaluators keep
  error shapes byte-identical.

CPU / allocs (bench_round12_micro):
- sized_json: pre-sized listing serialization replacing axum Json's 128 B
  seed + doubling-realloc chain on files/folder-resources/photos/search
  responses (1.40x, 13 -> 2 allocs per 500-row page; byte-identical).
- Security headers: 4 SetResponseHeaderLayer folded into the CSP middleware
  pass (5 layers -> 1; 1.43x per request, -26 allocs; header set gated
  byte-identical incl. 304s).
- Media capture-metadata: single-read extraction — nom-exif now parses the
  buffer kamadak already read (zero-copy Bytes) and videos open once with a
  kind() dispatch; per-image opens 2-3 -> 1 (1.44x warm geomean, 1.6-3.2x
  cold cache; extraction outputs gated identical incl. the MIME-mislabel
  track fallback).
- Chunked-upload session ops: owner gate folded into the operation's own
  DashMap lookup + stack-encoded uuid compare (5 -> 3 lookups, -2 allocs,
  1.28x per chunk).

Blob cache (bench_blob_cache_index + round-3 regression guard):
- CachedBlobBackend index: tokio::sync::Mutex<LruCache> -> moka::sync::Cache
  with byte weigher. The mutex serialized every cached chunk read and scaled
  NEGATIVELY (2.08 -> 1.07 Mops/s from 1 -> 2 readers); moka probes are
  lock-free (2.17x at K=2). Byte budget now enforced by moka (manual
  current_size + collect_evictions machinery deleted); eviction listener
  unlinks size-evicted files only (Replaced entries keep their file —
  gated). Single-flight miss gate unchanged (16 concurrent misses -> 1
  fetch re-verified via the round-3 harness).
- put_blob now populates the cache BEFORE the inner backend consumes the
  source file (the old order failed 100% of the time — local renames,
  S3/Azure delete the source — so the first read after a whole-file put
  re-downloaded from the remote); inner-put failure invalidates the entry.

Frontend (vitest gates):
- List-view thumbnails request the 150px icon rendition instead of 400px
  preview into a 40px slot (~7.1x fewer pixels, ~4-5x fewer bytes per
  thumbnail across list views); grid keeps preview.

Rejected by its own bench (kept as evidence in bench_round12_micro §2):
- Single-pass compression predicate: the monomorphized And-chain already
  costs ~4.6 ns / 0 allocs total; the fused node measured within noise.

New migration: 20260719000000_users_search_trgm.sql (trgm indexes).
Deferred with prepared design: grouped file/grid view virtualization
(single-VirtualRows flatten, the photos pattern) — next round's headline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfidAJD5AHw23jtvBUNamB
2026-07-19 01:32:00 +00:00
Claude 221c1f31b0 perf: round 11 — StoragePath joined-only, classifier fusion, memoized bodies, query-shape pack, SPA fine-grained stars
Backend (each change benchmark-gated with BEFORE replicas + equivalence
gates; see examples/bench_round11_micro.rs, bench_round11_queries.rs,
bench_log_writer.rs and benches/ROUND11.md — final numbers land in the
follow-up doc commit):

- StoragePath re-representation: single canonical joined String, segments
  derived on demand; File/Folder drop the duplicated path_string field
  (4000→1000 allocs per 500-row listing page)
- Display classifier fusion: classify_display shares one stack-lowered
  extension across the three decision trees; call sites in FileDto,
  folder/favorites/recent handlers, trash, path-resolver (+ interning
  where Arc::from was still used)
- /status.php and /openapi.json memoized into OnceLock<Bytes> (openapi
  rebuilt a 171 KiB spec per request: 2.8 ms → 18 ns)
- NC upload-session PROPFIND: write! + pre-sized body + stack RFC2822
  dates (2.3-2.6x, 2582→772 allocs at 256 chunks)
- REST download: dead FileDto clone removed (capture mime/size + move)
- CalendarEventDto/TrashedItem into_parts moves (11 KiB ical_data memcpy
  gone per CalDAV row); CardDAV getlastmodified stack render
- 4xx path: borrowed ErrorResponse serialize, ErrorKind::as_str,
  not_found/already_exists clone kill
- vCard emit via write!; search page moved out with into_iter skip/take;
  content-hit UUIDs parsed once; group last-user check via HashSet
- RateLimiter: lock-free get + insert (and_upsert_with variant REJECTED
  by benchmark); CSRF token borrow-compare + borrowed cookie extraction
- Thumbnail/preview ETags built from as_str (Debug-identical bytes)
- Encrypted backend: encrypt_in_place_detached single-buffer write path,
  chunk-sized reserve in collect_stream; retry labels made lazy
- PG: deferred upload registration 3→1 round-trips (persist_file CTE
  template); direct_grant_cache for Calendar/AddressBook/Playlist authz
  (single-flight + set_role/clear_role invalidation); expand_user
  tokio::join!; geo clusters min(uuid)::text; recluster face assignment
  batched into one UNNEST update
- People recluster cosine: norms precomputed once (bit-identical gate)
- NC capabilities poll logs demoted to debug; tracing-appender dep added
  for the log-writer benchmark

Frontend:
- ResourceList.selectedEntries O(N)-per-toggle → id-index projection
  O(k log k); favorites/recent consume the batchToolbar snippet param and
  drop their duplicate filter + dead selectedIds mirror
- Recent: star state via new favoriteIds prop — a star click no longer
  rebuilds all N entries
- admin timeAgo >30d fallback uses the cached Intl.DateTimeFormat
- vitest gates in src/lib/components/round11.bench.test.ts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABhTEHuGujvwoodh67Kga7
2026-07-18 22:02:00 +00:00
Claude c51af68432 perf: round 10 — auth alloc purge, parent-herd batching, query-shape pack, NC 304s
Benchmark-gated (benches/ROUND10.md; every change carries a BEFORE/AFTER
harness with equivalence/safety gates — two designs were rejected or
rewritten by their own benches before adoption):

- Auth hot path: TokenClaims/CurrentUser display fields to Arc<str>, role
  to inline SmolStr end-to-end (Bearer, cookie, Basic-auth cache) — 4→1
  allocs per authenticated request, 3→0 per warm DAV request; JWT
  Encoding/Decoding/Validation built once.
- Cold shared-album herd: leader-inline parent batching in PgAclEngine
  (+ cascade try_get_with single-flight) — 100→2 parent queries per
  100-thumb cold herd, herd wall 1.9x, sequential + warm paths unchanged,
  all ROUND8/9 safety gates plus new herd-equivalence gates.
- Query-shape pack: share download double-fetch 2→1 (2.18x), contact-group
  COUNT(*) 14.9x, save_faces UNNEST 3.9x, playlist reorder UNNEST 63.7x
  (now atomic), search files∥folders join! 1.45x, move drive-lookup join!
  2.14x, trash partial (drive_id, trashed_at) indexes, CalDAV event-gate
  narrow read, favorites/recents binary-decode port, dead count_files
  removed.
- NC surface: preview + avatar honour If-None-Match (e2e: 5 KB and 197 KB
  → 0 bytes per revalidation), avatar WebP→PNG transcode memoised,
  PROPFIND/trashbin integer+date emits on stack formatters, folder-header
  enrichment join!, chunk-PUT retry stat folded into create_new open.
- common::fmt integer rendering rewritten on the std 2-digit LUT after the
  round's own bench caught the div-loop losing to to_string (16.1 ns vs
  22.5; speeds every prior-round call site).
- Micro-pack: WebDAV scope probe borrow-only, ShareService base_url
  snapshot, cookie_secure OnceLock, Arc'd AES-GCM cipher, stack request-id,
  tantivy analyzer clone dropped.
- SPA: search stale-guard + AbortController (10→1 completed round-trips,
  stale-clobber gone), getFolder in-flight dedup, gridColumns matchMedia
  hoist (10k→0 style reads).

Backend: cargo fmt + clippy -D warnings clean, 524 tests green.
Frontend: npm run check clean, 301 vitest green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DdM7V7M3QPW7HEHg3gLov
2026-07-18 20:33:50 +00:00
Dionisio Pozo 2317d594e3 Merge pull request #610 from EdouardVanbelle/security/grants2 2026-07-18 16:07:17 +02:00
Claude f53fba42a1 Merge branch 'main' into claude/performance-optimization-round-6
Resolves the one conflict in file_blob_read_repository.rs's
suggest_files_by_name: main added the CALLER_CAN_READ_DRIVE authz scope
(caller_id param + drive-membership filter, AuthZ audit finding #1 — the
suggest query previously leaked names/paths across tenants), round 6
switched the same query's id/folder_id columns to binary UUID decode.
Kept both: main's authz structure (format! + CALLER_CAN_READ_DRIVE +
caller_id bind) with round 6's binary decode (fi.id / fi.folder_id, no
::text) so the query matches the FileRow = (Uuid, …) tuple. The
deliberately-text sites (min(fm.file_id::text), folder path lookup)
stay text. Verified: build + clippy -D warnings clean, 524 unit +
554 integration tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-18 10:07:42 +00:00
Claude 9729f033b2 perf: round 6 backend — CardDAV cursor streaming, borrowed NC id chain, binary UUID decode, one-alloc hex
Benchmark-gated (equivalence + BEFORE/AFTER in examples/bench_*, results
and reproduce commands in benches/ROUND6.md):

- CardDAV whole-book REPORT + depth-1 PROPFIND stream through a PG
  cursor (stream_contacts_by_book, 500-contact pages) instead of
  materialising every vCard twice: 8 000 contacts TTFB 37.4 → 7.6 ms
  (4.9x), peak heap 19.0 → 7.0 MiB (2.7x), wall -23%; REPORT and
  PROPFIND byte-identical to the buffered writers.

- NC numeric-id chain fully borrowed: get_or_create_file_ids/folder_ids
  take &[&str] and return HashMap<Uuid, i64>; batch_resolve_ids callers
  (PROPFIND pages, REPORT, trashbin, OCS search) pass id slices and look
  up via nc_id_of. 2.006 → 0.006 allocs/child (334x), 1.53x wall per
  500-child page. batch_check_favorites binds &[&str] as text[].

- file_blob_read_repository listing SELECTs drop id::text/folder_id::text
  server casts: rows decode binary Uuid (16 vs 36 bytes on the wire) and
  render once in row_to_file. A/B on 500-row pages: 1.225 → 1.044 ms
  mean (1.17x), p95 1.686 → 1.345 (bench_uuid_text_cast; single-row,
  param and min() sites left as-is deliberately).

- IncrementalHasher::finalize_hex renders through common::fmt::hex_lower
  instead of one format! per digest byte: 18 → 1 (md5) / 35 → 1 (sha256)
  allocs per chunk finalize, 14-15x wall.

- Share landing overlaps the access-count UPDATE with the unlock fetch
  via tokio::join! (one round-trip off every public link hit).

- REJECTED by benchmark and reverted: try_join_all fan-out of the
  batch-favorites authz pre-check — 42.6 → 56.4 ms cold, 0.15 → 0.23 ms
  warm against local-socket PG (bench_favorites_authz kept as evidence).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-18 09:03:33 +00:00
Dionisio Pozo 4f51cb7aab Merge pull request #613 from AtalayaLabs/claude/performance-optimization-round-5
perf: round 5 — CalDAV cursor streaming, SPA interning gaps, NC href prefix, per-request micro-allocs
2026-07-18 10:30:11 +02:00
Edouard Vanbelle b1276938d4 security(upload): add permission to upload_file_streaming() 2026-07-17 19:01:05 +02:00
Claude 63cf6646d0 perf: round 5 — CalDAV cursor streaming, SPA interning gaps, NC href prefix, per-request micro-allocs
Seven benchmark-gated changes (benches/ROUND5.md; BEFORE/AFTER bench +
equivalence gate each, rollback rule as ROUND2-4 — two intermediate
CalDAV shapes measured worse and were themselves rolled back before
shipping):

- CalDAV whole-calendar responses (REPORT no-range/sync-collection,
  depth-1 collection PROPFIND, .ics GET): buffered double-residency →
  ONE window-ordered scan (MIN(start_time) OVER (PARTITION BY ical_uid))
  streamed through a PG cursor, pages cut at UID boundaries. TTFB
  23.3→11.0 ms (2.1x), peak heap 14.2→8.0 MiB at 4k events / 45→24 MiB
  at 12k, wall +9-15% (documented trade, ZIP-streaming class); both
  multistatus and ICS byte-identical to the buffered output. Rejected
  shapes kept in the doc: per-page GROUP-BY keyset (3-4x wall) and
  per-uid ANY hydration (~20 µs/index descent).
- SPA listing interning gaps: folder/recent/favorites resources handlers
  (and the WebDAV pseudo-root) called raw Arc::from per row for the
  closed display set ROUND3 interned — now intern_display/intern_mime,
  4→0 allocs/row, byte-identical Arc contents.
- NC PROPFIND child hrefs: username + parent path encoded once per
  request instead of per child (543→165 ns/row, 13→4 allocs); native
  WebDAV href drops its intermediate encode String.
- suggest enrichment: entity clone + field re-clones per keystroke row →
  consume + move (166.5→126.8 µs/200 rows, 20→7 allocs/row).
- list_readable_by returns the cache's Arc (246→128 ns warm hit, 4→0
  allocs) — deep Vec clone per DAV-selector request removed.
- CardDAV REPORT: borrowed props, reused href buffer, exact-size etag
  quoting (3.04→2.34 ms per 5k-contact getetag poll).
- Auth span records: user_id.to_string() per request ×3 →
  tracing::field::display.

Checks: cargo fmt, clippy --all-features --all-targets -D warnings,
cargo test --workspace (523 passed). Follow-ups (CardDAV streaming,
&[&str] id batches, ::text UUID casts A/B, share-landing join) recorded
in benches/ROUND5.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-17 15:19:00 +00:00
Dionisio Pozo 8aa013d3ee Merge pull request #605 from AtalayaLabs/claude/performance-optimization-analysis-sisndc
ROUND4: Row-path allocs, drive cache, CalDAV parse, PROPFIND emit, N+1 hydration, Azure streaming, faces bound
2026-07-17 15:56:34 +02:00
Claude 12dc648cff perf: round 4 — one-pass row paths, drive-selector cache, CalDAV single-parse, streamed Azure, batched hydration
Nine benchmark-gated changes (benches/ROUND4.md; every one ships with a
BEFORE/AFTER bench + equivalence gate, rollback rule as ROUND2/3):

- Row→entity path build: one-pass StoragePath::from_folder_and_name /
  from_joined + normalize_storage_name_owned + alloc-free Display —
  743→417 ns/file-row (1.78x), −5 allocs/row on every listing surface.
- WebDAV drive-selector: per-user readable_cache (single-flight, 30 s
  TTL, explicit invalidation incl. membership + group changes) replaces
  the grants join per request — 441 µs → 0.8 µs (~550x), 0 queries warm.
- CalDAV from_ical/update_ical_data: 8 full IcalParser runs per VEVENT
  → 1 (7.1x per PUT, 4.4x on 50-event imports); alloc-free split_vevents,
  chunk scan without the whole-body uppercase copy (1.4x), borrowed-key
  UID grouping (1.3x), REPORT props no longer cloned.
- PROPFIND emit: partition Vecs dropped (single-pass 404 list) + stack
  rendered RFC 3339/2822 dates, sizes, quoted etags (common::fmt,
  chrono-byte-identical, sweep-tested) on both DAV surfaces — 1.22x
  per page, 17.9→12.0 allocs/row.
- Grant-listing hydration: calendars/address books/playlists batch
  hydrate via = ANY($1) — 15 serial queries → 1 (~13x per sync poll).
- user-flags cache: get→insert → try_get_with single-flight (32→1
  queries per cold herd).
- Azure downloads: whole-blob Vec buffering → streamed SDK pages —
  TTFB 349→4 ms (87x), peak heap 480→1.9 MiB (254x) on 256 MiB blobs;
  new OXICLOUD_AZURE_ENDPOINT_URL override (Azurite/bench hook).
- Face indexing: unbounded per-image tokio::spawn → core-count
  semaphore, permit before blob read — peak heap 1175→176 MiB (6.7x).

Checks: cargo fmt, clippy --all-features --all-targets -D warnings,
cargo test --workspace (523 passed) + --features test_utils. hurl API
suite and dockerized integration DB not runnable in this environment —
left to CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-17 13:48:37 +00:00
Dionisio Pozo 807536b945 Merge pull request #601 from EdouardVanbelle/security/grants
security/grants
2026-07-17 13:31:03 +02:00
Claude cd4c62042a perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):

DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
  folders/files UNION-ALL branches as sargable row-value comparisons with
  per-branch ORDER/LIMIT + two partial expression indexes
  (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
  (19.5x); other sort modes at parity or better. New migration
  20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
  on the timeline index, joins moved above the top-N. 50k-photo library:
  97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
  comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
  idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
  (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).

Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
  connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
  now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
  concurrent cold readers = 16 full remote downloads racing truncating
  writes on ONE deterministic .tmp (corruptible cache); now 1 download
  (16x less egress, 2.8x wall on a shared link) and torn files can never
  be renamed into the cache.

I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
  chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
  pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
  every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
  common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
  clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
  O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
  9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
  (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
  ~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
  nodes, three SDK stacks gone from every build). tokio "process" is now
  an explicit feature (was enabled transitively by aws-config).

Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
  4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
  asserts output identity across locales and a 3x floor.

Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
2026-07-17 11:10:27 +00:00
Edouard Vanbelle 7aea383588 feat(antienum): 403 when sub can read, 404 otherwise
this is a UX improvement, always return a 404 not found when subject do not have any access on the resource
    but returns an explicit 403 forbidden is subject try a forbidden action on a resourse it can read

    regarding performance, the role is already in cache for the second call with read perm
2026-07-16 22:09:45 +02:00
Edouard Vanbelle c1924c825b security(search): ensure that search suggenstion returns answer the user has access to 2026-07-16 21:07:18 +02:00
Claude 82ee7da0d2 perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):

- Range requests (REST/DAV/shares) answered from the moka content cache
  for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
  256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
  with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
  Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
  per-session counter (lazy rebuild on cold start). 1,000-chunk upload
  gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
  buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
  latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
  the source stream keeps flowing during PG pin + backend writes;
  rollback ledger shared + lock-serialized so compensation stays exact
  on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
  OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
  a bounded Web Worker pool (File handles by reference); vitest gate
  asserts the pool beats sequential (first gate draft posting buffers
  was 2.6x slower and was rewritten — copies dominated).

Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
2026-07-16 16:50:07 +00:00
Claude aba89c4f5d perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
2026-07-16 14:20:20 +00:00
Edouard Vanbelle 7966c7178a fix(528): pass3: PUT with RECURRENCE-ID 2026-07-14 20:42:18 +02:00
Edouard Vanbelle f331dbf0ee feat(account): upgrade external to internal 2026-07-14 11:10:23 +02:00
Edouard Vanbelle ba620166ee feat(grant): clean up expired grants 2026-07-12 18:37:13 +02:00
Edouard Vanbelle c1e46910b0 feat(music): move playlist to authz engine 2026-07-08 01:04:03 +02:00
Edouard Vanbelle 0e341758af feat(calendar,addressbook): remove old sharing code 2026-07-08 01:03:25 +02:00
Edouard Vanbelle 0fcd617fd1 feat(calendar,addressbook): migrate share to authz engine
migrate DB entries to authz engine
    and wire authz engine to caldav and carddav
2026-07-08 01:03:25 +02:00
Edouard Vanbelle b5881c7114 feat(calendar,addressbook): prepare authz engine 2026-07-08 01:03:25 +02:00
Edouard Vanbelle 0342bae300 security(nextcloud): add authz to PUT verb 2026-07-05 22:53:05 +02:00
Edouard Vanbelle 37467ed9d3 feat(drive): remove all owner_id from {File,Folder}Dto 2026-07-03 01:13:53 +02:00
Edouard Vanbelle f5f5b1167f feat(drive): remove _for_owner() and use authz
remove related IDOR protection as Hurl tests are covering this surface
2026-07-02 01:14:02 +02:00