The ROUND23-deferred download_zip N+1, given its own validated pass. The
individually-selected files were authorized + fetched one at a time via
get_file_with_perms (require + get = 2 serial round-trips/file) before any
streaming — a 200-file selection was 400 serial round-trips. AFTER routes the
whole multi-select through the new
FileRetrievalService::get_files_by_ids_with_perms: one check_files_read_batch
(the PgAclEngine resolves every file's drive in ONE query and primes the
resource->drive cache) + one get_files_by_ids. 2N round-trips -> 2.
Authorization is unchanged and still enforced BEFORE any ZIP entry is written:
- add_file_entry_streamed writes the entry header (the filename) before it opens
the authorized stream, so the pre-filter is load-bearing — a denied file must
never reach it or its name leaks into the archive. AFTER a denied/missing id
is absent from the authorized map and is skipped in the same input order,
exactly as the old loop skipped a denied get_file_with_perms; it never reaches
the entry write. The authz moved from a per-file require to one batch check
EARLIER in the same function, not into or after the stream.
- The stream open keeps its own per-file Read check (now a primed-cache hit) +
Recents recording; check_files_read_batch is documented + gated as identical
to looping require.
Because the change is authorization-sensitive, the gate is the security property
itself. bench_round24_zip_authz drives the real PgAclEngine over a seeded,
interleaved mix of owned (granted drive) + denied (other drive) + missing ids
and asserts: the batch inclusion set AND input order are identical to the
per-file require loop; the included set is exactly the caller's owned files; no
denied or missing id is ever included (the authz-regression tripwire); and the
batch fetch returns exactly the owned files. Latency (cold, 600-item 1/3-owned
selection): 559 -> 267 ms (2.10x; the realistic all-owned selection is O(1) ->
a larger win). See benches/ROUND24.md.
The folder selections are left as-is (root counts are small and there is no
check_folders_read_batch primitive to batch through).
Verified: cargo clippy --features bench --all-targets -D warnings clean, cargo
fmt --all --check clean, cargo test --lib --features bench = 529 passed / 0
failed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKyQ4AnYtgp1JtjzweyMeo
Benchmark-gated (equivalence + BEFORE/AFTER; results + reproduce commands in
benches/ROUND7.md):
- Photos timeline re-grouped + re-laid-out the whole accumulated library on
every 60-item page (both `groups` and `photoRows` were $derived over the
full list), Σ ≈ O(N²/60) main-thread work during a scroll. Pages arrive
newest-first so grouping is append-only: the new PhotoTimeline
(lib/utils/photoTimeline.ts) re-buckets only the fresh page and re-lays-out
only changed groups, reusing untouched groups' cached rows, falling back to
a full rebuild on any config/deletion/non-append change. The pure
buildPhotoRows is the verbatim reference the gate holds it equal to at every
page. 50×60 drain: 76 500 → 3 000 grouping ops (25.5x), 23.0 → 2.2 ms
(10.6x).
- Range downloads paid authz + access-notify twice: download_file_impl
resolves the file via get_file_with_perms, then the Range branch re-ran
require_file + notify_file_accessed per request. Media/PDF viewers fetch
exclusively via Range (one request per seek), so every seek in a scrub
re-authorized an already-cleared file. Now routed through the non-perms
get_file_range_preloaded (matching the share-landing + WebDAV range paths);
the unused _with_perms range method is removed. The request-level gate still
denies before the branch runs (bench asserts member granted, outsider
denied). Per seek removed: WARM 0.67 µs, COLD 1362.66 µs — a grant-cascade
drive-resolve query per seek for a shared-drive recipient on a cold cache.
- /api/folders/{id}/resources row→DTO mapping cloned row.name into the DTO
though the row is owned; folders move it (fixed icons), files compute the
name-derived icon/category classes first then move it. 500-row page:
10.004 → 9.004 allocs/row (500 clones removed), output identical.
Deferred with rationale in ROUND7.md: thumbnail ACL-before-304 (security
posture — needs a security review, not a perf tweak), batch_operations
Arc<str>→String widening, list-view O(N²) on smaller lists, and the serial→
join! pairs (decide-by-bench with injected latency, per the round-6 rejection).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):
- Range requests (REST/DAV/shares) answered from the moka content cache
for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
per-session counter (lazy rebuild on cold start). 1,000-chunk upload
gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
the source stream keeps flowing during PG pin + backend writes;
rollback ledger shared + lock-serialized so compensation stays exact
on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
a bounded Web Worker pool (File handles by reference); vitest gate
asserts the pool beats sequential (first gate draft posting buffers
was 2.6x slower and was rewritten — copies dominated).
Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
OptimizedFileContent::Mmap was constructed nowhere — the documented "Tier 2:
memory-mapped I/O (10-100 MB)" path was never wired, so optimized_inner only
ever returns Bytes (<10 MB) or Stream (>=10 MB). The variant survived only as
an enum case plus two dead match arms in the file and share download handlers.
Remove the variant and its arms, and fix the now-misleading retrieval-service
tier docs (everything >=10 MB streams via CDC chunk reassembly with the
backend read-ahead; there is no mmap tier). Behaviour is unchanged — the
deleted arms were unreachable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three list endpoints resolved each resource with one query per id:
- GET /api/grants/incoming and /api/grants/outgoing used
join_all(ids.map(get_file)) + join_all(ids.map(get_folder)), so a single
page (limit ≤ 200) could demand ~200 concurrent connections from the
20-connection primary pool, causing acquire-timeouts and head-of-line
blocking under load.
- The NextCloud favorites REPORT (oc:filter-files) fetched get_file/
get_folder once per favorite — up to N serial round-trips per sync.
Add by-ids batch reads that mirror the existing get_file/get_folder column
mapping and NOT is_trashed filter:
- FileBlobReadRepository::get_files_by_ids / FolderDbRepository::get_folders_by_ids
(one SELECT ... WHERE id = ANY($1)), exposed as FileRetrievalService::
get_files_by_ids / FolderService::get_folders_by_ids returning DTOs.
- Both grant handlers and the favorites REPORT now issue two batch queries
total and look results up by id, preserving original order. Missing ids
(stale grants whose resource was deleted, or trashed/removed favorites)
drop out exactly as before. No auth-semantics change: these paths already
resolved ids vetted by the authorization engine / favorites table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
FileContentCache (moka, 512 MiB) was keyed by the file UUID, so content that
the CDC store already deduplicates to ONE blob on disk was cached once PER
FILE in RAM: N files sharing a blob held N copies, all counting against the
512 MiB cap. With effective dedup the cache filled with duplicates and
thrashed.
Key it by the blob hash instead (already on FileDto::content_hash):
- The in-RAM cache now benefits from dedup like the disk does — each distinct
blob is cached once and shared across every file/user that references it,
so a download by user A warms the cache for user B's identical content.
- Content is immutable by hash, so entries never go stale; the existing
invalidate(file_id) calls become harmless no-ops (a UUID never matches a
hash key) and can be removed in a later cleanup.
- ETag is now the immutable content hash (strong validator).
- Guarded: a hash-less stub DTO disables caching for that request rather than
colliding every such file on the empty key.
Response Content-Type still comes from the DTO, not the cache, so keying does
not affect the served MIME (verified).
Benchmark (real moka, exact 512 MiB/weight config, 400 files x 4 MiB = 1600 MiB
working set, 4000 uniform-random accesses):
dedup 1x : file_id 35.8% hit / 2568 reads vs hash 35.6% / 2577 (no dedup -> no change; control)
dedup 5x : file_id 35.8% hit / 2570 reads vs hash 98.0% / 80 (32x fewer disk reads, RAM 512->320 MiB)
dedup 20x: file_id 35.1% hit / 2596 reads vs hash 99.5% / 20 (130x fewer disk reads, RAM 512->80 MiB)
The win scales with the dedup ratio; with no dedup it is a no-op.
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
- Add type aliases (FileRow, FolderRow, FolderRowPaginated, FolderRowOptUser) to reduce type complexity
- Simplify redundant closures in app_password_handler and webdav_handler
- Remove needless borrow in auth_handler
- Collapse nested if/let chains in login_lockout, webdav_lock, auth, rate_limit
- Box LockEntry in acquire() Err variant to fix large enum variant warning
- Rename DeviceCodeStatus::from_str to parse to avoid should_implement_trait lint
- Add #[allow(clippy::too_many_arguments)] and #[allow(clippy::result_unit_err)] where appropriate
- Convert integration_tests from cargo feature to custom cfg attribute
- Add check-cfg lint config in Cargo.toml for integration_tests cfg
Replace String with Arc<str> for fields that contain repeated static
values (mime_type, icon_class, icon_special_class, category) in
FileDto, FolderDto, and OptimizedFileContent.
These fields are computed from ~40 static lookup tables and cloned
on every request. With Arc<str>, clone becomes O(1) atomic increment
instead of O(n) heap allocation — saving thousands of allocations/s
under load.
Fields kept as String: id, name, path, folder_id, owner_id
(unique per item, rarely cloned).
Zero API impact — serde serializes Arc<str> identically to String.
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.
Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
Replace list_files_in_subtree (fetch_all → Vec) with stream_files_in_subtree
that returns a Pin<Box<dyn Stream<Item = Result<File/FileDto>>>> backed by a
PostgreSQL cursor via sqlx::fetch().
Changes:
- FileReadPort::stream_files_in_subtree() returns streaming cursor (no default)
- FileRetrievalUseCase::stream_files_in_subtree() maps File→FileDto on the fly
- FileBlobReadRepository: async_stream::try_stream! + sqlx::fetch() cursor
- batch_operations: consume stream into HashMap incrementally
- zip_service: consume stream into HashMap incrementally
- All stubs/mocks updated (return empty stream)
Eliminates:
- Double allocation: Vec<(9-tuple)> + Vec<File> materialized simultaneously
- Unbounded RAM proportional to subtree size (was ~500 bytes × N files)
- Latency: callers blocked until last row fetched from PG
RAM is now O(folders) for the HashMap, not O(files).
Replace String with Arc<str> for etag and content_type fields in the
content cache. String::clone() allocates and copies the full string on
every cache hit (O(n)), while Arc<str>::clone() is O(1) — just an atomic
ref-count increment.
This eliminates 2 heap allocations per cache hit on the hottest download
path. At 1000 req/s that is 2000 fewer alloc/dealloc cycles per second.
Changed files:
- cache_ports.rs: trait signatures String → Arc<str>
- file_content_cache.rs: CacheEntry fields, get/put methods, tests
- stubs.rs: StubContentCachePort signatures
- file_retrieval_service.rs: caller creates Arc<str> before put()
Replace BFS traversal that issued 2 SQL queries per folder (list_files +
list_folders) with 2 total queries using PostgreSQL ltree <@ operator:
1. list_subtree_folders: single GiST-indexed scan for all folders
2. list_files_in_subtree: single GiST-indexed join for all files
Files are grouped by folder_id in a HashMap, then iterated in directory
order (folders pre-sorted by path from SQL).
Changes across 4 architecture layers:
- Domain: FolderRepository::list_subtree_folders (default impl)
- Application ports: FolderUseCase, FileRetrievalUseCase, FileReadPort
- Application services: FolderService, FileRetrievalService passthroughs
- Infrastructure: PG implementations + ZipService rewrite
Query count: O(N) → O(1). Latency for 100-folder tree: ~200 round-trips → 3.
Standardize code formatting across all 173 Rust source files
using rustfmt. No functional changes - purely cosmetic.
This establishes a consistent code style baseline for the
project going forward.
- Remove Serialize/Deserialize from File, Folder, Session, User, Contact entities
- Create contact_persistence_dto.rs for JSONB persistence in infrastructure layer
- Update contact_pg_repository to use persistence DTOs
- Fix dependency on zip crate (downgrade from 7.2.0 to 2.1.0)
- Fix unused variable warnings in main.rs
- Move PathService import from domain to infrastructure
- Add missing fields to CoreServices and RepositoryServices
- Create proper service initialization in main.rs
Clean Architecture improvements:
- Domain layer no longer depends on serde framework
- Persistence concerns isolated to infrastructure layer
- TokenClaims in auth_service.rs is only exception (required for JWT)
Claude