Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):
- Range requests (REST/DAV/shares) answered from the moka content cache
for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
per-session counter (lazy rebuild on cold start). 1,000-chunk upload
gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
the source stream keeps flowing during PG pin + backend writes;
rollback ledger shared + lock-serialized so compensation stays exact
on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
a bounded Web Worker pool (File handles by reference); vitest gate
asserts the pool beats sequential (first gate draft posting buffers
was 2.6x slower and was rewritten — copies dominated).
Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
automatically create default Calendar and default addressbook per user
(no creation if user already have a such resource)
default name are "Personal"
this is using the user's life cycle like does the drives
answers to issue #545
- add resource kind filter (file, folder, drive) in shared section (localStorage stored)
- add user preferences serverside store
- add client side dotfile filter (show/hide dotfiles) (user perf stored, default: dotfiles are shown)
for security trashed dotfile are always displayed
protection added: if a folder has only hidden items, a notification invite user to display it
if a user rename or create a hidden item, a notification tells it to user
update_file_streaming_with_perms (the method behind every WebDAV/
NextCloud/WOPI PUT) never called the storage-usage-delta hook, so
drives.used_bytes and the RFC 4331 quota-used-bytes property never
reflected content written via WebDAV — only the REST multipart
upload path bumped usage. Extract apply_storage_usage_delta() from
maybe_update_storage_usage() and wire it into both branches: the
overwrite path applies new_size - old_size, the create path applies
the full size.
Also fixes the two RFC 4331 hurl tests that caught this:
nc_webdav_quota_properties.hurl had a Hurl parse error ([BasicAuth]
section keys can't mix literal+template, so the {user}~{folder}
composite marker is now pre-resolved via [Options] variable: before
being referenced as a single template), and both quota-properties
tests now retry the post-upload PROPFIND (matching the existing
drive_quota.hurl/user_envelope_quota.hurl pattern) since the delta
is applied fire-and-forget on a background task.
Drops the boolean is_personal() wrapper in favor of matching
DriveKind::Personal/Shared at the two call sites, matching the
exhaustive-match convention already used for DriveKind elsewhere
(as_str, parse, DriveKindDto::from).
ensure OIDC is supported during nextcloud login
flow is:
1. nextcloud
2. oxicloud login ( direct pass or OIDC according config)
3. drive picker (if user has multiple drive)
4. success page + backchannel login to nextcloud
strip_chroot_prefix replaces the hardcoded "Personal/" strip
in NC trashbin PROPFIND, OCS unified search, and REPORT
(favorites + search). Handles composed chroots, drops
cross-chroot items instead of surfacing malformed paths, and
fixes the leading-slash mismatch (FolderDto path has '/', DB
paths don't) that silently dropped every NC trashbin item
post-D3. OCS keeps a first-segment fallback (results
legitimately span drives, no single chroot).
uploads_handler switches to nc_to_internal_path(chroot, …)
for the two remaining hardcoded "Personal/" sites, closing
the D1 TODO markers.
RecentService::record_item_access is split from a new
record_item_access_internal (no authz) used by
RecentRecordingHook. Round 1's authz.require widened the
tokio::spawn race past tests/api/recent.hurl step 7; the
internal path skips the redundant Read gate — upstream
_with_perms already enforced it.
Tests: 8 unit tests pin strip_chroot_prefix (leading slash,
composed chroots, sibling-leak rejection, partial-prefix,
empty-chroot). drives_membership.hurl step 21b/22b cover
Editor upload → 201 / Viewer upload → 404 fresh + overwrite
with fixture cleanup at 30c. test_nc_move_copy_delete_trash
K1 pins the actual original-location value.
plan:
```
What step 5 is actually doing
Goal: stop writing to storage.files.user_id / storage.folders.user_id on new INSERTs. This is a prerequisite for step 6 (dropping the column entirely). The column has existed since the pre-D0 single-owner era; every read that used to key on it has already been migrated to drive-membership grants over the last N days.
What has to change together for this to be safe:
┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Piece │ What changes │ Why │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Swap storage.files uniqueness indexes from (folder_id, name, user_id) to (drive_id, folder_id, name) │ Otherwise, two new rows with user_id = NULL would both be allowed (PG treats NULLs as distinct) — │
│ (a) │ │ uniqueness silently breaks │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ ALTER user_id DROP NOT NULL on both tables │ Otherwise, dropping the INSERT bind violates NOT NULL and every write 500s │
│ (b) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Drop dead user_id-leading indexes │ Cheap cleanup — nothing scans them anymore │
│ (c) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ PL/pgSQL (d) │ Rewrite storage.copy_folder_tree without user_id in the INSERT column list │ Cross-drive copy runs entirely in SQL, needs the same treatment │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (e) │ ~12 INSERT column-list drops across folder_db_repository, file_blob_write_repository, drive_pg_repository, dedup_service, │ Actual write path │
│ │ folder_service, load-seed.rs │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (f) │ Simplify folder_db_repository::create_folder's parent lookup to only fetch drive_id (was fetching (user_id, drive_id)) │ It's fetching a value it no longer needs │
└──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
repl user_id by caller has read access in readonly functions
using CALLER_CAN_READ_DRIVE constant
ensure webdav preview is using the permission handler
add `include_in_photo_index` and `include_in_music_index` policies
both true for default personal drive
photo is implemented
music is not yet implemented
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
Edouard Vanbelle