this fix https://github.com/AtalayaLabs/OxiCloud/issues/607
which was introduced by commit 12dc648cff
when a user does activity in a drive, admin can invalidate cache via the internal
call /api/admin/internal/trigger-sweep
this permit end 2 end test to validte immediately that used_bytes corresponds to the expected result
this fix https://github.com/AtalayaLabs/OxiCloud/issues/607
which was introduced by commit 12dc648cff
when a user rename a root folder, this invalidate the cache
still some UX effect displaying phantom drive is grant is revoked,
cache is 30s of TTL so this UX glitch is acceptable
- nc chunked-upload MOVE assembly (#12): both branches now funnel through
update_file_streaming_with_perms, whose internal fork enforces Update on
the existing file OR Create on the parent folder / drive root. Pre-fix,
the create branch went through plain upload_file_streaming with no
authz.require — a Viewer on a shared drive could MKCOL → PUT chunks →
MOVE and land a brand-new file. Error mapping switched to AppError::from
so denials keep the graduated 403/404 shape.
- trash empty-for-drive: route through authz.require(Delete, Drive) instead
of the bespoke drives_with_delete_for check + hardcoded not_found. Viewer
now gets 403 (has Read), outsider stays 404 (no Read, anti-enum). Emits
the standard authz.denied event with visibility field instead of the
ad-hoc trash.empty_drive_rejected.
- tests/api/trash_per_drive.hurl: flip Viewer/Editor asserts 404 → 403;
new Step 11b regression pin for finding #10 (Editor restore + delete
attempts must 403 AND body must not contain "success":true — trips if
the historical substring-match-on-"not found" hack ever comes back).
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):
DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
folders/files UNION-ALL branches as sargable row-value comparisons with
per-branch ORDER/LIMIT + two partial expression indexes
(folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
(19.5x); other sort modes at parity or better. New migration
20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
on the timeline index, joins moved above the top-N. 50k-photo library:
97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
(5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).
Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
concurrent cold readers = 16 full remote downloads racing truncating
writes on ONE deterministic .tmp (corruptible cache); now 1 download
(16x less egress, 2.8x wall on a shared link) and torn files can never
be renamed into the cache.
I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
(OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
nodes, three SDK stacks gone from every build). tokio "process" is now
an explicit feature (was enabled transitively by aws-config).
Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
asserts output identity across locales and a 3x floor.
Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
this is a UX improvement, always return a 404 not found when subject do not have any access on the resource
but returns an explicit 403 forbidden is subject try a forbidden action on a resourse it can read
regarding performance, the role is already in cache for the second call with read perm
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):
- Range requests (REST/DAV/shares) answered from the moka content cache
for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
per-session counter (lazy rebuild on cold start). 1,000-chunk upload
gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
the source stream keeps flowing during PG pin + backend writes;
rollback ledger shared + lock-serialized so compensation stays exact
on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
a bounded Web Worker pool (File handles by reference); vitest gate
asserts the pool beats sequential (first gate draft posting buffers
was 2.6x slower and was rewritten — copies dominated).
Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
automatically create default Calendar and default addressbook per user
(no creation if user already have a such resource)
default name are "Personal"
this is using the user's life cycle like does the drives
answers to issue #545
- grants: permission moved to roles
- new resources (Drive, Caldav, Carddav, Playlist) now using ReBAC
- expired shared now cleaned up
- drive visible in Webdav
- new login/registration options (domain allow list, policies, etc)
- upgrade of external user into internal user
Edouard Vanbelle