Oxicloud

cjw/Oxicloud

Fork 0

Commit Graph

Author	SHA1	Message	Date
Claude	fdf445d2b0	perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade Benchmark-gated round (benches/ROUND9.md): every change carries a BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from the committed harnesses on 4 cores / local PG 16. Backend: - Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced + sync_blobs — the trait default had silently reinstated HEAD-before-PUT per chunk on decorated remote stacks, undoing ROUND3 §8. Full production stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3). - NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT (bench_nc_enrich_join, injected-latency decide-by-bench). - Search enrichment consumes its DTOs and carries the interned Arc<str> display fields end-to-end (SearchFileResultDto type change, OpenAPI shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC REPORT conversion stops re-running all three classifiers per row (bench_search_enrich). - NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs), Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser> + lazy span render (11 -> 6/build) (bench_nc_session). - Storage micro-pack: atomic create_new chunk writes (2.1x fresh), stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro). - OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0 allocs/poll, byte-identical (bench_capabilities_static). - Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive (bench_drive_is_empty). - favorites/recents row-map ROUND7 port: path/name/blob_hash moved, -2.75 allocs/row (bench_resource_row_map §2). - Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page fetch, honest verdict incl. one noise-band wash documented (bench_folder_uuid_decode). - Authz: file cascade decision decomposed into memoized folder-level decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl. new direct-grant sibling isolation, revoke-flush re-verified, full integration authz suite green (bench_thumbnail_cascade_cache). Frontend (vitest gates committed beside the code): - resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x (recipients.bench.test.ts). - ResourceList selection-prune effect skips when nothing is selected (100 -> 0 Set builds per drain) and the photos timeline reads a listener-fed mobile flag instead of matchMedia per recompute (listDerives.bench.test.ts). Verification: cargo fmt + clippy --all-features --all-targets -D warnings clean; 524 unit + 554 integration (--cfg integration_tests) tests pass; frontend npm run check clean with 293 vitest tests green. Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder (maintainer sign-off), per-page batched parent resolution, JWT-claims Arc<str>, batch_operations signature widening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn	2026-07-18 16:12:04 +00:00
Claude	cd4c62042a	perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr	2026-07-17 11:10:27 +00:00

Author

SHA1

Message

Date

Claude

fdf445d2b0

perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade

Benchmark-gated round (benches/ROUND9.md): every change carries a
BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from
the committed harnesses on 4 cores / local PG 16.

Backend:
- Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced
  + sync_blobs — the trait default had silently reinstated HEAD-before-PUT
  per chunk on decorated remote stacks, undoing ROUND3 §8. Full production
  stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3).
- NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead
  props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT
  (bench_nc_enrich_join, injected-latency decide-by-bench).
- Search enrichment consumes its DTOs and carries the interned Arc<str>
  display fields end-to-end (SearchFileResultDto type change, OpenAPI
  shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC
  REPORT conversion stops re-running all three classifiers per row
  (bench_search_enrich).
- NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs),
  Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser>
  + lazy span render (11 -> 6/build) (bench_nc_session).
- Storage micro-pack: atomic create_new chunk writes (2.1x fresh),
  stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the
  Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for
  chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro).
- OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0
  allocs/poll, byte-identical (bench_capabilities_static).
- Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive
  (bench_drive_is_empty).
- favorites/recents row-map ROUND7 port: path/name/blob_hash moved,
  -2.75 allocs/row (bench_resource_row_map §2).
- Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page
  fetch, honest verdict incl. one noise-band wash documented
  (bench_folder_uuid_decode).
- Authz: file cascade decision decomposed into memoized folder-level
  decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album
  first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl.
  new direct-grant sibling isolation, revoke-flush re-verified, full
  integration authz suite green (bench_thumbnail_cascade_cache).

Frontend (vitest gates committed beside the code):
- resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x
  (recipients.bench.test.ts).
- ResourceList selection-prune effect skips when nothing is selected
  (100 -> 0 Set builds per drain) and the photos timeline reads a
  listener-fed mobile flag instead of matchMedia per recompute
  (listDerives.bench.test.ts).

Verification: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 524 unit + 554 integration (--cfg integration_tests) tests pass;
frontend npm run check clean with 293 vitest tests green.

Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder
(maintainer sign-off), per-page batched parent resolution, JWT-claims
Arc<str>, batch_operations signature widening.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn

2026-07-18 16:12:04 +00:00

Claude

cd4c62042a

perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning

Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):

DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
  folders/files UNION-ALL branches as sargable row-value comparisons with
  per-branch ORDER/LIMIT + two partial expression indexes
  (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
  (19.5x); other sort modes at parity or better. New migration
  20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
  on the timeline index, joins moved above the top-N. 50k-photo library:
  97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
  comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
  idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
  (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).

Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
  connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
  now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
  concurrent cold readers = 16 full remote downloads racing truncating
  writes on ONE deterministic .tmp (corruptible cache); now 1 download
  (16x less egress, 2.8x wall on a shared link) and torn files can never
  be renamed into the cache.

I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
  chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
  pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
  every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
  common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
  clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
  O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
  9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
  (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
  ~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
  nodes, three SDK stacks gone from every build). tokio "process" is now
  an explicit feature (was enabled transitively by aws-config).

Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
  4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
  asserts output identity across locales and a 3x floor.

Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr

2026-07-17 11:10:27 +00:00

2 Commits