perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
+80
-3
@@ -8,7 +8,9 @@ default-run = "oxicloud"
|
||||
[dependencies]
|
||||
mimalloc = { version = "0.1.52", default-features = false }
|
||||
axum = { version = "0.8.9", features = ["multipart", "http1", "http2", "tokio", "macros"] }
|
||||
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs"] }
|
||||
# "process" was previously enabled implicitly through aws-config's feature
|
||||
# unification; ffmpeg_video_frame_service needs it, so declare it ourselves.
|
||||
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs", "process"] }
|
||||
tokio-util = { version = "0.7.18", features = ["io", "codec", "compat"] }
|
||||
tokio-stream = { version = "0.1.18", features = ["fs", "sync"] }
|
||||
bytes = "1.11.1"
|
||||
@@ -86,9 +88,12 @@ dashmap = "6.2.1"
|
||||
socket2 = { version = "0.6.4", features = ["all"] }
|
||||
urlencoding = "2.1.3"
|
||||
utoipa = { version = "5.5.0", features = ["axum_extras", "uuid", "chrono"] }
|
||||
# NOTE: aws-config and aws-smithy-types were removed as direct deps in the
|
||||
# round-3 perf pass — S3BlobBackend builds its client purely from
|
||||
# aws_sdk_s3::config with static credentials; nothing referenced either
|
||||
# crate, and aws-config alone pulled aws-sdk-sso/ssooidc/sts (~90 crates)
|
||||
# into every build (benches/ROUND3.md).
|
||||
aws-sdk-s3 = "1.136.0"
|
||||
aws-config = { version = "1.8.18", features = ["behavior-version-latest"] }
|
||||
aws-smithy-types = "1.5.0"
|
||||
azure_core = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
azure_storage = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
azure_storage_blobs = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
@@ -278,6 +283,78 @@ name = "bench_owner_cache"
|
||||
path = "examples/bench_owner_cache.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-3 battery ─────────────────────────────────────────────────────────────
|
||||
|
||||
# Web-UI folder listing — whole-folder rescan + top-N sort per page vs keyset
|
||||
# pushdown into the UNION-ALL branches + (folder_id, LOWER(name), id) indexes
|
||||
# (needs the dev Postgres up).
|
||||
[[example]]
|
||||
name = "bench_listing_keyset"
|
||||
path = "examples/bench_listing_keyset.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Photos timeline — full-library scan + top-N above the grants join vs
|
||||
# per-drive LATERAL top-N on the media-timeline index (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_photos_timeline"
|
||||
path = "examples/bench_photos_timeline.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# PROPFIND subfolder paging — LIMIT/OFFSET + COUNT(*) OVER() per page vs
|
||||
# keyset batch, mirroring the files-side PROPFIND-PAGING fix (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_folder_keyset"
|
||||
path = "examples/bench_folder_keyset.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Basic-auth thundering herd — K concurrent cache misses each paying Argon2id
|
||||
# vs single-flight try_get_with (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_auth_herd"
|
||||
path = "examples/bench_auth_herd.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# CachedBlobBackend — miss stampede (N duplicate remote fetches racing on one
|
||||
# .tmp) vs per-hash single-flight; warm-hit index throughput. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_blob_cache"
|
||||
path = "examples/bench_blob_cache.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Upload spool/assembly I/O — ReaderStream capacity sweep on part-file reads
|
||||
# and BufWriter vs bare-File frame writes on the chunk spool path. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_upload_spool"
|
||||
path = "examples/bench_upload_spool.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# S3 chunk PUT — HEAD-before-PUT vs unconditional PUT against a local axum
|
||||
# stub with injected latency; Azure Bytes-vs-to_vec copy micro. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_s3_put"
|
||||
path = "examples/bench_s3_put.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# File/Folder -> DTO mapping allocations — Arc<str> interning of closed-set
|
||||
# display fields, 1-alloc etag/size formatting. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_dto_map"
|
||||
path = "examples/bench_dto_map.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# CardDAV REPORT — dead per-contact vCard pre-generation + O(N^2) uid scan vs
|
||||
# single on-demand generation. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_carddav_report"
|
||||
path = "examples/bench_carddav_report.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Search-results cache RSS — entry-count capacity vs byte weigher. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_search_cache_mem"
|
||||
path = "examples/bench_search_cache_mem.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
|
||||
Reference in New Issue
Block a user