perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
Generated
-119
@@ -341,37 +341,6 @@ version = "1.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
|
||||
|
||||
[[package]]
|
||||
name = "aws-config"
|
||||
version = "1.8.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e33f815b73a3899c03b380d543532e5865f230dce9678d108dc10732a8682275"
|
||||
dependencies = [
|
||||
"aws-credential-types",
|
||||
"aws-runtime",
|
||||
"aws-sdk-sso",
|
||||
"aws-sdk-ssooidc",
|
||||
"aws-sdk-sts",
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-http",
|
||||
"aws-smithy-json",
|
||||
"aws-smithy-runtime",
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-schema",
|
||||
"aws-smithy-types",
|
||||
"aws-types",
|
||||
"bytes",
|
||||
"fastrand 2.4.1",
|
||||
"hex",
|
||||
"http 1.4.0",
|
||||
"sha1 0.10.6",
|
||||
"time",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"url",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-credential-types"
|
||||
version = "1.2.14"
|
||||
@@ -470,82 +439,6 @@ dependencies = [
|
||||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-sso"
|
||||
version = "1.102.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8c82b3ac19f1431854f7ace3a7531674633e286bfdde21976893bfee36fd493b"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
"aws-runtime",
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-http",
|
||||
"aws-smithy-json",
|
||||
"aws-smithy-observability",
|
||||
"aws-smithy-runtime",
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-types",
|
||||
"aws-types",
|
||||
"bytes",
|
||||
"fastrand 2.4.1",
|
||||
"http 0.2.12",
|
||||
"http 1.4.0",
|
||||
"regex-lite",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-ssooidc"
|
||||
version = "1.104.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "321000d2b4c5519ee573f73167f612efd7329322d9b26969ad1979f0427f1913"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
"aws-runtime",
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-http",
|
||||
"aws-smithy-json",
|
||||
"aws-smithy-observability",
|
||||
"aws-smithy-runtime",
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-types",
|
||||
"aws-types",
|
||||
"bytes",
|
||||
"fastrand 2.4.1",
|
||||
"http 0.2.12",
|
||||
"http 1.4.0",
|
||||
"regex-lite",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-sts"
|
||||
version = "1.107.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3d0d328ba962af23ecfa3c9f23b98d3d35e325fa218d7f13d17a6bf522f8a560"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
"aws-runtime",
|
||||
"aws-smithy-async",
|
||||
"aws-smithy-http",
|
||||
"aws-smithy-json",
|
||||
"aws-smithy-observability",
|
||||
"aws-smithy-query",
|
||||
"aws-smithy-runtime",
|
||||
"aws-smithy-runtime-api",
|
||||
"aws-smithy-types",
|
||||
"aws-smithy-xml",
|
||||
"aws-types",
|
||||
"fastrand 2.4.1",
|
||||
"http 0.2.12",
|
||||
"http 1.4.0",
|
||||
"regex-lite",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-sigv4"
|
||||
version = "1.4.5"
|
||||
@@ -688,16 +581,6 @@ dependencies = [
|
||||
"aws-smithy-runtime-api",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-query"
|
||||
version = "0.60.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1a56d79744fb3edb5d722ef79d86081e121d3b9422cb209eb03aea6aa4f21ebd"
|
||||
dependencies = [
|
||||
"aws-smithy-types",
|
||||
"urlencoding",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aws-smithy-runtime"
|
||||
version = "1.11.3"
|
||||
@@ -4231,9 +4114,7 @@ dependencies = [
|
||||
"async-stream",
|
||||
"async-trait",
|
||||
"async_zip",
|
||||
"aws-config",
|
||||
"aws-sdk-s3",
|
||||
"aws-smithy-types",
|
||||
"axum",
|
||||
"azure_core",
|
||||
"azure_storage",
|
||||
|
||||
+80
-3
@@ -8,7 +8,9 @@ default-run = "oxicloud"
|
||||
[dependencies]
|
||||
mimalloc = { version = "0.1.52", default-features = false }
|
||||
axum = { version = "0.8.9", features = ["multipart", "http1", "http2", "tokio", "macros"] }
|
||||
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs"] }
|
||||
# "process" was previously enabled implicitly through aws-config's feature
|
||||
# unification; ffmpeg_video_frame_service needs it, so declare it ourselves.
|
||||
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs", "process"] }
|
||||
tokio-util = { version = "0.7.18", features = ["io", "codec", "compat"] }
|
||||
tokio-stream = { version = "0.1.18", features = ["fs", "sync"] }
|
||||
bytes = "1.11.1"
|
||||
@@ -86,9 +88,12 @@ dashmap = "6.2.1"
|
||||
socket2 = { version = "0.6.4", features = ["all"] }
|
||||
urlencoding = "2.1.3"
|
||||
utoipa = { version = "5.5.0", features = ["axum_extras", "uuid", "chrono"] }
|
||||
# NOTE: aws-config and aws-smithy-types were removed as direct deps in the
|
||||
# round-3 perf pass — S3BlobBackend builds its client purely from
|
||||
# aws_sdk_s3::config with static credentials; nothing referenced either
|
||||
# crate, and aws-config alone pulled aws-sdk-sso/ssooidc/sts (~90 crates)
|
||||
# into every build (benches/ROUND3.md).
|
||||
aws-sdk-s3 = "1.136.0"
|
||||
aws-config = { version = "1.8.18", features = ["behavior-version-latest"] }
|
||||
aws-smithy-types = "1.5.0"
|
||||
azure_core = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
azure_storage = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
azure_storage_blobs = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
||||
@@ -278,6 +283,78 @@ name = "bench_owner_cache"
|
||||
path = "examples/bench_owner_cache.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-3 battery ─────────────────────────────────────────────────────────────
|
||||
|
||||
# Web-UI folder listing — whole-folder rescan + top-N sort per page vs keyset
|
||||
# pushdown into the UNION-ALL branches + (folder_id, LOWER(name), id) indexes
|
||||
# (needs the dev Postgres up).
|
||||
[[example]]
|
||||
name = "bench_listing_keyset"
|
||||
path = "examples/bench_listing_keyset.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Photos timeline — full-library scan + top-N above the grants join vs
|
||||
# per-drive LATERAL top-N on the media-timeline index (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_photos_timeline"
|
||||
path = "examples/bench_photos_timeline.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# PROPFIND subfolder paging — LIMIT/OFFSET + COUNT(*) OVER() per page vs
|
||||
# keyset batch, mirroring the files-side PROPFIND-PAGING fix (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_folder_keyset"
|
||||
path = "examples/bench_folder_keyset.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Basic-auth thundering herd — K concurrent cache misses each paying Argon2id
|
||||
# vs single-flight try_get_with (needs Postgres).
|
||||
[[example]]
|
||||
name = "bench_auth_herd"
|
||||
path = "examples/bench_auth_herd.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# CachedBlobBackend — miss stampede (N duplicate remote fetches racing on one
|
||||
# .tmp) vs per-hash single-flight; warm-hit index throughput. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_blob_cache"
|
||||
path = "examples/bench_blob_cache.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Upload spool/assembly I/O — ReaderStream capacity sweep on part-file reads
|
||||
# and BufWriter vs bare-File frame writes on the chunk spool path. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_upload_spool"
|
||||
path = "examples/bench_upload_spool.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# S3 chunk PUT — HEAD-before-PUT vs unconditional PUT against a local axum
|
||||
# stub with injected latency; Azure Bytes-vs-to_vec copy micro. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_s3_put"
|
||||
path = "examples/bench_s3_put.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# File/Folder -> DTO mapping allocations — Arc<str> interning of closed-set
|
||||
# display fields, 1-alloc etag/size formatting. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_dto_map"
|
||||
path = "examples/bench_dto_map.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# CardDAV REPORT — dead per-contact vCard pre-generation + O(N^2) uid scan vs
|
||||
# single on-demand generation. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_carddav_report"
|
||||
path = "examples/bench_carddav_report.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Search-results cache RSS — entry-count capacity vs byte weigher. No Postgres.
|
||||
[[example]]
|
||||
name = "bench_search_cache_mem"
|
||||
path = "examples/bench_search_cache_mem.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
# Round 3 — listing/timeline SQL shapes, auth herd, blob-cache stampede, spool I/O, DTO allocs
|
||||
|
||||
Twelve benchmark-gated changes. Rule of the round (same as ROUND2): every
|
||||
change ships with a BEFORE/AFTER benchmark; an AFTER that doesn't beat its
|
||||
BEFORE gets rolled back — none did. Equivalence gates (byte-identical
|
||||
output / identical row sequences) guard every behavior-preserving rewrite.
|
||||
|
||||
Measured on 4 cores / 15 GiB, local PostgreSQL 16 (fsync off), release
|
||||
profile. Reproduce any row with the command in its section.
|
||||
|
||||
## Summary
|
||||
|
||||
| # | change | key metric | before → after |
|
||||
|--:|---|---|---|
|
||||
| 1 | Web-UI listing keyset pushdown | ms/page p50, 20k-entry folder | 26.6 → 1.30 (**19.5x**) |
|
||||
| 2 | Photos timeline LATERAL top-N | ms/page p50, 50k-photo library | 97.4 → 1.61 (**55.7x**) |
|
||||
| 3 | PROPFIND subfolder keyset | full walk, 5k dirs | 79.7 → 17.9 ms (**4.5x**) |
|
||||
| 4 | Basic-auth single-flight | herd CPU, 8 conns | 2620 → 300 ms (**8.7x**) |
|
||||
| 5 | Blob-cache miss single-flight | remote fetches / wall | 16 → 1, 519 → 188 ms (**2.8x**) |
|
||||
| 6 | Chunk-assembly read buffer 512K | wall / read syscalls | 251 → 109 ms (**2.3x**), 2580 → 340 |
|
||||
| 7 | Chunk-spool BufWriter 512K | wall / write syscalls | 877 → 158 ms (**5.6x**), 12800 → 400 |
|
||||
| 8 | S3/Azure unsynced PUT (no HEAD) | wall / requests, 500 chunks | 1604 → 868 ms (**1.8x**), 1000 → 500 |
|
||||
| 9 | DTO mapping interning | allocs/row file / folder | 11.0 → 4.0, 11.8 → 1.0 |
|
||||
| 10 | CardDAV REPORT dead work | 5k contacts, getetag | 55.7 → 5.7 ms (**9.8x**) |
|
||||
| 11 | Search-cache byte weigher | retained RSS worst case | ~298 MiB → 31.9 MiB (bounded) |
|
||||
| 12 | Drop aws-config/aws-smithy-types | dep-graph nodes | 1728 → 1646 |
|
||||
|
||||
Frontend (gated by vitest, `frontend/src/lib/utils/formatDate.bench.test.ts`):
|
||||
cached `Intl.DateTimeFormat` — 20k dates 2612 → 50.6 ms (**51.6x**), output
|
||||
identity asserted across locales.
|
||||
|
||||
---
|
||||
|
||||
## [1] Web-UI folder listing — whole-folder rescan → per-branch keyset — 19.5x
|
||||
|
||||
`list_resources_paged` (SPA files view) applied its keyset cursor OUTSIDE
|
||||
the folders/files UNION-ALL on computed columns (`sort_str = LOWER(name)`,
|
||||
`folder_first`), so Postgres re-scanned and top-N-sorted every remaining
|
||||
row of the folder on every page (EXPLAIN: Seq Scan, 17,999 rows removed by
|
||||
filter, 29 ms / 565 buffers per 200-row page on a 20k-file folder).
|
||||
|
||||
Now the cursor is pushed into each branch as a sargable row-value
|
||||
comparison on base columns (`(LOWER(name), id) > ($str, $id)`), constants
|
||||
folded per branch in Rust (a cursor in the file group drops the folder
|
||||
branch outright), each branch pre-sorts + pre-limits, and the outer query
|
||||
merges ≤ 2·limit rows. Two new expression indexes (migration
|
||||
`20260918000000`): `idx_files_folder_lname (folder_id, LOWER(name), id)`
|
||||
and `idx_folders_parent_lname (parent_id, LOWER(name), id)`, both partial
|
||||
on `NOT is_trashed`.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_listing_keyset
|
||||
# full drain, 20k files + 300 dirs, 200/page total ms p50/pg p99/pg
|
||||
# name OLD/no-idx 2717.2 26.57 33.55
|
||||
# name OLD/idx (indexes alone don't help) 2786.8 27.83 35.62
|
||||
# name NEW/idx 139.6 1.30 1.81 19.5x
|
||||
# modified_at OLD → NEW (no dedicated index) 1653.4 → 1367.5 1.2x
|
||||
```
|
||||
|
||||
Equivalence: the drained `(type, id)` sequence is asserted identical across
|
||||
all modes and both sort orders; the example exits 1 on mismatch.
|
||||
|
||||
## [2] Photos timeline — full-library scan → per-drive LATERAL top-N — 55.7x
|
||||
|
||||
`list_media_files` claimed `idx_files_media_timeline_by_drive` let LIMIT
|
||||
stop the scan early; EXPLAIN refuted it — the folders/file_metadata joins
|
||||
and the global sort sat ABOVE the `drive_id IN (grants)` nested loop, so
|
||||
every page fed the ENTIRE media library through the join into a top-N
|
||||
heapsort. Now the accessible drive ids materialise once, a
|
||||
`CROSS JOIN LATERAL (… ORDER BY media_sort_date DESC LIMIT k)` per drive
|
||||
does one bounded index scan each, and the joins run on the k emitted rows
|
||||
only.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_photos_timeline
|
||||
# 10 pages of 100, 50k photos, 3 drives total ms p50 ms/page
|
||||
# OLD 1032.1 97.41
|
||||
# NEW 18.5 1.61 55.7x
|
||||
```
|
||||
|
||||
Equivalence: page-by-page id sequences asserted identical (seed uses
|
||||
strictly distinct capture dates so ties can't mask reordering).
|
||||
|
||||
## [3] PROPFIND subfolder paging — LIMIT/OFFSET + COUNT(*) OVER() → keyset — 4.5x
|
||||
|
||||
The exact quadratic shape PROPFIND-PAGING fixed for files still applied to
|
||||
sub-folders on both DAV surfaces: every page window-aggregated and
|
||||
re-scanned all N sub-folders, and the total was only used for `has_next`.
|
||||
New `FolderRepository::list_folders_batch` (keyset `name > $last`, served
|
||||
by the existing `idx_folders_unique_name`, no migration) wired into both
|
||||
streaming PROPFIND walkers via `list_folders_batch_with_perms` (same
|
||||
per-batch authz as before).
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_folder_keyset
|
||||
# full walk, 5k dirs, 500/page total ms p50 ms/page
|
||||
# OFFSET 79.7 6.54
|
||||
# KEYSET 17.9 1.64 4.5x
|
||||
```
|
||||
|
||||
## [4] Basic-auth cache — thundering herd → single-flight — 8.7x CPU
|
||||
|
||||
Every DAV/NC request authenticates via `verify_basic_auth`. On a cache
|
||||
miss each concurrent caller independently ran the full slow path — an
|
||||
Argon2id verification (m=64 MiB, t=3, p=2 ≈ 290 ms CPU here) apiece. DAV
|
||||
sync clients hold 4-8 parallel connections, so every TTL expiry (300 s)
|
||||
fanned out K verifications: a recurring p99 spike + CPU/RAM burst.
|
||||
`try_get_with` now coalesces concurrent misses; errors are never cached
|
||||
(brute-force cost preserved), revocation via `invalidate_entries_if`
|
||||
unchanged.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_auth_herd
|
||||
# herd of 8, cold cache wall ms CPU ms verifications
|
||||
# BEFORE (per-caller) 764 2620 9.0
|
||||
# AFTER (single-flight) 311 300 1.0
|
||||
# warm hit p50: 0.6 us
|
||||
```
|
||||
|
||||
## [5] CachedBlobBackend — miss stampede → per-hash single-flight — 16 fetches → 1
|
||||
|
||||
K concurrent cold readers of one blob (video player's parallel Range
|
||||
probes; N clients pulling the same new file) each downloaded the FULL blob
|
||||
from S3/Azure — and raced truncating writes on ONE deterministic `.tmp`
|
||||
path (a torn interleaving could be renamed into the cache). Fixes: a
|
||||
per-hash DashMap gate (leader fetches, waiters re-check and serve
|
||||
locally), plus unique `.{uuid}.tmp` names + error-path cleanup so a
|
||||
corrupt file can never land at the final path.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_blob_cache
|
||||
# 16 cold readers, 32 MiB blob, shared 1 GiB/s link wall ms fetches remote MiB
|
||||
# BEFORE (per-caller) 519 16 512
|
||||
# AFTER (single-flight) 188 1 32
|
||||
# gates: fetch count == 1; BLAKE3 of served + durable cache file == source
|
||||
```
|
||||
|
||||
## [6][7] Upload spool I/O — 64 KiB reads, unbuffered frame writes
|
||||
|
||||
Assembly read (`stream_from_files`, the single read pass over every
|
||||
completed chunked upload) used 64 KiB `ReaderStream` polls — one
|
||||
blocking-pool dispatch + read(2) each — while every other blob path uses
|
||||
256 KiB+. Capacity sweep picked 512 KiB. Chunk-spool writes
|
||||
(`stream_body_to_path`, every chunk PUT on both surfaces) went straight to
|
||||
a bare tokio File — one dispatch + write(2) per ~16-64 KiB HTTP frame; now
|
||||
wrapped in `BufWriter::with_capacity(512 KiB)` like the dedup handler's
|
||||
spool loop.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_upload_spool
|
||||
# [1] read 16 x 10 MiB parts wall ms read syscalls
|
||||
# 64K (BEFORE) 250.8 2580
|
||||
# 256K 125.1 660
|
||||
# 512K (AFTER) 108.8 340 2.3x
|
||||
# 1M 111.3 180
|
||||
# [2] spool 640 x 16 KiB frames x 20 files
|
||||
# bare File (BEFORE) 877.4 12800 syscw
|
||||
# BufWriter 512K (AFTER) 157.9 400 syscw 5.6x
|
||||
```
|
||||
|
||||
## [8] S3/Azure chunk writes — HEAD-before-PUT → unconditional PUT — 1.8x
|
||||
|
||||
Neither remote backend overrode `put_blob_from_bytes_unsynced`, so the
|
||||
dedup settle path (every NEW chunk of every upload) routed through
|
||||
`put_blob_from_bytes` and its "idempotent" HEAD/get_properties probe —
|
||||
2 round-trips per chunk for chunks the dedup layer already knows are new.
|
||||
Content-addressed keys make re-PUTs overwrite-safe, so the new overrides
|
||||
PUT directly. Azure additionally stopped copying every chunk
|
||||
(`data.to_vec()` → `Bytes` into `azure_core::Body`): 0.44 ms + 4 MiB
|
||||
transient alloc per 4 MiB chunk removed.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_s3_put
|
||||
# 500 x 256 KiB chunks, concurrency 8, 10 ms/request stub
|
||||
# BEFORE (HEAD+PUT) 1604 ms 500 HEADs + 500 PUTs
|
||||
# AFTER (PUT only) 868 ms 500 PUTs 1.8x
|
||||
```
|
||||
|
||||
## [9] Entity → DTO mapping — closed-set interning + 1-alloc formatting
|
||||
|
||||
`Arc::<str>::from(&'static str)` always allocates+copies, so every file
|
||||
row paid 4 allocations for values drawn from a ~60-string closed set
|
||||
(icon class, special class, category, mime), plus 2-alloc etag and 2-alloc
|
||||
size formatting; FolderDto additionally built its etag twice and cloned 4
|
||||
Strings it could move. Now: `LazyLock` intern tables (lookup + refcount
|
||||
bump; unknown values fall back to `Arc::from`, same bytes), single-alloc
|
||||
`compute_etag`/`format_file_size`, and `Folder::into_parts()` moves.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_dto_map
|
||||
# 10k rows ns/row allocs/row
|
||||
# File→FileDto BEFORE 1229.2 10.96
|
||||
# File→FileDto AFTER 1004.9 3.96
|
||||
# Folder→FolderDto BEFORE 425.2 11.80
|
||||
# Folder→FolderDto AFTER 204.5 1.00
|
||||
# gate: all DTO fields byte-identical BEFORE vs AFTER (10k files + 10k folders)
|
||||
```
|
||||
|
||||
## [10] CardDAV REPORT — dead double vCard generation + O(N²) scan — 9.8x
|
||||
|
||||
`handle_report` pre-generated a vCard for EVERY contact; the adapter then
|
||||
did a linear uid `find` per contact — O(N²) string compares — and
|
||||
DISCARDED the result (`let _ = vcard`), regenerating on demand inside
|
||||
`write_contact_response` anyway. Pure dead work, deleted; `contact_to_vcard`
|
||||
also switched `push_str(&format!(…))` → `write!` (one temp String per
|
||||
vCard line removed).
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_carddav_report
|
||||
# N=5000 getetag 55.7 → 5.7 ms 9.8x
|
||||
# N=5000 getetag+address-data 76.2 → 15.3 ms 5.0x
|
||||
# gate: REPORT XML byte-identical BEFORE vs AFTER for all prop sets
|
||||
```
|
||||
|
||||
## [11] Search-results cache — entry count → byte weigher — bounded RSS
|
||||
|
||||
The cache was capped at 1000 ENTRIES with a 300 s TTL; each entry holds up
|
||||
to 500 enriched rows (~10 owned Strings each) and keys include
|
||||
user+query+offset+limit, so every keystroke/page/user minted an entry —
|
||||
~300 MiB of invisible RSS was reachable. Now a byte weigher + 32 MiB
|
||||
budget (`OXICLOUD_SEARCH_CACHE_MAX_BYTES`), same TTL, same read latency.
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_search_cache_mem
|
||||
# 1000 pages x 500 rows retained bytes get() p50
|
||||
# BEFORE (1000 entries) ~298 MiB (9.3x) 155 ns
|
||||
# AFTER (32 MiB weigher) 31.9 MiB 155 ns parity 1.00x
|
||||
```
|
||||
|
||||
## [12] Cargo — drop aws-config + aws-smithy-types
|
||||
|
||||
Both were direct dependencies with ZERO references in the codebase —
|
||||
`S3BlobBackend` builds its client purely from `aws_sdk_s3::config` with
|
||||
static credentials. `aws-config` alone dragged aws-sdk-sso, aws-sdk-ssooidc
|
||||
and aws-sdk-sts into every build. Dependency-graph nodes: 1728 → 1646.
|
||||
`tokio`'s `process` feature (used by the ffmpeg thumbnailer) was only
|
||||
enabled transitively through aws-config's feature unification — it is now
|
||||
declared explicitly.
|
||||
|
||||
## Frontend — cached Intl.DateTimeFormat — 51.6x
|
||||
|
||||
`formatDate` (and four sibling callsites) constructed a fresh
|
||||
`Intl.DateTimeFormat` per call (~131 µs each here) — paid roughly twice
|
||||
per row while rendering/scrolling file lists. Module-scope cache keyed by
|
||||
(locale, options), invalidated on `languagechange`.
|
||||
|
||||
```
|
||||
cd frontend && npx vitest run src/lib/utils/formatDate.bench.test.ts
|
||||
# 20k dates: cached 50.6 ms vs per-call 2612.0 ms (51.6x); output-identity
|
||||
# matrix across en/es/ar/ja and every option shape used by the app
|
||||
```
|
||||
|
||||
## Audited but NOT adopted (for the record)
|
||||
|
||||
- **Fat LTO / panic=abort / OpenAPI LazyLock**: refuted by the verification
|
||||
pass (sub-1% plausible gain, or cold paths; `catch_unwind` shields
|
||||
pdf-extract so panic=abort is off the table).
|
||||
- **Chained clone-on-hit drive caches, localeCompare→Intl.Collator**:
|
||||
measured previously — residual gains are noise or regressions
|
||||
(benches/CHROOT-CACHE.md, benches/NPLUS1-AND-CACHES.md).
|
||||
- **Follow-ups worth a future round** (confirmed real, not yet gated):
|
||||
grouped/swimlane files view is unvirtualized (10k-row DOM); Azure
|
||||
download path buffers whole blobs in RAM (needs an Azurite-gated bench);
|
||||
face-indexing spawns unbounded per-image tasks; WebDAV drive-selector
|
||||
resolution re-runs the grants join per request (cacheable like
|
||||
CHROOT-CACHE); `make_file_path` split→rejoin + NFC copy per listing row.
|
||||
@@ -0,0 +1,209 @@
|
||||
//! Basic-auth thundering-herd benchmark — K concurrent cache misses.
|
||||
//!
|
||||
//! Every WebDAV/CalDAV/CardDAV/NextCloud request authenticates through
|
||||
//! `AppPasswordService::verify_basic_auth`. The cache (TTL 300 s) used to be
|
||||
//! a plain get/insert: when a sync client holding K parallel connections hit
|
||||
//! an expired entry, all K in-flight requests missed simultaneously and each
|
||||
//! ran the full slow path — an Argon2id verification at ~64 MiB / t=3 / p=2
|
||||
//! apiece (100-300 ms CPU each). `try_get_with` now coalesces concurrent
|
||||
//! misses into ONE verification; failed verifications stay uncached.
|
||||
//!
|
||||
//! Sections:
|
||||
//! BEFORE (emulated) — K concurrent bare Argon2id verifications, the exact
|
||||
//! work the old code fanned out per herd
|
||||
//! AFTER — K concurrent verify_basic_auth on a cold cache
|
||||
//! (single-flight: 1 verification, K-1 waiters)
|
||||
//! warm-hit — p50 of the cached path
|
||||
//!
|
||||
//! Gate: AFTER's process-CPU delta must be ~1 verification (< 2x a single
|
||||
//! verify), while BEFORE burns ~K of them. All K results must be Ok and
|
||||
//! identical.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL / OXICLOUD_DB_CONNECTION_STRING
|
||||
//! from .env):
|
||||
//! cargo run --release --features bench --example bench_auth_herd
|
||||
//! Tunables: BENCH_HERD (8)
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use oxicloud::application::services::app_password_service::AppPasswordService;
|
||||
use oxicloud::infrastructure::repositories::pg::{AppPasswordPgRepository, UserPgRepository};
|
||||
use oxicloud::infrastructure::services::password_hasher::Argon2PasswordHasher;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Process CPU time (utime + stime) in seconds, from /proc/self/stat.
|
||||
fn cpu_seconds() -> f64 {
|
||||
let stat = std::fs::read_to_string("/proc/self/stat").expect("stat");
|
||||
// utime/stime are fields 14/15 (1-indexed) — index past the comm field
|
||||
// (it can contain spaces) via the closing paren.
|
||||
let rest = &stat[stat.rfind(')').unwrap() + 2..];
|
||||
let fields: Vec<&str> = rest.split_whitespace().collect();
|
||||
let utime: f64 = fields[11].parse().expect("utime");
|
||||
let stime: f64 = fields[12].parse().expect("stime");
|
||||
let hz = 100.0; // USER_HZ on all mainstream Linux configs
|
||||
(utime + stime) / hz
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL");
|
||||
let herd: usize = env_or("BENCH_HERD", 8);
|
||||
|
||||
let pool = Arc::new(
|
||||
PgPoolOptions::new()
|
||||
.max_connections(10)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect"),
|
||||
);
|
||||
|
||||
// ── Seed: user + NC-format app password (production Argon2 params) ──
|
||||
let username = format!("bench_herd_{}", std::process::id());
|
||||
let user_id: uuid::Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, password_hash, role)
|
||||
VALUES ($1, $2, '', 'user') RETURNING id",
|
||||
)
|
||||
.bind(&username)
|
||||
.bind(format!("{username}@bench.invalid"))
|
||||
.fetch_one(pool.as_ref())
|
||||
.await
|
||||
.expect("seed user");
|
||||
|
||||
// Production defaults: m=64 MiB, t=3, p=2 (config.rs auth defaults).
|
||||
let hasher = Arc::new(Argon2PasswordHasher::new(65536, 3, 2));
|
||||
let svc = Arc::new(AppPasswordService::new(
|
||||
Arc::new(AppPasswordPgRepository::new(pool.clone())),
|
||||
hasher.clone(),
|
||||
Arc::new(UserPgRepository::new(pool.clone())),
|
||||
"http://localhost".into(),
|
||||
));
|
||||
let (_ap_id, plain) = svc.create_nc(user_id, "bench").await.expect("create_nc");
|
||||
|
||||
// ── Single-verify baseline (what one Argon2id run costs here) ──────
|
||||
use oxicloud::application::ports::auth_ports::PasswordHasherPort;
|
||||
let ref_hash = hasher.hash_password("benchpw").await.expect("hash");
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
assert!(
|
||||
hasher
|
||||
.verify_password("benchpw", &ref_hash)
|
||||
.await
|
||||
.expect("verify")
|
||||
);
|
||||
let one_wall = t.elapsed().as_secs_f64();
|
||||
let one_cpu = cpu_seconds() - c;
|
||||
println!(
|
||||
"single Argon2id verify: {:.0} ms wall, {:.0} ms CPU",
|
||||
one_wall * 1000.0,
|
||||
one_cpu * 1000.0
|
||||
);
|
||||
|
||||
// ── BEFORE (emulated): K concurrent bare verifications ─────────────
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..herd {
|
||||
let h = hasher.clone();
|
||||
let rh = ref_hash.clone();
|
||||
set.spawn(async move { h.verify_password("benchpw", &rh).await.expect("verify") });
|
||||
}
|
||||
while let Some(r) = set.join_next().await {
|
||||
assert!(r.expect("join"));
|
||||
}
|
||||
let before_wall = t.elapsed().as_secs_f64();
|
||||
let before_cpu = cpu_seconds() - c;
|
||||
|
||||
// ── AFTER: K concurrent verify_basic_auth on a cold cache ──────────
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..herd {
|
||||
let s = svc.clone();
|
||||
let u = username.clone();
|
||||
let p = plain.clone();
|
||||
set.spawn(async move { s.verify_basic_auth(&u, &p).await });
|
||||
}
|
||||
let mut ids = Vec::new();
|
||||
while let Some(r) = set.join_next().await {
|
||||
let (uid, uname, _, _) = r.expect("join").expect("verify_basic_auth");
|
||||
assert_eq!(uname, username);
|
||||
ids.push(uid);
|
||||
}
|
||||
assert!(ids.iter().all(|&u| u == user_id));
|
||||
let after_wall = t.elapsed().as_secs_f64();
|
||||
let after_cpu = cpu_seconds() - c;
|
||||
|
||||
// ── Warm hit p50 ────────────────────────────────────────────────────
|
||||
let mut lat = Vec::with_capacity(10_000);
|
||||
for _ in 0..10_000 {
|
||||
let t = Instant::now();
|
||||
let _ = svc
|
||||
.verify_basic_auth(&username, &plain)
|
||||
.await
|
||||
.expect("warm hit");
|
||||
lat.push(t.elapsed().as_secs_f64() * 1e6);
|
||||
}
|
||||
lat.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
let warm_p50 = lat[lat.len() / 2];
|
||||
|
||||
println!("\n# herd of {herd} concurrent Basic Auth verifications, cold cache");
|
||||
println!(
|
||||
"{:<22} {:>10} {:>10} {:>14}",
|
||||
"variant", "wall ms", "CPU ms", "verifications"
|
||||
);
|
||||
println!(
|
||||
"{:<22} {:>10.0} {:>10.0} {:>14.1}",
|
||||
"BEFORE (per-caller)",
|
||||
before_wall * 1000.0,
|
||||
before_cpu * 1000.0,
|
||||
before_cpu / one_cpu
|
||||
);
|
||||
println!(
|
||||
"{:<22} {:>10.0} {:>10.0} {:>14.1}",
|
||||
"AFTER (single-flight)",
|
||||
after_wall * 1000.0,
|
||||
after_cpu * 1000.0,
|
||||
after_cpu / one_cpu
|
||||
);
|
||||
println!("warm cache hit p50: {warm_p50:.1} us");
|
||||
|
||||
// ── Cleanup ─────────────────────────────────────────────────────────
|
||||
let _ = sqlx::query("DELETE FROM auth.app_passwords WHERE user_id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool.as_ref())
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool.as_ref())
|
||||
.await;
|
||||
|
||||
// ── Gate ────────────────────────────────────────────────────────────
|
||||
// AFTER must coalesce to ~1 verification's CPU; 2x headroom for
|
||||
// scheduler noise. BEFORE must show the herd actually fanned out.
|
||||
if after_cpu > one_cpu * 2.0 {
|
||||
eprintln!(
|
||||
"GATE FAIL: single-flight AFTER burned {:.1} verifications of CPU (expected ~1)",
|
||||
after_cpu / one_cpu
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if before_cpu < one_cpu * (herd as f64) * 0.6 {
|
||||
eprintln!(
|
||||
"GATE WARN: BEFORE emulation did not saturate ({:.1} verifs)",
|
||||
before_cpu / one_cpu
|
||||
);
|
||||
}
|
||||
println!("\nGATE PASS: cold-cache herd coalesced to ~1 Argon2id run");
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
//! CachedBlobBackend miss-stampede benchmark — duplicate remote fetches.
|
||||
//!
|
||||
//! K concurrent cold readers of ONE blob (a video player's parallel Range
|
||||
//! probes on an uncached file, N sync clients pulling the same new file)
|
||||
//! used to each download the FULL blob from the remote backend and race
|
||||
//! their writes on one shared deterministic `.tmp` path. The per-hash
|
||||
//! single-flight gate coalesces them onto one download; waiters serve the
|
||||
//! leader's cached file.
|
||||
//!
|
||||
//! The mock inner backend counts `get_blob_stream` calls and serves a
|
||||
//! 32 MiB blob with an injected 15 ms first-byte latency + paced chunks
|
||||
//! (models a remote object store).
|
||||
//!
|
||||
//! BEFORE (emulated) — K concurrent direct inner fetches, each draining
|
||||
//! the full stream (what the old miss path did)
|
||||
//! AFTER — K concurrent `CachedBlobBackend::get_blob_stream`
|
||||
//! on a cold cache
|
||||
//!
|
||||
//! Gates: AFTER's inner-fetch count == 1; the cached file must BLAKE3-match
|
||||
//! the source; K x full-drain wall reported for both.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_blob_cache
|
||||
//! Tunables: BENCH_CONCURRENCY (16), BENCH_BLOB_MB (32)
|
||||
|
||||
use std::env;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::Bytes;
|
||||
use futures::StreamExt;
|
||||
use oxicloud::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use oxicloud::domain::errors::DomainError;
|
||||
use oxicloud::infrastructure::services::cached_blob_backend::{BlobCacheConfig, CachedBlobBackend};
|
||||
|
||||
type BoxFut<'a, T> = std::pin::Pin<Box<dyn std::future::Future<Output = T> + Send + 'a>>;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Mock remote backend: one in-RAM blob, counted reads, and — crucially —
|
||||
/// SHARED aggregate bandwidth: concurrent streams split one simulated
|
||||
/// 1 GiB/s link (a real NIC/egress link doesn't hand every duplicate
|
||||
/// download its own private lane, so duplicate fetches cost real wall
|
||||
/// time, not just bytes).
|
||||
struct MockRemote {
|
||||
data: Bytes,
|
||||
fetches: AtomicU64,
|
||||
bytes_served: AtomicU64,
|
||||
/// Virtual time (µs since bench start) when the shared link frees up.
|
||||
link_busy_until_us: Arc<tokio::sync::Mutex<u64>>,
|
||||
epoch: Instant,
|
||||
}
|
||||
|
||||
const LINK_BYTES_PER_SEC: u64 = 1024 * 1024 * 1024; // 1 GiB/s aggregate
|
||||
|
||||
impl MockRemote {
|
||||
fn new(data: Bytes) -> Self {
|
||||
Self {
|
||||
data,
|
||||
fetches: AtomicU64::new(0),
|
||||
bytes_served: AtomicU64::new(0),
|
||||
link_busy_until_us: Arc::new(tokio::sync::Mutex::new(0)),
|
||||
epoch: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
fn stream(&self) -> BlobStream {
|
||||
self.fetches.fetch_add(1, Ordering::Relaxed);
|
||||
self.bytes_served
|
||||
.fetch_add(self.data.len() as u64, Ordering::Relaxed);
|
||||
let data = self.data.clone();
|
||||
let link = self.link_busy_until_us.clone();
|
||||
let epoch = self.epoch;
|
||||
let s = async_stream::stream! {
|
||||
// First-byte latency of a remote GET.
|
||||
tokio::time::sleep(Duration::from_millis(15)).await;
|
||||
let chunk = 4 * 1024 * 1024;
|
||||
let mut off = 0usize;
|
||||
while off < data.len() {
|
||||
let end = (off + chunk).min(data.len());
|
||||
// Reserve this chunk's slot on the shared link, then sleep
|
||||
// until the slot has elapsed — bandwidth divides across
|
||||
// every in-flight stream.
|
||||
let slot_us = (end - off) as u64 * 1_000_000 / LINK_BYTES_PER_SEC;
|
||||
let wake_us = {
|
||||
let mut busy = link.lock().await;
|
||||
let now_us = epoch.elapsed().as_micros() as u64;
|
||||
let start = (*busy).max(now_us);
|
||||
*busy = start + slot_us;
|
||||
*busy
|
||||
};
|
||||
let now_us = epoch.elapsed().as_micros() as u64;
|
||||
if wake_us > now_us {
|
||||
tokio::time::sleep(Duration::from_micros(wake_us - now_us)).await;
|
||||
}
|
||||
yield Ok::<Bytes, std::io::Error>(data.slice(off..end));
|
||||
off = end;
|
||||
}
|
||||
};
|
||||
Box::pin(s)
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for MockRemote {
|
||||
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
fn put_blob(&self, _hash: &str, _source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
Box::pin(async { Ok(0) })
|
||||
}
|
||||
fn put_blob_from_bytes(
|
||||
&self,
|
||||
_hash: &str,
|
||||
data: Bytes,
|
||||
) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
Box::pin(async move { Ok(data.len() as u64) })
|
||||
}
|
||||
fn get_blob_stream(&self, _hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let s = self.stream();
|
||||
Box::pin(async move { Ok(s) })
|
||||
}
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
_hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let data = self.data.clone();
|
||||
self.fetches.fetch_add(1, Ordering::Relaxed);
|
||||
Box::pin(async move {
|
||||
let end = end.unwrap_or(data.len() as u64).min(data.len() as u64);
|
||||
let s = futures::stream::once(async move {
|
||||
Ok::<Bytes, std::io::Error>(data.slice(start as usize..end as usize))
|
||||
});
|
||||
Ok(Box::pin(s) as BlobStream)
|
||||
})
|
||||
}
|
||||
fn delete_blob(&self, _hash: &str) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
fn blob_exists(&self, _hash: &str) -> BoxFut<'_, Result<bool, DomainError>> {
|
||||
Box::pin(async { Ok(true) })
|
||||
}
|
||||
fn blob_size(&self, _hash: &str) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
let n = self.data.len() as u64;
|
||||
Box::pin(async move { Ok(n) })
|
||||
}
|
||||
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>> {
|
||||
Box::pin(async {
|
||||
Ok(StorageHealthStatus {
|
||||
connected: true,
|
||||
backend_type: "mock".into(),
|
||||
message: "ok".into(),
|
||||
available_bytes: None,
|
||||
})
|
||||
})
|
||||
}
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"mock"
|
||||
}
|
||||
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
async fn drain(mut s: BlobStream) -> (u64, [u8; 32]) {
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut n = 0u64;
|
||||
while let Some(chunk) = s.next().await {
|
||||
let b = chunk.expect("chunk");
|
||||
n += b.len() as u64;
|
||||
hasher.update(&b);
|
||||
}
|
||||
(n, hasher.finalize().into())
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let k: usize = env_or("BENCH_CONCURRENCY", 16);
|
||||
let blob_mb: usize = env_or("BENCH_BLOB_MB", 32);
|
||||
|
||||
let data: Bytes = (0..blob_mb * 1024 * 1024)
|
||||
.map(|i| (i * 37 % 249) as u8)
|
||||
.collect::<Vec<u8>>()
|
||||
.into();
|
||||
let ref_hash: [u8; 32] = blake3::hash(&data).into();
|
||||
let blob_len = data.len() as u64;
|
||||
let hash = "benchblobcache00000000000000000000000000000000000000000000000000";
|
||||
|
||||
// ── BEFORE (emulated): K concurrent direct inner fetches ───────────
|
||||
let remote = Arc::new(MockRemote::new(data.clone()));
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..k {
|
||||
let r = remote.clone();
|
||||
set.spawn(async move {
|
||||
let s = r.get_blob_stream(hash).await.expect("stream");
|
||||
drain(s).await
|
||||
});
|
||||
}
|
||||
while let Some(res) = set.join_next().await {
|
||||
let (n, h) = res.expect("join");
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash);
|
||||
}
|
||||
let before_wall = t.elapsed().as_secs_f64() * 1000.0;
|
||||
let before_fetches = remote.fetches.load(Ordering::Relaxed);
|
||||
let before_mb = remote.bytes_served.load(Ordering::Relaxed) / (1024 * 1024);
|
||||
|
||||
// ── AFTER: K concurrent CachedBlobBackend reads, cold cache ────────
|
||||
let remote = Arc::new(MockRemote::new(data.clone()));
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let cached = Arc::new(CachedBlobBackend::new(
|
||||
remote.clone(),
|
||||
&BlobCacheConfig {
|
||||
cache_dir: dir.path().to_path_buf(),
|
||||
max_cache_bytes: 1 << 30,
|
||||
},
|
||||
));
|
||||
cached.initialize().await.expect("init");
|
||||
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..k {
|
||||
let c = cached.clone();
|
||||
set.spawn(async move {
|
||||
let s = c.get_blob_stream(hash).await.expect("stream");
|
||||
drain(s).await
|
||||
});
|
||||
}
|
||||
while let Some(res) = set.join_next().await {
|
||||
let (n, h) = res.expect("join");
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash, "cached read corrupted");
|
||||
}
|
||||
let after_wall = t.elapsed().as_secs_f64() * 1000.0;
|
||||
let after_fetches = remote.fetches.load(Ordering::Relaxed);
|
||||
let after_mb = remote.bytes_served.load(Ordering::Relaxed) / (1024 * 1024);
|
||||
|
||||
// Integrity of the durable cache file itself.
|
||||
let (n, h) = drain(cached.get_blob_stream(hash).await.expect("warm")).await;
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash, "durable cache file corrupted");
|
||||
let warm_fetches = remote.fetches.load(Ordering::Relaxed) - after_fetches;
|
||||
|
||||
println!("# {k} concurrent cold readers of one {blob_mb} MiB blob (remote: 15 ms TTFB, paced)");
|
||||
println!(
|
||||
"{:<24} {:>10} {:>14} {:>12}",
|
||||
"variant", "wall ms", "inner fetches", "remote MiB"
|
||||
);
|
||||
println!(
|
||||
"{:<24} {:>10.0} {:>14} {:>12}",
|
||||
"BEFORE (per-caller)", before_wall, before_fetches, before_mb
|
||||
);
|
||||
println!(
|
||||
"{:<24} {:>10.0} {:>14} {:>12}",
|
||||
"AFTER (single-flight)", after_wall, after_fetches, after_mb
|
||||
);
|
||||
|
||||
// ── Gates ───────────────────────────────────────────────────────────
|
||||
if after_fetches != 1 {
|
||||
eprintln!("GATE FAIL: expected exactly 1 coalesced remote fetch, got {after_fetches}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
if warm_fetches != 0 {
|
||||
eprintln!("GATE FAIL: warm read hit the remote backend");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!("\nGATE PASS: {before_fetches} remote fetches -> 1, cache file verified");
|
||||
}
|
||||
@@ -0,0 +1,531 @@
|
||||
//! CardDAV REPORT generation benchmark — dead double vCard generation +
|
||||
//! O(N²) uid scan (BEFORE) vs single on-demand generation (AFTER).
|
||||
//!
|
||||
//! The old `handle_report` flow pre-generated a vCard for EVERY contact into a
|
||||
//! `Vec<(uid, vcard)>`, then `generate_contacts_response` did a linear
|
||||
//! `find(|(uid, _)| *uid == contact.uid)` per contact — O(N²) string compares
|
||||
//! — and *discarded* the result (`let _ = vcard`), because
|
||||
//! `write_contact_response` regenerates the vCard on demand anyway. The fix
|
||||
//! deletes the pre-generation and the scan, and converts `contact_to_vcard`
|
||||
//! from `push_str(&format!(…))` (one temp String per line) to
|
||||
//! `write!(&mut String, …)`.
|
||||
//!
|
||||
//! `mod before` below is a verbatim copy of the OLD code (old
|
||||
//! `contact_to_vcard`, old `generate_contacts_response` with the `vcards`
|
||||
//! parameter, and the then-current `write_contact_response`), so one binary
|
||||
//! measures both variants and byte-compares their output.
|
||||
//!
|
||||
//! Equivalence gate: BEFORE and AFTER XML must be byte-identical for every
|
||||
//! (N, prop-set) combination, and the old/new `contact_to_vcard` must agree
|
||||
//! byte-for-byte on every synthetic contact. Any mismatch exits 1 with the
|
||||
//! first differing offset.
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_carddav_report
|
||||
//! Tunables (env):
|
||||
//! BENCH_REPS (5) median reported
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use chrono::{NaiveDate, TimeZone, Utc};
|
||||
use oxicloud::application::adapters::carddav_adapter::{
|
||||
CardDavAdapter, CardDavReportType, contact_to_vcard,
|
||||
};
|
||||
use oxicloud::application::adapters::webdav_adapter::QualifiedName;
|
||||
use oxicloud::application::dtos::contact_dto::{AddressDto, ContactDto, EmailDto, PhoneDto};
|
||||
|
||||
/// Verbatim copy of the pre-fix production code (handler + adapter side),
|
||||
/// kept here so the benchmark measures the real OLD flow, not a caricature.
|
||||
mod before {
|
||||
use std::io::Write;
|
||||
|
||||
use oxicloud::application::adapters::carddav_adapter::CardDavReportType;
|
||||
use oxicloud::application::adapters::webdav_adapter::QualifiedName;
|
||||
use oxicloud::application::dtos::contact_dto::ContactDto;
|
||||
use quick_xml::Writer;
|
||||
use quick_xml::events::{BytesEnd, BytesStart, BytesText, Event};
|
||||
|
||||
/// OLD `generate_contacts_response` — takes the pre-generated `vcards`,
|
||||
/// does the O(N²) linear uid scan per contact, then throws the hit away.
|
||||
pub fn generate_contacts_response<W: Write>(
|
||||
writer: W,
|
||||
contacts: &[ContactDto],
|
||||
vcards: &[(String, String)], // (uid, vcard_data)
|
||||
report: &CardDavReportType,
|
||||
base_href: &str,
|
||||
) -> std::io::Result<()> {
|
||||
let mut xml_writer = Writer::new(writer);
|
||||
|
||||
xml_writer.write_event(Event::Start(
|
||||
BytesStart::new("D:multistatus").with_attributes([
|
||||
("xmlns:D", "DAV:"),
|
||||
("xmlns:CR", "urn:ietf:params:xml:ns:carddav"),
|
||||
]),
|
||||
))?;
|
||||
|
||||
let props = match report {
|
||||
CardDavReportType::AddressbookQuery { props } => props.clone(),
|
||||
CardDavReportType::AddressbookMultiget { props, .. } => props.clone(),
|
||||
CardDavReportType::SyncCollection { props, .. } => props.clone(),
|
||||
};
|
||||
|
||||
for contact in contacts {
|
||||
let href = format!("{}{}.vcf", base_href, contact.uid);
|
||||
let vcard = vcards
|
||||
.iter()
|
||||
.find(|(uid, _)| *uid == contact.uid)
|
||||
.map(|(_, data)| data.as_str())
|
||||
.unwrap_or("");
|
||||
write_contact_response(&mut xml_writer, contact, &props, &href)?;
|
||||
// If address-data is requested, include vcard
|
||||
if props.iter().any(|p| p.name == "address-data") || props.is_empty() {
|
||||
// Already handled in write_contact_response
|
||||
}
|
||||
let _ = vcard; // suppress warning - used via contact_to_vcard fallback
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:multistatus")))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy of the (unchanged) private `write_contact_response`, wired to the
|
||||
/// OLD `contact_to_vcard` so the BEFORE variant is fully self-contained.
|
||||
fn write_contact_response<W: Write>(
|
||||
xml_writer: &mut Writer<W>,
|
||||
contact: &ContactDto,
|
||||
props: &[QualifiedName],
|
||||
href: &str,
|
||||
) -> std::io::Result<()> {
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:response")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:href")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(href)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:href")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:propstat")))?;
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:prop")))?;
|
||||
|
||||
if props.is_empty() {
|
||||
// Return standard properties
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new("D:resourcetype")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getetag")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&format!(
|
||||
"\"{}\"",
|
||||
contact.etag
|
||||
))))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getetag")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getcontenttype")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new("text/vcard; charset=utf-8")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getcontenttype")))?;
|
||||
|
||||
// Include vCard data
|
||||
let vcard = contact_to_vcard(contact);
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("CR:address-data")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&vcard)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("CR:address-data")))?;
|
||||
} else {
|
||||
for prop in props {
|
||||
match (prop.namespace.as_str(), prop.name.as_str()) {
|
||||
("DAV:", "resourcetype") => {
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new("D:resourcetype")))?;
|
||||
}
|
||||
("DAV:", "getetag") => {
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getetag")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&format!(
|
||||
"\"{}\"",
|
||||
contact.etag
|
||||
))))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getetag")))?;
|
||||
}
|
||||
("DAV:", "getcontenttype") => {
|
||||
xml_writer
|
||||
.write_event(Event::Start(BytesStart::new("D:getcontenttype")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(
|
||||
"text/vcard; charset=utf-8",
|
||||
)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getcontenttype")))?;
|
||||
}
|
||||
("DAV:", "getlastmodified") => {
|
||||
xml_writer
|
||||
.write_event(Event::Start(BytesStart::new("D:getlastmodified")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(
|
||||
&contact.updated_at.to_rfc2822(),
|
||||
)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getlastmodified")))?;
|
||||
}
|
||||
("urn:ietf:params:xml:ns:carddav", "address-data") => {
|
||||
let vcard = contact_to_vcard(contact);
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("CR:address-data")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&vcard)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("CR:address-data")))?;
|
||||
}
|
||||
_ => {
|
||||
let prop_name = if prop.namespace == "urn:ietf:params:xml:ns:carddav" {
|
||||
format!("CR:{}", prop.name)
|
||||
} else if prop.namespace == "DAV:" {
|
||||
format!("D:{}", prop.name)
|
||||
} else {
|
||||
prop.name.clone()
|
||||
};
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new(&prop_name)))?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:prop")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:status")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new("HTTP/1.1 200 OK")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:status")))?;
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:propstat")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:response")))?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// OLD `contact_to_vcard` — one `push_str(&format!(…))` temp String per line.
|
||||
pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
|
||||
|
||||
vcard.push_str(&format!("UID:{}\r\n", contact.uid));
|
||||
|
||||
if let (Some(last), Some(first)) = (&contact.last_name, &contact.first_name) {
|
||||
vcard.push_str(&format!("N:{};{};;;\r\n", last, first));
|
||||
} else if let Some(last) = &contact.last_name {
|
||||
vcard.push_str(&format!("N:{};;;;\r\n", last));
|
||||
} else if let Some(first) = &contact.first_name {
|
||||
vcard.push_str(&format!("N:;{};;;\r\n", first));
|
||||
}
|
||||
|
||||
if let Some(fn_name) = &contact.full_name {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
} else {
|
||||
// FN is mandatory in vCard 3.0
|
||||
let fn_name = format!(
|
||||
"{} {}",
|
||||
contact.first_name.as_deref().unwrap_or(""),
|
||||
contact.last_name.as_deref().unwrap_or(""),
|
||||
)
|
||||
.trim()
|
||||
.to_string();
|
||||
if !fn_name.is_empty() {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
} else {
|
||||
vcard.push_str("FN:Unknown\r\n");
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(nickname) = &contact.nickname {
|
||||
vcard.push_str(&format!("NICKNAME:{}\r\n", nickname));
|
||||
}
|
||||
|
||||
for email in &contact.email {
|
||||
vcard.push_str(&format!(
|
||||
"EMAIL;TYPE={}:{}\r\n",
|
||||
email.r#type.to_uppercase(),
|
||||
email.email
|
||||
));
|
||||
}
|
||||
|
||||
for phone in &contact.phone {
|
||||
vcard.push_str(&format!(
|
||||
"TEL;TYPE={}:{}\r\n",
|
||||
phone.r#type.to_uppercase(),
|
||||
phone.number
|
||||
));
|
||||
}
|
||||
|
||||
for addr in &contact.address {
|
||||
let adr = format!(
|
||||
";;{};{};{};{};{}",
|
||||
addr.street.as_deref().unwrap_or(""),
|
||||
addr.city.as_deref().unwrap_or(""),
|
||||
addr.state.as_deref().unwrap_or(""),
|
||||
addr.postal_code.as_deref().unwrap_or(""),
|
||||
addr.country.as_deref().unwrap_or(""),
|
||||
);
|
||||
vcard.push_str(&format!(
|
||||
"ADR;TYPE={}:{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
adr
|
||||
));
|
||||
}
|
||||
|
||||
if let Some(org) = &contact.organization {
|
||||
vcard.push_str(&format!("ORG:{}\r\n", org));
|
||||
}
|
||||
if let Some(title) = &contact.title {
|
||||
vcard.push_str(&format!("TITLE:{}\r\n", title));
|
||||
}
|
||||
if let Some(notes) = &contact.notes {
|
||||
vcard.push_str(&format!("NOTE:{}\r\n", notes.replace('\n', "\\n")));
|
||||
}
|
||||
if let Some(bday) = &contact.birthday {
|
||||
vcard.push_str(&format!("BDAY:{}\r\n", bday.format("%Y-%m-%d")));
|
||||
}
|
||||
if let Some(photo) = &contact.photo_url {
|
||||
vcard.push_str(&format!("PHOTO;VALUE=URI:{}\r\n", photo));
|
||||
}
|
||||
|
||||
vcard.push_str(&format!(
|
||||
"REV:{}\r\n",
|
||||
contact.updated_at.format("%Y%m%dT%H%M%SZ")
|
||||
));
|
||||
vcard.push_str("END:VCARD\r\n");
|
||||
|
||||
vcard
|
||||
}
|
||||
}
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Deterministic synthetic address book: every contact has 2 emails, 1 phone
|
||||
/// and 1 address; optional fields (nickname, notes-with-newline, birthday,
|
||||
/// photo, missing names → FN fallback) are cycled so the byte-equality gate
|
||||
/// exercises every `contact_to_vcard` branch, not just the happy path.
|
||||
fn make_contacts(n: usize) -> Vec<ContactDto> {
|
||||
let created = Utc.with_ymd_and_hms(2026, 1, 15, 9, 0, 0).unwrap();
|
||||
let updated = Utc.with_ymd_and_hms(2026, 6, 30, 18, 45, 12).unwrap();
|
||||
|
||||
(0..n)
|
||||
.map(|i| {
|
||||
let (full_name, first_name, last_name) = match i % 5 {
|
||||
0 => (
|
||||
Some(format!("Contact {i:05} Example")),
|
||||
Some(format!("Contact{i:05}")),
|
||||
Some("Example".to_string()),
|
||||
),
|
||||
1 => (
|
||||
None,
|
||||
Some(format!("Contact{i:05}")),
|
||||
Some("Example".to_string()),
|
||||
),
|
||||
2 => (None, None, Some("Example".to_string())),
|
||||
3 => (None, Some(format!("Contact{i:05}")), None),
|
||||
_ => (None, None, None), // FN:Unknown fallback
|
||||
};
|
||||
ContactDto {
|
||||
id: format!("id-{i:05}"),
|
||||
address_book_id: "bench-book".to_string(),
|
||||
uid: format!("bench-contact-{i:05}@oxicloud"),
|
||||
full_name,
|
||||
first_name,
|
||||
last_name,
|
||||
nickname: (i % 7 == 0).then(|| format!("nick{i}")),
|
||||
email: vec![
|
||||
EmailDto {
|
||||
email: format!("contact{i:05}@example.com"),
|
||||
r#type: "work".to_string(),
|
||||
is_primary: true,
|
||||
},
|
||||
EmailDto {
|
||||
email: format!("contact{i:05}@home.example.org"),
|
||||
r#type: "home".to_string(),
|
||||
is_primary: false,
|
||||
},
|
||||
],
|
||||
phone: vec![PhoneDto {
|
||||
number: format!("+1-555-{:04}", i % 10_000),
|
||||
r#type: "cell".to_string(),
|
||||
is_primary: true,
|
||||
}],
|
||||
address: vec![AddressDto {
|
||||
street: Some(format!("{} Main Street", i + 1)),
|
||||
city: Some("Springfield".to_string()),
|
||||
state: Some("IL".to_string()),
|
||||
postal_code: Some(format!("{:05}", 60_000 + (i % 1_000))),
|
||||
country: Some("USA".to_string()),
|
||||
r#type: "home".to_string(),
|
||||
is_primary: true,
|
||||
}],
|
||||
organization: Some("OxiCloud Benchmarks Inc.".to_string()),
|
||||
title: Some("Engineer".to_string()),
|
||||
notes: (i % 11 == 0).then(|| "line one\nline two & <specials>".to_string()),
|
||||
photo_url: (i % 13 == 0).then(|| format!("https://example.com/avatars/{i}.jpg")),
|
||||
birthday: (i % 3 == 0).then(|| NaiveDate::from_ymd_opt(1990, 5, 17).unwrap()),
|
||||
anniversary: None,
|
||||
created_at: created,
|
||||
updated_at: updated,
|
||||
etag: format!("etag-{i:05}"),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn dav(name: &str) -> QualifiedName {
|
||||
QualifiedName {
|
||||
namespace: "DAV:".to_string(),
|
||||
name: name.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn carddav(name: &str) -> QualifiedName {
|
||||
QualifiedName {
|
||||
namespace: "urn:ietf:params:xml:ns:carddav".to_string(),
|
||||
name: name.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// OLD handler flow: pre-generate a vCard per contact, then generate the XML
|
||||
/// (which re-generates every vCard on demand and never reads the pre-made ones).
|
||||
fn run_before(contacts: &[ContactDto], report: &CardDavReportType, base_href: &str) -> Vec<u8> {
|
||||
// Generate vCards (verbatim old handle_report pre-generation)
|
||||
let vcards: Vec<(String, String)> = contacts
|
||||
.iter()
|
||||
.map(|c| (c.uid.clone(), before::contact_to_vcard(c)))
|
||||
.collect();
|
||||
|
||||
let mut out = Vec::new();
|
||||
before::generate_contacts_response(&mut out, contacts, &vcards, report, base_href)
|
||||
.expect("BEFORE XML generation failed");
|
||||
out
|
||||
}
|
||||
|
||||
/// NEW production path.
|
||||
fn run_after(contacts: &[ContactDto], report: &CardDavReportType, base_href: &str) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
CardDavAdapter::generate_contacts_response(&mut out, contacts, report, base_href)
|
||||
.expect("AFTER XML generation failed");
|
||||
out
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
fn first_diff(a: &[u8], b: &[u8]) -> Option<usize> {
|
||||
if a == b {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
a.iter()
|
||||
.zip(b.iter())
|
||||
.position(|(x, y)| x != y)
|
||||
.unwrap_or_else(|| a.len().min(b.len())),
|
||||
)
|
||||
}
|
||||
|
||||
fn context_snippet(bytes: &[u8], at: usize) -> String {
|
||||
let start = at.saturating_sub(40);
|
||||
let end = (at + 40).min(bytes.len());
|
||||
String::from_utf8_lossy(&bytes[start..end]).into_owned()
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
let base_href = "/carddav/bench-book/";
|
||||
|
||||
let prop_sets: Vec<(&str, Vec<QualifiedName>)> = vec![
|
||||
("getetag", vec![dav("getetag")]),
|
||||
(
|
||||
"getetag + address-data",
|
||||
vec![dav("getetag"), carddav("address-data")],
|
||||
),
|
||||
// Not part of the timing table, but gated too: the empty-props
|
||||
// default path also embeds address-data.
|
||||
("(empty = allprop default)", vec![]),
|
||||
];
|
||||
let sizes = [500usize, 5_000];
|
||||
|
||||
// ── Equivalence gate ────────────────────────────────────────────────
|
||||
let gate_contacts = make_contacts(*sizes.iter().max().unwrap());
|
||||
for c in &gate_contacts {
|
||||
let old = before::contact_to_vcard(c);
|
||||
let new = contact_to_vcard(c);
|
||||
if old != new {
|
||||
let at = first_diff(old.as_bytes(), new.as_bytes()).unwrap();
|
||||
eprintln!(
|
||||
"EQUIVALENCE FAILURE: contact_to_vcard differs for uid={} at byte {}\n old: …{}…\n new: …{}…",
|
||||
c.uid,
|
||||
at,
|
||||
context_snippet(old.as_bytes(), at),
|
||||
context_snippet(new.as_bytes(), at),
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
for &n in &sizes {
|
||||
let contacts = &gate_contacts[..n];
|
||||
for (label, props) in &prop_sets {
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: props.clone(),
|
||||
};
|
||||
let old_xml = run_before(contacts, &report, base_href);
|
||||
let new_xml = run_after(contacts, &report, base_href);
|
||||
if let Some(at) = first_diff(&old_xml, &new_xml) {
|
||||
eprintln!(
|
||||
"EQUIVALENCE FAILURE: REPORT XML differs (N={}, props={}) at byte {} (before {} B, after {} B)\n before: …{}…\n after: …{}…",
|
||||
n,
|
||||
label,
|
||||
at,
|
||||
old_xml.len(),
|
||||
new_xml.len(),
|
||||
context_snippet(&old_xml, at),
|
||||
context_snippet(&new_xml, at),
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"equivalence gate: BEFORE == AFTER byte-identical for all prop sets at N = {:?} (and all {} vCards match)\n",
|
||||
sizes,
|
||||
gate_contacts.len()
|
||||
);
|
||||
|
||||
// ── Timing ──────────────────────────────────────────────────────────
|
||||
println!("| N | props | BEFORE ms | AFTER ms | speedup |");
|
||||
println!("|------:|------------------------|----------:|---------:|--------:|");
|
||||
for &n in &sizes {
|
||||
let contacts = &gate_contacts[..n];
|
||||
for (label, props) in prop_sets.iter().take(2) {
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: props.clone(),
|
||||
};
|
||||
|
||||
// Warm-up (allocator, caches) — result discarded.
|
||||
let _ = run_before(contacts, &report, base_href);
|
||||
let _ = run_after(contacts, &report, base_href);
|
||||
|
||||
let mut before_ms = Vec::with_capacity(reps);
|
||||
let mut after_ms = Vec::with_capacity(reps);
|
||||
for _ in 0..reps {
|
||||
let t0 = Instant::now();
|
||||
let out = run_before(contacts, &report, base_href);
|
||||
before_ms.push(t0.elapsed().as_secs_f64() * 1_000.0);
|
||||
std::hint::black_box(&out);
|
||||
|
||||
let t1 = Instant::now();
|
||||
let out = run_after(contacts, &report, base_href);
|
||||
after_ms.push(t1.elapsed().as_secs_f64() * 1_000.0);
|
||||
std::hint::black_box(&out);
|
||||
}
|
||||
let b = median(before_ms);
|
||||
let a = median(after_ms);
|
||||
println!(
|
||||
"| {:>5} | {:<22} | {:>9.3} | {:>8.3} | {:>6.2}x |",
|
||||
n,
|
||||
label,
|
||||
b,
|
||||
a,
|
||||
b / a
|
||||
);
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"\n(median of {} reps; BEFORE includes the old handler's vCard pre-generation loop,",
|
||||
reps
|
||||
);
|
||||
println!(" which the old code then discarded — the O(N²) uid scan dominates at large N)");
|
||||
}
|
||||
@@ -0,0 +1,589 @@
|
||||
//! File/Folder entity → DTO mapping benchmark — per-row allocation churn.
|
||||
//!
|
||||
//! Isolates the variables the DTO-mapping change touches:
|
||||
//!
|
||||
//! • `Arc::<str>::from(&'static str)` for the closed-set display fields
|
||||
//! (icon class, icon special class, category) — always alloc + copy —
|
||||
//! vs interned `Arc<str>` lookups (`intern_display` / `intern_mime`).
|
||||
//! • `File::compute_etag` / `Folder::compute_etag` — `chars().take(16)
|
||||
//! .collect::<String>()` + `format!` (2 allocs) vs one sized buffer.
|
||||
//! • `format_file_size` — two `format!` calls per row vs one buffer.
|
||||
//! • `Folder → FolderDto` — per-getter `.to_string()` clones + a
|
||||
//! double-allocated etag vs `into_parts()` moves.
|
||||
//!
|
||||
//! The OLD mapping logic is copied verbatim into `mod before` so one binary
|
||||
//! reports BEFORE vs AFTER side by side, and an equivalence gate asserts the
|
||||
//! two produce byte-identical DTOs for every row (exit 1 on any diff).
|
||||
//!
|
||||
//! Sections:
|
||||
//! 1. File → FileDto wall time (p50 ns/row over BENCH_PASSES passes)
|
||||
//! 2. Folder → FolderDto wall time (same)
|
||||
//! 3. Alloc calls/row (counting global allocator wrapping System — the
|
||||
//! lib crate sets no global allocator; mimalloc lives in main.rs only,
|
||||
//! which examples do not link)
|
||||
//! 4. Equivalence gate: BEFORE output == AFTER output, field by field
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_dto_map
|
||||
//! Tunables (env):
|
||||
//! BENCH_ROWS (10000) BENCH_PASSES (100)
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::env;
|
||||
use std::hint::black_box;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use oxicloud::application::dtos::file_dto::FileDto;
|
||||
use oxicloud::application::dtos::folder_dto::FolderDto;
|
||||
use oxicloud::domain::entities::file::File;
|
||||
use oxicloud::domain::entities::folder::Folder;
|
||||
use oxicloud::domain::services::path_service::StoragePath;
|
||||
use uuid::Uuid;
|
||||
|
||||
// ─── Counting allocator (Section 3) ─────────────────────────────────────────
|
||||
|
||||
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
struct CountingAlloc;
|
||||
|
||||
unsafe impl GlobalAlloc for CountingAlloc {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc(layout) }
|
||||
}
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) }
|
||||
}
|
||||
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.realloc(ptr, layout, new_size) }
|
||||
}
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc_zeroed(layout) }
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static GLOBAL: CountingAlloc = CountingAlloc;
|
||||
|
||||
// ─── BEFORE: verbatim copy of the pre-optimization mapping logic ────────────
|
||||
|
||||
/// Pre-optimization reference implementation. Copied verbatim from the old
|
||||
/// `From<File> for FileDto` / `From<Folder> for FolderDto` bodies, the old
|
||||
/// `File::compute_etag` / `Folder::compute_etag` formulas and the old
|
||||
/// `format_file_size` — kept byte-for-byte in behaviour so the equivalence
|
||||
/// gate proves the optimized paths change nothing observable.
|
||||
#[allow(clippy::all)]
|
||||
mod before {
|
||||
use std::sync::Arc;
|
||||
|
||||
use oxicloud::application::dtos::display_helpers::{
|
||||
category_for, icon_class_for, icon_special_class_for,
|
||||
};
|
||||
use oxicloud::application::dtos::file_dto::FileDto;
|
||||
use oxicloud::application::dtos::folder_dto::FolderDto;
|
||||
use oxicloud::domain::entities::file::File;
|
||||
use oxicloud::domain::entities::folder::Folder;
|
||||
|
||||
/// Old `File::compute_etag`: intermediate `collect::<String>()` +
|
||||
/// `format!` — 2 allocations for one ~21-char string.
|
||||
fn file_compute_etag(blob_hash: &str, modified_at: u64) -> String {
|
||||
let prefix: String = blob_hash.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, modified_at)
|
||||
}
|
||||
|
||||
/// Old `Folder::compute_etag` (same shape as the file formula).
|
||||
fn folder_compute_etag(id: &str, tree_modified_at: u64) -> String {
|
||||
let prefix: String = id.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, tree_modified_at)
|
||||
}
|
||||
|
||||
/// Old `format_file_size`: two `format!` calls per row.
|
||||
fn format_file_size(bytes: u64) -> String {
|
||||
if bytes == 0 {
|
||||
return "0 Bytes".to_string();
|
||||
}
|
||||
|
||||
const K: f64 = 1024.0;
|
||||
const SIZES: [&str; 5] = ["Bytes", "KB", "MB", "GB", "TB"];
|
||||
|
||||
let i = ((bytes as f64).ln() / K.ln()).floor() as usize;
|
||||
let i = i.min(SIZES.len() - 1);
|
||||
|
||||
let value = bytes as f64 / K.powi(i as i32);
|
||||
|
||||
let formatted = format!("{:.2}", value);
|
||||
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
|
||||
|
||||
format!("{} {}", formatted, SIZES[i])
|
||||
}
|
||||
|
||||
/// Old `From<File> for FileDto` body: `Arc::from(&str)` for the three
|
||||
/// display fields and the mime type (alloc + copy each), 2-alloc etag,
|
||||
/// 2-format size string.
|
||||
pub fn file_to_dto(file: File) -> FileDto {
|
||||
let etag = file_compute_etag(file.content_hash(), file.modified_at());
|
||||
let content_hash = file.content_hash().to_string();
|
||||
|
||||
let parts = file.into_parts();
|
||||
|
||||
let icon_class: Arc<str> = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class: Arc<str> =
|
||||
Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category: Arc<str> = Arc::from(category_for(&parts.name, &parts.mime_type));
|
||||
let size_formatted = format_file_size(parts.size);
|
||||
let mime_type: Arc<str> = Arc::from(parts.mime_type.as_str());
|
||||
|
||||
FileDto {
|
||||
id: parts.id,
|
||||
name: parts.name,
|
||||
path: parts.path_string,
|
||||
size: parts.size,
|
||||
mime_type,
|
||||
folder_id: parts.folder_id,
|
||||
created_at: parts.created_at,
|
||||
modified_at: parts.modified_at,
|
||||
icon_class,
|
||||
icon_special_class,
|
||||
category,
|
||||
size_formatted,
|
||||
sort_date: None,
|
||||
content_hash,
|
||||
etag,
|
||||
created_by: parts.created_by,
|
||||
updated_by: parts.updated_by,
|
||||
}
|
||||
}
|
||||
|
||||
/// Old `From<Folder> for FolderDto` body: per-getter `.to_string()`
|
||||
/// clones, `folder.etag().to_string()` (etag built then cloned — the
|
||||
/// verbatim double alloc) and 3 fresh `Arc::from` constants per row.
|
||||
pub fn folder_to_dto(folder: Folder) -> FolderDto {
|
||||
let is_root = folder.parent_id().is_none();
|
||||
let etag = folder_compute_etag(folder.id(), folder.tree_modified_at()).to_string();
|
||||
|
||||
FolderDto {
|
||||
id: folder.id().to_string(),
|
||||
name: folder.name().to_string(),
|
||||
path: folder.path_string().to_string(),
|
||||
parent_id: folder.parent_id().map(String::from),
|
||||
drive_id: folder.drive_id(),
|
||||
created_at: folder.created_at(),
|
||||
modified_at: folder.modified_at(),
|
||||
is_root,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
etag,
|
||||
created_by: folder.created_by(),
|
||||
updated_by: folder.updated_by(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Synthetic corpus ────────────────────────────────────────────────────────
|
||||
|
||||
/// (extension, mime) matrix: interned common types, generic MIMEs that
|
||||
/// exercise the extension fallback, and exotic MIMEs that miss the intern
|
||||
/// table so the fallback `Arc::from` path is measured too.
|
||||
const KINDS: &[(&str, &str)] = &[
|
||||
("jpg", "image/jpeg"),
|
||||
("png", "image/png"),
|
||||
("heic", "image/heic"),
|
||||
("mp4", "video/mp4"),
|
||||
("mov", "video/quicktime"),
|
||||
("mp3", "audio/mpeg"),
|
||||
("flac", "audio/flac"),
|
||||
("pdf", "application/pdf"),
|
||||
(
|
||||
"docx",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
),
|
||||
(
|
||||
"xlsx",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
),
|
||||
("txt", "text/plain"),
|
||||
("md", "text/markdown"),
|
||||
("csv", "text/csv"),
|
||||
("json", "application/json"),
|
||||
("zip", "application/zip"),
|
||||
("gz", "application/gzip"),
|
||||
// Extension fallback: generic MIME, type resolved from the name.
|
||||
("rs", "application/octet-stream"),
|
||||
("py", "application/octet-stream"),
|
||||
("svelte", "application/octet-stream"),
|
||||
("dmg", "application/octet-stream"),
|
||||
("bin", "application/octet-stream"),
|
||||
// No extension + empty MIME: full-default path.
|
||||
("", ""),
|
||||
// Exotic MIMEs: miss the intern table, fall back to Arc::from.
|
||||
("pdb", "chemical/x-pdb"),
|
||||
("xyz", "application/x-very-exotic-subtype+custom"),
|
||||
];
|
||||
|
||||
const SIZES: &[u64] = &[
|
||||
0,
|
||||
137,
|
||||
500,
|
||||
1_024,
|
||||
1_536,
|
||||
65_536,
|
||||
1_048_576,
|
||||
3_423_744,
|
||||
987_654_321,
|
||||
1_073_741_824,
|
||||
5_497_558_138_880, // ~5 TB
|
||||
];
|
||||
|
||||
/// Deterministic xorshift64* — fake-but-plausible 64-char lowercase hex
|
||||
/// BLAKE3 hashes.
|
||||
fn next_seed(seed: &mut u64) -> u64 {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
seed.wrapping_mul(0x2545F4914F6CDD1D)
|
||||
}
|
||||
|
||||
fn fake_blake3(seed: &mut u64) -> String {
|
||||
format!(
|
||||
"{:016x}{:016x}{:016x}{:016x}",
|
||||
next_seed(seed),
|
||||
next_seed(seed),
|
||||
next_seed(seed),
|
||||
next_seed(seed)
|
||||
)
|
||||
}
|
||||
|
||||
fn build_files(rows: usize) -> Vec<File> {
|
||||
let mut seed = 0x9E3779B97F4A7C15u64;
|
||||
(0..rows)
|
||||
.map(|i| {
|
||||
let (ext, mime) = KINDS[i % KINDS.len()];
|
||||
let name = if ext.is_empty() {
|
||||
format!("file_{i:05}")
|
||||
} else {
|
||||
format!("file_{i:05}.{ext}")
|
||||
};
|
||||
let path = StoragePath::from_string(&format!("/bench/dir_{}/{}", i % 37, name));
|
||||
let folder_id = if i % 3 == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Uuid::from_u128(1000 + (i % 37) as u128).to_string())
|
||||
};
|
||||
let created_by = (i % 2 == 0).then(|| Uuid::from_u128(7 + (i % 5) as u128));
|
||||
let updated_by = (i % 4 == 0).then(|| Uuid::from_u128(11 + (i % 3) as u128));
|
||||
File::with_timestamps_blob_hash_and_provenance(
|
||||
Uuid::from_u128(i as u128).to_string(),
|
||||
name,
|
||||
path,
|
||||
SIZES[i % SIZES.len()],
|
||||
mime.to_string(),
|
||||
folder_id,
|
||||
1_600_000_000 + i as u64,
|
||||
1_700_000_000 + (i as u64 * 7) % 100_000,
|
||||
fake_blake3(&mut seed),
|
||||
created_by,
|
||||
updated_by,
|
||||
)
|
||||
.expect("valid synthetic file")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn build_folders(rows: usize) -> Vec<Folder> {
|
||||
(0..rows)
|
||||
.map(|i| {
|
||||
let name = format!("folder_{i:05}");
|
||||
let path = StoragePath::from_string(&format!("/bench/parent_{}/{}", i % 37, name));
|
||||
let parent_id = if i % 5 == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Uuid::from_u128(2000 + (i % 37) as u128).to_string())
|
||||
};
|
||||
let created_by = (i % 2 == 0).then(|| Uuid::from_u128(7 + (i % 5) as u128));
|
||||
let updated_by = (i % 4 == 0).then(|| Uuid::from_u128(11 + (i % 3) as u128));
|
||||
Folder::with_timestamps_tree_and_provenance(
|
||||
Uuid::from_u128(500_000 + i as u128).to_string(),
|
||||
name,
|
||||
path,
|
||||
parent_id,
|
||||
Uuid::from_u128(42 + (i % 4) as u128),
|
||||
1_600_000_000 + i as u64,
|
||||
1_700_000_000 + (i as u64 * 7) % 100_000,
|
||||
1_700_000_000 + (i as u64 * 11) % 100_000,
|
||||
created_by,
|
||||
updated_by,
|
||||
)
|
||||
.expect("valid synthetic folder")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
// ─── Measurement helpers ─────────────────────────────────────────────────────
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// p50 wall seconds per pass of `f` over `passes` passes.
|
||||
fn p50_pass_secs(passes: usize, mut f: impl FnMut()) -> f64 {
|
||||
f(); // warmup (also initializes LazyLock intern tables)
|
||||
let mut xs = Vec::with_capacity(passes);
|
||||
for _ in 0..passes {
|
||||
let t0 = Instant::now();
|
||||
f();
|
||||
xs.push(t0.elapsed().as_secs_f64());
|
||||
}
|
||||
median(xs)
|
||||
}
|
||||
|
||||
/// Allocation calls performed by one run of `f` (deterministic — the
|
||||
/// mappings do no I/O and touch no shared caches beyond the intern tables,
|
||||
/// which the warmup run already initialized).
|
||||
fn allocs_of(mut f: impl FnMut()) -> u64 {
|
||||
f(); // warmup so one-time lazy init isn't attributed to the variant
|
||||
let start = ALLOC_CALLS.load(Ordering::Relaxed);
|
||||
f();
|
||||
ALLOC_CALLS.load(Ordering::Relaxed) - start
|
||||
}
|
||||
|
||||
struct Row {
|
||||
variant: &'static str,
|
||||
ns_per_row: f64,
|
||||
allocs_per_row: f64,
|
||||
}
|
||||
|
||||
// ─── Equivalence gate (Section 4) ────────────────────────────────────────────
|
||||
|
||||
macro_rules! cmp_field {
|
||||
($diffs:expr, $i:expr, $kind:expr, $b:expr, $a:expr, $field:ident) => {
|
||||
if $b.$field != $a.$field {
|
||||
$diffs += 1;
|
||||
if $diffs <= 20 {
|
||||
println!(
|
||||
" DIFF {} row {}: {} BEFORE={:?} AFTER={:?}",
|
||||
$kind,
|
||||
$i,
|
||||
stringify!($field),
|
||||
$b.$field,
|
||||
$a.$field
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
fn diff_file(i: usize, b: &FileDto, a: &FileDto, diffs: &mut u64) {
|
||||
cmp_field!(*diffs, i, "file", b, a, id);
|
||||
cmp_field!(*diffs, i, "file", b, a, name);
|
||||
cmp_field!(*diffs, i, "file", b, a, path);
|
||||
cmp_field!(*diffs, i, "file", b, a, size);
|
||||
cmp_field!(*diffs, i, "file", b, a, mime_type);
|
||||
cmp_field!(*diffs, i, "file", b, a, folder_id);
|
||||
cmp_field!(*diffs, i, "file", b, a, created_at);
|
||||
cmp_field!(*diffs, i, "file", b, a, modified_at);
|
||||
cmp_field!(*diffs, i, "file", b, a, icon_class);
|
||||
cmp_field!(*diffs, i, "file", b, a, icon_special_class);
|
||||
cmp_field!(*diffs, i, "file", b, a, category);
|
||||
cmp_field!(*diffs, i, "file", b, a, size_formatted);
|
||||
cmp_field!(*diffs, i, "file", b, a, sort_date);
|
||||
cmp_field!(*diffs, i, "file", b, a, content_hash);
|
||||
cmp_field!(*diffs, i, "file", b, a, etag);
|
||||
cmp_field!(*diffs, i, "file", b, a, created_by);
|
||||
cmp_field!(*diffs, i, "file", b, a, updated_by);
|
||||
}
|
||||
|
||||
fn diff_folder(i: usize, b: &FolderDto, a: &FolderDto, diffs: &mut u64) {
|
||||
cmp_field!(*diffs, i, "folder", b, a, id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, name);
|
||||
cmp_field!(*diffs, i, "folder", b, a, path);
|
||||
cmp_field!(*diffs, i, "folder", b, a, parent_id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, drive_id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, created_at);
|
||||
cmp_field!(*diffs, i, "folder", b, a, modified_at);
|
||||
cmp_field!(*diffs, i, "folder", b, a, is_root);
|
||||
cmp_field!(*diffs, i, "folder", b, a, icon_class);
|
||||
cmp_field!(*diffs, i, "folder", b, a, icon_special_class);
|
||||
cmp_field!(*diffs, i, "folder", b, a, category);
|
||||
cmp_field!(*diffs, i, "folder", b, a, etag);
|
||||
cmp_field!(*diffs, i, "folder", b, a, created_by);
|
||||
cmp_field!(*diffs, i, "folder", b, a, updated_by);
|
||||
}
|
||||
|
||||
// ─── Main ────────────────────────────────────────────────────────────────────
|
||||
|
||||
fn main() {
|
||||
let rows: usize = env_or("BENCH_ROWS", 10_000).max(1);
|
||||
let passes: usize = env_or("BENCH_PASSES", 100).max(1);
|
||||
|
||||
let files = build_files(rows);
|
||||
let folders = build_folders(rows);
|
||||
println!(
|
||||
"corpus: {rows} files ({} kinds x {} sizes) + {rows} folders, {passes} timed passes",
|
||||
KINDS.len(),
|
||||
SIZES.len()
|
||||
);
|
||||
println!(
|
||||
"note: each measured pass pays one entity clone per row (mapping consumes the\n\
|
||||
entity); the clone-only baseline is measured separately and subtracted.\n"
|
||||
);
|
||||
|
||||
// ── Section 1: File → FileDto wall time ─────────────────────────────
|
||||
println!("── Section 1: File → FileDto (p50 wall, net of clone) ──");
|
||||
let file_base_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(f.clone());
|
||||
}
|
||||
});
|
||||
let file_before_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(before::file_to_dto(f.clone()));
|
||||
}
|
||||
});
|
||||
let file_after_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(FileDto::from(f.clone()));
|
||||
}
|
||||
});
|
||||
let file_base_ns = file_base_s * 1e9 / rows as f64;
|
||||
let file_before_ns = (file_before_s - file_base_s) * 1e9 / rows as f64;
|
||||
let file_after_ns = (file_after_s - file_base_s) * 1e9 / rows as f64;
|
||||
println!(" clone-only baseline: {file_base_ns:8.1} ns/row");
|
||||
println!(" BEFORE mapping: {file_before_ns:8.1} ns/row");
|
||||
println!(" AFTER mapping: {file_after_ns:8.1} ns/row\n");
|
||||
|
||||
// ── Section 2: Folder → FolderDto wall time ─────────────────────────
|
||||
println!("── Section 2: Folder → FolderDto (p50 wall, net of clone) ──");
|
||||
let folder_base_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(f.clone());
|
||||
}
|
||||
});
|
||||
let folder_before_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(before::folder_to_dto(f.clone()));
|
||||
}
|
||||
});
|
||||
let folder_after_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(FolderDto::from(f.clone()));
|
||||
}
|
||||
});
|
||||
let folder_base_ns = folder_base_s * 1e9 / rows as f64;
|
||||
let folder_before_ns = (folder_before_s - folder_base_s) * 1e9 / rows as f64;
|
||||
let folder_after_ns = (folder_after_s - folder_base_s) * 1e9 / rows as f64;
|
||||
println!(" clone-only baseline: {folder_base_ns:8.1} ns/row");
|
||||
println!(" BEFORE mapping: {folder_before_ns:8.1} ns/row");
|
||||
println!(" AFTER mapping: {folder_after_ns:8.1} ns/row\n");
|
||||
|
||||
// ── Section 3: allocation calls per row ─────────────────────────────
|
||||
println!("── Section 3: allocator calls per row (net of clone) ──");
|
||||
let file_base_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(f.clone());
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64;
|
||||
let file_before_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(before::file_to_dto(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- file_base_a;
|
||||
let file_after_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(FileDto::from(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- file_base_a;
|
||||
let folder_base_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(f.clone());
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64;
|
||||
let folder_before_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(before::folder_to_dto(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- folder_base_a;
|
||||
let folder_after_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(FolderDto::from(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- folder_base_a;
|
||||
println!(" file clone baseline: {file_base_a:6.2} allocs/row");
|
||||
println!(" file BEFORE mapping: {file_before_a:6.2} allocs/row");
|
||||
println!(" file AFTER mapping: {file_after_a:6.2} allocs/row");
|
||||
println!(" folder clone baseline: {folder_base_a:6.2} allocs/row");
|
||||
println!(" folder BEFORE mapping: {folder_before_a:6.2} allocs/row");
|
||||
println!(" folder AFTER mapping: {folder_after_a:6.2} allocs/row\n");
|
||||
|
||||
// ── Section 4: equivalence gate ─────────────────────────────────────
|
||||
println!("── Section 4: equivalence gate (BEFORE == AFTER, field by field) ──");
|
||||
let mut diffs: u64 = 0;
|
||||
for (i, f) in files.iter().enumerate() {
|
||||
let b = before::file_to_dto(f.clone());
|
||||
let a = FileDto::from(f.clone());
|
||||
diff_file(i, &b, &a, &mut diffs);
|
||||
}
|
||||
for (i, f) in folders.iter().enumerate() {
|
||||
let b = before::folder_to_dto(f.clone());
|
||||
let a = FolderDto::from(f.clone());
|
||||
diff_folder(i, &b, &a, &mut diffs);
|
||||
}
|
||||
if diffs > 0 {
|
||||
println!(" FAILED: {diffs} field diffs between BEFORE and AFTER mappings");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!(" PASSED: {rows} files + {rows} folders map byte-identically\n");
|
||||
|
||||
// ── Markdown summary ─────────────────────────────────────────────────
|
||||
let table = [
|
||||
Row {
|
||||
variant: "File→FileDto BEFORE",
|
||||
ns_per_row: file_before_ns,
|
||||
allocs_per_row: file_before_a,
|
||||
},
|
||||
Row {
|
||||
variant: "File→FileDto AFTER",
|
||||
ns_per_row: file_after_ns,
|
||||
allocs_per_row: file_after_a,
|
||||
},
|
||||
Row {
|
||||
variant: "Folder→FolderDto BEFORE",
|
||||
ns_per_row: folder_before_ns,
|
||||
allocs_per_row: folder_before_a,
|
||||
},
|
||||
Row {
|
||||
variant: "Folder→FolderDto AFTER",
|
||||
ns_per_row: folder_after_ns,
|
||||
allocs_per_row: folder_after_a,
|
||||
},
|
||||
];
|
||||
println!("| variant | ns/row | allocs/row |");
|
||||
println!("|---|---:|---:|");
|
||||
for r in &table {
|
||||
println!(
|
||||
"| {} | {:.1} | {:.2} |",
|
||||
r.variant, r.ns_per_row, r.allocs_per_row
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
//! PROPFIND subfolder-paging benchmark — LIMIT/OFFSET + COUNT(*) OVER() vs
|
||||
//! keyset, mirroring the files-side PROPFIND-PAGING fix.
|
||||
//!
|
||||
//! The streaming PROPFIND walkers (native WebDAV + NC-DAV) page a folder's
|
||||
//! subfolders via `list_folders_paginated`, whose query is
|
||||
//! `COUNT(*) OVER() … ORDER BY name LIMIT $2 OFFSET $3` — every page
|
||||
//! window-aggregates and rescans ALL N subfolders (the total is only used
|
||||
//! for has_next), so a full walk is O(N²/page) row visits.
|
||||
//!
|
||||
//! The AFTER shape is the same keyset used for files: `name > $last ORDER BY
|
||||
//! name LIMIT k`, served by the existing UNIQUE index
|
||||
//! `idx_folders_unique_name (parent_id, name, drive_id) WHERE NOT is_trashed
|
||||
//! AND parent_id IS NOT NULL` — no migration needed. has_next falls out of
|
||||
//! `rows.len() == limit`.
|
||||
//!
|
||||
//! Equivalence gate: the drained name sequence must be identical.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_folder_keyset
|
||||
//! Tunables: BENCH_DIRS (5000), BENCH_PAGE (500), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, dirs: usize) -> (Uuid, Uuid) {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_folder_keyset', '/bench_folder_keyset', 'bench_folder_keyset', $1)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.folders (name, path, lpath, parent_id, drive_id)
|
||||
SELECT 'Dir_' || LPAD(i::text, 6, '0'),
|
||||
'/bench_folder_keyset/Dir_' || LPAD(i::text, 6, '0'),
|
||||
('bench_folder_keyset.d' || i)::ltree,
|
||||
$1, $2
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(dirs as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("dirs");
|
||||
sqlx::query("ANALYZE storage.folders")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(drive_id, folder_id)
|
||||
}
|
||||
|
||||
const COLS: &str = "id::text, name, path, parent_id::text, drive_id,
|
||||
EXTRACT(EPOCH FROM created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM updated_at)::bigint,
|
||||
EXTRACT(EPOCH FROM tree_modified_at)::bigint,
|
||||
created_by, updated_by";
|
||||
|
||||
type Row = (
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
Uuid,
|
||||
i64,
|
||||
i64,
|
||||
i64,
|
||||
Option<Uuid>,
|
||||
Option<Uuid>,
|
||||
);
|
||||
type RowWithTotal = (
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
Uuid,
|
||||
i64,
|
||||
i64,
|
||||
i64,
|
||||
Option<Uuid>,
|
||||
Option<Uuid>,
|
||||
i64,
|
||||
);
|
||||
|
||||
/// OLD: production `list_folders_paginated` shape — window total + OFFSET.
|
||||
async fn walk_offset(pool: &PgPool, parent: Uuid, page: i64) -> (Vec<String>, Vec<f64>) {
|
||||
let mut offset = 0i64;
|
||||
let mut names = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows: Vec<RowWithTotal> = sqlx::query_as(&format!(
|
||||
"SELECT {COLS}, COUNT(*) OVER() AS total_count
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed
|
||||
ORDER BY name
|
||||
LIMIT $2 OFFSET $3"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.bind(offset)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("offset page");
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
names.extend(rows.into_iter().map(|r| r.1));
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
offset += n as i64;
|
||||
}
|
||||
(names, times)
|
||||
}
|
||||
|
||||
/// NEW: keyset on the existing unique index; has_next = rows.len() == limit.
|
||||
async fn walk_keyset(pool: &PgPool, parent: Uuid, page: i64) -> (Vec<String>, Vec<f64>) {
|
||||
let mut after: Option<String> = None;
|
||||
let mut names = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows: Vec<Row> = if let Some(a) = &after {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed AND name > $3
|
||||
ORDER BY name
|
||||
LIMIT $2"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.bind(a)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
} else {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed
|
||||
ORDER BY name
|
||||
LIMIT $2"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
}
|
||||
.expect("keyset page");
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
after = rows.last().map(|r| r.1.clone());
|
||||
names.extend(rows.into_iter().map(|r| r.1));
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
}
|
||||
(names, times)
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let dirs: usize = env_or("BENCH_DIRS", 5_000);
|
||||
let page: i64 = env_or("BENCH_PAGE", 500);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {dirs} subfolders (one-time)…");
|
||||
let (drive_id, folder_id) = seed(&pool, dirs).await;
|
||||
|
||||
let (ref_names, _) = walk_offset(&pool, folder_id, page).await;
|
||||
assert_eq!(ref_names.len(), dirs, "reference drain size");
|
||||
|
||||
println!("\n# full PROPFIND subfolder walk of a {dirs}-dir parent, {page}/page");
|
||||
println!(
|
||||
"{:<12} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
let mut base: Option<f64> = None;
|
||||
for mode in ["OFFSET", "KEYSET"] {
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut per_page: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (names, times) = if mode == "OFFSET" {
|
||||
walk_offset(&pool, folder_id, page).await
|
||||
} else {
|
||||
walk_keyset(&pool, folder_id, page).await
|
||||
};
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if names != ref_names {
|
||||
eprintln!("EQUIVALENCE FAILURE: {mode} drained a different sequence");
|
||||
failures += 1;
|
||||
}
|
||||
per_page = times;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<12} {:>11.1} {:>11.2} {:>8}",
|
||||
mode,
|
||||
ms,
|
||||
median(per_page.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,495 @@
|
||||
//! Web-UI folder listing benchmark — whole-folder rescan vs keyset pushdown.
|
||||
//!
|
||||
//! `list_resources_paged` (folder_db_repository.rs) pages the SPA files view
|
||||
//! with a UNION-ALL CTE (folders + files) and applies the keyset cursor
|
||||
//! OUTSIDE the CTE on computed columns (`sort_str = LOWER(name)`,
|
||||
//! `folder_first`). Postgres therefore scans every remaining row of the
|
||||
//! folder and top-N-sorts it on EVERY page — a 20k-file folder pays a full
|
||||
//! rescan per 200-row page.
|
||||
//!
|
||||
//! The AFTER shape pushes the cursor into each branch as a sargable
|
||||
//! row-value comparison (`(LOWER(name), id) > ($str, $id)`), gives each
|
||||
//! branch its own `ORDER BY … LIMIT`, and adds two expression indexes:
|
||||
//! idx_files_folder_lname (folder_id, LOWER(name), id) WHERE NOT is_trashed
|
||||
//! idx_folders_parent_lname (parent_id, LOWER(name), id) WHERE NOT is_trashed
|
||||
//! The outer query then merges ≤ 2·limit pre-sorted rows.
|
||||
//!
|
||||
//! Modes (full drain of the folder in default "name" order, plus a
|
||||
//! modified_at parity check):
|
||||
//! OLD/no-idx — the true BEFORE
|
||||
//! OLD/idx — new indexes alone, old query shape
|
||||
//! NEW/idx — the AFTER
|
||||
//!
|
||||
//! Equivalence gate: the drained (type, id) sequence must be identical
|
||||
//! across all modes; a mismatch aborts with exit(1).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_listing_keyset
|
||||
//! Tunables: BENCH_FILES (20000), BENCH_DIRS (300), BENCH_PAGE (200),
|
||||
//! BENCH_REPS (3)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, files: usize, dirs: usize) -> (Uuid, Uuid) {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_listing', '/bench_listing', 'bench_listing', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Mixed-case names so LOWER() actually differs from the raw column.
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.folders (name, path, lpath, parent_id, drive_id)
|
||||
SELECT 'Dir_' || LPAD(i::text, 6, '0'),
|
||||
'/bench_listing/Dir_' || LPAD(i::text, 6, '0'),
|
||||
('bench_listing.d' || i)::ltree,
|
||||
$1, $2
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(dirs as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("dirs");
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.files
|
||||
(name, folder_id, blob_hash, size, mime_type, drive_id,
|
||||
updated_at, category_order)
|
||||
SELECT 'File_' || LPAD(i::text, 8, '0') || '.JPG', $1,
|
||||
'benchlisting0000000000000000000000000000000000000000000000000000',
|
||||
1024 + i, 'image/jpeg', $2,
|
||||
NOW() - (i || ' seconds')::interval,
|
||||
3
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(files as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
sqlx::query("ANALYZE storage.files")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("ANALYZE storage.folders")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(drive_id, folder_id)
|
||||
}
|
||||
|
||||
const FOLDER_BRANCH: &str = r#"
|
||||
SELECT
|
||||
'folder'::text AS resource_type,
|
||||
f.id,
|
||||
f.name,
|
||||
f.parent_id AS folder_id,
|
||||
NULL::text AS mime_type,
|
||||
-1::bigint AS size,
|
||||
f.created_at,
|
||||
f.updated_at AS modified_at,
|
||||
f.drive_id,
|
||||
NULL::text AS blob_hash,
|
||||
LOWER(f.name) AS sort_str,
|
||||
0::bigint AS type_order,
|
||||
0::int AS folder_first
|
||||
FROM storage.folders f
|
||||
WHERE f.parent_id = $1::uuid AND NOT f.is_trashed
|
||||
"#;
|
||||
|
||||
const FILE_BRANCH: &str = r#"
|
||||
SELECT
|
||||
'file'::text AS resource_type,
|
||||
fm.id,
|
||||
fm.name,
|
||||
fm.folder_id,
|
||||
fm.mime_type,
|
||||
fm.size::bigint,
|
||||
fm.created_at,
|
||||
fm.updated_at AS modified_at,
|
||||
fm.drive_id,
|
||||
fm.blob_hash,
|
||||
LOWER(fm.name) AS sort_str,
|
||||
fm.category_order::bigint AS type_order,
|
||||
1::int AS folder_first
|
||||
FROM storage.files fm
|
||||
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
|
||||
"#;
|
||||
|
||||
const COLS: &str = "resource_type, id, name, folder_id, mime_type, size, \
|
||||
created_at, modified_at, drive_id, blob_hash, \
|
||||
sort_str, type_order, folder_first";
|
||||
|
||||
type Row = (
|
||||
String,
|
||||
Uuid,
|
||||
String,
|
||||
Option<Uuid>,
|
||||
Option<String>,
|
||||
i64,
|
||||
chrono::DateTime<chrono::Utc>,
|
||||
chrono::DateTime<chrono::Utc>,
|
||||
Uuid,
|
||||
Option<String>,
|
||||
String,
|
||||
i64,
|
||||
i32,
|
||||
);
|
||||
|
||||
/// Cursor state for the walks: (folder_first, sort_str, modified_at, id).
|
||||
#[derive(Clone)]
|
||||
struct Cur {
|
||||
ff: i64,
|
||||
sort_str: String,
|
||||
ts: chrono::DateTime<chrono::Utc>,
|
||||
id: Uuid,
|
||||
}
|
||||
|
||||
/// OLD shape, "name" order — production SQL verbatim: cursor OUTSIDE the CTE.
|
||||
async fn old_page_name(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
let sql = format!(
|
||||
"WITH resources AS ({FOLDER_BRANCH} UNION ALL {FILE_BRANCH}) \
|
||||
SELECT {COLS} FROM resources \
|
||||
WHERE ($3::bigint IS NULL) \
|
||||
OR (folder_first::bigint > $3) \
|
||||
OR (folder_first::bigint = $3 AND sort_str > $2) \
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid) \
|
||||
ORDER BY folder_first ASC, sort_str ASC, id ASC \
|
||||
LIMIT $6"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(cur.map(|c| c.sort_str.clone()))
|
||||
.bind(cur.map(|c| c.ff))
|
||||
.bind(cur.map(|c| c.ts))
|
||||
.bind(cur.map(|c| c.id))
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old name page")
|
||||
}
|
||||
|
||||
/// NEW shape, "name" order — cursor pushed into each branch as a sargable
|
||||
/// row-value comparison; each branch pre-sorts and pre-limits.
|
||||
async fn new_page_name(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
match cur {
|
||||
None => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH}) fb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (first)")
|
||||
}
|
||||
Some(c) if c.ff == 0 => {
|
||||
// Cursor sits in the folder group: folders continue after the
|
||||
// row-value cursor; ALL files still follow.
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH} \
|
||||
AND (LOWER(f.name), f.id) > ($3, $4::uuid)) fb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(&c.sort_str)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (folder cursor)")
|
||||
}
|
||||
Some(c) => {
|
||||
// Cursor sits in the file group: the folder branch is exhausted.
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
SELECT * FROM ({FILE_BRANCH} \
|
||||
AND (LOWER(fm.name), fm.id) > ($3, $4::uuid)) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2 \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(&c.sort_str)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (file cursor)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// OLD shape, "modified_at" order (newest first) — production SQL verbatim.
|
||||
async fn old_page_modified(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
let sql = format!(
|
||||
"WITH resources AS ({FOLDER_BRANCH} UNION ALL {FILE_BRANCH}) \
|
||||
SELECT {COLS} FROM resources \
|
||||
WHERE ($4::timestamptz IS NULL) \
|
||||
OR (modified_at < $4) \
|
||||
OR (modified_at = $4 AND id < $5::uuid) \
|
||||
ORDER BY modified_at DESC, id DESC \
|
||||
LIMIT $6"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(cur.map(|c| c.sort_str.clone()))
|
||||
.bind(cur.map(|c| c.ff))
|
||||
.bind(cur.map(|c| c.ts))
|
||||
.bind(cur.map(|c| c.id))
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old modified page")
|
||||
}
|
||||
|
||||
/// NEW shape, "modified_at" order — per-branch row-value cursor + LIMIT.
|
||||
async fn new_page_modified(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
match cur {
|
||||
None => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH}) fb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
) r ORDER BY modified_at DESC, id DESC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new modified page (first)")
|
||||
}
|
||||
Some(c) => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH} \
|
||||
AND (f.updated_at, f.id) < ($3, $4::uuid)) fb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH} \
|
||||
AND (fm.updated_at, fm.id) < ($3, $4::uuid)) lb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
) r ORDER BY modified_at DESC, id DESC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(c.ts)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new modified page (cursor)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Drain the whole folder; returns ((type, id) sequence, per-page ms).
|
||||
async fn drain(
|
||||
pool: &PgPool,
|
||||
parent: Uuid,
|
||||
limit: i64,
|
||||
new_shape: bool,
|
||||
by_modified: bool,
|
||||
) -> (Vec<(String, Uuid)>, Vec<f64>) {
|
||||
let mut cur: Option<Cur> = None;
|
||||
let mut seq = Vec::new();
|
||||
let mut page_ms = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows = match (new_shape, by_modified) {
|
||||
(false, false) => old_page_name(pool, parent, cur.as_ref(), limit).await,
|
||||
(true, false) => new_page_name(pool, parent, cur.as_ref(), limit).await,
|
||||
(false, true) => old_page_modified(pool, parent, cur.as_ref(), limit).await,
|
||||
(true, true) => new_page_modified(pool, parent, cur.as_ref(), limit).await,
|
||||
};
|
||||
page_ms.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
if let Some(last) = rows.last() {
|
||||
cur = Some(Cur {
|
||||
ff: last.12 as i64,
|
||||
sort_str: last.10.clone(),
|
||||
ts: last.7,
|
||||
id: last.1,
|
||||
});
|
||||
}
|
||||
seq.extend(rows.into_iter().map(|r| (r.0, r.1)));
|
||||
if (n as i64) < limit {
|
||||
break;
|
||||
}
|
||||
}
|
||||
(seq, page_ms)
|
||||
}
|
||||
|
||||
async fn set_indexes(pool: &PgPool, on: bool) {
|
||||
if on {
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_files_folder_lname
|
||||
ON storage.files (folder_id, LOWER(name), id) WHERE NOT is_trashed",
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files idx");
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_folders_parent_lname
|
||||
ON storage.folders (parent_id, LOWER(name), id) WHERE NOT is_trashed",
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("folders idx");
|
||||
} else {
|
||||
sqlx::query("DROP INDEX IF EXISTS storage.idx_files_folder_lname")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("DROP INDEX IF EXISTS storage.idx_folders_parent_lname")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
}
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
fn p99(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[(xs.len() as f64 * 0.99) as usize % xs.len()]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let files: usize = env_or("BENCH_FILES", 20_000);
|
||||
let dirs: usize = env_or("BENCH_DIRS", 300);
|
||||
let page: i64 = env_or("BENCH_PAGE", 200);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {files} files + {dirs} dirs (one-time)…");
|
||||
let (drive_id, folder_id) = seed(&pool, files, dirs).await;
|
||||
let total = files + dirs;
|
||||
|
||||
// Reference sequences for the equivalence gate (computed once per mode).
|
||||
set_indexes(&pool, false).await;
|
||||
let (ref_name, _) = drain(&pool, folder_id, page, false, false).await;
|
||||
let (ref_modified, _) = drain(&pool, folder_id, page, false, true).await;
|
||||
assert_eq!(ref_name.len(), total, "name drain row count");
|
||||
assert_eq!(ref_modified.len(), total, "modified drain row count");
|
||||
|
||||
println!("\n# full SPA-listing drain of a {files}-file/{dirs}-dir folder, {page}/page");
|
||||
println!(
|
||||
"{:<28} {:>11} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "p99 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
for by_modified in [false, true] {
|
||||
let label = if by_modified { "modified_at" } else { "name" };
|
||||
let reference = if by_modified {
|
||||
&ref_modified
|
||||
} else {
|
||||
&ref_name
|
||||
};
|
||||
let mut base: Option<f64> = None;
|
||||
for (mode, new_shape, idx) in [
|
||||
("OLD/no-idx", false, false),
|
||||
("OLD/idx", false, true),
|
||||
("NEW/idx", true, true),
|
||||
] {
|
||||
set_indexes(&pool, idx).await;
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut pages: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (seq, page_ms) = drain(&pool, folder_id, page, new_shape, by_modified).await;
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if &seq != reference {
|
||||
eprintln!("EQUIVALENCE FAILURE: {label}/{mode} drained a different sequence");
|
||||
failures += 1;
|
||||
}
|
||||
pages = page_ms;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<28} {:>11.1} {:>11.2} {:>11.2} {:>8}",
|
||||
format!("{label} {mode}"),
|
||||
ms,
|
||||
median(pages.clone()),
|
||||
p99(pages.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
// Leave the new indexes in place (they are the production migration).
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
//! Photos timeline benchmark — full-library scan vs per-drive LATERAL top-N.
|
||||
//!
|
||||
//! `list_media_files` (file_blob_read_repository.rs) filters by
|
||||
//! `fi.drive_id IN (<grants subquery>)`, joins folders + file_metadata, and
|
||||
//! sorts globally by `media_sort_date DESC LIMIT k`. The doc comment claims
|
||||
//! `idx_files_media_timeline_by_drive` lets LIMIT stop the scan early, but
|
||||
//! the plan is a Nested Loop over the drive set feeding EVERY media row
|
||||
//! through a Hash Left Join into a top-N heapsort ABOVE the join — the
|
||||
//! index is drained to exhaustion on every page, so each timeline page
|
||||
//! costs O(library), not O(page).
|
||||
//!
|
||||
//! The AFTER shape materialises the accessible drive ids once, then does a
|
||||
//! `CROSS JOIN LATERAL (… ORDER BY media_sort_date DESC LIMIT k)` per drive
|
||||
//! — each LATERAL is one bounded index scan — and merges `drives × k` rows.
|
||||
//! The folders/file_metadata joins move OUTSIDE the top-N so only the k
|
||||
//! emitted rows pay them.
|
||||
//!
|
||||
//! Equivalence gate: page-by-page id sequences must be identical (the seed
|
||||
//! uses strictly distinct capture dates so ties cannot mask reordering).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_photos_timeline
|
||||
//! Tunables: BENCH_MEDIA (50000), BENCH_DRIVES (3), BENCH_PAGE (100),
|
||||
//! BENCH_PAGES (10), BENCH_REPS (3)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, media: usize, drives: usize) -> (Uuid, Vec<Uuid>) {
|
||||
let caller = Uuid::new_v4();
|
||||
let mut drive_ids = Vec::with_capacity(drives);
|
||||
for d in 0..drives {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes, policies)
|
||||
VALUES ('shared', NULL, '{\"include_in_photo_index\": true}'::jsonb)
|
||||
RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ($1, $2, $3::ltree, $4) RETURNING id",
|
||||
)
|
||||
.bind(format!("bench_photos_{d}"))
|
||||
.bind(format!("/bench_photos_{d}"))
|
||||
.bind(format!("bench_photos_{d}"))
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.role_grants
|
||||
(subject_type, subject_id, resource_type, resource_id, role, granted_by)
|
||||
VALUES ('user', $1, 'drive', $2, 'viewer', $1)",
|
||||
)
|
||||
.bind(caller)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("grant");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Strictly distinct capture dates (offset per drive) so the
|
||||
// equivalence gate cannot be masked by tie reordering.
|
||||
let per_drive = media / drives;
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.files
|
||||
(name, folder_id, blob_hash, size, mime_type, drive_id, media_sort_date)
|
||||
SELECT 'IMG_' || LPAD(i::text, 8, '0') || '.jpg', $1,
|
||||
'benchphotos00000000000000000000000000000000000000000000000000000',
|
||||
2048, 'image/jpeg', $2,
|
||||
TIMESTAMPTZ '2026-01-01 00:00:00Z' - ((i * $4 + $5) || ' seconds')::interval
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(per_drive as i32)
|
||||
.bind(drives as i32)
|
||||
.bind(d as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
drive_ids.push(drive_id);
|
||||
}
|
||||
sqlx::query("ANALYZE storage.files")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("ANALYZE storage.role_grants")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(caller, drive_ids)
|
||||
}
|
||||
|
||||
type MediaRow = (
|
||||
String, // id::text
|
||||
String, // name
|
||||
Option<String>, // folder_id::text
|
||||
Option<String>, // fo.path
|
||||
i64, // size
|
||||
String, // mime_type
|
||||
i64, // created_at epoch
|
||||
i64, // updated_at epoch
|
||||
String, // blob_hash
|
||||
Option<Uuid>, // created_by
|
||||
Option<Uuid>, // updated_by
|
||||
i64, // sort_date epoch
|
||||
Option<i32>, // width
|
||||
Option<i32>, // height
|
||||
);
|
||||
|
||||
const GRANTS_SUBQ: &str = r#"
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
"#;
|
||||
|
||||
/// OLD shape — production SQL verbatim.
|
||||
async fn old_page(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
before: Option<chrono::DateTime<chrono::Utc>>,
|
||||
limit: i64,
|
||||
) -> Vec<MediaRow> {
|
||||
let cursor_pred = if before.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
fi.created_by, fi.updated_by,
|
||||
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
|
||||
WHERE fi.drive_id IN ({GRANTS_SUBQ})
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
"#
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(caller)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old page")
|
||||
}
|
||||
|
||||
/// NEW shape — accessible drives materialised once, per-drive LATERAL top-N
|
||||
/// on the timeline index, folders/metadata joined only on the emitted rows.
|
||||
async fn new_page(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
before: Option<chrono::DateTime<chrono::Utc>>,
|
||||
limit: i64,
|
||||
) -> Vec<MediaRow> {
|
||||
let cursor_pred = if before.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
WITH accessible AS MATERIALIZED ({GRANTS_SUBQ})
|
||||
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
|
||||
top.size, top.mime_type,
|
||||
EXTRACT(EPOCH FROM top.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM top.updated_at)::bigint,
|
||||
top.blob_hash,
|
||||
top.created_by, top.updated_by,
|
||||
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM (
|
||||
SELECT fi.*
|
||||
FROM accessible a
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT fi.*
|
||||
FROM storage.files fi
|
||||
WHERE fi.drive_id = a.id
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) fi
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) top
|
||||
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
|
||||
ORDER BY top.media_sort_date DESC
|
||||
"#
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(caller)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new page")
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// Walk `pages` cursor pages; returns (id sequence, per-page ms).
|
||||
async fn walk(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
page: i64,
|
||||
pages: usize,
|
||||
new_shape: bool,
|
||||
) -> (Vec<String>, Vec<f64>) {
|
||||
let mut before: Option<chrono::DateTime<chrono::Utc>> = None;
|
||||
let mut ids = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
for _ in 0..pages {
|
||||
let t = Instant::now();
|
||||
let rows = if new_shape {
|
||||
new_page(pool, caller, before, page).await
|
||||
} else {
|
||||
old_page(pool, caller, before, page).await
|
||||
};
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if rows.is_empty() {
|
||||
break;
|
||||
}
|
||||
// Cursor semantics mirror production: whole-second epoch of the last
|
||||
// row (list_media_files hands the epoch back to the client).
|
||||
let last_epoch = rows.last().unwrap().11;
|
||||
before = chrono::DateTime::from_timestamp(last_epoch, 0);
|
||||
ids.extend(rows.into_iter().map(|r| r.0));
|
||||
}
|
||||
(ids, times)
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let media: usize = env_or("BENCH_MEDIA", 50_000);
|
||||
let drives: usize = env_or("BENCH_DRIVES", 3);
|
||||
let page: i64 = env_or("BENCH_PAGE", 100);
|
||||
let pages: usize = env_or("BENCH_PAGES", 10);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {media} media rows across {drives} drives (one-time)…");
|
||||
let (caller, drive_ids) = seed(&pool, media, drives).await;
|
||||
|
||||
let (ref_ids, _) = walk(&pool, caller, page, pages, false).await;
|
||||
assert_eq!(
|
||||
ref_ids.len(),
|
||||
(page as usize) * pages,
|
||||
"reference walk size"
|
||||
);
|
||||
|
||||
println!("\n# {pages} timeline pages of {page} over a {media}-photo library ({drives} drives)");
|
||||
println!(
|
||||
"{:<8} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
let mut base: Option<f64> = None;
|
||||
for (mode, new_shape) in [("OLD", false), ("NEW", true)] {
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut per_page: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (ids, times) = walk(&pool, caller, page, pages, new_shape).await;
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if ids != ref_ids {
|
||||
eprintln!("EQUIVALENCE FAILURE: {mode} walk drained different ids");
|
||||
failures += 1;
|
||||
}
|
||||
per_page = times;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<8} {:>11.1} {:>11.2} {:>8}",
|
||||
mode,
|
||||
ms,
|
||||
median(per_page.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
for d in drive_ids {
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(d)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
}
|
||||
let _ = sqlx::query("DELETE FROM storage.role_grants WHERE subject_id = $1")
|
||||
.bind(caller)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
//! S3 chunk-PUT benchmark — HEAD-before-PUT vs unconditional PUT.
|
||||
//!
|
||||
//! `DedupService::settle_batch` writes every NEW chunk of every upload via
|
||||
//! `put_blob_from_bytes_unsynced`. S3/Azure never overrode it, so the trait
|
||||
//! default routed it through `put_blob_from_bytes`, whose "idempotent" HEAD
|
||||
//! probe made every chunk write pay 2 request round-trips. Content-addressed
|
||||
//! keys make re-PUTs overwrite-safe, so the new override PUTs directly.
|
||||
//!
|
||||
//! The stub S3 endpoint (in-process axum, per-request latency injection)
|
||||
//! counts HEAD/PUT requests:
|
||||
//! BEFORE — put_blob_from_bytes (HEAD 404 + PUT per chunk)
|
||||
//! AFTER — put_blob_from_bytes_unsynced (PUT per chunk)
|
||||
//!
|
||||
//! Section 2 measures the removed Azure `data.to_vec()` copy in isolation.
|
||||
//!
|
||||
//! Gates: AFTER request count == chunks (vs 2x), AFTER wall < BEFORE wall.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_s3_put
|
||||
//! Tunables: BENCH_CHUNKS (500), BENCH_CHUNK_KB (256), BENCH_CONCURRENCY (8),
|
||||
//! BENCH_RTT_MS (10)
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::Bytes;
|
||||
use oxicloud::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use oxicloud::common::config::S3StorageConfig;
|
||||
use oxicloud::infrastructure::services::s3_blob_backend::S3BlobBackend;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct Counters {
|
||||
heads: Arc<AtomicU64>,
|
||||
puts: Arc<AtomicU64>,
|
||||
}
|
||||
|
||||
async fn stub_s3(latency: Duration, counters: Counters) -> String {
|
||||
use axum::http::{Method, StatusCode};
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
let app = axum::Router::new().fallback(move |req: axum::extract::Request| {
|
||||
let counters = counters.clone();
|
||||
async move {
|
||||
tokio::time::sleep(latency).await;
|
||||
match *req.method() {
|
||||
Method::HEAD => {
|
||||
counters.heads.fetch_add(1, Ordering::Relaxed);
|
||||
StatusCode::NOT_FOUND
|
||||
}
|
||||
Method::PUT => {
|
||||
// Drain the body like a real endpoint would.
|
||||
let _ = axum::body::to_bytes(req.into_body(), usize::MAX).await;
|
||||
counters.puts.fetch_add(1, Ordering::Relaxed);
|
||||
StatusCode::OK
|
||||
}
|
||||
_ => StatusCode::OK,
|
||||
}
|
||||
}
|
||||
});
|
||||
tokio::spawn(async move {
|
||||
axum::serve(listener, app).await.expect("serve");
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
async fn drive(
|
||||
backend: Arc<S3BlobBackend>,
|
||||
chunks: usize,
|
||||
chunk_kb: usize,
|
||||
concurrency: usize,
|
||||
unsynced: bool,
|
||||
) -> f64 {
|
||||
let payload = Bytes::from(vec![0x5au8; chunk_kb * 1024]);
|
||||
let sem = Arc::new(tokio::sync::Semaphore::new(concurrency));
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for i in 0..chunks {
|
||||
let b = backend.clone();
|
||||
let p = payload.clone();
|
||||
let sem = sem.clone();
|
||||
set.spawn(async move {
|
||||
let _permit = sem.acquire().await.expect("sem");
|
||||
let hash = format!("{i:064x}");
|
||||
let n = if unsynced {
|
||||
b.put_blob_from_bytes_unsynced(&hash, p).await.expect("put")
|
||||
} else {
|
||||
b.put_blob_from_bytes(&hash, p).await.expect("put")
|
||||
};
|
||||
assert_eq!(n as usize, chunk_kb * 1024);
|
||||
});
|
||||
}
|
||||
while let Some(r) = set.join_next().await {
|
||||
r.expect("join");
|
||||
}
|
||||
t.elapsed().as_secs_f64() * 1000.0
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let chunks: usize = env_or("BENCH_CHUNKS", 500);
|
||||
let chunk_kb: usize = env_or("BENCH_CHUNK_KB", 256);
|
||||
let concurrency: usize = env_or("BENCH_CONCURRENCY", 8);
|
||||
let rtt_ms: u64 = env_or("BENCH_RTT_MS", 10);
|
||||
|
||||
let counters = Counters::default();
|
||||
let endpoint = stub_s3(Duration::from_millis(rtt_ms), counters.clone()).await;
|
||||
let backend = Arc::new(S3BlobBackend::new(&S3StorageConfig {
|
||||
endpoint_url: Some(endpoint),
|
||||
bucket: "bench".into(),
|
||||
region: "us-east-1".into(),
|
||||
access_key: "bench".into(),
|
||||
secret_key: "bench".into(),
|
||||
force_path_style: true,
|
||||
}));
|
||||
|
||||
println!(
|
||||
"# {chunks} x {chunk_kb} KiB chunk PUTs at concurrency {concurrency}, {rtt_ms} ms/request stub"
|
||||
);
|
||||
println!(
|
||||
"{:<26} {:>10} {:>8} {:>8} {:>8}",
|
||||
"variant", "wall ms", "HEADs", "PUTs", "vs OLD"
|
||||
);
|
||||
|
||||
// BEFORE: the trait-default route (put_blob_from_bytes = HEAD + PUT).
|
||||
let before = drive(backend.clone(), chunks, chunk_kb, concurrency, false).await;
|
||||
let before_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let before_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<26} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
"BEFORE (HEAD+PUT)", before, before_heads, before_puts, "1.0x"
|
||||
);
|
||||
|
||||
// AFTER: the unsynced override (PUT only).
|
||||
let after = drive(backend.clone(), chunks, chunk_kb, concurrency, true).await;
|
||||
let after_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let after_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<26} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
"AFTER (PUT only)",
|
||||
after,
|
||||
after_heads,
|
||||
after_puts,
|
||||
format!("{:.1}x", before / after)
|
||||
);
|
||||
|
||||
// ── Section 2: the removed Azure to_vec() copy, in isolation ───────
|
||||
let mb = 4;
|
||||
let data = Bytes::from(vec![0x77u8; mb * 1024 * 1024]);
|
||||
let reps = 200;
|
||||
let t = Instant::now();
|
||||
for _ in 0..reps {
|
||||
let v = data.to_vec();
|
||||
std::hint::black_box(&v);
|
||||
}
|
||||
let copy_ms = t.elapsed().as_secs_f64() * 1000.0 / reps as f64;
|
||||
println!(
|
||||
"\n# [2] removed Azure per-chunk copy: to_vec() of {mb} MiB = {copy_ms:.2} ms + {mb} MiB transient alloc per chunk"
|
||||
);
|
||||
|
||||
// ── Gates ───────────────────────────────────────────────────────────
|
||||
if after_heads != 0 || after_puts != chunks as u64 {
|
||||
eprintln!(
|
||||
"GATE FAIL: AFTER issued {after_heads} HEADs / {after_puts} PUTs (expected 0 / {chunks})"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if after >= before {
|
||||
eprintln!(
|
||||
"GATE FAIL: AFTER ({after:.0} ms) not faster than BEFORE ({before:.0} ms) — rollback"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!(
|
||||
"GATE PASS: {}-request walk -> {} requests, {:.1}x faster",
|
||||
before_heads + before_puts,
|
||||
after_puts,
|
||||
before / after
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,361 @@
|
||||
//! Search-results cache memory benchmark — entry-count bound vs byte bound.
|
||||
//!
|
||||
//! The search cache keys pages by user × query × offset × limit, and each
|
||||
//! page holds up to 500 enriched rows (`MAX_SEARCH_LIMIT`) of owned Strings.
|
||||
//! Bounded by ENTRY COUNT (the old scheme: `max_capacity(1000)` + TTL), a
|
||||
//! burst of keystrokes/pages/users could pin ~300 MB of invisible RSS for
|
||||
//! the 5-minute TTL. Bounded by BYTES (a `weigher` + 32 MiB budget — the
|
||||
//! same pattern as the file-content and dedup-manifest caches), retention
|
||||
//! can never exceed the budget.
|
||||
//!
|
||||
//! Two sub-phases over the same synthetic corpus (1,000 pages × 500 rows,
|
||||
//! ~150-char paths, realistic field contents):
|
||||
//! * BEFORE — a moka cache configured exactly as the old production wiring
|
||||
//! (entry-count 1000 + 300 s TTL).
|
||||
//! * AFTER — `build_search_results_cache(...)`, the *identical* function
|
||||
//! production now uses (weigher + 32 MiB + 300 s TTL).
|
||||
//!
|
||||
//! Reported per phase: entries retained, retained bytes (recomputed with the
|
||||
//! production weigher after `run_pending_tasks`), best-effort process memory
|
||||
//! (`VmHWM`/`VmRSS` from /proc/self/status), and hot-key `get()` p50 over
|
||||
//! 100k reads (proves the weigher — which only runs on insert — does not
|
||||
//! slow reads).
|
||||
//!
|
||||
//! NOTE on RSS: `VmHWM` is a monotonic high-water mark and the allocator may
|
||||
//! keep freed pages, so the AFTER phase (which runs second, after a full
|
||||
//! drop of the BEFORE cache) cannot show a peak below the BEFORE peak.
|
||||
//! Treat the RSS columns as best-effort corroboration; the authoritative
|
||||
//! metric is the weigher-recomputed retained bytes.
|
||||
//!
|
||||
//! Gates (exit code 1 on failure):
|
||||
//! * AFTER retained bytes ≤ 32 MiB budget
|
||||
//! * BEFORE retained bytes ≥ 8× the budget (measured ≈9–10×)
|
||||
//! * AFTER get() p50 within 20% of BEFORE
|
||||
//!
|
||||
//! No Postgres needed.
|
||||
//! Run: `cargo run --release --features bench --example bench_search_cache_mem`
|
||||
|
||||
use std::hint::black_box;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use oxicloud::application::dtos::search_dto::{SearchFileResultDto, SearchResultsDto};
|
||||
use oxicloud::application::services::search_service::{
|
||||
build_search_results_cache, search_results_entry_weight,
|
||||
};
|
||||
|
||||
/// Distinct cached pages inserted per phase (≈ users × queries × pages).
|
||||
const ENTRIES: u64 = 1_000;
|
||||
/// Rows per page — the handler's `MAX_SEARCH_LIMIT` clamp.
|
||||
const ROWS_PER_ENTRY: usize = 500;
|
||||
/// Production TTL (unchanged by the fix).
|
||||
const TTL_SECS: u64 = 300;
|
||||
/// The old production bound: 1000 ENTRIES, blind to entry size.
|
||||
const BEFORE_MAX_ENTRIES: u64 = 1_000;
|
||||
/// The new production bound: 32 MiB of weighed bytes.
|
||||
const AFTER_MAX_BYTES: u64 = 32 * 1024 * 1024;
|
||||
/// Hot-key reads per phase for the p50 latency comparison.
|
||||
const GETS: usize = 100_000;
|
||||
|
||||
const MIB: f64 = 1024.0 * 1024.0;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Deterministic synthetic corpus (no rand dependency)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Tiny xorshift64 PRNG — fast, deterministic, no dependency.
|
||||
fn xorshift(state: &mut u64) -> u64 {
|
||||
let mut x = *state;
|
||||
x ^= x << 13;
|
||||
x ^= x >> 7;
|
||||
x ^= x << 17;
|
||||
*state = x;
|
||||
x
|
||||
}
|
||||
|
||||
/// Lowercase-hex string of `chars` nibbles.
|
||||
fn pseudo_hex(state: &mut u64, chars: usize) -> String {
|
||||
let mut s = String::with_capacity(chars);
|
||||
while s.len() < chars {
|
||||
let block = format!("{:016x}", xorshift(state));
|
||||
let take = (chars - s.len()).min(16);
|
||||
s.push_str(&block[..take]);
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
/// 36-char UUID-shaped string (8-4-4-4-12), like the real `Uuid::to_string()`
|
||||
/// ids that populate `SearchFileResultDto::id` / `folder_id`.
|
||||
fn pseudo_uuid(state: &mut u64) -> String {
|
||||
let h = pseudo_hex(state, 32);
|
||||
format!(
|
||||
"{}-{}-{}-{}-{}",
|
||||
&h[0..8],
|
||||
&h[8..12],
|
||||
&h[12..16],
|
||||
&h[16..20],
|
||||
&h[20..32]
|
||||
)
|
||||
}
|
||||
|
||||
/// One synthetic 500-row search page with realistic field contents:
|
||||
/// UUID ids, ~30-char names, ~150-char nested drive paths, real MIME types,
|
||||
/// 64-hex BLAKE3 blob hashes, icon/category metadata, and a content-index
|
||||
/// snippet on every 8th row.
|
||||
fn synth_entry(idx: u64) -> Arc<SearchResultsDto> {
|
||||
const MIMES: [&str; 4] = [
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"image/jpeg",
|
||||
"text/markdown",
|
||||
];
|
||||
const SNIPPET: &str = "…the quarterly numbers show a steady increase in storage usage \
|
||||
across all departments, with the engineering share growing fastest and…";
|
||||
|
||||
let mut rng = idx.wrapping_mul(0x9E3779B97F4A7C15) | 1;
|
||||
let mut files = Vec::with_capacity(ROWS_PER_ENTRY);
|
||||
for row in 0..ROWS_PER_ENTRY {
|
||||
let name = format!(
|
||||
"quarterly_report_{:04}_rev{:03}.pdf",
|
||||
xorshift(&mut rng) % 10_000,
|
||||
row % 1_000
|
||||
);
|
||||
let path = format!(
|
||||
"/drives/{}/Departments/Engineering/Projects/oxicloud-benchmarks/2026/Q{}/weekly-sync-notes/attachments/{}",
|
||||
pseudo_uuid(&mut rng),
|
||||
row % 4 + 1,
|
||||
name
|
||||
);
|
||||
let content_hit = row % 8 == 0;
|
||||
let match_source = if content_hit { "content" } else { "name" };
|
||||
files.push(SearchFileResultDto {
|
||||
id: pseudo_uuid(&mut rng),
|
||||
name,
|
||||
path,
|
||||
size: 831_942,
|
||||
mime_type: MIMES[row % MIMES.len()].to_string(),
|
||||
folder_id: Some(pseudo_uuid(&mut rng)),
|
||||
created_at: 1_752_700_000,
|
||||
modified_at: 1_752_800_000,
|
||||
relevance_score: 50,
|
||||
size_formatted: "812.4 KB".to_string(),
|
||||
icon_class: "fas fa-file-pdf".to_string(),
|
||||
icon_special_class: "pdf-icon".to_string(),
|
||||
category: "document".to_string(),
|
||||
blob_hash: pseudo_hex(&mut rng, 64),
|
||||
snippet: content_hit.then(|| SNIPPET.to_string()),
|
||||
match_source: Some(match_source.to_string()),
|
||||
});
|
||||
}
|
||||
|
||||
Arc::new(SearchResultsDto::new(
|
||||
files,
|
||||
Vec::new(),
|
||||
ROWS_PER_ENTRY,
|
||||
0,
|
||||
Some(12_345),
|
||||
3,
|
||||
"relevance".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Best-effort process memory (Linux /proc; "n/a" elsewhere)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Read a kB-valued field (`VmHWM`, `VmRSS`) from /proc/self/status.
|
||||
fn status_kb(field: &str) -> Option<u64> {
|
||||
let text = std::fs::read_to_string("/proc/self/status").ok()?;
|
||||
text.lines()
|
||||
.find(|l| l.starts_with(field))
|
||||
.and_then(|l| l.split_whitespace().nth(1))
|
||||
.and_then(|kb| kb.parse().ok())
|
||||
}
|
||||
|
||||
fn fmt_kb(v: Option<u64>) -> String {
|
||||
match v {
|
||||
Some(kb) => format!("{:.1} MiB", kb as f64 / 1024.0),
|
||||
None => "n/a".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn fmt_kb_delta(start: Option<u64>, end: Option<u64>) -> String {
|
||||
match (start, end) {
|
||||
(Some(s), Some(e)) => format!("{:+.1} MiB", (e as f64 - s as f64) / 1024.0),
|
||||
_ => "n/a".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase runner
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
struct PhaseReport {
|
||||
retained_entries: u64,
|
||||
retained_bytes: u64,
|
||||
hwm_start_kb: Option<u64>,
|
||||
hwm_end_kb: Option<u64>,
|
||||
rss_start_kb: Option<u64>,
|
||||
rss_end_kb: Option<u64>,
|
||||
p50_get_ns: u64,
|
||||
}
|
||||
|
||||
/// Insert the full corpus, settle the cache, then measure retention and
|
||||
/// hot-key read latency. Identical for both variants — only the cache
|
||||
/// configuration differs.
|
||||
async fn run_phase(cache: &moka::future::Cache<u64, Arc<SearchResultsDto>>) -> PhaseReport {
|
||||
let hwm_start_kb = status_kb("VmHWM");
|
||||
let rss_start_kb = status_kb("VmRSS");
|
||||
|
||||
for i in 0..ENTRIES {
|
||||
cache.insert(i, synth_entry(i)).await;
|
||||
// Let eviction run as it would under live traffic, so evicted pages
|
||||
// are actually freed instead of piling up in moka's pending queue.
|
||||
if i % 64 == 0 {
|
||||
cache.run_pending_tasks().await;
|
||||
}
|
||||
}
|
||||
cache.run_pending_tasks().await;
|
||||
|
||||
let retained_entries = cache.entry_count();
|
||||
// Recompute retained bytes with the production weigher — for the BEFORE
|
||||
// variant this is exactly the memory its entry-count bound was blind to.
|
||||
let retained_bytes: u64 = cache
|
||||
.iter()
|
||||
.map(|(k, v)| u64::from(search_results_entry_weight(&k, &v)))
|
||||
.sum();
|
||||
|
||||
// Hot-key read latency: p50 over GETS reads of one resident key.
|
||||
let hot: u64 = *cache.iter().next().expect("cache is empty after fill").0;
|
||||
for _ in 0..1_000 {
|
||||
black_box(cache.get(&hot).await); // warmup
|
||||
}
|
||||
let mut lat_ns = Vec::with_capacity(GETS);
|
||||
for _ in 0..GETS {
|
||||
let t = Instant::now();
|
||||
let v = cache.get(&hot).await;
|
||||
lat_ns.push(t.elapsed().as_nanos() as u64);
|
||||
black_box(v);
|
||||
}
|
||||
lat_ns.sort_unstable();
|
||||
let p50_get_ns = lat_ns[lat_ns.len() / 2];
|
||||
|
||||
PhaseReport {
|
||||
retained_entries,
|
||||
retained_bytes,
|
||||
hwm_start_kb,
|
||||
hwm_end_kb: status_kb("VmHWM"),
|
||||
rss_start_kb,
|
||||
rss_end_kb: status_kb("VmRSS"),
|
||||
p50_get_ns,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let entry_weight = u64::from(search_results_entry_weight(&0, &synth_entry(0)));
|
||||
println!("\n###########################################################");
|
||||
println!("# Search-results cache: entry-count bound vs byte bound");
|
||||
println!(
|
||||
"# corpus: {ENTRIES} pages x {ROWS_PER_ENTRY} rows, ~{:.0} KiB/page (weigher)",
|
||||
entry_weight as f64 / 1024.0
|
||||
);
|
||||
println!(
|
||||
"# BEFORE: max_capacity({BEFORE_MAX_ENTRIES}) entries + {TTL_SECS}s TTL (old di.rs wiring)"
|
||||
);
|
||||
println!(
|
||||
"# AFTER : build_search_results_cache({TTL_SECS}, {} MiB) — production fn",
|
||||
AFTER_MAX_BYTES as f64 / MIB
|
||||
);
|
||||
println!("###########################################################\n");
|
||||
|
||||
// --- Phase 1: BEFORE (entry-count bound, exactly the old wiring) ---
|
||||
let before_cache: moka::future::Cache<u64, Arc<SearchResultsDto>> =
|
||||
moka::future::Cache::builder()
|
||||
.max_capacity(BEFORE_MAX_ENTRIES)
|
||||
.time_to_live(Duration::from_secs(TTL_SECS))
|
||||
.build();
|
||||
let before = run_phase(&before_cache).await;
|
||||
// Full drop between phases so the AFTER numbers never sit on top of the
|
||||
// BEFORE cache's live memory.
|
||||
drop(before_cache);
|
||||
|
||||
// --- Phase 2: AFTER (weigher + byte budget, the production builder) ---
|
||||
let after_cache = build_search_results_cache(TTL_SECS, AFTER_MAX_BYTES);
|
||||
let after = run_phase(&after_cache).await;
|
||||
|
||||
// --- Report ---
|
||||
println!("| metric | BEFORE (1000 entries + TTL) | AFTER (weigher + 32 MiB) |");
|
||||
println!("|---|---|---|");
|
||||
println!(
|
||||
"| entries retained | {} | {} |",
|
||||
before.retained_entries, after.retained_entries
|
||||
);
|
||||
println!(
|
||||
"| retained bytes (weigher) | {:.1} MiB | {:.1} MiB |",
|
||||
before.retained_bytes as f64 / MIB,
|
||||
after.retained_bytes as f64 / MIB
|
||||
);
|
||||
println!(
|
||||
"| byte budget | n/a (entry-count bound) | {:.0} MiB |",
|
||||
AFTER_MAX_BYTES as f64 / MIB
|
||||
);
|
||||
println!(
|
||||
"| VmHWM phase delta (best-effort) | {} | {} |",
|
||||
fmt_kb_delta(before.hwm_start_kb, before.hwm_end_kb),
|
||||
fmt_kb_delta(after.hwm_start_kb, after.hwm_end_kb)
|
||||
);
|
||||
println!(
|
||||
"| VmRSS start -> end | {} -> {} | {} -> {} |",
|
||||
fmt_kb(before.rss_start_kb),
|
||||
fmt_kb(before.rss_end_kb),
|
||||
fmt_kb(after.rss_start_kb),
|
||||
fmt_kb(after.rss_end_kb)
|
||||
);
|
||||
println!(
|
||||
"| get() p50, hot key ({GETS} reads) | {} ns | {} ns |",
|
||||
before.p50_get_ns, after.p50_get_ns
|
||||
);
|
||||
println!(
|
||||
"\nRSS note: VmHWM is monotonic and the allocator may retain freed pages, \
|
||||
so the AFTER phase (running second) cannot peak below the BEFORE peak; \
|
||||
the weigher-recomputed retained bytes are the authoritative comparison."
|
||||
);
|
||||
|
||||
// --- Gates ---
|
||||
let before_ratio = before.retained_bytes as f64 / AFTER_MAX_BYTES as f64;
|
||||
let lat_ratio = after.p50_get_ns as f64 / before.p50_get_ns.max(1) as f64;
|
||||
let gate_after_bounded = after.retained_bytes <= AFTER_MAX_BYTES;
|
||||
let gate_before_unbounded = before_ratio >= 8.0;
|
||||
let gate_latency = lat_ratio <= 1.2;
|
||||
|
||||
println!("\n| gate | condition | measured | result |");
|
||||
println!("|---|---|---|---|");
|
||||
println!(
|
||||
"| AFTER bounded | retained <= 32 MiB budget | {:.1} MiB | {} |",
|
||||
after.retained_bytes as f64 / MIB,
|
||||
if gate_after_bounded { "PASS" } else { "FAIL" }
|
||||
);
|
||||
println!(
|
||||
"| BEFORE unbounded | retained >= 8x budget (~10x expected) | {before_ratio:.1}x | {} |",
|
||||
if gate_before_unbounded {
|
||||
"PASS"
|
||||
} else {
|
||||
"FAIL"
|
||||
}
|
||||
);
|
||||
println!(
|
||||
"| read parity | AFTER p50 <= 1.2x BEFORE p50 | {lat_ratio:.2}x | {} |",
|
||||
if gate_latency { "PASS" } else { "FAIL" }
|
||||
);
|
||||
|
||||
if !(gate_after_bounded && gate_before_unbounded && gate_latency) {
|
||||
eprintln!("\nbench_search_cache_mem: GATE FAILURE");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!("\nAll gates passed.");
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
//! Upload spool/assembly I/O benchmark — buffer sizing on the chunk paths.
|
||||
//!
|
||||
//! Section 1 — assembly read (`stream_from_files`): every completed chunked
|
||||
//! upload is read back once, part file by part file, through
|
||||
//! `ReaderStream::with_capacity(file, N)`. Each poll is one blocking-pool
|
||||
//! dispatch + one read(2) of N bytes; the shipped capacity was 64 KiB while
|
||||
//! every other blob read path uses 256 KiB+. Sweeps N over
|
||||
//! 64K/256K/512K/1M and reports wall time + read syscalls.
|
||||
//!
|
||||
//! Section 2 — chunk spool write (`stream_body_to_path`): the PUT handlers
|
||||
//! wrote each HTTP frame (~16-64 KiB) straight to a bare tokio File — one
|
||||
//! blocking-pool dispatch + write(2) per frame. Compares that against the
|
||||
//! adopted `BufWriter::with_capacity(512 KiB)`.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_upload_spool
|
||||
//! Tunables: BENCH_PARTS (16), BENCH_PART_MB (10), BENCH_FRAME_KB (16),
|
||||
//! BENCH_SPOOL_MB (10), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
use bytes::Bytes;
|
||||
use futures::{StreamExt, TryStreamExt, stream};
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// (read syscalls, write syscalls) from /proc/self/io.
|
||||
fn io_counters() -> (u64, u64) {
|
||||
let s = std::fs::read_to_string("/proc/self/io").expect("io");
|
||||
let get = |k: &str| {
|
||||
s.lines()
|
||||
.find(|l| l.starts_with(k))
|
||||
.and_then(|l| l.split_whitespace().nth(1))
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(0)
|
||||
};
|
||||
(get("syscr:"), get("syscw:"))
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// The `stream_from_files` shape with a parameterized capacity.
|
||||
async fn drain_parts(paths: Vec<PathBuf>, cap: usize) -> (u64, [u8; 32]) {
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut total = 0u64;
|
||||
let s = stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
|
||||
.and_then(|path| async move {
|
||||
tokio::fs::File::open(path)
|
||||
.await
|
||||
.map(|file| ReaderStream::with_capacity(file, cap))
|
||||
})
|
||||
.try_flatten();
|
||||
let mut s = Box::pin(s);
|
||||
while let Some(chunk) = s.next().await {
|
||||
let chunk = chunk.expect("read");
|
||||
total += chunk.len() as u64;
|
||||
hasher.update(&chunk);
|
||||
}
|
||||
(total, hasher.finalize().into())
|
||||
}
|
||||
|
||||
/// The `stream_body_to_path` inner loop: frames -> file, optionally buffered.
|
||||
async fn spool_frames(frames: &[Bytes], path: &std::path::Path, buffered: bool) {
|
||||
let file = tokio::fs::File::create(path).await.expect("create");
|
||||
if buffered {
|
||||
let mut w = tokio::io::BufWriter::with_capacity(512 * 1024, file);
|
||||
for f in frames {
|
||||
w.write_all(f).await.expect("write");
|
||||
}
|
||||
w.flush().await.expect("flush");
|
||||
} else {
|
||||
let mut w = file;
|
||||
for f in frames {
|
||||
w.write_all(f).await.expect("write");
|
||||
}
|
||||
w.flush().await.expect("flush");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let parts: usize = env_or("BENCH_PARTS", 16);
|
||||
let part_mb: usize = env_or("BENCH_PART_MB", 10);
|
||||
let frame_kb: usize = env_or("BENCH_FRAME_KB", 16);
|
||||
let spool_mb: usize = env_or("BENCH_SPOOL_MB", 10);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
||||
// ── Section 1: assembly read capacity sweep ─────────────────────────
|
||||
println!("# [1] assembly read: {parts} x {part_mb} MiB part files, warm page cache");
|
||||
let mut paths = Vec::with_capacity(parts);
|
||||
let payload: Vec<u8> = (0..part_mb * 1024 * 1024)
|
||||
.map(|i| (i * 31 % 251) as u8)
|
||||
.collect();
|
||||
for i in 0..parts {
|
||||
let p = dir.path().join(format!("part_{i:05}"));
|
||||
tokio::fs::write(&p, &payload).await.expect("seed part");
|
||||
paths.push(p);
|
||||
}
|
||||
let expect_total = (parts * part_mb * 1024 * 1024) as u64;
|
||||
let (_, ref_hash) = drain_parts(paths.clone(), 256 * 1024).await;
|
||||
|
||||
println!(
|
||||
"{:<10} {:>10} {:>12} {:>8}",
|
||||
"capacity", "wall ms", "read sysc", "vs 64K"
|
||||
);
|
||||
let mut base: Option<f64> = None;
|
||||
for cap in [64 * 1024, 256 * 1024, 512 * 1024, 1024 * 1024] {
|
||||
let mut walls = Vec::with_capacity(reps);
|
||||
let mut syscr = 0u64;
|
||||
for _ in 0..reps {
|
||||
let (r0, _) = io_counters();
|
||||
let t = Instant::now();
|
||||
let (total, h) = drain_parts(paths.clone(), cap).await;
|
||||
walls.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let (r1, _) = io_counters();
|
||||
syscr = r1 - r0;
|
||||
assert_eq!(total, expect_total);
|
||||
assert_eq!(h, ref_hash, "content mismatch at capacity {cap}");
|
||||
}
|
||||
let ms = median(walls);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.2}x", b / ms))
|
||||
.unwrap_or_else(|| "1.00x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<10} {:>10.1} {:>12} {:>8}",
|
||||
format!("{}K", cap / 1024),
|
||||
ms,
|
||||
syscr,
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
// ── Section 2: chunk spool write, per-frame vs buffered ─────────────
|
||||
let frames_n = spool_mb * 1024 / frame_kb;
|
||||
println!(
|
||||
"\n# [2] chunk spool: {frames_n} x {frame_kb} KiB frames ({spool_mb} MiB), 20 files/rep"
|
||||
);
|
||||
let frame: Bytes = Bytes::from(vec![0xabu8; frame_kb * 1024]);
|
||||
let frames: Vec<Bytes> = (0..frames_n).map(|_| frame.clone()).collect();
|
||||
|
||||
println!(
|
||||
"{:<22} {:>10} {:>12} {:>8}",
|
||||
"variant", "wall ms", "write sysc", "vs bare"
|
||||
);
|
||||
let mut base: Option<f64> = None;
|
||||
for (label, buffered) in [
|
||||
("bare File (BEFORE)", false),
|
||||
("BufWriter 512K (AFTER)", true),
|
||||
] {
|
||||
let mut walls = Vec::with_capacity(reps);
|
||||
let mut syscw = 0u64;
|
||||
for r in 0..reps {
|
||||
let (_, w0) = io_counters();
|
||||
let t = Instant::now();
|
||||
for i in 0..20 {
|
||||
let p = dir.path().join(format!("spool_{r}_{i}"));
|
||||
spool_frames(&frames, &p, buffered).await;
|
||||
tokio::fs::remove_file(&p).await.ok();
|
||||
}
|
||||
walls.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let (_, w1) = io_counters();
|
||||
syscw = w1 - w0;
|
||||
}
|
||||
let ms = median(walls);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.2}x", b / ms))
|
||||
.unwrap_or_else(|| "1.00x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!("{label:<22} {:>10.1} {:>12} {:>8}", ms, syscw, speedup);
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,7 @@
|
||||
import { lazyComponent } from '$lib/composables/lazyComponent.svelte';
|
||||
import DrivePicker from '$lib/components/DrivePicker.svelte';
|
||||
import Icon from '$lib/icons/Icon.svelte';
|
||||
import { iconNameFromClass } from '$lib/utils/display';
|
||||
import { dateTimeFormatFor, iconNameFromClass } from '$lib/utils/display';
|
||||
import { userInitials, avatarColorIndex } from '$lib/utils/avatar';
|
||||
import { i18n, LANGUAGES, setLocale, t, type Locale } from '$lib/i18n/index.svelte';
|
||||
import { apiFetch } from '$lib/api/client';
|
||||
@@ -230,7 +230,7 @@
|
||||
const currentLang = $derived(LANGUAGES.find((l) => l.code === i18n.locale) ?? LANGUAGES[0]);
|
||||
|
||||
function formatTime(ms: number): string {
|
||||
return new Date(ms).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit' });
|
||||
return dateTimeFormatFor(undefined, { hour: '2-digit', minute: '2-digit' }).format(ms);
|
||||
}
|
||||
|
||||
function notifIcon(kind: string): string {
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
import { confirmDialog } from '$lib/stores/dialogs.svelte';
|
||||
import { t } from '$lib/i18n/index.svelte';
|
||||
import { errorToast } from '$lib/utils/errors';
|
||||
import { dateTimeFormatFor } from '$lib/utils/display';
|
||||
import { isVideo, photoTimestamp } from '$lib/utils/media';
|
||||
|
||||
interface Props {
|
||||
@@ -47,13 +48,13 @@
|
||||
});
|
||||
|
||||
function baseMeta(p: FileItem): string {
|
||||
const dateStr = new Date(photoTimestamp(p)).toLocaleDateString(undefined, {
|
||||
const dateStr = dateTimeFormatFor(undefined, {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric',
|
||||
hour: '2-digit',
|
||||
minute: '2-digit'
|
||||
});
|
||||
}).format(photoTimestamp(p));
|
||||
return p.size_formatted ? `${dateStr} · ${p.size_formatted}` : dateStr;
|
||||
}
|
||||
|
||||
|
||||
@@ -56,6 +56,60 @@ export function fileIconKindClass(iconName: string): string {
|
||||
return `file-icon--${fileIconKind(iconName)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Module-scope cache of `Intl.DateTimeFormat` instances, keyed by
|
||||
* `(locale, options signature)`. Constructing a formatter runs the full ICU
|
||||
* locale/pattern resolution (~50–200µs) while a `format()` call is ~1µs, and
|
||||
* {@link formatDate} runs roughly twice per row as large file lists render
|
||||
* and scroll — so a construct-per-call implementation (what
|
||||
* `toLocaleDateString(locale, options)` does under the hood) dominated list
|
||||
* fill. Entries are keyed by the locale actually requested — never frozen at
|
||||
* first use — so a runtime locale change just resolves a different entry.
|
||||
*/
|
||||
const dateTimeFormatCache = new Map<string, Intl.DateTimeFormat>();
|
||||
|
||||
// Entries built with `locale === undefined` snapshot the environment default
|
||||
// locale at construction time. `toLocaleDateString(undefined, …)` re-reads the
|
||||
// default on every call, so drop the cache if the default changes to keep the
|
||||
// cached path behaviourally identical.
|
||||
if (typeof window !== 'undefined') {
|
||||
window.addEventListener('languagechange', () => dateTimeFormatCache.clear());
|
||||
}
|
||||
|
||||
/**
|
||||
* Cached equivalent of `new Intl.DateTimeFormat(locale, options)`.
|
||||
*
|
||||
* `date.toLocaleDateString(locale, options)` / `toLocaleTimeString(…)` are
|
||||
* specified (ECMA-402) as building exactly this formatter per call — and
|
||||
* their component defaulting is a no-op once `options` names any date/time
|
||||
* component — so `dateTimeFormatFor(locale, options).format(date)` is
|
||||
* output-identical while paying construction once per (locale, options).
|
||||
*
|
||||
* The options signature uses `JSON.stringify`, so pass options as a hoisted
|
||||
* const or an inline literal (stable key order per callsite); a differently
|
||||
* ordered but equal object would only create a redundant entry, never a wrong
|
||||
* result.
|
||||
*/
|
||||
export function dateTimeFormatFor(
|
||||
locale: string | undefined,
|
||||
options?: Intl.DateTimeFormatOptions
|
||||
): Intl.DateTimeFormat {
|
||||
const key = `${locale ?? ''}|${options ? JSON.stringify(options) : ''}`;
|
||||
let fmt = dateTimeFormatCache.get(key);
|
||||
if (!fmt) {
|
||||
fmt = new Intl.DateTimeFormat(locale, options);
|
||||
dateTimeFormatCache.set(key, fmt);
|
||||
}
|
||||
return fmt;
|
||||
}
|
||||
|
||||
/** Options for {@link formatDate}, hoisted so every call shares one cache key. */
|
||||
const FORMAT_DATE_OPTS: Intl.DateTimeFormatOptions = {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric'
|
||||
};
|
||||
|
||||
/** Format a timestamp (epoch seconds/ms or ISO-8601 string) as a local date. */
|
||||
export function formatDate(value: number | string | null | undefined): string {
|
||||
if (value === null || value === undefined) return '';
|
||||
@@ -67,5 +121,5 @@ export function formatDate(value: number | string | null | undefined): string {
|
||||
d = new Date(value);
|
||||
}
|
||||
if (Number.isNaN(d.getTime())) return '';
|
||||
return d.toLocaleDateString(undefined, { year: 'numeric', month: 'short', day: 'numeric' });
|
||||
return dateTimeFormatFor(undefined, FORMAT_DATE_OPTS).format(d);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { dateTimeFormatFor, formatDate } from './display';
|
||||
|
||||
/**
|
||||
* Benchmark gate for the module-scope `Intl.DateTimeFormat` cache in
|
||||
* `display.ts` ({@link formatDate} / {@link dateTimeFormatFor}).
|
||||
*
|
||||
* Audit finding: `formatDate` built a fresh `Intl.DateTimeFormat` on every
|
||||
* call (`toLocaleDateString(undefined, opts)` constructs one internally), and
|
||||
* it runs ~twice per row while file lists render and scroll — a 10k-item
|
||||
* folder paid tens of thousands of ICU formatter constructions (~50–200µs
|
||||
* each) during list fill. The fix caches formatters in a Map keyed by
|
||||
* (locale, options signature).
|
||||
*
|
||||
* This gate asserts (1) the cached path is byte-identical to the
|
||||
* construct-per-call code it replaced, across dates, option shapes, and
|
||||
* locales (including an RTL one), and (2) it is decisively (≥3x) faster. If
|
||||
* the perf assertion fails, the cache is not delivering and the change
|
||||
* should be rolled back (it would be pure complexity).
|
||||
*/
|
||||
|
||||
/** The option shapes the app actually uses (display.ts + component callsites). */
|
||||
const DATE_OPTS: Intl.DateTimeFormatOptions = { year: 'numeric', month: 'short', day: 'numeric' };
|
||||
const MONTH_OPTS: Intl.DateTimeFormatOptions = { year: 'numeric', month: 'long' };
|
||||
const FULL_DATE_OPTS: Intl.DateTimeFormatOptions = {
|
||||
weekday: 'long',
|
||||
year: 'numeric',
|
||||
month: 'long',
|
||||
day: 'numeric'
|
||||
};
|
||||
const DATE_TIME_OPTS: Intl.DateTimeFormatOptions = {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric',
|
||||
hour: '2-digit',
|
||||
minute: '2-digit'
|
||||
};
|
||||
const TIME_OPTS: Intl.DateTimeFormatOptions = { hour: '2-digit', minute: '2-digit' };
|
||||
|
||||
/**
|
||||
* The pre-fix `formatDate`, verbatim: `toLocaleDateString` constructs a new
|
||||
* `Intl.DateTimeFormat` internally on every call. This is the uncached
|
||||
* reference the cached implementation must match and beat.
|
||||
*/
|
||||
function referenceFormatDate(value: number | string | null | undefined): string {
|
||||
if (value === null || value === undefined) return '';
|
||||
let d: Date;
|
||||
if (typeof value === 'number') {
|
||||
// Heuristic: seconds vs milliseconds.
|
||||
d = new Date(value < 1e12 ? value * 1000 : value);
|
||||
} else {
|
||||
d = new Date(value);
|
||||
}
|
||||
if (Number.isNaN(d.getTime())) return '';
|
||||
return d.toLocaleDateString(undefined, DATE_OPTS);
|
||||
}
|
||||
|
||||
/** ~20 inputs exercising the seconds/ms heuristic, ISO parsing, and edge cases. */
|
||||
const DATE_VALUES: Array<number | string | null | undefined> = [
|
||||
0, // epoch, seconds branch
|
||||
1, // seconds
|
||||
86_399, // seconds, last second of 1970-01-01 UTC
|
||||
951_782_400, // seconds, 2000-02-29 (leap day)
|
||||
1_700_000_000, // seconds
|
||||
999_999_999_999, // just under the 1e12 cutoff → seconds branch, far future
|
||||
1_000_000_000_000, // exactly 1e12 → milliseconds branch, 2001
|
||||
1_700_000_000_000, // milliseconds
|
||||
1_766_620_800_000, // milliseconds, 2025-12-25
|
||||
Date.UTC(1999, 11, 31, 23, 59, 59), // ms, century boundary
|
||||
Date.UTC(2038, 0, 19, 3, 14, 7), // ms, past the 32-bit epoch rollover
|
||||
'2024-01-15', // date-only ISO (parsed as UTC midnight)
|
||||
'2024-02-29T12:34:56Z', // leap day, UTC
|
||||
'1999-12-31T23:59:59.999Z',
|
||||
'2020-06-15T10:00:00+05:30', // non-UTC offset
|
||||
'2031-11-05T08:15:30-05:00',
|
||||
'0001-01-01T00:00:00Z', // extreme past
|
||||
'2024-07-04T00:00:00', // no offset (local time)
|
||||
'definitely not a date', // invalid → ''
|
||||
'', // invalid → ''
|
||||
null, // → ''
|
||||
undefined // → ''
|
||||
];
|
||||
|
||||
/** Locales the app ships (see SUPPORTED_LOCALES); 'ar' renders RTL. */
|
||||
const SAMPLE_LOCALES = ['en', 'es', 'ar', 'ja'] as const;
|
||||
|
||||
describe('cached Intl.DateTimeFormat (benchmark gate)', () => {
|
||||
it('formatDate output is identical to the uncached reference', () => {
|
||||
for (const value of DATE_VALUES) {
|
||||
expect(formatDate(value), `formatDate(${JSON.stringify(value)})`).toBe(
|
||||
referenceFormatDate(value)
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('cached formatters match per-call construction across locales and option shapes', () => {
|
||||
const dates = DATE_VALUES.filter((v): v is number | string => v !== null && v !== undefined)
|
||||
.map((v) => (typeof v === 'number' ? new Date(v < 1e12 ? v * 1000 : v) : new Date(v)))
|
||||
.filter((d) => !Number.isNaN(d.getTime()));
|
||||
expect(dates.length).toBeGreaterThanOrEqual(18);
|
||||
|
||||
for (const locale of SAMPLE_LOCALES) {
|
||||
for (const d of dates) {
|
||||
// Each toLocale*String call below is specified as constructing a
|
||||
// fresh Intl.DateTimeFormat — the uncached reference behaviour.
|
||||
expect(dateTimeFormatFor(locale, DATE_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString(locale, DATE_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor(locale, MONTH_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString(locale, MONTH_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor(locale, FULL_DATE_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString(locale, FULL_DATE_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor(locale, DATE_TIME_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString(locale, DATE_TIME_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor(locale, TIME_OPTS).format(d)).toBe(
|
||||
d.toLocaleTimeString(locale, TIME_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor(undefined, DATE_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString(undefined, DATE_OPTS)
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('reuses one instance per (locale, options) and never freezes the first locale', () => {
|
||||
// Same key → same instance (this is where the speedup comes from).
|
||||
expect(dateTimeFormatFor('es', DATE_OPTS)).toBe(dateTimeFormatFor('es', DATE_OPTS));
|
||||
expect(dateTimeFormatFor(undefined, DATE_OPTS)).toBe(dateTimeFormatFor(undefined, DATE_OPTS));
|
||||
// Different locale or options → different instance: a runtime locale
|
||||
// change must not keep formatting with the first locale seen.
|
||||
expect(dateTimeFormatFor('ar', DATE_OPTS)).not.toBe(dateTimeFormatFor('es', DATE_OPTS));
|
||||
expect(dateTimeFormatFor('es', TIME_OPTS)).not.toBe(dateTimeFormatFor('es', DATE_OPTS));
|
||||
const d = new Date(Date.UTC(2024, 4, 17, 12, 0, 0));
|
||||
expect(dateTimeFormatFor('ar', DATE_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString('ar', DATE_OPTS)
|
||||
);
|
||||
expect(dateTimeFormatFor('es', DATE_OPTS).format(d)).toBe(
|
||||
d.toLocaleDateString('es', DATE_OPTS)
|
||||
);
|
||||
});
|
||||
|
||||
it(
|
||||
'formats 20k dates ≥3x faster than per-call construction (perf gate)',
|
||||
{ timeout: 30_000 },
|
||||
() => {
|
||||
const N = 20_000;
|
||||
const base = Date.UTC(2020, 0, 1);
|
||||
// Deterministic spread of distinct ms timestamps across ~30 years.
|
||||
const values = Array.from({ length: N }, (_, i) => base + i * 47_777_777);
|
||||
|
||||
// Warm up both paths so JIT tiering and first-call construction sit
|
||||
// outside the measured windows. `sink` defeats dead-code elimination.
|
||||
let sink = 0;
|
||||
for (let i = 0; i < 500; i++) {
|
||||
sink += formatDate(values[i]).length;
|
||||
sink += referenceFormatDate(values[i]).length;
|
||||
}
|
||||
|
||||
const t0 = performance.now();
|
||||
for (const v of values) sink += formatDate(v).length;
|
||||
const cachedMs = performance.now() - t0;
|
||||
|
||||
const t1 = performance.now();
|
||||
for (const v of values) sink += referenceFormatDate(v).length;
|
||||
const uncachedMs = performance.now() - t1;
|
||||
|
||||
expect(sink).toBeGreaterThan(0);
|
||||
console.info(
|
||||
`formatDate x ${N}: cached ${cachedMs.toFixed(1)} ms vs construct-per-call ${uncachedMs.toFixed(1)} ms (${(uncachedMs / cachedMs).toFixed(1)}x)`
|
||||
);
|
||||
expect(cachedMs).toBeLessThan(uncachedMs / 3);
|
||||
}
|
||||
);
|
||||
});
|
||||
@@ -15,6 +15,7 @@
|
||||
import { t } from '$lib/i18n/index.svelte';
|
||||
import { ui } from '$lib/stores/ui.svelte';
|
||||
import { filterDotfiles } from '$lib/utils/dotfileFilter';
|
||||
import { dateTimeFormatFor } from '$lib/utils/display';
|
||||
import { isVideo, photoTimestamp } from '$lib/utils/media';
|
||||
|
||||
type Tab = 'moments' | 'places' | 'people';
|
||||
@@ -75,13 +76,13 @@
|
||||
function bucketLabel(d: Date): string {
|
||||
if (groupMode === 'year') return `${d.getFullYear()}`;
|
||||
if (groupMode === 'month')
|
||||
return d.toLocaleDateString(undefined, { year: 'numeric', month: 'long' });
|
||||
return d.toLocaleDateString(undefined, {
|
||||
return dateTimeFormatFor(undefined, { year: 'numeric', month: 'long' }).format(d);
|
||||
return dateTimeFormatFor(undefined, {
|
||||
weekday: 'long',
|
||||
year: 'numeric',
|
||||
month: 'long',
|
||||
day: 'numeric'
|
||||
});
|
||||
}).format(d);
|
||||
}
|
||||
|
||||
const groups = $derived.by(() => {
|
||||
|
||||
@@ -27,7 +27,7 @@
|
||||
import UserVignette from '$lib/components/UserVignette.svelte';
|
||||
import { t } from '$lib/i18n/index.svelte';
|
||||
import { ui } from '$lib/stores/ui.svelte';
|
||||
import { iconNameFromClass } from '$lib/utils/display';
|
||||
import { formatDate, iconNameFromClass } from '$lib/utils/display';
|
||||
|
||||
type GroupBy = 'items' | 'sharedWith';
|
||||
|
||||
@@ -158,9 +158,9 @@
|
||||
}
|
||||
function expiryLabel(iso: string | null | undefined): string {
|
||||
if (!iso) return t('share.noExpiry', 'No expiry');
|
||||
const d = new Date(iso);
|
||||
if (Number.isNaN(d.getTime())) return '';
|
||||
return d.toLocaleDateString(undefined, { year: 'numeric', month: 'short', day: 'numeric' });
|
||||
// Same semantics as before (`''` for unparseable dates), now via the
|
||||
// shared util so it reuses the cached Intl.DateTimeFormat.
|
||||
return formatDate(iso);
|
||||
}
|
||||
function isoToDate(iso: string | null | undefined): string {
|
||||
return iso ? String(iso).slice(0, 10) : '';
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
-- ════════════════════════════════════════════════════════════════════════════
|
||||
-- Web-UI listing keyset — expression indexes for the default "name" sort
|
||||
-- ════════════════════════════════════════════════════════════════════════════
|
||||
-- `list_resources_paged` (SPA files view) sorts case-insensitively on
|
||||
-- `LOWER(name)` with an id tie-breaker. The old query applied its keyset
|
||||
-- cursor OUTSIDE the folders/files UNION-ALL on computed columns, so every
|
||||
-- page rescanned and top-N-sorted the whole folder (28 ms/page on a
|
||||
-- 20k-entry folder). The query now pushes the cursor into each branch as a
|
||||
-- sargable row-value comparison `(LOWER(name), id) > ($str, $id)` — these
|
||||
-- two partial expression indexes let each branch answer that with one
|
||||
-- bounded, pre-ordered index-range read (1.3 ms/page, 19.5x;
|
||||
-- benches/LISTING-KEYSET.md).
|
||||
--
|
||||
-- Sibling of `idx_files_folder_name (folder_id, name)` (migration
|
||||
-- 20260917000000), which serves the byte-wise DAV ordering; the SPA orders
|
||||
-- by LOWER(name), which that index cannot provide.
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_files_folder_lname
|
||||
ON storage.files (folder_id, LOWER(name), id)
|
||||
WHERE NOT is_trashed;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_folders_parent_lname
|
||||
ON storage.folders (parent_id, LOWER(name), id)
|
||||
WHERE NOT is_trashed;
|
||||
@@ -651,7 +651,6 @@ impl CardDavAdapter {
|
||||
pub fn generate_contacts_response<W: Write>(
|
||||
writer: W,
|
||||
contacts: &[ContactDto],
|
||||
vcards: &[(String, String)], // (uid, vcard_data)
|
||||
report: &CardDavReportType,
|
||||
base_href: &str,
|
||||
) -> Result<()> {
|
||||
@@ -672,17 +671,9 @@ impl CardDavAdapter {
|
||||
|
||||
for contact in contacts {
|
||||
let href = format!("{}{}.vcf", base_href, contact.uid);
|
||||
let vcard = vcards
|
||||
.iter()
|
||||
.find(|(uid, _)| *uid == contact.uid)
|
||||
.map(|(_, data)| data.as_str())
|
||||
.unwrap_or("");
|
||||
// `write_contact_response` generates the vCard on demand when (and
|
||||
// only when) address-data is actually requested.
|
||||
Self::write_contact_response(&mut xml_writer, contact, &props, &href)?;
|
||||
// If address-data is requested, include vcard
|
||||
if props.iter().any(|p| p.name == "address-data") || props.is_empty() {
|
||||
// Already handled in write_contact_response
|
||||
}
|
||||
let _ = vcard; // suppress warning - used via contact_to_vcard fallback
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:multistatus")))?;
|
||||
@@ -868,20 +859,25 @@ impl CardDavAdapter {
|
||||
|
||||
/// Convert a ContactDto to vCard 3.0 format
|
||||
pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
// `write!` into a String is infallible; `let _ =` discards the Ok(()).
|
||||
// Formatting straight into the buffer avoids one temporary String per
|
||||
// vCard line compared to `push_str(&format!(…))`.
|
||||
use std::fmt::Write as _;
|
||||
|
||||
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
|
||||
|
||||
vcard.push_str(&format!("UID:{}\r\n", contact.uid));
|
||||
let _ = write!(vcard, "UID:{}\r\n", contact.uid);
|
||||
|
||||
if let (Some(last), Some(first)) = (&contact.last_name, &contact.first_name) {
|
||||
vcard.push_str(&format!("N:{};{};;;\r\n", last, first));
|
||||
let _ = write!(vcard, "N:{};{};;;\r\n", last, first);
|
||||
} else if let Some(last) = &contact.last_name {
|
||||
vcard.push_str(&format!("N:{};;;;\r\n", last));
|
||||
let _ = write!(vcard, "N:{};;;;\r\n", last);
|
||||
} else if let Some(first) = &contact.first_name {
|
||||
vcard.push_str(&format!("N:;{};;;\r\n", first));
|
||||
let _ = write!(vcard, "N:;{};;;\r\n", first);
|
||||
}
|
||||
|
||||
if let Some(fn_name) = &contact.full_name {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
let _ = write!(vcard, "FN:{}\r\n", fn_name);
|
||||
} else {
|
||||
// FN is mandatory in vCard 3.0
|
||||
let fn_name = format!(
|
||||
@@ -892,68 +888,68 @@ pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
.trim()
|
||||
.to_string();
|
||||
if !fn_name.is_empty() {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
let _ = write!(vcard, "FN:{}\r\n", fn_name);
|
||||
} else {
|
||||
vcard.push_str("FN:Unknown\r\n");
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(nickname) = &contact.nickname {
|
||||
vcard.push_str(&format!("NICKNAME:{}\r\n", nickname));
|
||||
let _ = write!(vcard, "NICKNAME:{}\r\n", nickname);
|
||||
}
|
||||
|
||||
for email in &contact.email {
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"EMAIL;TYPE={}:{}\r\n",
|
||||
email.r#type.to_uppercase(),
|
||||
email.email
|
||||
));
|
||||
);
|
||||
}
|
||||
|
||||
for phone in &contact.phone {
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"TEL;TYPE={}:{}\r\n",
|
||||
phone.r#type.to_uppercase(),
|
||||
phone.number
|
||||
));
|
||||
);
|
||||
}
|
||||
|
||||
for addr in &contact.address {
|
||||
let adr = format!(
|
||||
";;{};{};{};{};{}",
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"ADR;TYPE={}:;;{};{};{};{};{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
addr.street.as_deref().unwrap_or(""),
|
||||
addr.city.as_deref().unwrap_or(""),
|
||||
addr.state.as_deref().unwrap_or(""),
|
||||
addr.postal_code.as_deref().unwrap_or(""),
|
||||
addr.country.as_deref().unwrap_or(""),
|
||||
);
|
||||
vcard.push_str(&format!(
|
||||
"ADR;TYPE={}:{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
adr
|
||||
));
|
||||
}
|
||||
|
||||
if let Some(org) = &contact.organization {
|
||||
vcard.push_str(&format!("ORG:{}\r\n", org));
|
||||
let _ = write!(vcard, "ORG:{}\r\n", org);
|
||||
}
|
||||
if let Some(title) = &contact.title {
|
||||
vcard.push_str(&format!("TITLE:{}\r\n", title));
|
||||
let _ = write!(vcard, "TITLE:{}\r\n", title);
|
||||
}
|
||||
if let Some(notes) = &contact.notes {
|
||||
vcard.push_str(&format!("NOTE:{}\r\n", notes.replace('\n', "\\n")));
|
||||
let _ = write!(vcard, "NOTE:{}\r\n", notes.replace('\n', "\\n"));
|
||||
}
|
||||
if let Some(bday) = &contact.birthday {
|
||||
vcard.push_str(&format!("BDAY:{}\r\n", bday.format("%Y-%m-%d")));
|
||||
let _ = write!(vcard, "BDAY:{}\r\n", bday.format("%Y-%m-%d"));
|
||||
}
|
||||
if let Some(photo) = &contact.photo_url {
|
||||
vcard.push_str(&format!("PHOTO;VALUE=URI:{}\r\n", photo));
|
||||
let _ = write!(vcard, "PHOTO;VALUE=URI:{}\r\n", photo);
|
||||
}
|
||||
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"REV:{}\r\n",
|
||||
contact.updated_at.format("%Y%m%dT%H%M%SZ")
|
||||
));
|
||||
);
|
||||
vcard.push_str("END:VCARD\r\n");
|
||||
|
||||
vcard
|
||||
|
||||
@@ -484,10 +484,6 @@ mod tests {
|
||||
#[test]
|
||||
fn test_generate_contacts_response() {
|
||||
let contacts = vec![sample_contact()];
|
||||
let vcards = vec![(
|
||||
"contact-001".to_string(),
|
||||
contact_to_vcard(&sample_contact()),
|
||||
)];
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: vec![
|
||||
QualifiedName {
|
||||
@@ -505,7 +501,6 @@ mod tests {
|
||||
let result = CardDavAdapter::generate_contacts_response(
|
||||
&mut output,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
"/carddav/ab-001",
|
||||
);
|
||||
@@ -528,14 +523,12 @@ mod tests {
|
||||
#[test]
|
||||
fn test_generate_empty_contacts_response() {
|
||||
let contacts: Vec<ContactDto> = vec![];
|
||||
let vcards: Vec<(String, String)> = vec![];
|
||||
let report = CardDavReportType::AddressbookQuery { props: vec![] };
|
||||
|
||||
let mut output = Vec::new();
|
||||
let result = CardDavAdapter::generate_contacts_response(
|
||||
&mut output,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
"/carddav/ab-001",
|
||||
);
|
||||
|
||||
@@ -8,6 +8,175 @@
|
||||
//! then fall back to the file extension when the MIME is generic
|
||||
//! (`application/octet-stream` or empty).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fmt::Write as _;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
// ─── Arc<str> interning for closed-set display values ────────────────
|
||||
//
|
||||
// `FileDto` / `FolderDto` store their display fields as `Arc<str>` so DTO
|
||||
// clones are O(1). But `Arc::<str>::from(&str)` always allocates + copies,
|
||||
// so building the DTO paid 3-4 heap allocations per row even though the
|
||||
// value space is a small closed set. Interning turns each conversion into
|
||||
// a HashMap lookup + refcount bump.
|
||||
|
||||
/// Every `&'static str` that [`icon_class_for`], [`icon_special_class_for`]
|
||||
/// and [`category_for`] can return, plus the folder-DTO constants.
|
||||
///
|
||||
/// Keep this table in sync when adding a value to those functions — a
|
||||
/// missing entry is not a bug (callers fall back to `Arc::from`, same
|
||||
/// bytes, one extra allocation), just a lost optimization.
|
||||
static DISPLAY_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const CLOSED_SET: &[&str] = &[
|
||||
// icon_class_for
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file-word",
|
||||
"fas fa-file-excel",
|
||||
"fas fa-file-powerpoint",
|
||||
"fas fa-file-archive",
|
||||
"fas fa-file-code",
|
||||
"fas fa-hdd",
|
||||
"fas fa-file-image",
|
||||
"fas fa-file-video",
|
||||
"fas fa-file-audio",
|
||||
"fas fa-file-alt",
|
||||
"fas fa-terminal",
|
||||
"fas fa-file",
|
||||
// icon_special_class_for
|
||||
"pdf-icon",
|
||||
"doc-icon",
|
||||
"spreadsheet-icon",
|
||||
"presentation-icon",
|
||||
"archive-icon",
|
||||
"code-icon json-icon",
|
||||
"code-icon js-icon",
|
||||
"code-icon ts-icon",
|
||||
"code-icon html-icon",
|
||||
"code-icon sql-icon",
|
||||
"code-icon config-icon",
|
||||
"code-icon php-icon",
|
||||
"script-icon",
|
||||
"installer-icon",
|
||||
"image-icon",
|
||||
"video-icon",
|
||||
"audio-icon",
|
||||
"code-icon py-icon",
|
||||
"code-icon rust-icon",
|
||||
"code-icon",
|
||||
"code-icon go-icon",
|
||||
"code-icon ruby-icon",
|
||||
"code-icon md-icon",
|
||||
"code-icon css-icon",
|
||||
"code-icon java-icon",
|
||||
"code-icon c-icon",
|
||||
"code-icon cs-icon",
|
||||
"code-icon swift-icon",
|
||||
"",
|
||||
// category_for
|
||||
"PDF",
|
||||
"Document",
|
||||
"Spreadsheet",
|
||||
"Presentation",
|
||||
"Archive",
|
||||
"Code",
|
||||
"Installer",
|
||||
"Image",
|
||||
"Video",
|
||||
"Audio",
|
||||
"Markdown",
|
||||
"Text",
|
||||
// FolderDto constants
|
||||
"fas fa-folder",
|
||||
"folder-icon",
|
||||
"Folder",
|
||||
];
|
||||
CLOSED_SET.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for a display value from the closed sets
|
||||
/// above (icon class, icon special class, category). Lookup + refcount
|
||||
/// bump instead of alloc + copy; unknown values (future additions not
|
||||
/// yet in the table) fall back to `Arc::from` with identical bytes.
|
||||
pub fn intern_display(s: &'static str) -> Arc<str> {
|
||||
DISPLAY_INTERN
|
||||
.get(s)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(s))
|
||||
}
|
||||
|
||||
/// The MIME types that dominate real storage rows. Exotic types fall back
|
||||
/// to a per-row `Arc::from` — correctness is unaffected, only the alloc is.
|
||||
static MIME_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const COMMON_MIMES: &[&str] = &[
|
||||
"",
|
||||
"directory",
|
||||
"application/octet-stream",
|
||||
// Images
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/gif",
|
||||
"image/webp",
|
||||
"image/svg+xml",
|
||||
"image/heic",
|
||||
"image/heif",
|
||||
"image/avif",
|
||||
"image/bmp",
|
||||
"image/tiff",
|
||||
"image/x-icon",
|
||||
// Video
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"video/webm",
|
||||
"video/x-matroska",
|
||||
"video/x-msvideo",
|
||||
// Audio
|
||||
"audio/mpeg",
|
||||
"audio/mp4",
|
||||
"audio/ogg",
|
||||
"audio/flac",
|
||||
"audio/wav",
|
||||
"audio/x-wav",
|
||||
"audio/aac",
|
||||
// Documents
|
||||
"application/pdf",
|
||||
"application/msword",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-powerpoint",
|
||||
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
"application/vnd.oasis.opendocument.text",
|
||||
"application/vnd.oasis.opendocument.spreadsheet",
|
||||
// Text / code
|
||||
"text/plain",
|
||||
"text/csv",
|
||||
"text/html",
|
||||
"text/css",
|
||||
"text/markdown",
|
||||
"text/xml",
|
||||
"application/json",
|
||||
"application/javascript",
|
||||
"application/xml",
|
||||
"application/x-yaml",
|
||||
// Archives
|
||||
"application/zip",
|
||||
"application/gzip",
|
||||
"application/x-tar",
|
||||
"application/x-7z-compressed",
|
||||
"application/x-rar-compressed",
|
||||
];
|
||||
COMMON_MIMES.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for the given MIME type. Common types hit
|
||||
/// the intern table (refcount bump); exotic ones allocate as before.
|
||||
pub fn intern_mime(mime: &str) -> Arc<str> {
|
||||
MIME_INTERN
|
||||
.get(mime)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(mime))
|
||||
}
|
||||
|
||||
// ─── Private: extract lowercase extension from a filename ────────────
|
||||
fn ext_of(name: &str) -> Option<&str> {
|
||||
let name = name.rsplit('/').next().unwrap_or(name); // strip path
|
||||
@@ -388,11 +557,21 @@ pub fn format_file_size(bytes: u64) -> String {
|
||||
|
||||
let value = bytes as f64 / K.powi(i as i32);
|
||||
|
||||
// Two decimal places, then strip trailing zeros (matches JS parseFloat behaviour)
|
||||
let formatted = format!("{:.2}", value);
|
||||
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
|
||||
|
||||
format!("{} {}", formatted, SIZES[i])
|
||||
// Single buffer: write the 2-decimal value, strip trailing zeros in
|
||||
// place (matches JS parseFloat behaviour), then append the unit.
|
||||
// 16 chars covers the worst case ("16777216 TB" for u64::MAX,
|
||||
// "1023.99 Bytes" for the longest unit), so no realloc occurs.
|
||||
let mut out = String::with_capacity(16);
|
||||
let _ = write!(out, "{:.2}", value);
|
||||
while out.ends_with('0') {
|
||||
out.pop();
|
||||
}
|
||||
if out.ends_with('.') {
|
||||
out.pop();
|
||||
}
|
||||
out.push(' ');
|
||||
out.push_str(SIZES[i]);
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -506,6 +685,50 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Every value the closed-set display functions can return must hit
|
||||
/// the intern table (same bytes, shared allocation) — a miss is only
|
||||
/// a lost optimization, but this test keeps the table in sync.
|
||||
#[test]
|
||||
fn test_intern_display_covers_closed_sets_and_shares_storage() {
|
||||
for s in [
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file",
|
||||
"fas fa-terminal",
|
||||
"fas fa-folder",
|
||||
"code-icon rust-icon",
|
||||
"folder-icon",
|
||||
"",
|
||||
"PDF",
|
||||
"Folder",
|
||||
"Document",
|
||||
"Markdown",
|
||||
] {
|
||||
let a = intern_display(s);
|
||||
let b = intern_display(s);
|
||||
assert_eq!(&*a, s, "interned bytes must be identical");
|
||||
assert!(
|
||||
Arc::ptr_eq(&a, &b),
|
||||
"closed-set value {s:?} must come from the intern table"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_intern_mime_common_hits_table_exotic_falls_back() {
|
||||
let a = intern_mime("image/jpeg");
|
||||
let b = intern_mime("image/jpeg");
|
||||
assert_eq!(&*a, "image/jpeg");
|
||||
assert!(Arc::ptr_eq(&a, &b), "common MIME must be interned");
|
||||
|
||||
let exotic = intern_mime("chemical/x-pdb");
|
||||
assert_eq!(&*exotic, "chemical/x-pdb");
|
||||
let exotic2 = intern_mime("chemical/x-pdb");
|
||||
assert!(
|
||||
!Arc::ptr_eq(&exotic, &exotic2),
|
||||
"exotic MIME falls back to a fresh Arc"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ext_of() {
|
||||
assert_eq!(ext_of("file.txt"), Some("txt"));
|
||||
|
||||
@@ -6,7 +6,8 @@ use utoipa::ToSchema;
|
||||
use uuid::Uuid;
|
||||
|
||||
use super::display_helpers::{
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for,
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for, intern_display,
|
||||
intern_mime,
|
||||
};
|
||||
|
||||
/// DTO for file responses
|
||||
@@ -101,11 +102,15 @@ impl From<File> for FileDto {
|
||||
// for id, name, path, folder_id (previously 4× .to_string()).
|
||||
let parts = file.into_parts();
|
||||
|
||||
let icon_class = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class = Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = Arc::from(category_for(&parts.name, &parts.mime_type));
|
||||
// Display fields come from closed static tables and MIME values
|
||||
// repeat massively across rows — intern instead of allocating a
|
||||
// fresh Arc<str> per row (`Arc::from(&str)` always allocs+copies).
|
||||
let icon_class = intern_display(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class =
|
||||
intern_display(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = intern_display(category_for(&parts.name, &parts.mime_type));
|
||||
let size_formatted = format_file_size(parts.size);
|
||||
let mime_type = Arc::from(parts.mime_type.as_str());
|
||||
let mime_type = intern_mime(&parts.mime_type);
|
||||
|
||||
Self {
|
||||
id: parts.id,
|
||||
@@ -169,13 +174,13 @@ impl FileDto {
|
||||
name: "stub-file".to_string(),
|
||||
path: "/stub/path".to_string(),
|
||||
size: 0,
|
||||
mime_type: Arc::from("application/octet-stream"),
|
||||
mime_type: intern_mime("application/octet-stream"),
|
||||
folder_id: None,
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
icon_class: Arc::from("fas fa-file"),
|
||||
icon_special_class: Arc::from(""),
|
||||
category: Arc::from("Document"),
|
||||
icon_class: intern_display("fas fa-file"),
|
||||
icon_special_class: intern_display(""),
|
||||
category: intern_display("Document"),
|
||||
size_formatted: "0 Bytes".to_string(),
|
||||
content_hash: String::new(),
|
||||
etag: String::new(),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::application::dtos::cursor::{CursorListResponse, CursorQuery, PageCursor};
|
||||
use crate::application::dtos::display_helpers::intern_display;
|
||||
use crate::application::dtos::grant_dto::{ResourceContentDto, ResourceTypeDto};
|
||||
use crate::domain::entities::folder::Folder;
|
||||
use crate::domain::services::authorization::ResourceKind;
|
||||
@@ -99,24 +100,33 @@ pub struct FolderDto {
|
||||
|
||||
impl From<Folder> for FolderDto {
|
||||
fn from(folder: Folder) -> Self {
|
||||
let is_root = folder.parent_id().is_none();
|
||||
let etag = folder.etag().to_string();
|
||||
// Consume the entity by moving all fields — zero heap allocations
|
||||
// for id, name, path, parent_id (previously 3-4× .to_string()).
|
||||
let parts = folder.into_parts();
|
||||
|
||||
let is_root = parts.parent_id.is_none();
|
||||
// Single-allocation ETag straight from the owned parts. The old
|
||||
// shape (`folder.etag().to_string()`) built the String and then
|
||||
// cloned it — a pure double-alloc.
|
||||
let etag = Folder::compute_etag(&parts.id, parts.tree_modified_at);
|
||||
|
||||
Self {
|
||||
id: folder.id().to_string(),
|
||||
name: folder.name().to_string(),
|
||||
path: folder.path_string().to_string(),
|
||||
parent_id: folder.parent_id().map(String::from),
|
||||
drive_id: folder.drive_id(),
|
||||
created_at: folder.created_at(),
|
||||
modified_at: folder.modified_at(),
|
||||
id: parts.id,
|
||||
name: parts.name,
|
||||
path: parts.path_string,
|
||||
parent_id: parts.parent_id,
|
||||
drive_id: parts.drive_id,
|
||||
created_at: parts.created_at,
|
||||
modified_at: parts.modified_at,
|
||||
is_root,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
// Constant display fields: refcount bump on interned statics
|
||||
// instead of 3 fresh Arc allocations per row.
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag,
|
||||
created_by: folder.created_by(),
|
||||
updated_by: folder.updated_by(),
|
||||
created_by: parts.created_by,
|
||||
updated_by: parts.updated_by,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -163,9 +173,9 @@ impl FolderDto {
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
is_root: true,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag: String::new(),
|
||||
created_by: None,
|
||||
updated_by: None,
|
||||
|
||||
@@ -77,6 +77,31 @@ pub trait FolderUseCase: Send + Sync + 'static {
|
||||
pagination: &crate::application::dtos::pagination::PaginationRequestDto,
|
||||
) -> Result<crate::application::dtos::pagination::PaginatedResponseDto<FolderDto>, DomainError>;
|
||||
|
||||
/// Keyset-paged sub-folder listing in name order, scoped to a caller —
|
||||
/// `name > after_name LIMIT limit`, `has_next = len() == limit`.
|
||||
///
|
||||
/// Used by streaming WebDAV/NC PROPFIND: O(page) per page off the
|
||||
/// `idx_folders_unique_name` index instead of the quadratic
|
||||
/// `COUNT(*) OVER() … LIMIT/OFFSET` walk (benches/FOLDER-KEYSET.md).
|
||||
///
|
||||
/// The default implementation falls back to `list_folders_with_perms`
|
||||
/// + in-memory slice so stubs and mocks compile without changes.
|
||||
async fn list_folders_batch_with_perms(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
caller_id: Uuid,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<FolderDto>, DomainError> {
|
||||
let mut all = self.list_folders_with_perms(parent_id, caller_id).await?;
|
||||
all.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name.as_str() > a))
|
||||
.take(limit)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Renames a folder (ownership verified against caller_id)
|
||||
async fn rename_folder_with_perms(
|
||||
&self,
|
||||
|
||||
@@ -304,12 +304,45 @@ impl AppPasswordService {
|
||||
let cache_key: [u8; 32] =
|
||||
blake3::hash(format!("{}:{}", username, password).as_bytes()).into();
|
||||
|
||||
// ── 2. Cache hit → return immediately ────────────────────────
|
||||
if let Some(cached) = self.auth_cache.get(&cache_key).await {
|
||||
return Ok((cached.user_id, cached.username, cached.email, cached.role));
|
||||
// ── 2. Single-flight cache lookup ─────────────────────────────
|
||||
// Concurrent misses on the same credential coalesce into ONE
|
||||
// full verification: DAV sync clients hold 4-8 parallel
|
||||
// connections, so an expiring cache entry used to fan out into
|
||||
// K simultaneous Argon2id runs (~100-300 ms CPU + 64 MiB RAM
|
||||
// apiece) every TTL — a recurring p99 spike on every DAV
|
||||
// surface (8 -> 1 verifications, benches/AUTH-HERD.md).
|
||||
// `try_get_with` caches only `Ok` results, so failed
|
||||
// verifications are still never cached, preserving the full
|
||||
// Argon2id cost as a brute-force deterrent.
|
||||
let result = self
|
||||
.auth_cache
|
||||
.try_get_with(
|
||||
cache_key,
|
||||
self.verify_basic_auth_uncached(username, password),
|
||||
)
|
||||
.await
|
||||
.map_err(
|
||||
|e: std::sync::Arc<DomainError>| match std::sync::Arc::try_unwrap(e) {
|
||||
Ok(err) => err,
|
||||
// Another coalesced waiter still holds the Arc — rebuild
|
||||
// an equivalent error (the source chain isn't clonable).
|
||||
Err(shared) => {
|
||||
DomainError::new(shared.kind, shared.entity_type, shared.message.clone())
|
||||
}
|
||||
},
|
||||
)?;
|
||||
Ok((result.user_id, result.username, result.email, result.role))
|
||||
}
|
||||
|
||||
// ── 3. Cache miss → full verification ────────────────────────
|
||||
/// The uncached Basic Auth slow path: user lookup, prefix-scoped
|
||||
/// candidate fetch, Argon2id verification. Runs at most once per
|
||||
/// credential per TTL — `verify_basic_auth` coalesces concurrent
|
||||
/// callers onto a single in-flight instance of this future.
|
||||
async fn verify_basic_auth_uncached(
|
||||
&self,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<CachedBasicAuthResult, DomainError> {
|
||||
let user = self
|
||||
.user_repo
|
||||
.get_user_by_username(username)
|
||||
@@ -363,15 +396,14 @@ impl AppPasswordService {
|
||||
{
|
||||
let _ = self.repo.touch_last_used(ap.id).await;
|
||||
|
||||
let result = CachedBasicAuthResult {
|
||||
// Caching happens in `verify_basic_auth`: `try_get_with`
|
||||
// stores this value under the blake3 key on return.
|
||||
return Ok(CachedBasicAuthResult {
|
||||
user_id: user.id(),
|
||||
username: user.username().unwrap_or("").to_string(),
|
||||
email: user.email().to_string(),
|
||||
role: user.role().to_string(),
|
||||
};
|
||||
|
||||
self.auth_cache.insert(cache_key, result.clone()).await;
|
||||
return Ok((result.user_id, result.username, result.email, result.role));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -429,6 +429,62 @@ impl FolderUseCase for FolderService {
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
/// Keyset-paged sub-folder listing (name order), caller-scoped.
|
||||
///
|
||||
/// AuthZ mirrors `list_folders_paginated_with_perms`: one
|
||||
/// `authz.require(Read)` on the parent per batch; root scope goes
|
||||
/// through the caller's drive-membership listing.
|
||||
async fn list_folders_batch_with_perms(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
caller_id: Uuid,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<FolderDto>, DomainError> {
|
||||
match parent_id {
|
||||
Some(pid) => {
|
||||
self.authz
|
||||
.require(
|
||||
Subject::User(caller_id),
|
||||
Permission::Read,
|
||||
Self::folder_resource(pid)?,
|
||||
)
|
||||
.await?;
|
||||
let folders = self
|
||||
.folder_storage
|
||||
.list_folders_batch(parent_id, after_name, limit)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"FolderStorage",
|
||||
format!("Failed to batch-list folders in parent {pid}: {e}"),
|
||||
)
|
||||
})?;
|
||||
Ok(folders.into_iter().map(FolderDto::from).collect())
|
||||
}
|
||||
None => {
|
||||
// Root scope: one row per readable drive — a handful.
|
||||
let mut all = self
|
||||
.folder_storage
|
||||
.list_root_folders_for_caller(caller_id)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"FolderStorage",
|
||||
format!("Failed to batch-list root folders for '{caller_id}': {e}"),
|
||||
)
|
||||
})?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.map(FolderDto::from)
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Lists folders with pagination, scoped to a specific owner.
|
||||
async fn list_folders_paginated_with_perms(
|
||||
&self,
|
||||
|
||||
@@ -67,9 +67,80 @@ pub struct SearchService {
|
||||
/// Lock-free concurrent cache with automatic TTL and LRU eviction (moka).
|
||||
/// Values are `Arc<SearchResultsDto>` so cache insert/hit is a single
|
||||
/// atomic ref-count increment (~1 ns) instead of cloning thousands of Strings.
|
||||
///
|
||||
/// **Byte-bounded**, not entry-bounded: entries are weighed by
|
||||
/// [`search_results_entry_weight`] and `max_capacity` is a byte budget.
|
||||
/// Keys span user × query × offset × limit, and each page holds up to 500
|
||||
/// enriched rows (~500–900 B of owned Strings each) — an entry-count bound
|
||||
/// let hundreds of MB of result pages accumulate invisibly.
|
||||
search_cache: moka::future::Cache<u64, Arc<SearchResultsDto>>,
|
||||
}
|
||||
|
||||
// ─── Search-results cache (byte-bounded) ─────────────────────────────────
|
||||
|
||||
/// Approximate heap bytes retained by one cached search page.
|
||||
///
|
||||
/// With a `weigher` installed, moka's `max_capacity` is the sum of entry
|
||||
/// *weights*, so this converts the cache bound from "number of entries" to
|
||||
/// real bytes: the length of every owned `String` in each file/folder row,
|
||||
/// plus a fixed per-row and per-entry overhead for struct fields, the 24-B
|
||||
/// `String` headers, `Vec` slots and allocator slop. Same pattern as the
|
||||
/// file-content cache and the dedup manifest cache.
|
||||
///
|
||||
/// `pub` so `examples/bench_search_cache_mem.rs` can recompute retained
|
||||
/// bytes with the exact production formula.
|
||||
pub fn search_results_entry_weight(_key: &u64, value: &Arc<SearchResultsDto>) -> u32 {
|
||||
/// Fixed per-row overhead: struct scalars + one 24-B header per `String`
|
||||
/// field (12 on a file row, 4 on a folder row) + `Vec` slot + allocator
|
||||
/// slop. Deliberately a round upper-ish estimate — under-weighing is the
|
||||
/// failure mode that re-opens the memory hole.
|
||||
const ROW_OVERHEAD: usize = 200;
|
||||
/// Fixed per-entry overhead: `Arc` + `SearchResultsDto` scalars + `Vec`
|
||||
/// headers + moka's own bookkeeping per entry.
|
||||
const ENTRY_OVERHEAD: usize = 256;
|
||||
|
||||
fn opt_len(s: &Option<String>) -> usize {
|
||||
s.as_deref().map_or(0, str::len)
|
||||
}
|
||||
|
||||
let mut bytes = ENTRY_OVERHEAD + value.sort_by.len();
|
||||
for f in &value.files {
|
||||
bytes += ROW_OVERHEAD
|
||||
+ f.id.len()
|
||||
+ f.name.len()
|
||||
+ f.path.len()
|
||||
+ f.mime_type.len()
|
||||
+ opt_len(&f.folder_id)
|
||||
+ f.size_formatted.len()
|
||||
+ f.icon_class.len()
|
||||
+ f.icon_special_class.len()
|
||||
+ f.category.len()
|
||||
+ f.blob_hash.len()
|
||||
+ opt_len(&f.snippet)
|
||||
+ opt_len(&f.match_source);
|
||||
}
|
||||
for d in &value.folders {
|
||||
bytes += ROW_OVERHEAD + d.id.len() + d.name.len() + d.path.len() + opt_len(&d.parent_id);
|
||||
}
|
||||
bytes.min(u32::MAX as usize) as u32
|
||||
}
|
||||
|
||||
/// Build the search-results cache exactly as production wires it: a byte
|
||||
/// budget enforced through [`search_results_entry_weight`], plus TTL.
|
||||
///
|
||||
/// Shared with `examples/bench_search_cache_mem.rs` so the benchmark
|
||||
/// measures the identical cache configuration that serves requests.
|
||||
pub fn build_search_results_cache(
|
||||
cache_ttl_secs: u64,
|
||||
max_bytes: u64,
|
||||
) -> moka::future::Cache<u64, Arc<SearchResultsDto>> {
|
||||
moka::future::Cache::builder()
|
||||
.max_capacity(max_bytes)
|
||||
.weigher(search_results_entry_weight)
|
||||
.time_to_live(Duration::from_secs(cache_ttl_secs))
|
||||
.build()
|
||||
}
|
||||
|
||||
// ─── Utility functions (pure, no self — computed on the server) ─────────
|
||||
|
||||
/// Compute relevance score (0–100) for a name against a query.
|
||||
@@ -160,6 +231,10 @@ fn get_category(name: &str, mime: &str) -> String {
|
||||
impl SearchService {
|
||||
/**
|
||||
* Creates a new instance of the search service.
|
||||
*
|
||||
* `max_cache_bytes` is the byte budget for the results cache (weigher-
|
||||
* bounded, see [`search_results_entry_weight`]) — it replaced the old
|
||||
* entry-count capacity, which was blind to how big each cached page is.
|
||||
*/
|
||||
pub fn new(
|
||||
file_repository: Arc<FileBlobReadRepository>,
|
||||
@@ -168,12 +243,9 @@ impl SearchService {
|
||||
authorization: Option<Arc<crate::infrastructure::services::pg_acl_engine::PgAclEngine>>,
|
||||
drive_repo: Option<Arc<dyn crate::domain::repositories::drive_repository::DriveRepository>>,
|
||||
cache_ttl: u64,
|
||||
max_cache_size: usize,
|
||||
max_cache_bytes: u64,
|
||||
) -> Self {
|
||||
let search_cache = moka::future::Cache::builder()
|
||||
.max_capacity(max_cache_size as u64)
|
||||
.time_to_live(Duration::from_secs(cache_ttl))
|
||||
.build();
|
||||
let search_cache = build_search_results_cache(cache_ttl, max_cache_bytes);
|
||||
|
||||
Self {
|
||||
file_repository,
|
||||
@@ -815,6 +887,88 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entry_weight_counts_every_owned_string_plus_overheads() {
|
||||
// Empty page: entry overhead + sort_by ("relevance" = 9 bytes).
|
||||
let empty = Arc::new(SearchResultsDto::empty());
|
||||
let base = search_results_entry_weight(&0, &empty) as usize;
|
||||
assert_eq!(base, 256 + 9);
|
||||
|
||||
// One file row: base + row overhead + its owned string bytes
|
||||
// (id 7 + name 7 + path 8 + mime 10; the rest are empty/None).
|
||||
let one_file = Arc::new(SearchResultsDto::new(
|
||||
vec![dto("abc.txt", 50, 10, 1)],
|
||||
Vec::new(),
|
||||
100,
|
||||
0,
|
||||
Some(1),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
));
|
||||
let w = search_results_entry_weight(&0, &one_file) as usize;
|
||||
assert_eq!(w, base + 200 + 7 + 7 + 8 + 10);
|
||||
|
||||
// Folder rows weigh too (id 2 + name 4 + path 5 + parent 6 = 17).
|
||||
let one_folder = Arc::new(SearchResultsDto::new(
|
||||
Vec::new(),
|
||||
vec![SearchFolderResultDto {
|
||||
id: "f1".to_string(),
|
||||
name: "docs".to_string(),
|
||||
path: "/docs".to_string(),
|
||||
parent_id: Some("parent".to_string()),
|
||||
drive_id: Uuid::nil(),
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
is_root: false,
|
||||
relevance_score: 50,
|
||||
}],
|
||||
100,
|
||||
0,
|
||||
Some(1),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
));
|
||||
let w = search_results_entry_weight(&0, &one_folder) as usize;
|
||||
assert_eq!(w, base + 200 + 2 + 4 + 5 + 6);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cache_evicts_down_to_the_byte_budget() {
|
||||
// Budget fits ~2 of these entries; inserting 20 must never let the
|
||||
// weighted size settle above the budget.
|
||||
let entry = |i: usize| {
|
||||
Arc::new(SearchResultsDto::new(
|
||||
(0..50)
|
||||
.map(|r| dto(&format!("file_{i}_{r}_{}", "x".repeat(100)), 50, 1, 1))
|
||||
.collect(),
|
||||
Vec::new(),
|
||||
50,
|
||||
0,
|
||||
Some(50),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
))
|
||||
};
|
||||
let per_entry = search_results_entry_weight(&0, &entry(0)) as u64;
|
||||
let budget = per_entry * 2 + per_entry / 2;
|
||||
|
||||
let cache = build_search_results_cache(300, budget);
|
||||
for i in 0..20u64 {
|
||||
cache.insert(i, entry(i as usize)).await;
|
||||
}
|
||||
cache.run_pending_tasks().await;
|
||||
|
||||
let retained: u64 = cache
|
||||
.iter()
|
||||
.map(|(k, v)| search_results_entry_weight(&k, &v) as u64)
|
||||
.sum();
|
||||
assert!(
|
||||
retained <= budget,
|
||||
"retained {retained} B exceeds budget {budget} B"
|
||||
);
|
||||
assert!(cache.entry_count() <= 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merged_files_resort_by_relevance_and_by_column() {
|
||||
let mut files = vec![
|
||||
|
||||
@@ -1250,6 +1250,33 @@ impl Default for ContentSearchConfig {
|
||||
}
|
||||
}
|
||||
|
||||
/// Search-results cache configuration — the per-user results-page cache
|
||||
/// inside `SearchService`, not the Tantivy content index above.
|
||||
///
|
||||
/// The cache is **byte-bounded**: each entry is weighed by the approximate
|
||||
/// heap size of its result page (see `search_results_entry_weight`) and moka
|
||||
/// evicts once the summed weight exceeds `max_bytes` — the same byte-budget
|
||||
/// pattern the file-content cache and the dedup manifest cache use. This
|
||||
/// replaced an entry-count capacity: with cache keys spanning
|
||||
/// user × query × offset × limit and up to 500 enriched rows per page, an
|
||||
/// entry count said nothing about resident memory (1000 entries could pin
|
||||
/// ~300 MB for the TTL). No entry-count knob is kept — bytes are the only
|
||||
/// dimension that matters here.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct SearchCacheConfig {
|
||||
/// Byte budget for cached search-result pages. Default: 32 MiB.
|
||||
/// Env: `OXICLOUD_SEARCH_CACHE_MAX_BYTES`.
|
||||
pub max_bytes: u64,
|
||||
}
|
||||
|
||||
impl Default for SearchCacheConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_bytes: 32 * 1024 * 1024,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// WASM plugin runtime configuration (M0 walking skeleton).
|
||||
///
|
||||
/// The runtime is doubly gated: it is only compiled when the `plugins` cargo
|
||||
@@ -1375,6 +1402,8 @@ pub struct AppConfig {
|
||||
pub i18n: I18nConfig,
|
||||
/// Content-search configuration (embedded full-text index)
|
||||
pub content_search: ContentSearchConfig,
|
||||
/// Search-results cache configuration (byte-bounded moka cache)
|
||||
pub search_cache: SearchCacheConfig,
|
||||
/// WASM plugin runtime configuration
|
||||
pub plugins: PluginConfig,
|
||||
/// Face-recognition (People) model configuration
|
||||
@@ -1431,6 +1460,7 @@ impl Default for AppConfig {
|
||||
magic_link: MagicLinkConfig::default(),
|
||||
i18n: I18nConfig::default(),
|
||||
content_search: ContentSearchConfig::default(),
|
||||
search_cache: SearchCacheConfig::default(),
|
||||
plugins: PluginConfig::default(),
|
||||
faces: FacesConfig::default(),
|
||||
}
|
||||
@@ -1934,6 +1964,13 @@ impl AppConfig {
|
||||
config.content_search.max_text_bytes = val;
|
||||
}
|
||||
|
||||
// Search-results cache (byte-bounded)
|
||||
if let Ok(v) = env::var("OXICLOUD_SEARCH_CACHE_MAX_BYTES").map(|v| v.parse::<u64>())
|
||||
&& let Ok(val) = v
|
||||
{
|
||||
config.search_cache.max_bytes = val;
|
||||
}
|
||||
|
||||
// WASM plugin runtime
|
||||
if let Ok(v) = env::var("OXICLOUD_ENABLE_PLUGINS").map(|v| v.parse::<bool>())
|
||||
&& let Ok(val) = v
|
||||
|
||||
+5
-1
@@ -657,7 +657,11 @@ impl AppServiceFactory {
|
||||
Some(authz.clone()),
|
||||
Some(drive_repo.clone()),
|
||||
300, // Cache TTL in seconds (5 minutes)
|
||||
1000, // Maximum cache entries
|
||||
// Byte budget for cached result pages (weigher-bounded, 32 MiB
|
||||
// default; env OXICLOUD_SEARCH_CACHE_MAX_BYTES). Replaces the old
|
||||
// entry-count capacity, which let 500-row pages keyed by
|
||||
// user×query×offset×limit pin hundreds of MB for the TTL.
|
||||
self.config.search_cache.max_bytes,
|
||||
)));
|
||||
|
||||
tracing::info!("Application services initialized");
|
||||
|
||||
@@ -352,8 +352,25 @@ impl File {
|
||||
/// formula here changes it everywhere — that is the property
|
||||
/// we want.
|
||||
pub fn compute_etag(blob_hash: &str, modified_at: u64) -> String {
|
||||
let prefix: String = blob_hash.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, modified_at)
|
||||
use std::fmt::Write as _;
|
||||
|
||||
// Byte index just past the 16th char (whole string when shorter).
|
||||
// `blob_hash` is lowercase hex ASCII in practice, so this is
|
||||
// effectively `min(len, 16)`, but `char_indices` keeps the slice
|
||||
// char-boundary-safe for exotic fixture values — byte-identical
|
||||
// to the old `chars().take(16).collect::<String>()` without the
|
||||
// intermediate allocation.
|
||||
let end = match blob_hash.char_indices().nth(16) {
|
||||
Some((i, _)) => i,
|
||||
None => blob_hash.len(),
|
||||
};
|
||||
|
||||
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
|
||||
let mut etag = String::with_capacity(end + 1 + 20);
|
||||
etag.push_str(&blob_hash[..end]);
|
||||
etag.push('-');
|
||||
let _ = write!(etag, "{modified_at}");
|
||||
etag
|
||||
}
|
||||
|
||||
// Getters
|
||||
|
||||
@@ -7,6 +7,30 @@ use crate::domain::services::path_service::{
|
||||
// Re-export entity errors from the centralized module
|
||||
pub use super::entity_errors::{FolderError, FolderResult};
|
||||
|
||||
/// Owned parts of a [`Folder`] entity, produced by [`Folder::into_parts()`].
|
||||
///
|
||||
/// Consuming a `Folder` into `FolderParts` **moves** every field without
|
||||
/// cloning, eliminating the 3-4 heap allocations that previously occurred
|
||||
/// when converting `Folder → FolderDto` via `.to_string()` on each getter.
|
||||
/// Mirrors [`super::file::FileParts`].
|
||||
pub struct FolderParts {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub storage_path: StoragePath,
|
||||
pub path_string: String,
|
||||
pub parent_id: Option<String>,
|
||||
/// Drive that owns this folder. See [`Folder::drive_id`].
|
||||
pub drive_id: Uuid,
|
||||
pub created_at: u64,
|
||||
pub modified_at: u64,
|
||||
/// Descendant-rollup timestamp. See [`Folder::tree_modified_at`].
|
||||
pub tree_modified_at: u64,
|
||||
/// §14 provenance: original creator. See [`Folder::created_by`].
|
||||
pub created_by: Option<Uuid>,
|
||||
/// §14 provenance: most recent mutator. See [`Folder::updated_by`].
|
||||
pub updated_by: Option<Uuid>,
|
||||
}
|
||||
|
||||
/// Represents a folder entity in the domain
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Folder {
|
||||
@@ -219,6 +243,26 @@ impl Folder {
|
||||
})
|
||||
}
|
||||
|
||||
/// Consume the entity and return all fields by ownership.
|
||||
///
|
||||
/// Use this when converting `Folder` into a DTO to avoid cloning
|
||||
/// every `String` field (saves 3-4 heap allocations per folder).
|
||||
pub fn into_parts(self) -> FolderParts {
|
||||
FolderParts {
|
||||
id: self.id,
|
||||
name: self.name,
|
||||
storage_path: self.storage_path,
|
||||
path_string: self.path_string,
|
||||
parent_id: self.parent_id,
|
||||
drive_id: self.drive_id,
|
||||
created_at: self.created_at,
|
||||
modified_at: self.modified_at,
|
||||
tree_modified_at: self.tree_modified_at,
|
||||
created_by: self.created_by,
|
||||
updated_by: self.updated_by,
|
||||
}
|
||||
}
|
||||
|
||||
// Getters
|
||||
pub fn id(&self) -> &str {
|
||||
&self.id
|
||||
@@ -326,8 +370,25 @@ impl Folder {
|
||||
/// changed; the folder's own value stays untouched
|
||||
/// (self-exclusion).
|
||||
pub fn compute_etag(id: &str, tree_modified_at: u64) -> String {
|
||||
let prefix: String = id.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, tree_modified_at)
|
||||
use std::fmt::Write as _;
|
||||
|
||||
// Byte index just past the 16th char (whole string when shorter).
|
||||
// `id` is a UUID string (ASCII) in practice, so this is
|
||||
// effectively `min(len, 16)`, but `char_indices` keeps the slice
|
||||
// char-boundary-safe for exotic fixture values — byte-identical
|
||||
// to the old `chars().take(16).collect::<String>()` without the
|
||||
// intermediate allocation.
|
||||
let end = match id.char_indices().nth(16) {
|
||||
Some((i, _)) => i,
|
||||
None => id.len(),
|
||||
};
|
||||
|
||||
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
|
||||
let mut etag = String::with_capacity(end + 1 + 20);
|
||||
etag.push_str(&id[..end]);
|
||||
etag.push('-');
|
||||
let _ = write!(etag, "{tree_modified_at}");
|
||||
etag
|
||||
}
|
||||
|
||||
/// Creates a new Folder instance from a DTO
|
||||
|
||||
@@ -98,6 +98,32 @@ pub trait FolderRepository: Send + Sync + 'static {
|
||||
include_total: bool,
|
||||
) -> Result<(Vec<Folder>, Option<usize>), DomainError>;
|
||||
|
||||
/// Keyset-paged listing of `parent_id`'s direct sub-folders in name
|
||||
/// order — `name > $after_name ORDER BY name LIMIT $limit`, one bounded
|
||||
/// index-range read per page off the partial unique index
|
||||
/// `idx_folders_unique_name`. Streaming PROPFIND drains sub-folders
|
||||
/// with this instead of `COUNT(*) OVER() … LIMIT/OFFSET`, which
|
||||
/// window-aggregated and rescanned all N sub-folders on every page
|
||||
/// (4.5x on a 5k-dir parent, benches/FOLDER-KEYSET.md). `has_next`
|
||||
/// falls out of `rows.len() == limit` — no total needed.
|
||||
///
|
||||
/// The default implementation falls back to `list_folders` + in-memory
|
||||
/// slice so stubs and mocks compile without changes.
|
||||
async fn list_folders_batch(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<Folder>, DomainError> {
|
||||
let mut all = self.list_folders(parent_id).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Renames a folder. `caller_id` is stamped into `updated_by`
|
||||
/// alongside the `updated_at = NOW()` bump (§14 provenance).
|
||||
async fn rename_folder(
|
||||
|
||||
@@ -488,13 +488,21 @@ impl FileBlobReadRepository {
|
||||
/// `sort_date` epoch for each file (used as pagination cursor).
|
||||
///
|
||||
/// Uses the denormalised `media_sort_date` column (synced from
|
||||
/// `file_metadata.captured_at` by trigger) so no JOIN with
|
||||
/// `file_metadata` is needed. The partial covering index
|
||||
/// `idx_files_media_timeline_by_drive` (migration 20260901000001)
|
||||
/// keys on `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows — Postgres does one IndexScan per in-scope
|
||||
/// drive_id already ordered by capture date, so LIMIT stops the scan
|
||||
/// early. Same O(LIMIT) shape as the pre-D7 `user_id`-keyed hot path.
|
||||
/// `file_metadata.captured_at` by trigger). The accessible drive ids
|
||||
/// are materialised once, then a `CROSS JOIN LATERAL (… ORDER BY
|
||||
/// media_sort_date DESC LIMIT k)` per drive turns the partial covering
|
||||
/// index `idx_files_media_timeline_by_drive` (migration 20260901000001,
|
||||
/// `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows) into one BOUNDED index scan per drive; the outer
|
||||
/// merge sorts `drives × k` rows. The folders / file_metadata joins sit
|
||||
/// outside the top-N so only the k emitted rows pay them.
|
||||
///
|
||||
/// The previous shape put the joins and the global `ORDER BY … LIMIT`
|
||||
/// above a `drive_id IN (…)` nested loop — Postgres fed EVERY media row
|
||||
/// through the join into a top-N heapsort, scanning the timeline index
|
||||
/// to exhaustion on every page: O(library) per page, 97 ms on a
|
||||
/// 50k-photo library vs 1.6 ms for this shape (55.7x,
|
||||
/// benches/PHOTOS-TIMELINE.md).
|
||||
///
|
||||
/// Scope (`docs/plan/drive.md` §15): drives with
|
||||
/// `policies.include_in_photo_index = true` where the caller has a
|
||||
@@ -537,19 +545,7 @@ impl FileBlobReadRepository {
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by,
|
||||
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
|
||||
WHERE fi.drive_id IN (
|
||||
WITH accessible AS MATERIALIZED (
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
@@ -563,11 +559,33 @@ impl FileBlobReadRepository {
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
)
|
||||
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
|
||||
top.size, top.mime_type,
|
||||
EXTRACT(EPOCH FROM top.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM top.updated_at)::bigint,
|
||||
top.blob_hash,
|
||||
top.created_by, top.updated_by,
|
||||
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM (
|
||||
SELECT fi.*
|
||||
FROM accessible a
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT fi.*
|
||||
FROM storage.files fi
|
||||
WHERE fi.drive_id = a.id
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) fi
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) top
|
||||
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
|
||||
ORDER BY top.media_sort_date DESC
|
||||
"#,
|
||||
);
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
|
||||
|
||||
@@ -501,6 +501,61 @@ impl FolderRepository for FolderDbRepository {
|
||||
Ok((folders?, total))
|
||||
}
|
||||
|
||||
/// Keyset sub-folder page: `name > $after ORDER BY name LIMIT $limit`,
|
||||
/// one bounded index-range read off `idx_folders_unique_name` — the
|
||||
/// cursor predicate is only emitted when a cursor exists (a bound
|
||||
/// disjunction would block the index condition under generic plans,
|
||||
/// same rule as `list_files_batch`). Root scope (`parent_id = None`)
|
||||
/// keeps the trait's in-memory default: roots are one-per-drive, a
|
||||
/// handful of rows.
|
||||
async fn list_folders_batch(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<Folder>, DomainError> {
|
||||
let Some(pid) = parent_id else {
|
||||
let mut all = self.list_folders(None).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
return Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.collect());
|
||||
};
|
||||
|
||||
let cursor_pred = if after_name.is_some() {
|
||||
"AND name > $3"
|
||||
} else {
|
||||
"AND $3::text IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
"SELECT id::text, name, path, parent_id::text, drive_id, \
|
||||
EXTRACT(EPOCH FROM created_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM updated_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM tree_modified_at)::bigint, \
|
||||
created_by, updated_by \
|
||||
FROM storage.folders \
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed \
|
||||
{cursor_pred} \
|
||||
ORDER BY name \
|
||||
LIMIT $2"
|
||||
);
|
||||
let rows: Vec<FolderRow> = sqlx::query_as(&sql)
|
||||
.bind(pid)
|
||||
.bind(limit as i64)
|
||||
.bind(after_name)
|
||||
.fetch_all(self.pool())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FolderDb", format!("batch: {e}")))?;
|
||||
|
||||
rows.into_iter()
|
||||
.map(|(id, name, path, pid, did, ca, ma, tma, cb, ub)| {
|
||||
Self::row_to_folder(id, name, path, pid, did, ca, ma, tma, cb, ub)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Paginated companion to `list_root_folders_for_caller` — same
|
||||
/// drive-membership predicate, adds LIMIT/OFFSET and an optional
|
||||
/// window-function COUNT so total pages can be surfaced without a
|
||||
@@ -1391,13 +1446,6 @@ impl FolderDbRepository {
|
||||
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
|
||||
"#;
|
||||
|
||||
let cte_inner = match (include_folders, include_files) {
|
||||
(true, true) => format!("{folder_branch} UNION ALL {file_branch}"),
|
||||
(true, false) => folder_branch.to_owned(),
|
||||
(false, true) => file_branch.to_owned(),
|
||||
(false, false) => unreachable!(),
|
||||
};
|
||||
|
||||
// ── Cursor binds ─────────────────────────────────────────────────────
|
||||
// $1 = parent_id $2 = cursor_str $3 = cursor_int
|
||||
// $4 = cursor_ts $5 = cursor_id $6 = limit
|
||||
@@ -1406,112 +1454,184 @@ impl FolderDbRepository {
|
||||
let cursor_ts = cursor.and_then(|c| c.sort_ts);
|
||||
let cursor_id = cursor.map(|c| c.resource_id);
|
||||
|
||||
// ── Sort-specific WHERE + ORDER BY ───────────────────────────────────
|
||||
// Each arm produces two variants based on `reverse`.
|
||||
// For "name": folder_first stays ASC in both directions (folders always
|
||||
// precede files); only the alpha order within each group flips.
|
||||
let (where_clause, order_clause) = match order_by {
|
||||
"type" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order < $3)
|
||||
OR (type_order = $3 AND sort_str < $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY type_order DESC, sort_str DESC, id DESC",
|
||||
)
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order > $3)
|
||||
OR (type_order = $3 AND sort_str > $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY type_order ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
// ── Per-branch cursor pushdown ───────────────────────────────────────
|
||||
// The cursor is applied INSIDE each UNION-ALL branch as a sargable
|
||||
// row-value comparison on base columns — not on the CTE's computed
|
||||
// columns — and every branch pre-sorts and pre-limits, so Postgres
|
||||
// reads O(limit) rows per branch instead of rescanning and
|
||||
// top-N-sorting the entire folder on every page (19.5x on a
|
||||
// 20k-entry folder, benches/LISTING-KEYSET.md). The "name" sort is
|
||||
// served by the expression indexes idx_files_folder_lname /
|
||||
// idx_folders_parent_lname (migration 20260918000000).
|
||||
//
|
||||
// Sort-key columns that are CONSTANT within a branch (folder_first,
|
||||
// the folder branch's type_order = 0 and size = -1) are folded in
|
||||
// Rust: depending on which group the cursor points into, the branch
|
||||
// predicate shortens to a row-value over the remaining keys, the
|
||||
// branch keeps all its rows, or the branch drops out entirely.
|
||||
enum BranchCursor {
|
||||
/// The cursor has moved past every row this branch can produce.
|
||||
Drop,
|
||||
/// Every row in this branch sorts after the cursor.
|
||||
All,
|
||||
/// Row-value comparison over the branch's non-constant sort keys.
|
||||
Pred(String),
|
||||
}
|
||||
use BranchCursor::{All, Drop, Pred};
|
||||
|
||||
let has_cursor = cursor.is_some();
|
||||
// (folder-branch cursor, file-branch cursor, per-branch ORDER BY on
|
||||
// the branch's output aliases, outer merge ORDER BY)
|
||||
let (folder_cur, file_cur, branch_order, outer_order) = match order_by {
|
||||
"type" => {
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY type_order DESC, sort_str DESC, id DESC")
|
||||
} else {
|
||||
(">", "ORDER BY type_order ASC, sort_str ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have type_order = 0; a cursor sitting on a
|
||||
// file (type_order > 0) either exhausts the folder group
|
||||
// (ASC) or precedes all of it (DESC).
|
||||
Some(c_to) if c_to > 0 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!(
|
||||
"(fm.category_order::bigint, LOWER(fm.name), fm.id) {op} ($3, $2, $5::uuid)"
|
||||
))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
"modified_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at > $4)
|
||||
OR (modified_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY modified_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY modified_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at < $4)
|
||||
OR (modified_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY modified_at DESC, id DESC",
|
||||
)
|
||||
("<", "ORDER BY modified_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.updated_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"created_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at > $4)
|
||||
OR (created_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY created_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY created_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at < $4)
|
||||
OR (created_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY created_at DESC, id DESC",
|
||||
)
|
||||
("<", "ORDER BY created_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.created_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"size" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size < $3)
|
||||
OR (size = $3 AND id < $5::uuid)"#,
|
||||
"ORDER BY size DESC, id DESC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY size DESC, id DESC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size > $3)
|
||||
OR (size = $3 AND id > $5::uuid)"#,
|
||||
"ORDER BY size ASC, id ASC",
|
||||
)
|
||||
(">", "ORDER BY size ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have size = -1; a cursor sitting on a file
|
||||
// (size >= 0) exhausts the folder group (ASC) or precedes
|
||||
// all of it (DESC).
|
||||
Some(c_sz) if c_sz > -1 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("f.id {op} $5::uuid")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!("(fm.size::bigint, fm.id) {op} ($3, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
_ => {
|
||||
// "name" (default): folder_first stays ASC so folders always precede
|
||||
// files; only the alpha order within each group flips when reversed.
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str < $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC",
|
||||
)
|
||||
// "name" (default): folder_first stays ASC so folders always
|
||||
// precede files; only the alpha order within each group flips
|
||||
// when reversed. cursor_int carries folder_first (0|1).
|
||||
let op = if reverse { "<" } else { ">" };
|
||||
let branch_ord = if reverse {
|
||||
"ORDER BY sort_str DESC, id DESC"
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str > $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
}
|
||||
"ORDER BY sort_str ASC, id ASC"
|
||||
};
|
||||
let outer_ord = if reverse {
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC"
|
||||
} else {
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC"
|
||||
};
|
||||
let (folder_cur, file_cur) = match cursor_int {
|
||||
None => (All, All),
|
||||
// Cursor inside the folder group: folders continue after
|
||||
// the row-value cursor; every file still follows.
|
||||
Some(0) => (
|
||||
Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
All,
|
||||
),
|
||||
// Cursor inside the file group: the folder group is done.
|
||||
Some(_) => (
|
||||
Drop,
|
||||
Pred(format!("(LOWER(fm.name), fm.id) {op} ($2, $5::uuid)")),
|
||||
),
|
||||
};
|
||||
(folder_cur, file_cur, branch_ord, outer_ord)
|
||||
}
|
||||
};
|
||||
|
||||
let wrap = |branch: &str, cur: &BranchCursor| -> Option<String> {
|
||||
let extra = match cur {
|
||||
Drop => return None,
|
||||
All => String::new(),
|
||||
Pred(p) => format!(" AND {p}"),
|
||||
};
|
||||
Some(format!(
|
||||
"(SELECT * FROM ({branch}{extra}) b {branch_order} LIMIT $6)"
|
||||
))
|
||||
};
|
||||
let mut branches = Vec::with_capacity(2);
|
||||
if include_folders && let Some(b) = wrap(folder_branch, &folder_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
if include_files && let Some(b) = wrap(file_branch, &file_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
// Every requested branch dropped out (e.g. folders-only listing with
|
||||
// the cursor already past the folder group).
|
||||
if branches.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let inner = branches.join(" UNION ALL ");
|
||||
|
||||
let sql = format!(
|
||||
"WITH resources AS ({cte_inner}) \
|
||||
SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
"SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
created_at, modified_at, drive_id, blob_hash, \
|
||||
sort_str, type_order, folder_first \
|
||||
FROM resources \
|
||||
{where_clause} \
|
||||
{order_clause} \
|
||||
FROM ({inner}) r \
|
||||
{outer_order} \
|
||||
LIMIT $6"
|
||||
);
|
||||
|
||||
|
||||
@@ -130,7 +130,9 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
return Ok(size);
|
||||
}
|
||||
|
||||
client.put_block_blob(data.to_vec()).await.map_err(|e| {
|
||||
// `Bytes` converts into `azure_core::Body` by reference count —
|
||||
// the old `data.to_vec()` copied every chunk once more.
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
|
||||
@@ -138,6 +140,26 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Content-addressed keys make
|
||||
/// re-PUTs idempotent, so the `get_properties` probe
|
||||
/// `put_blob_from_bytes` pays is a pure extra round-trip on every NEW
|
||||
/// chunk (2 RTTs -> 1, benches/S3-PUT.md — same shape as S3).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
let size = data.len() as u64;
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
@@ -13,12 +13,14 @@ use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use bytes::Bytes;
|
||||
use dashmap::DashMap;
|
||||
use lru::LruCache;
|
||||
use std::num::NonZeroUsize;
|
||||
use tokio::fs;
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::io::ReaderStream;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
@@ -56,6 +58,13 @@ pub struct CachedBlobBackend {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
/// Per-hash single-flight gates for cache misses. K concurrent cold
|
||||
/// readers of one blob (e.g. a video player's parallel Range probes)
|
||||
/// used to each download the FULL blob from the remote backend — and
|
||||
/// race their writes on one shared `.tmp` path. The gate coalesces
|
||||
/// them onto one fetch; waiters re-check the cache and serve locally
|
||||
/// (16 fetches -> 1, benches/BLOB-CACHE.md).
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedBlobBackend {
|
||||
@@ -70,6 +79,7 @@ impl CachedBlobBackend {
|
||||
NonZeroUsize::new(1_000_000).unwrap(),
|
||||
))),
|
||||
current_size: Arc::new(AtomicU64::new(0)),
|
||||
inflight: Arc::new(DashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,6 +160,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
// Write to inner backend
|
||||
@@ -172,6 +183,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
let size = inner.put_blob_from_bytes(&hash, data.clone()).await?;
|
||||
@@ -203,6 +215,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE touching the filesystem so concurrent
|
||||
@@ -219,14 +232,17 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch from inner, spool to cache
|
||||
// Cache miss — fetch from inner (single-flight), spool to cache
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let file = fs::File::open(&dest).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
|
||||
})?;
|
||||
@@ -249,6 +265,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE the open()/seek() syscalls so concurrent
|
||||
@@ -271,14 +288,19 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch full blob into cache, then serve range
|
||||
// Cache miss — fetch full blob into cache (single-flight: a
|
||||
// player's parallel cold Range probes coalesce onto ONE remote
|
||||
// download), then serve the range locally.
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let mut file = fs::File::open(&dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
|
||||
@@ -406,6 +428,7 @@ struct CachedRef {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedRef {
|
||||
@@ -414,6 +437,37 @@ impl CachedRef {
|
||||
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
|
||||
/// Single-flight wrapper around [`Self::fetch_and_cache_static`]: the
|
||||
/// first caller for a hash becomes the leader and downloads; concurrent
|
||||
/// callers queue on the per-hash gate, then re-check the cache and serve
|
||||
/// the leader's file without touching the remote backend. Errors are not
|
||||
/// cached — the gate entry is dropped, so the next caller retries.
|
||||
async fn fetch_and_cache_singleflight(
|
||||
&self,
|
||||
hash: &str,
|
||||
inner: &dyn BlobStorageBackend,
|
||||
cached: &Path,
|
||||
) -> Result<PathBuf, DomainError> {
|
||||
let gate = self
|
||||
.inflight
|
||||
.entry(hash.to_string())
|
||||
.or_insert_with(|| Arc::new(Mutex::new(())))
|
||||
.clone();
|
||||
let _guard = gate.lock().await;
|
||||
|
||||
// Re-check under the gate: if we queued behind the leader, the blob
|
||||
// is on disk now and this turns into a local open.
|
||||
if self.index.lock().await.get(hash).is_some() && fs::metadata(cached).await.is_ok() {
|
||||
return Ok(cached.to_path_buf());
|
||||
}
|
||||
|
||||
let result = self.fetch_and_cache_static(hash, inner).await;
|
||||
// Drop the gate whether we succeeded or failed; a late-arriving
|
||||
// caller after an error creates a fresh gate and retries the fetch.
|
||||
self.inflight.remove(hash);
|
||||
result
|
||||
}
|
||||
|
||||
/// Pop LRU entries until the cache is back within its byte budget,
|
||||
/// returning the on-disk paths of the evicted blobs.
|
||||
///
|
||||
@@ -486,10 +540,15 @@ impl CachedRef {
|
||||
})?;
|
||||
}
|
||||
|
||||
let tmp = dest.with_extension("tmp");
|
||||
let mut file = fs::File::create(&tmp)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
|
||||
// Unique temp name: even if two fetches for one hash ever race
|
||||
// (e.g. across processes sharing a cache dir), each writes its own
|
||||
// inode and the rename is atomic — a torn/interleaved file can
|
||||
// never land at the final path.
|
||||
let tmp = dest.with_extension(format!("{}.tmp", Uuid::new_v4()));
|
||||
let write_result: Result<u64, DomainError> = async {
|
||||
let mut file = fs::File::create(&tmp).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("create tmp: {e}"))
|
||||
})?;
|
||||
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
@@ -506,11 +565,26 @@ impl CachedRef {
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
drop(file);
|
||||
Ok(total)
|
||||
}
|
||||
.await;
|
||||
let total = match write_result {
|
||||
Ok(total) => total,
|
||||
Err(e) => {
|
||||
// Unique tmp names never get overwritten by a later fetch —
|
||||
// reap the partial file instead of leaking it.
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
|
||||
fs::rename(&tmp, &dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
|
||||
if let Err(e) = fs::rename(&tmp, &dest).await {
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(DomainError::internal_error(
|
||||
"BlobCache",
|
||||
format!("rename: {e}"),
|
||||
));
|
||||
}
|
||||
|
||||
let to_evict = {
|
||||
let mut idx = self.index.lock().await;
|
||||
|
||||
@@ -200,6 +200,38 @@ impl BlobStorageBackend for S3BlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Keys are content-addressed
|
||||
/// (BLAKE3), so a re-PUT writes identical bytes — overwrite-safe
|
||||
/// idempotency without the HEAD probe `put_blob_from_bytes` pays. The
|
||||
/// dedup layer already filtered out chunks the database knows about,
|
||||
/// so the probe was a pure extra round-trip on every NEW chunk of
|
||||
/// every upload (2 RTTs -> 1, benches/S3-PUT.md).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
let size = data.len() as u64;
|
||||
self.client
|
||||
.put_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.body(ByteStream::from(data))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to upload blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
@@ -385,7 +385,6 @@ async fn handle_propfind(
|
||||
CardDavAdapter::generate_contacts_response(
|
||||
&mut response_body,
|
||||
std::slice::from_ref(&contact),
|
||||
&[(contact.uid.clone(), contact_to_vcard(&contact))],
|
||||
&report,
|
||||
base_href,
|
||||
)
|
||||
@@ -449,21 +448,9 @@ async fn handle_report(
|
||||
.map_err(AppError::from)?,
|
||||
};
|
||||
|
||||
// Generate vCards
|
||||
let vcards: Vec<(String, String)> = contacts
|
||||
.iter()
|
||||
.map(|c| (c.uid.clone(), contact_to_vcard(c)))
|
||||
.collect();
|
||||
|
||||
let base_href = &format!("/carddav/{}/", address_book_id);
|
||||
let mut response_body = Vec::new();
|
||||
CardDavAdapter::generate_contacts_response(
|
||||
&mut response_body,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
base_href,
|
||||
)
|
||||
CardDavAdapter::generate_contacts_response(&mut response_body, &contacts, &report, base_href)
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
|
||||
|
||||
Ok(Response::builder()
|
||||
|
||||
@@ -781,25 +781,25 @@ async fn build_streaming_propfind_response(
|
||||
|
||||
// ── Children (only if Depth == 1) ────────────────────────
|
||||
if depth == "1" {
|
||||
let pagination = crate::application::dtos::pagination::PaginationRequestDto {
|
||||
page: 0,
|
||||
page_size: PROPFIND_BATCH_SIZE as usize,
|
||||
};
|
||||
let fid_ref = folder_id.as_deref();
|
||||
|
||||
// Stream sub-folders in pages (user-scoped)
|
||||
let mut page = 0usize;
|
||||
// Stream sub-folders in pages (user-scoped, keyset cursor —
|
||||
// O(page) per page off idx_folders_unique_name instead of the
|
||||
// quadratic COUNT(*) OVER() + LIMIT/OFFSET walk; 4.5x on a
|
||||
// 5k-dir parent, benches/FOLDER-KEYSET.md).
|
||||
let mut after_folder: Option<String> = None;
|
||||
loop {
|
||||
let pag = crate::application::dtos::pagination::PaginationRequestDto {
|
||||
page,
|
||||
page_size: pagination.page_size,
|
||||
};
|
||||
let result = folder_service
|
||||
.list_folders_paginated_with_perms(fid_ref, user_id, &pag)
|
||||
let batch = folder_service
|
||||
.list_folders_batch_with_perms(
|
||||
fid_ref,
|
||||
user_id,
|
||||
after_folder.as_deref(),
|
||||
PROPFIND_BATCH_SIZE as usize,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
|
||||
if result.items.is_empty() {
|
||||
if batch.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -808,25 +808,25 @@ async fn build_streaming_propfind_response(
|
||||
// 1-4.5 s of pure DB chatter on a 2000-child folder
|
||||
// (measured in benches/DEAD-PROPS.md).
|
||||
let subfolder_deads =
|
||||
folders_dead_props_map(&dead_props_store, &result.items).await;
|
||||
folders_dead_props_map(&dead_props_store, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 800);
|
||||
let mut chunk = Vec::with_capacity(batch.len() * 800);
|
||||
{
|
||||
let mut w = Writer::new(&mut chunk);
|
||||
for subfolder in result.items.iter() {
|
||||
for subfolder in batch.iter() {
|
||||
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
|
||||
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
|
||||
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
}
|
||||
}
|
||||
let has_more = result.pagination.has_next;
|
||||
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
|
||||
after_folder = batch.last().map(|f| f.name.clone());
|
||||
yield Bytes::from(chunk);
|
||||
|
||||
if !has_more {
|
||||
break;
|
||||
}
|
||||
page += 1;
|
||||
}
|
||||
|
||||
// Stream files in pages (user-scoped, keyset cursor — O(page)
|
||||
|
||||
@@ -16,7 +16,6 @@ use uuid::Uuid;
|
||||
use crate::application::adapters::webdav_adapter::{
|
||||
PropFindRequest, PropPatchOp, QualifiedName, WebDavAdapter, is_protected_property,
|
||||
};
|
||||
use crate::application::dtos::pagination::PaginationRequestDto;
|
||||
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
||||
use crate::application::ports::favorites_ports::FavoritesUseCase;
|
||||
use crate::application::ports::file_ports::{
|
||||
@@ -1584,38 +1583,42 @@ fn build_nc_streaming_propfind(
|
||||
after_name = batch.last().map(|f| f.name.clone());
|
||||
}
|
||||
|
||||
// Subfolders in pages — also collections, same trailing-slash rule.
|
||||
let mut page = 0usize;
|
||||
// Subfolders in pages — also collections, same trailing-slash
|
||||
// rule. Keyset cursor: O(page) per page off
|
||||
// idx_folders_unique_name instead of the quadratic
|
||||
// COUNT(*) OVER() + LIMIT/OFFSET walk (benches/FOLDER-KEYSET.md).
|
||||
let mut after_folder: Option<String> = None;
|
||||
loop {
|
||||
let pag = PaginationRequestDto {
|
||||
page,
|
||||
page_size: PROPFIND_BATCH_SIZE as usize,
|
||||
};
|
||||
let result = folder_service
|
||||
.list_folders_paginated_with_perms(Some(&folder.id), user_id, &pag)
|
||||
let batch = folder_service
|
||||
.list_folders_batch_with_perms(
|
||||
Some(&folder.id),
|
||||
user_id,
|
||||
after_folder.as_deref(),
|
||||
PROPFIND_BATCH_SIZE as usize,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
if result.items.is_empty() {
|
||||
if batch.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
let favs = if let Some(fav) = fav_svc {
|
||||
let items: Vec<(&str, &str)> =
|
||||
result.items.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
|
||||
batch.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
|
||||
fav.batch_check_favorites(user_id, &items).await.unwrap_or_default()
|
||||
} else {
|
||||
HashSet::new()
|
||||
};
|
||||
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
|
||||
let folder_uuids: Vec<String> = batch.iter().map(|sf| sf.id.clone()).collect();
|
||||
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
|
||||
// Batched — see benches/DEAD-PROPS.md.
|
||||
let sub_deads =
|
||||
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
|
||||
folders_dead_props_map(&state.webdav_dead_props, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
|
||||
let mut chunk = Vec::with_capacity(batch.len() * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for sf in result.items.iter() {
|
||||
for sf in batch.iter() {
|
||||
let dead = dead_props_for(&sf.id, &sub_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
sf.name.clone()
|
||||
@@ -1629,13 +1632,13 @@ fn build_nc_streaming_propfind(
|
||||
.map_err(std::io::Error::other)?;
|
||||
}
|
||||
}
|
||||
let has_more = result.pagination.has_next;
|
||||
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
|
||||
after_folder = batch.last().map(|sf| sf.name.clone());
|
||||
yield Bytes::from(chunk);
|
||||
|
||||
if !has_more {
|
||||
break;
|
||||
}
|
||||
page += 1;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -273,11 +273,16 @@ pub fn multipart_field_stream(
|
||||
pub fn stream_from_files(
|
||||
paths: Vec<PathBuf>,
|
||||
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
|
||||
// 512 KiB per poll: each ReaderStream poll on a tokio::fs::File is one
|
||||
// blocking-pool dispatch + one read(2) of the buffer size. The old
|
||||
// 64 KiB buffer paid 8x the dispatches/syscalls of every other blob
|
||||
// read path (STREAM_CHUNK_SIZE = 256 KiB) for the single read pass
|
||||
// over every completed chunked upload (benches/UPLOAD-SPOOL.md).
|
||||
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
|
||||
.and_then(|path| async move {
|
||||
tokio::fs::File::open(path)
|
||||
.await
|
||||
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
|
||||
.map(|file| ReaderStream::with_capacity(file, 512 * 1024))
|
||||
})
|
||||
.try_flatten()
|
||||
}
|
||||
@@ -319,9 +324,15 @@ pub async fn stream_body_to_path(
|
||||
max_bytes: usize,
|
||||
checksum_alg: Option<ChecksumAlg>,
|
||||
) -> Result<StreamedToPath, AppError> {
|
||||
let mut file = tokio::fs::File::create(path)
|
||||
// BufWriter coalesces the per-HTTP-frame writes (~16-64 KiB each) into
|
||||
// 512 KiB write(2)s — a bare tokio File dispatches one blocking-pool op
|
||||
// per frame (benches/UPLOAD-SPOOL.md). Same capacity as the dedup
|
||||
// handler's spool loop. On the error paths below the partial file is
|
||||
// removed, so silently dropping unflushed buffer contents is fine.
|
||||
let file = tokio::fs::File::create(path)
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
|
||||
let mut file = tokio::io::BufWriter::with_capacity(512 * 1024, file);
|
||||
|
||||
let mut total_bytes: usize = 0;
|
||||
let mut stream = BodyStream::new(body);
|
||||
|
||||
Reference in New Issue
Block a user