From 7b26ff014c1aa0224604497463dfb69aa7b0ba6d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 22 Jun 2026 08:38:27 +0000 Subject: [PATCH] docs(bench): record blob-prefetch + tokio-runtime benchmark results Companion benches/*.md (matching the repo convention) capturing the before/after numbers and the honest interpretation behind the read_prefetch 1->2 tuning and the runtime pool sizing. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez --- benches/BLOB-PREFETCH.md | 53 +++++++++++++++++++++++ benches/RUNTIME.md | 93 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 benches/BLOB-PREFETCH.md create mode 100644 benches/RUNTIME.md diff --git a/benches/BLOB-PREFETCH.md b/benches/BLOB-PREFETCH.md new file mode 100644 index 00000000..77c00a58 --- /dev/null +++ b/benches/BLOB-PREFETCH.md @@ -0,0 +1,53 @@ +# Blob download read-ahead benchmark + +Measures the local backend's chunk read-ahead depth — the `buffered(N)` +read-ahead in `DedupService::stream_chunks` fed by +`BlobStorageBackend::read_prefetch()`. Reproduces the exact production reassembly +combinator (`stream::iter(hashes).map(get_blob_stream).buffered(N).try_flatten()`) +over a real `LocalBlobBackend` whose chunk files are scattered across the 256 +hash-prefix dirs, then drains it and reports throughput. `N = 1` is the old +production default ("antes"); higher N is the change ("después"). + +## Reproduce + +```bash +cargo run --release --features bench --example bench_blob_prefetch +# tunables: BENCH_FILE_MB=192 BENCH_CHUNK_KB=256 BENCH_PREFETCH=1,2,4,8,16 +# BENCH_THROTTLE_MBPS=0,300,100 BENCH_REPS=5 BENCH_COLD=1 +``` + +## Results (4-core box, SSD-class storage, 192 MiB in 768×256 KiB chunks) + +Median MB/s over 5 reps; `vs N=1` is the read-ahead gain over the old default. + +| scenario | N=1 | N=2 | N=4 | N=8 | N=16 | +|-------------------------|----:|-----:|-----:|-----:|-----:| +| warm / unthrottled |1306 | **1460** |1394 |1357 |1248 | +| cold / unthrottled | 456 | **489** | 469 | 478 | 473 | +| warm / throttled@300MB/s| 167 | 166 | 166 | 166 | 166 | +| cold / throttled@300MB/s| 138 | 140 | 135 | 135 | 136 | +| warm / throttled@100MB/s| 62 | 62 | 62 | 62 | 62 | +| cold / throttled@100MB/s| 57 | 58 | 57 | 57 | 57 | + +(`vs N=1` for the best column N=2: warm/unthrottled **+11.8 %**, cold/unthrottled +**+7.2 %**; throttled rows ≈ 0 %. N=16 regresses warm −4.4 %.) + +## Conclusions + +1. **N=2 is the sweet spot, not 8.** It wins or ties in 5 of 6 scenarios at the + lowest fan-out: +11.8 % warm and +7.2 % cold on disk-bound reads, neutral when + the consumer is the bottleneck. N=8 gives only +3.9 %/+4.8 %; N=16 regresses. + So local now defaults to 2 (was 1); S3/Azure keep 8 (request-latency bound). + +2. **The win is disk-bound, not network-bound.** Throttled (network-bound) rows + are flat because `buffered(N)` here overlaps the per-chunk `File::open` + (cheap on local disk), **not** the data read (which `try_flatten` polls + sequentially). The disk-bound rows cover localhost/LAN downloads *and* the + internal blob reads that drain as fast as the disk delivers — thumbnail + render, transcode, ZIP export, content extraction — all via `stream_chunks`. + +3. **No cold regression on SSD.** The trait doc's "slower cold" worry (concurrent + opens → random I/O over scattered chunk files) is an HDD seek-thrash concern; + on SSD-class storage cold reads *improved* at N=2. Operators on spinning disks + can restore the old behaviour with `OXICLOUD_LOCAL_READ_PREFETCH=1`; NVMe + arrays can raise it. diff --git a/benches/RUNTIME.md b/benches/RUNTIME.md new file mode 100644 index 00000000..1de62f66 --- /dev/null +++ b/benches/RUNTIME.md @@ -0,0 +1,93 @@ +# Tokio runtime tuning benchmark + +Measures the two things `build_runtime` (`src/main.rs`) changes versus the bare +`#[tokio::main]` defaults, sized by `common::runtime::runtime_pool_sizes`: + +- **Worker count.** `#[tokio::main]` defaults to `available_parallelism()`, which + honours CPU *affinity* (`sched_getaffinity`: cpuset, `taskset`) but **ignores + the CFS bandwidth quota** (`docker --cpus`, cgroup v2 `cpu.max`, v1 + `cpu.cfs_quota_us`). On a 2-core-quota container on a many-core host it spawns + one worker per *host* core. `effective_parallelism()` folds the quota back in. +- **Blocking pool.** `#[tokio::main]` defaults to a flat `max_blocking_threads = + 512` — a multi-GB RSS blast radius for this heavy `spawn_blocking` user + (thumbnails, transcode, zip, PDF/text extraction, Argon2 ≈19 MB/hash). The + builder caps it at `max(32, 8 × workers)`. + +## Reproduce + +```bash +cargo build --release --features bench --example bench_tokio_runtime +# Pin to 2 cores to model a 2-core CPU quota on a bigger host: +taskset -c 0,1 ./target/release/examples/bench_tokio_runtime +# Part B uses a fixed glibc mmap threshold for a clean RSS read: +MALLOC_MMAP_THRESHOLD_=131072 MALLOC_TRIM_THRESHOLD_=131072 \ + taskset -c 0,1 ./target/release/examples/bench_tokio_runtime +# tunables: BENCH_CONCURRENCY=96 BENCH_SECONDS=4 BENCH_BURN_KB=256 +# BENCH_WORKERS_BEFORE=32 BENCH_BLOCKING_TASKS=96 BENCH_ALLOC_MB=16 BENCH_MAX_BLOCKING_AFTER=16 +``` + +## Results (4-core box, pinned to 2 cores via `taskset -c 0,1`) + +### [A] Worker over-subscription under CPU contention + +96 concurrent async "requests", each an async hop + a 256 KiB BLAKE3 (models a +handler that interleaves I/O with on-worker compute), over 4 s. + +| runtime | req/s | p50 µs | p99 µs | +|-----------------------|-------:|-------:|-------:| +| before: 32 workers | 46 854 | 121 | 60 360 | +| after: 2 workers | 42 893 | 2 140 | 4 962 | + +→ **throughput −8.5 %, p99 latency −91.8 %** (after vs before) + +### [B] Blocking-pool RSS blast radius + +96 concurrent `spawn_blocking` tasks, 16 MiB resident each, held 120 ms +(fixed glibc mmap threshold so freed allocations leave RSS promptly). + +| max_blocking_threads | peak RSS MiB | vs default | +|-----------------------------|-------------:|-----------:| +| before: 512 (tokio default) | 1 231 | — | +| after: 16 (bounded) | 261 | −970 MiB | + +## Conclusions + +1. **Blocking-pool cap — clear win, no downside.** Bounding 512→16 cut peak RSS + under a 96-task flood from **1231 MiB to 261 MiB (−970 MiB)**. The cap only + engages under a pile-up; steady-state operation is unaffected, and the app's + heaviest blocking consumers are already semaphore-limited (Argon2 = 2, + thumbnail decode ≈ cores), so `max(32, 8×workers)` is generous headroom that + simply removes the unbounded tail that can OOM-kill the process under a spike. + +2. **Worker sizing — a latency/throughput trade, favourable for a server.** + Over-subscription (32 workers on 2 cores, what tokio's default does under a + CFS quota) won **+8.5 % peak throughput** but at a **catastrophic p99 of + 60 ms** (12× the tuned 5 ms) with a bimodal distribution — some requests fly + (p50 121 µs), others starve. Sizing to the quota (2 workers) gives uniform, + predictable latency at a small throughput cost. For an interactive file + server, p99 dominates UX (timeouts, head-of-line blocking), so this is the + right trade. + +3. **This microbenchmark is a worst case *for* the tuned config.** It is pure + on-worker CPU, which is exactly where over-subscription's throughput edge + shows. Real OxiCloud handlers push CPU to `spawn_blocking` and the async + workers mostly await I/O (DB, disk) — there the over-subscription throughput + edge evaporates (idle workers just park) while its tail-latency penalty + remains. Production should see the worker change as ≥ neutral on throughput + and strictly better on tail latency. + +4. **No regression off-quota.** `effective_parallelism()` == `available_ + parallelism()` whenever there is no CFS quota (or affinity already restricts + the process), so on bare metal / affinity-pinned deployments the worker count + is unchanged from the old default. The change only bites under a CFS quota — + precisely the case it fixes. + +5. **Follow-up:** the same `available_parallelism()` blind spot affects the + image/rayon pools (`thumbnail_service.rs`, `image_transcode_service.rs`, + `di.rs` video) — they over-spawn under a CFS quota too. Switching those to + `common::runtime::effective_parallelism()` is the natural next step (left out + here to keep this change focused on the runtime). + +Both knobs are env-overridable (`OXICLOUD_WORKER_THREADS` / +`OXICLOUD_MAX_BLOCKING_THREADS`) and logged at startup ("Tokio runtime pools +sized"), so operators can see and tune what is in effect.