Files
Oxicloud/examples/bench_static_precompress.rs
T
Claude aba89c4f5d perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
2026-07-16 14:20:20 +00:00

171 lines
6.5 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Static-asset compression benchmark — on-the-fly Brotli per request vs
//! serving a precompressed sibling.
//!
//! The SPA router compressed every compressible static response on the fly
//! (tower-http `CompressionLayer`, backed by `async-compression`'s Brotli at
//! `Level::Default`) — the same immutable `/_app/immutable` bundle re-encoded
//! on EVERY request. The change teaches `ServeDir` to serve build-time
//! `.br`/`.gz` siblings (`precompressed_br()/precompressed_gzip()` +
//! `frontend/scripts/precompress.mjs`), so a request costs a file read.
//!
//! This isolates exactly that per-request delta on a JS-bundle-like payload:
//! BEFORE — Brotli-encode the asset with async-compression Level::Default
//! (what the layer does per request)
//! AFTER — read the precompressed sibling from disk (what ServeDir does)
//!
//! Run (no Postgres needed):
//! cargo run --release --features bench --example bench_static_precompress
//! Tunables: BENCH_ASSET_KB (700), BENCH_REPS (30)
use std::env;
use std::io::Write as _;
use std::time::Instant;
use tokio::io::AsyncReadExt;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
/// JS-like corpus: repetitive identifiers + literals, compresses like a real
/// minified bundle (roughly 3-5×).
fn synth_js(len: usize, seed: &mut u64) -> Vec<u8> {
const FRAGS: &[&str] = &[
"function(e,t,n){var r=this;",
"return Object.assign({},",
"const a=document.querySelector(",
"export default{data(){return{",
"await fetch(url,{method:'POST',headers:",
".map(function(x){return x.id});",
"if(void 0!==e&&null!==t){",
"console.error('unhandled',err);",
];
let mut out = Vec::with_capacity(len);
while out.len() < len {
*seed ^= *seed << 13;
*seed ^= *seed >> 7;
*seed ^= *seed << 17;
out.extend_from_slice(FRAGS[(*seed as usize) % FRAGS.len()].as_bytes());
// sprinkle some varying identifiers so it's not pathological
let _ = write!(out, "v{}", *seed % 1000);
}
out.truncate(len);
out
}
fn median(mut xs: Vec<f64>) -> f64 {
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
xs[xs.len() / 2]
}
#[tokio::main(flavor = "multi_thread")]
async fn main() {
let asset_kb: usize = env_or("BENCH_ASSET_KB", 700);
let reps: usize = env_or("BENCH_REPS", 30);
let mut seed = 0xC0FFEEu64;
let asset = synth_js(asset_kb * 1024, &mut seed);
// Precompress once (build-time cost, paid once per deploy).
let dir = tempfile::tempdir().expect("tempdir");
let br_path = dir.path().join("bundle.js.br");
let t = Instant::now();
let precompressed = {
use async_compression::tokio::bufread::BrotliEncoder;
let mut enc = BrotliEncoder::new(std::io::Cursor::new(asset.clone()));
let mut out = Vec::new();
enc.read_to_end(&mut out).await.expect("precompress");
out
};
let build_ms = t.elapsed().as_secs_f64() * 1000.0;
std::fs::write(&br_path, &precompressed).expect("write .br");
println!(
"asset: {} KiB JS-like → {} KiB brotli ({}% smaller); one-time build cost {:.1} ms\n",
asset.len() / 1024,
precompressed.len() / 1024,
100 - precompressed.len() * 100 / asset.len(),
build_ms
);
// BEFORE: per-request Brotli at the layer's default level.
let mut enc_times = Vec::with_capacity(reps);
for _ in 0..reps {
let t = Instant::now();
use async_compression::tokio::bufread::BrotliEncoder;
let mut enc = BrotliEncoder::new(std::io::Cursor::new(asset.clone()));
let mut out = Vec::new();
enc.read_to_end(&mut out).await.expect("encode");
std::hint::black_box(&out);
enc_times.push(t.elapsed().as_secs_f64() * 1000.0);
}
// AFTER: per-request read of the precompressed sibling.
let mut read_times = Vec::with_capacity(reps);
for _ in 0..reps {
let t = Instant::now();
let mut f = tokio::fs::File::open(&br_path).await.expect("open");
let mut out = Vec::new();
f.read_to_end(&mut out).await.expect("read");
std::hint::black_box(&out);
read_times.push(t.elapsed().as_secs_f64() * 1000.0);
}
// ── Dynamic-response level sweep ─────────────────────────────────────
// The global API CompressionLayer (main.rs) compresses JSON responses
// per request. async-compression's Level::Default for Brotli is
// QUALITY 11 (brotli-8.0.2 encode.rs:323 via compression-codecs) — a
// deploy-grade setting on a per-request path. Sweep levels on a
// JSON-like 64 KiB body to pick the runtime quality.
let json_body = synth_js(64 * 1024, &mut seed); // JSON compresses like JS
println!("\n# per-request Brotli level on a 64 KiB JSON-like API response");
println!("{:<22} {:>10} {:>12}", "level", "ms/resp", "out KiB");
for (label, level) in [
("Default (= q11!)", async_compression::Level::Default),
("Precise(4)", async_compression::Level::Precise(4)),
("Fastest", async_compression::Level::Fastest),
] {
let mut times = Vec::with_capacity(reps);
let mut out_len = 0;
for _ in 0..reps {
let t = Instant::now();
use async_compression::tokio::bufread::BrotliEncoder;
let mut enc =
BrotliEncoder::with_quality(std::io::Cursor::new(json_body.clone()), level);
let mut out = Vec::new();
enc.read_to_end(&mut out).await.expect("encode");
out_len = out.len();
std::hint::black_box(&out);
times.push(t.elapsed().as_secs_f64() * 1000.0);
}
println!(
"{:<22} {:>10.2} {:>12.1}",
label,
median(times),
out_len as f64 / 1024.0
);
}
let enc = median(enc_times);
let read = median(read_times);
println!(
"{:<34} {:>10} {:>9}",
"mode (per request)", "ms", "vs BEFORE"
);
println!(
"{:<34} {:>10.2} {:>9}",
"BEFORE on-the-fly Brotli", enc, "1.0x"
);
println!(
"{:<34} {:>10.3} {:>8.0}x",
"AFTER precompressed read",
read,
enc / read
);
println!("\n(BEFORE also holds ~1 tokio task busy for the duration on every request;");
println!(" AFTER additionally ships the deploy-time q11 encoding, usually smaller than");
println!(" the runtime default level.)");
}