perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -0,0 +1,213 @@
|
||||
//! People-tab benchmark — full faces scan (embeddings included) vs grouped COUNT.
|
||||
//!
|
||||
//! `PeopleService::list_people` used to call `faces_for_user`, dragging every
|
||||
//! face row — each with a 2,048-byte embedding BYTEA — across the wire and
|
||||
//! decoding it into a fresh `Vec<f32>`, only to (a) count faces per person and
|
||||
//! (b) resolve ~a-handful of cover faces to file ids. The change replaces it
|
||||
//! with `person_face_stats` (grouped COUNT) + `file_ids_for_faces` (one
|
||||
//! `= ANY` over just the cover ids).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_people_list
|
||||
//! Tunables: BENCH_FACES (10000), BENCH_PERSONS (20), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Seeded {
|
||||
user_id: Uuid,
|
||||
drive_id: Uuid,
|
||||
cover_ids: Vec<Uuid>,
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, faces: usize, persons: usize) -> Seeded {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let user_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role)
|
||||
VALUES ('bench_people', 'bench_people@bench.invalid', 'user') RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("user");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, default_for_user) VALUES ('personal', $1) RETURNING id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_people', '/bench_people', 'bench_people', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp root");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Photo files the faces point at.
|
||||
let file_ids: Vec<Uuid> = sqlx::query_scalar(
|
||||
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
SELECT 'p' || i, $1, 'benchpeople0000000000000000000000000000000000000000000000000000',
|
||||
1024, 'image/jpeg', $2
|
||||
FROM generate_series(1, $3) AS i
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(faces as i32)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
|
||||
// Persons + faces (2 KiB embedding each, like the real 512×f32).
|
||||
let mut person_ids = Vec::with_capacity(persons);
|
||||
for i in 0..persons {
|
||||
let pid: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO faces.persons (user_id, display_name) VALUES ($1, $2) RETURNING id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(format!("Person {i}"))
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("person");
|
||||
person_ids.push(pid);
|
||||
}
|
||||
|
||||
let embedding = vec![0u8; 2048];
|
||||
let mut cover_ids = Vec::with_capacity(persons);
|
||||
for (i, file_id) in file_ids.iter().enumerate() {
|
||||
let pid = person_ids[i % persons];
|
||||
let face_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO faces.faces
|
||||
(file_id, user_id, person_id, bbox, det_score, quality, embedding, blob_hash)
|
||||
VALUES ($1, $2, $3, ARRAY[0.1,0.1,0.2,0.2]::real[], 0.99, 0.9, $4,
|
||||
'benchpeople0000000000000000000000000000000000000000000000000000')
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(file_id)
|
||||
.bind(user_id)
|
||||
.bind(pid)
|
||||
.bind(&embedding)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("face");
|
||||
if i < persons {
|
||||
cover_ids.push(face_id);
|
||||
}
|
||||
}
|
||||
sqlx::query("ANALYZE faces.faces").execute(pool).await.ok();
|
||||
|
||||
Seeded {
|
||||
user_id,
|
||||
drive_id,
|
||||
cover_ids,
|
||||
}
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, s: &Seeded) {
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(s.drive_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(s.user_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let faces: usize = env_or("BENCH_FACES", 10_000);
|
||||
let persons: usize = env_or("BENCH_PERSONS", 20);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {faces} faces / {persons} persons (one-time)…");
|
||||
let seeded = seed(&pool, faces, persons).await;
|
||||
|
||||
println!(
|
||||
"\n# GET /api/people data fetch: BEFORE (full face rows) vs AFTER (COUNT + cover ANY)"
|
||||
);
|
||||
println!("{:<28} {:>12} {:>14}", "mode", "total ms", "bytes moved");
|
||||
|
||||
let mut base = None;
|
||||
for mode in ["BEFORE full-rows", "AFTER count+covers"] {
|
||||
let mut times = Vec::with_capacity(reps);
|
||||
let mut bytes = 0usize;
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
if mode.starts_with("BEFORE") {
|
||||
// faces_for_user shape: every column incl. embedding.
|
||||
let rows: Vec<(Uuid, Uuid, Option<Uuid>, Vec<u8>)> = sqlx::query_as(
|
||||
"SELECT id, file_id, person_id, embedding FROM faces.faces WHERE user_id = $1",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("full rows");
|
||||
bytes = rows.iter().map(|r| r.3.len() + 48).sum();
|
||||
assert_eq!(rows.len(), faces);
|
||||
} else {
|
||||
let stats: Vec<(Uuid, i64)> = sqlx::query_as(
|
||||
"SELECT person_id, COUNT(*) FROM faces.faces
|
||||
WHERE user_id = $1 AND person_id IS NOT NULL GROUP BY person_id",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("stats");
|
||||
let covers: Vec<(Uuid, Uuid)> = sqlx::query_as(
|
||||
"SELECT id, file_id FROM faces.faces WHERE user_id = $1 AND id = ANY($2)",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.bind(&seeded.cover_ids)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("covers");
|
||||
bytes = (stats.len() + covers.len()) * 32;
|
||||
assert_eq!(stats.len(), persons);
|
||||
}
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
let ms = median(times);
|
||||
let speedup = base
|
||||
.map(|b: f64| format!("({:.1}x)", b / ms))
|
||||
.unwrap_or_default();
|
||||
println!("{mode:<28} {ms:>12.2} {bytes:>14} {speedup}");
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
}
|
||||
|
||||
cleanup(&pool, &seeded).await;
|
||||
}
|
||||
Reference in New Issue
Block a user