perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
@@ -10,7 +10,9 @@
|
||||
//! schema and `docs/plan/drive.md` §3 / §15 for the locked design.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use moka::future::Cache;
|
||||
use sqlx::{PgPool, Row, types::Uuid};
|
||||
|
||||
use crate::domain::entities::drive::{Drive, DriveKind};
|
||||
@@ -18,13 +20,38 @@ use crate::domain::repositories::drive_repository::{
|
||||
DriveRepository, DriveRepositoryError, DriveWithRootName,
|
||||
};
|
||||
|
||||
/// `default_drive_cache` TTL. The default-drive → root-folder binding is
|
||||
/// nearly immutable (changes only on provisioning / drive deletion /
|
||||
/// policy edits — all of which invalidate explicitly below), yet it is
|
||||
/// re-resolved on EVERY NextCloud request (basic-auth chroot), every
|
||||
/// native `/webdav` request (Mode-B scope resolution) and every WOPI
|
||||
/// call. 30 s mirrors `drive_role_cache` in `pg_acl_engine.rs` and bounds
|
||||
/// the one non-invalidated staleness source: a root-folder *rename*,
|
||||
/// which doesn't pass through this repository. Measured in
|
||||
/// `benches/CHROOT-CACHE.md`.
|
||||
const DEFAULT_DRIVE_CACHE_TTL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// One entry per active user; entries are small (a `Drive` + a name).
|
||||
const DEFAULT_DRIVE_CACHE_CAPACITY: u64 = 100_000;
|
||||
|
||||
pub struct DrivePgRepository {
|
||||
pool: Arc<PgPool>,
|
||||
/// user_id → default drive (+ root folder name). See
|
||||
/// [`DEFAULT_DRIVE_CACHE_TTL`]. Only `Ok` results are cached, so the
|
||||
/// provisioning idempotency check (`NotFound` → create) always sees
|
||||
/// the live table.
|
||||
default_drive_cache: Cache<Uuid, DriveWithRootName>,
|
||||
}
|
||||
|
||||
impl DrivePgRepository {
|
||||
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||
Self { pool }
|
||||
Self {
|
||||
pool,
|
||||
default_drive_cache: Cache::builder()
|
||||
.max_capacity(DEFAULT_DRIVE_CACHE_CAPACITY)
|
||||
.time_to_live(DEFAULT_DRIVE_CACHE_TTL)
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
|
||||
fn map_sqlx_err(context: &'static str, e: sqlx::Error) -> DriveRepositoryError {
|
||||
@@ -209,6 +236,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("create_personal_drive_atomic.commit", e))?;
|
||||
|
||||
// Drop any cached default-drive resolution for this user (a stale
|
||||
// NotFound is never cached, but be explicit about the write path).
|
||||
self.default_drive_cache.invalidate(&owner_id).await;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
}
|
||||
|
||||
@@ -394,6 +425,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
tx.commit()
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("delete_atomic.commit", e))?;
|
||||
// We only have the drive id here; the cache is keyed by user.
|
||||
// Deletion is rare — clearing the whole cache is the simple,
|
||||
// always-correct move (repopulates at one query per active user).
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -448,6 +483,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
) -> Result<DriveWithRootName, DriveRepositoryError> {
|
||||
if let Some(cached) = self.default_drive_cache.get(&user_id).await {
|
||||
return Ok(cached);
|
||||
}
|
||||
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT d.id, d.kind, d.default_for_user, d.root_folder_id,
|
||||
@@ -465,7 +504,9 @@ impl DriveRepository for DrivePgRepository {
|
||||
.map_err(|e| Self::map_sqlx_err("find_default_for_user", e))?
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(user_id.to_string()))?;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
let dwr = Self::row_to_drive_with_name(&row)?;
|
||||
self.default_drive_cache.insert(user_id, dwr.clone()).await;
|
||||
Ok(dwr)
|
||||
}
|
||||
|
||||
async fn list_readable_by(
|
||||
@@ -675,6 +716,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
let raw = row
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(drive_id.to_string()))?
|
||||
.0;
|
||||
// Policy edits must not serve a stale `policies` bag from the
|
||||
// default-drive cache (keyed by user, and we only have the drive
|
||||
// id) — clear it; policy edits are admin-rare.
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(crate::domain::entities::drive::DrivePolicies::from_value(
|
||||
&raw,
|
||||
))
|
||||
|
||||
Reference in New Issue
Block a user