perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
@@ -29,6 +29,30 @@ pub trait AuthorizationEngine: Send + Sync + 'static {
|
||||
resource: Resource,
|
||||
) -> Result<bool, DomainError>;
|
||||
|
||||
/// Batched `check(subject, Read, File(id))` over a result page: returns
|
||||
/// the subset of `file_ids` the subject may read. Semantically identical
|
||||
/// to looping [`Self::check`] (the default does exactly that); the
|
||||
/// `PgAclEngine` override resolves every file's drive in ONE query and
|
||||
/// reuses the per-drive role cache, so verifying a 200-hit search page
|
||||
/// costs 1 SQL round-trip instead of up to 200 sequential ones
|
||||
/// (benches/SEARCH-REBAC.md).
|
||||
async fn check_files_read_batch(
|
||||
&self,
|
||||
subject: Subject,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashSet<Uuid>, DomainError> {
|
||||
let mut allowed = std::collections::HashSet::with_capacity(file_ids.len());
|
||||
for id in file_ids {
|
||||
if self
|
||||
.check(subject, Permission::Read, Resource::File(*id))
|
||||
.await?
|
||||
{
|
||||
allowed.insert(*id);
|
||||
}
|
||||
}
|
||||
Ok(allowed)
|
||||
}
|
||||
|
||||
/// Convenience wrapper around `check`: returns `Ok(())` when allowed and
|
||||
/// `DomainError::not_found` when denied (anti-enumeration — same error as
|
||||
/// "resource doesn't exist" so attackers can't probe IDs by error shape).
|
||||
|
||||
@@ -39,6 +39,23 @@ pub trait FaceRepository: Send + Sync + 'static {
|
||||
user_id: Uuid,
|
||||
blob_hash: &str,
|
||||
) -> Result<Vec<Face>, DomainError>;
|
||||
/// `(person_id, face_count)` per non-empty cluster — a grouped COUNT
|
||||
/// instead of dragging every face row (each with a 2 KiB embedding
|
||||
/// BYTEA) across the wire just to count them. See benches/PEOPLE-LIST.md.
|
||||
async fn person_face_stats(&self, user_id: Uuid) -> Result<Vec<(Uuid, i64)>, DomainError>;
|
||||
/// face id → file id for the given faces (cover-photo resolution).
|
||||
async fn file_ids_for_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
face_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashMap<Uuid, Uuid>, DomainError>;
|
||||
/// Reassign every face of `from` to `into` in one statement (merge).
|
||||
async fn reassign_person_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
from: Uuid,
|
||||
into: Uuid,
|
||||
) -> Result<u64, DomainError>;
|
||||
async fn assign_person(
|
||||
&self,
|
||||
face_id: Uuid,
|
||||
|
||||
@@ -235,13 +235,14 @@ pub trait FileRetrievalUseCase: Send + Sync + 'static {
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
let all = self.list_files(folder_id).await?;
|
||||
let mut all = self.list_files(folder_id).await?;
|
||||
all.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.skip(offset as usize)
|
||||
.filter(|f| after_name.is_none_or(|a| f.name.as_str() > a))
|
||||
.take(limit as usize)
|
||||
.collect())
|
||||
}
|
||||
@@ -257,10 +258,10 @@ pub trait FileRetrievalUseCase: Send + Sync + 'static {
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
_owner_id: Uuid,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
self.list_files_batch(folder_id, offset, limit).await
|
||||
self.list_files_batch(folder_id, after_name, limit).await
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -99,6 +99,28 @@ pub trait ShareStoragePort: Send + Sync + 'static {
|
||||
share: &crate::domain::entities::share::Share,
|
||||
) -> Result<crate::domain::entities::share::Share, DomainError>;
|
||||
|
||||
/// Atomically bump a link's access counter (public share landing).
|
||||
/// Returns the number of rows updated — 0 means "no live share for
|
||||
/// this token" (missing OR expired).
|
||||
///
|
||||
/// The default is the legacy read-modify-write (kept for test mocks);
|
||||
/// `SharePgRepository` overrides it with a single `UPDATE … SET
|
||||
/// access_count = access_count + 1`, replacing 2 correlated-subquery
|
||||
/// round-trips per anonymous visit with 1 and removing the lost-update
|
||||
/// race between concurrent visitors (benches/SHARE-ACCESS.md).
|
||||
async fn increment_access_count(&self, token: &str) -> Result<u64, DomainError> {
|
||||
let share = match self.find_share_by_token(token).await {
|
||||
Ok(s) => s,
|
||||
Err(e) if e.kind == crate::common::errors::ErrorKind::NotFound => return Ok(0),
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
if share.is_expired() {
|
||||
return Ok(0);
|
||||
}
|
||||
self.update_share(&share.increment_access_count()).await?;
|
||||
Ok(1)
|
||||
}
|
||||
|
||||
async fn find_shares_by_user(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
|
||||
@@ -107,20 +107,30 @@ pub trait FileReadPort: Send + Sync + 'static {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// Lists files in a folder with LIMIT/OFFSET pagination.
|
||||
/// Lists files in a folder in name order, keyset-paginated.
|
||||
///
|
||||
/// Used by streaming WebDAV PROPFIND to avoid loading all files at once.
|
||||
/// `after_name` is the last name of the previous page (`None` = first
|
||||
/// page); names are unique within a folder (unique index on
|
||||
/// `(drive_id, folder_id, name)`), so `name > after_name` is a total,
|
||||
/// stable cursor. Unlike LIMIT/OFFSET, every page is O(page) — the old
|
||||
/// offset shape re-scanned and re-sorted the whole folder per page
|
||||
/// (benches/PROPFIND-PAGING.md).
|
||||
///
|
||||
/// Default: falls back to `list_files` (loads all, then slices in memory).
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<File>, DomainError> {
|
||||
let all = self.list_files(folder_id).await?;
|
||||
let start = (offset as usize).min(all.len());
|
||||
let end = (start + limit as usize).min(all.len());
|
||||
Ok(all.into_iter().skip(start).take(end - start).collect())
|
||||
let mut all = self.list_files(folder_id).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit as usize)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Streams every file in the subtree rooted at `folder_id`.
|
||||
|
||||
Reference in New Issue
Block a user