perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes

Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
2026-07-16 14:20:20 +00:00
parent b69c18b934
commit aba89c4f5d
52 changed files with 3262 additions and 444 deletions
@@ -10,7 +10,9 @@
//! schema and `docs/plan/drive.md` §3 / §15 for the locked design.
use std::sync::Arc;
use std::time::Duration;
use moka::future::Cache;
use sqlx::{PgPool, Row, types::Uuid};
use crate::domain::entities::drive::{Drive, DriveKind};
@@ -18,13 +20,38 @@ use crate::domain::repositories::drive_repository::{
DriveRepository, DriveRepositoryError, DriveWithRootName,
};
/// `default_drive_cache` TTL. The default-drive → root-folder binding is
/// nearly immutable (changes only on provisioning / drive deletion /
/// policy edits — all of which invalidate explicitly below), yet it is
/// re-resolved on EVERY NextCloud request (basic-auth chroot), every
/// native `/webdav` request (Mode-B scope resolution) and every WOPI
/// call. 30 s mirrors `drive_role_cache` in `pg_acl_engine.rs` and bounds
/// the one non-invalidated staleness source: a root-folder *rename*,
/// which doesn't pass through this repository. Measured in
/// `benches/CHROOT-CACHE.md`.
const DEFAULT_DRIVE_CACHE_TTL: Duration = Duration::from_secs(30);
/// One entry per active user; entries are small (a `Drive` + a name).
const DEFAULT_DRIVE_CACHE_CAPACITY: u64 = 100_000;
pub struct DrivePgRepository {
pool: Arc<PgPool>,
/// user_id → default drive (+ root folder name). See
/// [`DEFAULT_DRIVE_CACHE_TTL`]. Only `Ok` results are cached, so the
/// provisioning idempotency check (`NotFound` → create) always sees
/// the live table.
default_drive_cache: Cache<Uuid, DriveWithRootName>,
}
impl DrivePgRepository {
pub fn new(pool: Arc<PgPool>) -> Self {
Self { pool }
Self {
pool,
default_drive_cache: Cache::builder()
.max_capacity(DEFAULT_DRIVE_CACHE_CAPACITY)
.time_to_live(DEFAULT_DRIVE_CACHE_TTL)
.build(),
}
}
fn map_sqlx_err(context: &'static str, e: sqlx::Error) -> DriveRepositoryError {
@@ -209,6 +236,10 @@ impl DriveRepository for DrivePgRepository {
.await
.map_err(|e| Self::map_sqlx_err("create_personal_drive_atomic.commit", e))?;
// Drop any cached default-drive resolution for this user (a stale
// NotFound is never cached, but be explicit about the write path).
self.default_drive_cache.invalidate(&owner_id).await;
Self::row_to_drive_with_name(&row)
}
@@ -394,6 +425,10 @@ impl DriveRepository for DrivePgRepository {
tx.commit()
.await
.map_err(|e| Self::map_sqlx_err("delete_atomic.commit", e))?;
// We only have the drive id here; the cache is keyed by user.
// Deletion is rare — clearing the whole cache is the simple,
// always-correct move (repopulates at one query per active user).
self.default_drive_cache.invalidate_all();
Ok(())
}
@@ -448,6 +483,10 @@ impl DriveRepository for DrivePgRepository {
&self,
user_id: Uuid,
) -> Result<DriveWithRootName, DriveRepositoryError> {
if let Some(cached) = self.default_drive_cache.get(&user_id).await {
return Ok(cached);
}
let row = sqlx::query(
r#"
SELECT d.id, d.kind, d.default_for_user, d.root_folder_id,
@@ -465,7 +504,9 @@ impl DriveRepository for DrivePgRepository {
.map_err(|e| Self::map_sqlx_err("find_default_for_user", e))?
.ok_or_else(|| DriveRepositoryError::NotFound(user_id.to_string()))?;
Self::row_to_drive_with_name(&row)
let dwr = Self::row_to_drive_with_name(&row)?;
self.default_drive_cache.insert(user_id, dwr.clone()).await;
Ok(dwr)
}
async fn list_readable_by(
@@ -675,6 +716,10 @@ impl DriveRepository for DrivePgRepository {
let raw = row
.ok_or_else(|| DriveRepositoryError::NotFound(drive_id.to_string()))?
.0;
// Policy edits must not serve a stale `policies` bag from the
// default-drive cache (keyed by user, and we only have the drive
// id) — clear it; policy edits are admin-rare.
self.default_drive_cache.invalidate_all();
Ok(crate::domain::entities::drive::DrivePolicies::from_value(
&raw,
))
@@ -186,6 +186,60 @@ impl FaceRepository for FacePgRepository {
Ok(rows.into_iter().map(row_to_face).collect())
}
async fn person_face_stats(&self, user_id: Uuid) -> Result<Vec<(Uuid, i64)>, DomainError> {
// Grouped COUNT — the People tab only needs per-person counts, so
// this replaces a full faces_for_user scan that shipped a 2 KiB
// embedding BYTEA per row (benches/PEOPLE-LIST.md).
let rows: Vec<(Uuid, i64)> = sqlx::query_as(
"SELECT person_id, COUNT(*) FROM faces.faces
WHERE user_id = $1 AND person_id IS NOT NULL
GROUP BY person_id",
)
.bind(user_id)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| db_err("person_face_stats", e))?;
Ok(rows)
}
async fn file_ids_for_faces(
&self,
user_id: Uuid,
face_ids: &[Uuid],
) -> Result<std::collections::HashMap<Uuid, Uuid>, DomainError> {
if face_ids.is_empty() {
return Ok(std::collections::HashMap::new());
}
let rows: Vec<(Uuid, Uuid)> = sqlx::query_as(
"SELECT id, file_id FROM faces.faces WHERE user_id = $1 AND id = ANY($2)",
)
.bind(user_id)
.bind(face_ids)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| db_err("file_ids_for_faces", e))?;
Ok(rows.into_iter().collect())
}
async fn reassign_person_faces(
&self,
user_id: Uuid,
from: Uuid,
into: Uuid,
) -> Result<u64, DomainError> {
let result = sqlx::query(
"UPDATE faces.faces SET person_id = $3
WHERE user_id = $1 AND person_id = $2",
)
.bind(user_id)
.bind(from)
.bind(into)
.execute(self.pool.as_ref())
.await
.map_err(|e| db_err("reassign_person_faces", e))?;
Ok(result.rows_affected())
}
async fn assign_person(
&self,
face_id: Uuid,
@@ -366,6 +366,30 @@ impl FileBlobReadRepository {
.ok_or_else(|| DomainError::not_found("File", file_id))
}
/// Batched variant of [`Self::get_file_drive_id`]: one `= ANY($1)`
/// round-trip for a whole result page. Missing / unknown ids are simply
/// absent from the output (the single-id variant maps them to
/// `NotFound`). Used by `PgAclEngine::check_files_read_batch` — the
/// per-hit loop cost up to 200 sequential point SELECTs per content
/// search (benches/SEARCH-REBAC.md).
pub async fn get_file_drive_ids(
&self,
file_ids: &[uuid::Uuid],
) -> Result<Vec<(uuid::Uuid, uuid::Uuid)>, DomainError> {
if file_ids.is_empty() {
return Ok(Vec::new());
}
sqlx::query_as::<_, (uuid::Uuid, uuid::Uuid)>(
"SELECT id, drive_id FROM storage.files WHERE id = ANY($1)",
)
.bind(file_ids)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| {
DomainError::internal_error("FileBlobRead", format!("drive_id batch lookup: {e}"))
})
}
/// Creates a stub instance for testing — never hits PG.
/// Available in both standard unit-test (`cfg(test)`) and integration
/// (`cfg(integration_tests)`) builds; `PgAclEngine::new_stub` chains
@@ -494,7 +518,24 @@ impl FileBlobReadRepository {
before: Option<i64>,
limit: i64,
) -> Result<(Vec<File>, Vec<i64>, Vec<(Option<i32>, Option<i32>)>), DomainError> {
let rows: Vec<MediaFileRow> = sqlx::query_as(
// Sargable keyset cursor: compare the RAW `media_sort_date` column
// against a timestamptz bind so the planner can use the cursor as
// an index boundary condition on `idx_files_media_timeline_by_drive`.
// The old shape wrapped the column in `EXTRACT(EPOCH …)::bigint`
// (plus an `IS NULL OR` disjunction), which degraded the cursor to
// a per-row Filter: page k re-read and discarded all k·limit rows
// already scrolled past (benches/PHOTOS-CURSOR.md). Since `before`
// is whole seconds, `media_sort_date < to_timestamp(before)` admits
// exactly the same rows as the old truncated comparison. The
// predicate is emitted only when a cursor exists — a bound
// disjunction would block the index condition under generic plans.
let cursor_ts = before.and_then(|s| chrono::DateTime::from_timestamp(s, 0));
let cursor_pred = if cursor_ts.is_some() {
"AND fi.media_sort_date < $2"
} else {
"AND $2::timestamptz IS NULL"
};
let sql = format!(
r#"
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
fi.size, fi.mime_type,
@@ -524,18 +565,18 @@ impl FileBlobReadRepository {
)
AND NOT fi.is_trashed
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
AND ($2::bigint IS NULL
OR EXTRACT(EPOCH FROM fi.media_sort_date)::bigint < $2::bigint)
{cursor_pred}
ORDER BY fi.media_sort_date DESC
LIMIT $3
"#,
)
.bind(caller_id)
.bind(before)
.bind(limit)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
);
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
.bind(caller_id)
.bind(cursor_ts)
.bind(limit)
.fetch_all(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
let mut files = Vec::with_capacity(rows.len());
let mut sort_dates = Vec::with_capacity(rows.len());
@@ -768,62 +809,57 @@ impl FileReadPort for FileBlobReadRepository {
self.resolve_blob_hash(file_id).await
}
/// Paginated file listing — fetches only `limit` rows starting at `offset`.
/// Keyset-paginated file listing in name order — fetches only `limit`
/// rows after `after_name` (exclusive).
///
/// Uses a single SQL query with `LIMIT/OFFSET` to avoid loading the full
/// folder contents into memory. Ideal for streaming WebDAV PROPFIND.
/// Names are unique per folder, so `name > $after` is a total cursor.
/// Served by `idx_files_folder_name (folder_id, name) WHERE NOT
/// is_trashed` as a pure index-range read: O(page) per page with no
/// sort, where the old `LIMIT/OFFSET` shape re-scanned and re-sorted
/// the entire folder for every page (benches/PROPFIND-PAGING.md). The
/// cursor predicate is emitted only when a cursor exists — a
/// `$2 IS NULL OR name > $2` disjunction would block the index
/// condition under the extended protocol's generic plans.
#[allow(clippy::type_complexity)]
async fn list_files_batch(
&self,
folder_id: Option<&str>,
offset: i64,
after_name: Option<&str>,
limit: i64,
) -> Result<Vec<File>, DomainError> {
let rows: Vec<FileRow> = if let Some(fid) = folder_id {
sqlx::query_as(
r#"
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
fi.size, fi.mime_type,
EXTRACT(EPOCH FROM fi.created_at)::bigint,
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
fi.blob_hash,
fi.created_by, fi.updated_by
FROM storage.files fi
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
WHERE fi.folder_id = $1::uuid AND NOT fi.is_trashed
ORDER BY fi.name
LIMIT $2 OFFSET $3
"#,
)
.bind(fid)
.bind(limit)
.bind(offset)
.fetch_all(self.pool.as_ref())
.await
let folder_pred = if folder_id.is_some() {
"fi.folder_id = $1::uuid"
} else {
sqlx::query_as(
r#"
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
fi.size, fi.mime_type,
EXTRACT(EPOCH FROM fi.created_at)::bigint,
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
fi.blob_hash,
"fi.folder_id IS NULL AND $1::uuid IS NULL"
};
let cursor_pred = if after_name.is_some() {
"AND fi.name > $3"
} else {
"AND $3::text IS NULL"
};
let sql = format!(
r#"
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
fi.size, fi.mime_type,
EXTRACT(EPOCH FROM fi.created_at)::bigint,
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
fi.blob_hash,
fi.created_by, fi.updated_by
FROM storage.files fi
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
WHERE fi.folder_id IS NULL AND NOT fi.is_trashed
ORDER BY fi.name
LIMIT $1 OFFSET $2
"#,
)
fi.created_by, fi.updated_by
FROM storage.files fi
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
WHERE {folder_pred} AND NOT fi.is_trashed {cursor_pred}
ORDER BY fi.name
LIMIT $2
"#,
);
let rows: Vec<FileRow> = sqlx::query_as(&sql)
.bind(folder_id)
.bind(limit)
.bind(offset)
.bind(after_name)
.fetch_all(self.pool.as_ref())
.await
}
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
rows.into_iter()
.map(
@@ -122,6 +122,35 @@ impl ShareStoragePort for SharePgRepository {
Self::row_to_entity(&row)
}
async fn increment_access_count(&self, token: &str) -> Result<u64, DomainError> {
// One atomic statement — the relative bump can't lose concurrent
// increments and never rewrites unrelated columns (the legacy
// read-modify-write wrote back item_name/password_hash wholesale,
// silently clobbering concurrent owner edits). The expiry guard
// mirrors find_share_by_token's MIN(expires_at) subquery: NULL =
// never expires.
let result = sqlx::query(
r#"
UPDATE storage.shares s
SET access_count = s.access_count + 1
WHERE s.token = $1
AND COALESCE(
(SELECT MIN(ag.expires_at)
FROM storage.role_grants ag
WHERE ag.subject_type = 'token' AND ag.subject_id = s.id) > NOW(),
TRUE)
"#,
)
.bind(token)
.execute(&*self.db_pool)
.await
.map_err(|e| {
tracing::error!("Database error incrementing share access count: {}", e);
DomainError::internal_error("Share", format!("Failed to register access: {e}"))
})?;
Ok(result.rows_affected())
}
async fn find_share_by_token(&self, token: &str) -> Result<Share, DomainError> {
let row = sqlx::query(
r#"
@@ -85,6 +85,33 @@ impl UserPgRepository {
})
}
/// Fetch only `(storage_used_bytes, storage_quota_bytes)`. Not part of
/// the `UserRepository` trait — called from `StorageUsageService`.
///
/// Same rationale as [`Self::get_user_flags`]: the full-row SELECT drags
/// `image` (a data URI of up to 512 KiB), `password_hash`,
/// `ui_preferences`, … across the wire, and the quota path runs on every
/// folder PROPFIND and every upload quota check just to read two i64s.
/// Measured in `benches/QUOTA-PATH.md`.
pub async fn get_storage_usage(&self, id: Uuid) -> UserRepositoryResult<(i64, i64)> {
let row = sqlx::query(
r#"
SELECT storage_used_bytes, storage_quota_bytes
FROM auth.users
WHERE id = $1
"#,
)
.bind(id)
.fetch_one(&*self.pool)
.await
.map_err(Self::map_sqlx_error)?;
Ok((
row.get("storage_used_bytes"),
row.get("storage_quota_bytes"),
))
}
/// Updates a user's profile image (URL or data URI). Not part of the
/// `UserRepository` trait — called directly from `AuthApplicationService`.
pub async fn update_image(