perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -836,10 +836,8 @@ impl AppServiceFactory {
|
||||
let service = Arc::new(
|
||||
TrashService::new(
|
||||
trash_repo.clone(),
|
||||
repos.file_read_repository.clone(),
|
||||
repos.file_write_repository.clone(),
|
||||
repos.folder_repository.clone(),
|
||||
self.config.storage.trash_retention_days,
|
||||
core.dedup_service.clone(),
|
||||
Some(core.file_content_cache.clone()),
|
||||
authz.clone(),
|
||||
|
||||
@@ -108,10 +108,117 @@ pub async fn refine_content_type_from_file(
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a MIME type identifies content that is already compressed, so
|
||||
/// running Deflate over it burns CPU for ~0 % size gain.
|
||||
///
|
||||
/// Used by the ZIP export paths (`ZipService`, `BatchOperations`) to pick
|
||||
/// `Compression::Stored` per entry instead of deflating JPEG/MP4/… bytes.
|
||||
/// The set mirrors the HTTP `CompressionLayer` exclusion list in `main.rs`
|
||||
/// (keep the two in sync), minus entries that are containers of possibly
|
||||
/// incompressible data rather than compressed formats themselves
|
||||
/// (`application/x-tar`, `application/octet-stream`) — those stay on Deflate
|
||||
/// so unknown-but-compressible content is never stored uncompressed.
|
||||
pub fn is_precompressed_mime(mime: &str) -> bool {
|
||||
// Strip any parameters ("; charset=…") and normalize case.
|
||||
let essence = mime.split(';').next().unwrap_or(mime).trim();
|
||||
|
||||
// Compressed families: every common video/audio codec container.
|
||||
if essence.starts_with("video/") || essence.starts_with("audio/") {
|
||||
return true;
|
||||
}
|
||||
// Zip-based document bundles (docx/xlsx/pptx, odt/ods/odp, …).
|
||||
if essence.starts_with("application/vnd.openxmlformats-officedocument")
|
||||
|| essence.starts_with("application/vnd.oasis.opendocument")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
matches!(
|
||||
essence,
|
||||
// Raster images with built-in compression (SVG intentionally absent).
|
||||
"image/jpeg"
|
||||
| "image/png"
|
||||
| "image/gif"
|
||||
| "image/webp"
|
||||
| "image/avif"
|
||||
| "image/heic"
|
||||
| "image/heif"
|
||||
| "image/jp2"
|
||||
// Already-compressed web fonts; ttf/otf left compressible.
|
||||
| "font/woff"
|
||||
| "font/woff2"
|
||||
| "application/font-woff"
|
||||
// Archives & compressed containers.
|
||||
| "application/zip"
|
||||
| "application/gzip"
|
||||
| "application/x-gzip"
|
||||
| "application/x-7z-compressed"
|
||||
| "application/x-rar-compressed"
|
||||
| "application/x-bzip2"
|
||||
| "application/zstd"
|
||||
| "application/x-xz"
|
||||
| "application/epub+zip"
|
||||
| "application/java-archive"
|
||||
| "application/vnd.android.package-archive"
|
||||
// PDF: internal streams are usually already deflated.
|
||||
| "application/pdf"
|
||||
)
|
||||
}
|
||||
|
||||
/// ZIP entry compression for a file of the given MIME type: `Stored` for
|
||||
/// already-compressed content, `Deflate` otherwise. Shared by every ZIP
|
||||
/// export path (`ZipService`, `BatchOperations`).
|
||||
pub fn zip_entry_compression(mime: &str) -> async_zip::Compression {
|
||||
if is_precompressed_mime(mime) {
|
||||
async_zip::Compression::Stored
|
||||
} else {
|
||||
async_zip::Compression::Deflate
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// ── is_precompressed_mime ───────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn media_and_archives_are_precompressed() {
|
||||
for mime in [
|
||||
"image/jpeg",
|
||||
"image/webp",
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"audio/mpeg",
|
||||
"application/zip",
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"font/woff2",
|
||||
] {
|
||||
assert!(is_precompressed_mime(mime), "{mime} should be Stored");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compressible_types_keep_deflate() {
|
||||
for mime in [
|
||||
"text/plain",
|
||||
"text/html",
|
||||
"application/json",
|
||||
"image/svg+xml",
|
||||
"application/x-tar",
|
||||
"application/octet-stream",
|
||||
"",
|
||||
] {
|
||||
assert!(!is_precompressed_mime(mime), "{mime} should stay Deflate");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mime_parameters_are_ignored() {
|
||||
assert!(is_precompressed_mime("image/jpeg; charset=binary"));
|
||||
}
|
||||
|
||||
// ── refine_content_type (sync) ──────────────────────────────
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user