perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -9,7 +9,7 @@
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
"build": "vite build",
|
||||
"postbuild": "node scripts/emit-askama-common.mjs",
|
||||
"postbuild": "node scripts/emit-askama-common.mjs && node scripts/precompress.mjs ../static-dist",
|
||||
"preview": "vite preview",
|
||||
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json && eslint . && stylelint \"src/**/*.{css,svelte}\" && prettier --check .",
|
||||
"check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch",
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
// Precompress built SPA assets so the Rust web layer can serve them with
|
||||
// `ServeDir::precompressed_br()/precompressed_gzip()` instead of re-running
|
||||
// Brotli over the same immutable bundle on every request (the tower-http
|
||||
// CompressionLayer stays as the on-the-fly fallback for anything without a
|
||||
// sibling). Runs as the `build` script's final step; uses only node:zlib —
|
||||
// no dependencies. See benches/STATIC-PRECOMPRESSED.md for the measured win.
|
||||
import { promises as fs } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import zlib from 'node:zlib';
|
||||
|
||||
const OUT_DIR = process.argv[2] ?? '../static-dist';
|
||||
// Compressible text assets; media formats are already compressed.
|
||||
const EXTENSIONS = new Set([
|
||||
'.js',
|
||||
'.mjs',
|
||||
'.css',
|
||||
'.html',
|
||||
'.svg',
|
||||
'.json',
|
||||
'.txt',
|
||||
'.xml',
|
||||
'.map',
|
||||
'.webmanifest'
|
||||
]);
|
||||
// Below this size the encoding overhead outweighs the transfer win
|
||||
// (mirrors the server's SizeAbove(256) predicate).
|
||||
const MIN_BYTES = 256;
|
||||
|
||||
async function* walk(dir) {
|
||||
for (const entry of await fs.readdir(dir, { withFileTypes: true })) {
|
||||
const p = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) yield* walk(p);
|
||||
else yield p;
|
||||
}
|
||||
}
|
||||
|
||||
let files = 0;
|
||||
let inBytes = 0;
|
||||
let brBytes = 0;
|
||||
for await (const file of walk(OUT_DIR)) {
|
||||
if (!EXTENSIONS.has(path.extname(file))) continue;
|
||||
const data = await fs.readFile(file);
|
||||
if (data.length < MIN_BYTES) continue;
|
||||
const br = zlib.brotliCompressSync(data, {
|
||||
params: {
|
||||
[zlib.constants.BROTLI_PARAM_QUALITY]: 11,
|
||||
[zlib.constants.BROTLI_PARAM_SIZE_HINT]: data.length
|
||||
}
|
||||
});
|
||||
const gz = zlib.gzipSync(data, { level: 9 });
|
||||
// Only keep siblings that actually shrink the asset.
|
||||
if (br.length < data.length) await fs.writeFile(`${file}.br`, br);
|
||||
if (gz.length < data.length) await fs.writeFile(`${file}.gz`, gz);
|
||||
files += 1;
|
||||
inBytes += data.length;
|
||||
brBytes += Math.min(br.length, data.length);
|
||||
}
|
||||
console.log(
|
||||
`precompress: ${files} assets, ${(inBytes / 1024).toFixed(0)} KiB → ${(brBytes / 1024).toFixed(0)} KiB brotli (${inBytes ? ((1 - brBytes / inBytes) * 100).toFixed(0) : 0}% smaller)`
|
||||
);
|
||||
@@ -108,7 +108,19 @@ export async function getFolder(id: string): Promise<FolderItem> {
|
||||
*/
|
||||
export async function fetchFolderListing(
|
||||
folderId: string,
|
||||
opts: { etag?: string; forceRefresh?: boolean } = {}
|
||||
opts: {
|
||||
etag?: string;
|
||||
forceRefresh?: boolean;
|
||||
/**
|
||||
* Progressive render hook: invoked after EVERY page with the
|
||||
* accumulated listing so far (the arrays are fresh copies — safe to
|
||||
* hand to reactive state). Without it, a 2,000-item folder waited
|
||||
* for all ⌈N/200⌉ sequential round-trips before the first row
|
||||
* painted; with it the view paints after page one (~200 items) and
|
||||
* fills in as the tail pages land.
|
||||
*/
|
||||
onPage?: (partial: FolderListing, done: boolean) => void;
|
||||
} = {}
|
||||
): Promise<FolderListingResult> {
|
||||
const folders: FolderItem[] = [];
|
||||
const files: FileItem[] = [];
|
||||
@@ -132,6 +144,10 @@ export async function fetchFolderListing(
|
||||
else files.push(it.resource as FileItem);
|
||||
}
|
||||
cursor = page.next_cursor;
|
||||
opts.onPage?.(
|
||||
{ folders: [...folders], files: [...files], favoriteIds: [], sharedIds: [] },
|
||||
!cursor
|
||||
);
|
||||
} while (cursor);
|
||||
|
||||
return { status: 200, listing: { folders, files, favoriteIds: [], sharedIds: [] } };
|
||||
|
||||
@@ -308,7 +308,22 @@
|
||||
});
|
||||
|
||||
try {
|
||||
const res = await fetchFolderListing(folderId, { etag: cached?.etag });
|
||||
const res = await fetchFolderListing(folderId, {
|
||||
etag: cached?.etag,
|
||||
// Paint page one (~200 items) immediately instead of waiting
|
||||
// for every sequential page of a large folder; later pages
|
||||
// extend the view as they land. Skip when a cached copy is
|
||||
// already on screen — replacing it with a partial list would
|
||||
// briefly shrink the view.
|
||||
onPage: cached
|
||||
? undefined
|
||||
: (partial, done) => {
|
||||
if (seq !== loadSeq || done) return; // final state applied below
|
||||
applyListing(partial);
|
||||
loading = false;
|
||||
showSkeleton = false;
|
||||
}
|
||||
});
|
||||
if (seq !== loadSeq) return; // superseded by a newer navigation
|
||||
if (res.status === 200 && res.listing) {
|
||||
applyListing(res.listing);
|
||||
|
||||
Reference in New Issue
Block a user