perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes

Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w

src/infrastructure/services/dedup_service.rs

@@ -240,6 +240,16 @@ impl Drop for IngestGuard {
 /// in the [`BlobStorageBackend`], and maintains a manifest in PostgreSQL
 /// mapping file_hash → \[chunk_hashes\].  BLAKE3 hashing, ref-counting
 /// and the PostgreSQL dedup index all live here.
+/// Immutable chunk map of one CDC blob (`storage.chunk_manifests` row,
+/// minus the mutable `ref_count`). Content-addressed: for a given
+/// `file_hash` the chunk list and total size never change, which is what
+/// makes [`DedupService::manifest_cached`] safe.
+pub struct ChunkManifest {
+    pub chunk_hashes: Vec<String>,
+    pub chunk_sizes: Vec<i64>,
+    pub total_size: i64,
+}
+
 pub struct DedupService {
     /// Pluggable blob storage backend (local FS, S3, …).
     backend: Arc<dyn BlobStorageBackend>,
@@ -251,6 +261,13 @@ pub struct DedupService {
     maintenance_pool: Arc<PgPool>,
     /// Single lifecycle dispatcher — fired on blob created / deleted.
     blob_lifecycle: Option<Arc<BlobLifecycleService>>,
+    /// `file_hash → ChunkManifest` for the read path — every stream / range
+    /// / full read of a CDC blob used to pay one manifest query first, even
+    /// for the media the gallery re-reads constantly. Positive-only (a
+    /// legacy blob gaining a manifest via background rechunking must be
+    /// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
+    /// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
+    manifest_cache: moka::future::Cache<String, Arc<ChunkManifest>>,
 }
 
 impl DedupService {
@@ -269,9 +286,22 @@ impl DedupService {
             pool,
             maintenance_pool,
             blob_lifecycle: None,
+            manifest_cache: Self::build_manifest_cache(),
         }
     }
 
+    /// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
+    /// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
+    fn build_manifest_cache() -> moka::future::Cache<String, Arc<ChunkManifest>> {
+        moka::future::Cache::builder()
+            .weigher(|key: &String, value: &Arc<ChunkManifest>| {
+                (key.len() + value.chunk_hashes.len() * 80 + 64) as u32
+            })
+            .max_capacity(32 * 1024 * 1024)
+            .time_to_live(std::time::Duration::from_secs(60))
+            .build()
+    }
+
     /// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
     pub fn with_blob_lifecycle(mut self, lifecycle: Arc<BlobLifecycleService>) -> Self {
         self.blob_lifecycle = Some(lifecycle);
@@ -311,6 +341,7 @@ impl DedupService {
             pool: stub_pool.clone(),
             maintenance_pool: stub_pool,
             blob_lifecycle: None,
+            manifest_cache: Self::build_manifest_cache(),
         }
     }
 
@@ -1385,6 +1416,10 @@ impl DedupService {
                 .await
                 .map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?;
 
+            // Post-commit so a concurrent read can't re-cache the manifest
+            // between invalidation and the delete becoming visible.
+            self.manifest_cache.invalidate(file_hash).await;
+
             // File content is gone — drop its blob-keyed thumbnails now.
             self.fire_blob_hooks(file_hash);
 
@@ -1606,27 +1641,49 @@ impl DedupService {
         Box::pin(chunk_stream)
     }
 
+    /// Cached manifest fetch for the read path (see the `manifest_cache`
+    /// field docs). `None` = legacy whole-file blob — never cached, so a
+    /// background rechunk that creates a manifest is honoured immediately.
+    async fn manifest_cached(&self, hash: &str) -> Result<Option<Arc<ChunkManifest>>, DomainError> {
+        if let Some(m) = self.manifest_cache.get(hash).await {
+            return Ok(Some(m));
+        }
+        let row = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
+            "SELECT chunk_hashes, chunk_sizes, total_size
+             FROM storage.chunk_manifests WHERE file_hash = $1",
+        )
+        .bind(hash)
+        .fetch_optional(self.pool.as_ref())
+        .await
+        .map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
+        match row {
+            Some((chunk_hashes, chunk_sizes, total_size)) => {
+                let m = Arc::new(ChunkManifest {
+                    chunk_hashes,
+                    chunk_sizes,
+                    total_size,
+                });
+                self.manifest_cache
+                    .insert(hash.to_string(), m.clone())
+                    .await;
+                Ok(Some(m))
+            }
+            None => Ok(None),
+        }
+    }
+
     /// Stream blob content — CDC-aware with legacy fallback.
     ///
-    /// For CDC files: looks up the manifest, then streams chunks in order,
-    /// concatenating them into a single byte stream.
+    /// For CDC files: looks up the manifest (RAM-cached), then streams
+    /// chunks in order, concatenating them into a single byte stream.
     /// For legacy blobs: delegates directly to the backend.
     pub async fn read_blob_stream(
         &self,
         hash: &str,
     ) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
     {
-        // Check manifest
-        let manifest = sqlx::query_scalar::<_, Vec<String>>(
-            "SELECT chunk_hashes FROM storage.chunk_manifests WHERE file_hash = $1",
-        )
-        .bind(hash)
-        .fetch_optional(self.pool.as_ref())
-        .await
-        .map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
-
-        match manifest {
-            Some(chunk_hashes) => Ok(self.stream_chunks(chunk_hashes)),
+        match self.manifest_cached(hash).await? {
+            Some(m) => Ok(self.stream_chunks(m.chunk_hashes.clone())),
             // Legacy whole-file blob
             None => self.backend.get_blob_stream(hash).await,
         }
@@ -1644,18 +1701,11 @@ impl DedupService {
     /// `blob_size` + `read_blob_stream`) doubled the manifest round-trips on
     /// every full-blob read (e.g. 2N queries for an N-image gallery cold load).
     pub async fn read_blob_bytes(&self, hash: &str) -> Result<Bytes, DomainError> {
-        let manifest = sqlx::query_as::<_, (Vec<String>, i64)>(
-            "SELECT chunk_hashes, total_size FROM storage.chunk_manifests WHERE file_hash = $1",
-        )
-        .bind(hash)
-        .fetch_optional(self.pool.as_ref())
-        .await
-        .map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
-
-        let (mut stream, expected_size) = match manifest {
-            Some((chunk_hashes, total_size)) => {
-                (self.stream_chunks(chunk_hashes), total_size.max(0) as usize)
-            }
+        let (mut stream, expected_size) = match self.manifest_cached(hash).await? {
+            Some(m) => (
+                self.stream_chunks(m.chunk_hashes.clone()),
+                m.total_size.max(0) as usize,
+            ),
             None => {
                 // Legacy whole-file blob: size + stream straight from the backend.
                 let size = self.backend.blob_size(hash).await? as usize;
@@ -1685,17 +1735,9 @@ impl DedupService {
         end: Option<u64>,
     ) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
     {
-        // Check manifest
-        let manifest = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
-            "SELECT chunk_hashes, chunk_sizes, total_size
-             FROM storage.chunk_manifests WHERE file_hash = $1",
-        )
-        .bind(hash)
-        .fetch_optional(self.pool.as_ref())
-        .await
-        .map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
-
-        if let Some((chunk_hashes, chunk_sizes, total_size)) = manifest {
+        if let Some(m) = self.manifest_cached(hash).await? {
+            let (chunk_hashes, chunk_sizes, total_size) =
+                (&m.chunk_hashes, &m.chunk_sizes, m.total_size);
             let end = end.unwrap_or(total_size as u64);
 
             // Calculate which chunks overlap [start, end)
@@ -1749,17 +1791,9 @@ impl DedupService {
 
     /// Get blob size — manifest-aware with legacy fallback.
     pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
-        // Check manifest first (O(1) from PG)
-        let manifest_size = sqlx::query_scalar::<_, i64>(
-            "SELECT total_size FROM storage.chunk_manifests WHERE file_hash = $1",
-        )
-        .bind(hash)
-        .fetch_optional(self.pool.as_ref())
-        .await
-        .map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
-
-        if let Some(size) = manifest_size {
-            return Ok(size as u64);
+        // Check manifest first (RAM cache, else one O(1) PG row)
+        if let Some(m) = self.manifest_cached(hash).await? {
+            return Ok(m.total_size as u64);
         }
 
         // Legacy: delegate to backend
@@ -2048,6 +2082,7 @@ impl DedupService {
             }
 
             for (file_hash, chunk_hashes, size) in &batch {
+                self.manifest_cache.invalidate(file_hash).await;
                 // Decrement chunk ref_counts. GREATEST(.., 0) guards against the
                 // single-chunk file case where the PG file-delete trigger already
                 // decremented blobs.ref_count (because file_hash == chunk_hash);