perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -10,7 +10,9 @@
|
||||
//! schema and `docs/plan/drive.md` §3 / §15 for the locked design.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use moka::future::Cache;
|
||||
use sqlx::{PgPool, Row, types::Uuid};
|
||||
|
||||
use crate::domain::entities::drive::{Drive, DriveKind};
|
||||
@@ -18,13 +20,38 @@ use crate::domain::repositories::drive_repository::{
|
||||
DriveRepository, DriveRepositoryError, DriveWithRootName,
|
||||
};
|
||||
|
||||
/// `default_drive_cache` TTL. The default-drive → root-folder binding is
|
||||
/// nearly immutable (changes only on provisioning / drive deletion /
|
||||
/// policy edits — all of which invalidate explicitly below), yet it is
|
||||
/// re-resolved on EVERY NextCloud request (basic-auth chroot), every
|
||||
/// native `/webdav` request (Mode-B scope resolution) and every WOPI
|
||||
/// call. 30 s mirrors `drive_role_cache` in `pg_acl_engine.rs` and bounds
|
||||
/// the one non-invalidated staleness source: a root-folder *rename*,
|
||||
/// which doesn't pass through this repository. Measured in
|
||||
/// `benches/CHROOT-CACHE.md`.
|
||||
const DEFAULT_DRIVE_CACHE_TTL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// One entry per active user; entries are small (a `Drive` + a name).
|
||||
const DEFAULT_DRIVE_CACHE_CAPACITY: u64 = 100_000;
|
||||
|
||||
pub struct DrivePgRepository {
|
||||
pool: Arc<PgPool>,
|
||||
/// user_id → default drive (+ root folder name). See
|
||||
/// [`DEFAULT_DRIVE_CACHE_TTL`]. Only `Ok` results are cached, so the
|
||||
/// provisioning idempotency check (`NotFound` → create) always sees
|
||||
/// the live table.
|
||||
default_drive_cache: Cache<Uuid, DriveWithRootName>,
|
||||
}
|
||||
|
||||
impl DrivePgRepository {
|
||||
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||
Self { pool }
|
||||
Self {
|
||||
pool,
|
||||
default_drive_cache: Cache::builder()
|
||||
.max_capacity(DEFAULT_DRIVE_CACHE_CAPACITY)
|
||||
.time_to_live(DEFAULT_DRIVE_CACHE_TTL)
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
|
||||
fn map_sqlx_err(context: &'static str, e: sqlx::Error) -> DriveRepositoryError {
|
||||
@@ -209,6 +236,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("create_personal_drive_atomic.commit", e))?;
|
||||
|
||||
// Drop any cached default-drive resolution for this user (a stale
|
||||
// NotFound is never cached, but be explicit about the write path).
|
||||
self.default_drive_cache.invalidate(&owner_id).await;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
}
|
||||
|
||||
@@ -394,6 +425,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
tx.commit()
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("delete_atomic.commit", e))?;
|
||||
// We only have the drive id here; the cache is keyed by user.
|
||||
// Deletion is rare — clearing the whole cache is the simple,
|
||||
// always-correct move (repopulates at one query per active user).
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -448,6 +483,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
) -> Result<DriveWithRootName, DriveRepositoryError> {
|
||||
if let Some(cached) = self.default_drive_cache.get(&user_id).await {
|
||||
return Ok(cached);
|
||||
}
|
||||
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT d.id, d.kind, d.default_for_user, d.root_folder_id,
|
||||
@@ -465,7 +504,9 @@ impl DriveRepository for DrivePgRepository {
|
||||
.map_err(|e| Self::map_sqlx_err("find_default_for_user", e))?
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(user_id.to_string()))?;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
let dwr = Self::row_to_drive_with_name(&row)?;
|
||||
self.default_drive_cache.insert(user_id, dwr.clone()).await;
|
||||
Ok(dwr)
|
||||
}
|
||||
|
||||
async fn list_readable_by(
|
||||
@@ -675,6 +716,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
let raw = row
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(drive_id.to_string()))?
|
||||
.0;
|
||||
// Policy edits must not serve a stale `policies` bag from the
|
||||
// default-drive cache (keyed by user, and we only have the drive
|
||||
// id) — clear it; policy edits are admin-rare.
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(crate::domain::entities::drive::DrivePolicies::from_value(
|
||||
&raw,
|
||||
))
|
||||
|
||||
@@ -186,6 +186,60 @@ impl FaceRepository for FacePgRepository {
|
||||
Ok(rows.into_iter().map(row_to_face).collect())
|
||||
}
|
||||
|
||||
async fn person_face_stats(&self, user_id: Uuid) -> Result<Vec<(Uuid, i64)>, DomainError> {
|
||||
// Grouped COUNT — the People tab only needs per-person counts, so
|
||||
// this replaces a full faces_for_user scan that shipped a 2 KiB
|
||||
// embedding BYTEA per row (benches/PEOPLE-LIST.md).
|
||||
let rows: Vec<(Uuid, i64)> = sqlx::query_as(
|
||||
"SELECT person_id, COUNT(*) FROM faces.faces
|
||||
WHERE user_id = $1 AND person_id IS NOT NULL
|
||||
GROUP BY person_id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("person_face_stats", e))?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
async fn file_ids_for_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
face_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashMap<Uuid, Uuid>, DomainError> {
|
||||
if face_ids.is_empty() {
|
||||
return Ok(std::collections::HashMap::new());
|
||||
}
|
||||
let rows: Vec<(Uuid, Uuid)> = sqlx::query_as(
|
||||
"SELECT id, file_id FROM faces.faces WHERE user_id = $1 AND id = ANY($2)",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(face_ids)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("file_ids_for_faces", e))?;
|
||||
Ok(rows.into_iter().collect())
|
||||
}
|
||||
|
||||
async fn reassign_person_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
from: Uuid,
|
||||
into: Uuid,
|
||||
) -> Result<u64, DomainError> {
|
||||
let result = sqlx::query(
|
||||
"UPDATE faces.faces SET person_id = $3
|
||||
WHERE user_id = $1 AND person_id = $2",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(from)
|
||||
.bind(into)
|
||||
.execute(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("reassign_person_faces", e))?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
async fn assign_person(
|
||||
&self,
|
||||
face_id: Uuid,
|
||||
|
||||
@@ -366,6 +366,30 @@ impl FileBlobReadRepository {
|
||||
.ok_or_else(|| DomainError::not_found("File", file_id))
|
||||
}
|
||||
|
||||
/// Batched variant of [`Self::get_file_drive_id`]: one `= ANY($1)`
|
||||
/// round-trip for a whole result page. Missing / unknown ids are simply
|
||||
/// absent from the output (the single-id variant maps them to
|
||||
/// `NotFound`). Used by `PgAclEngine::check_files_read_batch` — the
|
||||
/// per-hit loop cost up to 200 sequential point SELECTs per content
|
||||
/// search (benches/SEARCH-REBAC.md).
|
||||
pub async fn get_file_drive_ids(
|
||||
&self,
|
||||
file_ids: &[uuid::Uuid],
|
||||
) -> Result<Vec<(uuid::Uuid, uuid::Uuid)>, DomainError> {
|
||||
if file_ids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
sqlx::query_as::<_, (uuid::Uuid, uuid::Uuid)>(
|
||||
"SELECT id, drive_id FROM storage.files WHERE id = ANY($1)",
|
||||
)
|
||||
.bind(file_ids)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("FileBlobRead", format!("drive_id batch lookup: {e}"))
|
||||
})
|
||||
}
|
||||
|
||||
/// Creates a stub instance for testing — never hits PG.
|
||||
/// Available in both standard unit-test (`cfg(test)`) and integration
|
||||
/// (`cfg(integration_tests)`) builds; `PgAclEngine::new_stub` chains
|
||||
@@ -494,7 +518,24 @@ impl FileBlobReadRepository {
|
||||
before: Option<i64>,
|
||||
limit: i64,
|
||||
) -> Result<(Vec<File>, Vec<i64>, Vec<(Option<i32>, Option<i32>)>), DomainError> {
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(
|
||||
// Sargable keyset cursor: compare the RAW `media_sort_date` column
|
||||
// against a timestamptz bind so the planner can use the cursor as
|
||||
// an index boundary condition on `idx_files_media_timeline_by_drive`.
|
||||
// The old shape wrapped the column in `EXTRACT(EPOCH …)::bigint`
|
||||
// (plus an `IS NULL OR` disjunction), which degraded the cursor to
|
||||
// a per-row Filter: page k re-read and discarded all k·limit rows
|
||||
// already scrolled past (benches/PHOTOS-CURSOR.md). Since `before`
|
||||
// is whole seconds, `media_sort_date < to_timestamp(before)` admits
|
||||
// exactly the same rows as the old truncated comparison. The
|
||||
// predicate is emitted only when a cursor exists — a bound
|
||||
// disjunction would block the index condition under generic plans.
|
||||
let cursor_ts = before.and_then(|s| chrono::DateTime::from_timestamp(s, 0));
|
||||
let cursor_pred = if cursor_ts.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
@@ -524,18 +565,18 @@ impl FileBlobReadRepository {
|
||||
)
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
AND ($2::bigint IS NULL
|
||||
OR EXTRACT(EPOCH FROM fi.media_sort_date)::bigint < $2::bigint)
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
"#,
|
||||
)
|
||||
.bind(caller_id)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
|
||||
);
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
|
||||
.bind(caller_id)
|
||||
.bind(cursor_ts)
|
||||
.bind(limit)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
|
||||
|
||||
let mut files = Vec::with_capacity(rows.len());
|
||||
let mut sort_dates = Vec::with_capacity(rows.len());
|
||||
@@ -768,62 +809,57 @@ impl FileReadPort for FileBlobReadRepository {
|
||||
self.resolve_blob_hash(file_id).await
|
||||
}
|
||||
|
||||
/// Paginated file listing — fetches only `limit` rows starting at `offset`.
|
||||
/// Keyset-paginated file listing in name order — fetches only `limit`
|
||||
/// rows after `after_name` (exclusive).
|
||||
///
|
||||
/// Uses a single SQL query with `LIMIT/OFFSET` to avoid loading the full
|
||||
/// folder contents into memory. Ideal for streaming WebDAV PROPFIND.
|
||||
/// Names are unique per folder, so `name > $after` is a total cursor.
|
||||
/// Served by `idx_files_folder_name (folder_id, name) WHERE NOT
|
||||
/// is_trashed` as a pure index-range read: O(page) per page with no
|
||||
/// sort, where the old `LIMIT/OFFSET` shape re-scanned and re-sorted
|
||||
/// the entire folder for every page (benches/PROPFIND-PAGING.md). The
|
||||
/// cursor predicate is emitted only when a cursor exists — a
|
||||
/// `$2 IS NULL OR name > $2` disjunction would block the index
|
||||
/// condition under the extended protocol's generic plans.
|
||||
#[allow(clippy::type_complexity)]
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<File>, DomainError> {
|
||||
let rows: Vec<FileRow> = if let Some(fid) = folder_id {
|
||||
sqlx::query_as(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id = $1::uuid AND NOT fi.is_trashed
|
||||
ORDER BY fi.name
|
||||
LIMIT $2 OFFSET $3
|
||||
"#,
|
||||
)
|
||||
.bind(fid)
|
||||
.bind(limit)
|
||||
.bind(offset)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
let folder_pred = if folder_id.is_some() {
|
||||
"fi.folder_id = $1::uuid"
|
||||
} else {
|
||||
sqlx::query_as(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
"fi.folder_id IS NULL AND $1::uuid IS NULL"
|
||||
};
|
||||
let cursor_pred = if after_name.is_some() {
|
||||
"AND fi.name > $3"
|
||||
} else {
|
||||
"AND $3::text IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id IS NULL AND NOT fi.is_trashed
|
||||
ORDER BY fi.name
|
||||
LIMIT $1 OFFSET $2
|
||||
"#,
|
||||
)
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE {folder_pred} AND NOT fi.is_trashed {cursor_pred}
|
||||
ORDER BY fi.name
|
||||
LIMIT $2
|
||||
"#,
|
||||
);
|
||||
let rows: Vec<FileRow> = sqlx::query_as(&sql)
|
||||
.bind(folder_id)
|
||||
.bind(limit)
|
||||
.bind(offset)
|
||||
.bind(after_name)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
}
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
|
||||
|
||||
rows.into_iter()
|
||||
.map(
|
||||
|
||||
@@ -122,6 +122,35 @@ impl ShareStoragePort for SharePgRepository {
|
||||
Self::row_to_entity(&row)
|
||||
}
|
||||
|
||||
async fn increment_access_count(&self, token: &str) -> Result<u64, DomainError> {
|
||||
// One atomic statement — the relative bump can't lose concurrent
|
||||
// increments and never rewrites unrelated columns (the legacy
|
||||
// read-modify-write wrote back item_name/password_hash wholesale,
|
||||
// silently clobbering concurrent owner edits). The expiry guard
|
||||
// mirrors find_share_by_token's MIN(expires_at) subquery: NULL =
|
||||
// never expires.
|
||||
let result = sqlx::query(
|
||||
r#"
|
||||
UPDATE storage.shares s
|
||||
SET access_count = s.access_count + 1
|
||||
WHERE s.token = $1
|
||||
AND COALESCE(
|
||||
(SELECT MIN(ag.expires_at)
|
||||
FROM storage.role_grants ag
|
||||
WHERE ag.subject_type = 'token' AND ag.subject_id = s.id) > NOW(),
|
||||
TRUE)
|
||||
"#,
|
||||
)
|
||||
.bind(token)
|
||||
.execute(&*self.db_pool)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
tracing::error!("Database error incrementing share access count: {}", e);
|
||||
DomainError::internal_error("Share", format!("Failed to register access: {e}"))
|
||||
})?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
async fn find_share_by_token(&self, token: &str) -> Result<Share, DomainError> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
|
||||
@@ -85,6 +85,33 @@ impl UserPgRepository {
|
||||
})
|
||||
}
|
||||
|
||||
/// Fetch only `(storage_used_bytes, storage_quota_bytes)`. Not part of
|
||||
/// the `UserRepository` trait — called from `StorageUsageService`.
|
||||
///
|
||||
/// Same rationale as [`Self::get_user_flags`]: the full-row SELECT drags
|
||||
/// `image` (a data URI of up to 512 KiB), `password_hash`,
|
||||
/// `ui_preferences`, … across the wire, and the quota path runs on every
|
||||
/// folder PROPFIND and every upload quota check just to read two i64s.
|
||||
/// Measured in `benches/QUOTA-PATH.md`.
|
||||
pub async fn get_storage_usage(&self, id: Uuid) -> UserRepositoryResult<(i64, i64)> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT storage_used_bytes, storage_quota_bytes
|
||||
FROM auth.users
|
||||
WHERE id = $1
|
||||
"#,
|
||||
)
|
||||
.bind(id)
|
||||
.fetch_one(&*self.pool)
|
||||
.await
|
||||
.map_err(Self::map_sqlx_error)?;
|
||||
|
||||
Ok((
|
||||
row.get("storage_used_bytes"),
|
||||
row.get("storage_quota_bytes"),
|
||||
))
|
||||
}
|
||||
|
||||
/// Updates a user's profile image (URL or data URI). Not part of the
|
||||
/// `UserRepository` trait — called directly from `AuthApplicationService`.
|
||||
pub async fn update_image(
|
||||
|
||||
@@ -240,6 +240,16 @@ impl Drop for IngestGuard {
|
||||
/// in the [`BlobStorageBackend`], and maintains a manifest in PostgreSQL
|
||||
/// mapping file_hash → \[chunk_hashes\]. BLAKE3 hashing, ref-counting
|
||||
/// and the PostgreSQL dedup index all live here.
|
||||
/// Immutable chunk map of one CDC blob (`storage.chunk_manifests` row,
|
||||
/// minus the mutable `ref_count`). Content-addressed: for a given
|
||||
/// `file_hash` the chunk list and total size never change, which is what
|
||||
/// makes [`DedupService::manifest_cached`] safe.
|
||||
pub struct ChunkManifest {
|
||||
pub chunk_hashes: Vec<String>,
|
||||
pub chunk_sizes: Vec<i64>,
|
||||
pub total_size: i64,
|
||||
}
|
||||
|
||||
pub struct DedupService {
|
||||
/// Pluggable blob storage backend (local FS, S3, …).
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
@@ -251,6 +261,13 @@ pub struct DedupService {
|
||||
maintenance_pool: Arc<PgPool>,
|
||||
/// Single lifecycle dispatcher — fired on blob created / deleted.
|
||||
blob_lifecycle: Option<Arc<BlobLifecycleService>>,
|
||||
/// `file_hash → ChunkManifest` for the read path — every stream / range
|
||||
/// / full read of a CDC blob used to pay one manifest query first, even
|
||||
/// for the media the gallery re-reads constantly. Positive-only (a
|
||||
/// legacy blob gaining a manifest via background rechunking must be
|
||||
/// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
|
||||
/// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
|
||||
manifest_cache: moka::future::Cache<String, Arc<ChunkManifest>>,
|
||||
}
|
||||
|
||||
impl DedupService {
|
||||
@@ -269,9 +286,22 @@ impl DedupService {
|
||||
pool,
|
||||
maintenance_pool,
|
||||
blob_lifecycle: None,
|
||||
manifest_cache: Self::build_manifest_cache(),
|
||||
}
|
||||
}
|
||||
|
||||
/// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
|
||||
/// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
|
||||
fn build_manifest_cache() -> moka::future::Cache<String, Arc<ChunkManifest>> {
|
||||
moka::future::Cache::builder()
|
||||
.weigher(|key: &String, value: &Arc<ChunkManifest>| {
|
||||
(key.len() + value.chunk_hashes.len() * 80 + 64) as u32
|
||||
})
|
||||
.max_capacity(32 * 1024 * 1024)
|
||||
.time_to_live(std::time::Duration::from_secs(60))
|
||||
.build()
|
||||
}
|
||||
|
||||
/// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
|
||||
pub fn with_blob_lifecycle(mut self, lifecycle: Arc<BlobLifecycleService>) -> Self {
|
||||
self.blob_lifecycle = Some(lifecycle);
|
||||
@@ -311,6 +341,7 @@ impl DedupService {
|
||||
pool: stub_pool.clone(),
|
||||
maintenance_pool: stub_pool,
|
||||
blob_lifecycle: None,
|
||||
manifest_cache: Self::build_manifest_cache(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1385,6 +1416,10 @@ impl DedupService {
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?;
|
||||
|
||||
// Post-commit so a concurrent read can't re-cache the manifest
|
||||
// between invalidation and the delete becoming visible.
|
||||
self.manifest_cache.invalidate(file_hash).await;
|
||||
|
||||
// File content is gone — drop its blob-keyed thumbnails now.
|
||||
self.fire_blob_hooks(file_hash);
|
||||
|
||||
@@ -1606,27 +1641,49 @@ impl DedupService {
|
||||
Box::pin(chunk_stream)
|
||||
}
|
||||
|
||||
/// Cached manifest fetch for the read path (see the `manifest_cache`
|
||||
/// field docs). `None` = legacy whole-file blob — never cached, so a
|
||||
/// background rechunk that creates a manifest is honoured immediately.
|
||||
async fn manifest_cached(&self, hash: &str) -> Result<Option<Arc<ChunkManifest>>, DomainError> {
|
||||
if let Some(m) = self.manifest_cache.get(hash).await {
|
||||
return Ok(Some(m));
|
||||
}
|
||||
let row = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
|
||||
"SELECT chunk_hashes, chunk_sizes, total_size
|
||||
FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
match row {
|
||||
Some((chunk_hashes, chunk_sizes, total_size)) => {
|
||||
let m = Arc::new(ChunkManifest {
|
||||
chunk_hashes,
|
||||
chunk_sizes,
|
||||
total_size,
|
||||
});
|
||||
self.manifest_cache
|
||||
.insert(hash.to_string(), m.clone())
|
||||
.await;
|
||||
Ok(Some(m))
|
||||
}
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Stream blob content — CDC-aware with legacy fallback.
|
||||
///
|
||||
/// For CDC files: looks up the manifest, then streams chunks in order,
|
||||
/// concatenating them into a single byte stream.
|
||||
/// For CDC files: looks up the manifest (RAM-cached), then streams
|
||||
/// chunks in order, concatenating them into a single byte stream.
|
||||
/// For legacy blobs: delegates directly to the backend.
|
||||
pub async fn read_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
// Check manifest
|
||||
let manifest = sqlx::query_scalar::<_, Vec<String>>(
|
||||
"SELECT chunk_hashes FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
match manifest {
|
||||
Some(chunk_hashes) => Ok(self.stream_chunks(chunk_hashes)),
|
||||
match self.manifest_cached(hash).await? {
|
||||
Some(m) => Ok(self.stream_chunks(m.chunk_hashes.clone())),
|
||||
// Legacy whole-file blob
|
||||
None => self.backend.get_blob_stream(hash).await,
|
||||
}
|
||||
@@ -1644,18 +1701,11 @@ impl DedupService {
|
||||
/// `blob_size` + `read_blob_stream`) doubled the manifest round-trips on
|
||||
/// every full-blob read (e.g. 2N queries for an N-image gallery cold load).
|
||||
pub async fn read_blob_bytes(&self, hash: &str) -> Result<Bytes, DomainError> {
|
||||
let manifest = sqlx::query_as::<_, (Vec<String>, i64)>(
|
||||
"SELECT chunk_hashes, total_size FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
let (mut stream, expected_size) = match manifest {
|
||||
Some((chunk_hashes, total_size)) => {
|
||||
(self.stream_chunks(chunk_hashes), total_size.max(0) as usize)
|
||||
}
|
||||
let (mut stream, expected_size) = match self.manifest_cached(hash).await? {
|
||||
Some(m) => (
|
||||
self.stream_chunks(m.chunk_hashes.clone()),
|
||||
m.total_size.max(0) as usize,
|
||||
),
|
||||
None => {
|
||||
// Legacy whole-file blob: size + stream straight from the backend.
|
||||
let size = self.backend.blob_size(hash).await? as usize;
|
||||
@@ -1685,17 +1735,9 @@ impl DedupService {
|
||||
end: Option<u64>,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
// Check manifest
|
||||
let manifest = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
|
||||
"SELECT chunk_hashes, chunk_sizes, total_size
|
||||
FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
if let Some((chunk_hashes, chunk_sizes, total_size)) = manifest {
|
||||
if let Some(m) = self.manifest_cached(hash).await? {
|
||||
let (chunk_hashes, chunk_sizes, total_size) =
|
||||
(&m.chunk_hashes, &m.chunk_sizes, m.total_size);
|
||||
let end = end.unwrap_or(total_size as u64);
|
||||
|
||||
// Calculate which chunks overlap [start, end)
|
||||
@@ -1749,17 +1791,9 @@ impl DedupService {
|
||||
|
||||
/// Get blob size — manifest-aware with legacy fallback.
|
||||
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
|
||||
// Check manifest first (O(1) from PG)
|
||||
let manifest_size = sqlx::query_scalar::<_, i64>(
|
||||
"SELECT total_size FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
if let Some(size) = manifest_size {
|
||||
return Ok(size as u64);
|
||||
// Check manifest first (RAM cache, else one O(1) PG row)
|
||||
if let Some(m) = self.manifest_cached(hash).await? {
|
||||
return Ok(m.total_size as u64);
|
||||
}
|
||||
|
||||
// Legacy: delegate to backend
|
||||
@@ -2048,6 +2082,7 @@ impl DedupService {
|
||||
}
|
||||
|
||||
for (file_hash, chunk_hashes, size) in &batch {
|
||||
self.manifest_cache.invalidate(file_hash).await;
|
||||
// Decrement chunk ref_counts. GREATEST(.., 0) guards against the
|
||||
// single-chunk file case where the PG file-delete trigger already
|
||||
// decremented blobs.ref_count (because file_hash == chunk_hash);
|
||||
|
||||
@@ -1112,6 +1112,78 @@ impl AuthorizationEngine for PgAclEngine {
|
||||
result
|
||||
}
|
||||
|
||||
/// Batched Read check over a page of file ids (see the trait docs).
|
||||
///
|
||||
/// Decision-equivalent to looping `check`: (1) resolve every file's
|
||||
/// drive in one `= ANY($1)` query (same rows as N ×
|
||||
/// `get_file_drive_id`; absent ids decide `false` exactly like the
|
||||
/// per-file `NotFound` path), (2) evaluate the drive-role floor once
|
||||
/// per distinct drive through the same `drive_role_cache`, (3) send
|
||||
/// only the drive-floor misses through the full per-file cascade —
|
||||
/// preserving per-file grant resolution. `Read` is never gated by the
|
||||
/// read-only drive freeze, so skipping that branch changes nothing.
|
||||
async fn check_files_read_batch(
|
||||
&self,
|
||||
subject: Subject,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashSet<Uuid>, DomainError> {
|
||||
use std::collections::{HashMap, HashSet};
|
||||
let start = std::time::Instant::now();
|
||||
let counters = QueryCounters::default();
|
||||
|
||||
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
|
||||
let pairs = self.file_repo.get_file_drive_ids(file_ids).await?;
|
||||
|
||||
// Prime the resource→drive cache — later single checks on these
|
||||
// files (download, share) skip their point lookup too.
|
||||
for (file_id, drive_id) in &pairs {
|
||||
self.owner_cache
|
||||
.insert(Resource::File(*file_id), *drive_id)
|
||||
.await;
|
||||
}
|
||||
|
||||
let mut drive_readable: HashMap<Uuid, bool> = HashMap::new();
|
||||
for (_, drive_id) in &pairs {
|
||||
if !drive_readable.contains_key(drive_id) {
|
||||
let ok = self
|
||||
.caller_role_on_drive_cached(subject, *drive_id, &counters)
|
||||
.await?
|
||||
.is_some_and(|role| role.expand().contains(&Permission::Read));
|
||||
drive_readable.insert(*drive_id, ok);
|
||||
}
|
||||
}
|
||||
|
||||
let mut allowed: HashSet<Uuid> = HashSet::with_capacity(pairs.len());
|
||||
for (file_id, drive_id) in &pairs {
|
||||
if drive_readable.get(drive_id).copied().unwrap_or(false) {
|
||||
allowed.insert(*file_id);
|
||||
} else if self
|
||||
.check_inner(
|
||||
subject,
|
||||
Permission::Read,
|
||||
Resource::File(*file_id),
|
||||
&counters,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
// Per-file / folder-cascade grant inside a drive the caller
|
||||
// has no role on — rare, but must keep resolving.
|
||||
allowed.insert(*file_id);
|
||||
}
|
||||
}
|
||||
|
||||
tracing::debug!(
|
||||
target: "oxicloud::authz",
|
||||
event = "authz.check_files_read_batch",
|
||||
subject = %subject,
|
||||
files = file_ids.len(),
|
||||
allowed = allowed.len(),
|
||||
duration_us = start.elapsed().as_micros() as u64,
|
||||
sql_queries = counters.sql_queries.load(Ordering::Relaxed),
|
||||
);
|
||||
Ok(allowed)
|
||||
}
|
||||
|
||||
async fn list_incoming_grants(&self, subject: Subject) -> Result<Vec<Grant>, DomainError> {
|
||||
let counters = QueryCounters::default();
|
||||
let (subject_types, subject_ids) = self.subject_match_set(subject, &counters).await?;
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
//! it was not handled by the path-based store either, so this is a
|
||||
//! parity decision, not a regression.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use sqlx::{PgPool, Row};
|
||||
@@ -126,17 +127,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
|
||||
/// Delete a specific dead property. No-op if not present.
|
||||
///
|
||||
/// Filters on the concrete id column (`folder_id = $1` / `file_id = $1`)
|
||||
/// rather than the old `IS NOT DISTINCT FROM` pair — PostgreSQL cannot
|
||||
/// serve `IS NOT DISTINCT FROM` from a B-tree index, so every lookup
|
||||
/// degraded to a sequential scan as the table grew. The `=` shape is
|
||||
/// served by the partial unique indexes from migration 20260830000001.
|
||||
/// (Same rationale for `get_all` / `get` / the batched readers below —
|
||||
/// measured in `benches/DEAD-PROPS.md`.)
|
||||
pub async fn remove(&self, r: ResourceRef, name: &QualifiedName) -> Result<(), DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
sqlx::query(&format!(
|
||||
"DELETE FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.execute(&*self.pool)
|
||||
@@ -150,28 +157,64 @@ impl DeadPropertyStore {
|
||||
&self,
|
||||
r: ResourceRef,
|
||||
) -> Result<Vec<(QualifiedName, Option<String>)>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let rows = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1",
|
||||
))
|
||||
.bind(id)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("DeadPropertyStore", format!("get_all: {e}")))?;
|
||||
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.map(|r| {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
})
|
||||
.collect())
|
||||
Ok(rows.into_iter().map(row_to_prop).collect())
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every file in a PROPFIND page:
|
||||
/// ONE `file_id = ANY($1)` round-trip instead of N sequential queries.
|
||||
/// Files with no dead properties are simply absent from the map.
|
||||
pub async fn get_all_for_files(
|
||||
&self,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("file_id", file_ids).await
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every subfolder in a PROPFIND page.
|
||||
pub async fn get_all_for_folders(
|
||||
&self,
|
||||
folder_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("folder_id", folder_ids).await
|
||||
}
|
||||
|
||||
async fn get_all_batched(
|
||||
&self,
|
||||
column: &str,
|
||||
ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
if ids.is_empty() {
|
||||
return Ok(HashMap::new());
|
||||
}
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT {column} AS resource_id, namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE {column} = ANY($1)",
|
||||
))
|
||||
.bind(ids)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("DeadPropertyStore", format!("get_all_batched: {e}"))
|
||||
})?;
|
||||
|
||||
let mut map: HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> = HashMap::new();
|
||||
for row in rows {
|
||||
let resource_id: Uuid = row.get("resource_id");
|
||||
map.entry(resource_id).or_default().push(row_to_prop(row));
|
||||
}
|
||||
Ok(map)
|
||||
}
|
||||
|
||||
/// Return a specific dead property, or `None` if not stored.
|
||||
@@ -181,16 +224,14 @@ impl DeadPropertyStore {
|
||||
r: ResourceRef,
|
||||
name: &QualifiedName,
|
||||
) -> Result<Option<Option<String>>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let row = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let row = sqlx::query(&format!(
|
||||
"SELECT value FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.fetch_optional(&*self.pool)
|
||||
@@ -201,16 +242,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Splits a `ResourceRef` into `(folder_id, file_id)` Option pairs for
|
||||
/// binding into SQL. The unused slot is `None` so `IS NOT DISTINCT FROM`
|
||||
/// matches the NULL stored in the unused column.
|
||||
fn split_ref(r: ResourceRef) -> (Option<Uuid>, Option<Uuid>) {
|
||||
/// Maps a `ResourceRef` onto the column that stores it plus the id to bind.
|
||||
/// The column name is one of two compile-time literals — never user input —
|
||||
/// so interpolating it into the SQL text is safe.
|
||||
fn split_ref(r: ResourceRef) -> (&'static str, Uuid) {
|
||||
match r {
|
||||
ResourceRef::Folder(id) => (Some(id), None),
|
||||
ResourceRef::File(id) => (None, Some(id)),
|
||||
ResourceRef::Folder(id) => ("folder_id", id),
|
||||
ResourceRef::File(id) => ("file_id", id),
|
||||
}
|
||||
}
|
||||
|
||||
fn row_to_prop(r: sqlx::postgres::PgRow) -> (QualifiedName, Option<String>) {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
}
|
||||
|
||||
pub fn create_dead_property_store(pool: Arc<PgPool>) -> Arc<DeadPropertyStore> {
|
||||
Arc::new(DeadPropertyStore::new(pool))
|
||||
}
|
||||
|
||||
@@ -52,7 +52,13 @@ enum ZipPlanEntry {
|
||||
/// Directory entry (Stored, zero-length body).
|
||||
Dir(String),
|
||||
/// File entry: ZIP-relative path + file id to stream from the blob store.
|
||||
File { zip_path: String, file_id: String },
|
||||
/// `compression` is picked from the file's MIME type at plan time —
|
||||
/// `Stored` for already-compressed media (JPEG/MP4/…), `Deflate` otherwise.
|
||||
File {
|
||||
zip_path: String,
|
||||
file_id: String,
|
||||
compression: Compression,
|
||||
},
|
||||
}
|
||||
|
||||
/// Message protocol from the prefetch task to the ZIP writer. For each
|
||||
@@ -74,8 +80,11 @@ const PREFETCH_BUFFER_CHUNKS: usize = 64;
|
||||
///
|
||||
/// Uses `async_zip` for fully-async archive creation. Every write (headers,
|
||||
/// compressed chunk data, central directory) goes through
|
||||
/// `tokio::io::BufWriter` → `tokio::fs::File`, so **no Tokio worker is ever
|
||||
/// blocked** by disk I/O or compression.
|
||||
/// `tokio::io::BufWriter` → `tokio::fs::File`, so no Tokio worker is ever
|
||||
/// blocked by disk I/O. Deflate itself DOES run inline on the writing task
|
||||
/// (async_zip compresses inside `poll_write`), which is why entries whose
|
||||
/// MIME says the content is already compressed are `Stored` instead — that
|
||||
/// turns the archive hot path from ~1 CPU core per download into CRC + memcpy.
|
||||
///
|
||||
/// Archive creation is a 2-stage pipeline: a prefetch task reads file
|
||||
/// content from the blob store ahead of the writer, so the next file's
|
||||
@@ -183,6 +192,9 @@ impl ZipService {
|
||||
plan.push(ZipPlanEntry::File {
|
||||
zip_path: format!("{}{}", zip_dir, file.name),
|
||||
file_id: file.id.to_string(),
|
||||
compression: crate::common::mime_detect::zip_entry_compression(
|
||||
&file.mime_type,
|
||||
),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -228,8 +240,12 @@ impl ZipService {
|
||||
}
|
||||
}
|
||||
}
|
||||
ZipPlanEntry::File { zip_path, .. } => {
|
||||
Self::write_prefetched_file(&mut zip, zip_path, &mut rx).await?;
|
||||
ZipPlanEntry::File {
|
||||
zip_path,
|
||||
compression,
|
||||
..
|
||||
} => {
|
||||
Self::write_prefetched_file(&mut zip, zip_path, *compression, &mut rx).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -282,17 +298,19 @@ impl ZipService {
|
||||
}
|
||||
}
|
||||
|
||||
/// Writer stage: drains one file's prefetched chunks into a Deflate
|
||||
/// ZIP entry. Peak memory stays bounded by the channel, independent
|
||||
/// of individual file sizes.
|
||||
/// Writer stage: drains one file's prefetched chunks into a ZIP entry
|
||||
/// (`Stored` for already-compressed media, `Deflate` otherwise — see
|
||||
/// `entry_compression`). Peak memory stays bounded by the channel,
|
||||
/// independent of individual file sizes.
|
||||
async fn write_prefetched_file(
|
||||
zip: &mut AsyncZipWriter,
|
||||
zip_path: &str,
|
||||
compression: Compression,
|
||||
rx: &mut tokio::sync::mpsc::Receiver<Prefetched>,
|
||||
) -> Result<()> {
|
||||
info!("Adding file to ZIP: {}", zip_path);
|
||||
|
||||
let entry = ZipEntryBuilder::new(zip_path.to_string().into(), Compression::Deflate);
|
||||
let entry = ZipEntryBuilder::new(zip_path.to_string().into(), compression);
|
||||
let mut entry_writer = zip
|
||||
.write_entry_stream(entry)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user