perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
@@ -129,6 +129,30 @@ pub struct PropFindRequest {
|
||||
pub prop_find_type: PropFindType,
|
||||
}
|
||||
|
||||
impl PropFindRequest {
|
||||
/// Whether answering this PROPFIND requires resolving the account /
|
||||
/// drive quota at all.
|
||||
///
|
||||
/// `resolve_webdav_quota` costs two DB round-trips per request; sync
|
||||
/// clients poll folders with an explicit `<D:prop>` list that most of
|
||||
/// the time names only etag/length/type props — computing quota there
|
||||
/// is pure waste (the response never mentions it). `AllProp` and
|
||||
/// `PropName` keep quota: the writers emit RFC 4331 props for both.
|
||||
/// Measured in `benches/QUOTA-PATH.md`.
|
||||
pub fn wants_quota(&self) -> bool {
|
||||
match &self.prop_find_type {
|
||||
PropFindType::AllProp | PropFindType::PropName => true,
|
||||
PropFindType::Prop(props) => props.iter().any(|p| {
|
||||
p.namespace == "DAV:"
|
||||
&& matches!(
|
||||
p.name.as_str(),
|
||||
"quota-used-bytes" | "quota-available-bytes"
|
||||
)
|
||||
}),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// WebDAV property value
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PropValue {
|
||||
|
||||
@@ -29,6 +29,30 @@ pub trait AuthorizationEngine: Send + Sync + 'static {
|
||||
resource: Resource,
|
||||
) -> Result<bool, DomainError>;
|
||||
|
||||
/// Batched `check(subject, Read, File(id))` over a result page: returns
|
||||
/// the subset of `file_ids` the subject may read. Semantically identical
|
||||
/// to looping [`Self::check`] (the default does exactly that); the
|
||||
/// `PgAclEngine` override resolves every file's drive in ONE query and
|
||||
/// reuses the per-drive role cache, so verifying a 200-hit search page
|
||||
/// costs 1 SQL round-trip instead of up to 200 sequential ones
|
||||
/// (benches/SEARCH-REBAC.md).
|
||||
async fn check_files_read_batch(
|
||||
&self,
|
||||
subject: Subject,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashSet<Uuid>, DomainError> {
|
||||
let mut allowed = std::collections::HashSet::with_capacity(file_ids.len());
|
||||
for id in file_ids {
|
||||
if self
|
||||
.check(subject, Permission::Read, Resource::File(*id))
|
||||
.await?
|
||||
{
|
||||
allowed.insert(*id);
|
||||
}
|
||||
}
|
||||
Ok(allowed)
|
||||
}
|
||||
|
||||
/// Convenience wrapper around `check`: returns `Ok(())` when allowed and
|
||||
/// `DomainError::not_found` when denied (anti-enumeration — same error as
|
||||
/// "resource doesn't exist" so attackers can't probe IDs by error shape).
|
||||
|
||||
@@ -39,6 +39,23 @@ pub trait FaceRepository: Send + Sync + 'static {
|
||||
user_id: Uuid,
|
||||
blob_hash: &str,
|
||||
) -> Result<Vec<Face>, DomainError>;
|
||||
/// `(person_id, face_count)` per non-empty cluster — a grouped COUNT
|
||||
/// instead of dragging every face row (each with a 2 KiB embedding
|
||||
/// BYTEA) across the wire just to count them. See benches/PEOPLE-LIST.md.
|
||||
async fn person_face_stats(&self, user_id: Uuid) -> Result<Vec<(Uuid, i64)>, DomainError>;
|
||||
/// face id → file id for the given faces (cover-photo resolution).
|
||||
async fn file_ids_for_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
face_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashMap<Uuid, Uuid>, DomainError>;
|
||||
/// Reassign every face of `from` to `into` in one statement (merge).
|
||||
async fn reassign_person_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
from: Uuid,
|
||||
into: Uuid,
|
||||
) -> Result<u64, DomainError>;
|
||||
async fn assign_person(
|
||||
&self,
|
||||
face_id: Uuid,
|
||||
|
||||
@@ -235,13 +235,14 @@ pub trait FileRetrievalUseCase: Send + Sync + 'static {
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
let all = self.list_files(folder_id).await?;
|
||||
let mut all = self.list_files(folder_id).await?;
|
||||
all.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.skip(offset as usize)
|
||||
.filter(|f| after_name.is_none_or(|a| f.name.as_str() > a))
|
||||
.take(limit as usize)
|
||||
.collect())
|
||||
}
|
||||
@@ -257,10 +258,10 @@ pub trait FileRetrievalUseCase: Send + Sync + 'static {
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
_owner_id: Uuid,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
self.list_files_batch(folder_id, offset, limit).await
|
||||
self.list_files_batch(folder_id, after_name, limit).await
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -99,6 +99,28 @@ pub trait ShareStoragePort: Send + Sync + 'static {
|
||||
share: &crate::domain::entities::share::Share,
|
||||
) -> Result<crate::domain::entities::share::Share, DomainError>;
|
||||
|
||||
/// Atomically bump a link's access counter (public share landing).
|
||||
/// Returns the number of rows updated — 0 means "no live share for
|
||||
/// this token" (missing OR expired).
|
||||
///
|
||||
/// The default is the legacy read-modify-write (kept for test mocks);
|
||||
/// `SharePgRepository` overrides it with a single `UPDATE … SET
|
||||
/// access_count = access_count + 1`, replacing 2 correlated-subquery
|
||||
/// round-trips per anonymous visit with 1 and removing the lost-update
|
||||
/// race between concurrent visitors (benches/SHARE-ACCESS.md).
|
||||
async fn increment_access_count(&self, token: &str) -> Result<u64, DomainError> {
|
||||
let share = match self.find_share_by_token(token).await {
|
||||
Ok(s) => s,
|
||||
Err(e) if e.kind == crate::common::errors::ErrorKind::NotFound => return Ok(0),
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
if share.is_expired() {
|
||||
return Ok(0);
|
||||
}
|
||||
self.update_share(&share.increment_access_count()).await?;
|
||||
Ok(1)
|
||||
}
|
||||
|
||||
async fn find_shares_by_user(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
|
||||
@@ -107,20 +107,30 @@ pub trait FileReadPort: Send + Sync + 'static {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// Lists files in a folder with LIMIT/OFFSET pagination.
|
||||
/// Lists files in a folder in name order, keyset-paginated.
|
||||
///
|
||||
/// Used by streaming WebDAV PROPFIND to avoid loading all files at once.
|
||||
/// `after_name` is the last name of the previous page (`None` = first
|
||||
/// page); names are unique within a folder (unique index on
|
||||
/// `(drive_id, folder_id, name)`), so `name > after_name` is a total,
|
||||
/// stable cursor. Unlike LIMIT/OFFSET, every page is O(page) — the old
|
||||
/// offset shape re-scanned and re-sorted the whole folder per page
|
||||
/// (benches/PROPFIND-PAGING.md).
|
||||
///
|
||||
/// Default: falls back to `list_files` (loads all, then slices in memory).
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<File>, DomainError> {
|
||||
let all = self.list_files(folder_id).await?;
|
||||
let start = (offset as usize).min(all.len());
|
||||
let end = (start + limit as usize).min(all.len());
|
||||
Ok(all.into_iter().skip(start).take(end - start).collect())
|
||||
let mut all = self.list_files(folder_id).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit as usize)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Streams every file in the subtree rooted at `folder_id`.
|
||||
|
||||
@@ -734,7 +734,13 @@ impl BatchOperationService {
|
||||
{
|
||||
Ok(file_dto) => {
|
||||
match self
|
||||
.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, user_id)
|
||||
.add_file_entry_streamed(
|
||||
&mut zip,
|
||||
file_id,
|
||||
&file_dto.name,
|
||||
&file_dto.mime_type,
|
||||
Some(user_id),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => items_added += 1,
|
||||
@@ -758,7 +764,7 @@ impl BatchOperationService {
|
||||
{
|
||||
Ok(root_folder) => {
|
||||
match self
|
||||
.add_folder_subtree_to_zip(&mut zip, folder_id, &root_folder, user_id)
|
||||
.add_folder_subtree_to_zip(&mut zip, folder_id, &root_folder)
|
||||
.await
|
||||
{
|
||||
Ok(_) => items_added += 1,
|
||||
@@ -803,24 +809,44 @@ impl BatchOperationService {
|
||||
}
|
||||
|
||||
/// Streams a single file into an async ZIP entry (~64 KB peak RAM per file).
|
||||
///
|
||||
/// Already-compressed content (per its MIME type) is `Stored` — deflating
|
||||
/// JPEG/MP4/… burns ~a CPU core per download for ~0 % size gain.
|
||||
///
|
||||
/// `caller_id = Some(uid)` enforces the per-file Read check and records
|
||||
/// the access in Recents (explicitly-selected top-level files).
|
||||
/// `None` = the file was enumerated from a folder subtree whose ROOT the
|
||||
/// caller already passed `get_folder_with_perms` for — per-file
|
||||
/// re-authorization and per-file Recent spam (2 writes/file via the
|
||||
/// recent hook) are skipped, mirroring `ZipService::create_folder_zip`
|
||||
/// on the native folder-download path (benches/ZIP-BATCH-AUTHZ.md).
|
||||
async fn add_file_entry_streamed(
|
||||
&self,
|
||||
zip: &mut ZipFileWriter<tokio_util::compat::Compat<BufWriter<tokio::fs::File>>>,
|
||||
file_id: &str,
|
||||
entry_name: &str,
|
||||
caller_id: Uuid,
|
||||
mime_type: &str,
|
||||
caller_id: Option<Uuid>,
|
||||
) -> Result<(), BatchOperationError> {
|
||||
let entry = ZipEntryBuilder::new(entry_name.to_string().into(), Compression::Deflate);
|
||||
let compression = crate::common::mime_detect::zip_entry_compression(mime_type);
|
||||
let entry = ZipEntryBuilder::new(entry_name.to_string().into(), compression);
|
||||
let mut writer = zip
|
||||
.write_entry_stream(entry)
|
||||
.await
|
||||
.map_err(|e| BatchOperationError::Internal(format!("zip entry start: {}", e)))?;
|
||||
|
||||
let stream = self
|
||||
.file_retrieval
|
||||
.get_file_stream_with_perms(file_id, caller_id)
|
||||
.await
|
||||
.map_err(BatchOperationError::Domain)?;
|
||||
let stream = match caller_id {
|
||||
Some(uid) => self
|
||||
.file_retrieval
|
||||
.get_file_stream_with_perms(file_id, uid)
|
||||
.await
|
||||
.map_err(BatchOperationError::Domain)?,
|
||||
None => self
|
||||
.file_retrieval
|
||||
.get_file_stream(file_id)
|
||||
.await
|
||||
.map_err(BatchOperationError::Domain)?,
|
||||
};
|
||||
let mut stream = std::pin::Pin::from(stream);
|
||||
|
||||
while let Some(chunk) = stream.next().await {
|
||||
@@ -849,7 +875,6 @@ impl BatchOperationService {
|
||||
zip: &mut ZipFileWriter<tokio_util::compat::Compat<BufWriter<tokio::fs::File>>>,
|
||||
folder_id: &str,
|
||||
root_folder: &FolderDto,
|
||||
caller_id: Uuid,
|
||||
) -> Result<(), BatchOperationError> {
|
||||
// Bulk-fetch folder tree (small — one entry per folder)
|
||||
let all_folders = self
|
||||
@@ -903,8 +928,10 @@ impl BatchOperationService {
|
||||
if let Some(files) = files_by_folder.get(&folder.id) {
|
||||
for file in files {
|
||||
let file_path = format!("{}{}", zip_dir, file.name);
|
||||
// Subtree pre-authorized at the root folder — see
|
||||
// `add_file_entry_streamed` docs for why `None`.
|
||||
if let Err(e) = self
|
||||
.add_file_entry_streamed(zip, &file.id, &file_path, caller_id)
|
||||
.add_file_entry_streamed(zip, &file.id, &file_path, &file.mime_type, None)
|
||||
.await
|
||||
{
|
||||
info!("Could not add file {} to ZIP: {}", file.name, e);
|
||||
|
||||
@@ -451,12 +451,12 @@ impl FileRetrievalUseCase for FileRetrievalService {
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
let files = self
|
||||
.file_read
|
||||
.list_files_batch(folder_id, offset, limit)
|
||||
.list_files_batch(folder_id, after_name, limit)
|
||||
.await?;
|
||||
Ok(files.into_iter().map(FileDto::from).collect())
|
||||
}
|
||||
@@ -465,7 +465,7 @@ impl FileRetrievalUseCase for FileRetrievalService {
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
owner_id: Uuid,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
// Post-D0: every file lives in a folder — `storage.files.folder_id`
|
||||
@@ -482,7 +482,7 @@ impl FileRetrievalUseCase for FileRetrievalService {
|
||||
.await?;
|
||||
let files = self
|
||||
.file_read
|
||||
.list_files_batch(folder_id, offset, limit)
|
||||
.list_files_batch(folder_id, after_name, limit)
|
||||
.await?;
|
||||
Ok(files.into_iter().map(FileDto::from).collect())
|
||||
}
|
||||
|
||||
@@ -166,18 +166,23 @@ impl PeopleService {
|
||||
}
|
||||
|
||||
/// People (non-empty clusters), most-photographed first.
|
||||
///
|
||||
/// Counts come from a grouped-COUNT query and cover photos from one
|
||||
/// batched lookup of just the cover face ids — the previous
|
||||
/// `faces_for_user` shipped every face row (2 KiB embedding included)
|
||||
/// only to count them: ~20 MB of BYTEA per request on a 10k-face
|
||||
/// library (benches/PEOPLE-LIST.md).
|
||||
pub async fn list_people(&self, caller_id: Uuid) -> Result<Vec<PersonDto>, DomainError> {
|
||||
let persons = self.repo.persons_for_user(caller_id).await?;
|
||||
let faces = self.repo.faces_for_user(caller_id).await?;
|
||||
|
||||
let mut count: HashMap<Uuid, i64> = HashMap::new();
|
||||
let mut face_file: HashMap<Uuid, Uuid> = HashMap::new();
|
||||
for f in &faces {
|
||||
if let Some(pid) = f.person_id {
|
||||
*count.entry(pid).or_default() += 1;
|
||||
}
|
||||
face_file.insert(f.id, f.file_id);
|
||||
}
|
||||
let count: HashMap<Uuid, i64> = self
|
||||
.repo
|
||||
.person_face_stats(caller_id)
|
||||
.await?
|
||||
.into_iter()
|
||||
.collect();
|
||||
let cover_ids: Vec<Uuid> = persons.iter().filter_map(|p| p.cover_face_id).collect();
|
||||
let face_file: HashMap<Uuid, Uuid> =
|
||||
self.repo.file_ids_for_faces(caller_id, &cover_ids).await?;
|
||||
|
||||
let mut out: Vec<PersonDto> = persons
|
||||
.into_iter()
|
||||
@@ -245,11 +250,13 @@ impl PeopleService {
|
||||
|
||||
/// Merge `from` into `into` by reassigning all of `from`'s faces. The
|
||||
/// now-empty `from` person is hidden by `list_people`.
|
||||
///
|
||||
/// One set-based UPDATE — the previous shape loaded every face row
|
||||
/// (embeddings included) and issued one UPDATE per matching face.
|
||||
pub async fn merge(&self, caller_id: Uuid, into: Uuid, from: Uuid) -> Result<(), DomainError> {
|
||||
let faces = self.repo.faces_for_user(caller_id).await?;
|
||||
for f in faces.into_iter().filter(|f| f.person_id == Some(from)) {
|
||||
self.repo.assign_person(f.id, Some(into)).await?;
|
||||
}
|
||||
self.repo
|
||||
.reassign_person_faces(caller_id, from, into)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -260,7 +260,7 @@ impl SearchService {
|
||||
user_id: Uuid,
|
||||
) -> Vec<ContentHitDto> {
|
||||
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
||||
use crate::domain::services::authorization::{Permission, Resource, Subject};
|
||||
use crate::domain::services::authorization::Subject;
|
||||
|
||||
let Some(index) = &self.content_index else {
|
||||
return Vec::new();
|
||||
@@ -316,36 +316,42 @@ impl SearchService {
|
||||
// drive the caller doesn't otherwise have. The Tantivy
|
||||
// filter is drive-only; this re-check restores per-file
|
||||
// resolution.
|
||||
// Failures degrade conservatively (drop the hit, log it) —
|
||||
// never leak.
|
||||
let mut verified = Vec::with_capacity(hits.len());
|
||||
for hit in hits {
|
||||
let file_uuid = match Uuid::parse_str(&hit.file_id) {
|
||||
Ok(u) => u,
|
||||
// Failures degrade conservatively (drop the hit / the page,
|
||||
// log it) — never leak. Batched: one drive-resolution query for
|
||||
// the whole page instead of up to CONTENT_HITS_LIMIT sequential
|
||||
// point SELECTs (benches/SEARCH-REBAC.md).
|
||||
let mut hit_ids = Vec::with_capacity(hits.len());
|
||||
for hit in &hits {
|
||||
match Uuid::parse_str(&hit.file_id) {
|
||||
Ok(u) => hit_ids.push(u),
|
||||
Err(_) => {
|
||||
tracing::warn!("Content-index hit had non-UUID file_id: {}", hit.file_id);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
let allowed = match authz
|
||||
.check_files_read_batch(Subject::User(user_id), &hit_ids)
|
||||
.await
|
||||
{
|
||||
Ok(set) => set,
|
||||
Err(e) => {
|
||||
tracing::warn!("ReBAC re-check failed for content hits: {e}");
|
||||
return Vec::new();
|
||||
}
|
||||
};
|
||||
let mut verified = Vec::with_capacity(hits.len());
|
||||
for hit in hits {
|
||||
let Ok(file_uuid) = Uuid::parse_str(&hit.file_id) else {
|
||||
continue; // already warned above
|
||||
};
|
||||
match authz
|
||||
.check(
|
||||
Subject::User(user_id),
|
||||
Permission::Read,
|
||||
Resource::File(file_uuid),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(true) => verified.push(hit),
|
||||
Ok(false) => {
|
||||
tracing::debug!(
|
||||
target: "oxicloud::search",
|
||||
file_id = %file_uuid,
|
||||
"dropping content-index hit: ReBAC denies Read after Tantivy filter",
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!("ReBAC re-check failed for {file_uuid}: {e}");
|
||||
}
|
||||
if allowed.contains(&file_uuid) {
|
||||
verified.push(hit);
|
||||
} else {
|
||||
tracing::debug!(
|
||||
target: "oxicloud::search",
|
||||
file_id = %file_uuid,
|
||||
"dropping content-index hit: ReBAC denies Read after Tantivy filter",
|
||||
);
|
||||
}
|
||||
}
|
||||
verified
|
||||
|
||||
@@ -510,29 +510,18 @@ impl ShareUseCase for ShareService {
|
||||
}
|
||||
|
||||
async fn register_shared_link_access(&self, token: &str) -> Result<(), DomainError> {
|
||||
// Find the shared link by its token
|
||||
let share = self
|
||||
.share_repository
|
||||
.find_share_by_token(token)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
ShareServiceError::NotFound(format!("Share with token {} not found: {}", token, e))
|
||||
})?;
|
||||
|
||||
// Check if it has expired
|
||||
if share.is_expired() {
|
||||
return Err(ShareServiceError::Expired.into());
|
||||
// One atomic UPDATE (see `ShareStoragePort::increment_access_count`).
|
||||
// 0 rows = missing or expired — collapsed into NotFound, same
|
||||
// response shape either way (anti-enumeration; the landing handler
|
||||
// discards this result regardless).
|
||||
let updated = self.share_repository.increment_access_count(token).await?;
|
||||
if updated == 0 {
|
||||
return Err(ShareServiceError::NotFound(format!(
|
||||
"Share with token {} not found or expired",
|
||||
token
|
||||
))
|
||||
.into());
|
||||
}
|
||||
|
||||
// Increment the access counter
|
||||
let updated_share = share.increment_access_count();
|
||||
|
||||
// Save the changes
|
||||
self.share_repository
|
||||
.update_share(&updated_share)
|
||||
.await
|
||||
.map_err(|e| ShareServiceError::Repository(e.to_string()))?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
use crate::application::ports::auth_ports::UserStoragePort;
|
||||
use crate::application::ports::storage_ports::StorageUsagePort;
|
||||
use crate::common::errors::DomainError;
|
||||
use crate::infrastructure::repositories::pg::UserPgRepository;
|
||||
@@ -512,9 +511,9 @@ impl StorageUsagePort for StorageUsageService {
|
||||
user_id: Uuid,
|
||||
additional_bytes: u64,
|
||||
) -> Result<(), DomainError> {
|
||||
let user = self.user_repository.get_user_by_id(user_id).await?;
|
||||
let quota = user.storage_quota_bytes();
|
||||
let used = user.storage_used_bytes();
|
||||
// Narrow 2-column read — the full user row carries the up-to-512 KiB
|
||||
// avatar `image` column, paid on every upload quota check otherwise.
|
||||
let (used, quota) = self.user_repository.get_storage_usage(user_id).await?;
|
||||
|
||||
// Quota of 0 means unlimited
|
||||
if quota <= 0 {
|
||||
@@ -548,8 +547,9 @@ impl StorageUsagePort for StorageUsageService {
|
||||
}
|
||||
|
||||
async fn get_user_storage_info(&self, user_id: Uuid) -> Result<(i64, i64), DomainError> {
|
||||
let user = self.user_repository.get_user_by_id(user_id).await?;
|
||||
Ok((user.storage_used_bytes(), user.storage_quota_bytes()))
|
||||
// Narrow 2-column read (avatar-free) — runs on every folder PROPFIND
|
||||
// that reports quota. See benches/QUOTA-PATH.md.
|
||||
Ok(self.user_repository.get_storage_usage(user_id).await?)
|
||||
}
|
||||
|
||||
async fn add_drive_storage_usage_delta(
|
||||
|
||||
@@ -14,7 +14,7 @@ use crate::application::dtos::trash_dto::{
|
||||
};
|
||||
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
||||
use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
||||
use crate::application::ports::storage_ports::{FileReadPort, FileWritePort};
|
||||
use crate::application::ports::storage_ports::FileWritePort;
|
||||
use crate::application::ports::trash_ports::TrashUseCase;
|
||||
use crate::common::errors::{DomainError, ErrorKind, Result};
|
||||
use crate::domain::entities::file::File;
|
||||
@@ -24,7 +24,6 @@ use crate::domain::repositories::folder_repository::FolderRepository;
|
||||
use crate::domain::repositories::trash_repository::TrashRepository;
|
||||
use crate::domain::services::authorization::ResourceKind;
|
||||
use crate::domain::services::authorization::{Permission, Resource, Subject};
|
||||
use crate::infrastructure::repositories::pg::file_blob_read_repository::FileBlobReadRepository;
|
||||
use crate::infrastructure::repositories::pg::file_blob_write_repository::FileBlobWriteRepository;
|
||||
use crate::infrastructure::repositories::pg::folder_db_repository::FolderDbRepository;
|
||||
use crate::infrastructure::repositories::pg::trash_db_repository::TrashDbRepository;
|
||||
@@ -49,9 +48,6 @@ pub struct TrashService {
|
||||
/// Repository for trash-specific operations like listing and retrieving trashed items
|
||||
trash_repository: Arc<TrashDbRepository>,
|
||||
|
||||
/// Port for file read operations (get file metadata)
|
||||
file_read_port: Arc<FileBlobReadRepository>,
|
||||
|
||||
/// Port for file write operations (trash, restore, delete)
|
||||
file_write_port: Arc<FileBlobWriteRepository>,
|
||||
|
||||
@@ -75,19 +71,14 @@ pub struct TrashService {
|
||||
/// so trash listings filter by drive membership instead of the legacy
|
||||
/// per-user scope.
|
||||
drive_repo: Arc<crate::infrastructure::repositories::pg::DrivePgRepository>,
|
||||
|
||||
/// Number of days items should be kept in trash before automatic cleanup
|
||||
retention_days: u32,
|
||||
}
|
||||
|
||||
impl TrashService {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
trash_repository: Arc<TrashDbRepository>,
|
||||
file_read_port: Arc<FileBlobReadRepository>,
|
||||
file_write_port: Arc<FileBlobWriteRepository>,
|
||||
folder_storage_port: Arc<FolderDbRepository>,
|
||||
retention_days: u32,
|
||||
dedup_service: Arc<DedupService>,
|
||||
content_cache: Option<Arc<FileContentCache>>,
|
||||
authz: Arc<PgAclEngine>,
|
||||
@@ -95,7 +86,6 @@ impl TrashService {
|
||||
) -> Self {
|
||||
Self {
|
||||
trash_repository,
|
||||
file_read_port,
|
||||
file_write_port,
|
||||
folder_storage_port,
|
||||
dedup_service,
|
||||
@@ -103,7 +93,6 @@ impl TrashService {
|
||||
content_cache,
|
||||
authz,
|
||||
drive_repo,
|
||||
retention_days,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -177,23 +166,17 @@ impl TrashUseCase for TrashService {
|
||||
// Note: We now verify file/folder ownership BEFORE moving to trash.
|
||||
// This prevents users from trashing items they do not own (IDOR).
|
||||
|
||||
// Parse UUIDs with detailed error handling
|
||||
// Parse UUIDs with detailed error handling. The parsed value is
|
||||
// re-derived per branch below; this early check preserves the 400
|
||||
// (validation) error shape for malformed ids.
|
||||
debug!("Validating item UUID: {}", item_id);
|
||||
let item_uuid = match Uuid::parse_str(item_id) {
|
||||
Ok(uuid) => {
|
||||
debug!("Valid item UUID: {}", uuid);
|
||||
uuid
|
||||
}
|
||||
Err(e) => {
|
||||
error!("Invalid item UUID: {} - Error: {}", item_id, e);
|
||||
return Err(DomainError::validation_error(format!(
|
||||
"Invalid item ID: {}",
|
||||
e
|
||||
)));
|
||||
}
|
||||
};
|
||||
|
||||
let user_uuid = user_id;
|
||||
if let Err(e) = Uuid::parse_str(item_id) {
|
||||
error!("Invalid item UUID: {} - Error: {}", item_id, e);
|
||||
return Err(DomainError::validation_error(format!(
|
||||
"Invalid item ID: {}",
|
||||
e
|
||||
)));
|
||||
}
|
||||
|
||||
match item_type {
|
||||
"file" => {
|
||||
@@ -209,59 +192,13 @@ impl TrashUseCase for TrashService {
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Authz already passed — use the non-owner-scoped read so that
|
||||
// grantees with Delete permission can trash files they don't own.
|
||||
// The file's user_id in storage.files is unchanged, so the item
|
||||
// will appear in the original owner's trash view.
|
||||
let file = match self.file_read_port.get_file(item_id).await {
|
||||
Ok(file) => {
|
||||
debug!("File found: {} ({})", file.name(), item_id);
|
||||
file
|
||||
}
|
||||
Err(e) => {
|
||||
error!("Error getting file: {} - {}", item_id, e);
|
||||
return Err(DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"File",
|
||||
format!("Error retrieving file {}: {}", item_id, e),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
let original_path = file.storage_path().to_string();
|
||||
debug!("Original file path: {}", original_path);
|
||||
|
||||
debug!("Creating TrashedItem object for the file");
|
||||
let trashed_item = TrashedItem::new(
|
||||
item_uuid,
|
||||
user_uuid,
|
||||
TrashedItemType::File,
|
||||
file.name().to_string(),
|
||||
original_path,
|
||||
self.retention_days,
|
||||
);
|
||||
debug!(
|
||||
"TrashedItem created successfully: {} -> {}",
|
||||
file.name(),
|
||||
trashed_item.id()
|
||||
);
|
||||
|
||||
// First add to trash index to register the item
|
||||
info!("Adding file {} to trash index", item_id);
|
||||
match self.trash_repository.add_to_trash(&trashed_item).await {
|
||||
Ok(_) => {
|
||||
debug!("File added to trash index successfully");
|
||||
}
|
||||
Err(e) => {
|
||||
error!("Error adding file to trash index: {}", e);
|
||||
return Err(DomainError::internal_error(
|
||||
"TrashRepository",
|
||||
format!("Failed to add file to trash: {}", e),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
// Then physically move the file to trash.
|
||||
// Soft-delete model: the is_trashed flag on the row IS the
|
||||
// trash membership — there is no separate trash index to
|
||||
// register into (`TrashRepository::add_to_trash` is a
|
||||
// documented no-op). The previous shape still fetched the
|
||||
// full file entity and built a `TrashedItem` only to feed
|
||||
// that no-op: one wasted SELECT per trash operation.
|
||||
//
|
||||
// §14: caller_id stamps `updated_by` on the trashed row.
|
||||
info!("Physically moving file to trash: {}", item_id);
|
||||
match self.file_write_port.move_to_trash(item_id, user_id).await {
|
||||
@@ -293,43 +230,10 @@ impl TrashUseCase for TrashService {
|
||||
)
|
||||
.await?;
|
||||
|
||||
let folder = self
|
||||
.folder_storage_port
|
||||
.get_folder(item_id)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorKind::NotFound,
|
||||
"Folder",
|
||||
format!("Error retrieving folder {}: {}", item_id, e),
|
||||
)
|
||||
})?;
|
||||
|
||||
let original_path = folder.storage_path().to_string();
|
||||
|
||||
let trashed_item = TrashedItem::new(
|
||||
item_uuid,
|
||||
user_uuid,
|
||||
TrashedItemType::Folder,
|
||||
folder.name().to_string(),
|
||||
original_path,
|
||||
self.retention_days,
|
||||
);
|
||||
|
||||
// First add to trash index to register the item
|
||||
debug!("Adding folder {} to trash repository", item_id);
|
||||
match self.trash_repository.add_to_trash(&trashed_item).await {
|
||||
Ok(_) => debug!("Successfully added folder to trash repository"),
|
||||
Err(e) => {
|
||||
error!("Failed to add folder to trash repository: {}", e);
|
||||
return Err(DomainError::internal_error(
|
||||
"TrashRepository",
|
||||
format!("Failed to add folder to trash: {}", e),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
// Then physically move the folder to trash.
|
||||
// Soft-delete model — same as the file branch above: the
|
||||
// cascade UPDATE below is the whole operation; no folder
|
||||
// fetch or trash-index write needed.
|
||||
//
|
||||
// §14: caller_id stamps `updated_by` on every cascade-trashed row.
|
||||
self.folder_storage_port
|
||||
.move_to_trash(item_id, user_id)
|
||||
|
||||
@@ -836,10 +836,8 @@ impl AppServiceFactory {
|
||||
let service = Arc::new(
|
||||
TrashService::new(
|
||||
trash_repo.clone(),
|
||||
repos.file_read_repository.clone(),
|
||||
repos.file_write_repository.clone(),
|
||||
repos.folder_repository.clone(),
|
||||
self.config.storage.trash_retention_days,
|
||||
core.dedup_service.clone(),
|
||||
Some(core.file_content_cache.clone()),
|
||||
authz.clone(),
|
||||
|
||||
@@ -108,10 +108,117 @@ pub async fn refine_content_type_from_file(
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a MIME type identifies content that is already compressed, so
|
||||
/// running Deflate over it burns CPU for ~0 % size gain.
|
||||
///
|
||||
/// Used by the ZIP export paths (`ZipService`, `BatchOperations`) to pick
|
||||
/// `Compression::Stored` per entry instead of deflating JPEG/MP4/… bytes.
|
||||
/// The set mirrors the HTTP `CompressionLayer` exclusion list in `main.rs`
|
||||
/// (keep the two in sync), minus entries that are containers of possibly
|
||||
/// incompressible data rather than compressed formats themselves
|
||||
/// (`application/x-tar`, `application/octet-stream`) — those stay on Deflate
|
||||
/// so unknown-but-compressible content is never stored uncompressed.
|
||||
pub fn is_precompressed_mime(mime: &str) -> bool {
|
||||
// Strip any parameters ("; charset=…") and normalize case.
|
||||
let essence = mime.split(';').next().unwrap_or(mime).trim();
|
||||
|
||||
// Compressed families: every common video/audio codec container.
|
||||
if essence.starts_with("video/") || essence.starts_with("audio/") {
|
||||
return true;
|
||||
}
|
||||
// Zip-based document bundles (docx/xlsx/pptx, odt/ods/odp, …).
|
||||
if essence.starts_with("application/vnd.openxmlformats-officedocument")
|
||||
|| essence.starts_with("application/vnd.oasis.opendocument")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
matches!(
|
||||
essence,
|
||||
// Raster images with built-in compression (SVG intentionally absent).
|
||||
"image/jpeg"
|
||||
| "image/png"
|
||||
| "image/gif"
|
||||
| "image/webp"
|
||||
| "image/avif"
|
||||
| "image/heic"
|
||||
| "image/heif"
|
||||
| "image/jp2"
|
||||
// Already-compressed web fonts; ttf/otf left compressible.
|
||||
| "font/woff"
|
||||
| "font/woff2"
|
||||
| "application/font-woff"
|
||||
// Archives & compressed containers.
|
||||
| "application/zip"
|
||||
| "application/gzip"
|
||||
| "application/x-gzip"
|
||||
| "application/x-7z-compressed"
|
||||
| "application/x-rar-compressed"
|
||||
| "application/x-bzip2"
|
||||
| "application/zstd"
|
||||
| "application/x-xz"
|
||||
| "application/epub+zip"
|
||||
| "application/java-archive"
|
||||
| "application/vnd.android.package-archive"
|
||||
// PDF: internal streams are usually already deflated.
|
||||
| "application/pdf"
|
||||
)
|
||||
}
|
||||
|
||||
/// ZIP entry compression for a file of the given MIME type: `Stored` for
|
||||
/// already-compressed content, `Deflate` otherwise. Shared by every ZIP
|
||||
/// export path (`ZipService`, `BatchOperations`).
|
||||
pub fn zip_entry_compression(mime: &str) -> async_zip::Compression {
|
||||
if is_precompressed_mime(mime) {
|
||||
async_zip::Compression::Stored
|
||||
} else {
|
||||
async_zip::Compression::Deflate
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// ── is_precompressed_mime ───────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn media_and_archives_are_precompressed() {
|
||||
for mime in [
|
||||
"image/jpeg",
|
||||
"image/webp",
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"audio/mpeg",
|
||||
"application/zip",
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"font/woff2",
|
||||
] {
|
||||
assert!(is_precompressed_mime(mime), "{mime} should be Stored");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compressible_types_keep_deflate() {
|
||||
for mime in [
|
||||
"text/plain",
|
||||
"text/html",
|
||||
"application/json",
|
||||
"image/svg+xml",
|
||||
"application/x-tar",
|
||||
"application/octet-stream",
|
||||
"",
|
||||
] {
|
||||
assert!(!is_precompressed_mime(mime), "{mime} should stay Deflate");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mime_parameters_are_ignored() {
|
||||
assert!(is_precompressed_mime("image/jpeg; charset=binary"));
|
||||
}
|
||||
|
||||
// ── refine_content_type (sync) ──────────────────────────────
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
//! infrastructure/services/path_service.rs because it has file system dependencies.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use unicode_normalization::UnicodeNormalization;
|
||||
use unicode_normalization::{IsNormalized, UnicodeNormalization, is_nfc_quick};
|
||||
|
||||
/// NFC-normalize a single file or folder name component.
|
||||
///
|
||||
@@ -25,7 +25,18 @@ use unicode_normalization::UnicodeNormalization;
|
||||
/// (`migrate-nfc-filenames`) cleans up rows that pre-date this rule.
|
||||
///
|
||||
/// Pure function — no I/O, allocates one `String`.
|
||||
///
|
||||
/// Fast path: `is_nfc_quick` is a per-char table lookup that answers
|
||||
/// `Yes` for virtually every name already in NFC — which is every name
|
||||
/// loaded back from PostgreSQL (the DB invariant above) and every
|
||||
/// ASCII name. That skips the full decompose/recompose state machine
|
||||
/// this function otherwise runs once per row on every listing
|
||||
/// (PROPFIND, folder listing, photos timeline). `Maybe`/`No` fall
|
||||
/// through to the full pipeline.
|
||||
pub fn normalize_storage_name(name: &str) -> String {
|
||||
if is_nfc_quick(name.chars()) == IsNormalized::Yes {
|
||||
return name.to_string();
|
||||
}
|
||||
name.nfc().collect()
|
||||
}
|
||||
|
||||
|
||||
@@ -10,7 +10,9 @@
|
||||
//! schema and `docs/plan/drive.md` §3 / §15 for the locked design.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use moka::future::Cache;
|
||||
use sqlx::{PgPool, Row, types::Uuid};
|
||||
|
||||
use crate::domain::entities::drive::{Drive, DriveKind};
|
||||
@@ -18,13 +20,38 @@ use crate::domain::repositories::drive_repository::{
|
||||
DriveRepository, DriveRepositoryError, DriveWithRootName,
|
||||
};
|
||||
|
||||
/// `default_drive_cache` TTL. The default-drive → root-folder binding is
|
||||
/// nearly immutable (changes only on provisioning / drive deletion /
|
||||
/// policy edits — all of which invalidate explicitly below), yet it is
|
||||
/// re-resolved on EVERY NextCloud request (basic-auth chroot), every
|
||||
/// native `/webdav` request (Mode-B scope resolution) and every WOPI
|
||||
/// call. 30 s mirrors `drive_role_cache` in `pg_acl_engine.rs` and bounds
|
||||
/// the one non-invalidated staleness source: a root-folder *rename*,
|
||||
/// which doesn't pass through this repository. Measured in
|
||||
/// `benches/CHROOT-CACHE.md`.
|
||||
const DEFAULT_DRIVE_CACHE_TTL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// One entry per active user; entries are small (a `Drive` + a name).
|
||||
const DEFAULT_DRIVE_CACHE_CAPACITY: u64 = 100_000;
|
||||
|
||||
pub struct DrivePgRepository {
|
||||
pool: Arc<PgPool>,
|
||||
/// user_id → default drive (+ root folder name). See
|
||||
/// [`DEFAULT_DRIVE_CACHE_TTL`]. Only `Ok` results are cached, so the
|
||||
/// provisioning idempotency check (`NotFound` → create) always sees
|
||||
/// the live table.
|
||||
default_drive_cache: Cache<Uuid, DriveWithRootName>,
|
||||
}
|
||||
|
||||
impl DrivePgRepository {
|
||||
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||
Self { pool }
|
||||
Self {
|
||||
pool,
|
||||
default_drive_cache: Cache::builder()
|
||||
.max_capacity(DEFAULT_DRIVE_CACHE_CAPACITY)
|
||||
.time_to_live(DEFAULT_DRIVE_CACHE_TTL)
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
|
||||
fn map_sqlx_err(context: &'static str, e: sqlx::Error) -> DriveRepositoryError {
|
||||
@@ -209,6 +236,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("create_personal_drive_atomic.commit", e))?;
|
||||
|
||||
// Drop any cached default-drive resolution for this user (a stale
|
||||
// NotFound is never cached, but be explicit about the write path).
|
||||
self.default_drive_cache.invalidate(&owner_id).await;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
}
|
||||
|
||||
@@ -394,6 +425,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
tx.commit()
|
||||
.await
|
||||
.map_err(|e| Self::map_sqlx_err("delete_atomic.commit", e))?;
|
||||
// We only have the drive id here; the cache is keyed by user.
|
||||
// Deletion is rare — clearing the whole cache is the simple,
|
||||
// always-correct move (repopulates at one query per active user).
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -448,6 +483,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
) -> Result<DriveWithRootName, DriveRepositoryError> {
|
||||
if let Some(cached) = self.default_drive_cache.get(&user_id).await {
|
||||
return Ok(cached);
|
||||
}
|
||||
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT d.id, d.kind, d.default_for_user, d.root_folder_id,
|
||||
@@ -465,7 +504,9 @@ impl DriveRepository for DrivePgRepository {
|
||||
.map_err(|e| Self::map_sqlx_err("find_default_for_user", e))?
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(user_id.to_string()))?;
|
||||
|
||||
Self::row_to_drive_with_name(&row)
|
||||
let dwr = Self::row_to_drive_with_name(&row)?;
|
||||
self.default_drive_cache.insert(user_id, dwr.clone()).await;
|
||||
Ok(dwr)
|
||||
}
|
||||
|
||||
async fn list_readable_by(
|
||||
@@ -675,6 +716,10 @@ impl DriveRepository for DrivePgRepository {
|
||||
let raw = row
|
||||
.ok_or_else(|| DriveRepositoryError::NotFound(drive_id.to_string()))?
|
||||
.0;
|
||||
// Policy edits must not serve a stale `policies` bag from the
|
||||
// default-drive cache (keyed by user, and we only have the drive
|
||||
// id) — clear it; policy edits are admin-rare.
|
||||
self.default_drive_cache.invalidate_all();
|
||||
Ok(crate::domain::entities::drive::DrivePolicies::from_value(
|
||||
&raw,
|
||||
))
|
||||
|
||||
@@ -186,6 +186,60 @@ impl FaceRepository for FacePgRepository {
|
||||
Ok(rows.into_iter().map(row_to_face).collect())
|
||||
}
|
||||
|
||||
async fn person_face_stats(&self, user_id: Uuid) -> Result<Vec<(Uuid, i64)>, DomainError> {
|
||||
// Grouped COUNT — the People tab only needs per-person counts, so
|
||||
// this replaces a full faces_for_user scan that shipped a 2 KiB
|
||||
// embedding BYTEA per row (benches/PEOPLE-LIST.md).
|
||||
let rows: Vec<(Uuid, i64)> = sqlx::query_as(
|
||||
"SELECT person_id, COUNT(*) FROM faces.faces
|
||||
WHERE user_id = $1 AND person_id IS NOT NULL
|
||||
GROUP BY person_id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("person_face_stats", e))?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
async fn file_ids_for_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
face_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashMap<Uuid, Uuid>, DomainError> {
|
||||
if face_ids.is_empty() {
|
||||
return Ok(std::collections::HashMap::new());
|
||||
}
|
||||
let rows: Vec<(Uuid, Uuid)> = sqlx::query_as(
|
||||
"SELECT id, file_id FROM faces.faces WHERE user_id = $1 AND id = ANY($2)",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(face_ids)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("file_ids_for_faces", e))?;
|
||||
Ok(rows.into_iter().collect())
|
||||
}
|
||||
|
||||
async fn reassign_person_faces(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
from: Uuid,
|
||||
into: Uuid,
|
||||
) -> Result<u64, DomainError> {
|
||||
let result = sqlx::query(
|
||||
"UPDATE faces.faces SET person_id = $3
|
||||
WHERE user_id = $1 AND person_id = $2",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(from)
|
||||
.bind(into)
|
||||
.execute(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| db_err("reassign_person_faces", e))?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
async fn assign_person(
|
||||
&self,
|
||||
face_id: Uuid,
|
||||
|
||||
@@ -366,6 +366,30 @@ impl FileBlobReadRepository {
|
||||
.ok_or_else(|| DomainError::not_found("File", file_id))
|
||||
}
|
||||
|
||||
/// Batched variant of [`Self::get_file_drive_id`]: one `= ANY($1)`
|
||||
/// round-trip for a whole result page. Missing / unknown ids are simply
|
||||
/// absent from the output (the single-id variant maps them to
|
||||
/// `NotFound`). Used by `PgAclEngine::check_files_read_batch` — the
|
||||
/// per-hit loop cost up to 200 sequential point SELECTs per content
|
||||
/// search (benches/SEARCH-REBAC.md).
|
||||
pub async fn get_file_drive_ids(
|
||||
&self,
|
||||
file_ids: &[uuid::Uuid],
|
||||
) -> Result<Vec<(uuid::Uuid, uuid::Uuid)>, DomainError> {
|
||||
if file_ids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
sqlx::query_as::<_, (uuid::Uuid, uuid::Uuid)>(
|
||||
"SELECT id, drive_id FROM storage.files WHERE id = ANY($1)",
|
||||
)
|
||||
.bind(file_ids)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("FileBlobRead", format!("drive_id batch lookup: {e}"))
|
||||
})
|
||||
}
|
||||
|
||||
/// Creates a stub instance for testing — never hits PG.
|
||||
/// Available in both standard unit-test (`cfg(test)`) and integration
|
||||
/// (`cfg(integration_tests)`) builds; `PgAclEngine::new_stub` chains
|
||||
@@ -494,7 +518,24 @@ impl FileBlobReadRepository {
|
||||
before: Option<i64>,
|
||||
limit: i64,
|
||||
) -> Result<(Vec<File>, Vec<i64>, Vec<(Option<i32>, Option<i32>)>), DomainError> {
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(
|
||||
// Sargable keyset cursor: compare the RAW `media_sort_date` column
|
||||
// against a timestamptz bind so the planner can use the cursor as
|
||||
// an index boundary condition on `idx_files_media_timeline_by_drive`.
|
||||
// The old shape wrapped the column in `EXTRACT(EPOCH …)::bigint`
|
||||
// (plus an `IS NULL OR` disjunction), which degraded the cursor to
|
||||
// a per-row Filter: page k re-read and discarded all k·limit rows
|
||||
// already scrolled past (benches/PHOTOS-CURSOR.md). Since `before`
|
||||
// is whole seconds, `media_sort_date < to_timestamp(before)` admits
|
||||
// exactly the same rows as the old truncated comparison. The
|
||||
// predicate is emitted only when a cursor exists — a bound
|
||||
// disjunction would block the index condition under generic plans.
|
||||
let cursor_ts = before.and_then(|s| chrono::DateTime::from_timestamp(s, 0));
|
||||
let cursor_pred = if cursor_ts.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
@@ -524,18 +565,18 @@ impl FileBlobReadRepository {
|
||||
)
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
AND ($2::bigint IS NULL
|
||||
OR EXTRACT(EPOCH FROM fi.media_sort_date)::bigint < $2::bigint)
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
"#,
|
||||
)
|
||||
.bind(caller_id)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
|
||||
);
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
|
||||
.bind(caller_id)
|
||||
.bind(cursor_ts)
|
||||
.bind(limit)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_media: {e}")))?;
|
||||
|
||||
let mut files = Vec::with_capacity(rows.len());
|
||||
let mut sort_dates = Vec::with_capacity(rows.len());
|
||||
@@ -768,62 +809,57 @@ impl FileReadPort for FileBlobReadRepository {
|
||||
self.resolve_blob_hash(file_id).await
|
||||
}
|
||||
|
||||
/// Paginated file listing — fetches only `limit` rows starting at `offset`.
|
||||
/// Keyset-paginated file listing in name order — fetches only `limit`
|
||||
/// rows after `after_name` (exclusive).
|
||||
///
|
||||
/// Uses a single SQL query with `LIMIT/OFFSET` to avoid loading the full
|
||||
/// folder contents into memory. Ideal for streaming WebDAV PROPFIND.
|
||||
/// Names are unique per folder, so `name > $after` is a total cursor.
|
||||
/// Served by `idx_files_folder_name (folder_id, name) WHERE NOT
|
||||
/// is_trashed` as a pure index-range read: O(page) per page with no
|
||||
/// sort, where the old `LIMIT/OFFSET` shape re-scanned and re-sorted
|
||||
/// the entire folder for every page (benches/PROPFIND-PAGING.md). The
|
||||
/// cursor predicate is emitted only when a cursor exists — a
|
||||
/// `$2 IS NULL OR name > $2` disjunction would block the index
|
||||
/// condition under the extended protocol's generic plans.
|
||||
#[allow(clippy::type_complexity)]
|
||||
async fn list_files_batch(
|
||||
&self,
|
||||
folder_id: Option<&str>,
|
||||
offset: i64,
|
||||
after_name: Option<&str>,
|
||||
limit: i64,
|
||||
) -> Result<Vec<File>, DomainError> {
|
||||
let rows: Vec<FileRow> = if let Some(fid) = folder_id {
|
||||
sqlx::query_as(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id = $1::uuid AND NOT fi.is_trashed
|
||||
ORDER BY fi.name
|
||||
LIMIT $2 OFFSET $3
|
||||
"#,
|
||||
)
|
||||
.bind(fid)
|
||||
.bind(limit)
|
||||
.bind(offset)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
let folder_pred = if folder_id.is_some() {
|
||||
"fi.folder_id = $1::uuid"
|
||||
} else {
|
||||
sqlx::query_as(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
"fi.folder_id IS NULL AND $1::uuid IS NULL"
|
||||
};
|
||||
let cursor_pred = if after_name.is_some() {
|
||||
"AND fi.name > $3"
|
||||
} else {
|
||||
"AND $3::text IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id IS NULL AND NOT fi.is_trashed
|
||||
ORDER BY fi.name
|
||||
LIMIT $1 OFFSET $2
|
||||
"#,
|
||||
)
|
||||
fi.created_by, fi.updated_by
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE {folder_pred} AND NOT fi.is_trashed {cursor_pred}
|
||||
ORDER BY fi.name
|
||||
LIMIT $2
|
||||
"#,
|
||||
);
|
||||
let rows: Vec<FileRow> = sqlx::query_as(&sql)
|
||||
.bind(folder_id)
|
||||
.bind(limit)
|
||||
.bind(offset)
|
||||
.bind(after_name)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
}
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
|
||||
.map_err(|e| DomainError::internal_error("FileBlobRead", format!("list_batch: {e}")))?;
|
||||
|
||||
rows.into_iter()
|
||||
.map(
|
||||
|
||||
@@ -122,6 +122,35 @@ impl ShareStoragePort for SharePgRepository {
|
||||
Self::row_to_entity(&row)
|
||||
}
|
||||
|
||||
async fn increment_access_count(&self, token: &str) -> Result<u64, DomainError> {
|
||||
// One atomic statement — the relative bump can't lose concurrent
|
||||
// increments and never rewrites unrelated columns (the legacy
|
||||
// read-modify-write wrote back item_name/password_hash wholesale,
|
||||
// silently clobbering concurrent owner edits). The expiry guard
|
||||
// mirrors find_share_by_token's MIN(expires_at) subquery: NULL =
|
||||
// never expires.
|
||||
let result = sqlx::query(
|
||||
r#"
|
||||
UPDATE storage.shares s
|
||||
SET access_count = s.access_count + 1
|
||||
WHERE s.token = $1
|
||||
AND COALESCE(
|
||||
(SELECT MIN(ag.expires_at)
|
||||
FROM storage.role_grants ag
|
||||
WHERE ag.subject_type = 'token' AND ag.subject_id = s.id) > NOW(),
|
||||
TRUE)
|
||||
"#,
|
||||
)
|
||||
.bind(token)
|
||||
.execute(&*self.db_pool)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
tracing::error!("Database error incrementing share access count: {}", e);
|
||||
DomainError::internal_error("Share", format!("Failed to register access: {e}"))
|
||||
})?;
|
||||
Ok(result.rows_affected())
|
||||
}
|
||||
|
||||
async fn find_share_by_token(&self, token: &str) -> Result<Share, DomainError> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
|
||||
@@ -85,6 +85,33 @@ impl UserPgRepository {
|
||||
})
|
||||
}
|
||||
|
||||
/// Fetch only `(storage_used_bytes, storage_quota_bytes)`. Not part of
|
||||
/// the `UserRepository` trait — called from `StorageUsageService`.
|
||||
///
|
||||
/// Same rationale as [`Self::get_user_flags`]: the full-row SELECT drags
|
||||
/// `image` (a data URI of up to 512 KiB), `password_hash`,
|
||||
/// `ui_preferences`, … across the wire, and the quota path runs on every
|
||||
/// folder PROPFIND and every upload quota check just to read two i64s.
|
||||
/// Measured in `benches/QUOTA-PATH.md`.
|
||||
pub async fn get_storage_usage(&self, id: Uuid) -> UserRepositoryResult<(i64, i64)> {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT storage_used_bytes, storage_quota_bytes
|
||||
FROM auth.users
|
||||
WHERE id = $1
|
||||
"#,
|
||||
)
|
||||
.bind(id)
|
||||
.fetch_one(&*self.pool)
|
||||
.await
|
||||
.map_err(Self::map_sqlx_error)?;
|
||||
|
||||
Ok((
|
||||
row.get("storage_used_bytes"),
|
||||
row.get("storage_quota_bytes"),
|
||||
))
|
||||
}
|
||||
|
||||
/// Updates a user's profile image (URL or data URI). Not part of the
|
||||
/// `UserRepository` trait — called directly from `AuthApplicationService`.
|
||||
pub async fn update_image(
|
||||
|
||||
@@ -240,6 +240,16 @@ impl Drop for IngestGuard {
|
||||
/// in the [`BlobStorageBackend`], and maintains a manifest in PostgreSQL
|
||||
/// mapping file_hash → \[chunk_hashes\]. BLAKE3 hashing, ref-counting
|
||||
/// and the PostgreSQL dedup index all live here.
|
||||
/// Immutable chunk map of one CDC blob (`storage.chunk_manifests` row,
|
||||
/// minus the mutable `ref_count`). Content-addressed: for a given
|
||||
/// `file_hash` the chunk list and total size never change, which is what
|
||||
/// makes [`DedupService::manifest_cached`] safe.
|
||||
pub struct ChunkManifest {
|
||||
pub chunk_hashes: Vec<String>,
|
||||
pub chunk_sizes: Vec<i64>,
|
||||
pub total_size: i64,
|
||||
}
|
||||
|
||||
pub struct DedupService {
|
||||
/// Pluggable blob storage backend (local FS, S3, …).
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
@@ -251,6 +261,13 @@ pub struct DedupService {
|
||||
maintenance_pool: Arc<PgPool>,
|
||||
/// Single lifecycle dispatcher — fired on blob created / deleted.
|
||||
blob_lifecycle: Option<Arc<BlobLifecycleService>>,
|
||||
/// `file_hash → ChunkManifest` for the read path — every stream / range
|
||||
/// / full read of a CDC blob used to pay one manifest query first, even
|
||||
/// for the media the gallery re-reads constantly. Positive-only (a
|
||||
/// legacy blob gaining a manifest via background rechunking must be
|
||||
/// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
|
||||
/// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
|
||||
manifest_cache: moka::future::Cache<String, Arc<ChunkManifest>>,
|
||||
}
|
||||
|
||||
impl DedupService {
|
||||
@@ -269,9 +286,22 @@ impl DedupService {
|
||||
pool,
|
||||
maintenance_pool,
|
||||
blob_lifecycle: None,
|
||||
manifest_cache: Self::build_manifest_cache(),
|
||||
}
|
||||
}
|
||||
|
||||
/// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
|
||||
/// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
|
||||
fn build_manifest_cache() -> moka::future::Cache<String, Arc<ChunkManifest>> {
|
||||
moka::future::Cache::builder()
|
||||
.weigher(|key: &String, value: &Arc<ChunkManifest>| {
|
||||
(key.len() + value.chunk_hashes.len() * 80 + 64) as u32
|
||||
})
|
||||
.max_capacity(32 * 1024 * 1024)
|
||||
.time_to_live(std::time::Duration::from_secs(60))
|
||||
.build()
|
||||
}
|
||||
|
||||
/// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
|
||||
pub fn with_blob_lifecycle(mut self, lifecycle: Arc<BlobLifecycleService>) -> Self {
|
||||
self.blob_lifecycle = Some(lifecycle);
|
||||
@@ -311,6 +341,7 @@ impl DedupService {
|
||||
pool: stub_pool.clone(),
|
||||
maintenance_pool: stub_pool,
|
||||
blob_lifecycle: None,
|
||||
manifest_cache: Self::build_manifest_cache(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1385,6 +1416,10 @@ impl DedupService {
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?;
|
||||
|
||||
// Post-commit so a concurrent read can't re-cache the manifest
|
||||
// between invalidation and the delete becoming visible.
|
||||
self.manifest_cache.invalidate(file_hash).await;
|
||||
|
||||
// File content is gone — drop its blob-keyed thumbnails now.
|
||||
self.fire_blob_hooks(file_hash);
|
||||
|
||||
@@ -1606,27 +1641,49 @@ impl DedupService {
|
||||
Box::pin(chunk_stream)
|
||||
}
|
||||
|
||||
/// Cached manifest fetch for the read path (see the `manifest_cache`
|
||||
/// field docs). `None` = legacy whole-file blob — never cached, so a
|
||||
/// background rechunk that creates a manifest is honoured immediately.
|
||||
async fn manifest_cached(&self, hash: &str) -> Result<Option<Arc<ChunkManifest>>, DomainError> {
|
||||
if let Some(m) = self.manifest_cache.get(hash).await {
|
||||
return Ok(Some(m));
|
||||
}
|
||||
let row = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
|
||||
"SELECT chunk_hashes, chunk_sizes, total_size
|
||||
FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
match row {
|
||||
Some((chunk_hashes, chunk_sizes, total_size)) => {
|
||||
let m = Arc::new(ChunkManifest {
|
||||
chunk_hashes,
|
||||
chunk_sizes,
|
||||
total_size,
|
||||
});
|
||||
self.manifest_cache
|
||||
.insert(hash.to_string(), m.clone())
|
||||
.await;
|
||||
Ok(Some(m))
|
||||
}
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Stream blob content — CDC-aware with legacy fallback.
|
||||
///
|
||||
/// For CDC files: looks up the manifest, then streams chunks in order,
|
||||
/// concatenating them into a single byte stream.
|
||||
/// For CDC files: looks up the manifest (RAM-cached), then streams
|
||||
/// chunks in order, concatenating them into a single byte stream.
|
||||
/// For legacy blobs: delegates directly to the backend.
|
||||
pub async fn read_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
// Check manifest
|
||||
let manifest = sqlx::query_scalar::<_, Vec<String>>(
|
||||
"SELECT chunk_hashes FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
match manifest {
|
||||
Some(chunk_hashes) => Ok(self.stream_chunks(chunk_hashes)),
|
||||
match self.manifest_cached(hash).await? {
|
||||
Some(m) => Ok(self.stream_chunks(m.chunk_hashes.clone())),
|
||||
// Legacy whole-file blob
|
||||
None => self.backend.get_blob_stream(hash).await,
|
||||
}
|
||||
@@ -1644,18 +1701,11 @@ impl DedupService {
|
||||
/// `blob_size` + `read_blob_stream`) doubled the manifest round-trips on
|
||||
/// every full-blob read (e.g. 2N queries for an N-image gallery cold load).
|
||||
pub async fn read_blob_bytes(&self, hash: &str) -> Result<Bytes, DomainError> {
|
||||
let manifest = sqlx::query_as::<_, (Vec<String>, i64)>(
|
||||
"SELECT chunk_hashes, total_size FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
let (mut stream, expected_size) = match manifest {
|
||||
Some((chunk_hashes, total_size)) => {
|
||||
(self.stream_chunks(chunk_hashes), total_size.max(0) as usize)
|
||||
}
|
||||
let (mut stream, expected_size) = match self.manifest_cached(hash).await? {
|
||||
Some(m) => (
|
||||
self.stream_chunks(m.chunk_hashes.clone()),
|
||||
m.total_size.max(0) as usize,
|
||||
),
|
||||
None => {
|
||||
// Legacy whole-file blob: size + stream straight from the backend.
|
||||
let size = self.backend.blob_size(hash).await? as usize;
|
||||
@@ -1685,17 +1735,9 @@ impl DedupService {
|
||||
end: Option<u64>,
|
||||
) -> Result<Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>, DomainError>
|
||||
{
|
||||
// Check manifest
|
||||
let manifest = sqlx::query_as::<_, (Vec<String>, Vec<i64>, i64)>(
|
||||
"SELECT chunk_hashes, chunk_sizes, total_size
|
||||
FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
if let Some((chunk_hashes, chunk_sizes, total_size)) = manifest {
|
||||
if let Some(m) = self.manifest_cached(hash).await? {
|
||||
let (chunk_hashes, chunk_sizes, total_size) =
|
||||
(&m.chunk_hashes, &m.chunk_sizes, m.total_size);
|
||||
let end = end.unwrap_or(total_size as u64);
|
||||
|
||||
// Calculate which chunks overlap [start, end)
|
||||
@@ -1749,17 +1791,9 @@ impl DedupService {
|
||||
|
||||
/// Get blob size — manifest-aware with legacy fallback.
|
||||
pub async fn blob_size(&self, hash: &str) -> Result<u64, DomainError> {
|
||||
// Check manifest first (O(1) from PG)
|
||||
let manifest_size = sqlx::query_scalar::<_, i64>(
|
||||
"SELECT total_size FROM storage.chunk_manifests WHERE file_hash = $1",
|
||||
)
|
||||
.bind(hash)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Manifest lookup: {}", e)))?;
|
||||
|
||||
if let Some(size) = manifest_size {
|
||||
return Ok(size as u64);
|
||||
// Check manifest first (RAM cache, else one O(1) PG row)
|
||||
if let Some(m) = self.manifest_cached(hash).await? {
|
||||
return Ok(m.total_size as u64);
|
||||
}
|
||||
|
||||
// Legacy: delegate to backend
|
||||
@@ -2048,6 +2082,7 @@ impl DedupService {
|
||||
}
|
||||
|
||||
for (file_hash, chunk_hashes, size) in &batch {
|
||||
self.manifest_cache.invalidate(file_hash).await;
|
||||
// Decrement chunk ref_counts. GREATEST(.., 0) guards against the
|
||||
// single-chunk file case where the PG file-delete trigger already
|
||||
// decremented blobs.ref_count (because file_hash == chunk_hash);
|
||||
|
||||
@@ -1112,6 +1112,78 @@ impl AuthorizationEngine for PgAclEngine {
|
||||
result
|
||||
}
|
||||
|
||||
/// Batched Read check over a page of file ids (see the trait docs).
|
||||
///
|
||||
/// Decision-equivalent to looping `check`: (1) resolve every file's
|
||||
/// drive in one `= ANY($1)` query (same rows as N ×
|
||||
/// `get_file_drive_id`; absent ids decide `false` exactly like the
|
||||
/// per-file `NotFound` path), (2) evaluate the drive-role floor once
|
||||
/// per distinct drive through the same `drive_role_cache`, (3) send
|
||||
/// only the drive-floor misses through the full per-file cascade —
|
||||
/// preserving per-file grant resolution. `Read` is never gated by the
|
||||
/// read-only drive freeze, so skipping that branch changes nothing.
|
||||
async fn check_files_read_batch(
|
||||
&self,
|
||||
subject: Subject,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<std::collections::HashSet<Uuid>, DomainError> {
|
||||
use std::collections::{HashMap, HashSet};
|
||||
let start = std::time::Instant::now();
|
||||
let counters = QueryCounters::default();
|
||||
|
||||
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
|
||||
let pairs = self.file_repo.get_file_drive_ids(file_ids).await?;
|
||||
|
||||
// Prime the resource→drive cache — later single checks on these
|
||||
// files (download, share) skip their point lookup too.
|
||||
for (file_id, drive_id) in &pairs {
|
||||
self.owner_cache
|
||||
.insert(Resource::File(*file_id), *drive_id)
|
||||
.await;
|
||||
}
|
||||
|
||||
let mut drive_readable: HashMap<Uuid, bool> = HashMap::new();
|
||||
for (_, drive_id) in &pairs {
|
||||
if !drive_readable.contains_key(drive_id) {
|
||||
let ok = self
|
||||
.caller_role_on_drive_cached(subject, *drive_id, &counters)
|
||||
.await?
|
||||
.is_some_and(|role| role.expand().contains(&Permission::Read));
|
||||
drive_readable.insert(*drive_id, ok);
|
||||
}
|
||||
}
|
||||
|
||||
let mut allowed: HashSet<Uuid> = HashSet::with_capacity(pairs.len());
|
||||
for (file_id, drive_id) in &pairs {
|
||||
if drive_readable.get(drive_id).copied().unwrap_or(false) {
|
||||
allowed.insert(*file_id);
|
||||
} else if self
|
||||
.check_inner(
|
||||
subject,
|
||||
Permission::Read,
|
||||
Resource::File(*file_id),
|
||||
&counters,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
// Per-file / folder-cascade grant inside a drive the caller
|
||||
// has no role on — rare, but must keep resolving.
|
||||
allowed.insert(*file_id);
|
||||
}
|
||||
}
|
||||
|
||||
tracing::debug!(
|
||||
target: "oxicloud::authz",
|
||||
event = "authz.check_files_read_batch",
|
||||
subject = %subject,
|
||||
files = file_ids.len(),
|
||||
allowed = allowed.len(),
|
||||
duration_us = start.elapsed().as_micros() as u64,
|
||||
sql_queries = counters.sql_queries.load(Ordering::Relaxed),
|
||||
);
|
||||
Ok(allowed)
|
||||
}
|
||||
|
||||
async fn list_incoming_grants(&self, subject: Subject) -> Result<Vec<Grant>, DomainError> {
|
||||
let counters = QueryCounters::default();
|
||||
let (subject_types, subject_ids) = self.subject_match_set(subject, &counters).await?;
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
//! it was not handled by the path-based store either, so this is a
|
||||
//! parity decision, not a regression.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use sqlx::{PgPool, Row};
|
||||
@@ -126,17 +127,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
|
||||
/// Delete a specific dead property. No-op if not present.
|
||||
///
|
||||
/// Filters on the concrete id column (`folder_id = $1` / `file_id = $1`)
|
||||
/// rather than the old `IS NOT DISTINCT FROM` pair — PostgreSQL cannot
|
||||
/// serve `IS NOT DISTINCT FROM` from a B-tree index, so every lookup
|
||||
/// degraded to a sequential scan as the table grew. The `=` shape is
|
||||
/// served by the partial unique indexes from migration 20260830000001.
|
||||
/// (Same rationale for `get_all` / `get` / the batched readers below —
|
||||
/// measured in `benches/DEAD-PROPS.md`.)
|
||||
pub async fn remove(&self, r: ResourceRef, name: &QualifiedName) -> Result<(), DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
sqlx::query(&format!(
|
||||
"DELETE FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.execute(&*self.pool)
|
||||
@@ -150,28 +157,64 @@ impl DeadPropertyStore {
|
||||
&self,
|
||||
r: ResourceRef,
|
||||
) -> Result<Vec<(QualifiedName, Option<String>)>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let rows = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1",
|
||||
))
|
||||
.bind(id)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("DeadPropertyStore", format!("get_all: {e}")))?;
|
||||
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.map(|r| {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
})
|
||||
.collect())
|
||||
Ok(rows.into_iter().map(row_to_prop).collect())
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every file in a PROPFIND page:
|
||||
/// ONE `file_id = ANY($1)` round-trip instead of N sequential queries.
|
||||
/// Files with no dead properties are simply absent from the map.
|
||||
pub async fn get_all_for_files(
|
||||
&self,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("file_id", file_ids).await
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every subfolder in a PROPFIND page.
|
||||
pub async fn get_all_for_folders(
|
||||
&self,
|
||||
folder_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("folder_id", folder_ids).await
|
||||
}
|
||||
|
||||
async fn get_all_batched(
|
||||
&self,
|
||||
column: &str,
|
||||
ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
if ids.is_empty() {
|
||||
return Ok(HashMap::new());
|
||||
}
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT {column} AS resource_id, namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE {column} = ANY($1)",
|
||||
))
|
||||
.bind(ids)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("DeadPropertyStore", format!("get_all_batched: {e}"))
|
||||
})?;
|
||||
|
||||
let mut map: HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> = HashMap::new();
|
||||
for row in rows {
|
||||
let resource_id: Uuid = row.get("resource_id");
|
||||
map.entry(resource_id).or_default().push(row_to_prop(row));
|
||||
}
|
||||
Ok(map)
|
||||
}
|
||||
|
||||
/// Return a specific dead property, or `None` if not stored.
|
||||
@@ -181,16 +224,14 @@ impl DeadPropertyStore {
|
||||
r: ResourceRef,
|
||||
name: &QualifiedName,
|
||||
) -> Result<Option<Option<String>>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let row = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let row = sqlx::query(&format!(
|
||||
"SELECT value FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.fetch_optional(&*self.pool)
|
||||
@@ -201,16 +242,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Splits a `ResourceRef` into `(folder_id, file_id)` Option pairs for
|
||||
/// binding into SQL. The unused slot is `None` so `IS NOT DISTINCT FROM`
|
||||
/// matches the NULL stored in the unused column.
|
||||
fn split_ref(r: ResourceRef) -> (Option<Uuid>, Option<Uuid>) {
|
||||
/// Maps a `ResourceRef` onto the column that stores it plus the id to bind.
|
||||
/// The column name is one of two compile-time literals — never user input —
|
||||
/// so interpolating it into the SQL text is safe.
|
||||
fn split_ref(r: ResourceRef) -> (&'static str, Uuid) {
|
||||
match r {
|
||||
ResourceRef::Folder(id) => (Some(id), None),
|
||||
ResourceRef::File(id) => (None, Some(id)),
|
||||
ResourceRef::Folder(id) => ("folder_id", id),
|
||||
ResourceRef::File(id) => ("file_id", id),
|
||||
}
|
||||
}
|
||||
|
||||
fn row_to_prop(r: sqlx::postgres::PgRow) -> (QualifiedName, Option<String>) {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
}
|
||||
|
||||
pub fn create_dead_property_store(pool: Arc<PgPool>) -> Arc<DeadPropertyStore> {
|
||||
Arc::new(DeadPropertyStore::new(pool))
|
||||
}
|
||||
|
||||
@@ -52,7 +52,13 @@ enum ZipPlanEntry {
|
||||
/// Directory entry (Stored, zero-length body).
|
||||
Dir(String),
|
||||
/// File entry: ZIP-relative path + file id to stream from the blob store.
|
||||
File { zip_path: String, file_id: String },
|
||||
/// `compression` is picked from the file's MIME type at plan time —
|
||||
/// `Stored` for already-compressed media (JPEG/MP4/…), `Deflate` otherwise.
|
||||
File {
|
||||
zip_path: String,
|
||||
file_id: String,
|
||||
compression: Compression,
|
||||
},
|
||||
}
|
||||
|
||||
/// Message protocol from the prefetch task to the ZIP writer. For each
|
||||
@@ -74,8 +80,11 @@ const PREFETCH_BUFFER_CHUNKS: usize = 64;
|
||||
///
|
||||
/// Uses `async_zip` for fully-async archive creation. Every write (headers,
|
||||
/// compressed chunk data, central directory) goes through
|
||||
/// `tokio::io::BufWriter` → `tokio::fs::File`, so **no Tokio worker is ever
|
||||
/// blocked** by disk I/O or compression.
|
||||
/// `tokio::io::BufWriter` → `tokio::fs::File`, so no Tokio worker is ever
|
||||
/// blocked by disk I/O. Deflate itself DOES run inline on the writing task
|
||||
/// (async_zip compresses inside `poll_write`), which is why entries whose
|
||||
/// MIME says the content is already compressed are `Stored` instead — that
|
||||
/// turns the archive hot path from ~1 CPU core per download into CRC + memcpy.
|
||||
///
|
||||
/// Archive creation is a 2-stage pipeline: a prefetch task reads file
|
||||
/// content from the blob store ahead of the writer, so the next file's
|
||||
@@ -183,6 +192,9 @@ impl ZipService {
|
||||
plan.push(ZipPlanEntry::File {
|
||||
zip_path: format!("{}{}", zip_dir, file.name),
|
||||
file_id: file.id.to_string(),
|
||||
compression: crate::common::mime_detect::zip_entry_compression(
|
||||
&file.mime_type,
|
||||
),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -228,8 +240,12 @@ impl ZipService {
|
||||
}
|
||||
}
|
||||
}
|
||||
ZipPlanEntry::File { zip_path, .. } => {
|
||||
Self::write_prefetched_file(&mut zip, zip_path, &mut rx).await?;
|
||||
ZipPlanEntry::File {
|
||||
zip_path,
|
||||
compression,
|
||||
..
|
||||
} => {
|
||||
Self::write_prefetched_file(&mut zip, zip_path, *compression, &mut rx).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -282,17 +298,19 @@ impl ZipService {
|
||||
}
|
||||
}
|
||||
|
||||
/// Writer stage: drains one file's prefetched chunks into a Deflate
|
||||
/// ZIP entry. Peak memory stays bounded by the channel, independent
|
||||
/// of individual file sizes.
|
||||
/// Writer stage: drains one file's prefetched chunks into a ZIP entry
|
||||
/// (`Stored` for already-compressed media, `Deflate` otherwise — see
|
||||
/// `entry_compression`). Peak memory stays bounded by the channel,
|
||||
/// independent of individual file sizes.
|
||||
async fn write_prefetched_file(
|
||||
zip: &mut AsyncZipWriter,
|
||||
zip_path: &str,
|
||||
compression: Compression,
|
||||
rx: &mut tokio::sync::mpsc::Receiver<Prefetched>,
|
||||
) -> Result<()> {
|
||||
info!("Adding file to ZIP: {}", zip_path);
|
||||
|
||||
let entry = ZipEntryBuilder::new(zip_path.to_string().into(), Compression::Deflate);
|
||||
let entry = ZipEntryBuilder::new(zip_path.to_string().into(), compression);
|
||||
let mut entry_writer = zip
|
||||
.write_entry_stream(entry)
|
||||
.await
|
||||
|
||||
@@ -38,6 +38,7 @@ use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
|
||||
use crate::interfaces::range_requests::{not_modified_response, range_response};
|
||||
use percent_encoding::{AsciiSet, NON_ALPHANUMERIC, percent_decode_str, utf8_percent_encode};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Characters that MUST NOT be percent-encoded inside a URI path segment.
|
||||
@@ -557,7 +558,13 @@ async fn handle_propfind(
|
||||
created_by: None,
|
||||
updated_by: None,
|
||||
};
|
||||
let quota = state.resolve_webdav_quota(user.id, Uuid::nil()).await;
|
||||
// Skip the 2-query quota resolution when the request's prop list
|
||||
// never mentions quota (benches/QUOTA-PATH.md).
|
||||
let quota = if propfind_request.wants_quota() {
|
||||
state.resolve_webdav_quota(user.id, Uuid::nil()).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
return build_streaming_propfind_response(
|
||||
root_folder,
|
||||
None, // folder_id = None → root children (drive-root folders)
|
||||
@@ -597,7 +604,11 @@ async fn handle_propfind(
|
||||
)
|
||||
.await?;
|
||||
let folder_id = folder.id.clone();
|
||||
let quota = state.resolve_webdav_quota(user.id, drive_id).await;
|
||||
let quota = if propfind_request.wants_quota() {
|
||||
state.resolve_webdav_quota(user.id, drive_id).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
return build_streaming_propfind_response(
|
||||
folder,
|
||||
Some(folder_id),
|
||||
@@ -663,7 +674,11 @@ async fn handle_propfind(
|
||||
)
|
||||
.await?;
|
||||
let folder_id = folder.id.clone();
|
||||
let quota = state.resolve_webdav_quota(user.id, drive_id).await;
|
||||
let quota = if propfind_request.wants_quota() {
|
||||
state.resolve_webdav_quota(user.id, drive_id).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
return build_streaming_propfind_response(
|
||||
folder,
|
||||
Some(folder_id),
|
||||
@@ -788,19 +803,18 @@ async fn build_streaming_propfind_response(
|
||||
break;
|
||||
}
|
||||
|
||||
// Materialise dead-props for the whole page before
|
||||
// we start writing — keeps the borrow checker happy
|
||||
// (the writer borrows the FolderDto and the dead-props
|
||||
// vec for the duration of write_folder_entry_*).
|
||||
let mut subfolder_deads = Vec::with_capacity(result.items.len());
|
||||
for subfolder in &result.items {
|
||||
subfolder_deads.push(folder_dead_props(&dead_props_store, subfolder).await);
|
||||
}
|
||||
// ONE batched dead-props query per page instead of a
|
||||
// sequential per-child round-trip — the N+1 shape cost
|
||||
// 1-4.5 s of pure DB chatter on a 2000-child folder
|
||||
// (measured in benches/DEAD-PROPS.md).
|
||||
let subfolder_deads =
|
||||
folders_dead_props_map(&dead_props_store, &result.items).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 800);
|
||||
{
|
||||
let mut w = Writer::new(&mut chunk);
|
||||
for (subfolder, child_dead) in result.items.iter().zip(subfolder_deads.iter()) {
|
||||
for subfolder in result.items.iter() {
|
||||
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
|
||||
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
|
||||
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
@@ -815,11 +829,17 @@ async fn build_streaming_propfind_response(
|
||||
page += 1;
|
||||
}
|
||||
|
||||
// Stream files in pages (user-scoped)
|
||||
let mut offset: i64 = 0;
|
||||
// Stream files in pages (user-scoped, keyset cursor — O(page)
|
||||
// per page instead of the quadratic LIMIT/OFFSET walk).
|
||||
let mut after_name: Option<String> = None;
|
||||
loop {
|
||||
let batch: Vec<FileDto> = file_retrieval_service
|
||||
.list_files_batch_with_perms(fid_ref, user_id, offset, PROPFIND_BATCH_SIZE)
|
||||
.list_files_batch_with_perms(
|
||||
fid_ref,
|
||||
user_id,
|
||||
after_name.as_deref(),
|
||||
PROPFIND_BATCH_SIZE,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
|
||||
@@ -828,15 +848,14 @@ async fn build_streaming_propfind_response(
|
||||
}
|
||||
|
||||
let batch_len = batch.len();
|
||||
let mut file_deads = Vec::with_capacity(batch_len);
|
||||
for file in &batch {
|
||||
file_deads.push(streamed_file_dead_props(&dead_props_store, file).await);
|
||||
}
|
||||
// Batched: one = ANY($1) query per 500-file page.
|
||||
let file_deads = files_dead_props_map(&dead_props_store, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(batch_len * 800);
|
||||
{
|
||||
let mut w = Writer::new(&mut chunk);
|
||||
for (file, child_dead) in batch.iter().zip(file_deads.iter()) {
|
||||
for file in batch.iter() {
|
||||
let child_dead = dead_props_for(&file.id, &file_deads);
|
||||
let href = format!("{}{}", base_href, encode_path_segment(&file.name));
|
||||
WebDavAdapter::write_file_entry_with_dead_props(&mut w, file, &propfind_request, &href, child_dead)
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
@@ -847,7 +866,7 @@ async fn build_streaming_propfind_response(
|
||||
if (batch_len as i64) < PROPFIND_BATCH_SIZE {
|
||||
break;
|
||||
}
|
||||
offset += batch_len as i64;
|
||||
after_name = batch.last().map(|f| f.name.clone());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1381,20 +1400,45 @@ pub(crate) async fn folder_dead_props(
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// File-leaf variant for the streaming walker (takes a `&DeadPropertyStore`
|
||||
/// rather than the full `&Arc<AppState>` so it can be called from inside
|
||||
/// the async-stream future without cloning state).
|
||||
pub(crate) async fn streamed_file_dead_props(
|
||||
/// Batched dead-props fetch for a whole PROPFIND page of files: ONE
|
||||
/// `file_id = ANY($1)` round-trip instead of one query per child (the old
|
||||
/// per-child `streamed_file_dead_props` loop cost seconds on large folders —
|
||||
/// benches/DEAD-PROPS.md). Same leniency as the single-resource helpers:
|
||||
/// any failure → empty map, so the PROPFIND still emits live properties.
|
||||
pub(crate) async fn files_dead_props_map(
|
||||
store: &DeadPropertyStore,
|
||||
file: &FileDto,
|
||||
) -> Vec<(QualifiedName, Option<String>)> {
|
||||
let Ok(file_id) = Uuid::parse_str(&file.id) else {
|
||||
return Vec::new();
|
||||
};
|
||||
store
|
||||
.get_all(ResourceRef::File(file_id))
|
||||
.await
|
||||
.unwrap_or_default()
|
||||
files: &[FileDto],
|
||||
) -> HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> {
|
||||
let ids: Vec<Uuid> = files
|
||||
.iter()
|
||||
.filter_map(|f| Uuid::parse_str(&f.id).ok())
|
||||
.collect();
|
||||
store.get_all_for_files(&ids).await.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Folder-page variant of [`files_dead_props_map`].
|
||||
pub(crate) async fn folders_dead_props_map(
|
||||
store: &DeadPropertyStore,
|
||||
folders: &[FolderDto],
|
||||
) -> HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> {
|
||||
let ids: Vec<Uuid> = folders
|
||||
.iter()
|
||||
.filter_map(|f| Uuid::parse_str(&f.id).ok())
|
||||
.collect();
|
||||
store.get_all_for_folders(&ids).await.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Looks up one resource's dead props in a batched map (resources with no
|
||||
/// dead properties are absent from the map → empty slice).
|
||||
pub(crate) fn dead_props_for<'a>(
|
||||
id: &str,
|
||||
map: &'a HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>,
|
||||
) -> &'a [(QualifiedName, Option<String>)] {
|
||||
Uuid::parse_str(id)
|
||||
.ok()
|
||||
.and_then(|u| map.get(&u))
|
||||
.map(|v| v.as_slice())
|
||||
.unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// A single condition inside a `List` of the WebDAV `If:` header
|
||||
|
||||
@@ -5,11 +5,36 @@ use axum::{
|
||||
response::{IntoResponse, Response},
|
||||
};
|
||||
use base64::Engine;
|
||||
use std::sync::Arc;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
use std::time::Duration;
|
||||
|
||||
use crate::application::dtos::folder_dto::FolderDto;
|
||||
use crate::common::di::AppState;
|
||||
use crate::interfaces::middleware::auth::CurrentUser;
|
||||
|
||||
/// Markerless-chroot cache: default-drive root folder id → `FolderDto`.
|
||||
///
|
||||
/// This middleware wraps EVERY protected NextCloud route (DAV files,
|
||||
/// per-chunk uploads, trashbin, previews, avatars, OCS polls). With the
|
||||
/// app-password verification already cached, the chroot resolution was the
|
||||
/// last per-request DB work: `find_default_for_user` (now cached in
|
||||
/// `DrivePgRepository`) plus this folder-by-PK fetch. A desktop sync run
|
||||
/// issues hundreds of these per minute for a value that changes only on a
|
||||
/// root-folder rename — the 30 s TTL bounds that staleness (mirrors
|
||||
/// `drive_role_cache` / the default-drive cache; measured in
|
||||
/// `benches/CHROOT-CACHE.md`).
|
||||
///
|
||||
/// Only the MARKERLESS branch is cached: it targets the caller's own
|
||||
/// default drive root, so no per-request authorization decision is being
|
||||
/// skipped. The drive-marker branch keeps its `get_folder_with_perms`
|
||||
/// check on every request.
|
||||
static NC_CHROOT_CACHE: LazyLock<moka::sync::Cache<uuid::Uuid, FolderDto>> = LazyLock::new(|| {
|
||||
moka::sync::Cache::builder()
|
||||
.max_capacity(100_000)
|
||||
.time_to_live(Duration::from_secs(30))
|
||||
.build()
|
||||
});
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum NextcloudAuthError {
|
||||
#[error("Unauthorized")]
|
||||
@@ -191,12 +216,24 @@ pub async fn basic_auth_middleware(
|
||||
.find_default_for_user(current_user.id)
|
||||
.await
|
||||
{
|
||||
Ok(drive_with_name) => state
|
||||
.applications
|
||||
.folder_service
|
||||
.get_folder(&drive_with_name.drive.root_folder_id.to_string())
|
||||
.await
|
||||
.ok(),
|
||||
Ok(drive_with_name) => {
|
||||
let root_id = drive_with_name.drive.root_folder_id;
|
||||
match NC_CHROOT_CACHE.get(&root_id) {
|
||||
Some(cached) => Some(cached),
|
||||
None => {
|
||||
let fetched = state
|
||||
.applications
|
||||
.folder_service
|
||||
.get_folder(&root_id.to_string())
|
||||
.await
|
||||
.ok();
|
||||
if let Some(f) = &fetched {
|
||||
NC_CHROOT_CACHE.insert(root_id, f.clone());
|
||||
}
|
||||
fetched
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,7 +21,9 @@ use crate::application::ports::folder_ports::FolderUseCase;
|
||||
use crate::application::ports::inbound::SearchUseCase;
|
||||
use crate::common::di::AppState;
|
||||
use crate::domain::entities::file::File;
|
||||
use crate::interfaces::api::handlers::webdav_handler::{file_dead_props, folder_dead_props};
|
||||
use crate::interfaces::api::handlers::webdav_handler::{
|
||||
dead_props_for, files_dead_props_map, folders_dead_props_map,
|
||||
};
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::nextcloud::webdav_handler::{
|
||||
batch_resolve_ids, format_oc_id, nc_href, write_file_response, write_folder_response,
|
||||
@@ -160,6 +162,11 @@ async fn handle_filter_files(
|
||||
|
||||
write_multistatus_start(&mut xml)?;
|
||||
|
||||
// Batched dead-props: one = ANY($1) query per type, not one per
|
||||
// result (benches/DEAD-PROPS.md).
|
||||
let file_deads = files_dead_props_map(&state.webdav_dead_props, &files).await;
|
||||
let folder_deads = folders_dead_props_map(&state.webdav_dead_props, &folders).await;
|
||||
|
||||
// Keep main's batched-resolution structure (one batch query
|
||||
// per type, not 2N round-trips). Hrefs use `url_user` so the
|
||||
// multi-drive `~{drive}` form is echoed back to the client;
|
||||
@@ -179,7 +186,7 @@ async fn handle_filter_files(
|
||||
let href = nc_href(url_user, subpath);
|
||||
let fid = file_id_map.get(&file.id).copied();
|
||||
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
|
||||
let dead = file_dead_props(&state, file).await;
|
||||
let dead = dead_props_for(&file.id, &file_deads);
|
||||
write_file_response(
|
||||
&mut xml,
|
||||
file,
|
||||
@@ -187,7 +194,7 @@ async fn handle_filter_files(
|
||||
(fid, oc_id.as_deref()),
|
||||
&user.username,
|
||||
&favorite_ids,
|
||||
&dead,
|
||||
dead,
|
||||
)
|
||||
.map_err(|e| AppError::internal_error(format!("XML write error: {}", e)))?;
|
||||
}
|
||||
@@ -205,7 +212,7 @@ async fn handle_filter_files(
|
||||
let href = format!("{}/", nc_href(url_user, subpath));
|
||||
let fid = folder_id_map.get(&folder.id).copied();
|
||||
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
|
||||
let dead = folder_dead_props(&state.webdav_dead_props, folder).await;
|
||||
let dead = dead_props_for(&folder.id, &folder_deads);
|
||||
write_folder_response(
|
||||
&mut xml,
|
||||
folder,
|
||||
@@ -217,7 +224,7 @@ async fn handle_filter_files(
|
||||
// PROPFIND on a specific collection — quota isn't
|
||||
// meaningful here (see `AppState::resolve_webdav_quota`).
|
||||
None,
|
||||
&dead,
|
||||
dead,
|
||||
)
|
||||
.map_err(|e| AppError::internal_error(format!("XML write error: {}", e)))?;
|
||||
}
|
||||
@@ -301,6 +308,11 @@ async fn handle_search(
|
||||
|
||||
write_multistatus_start(&mut xml)?;
|
||||
|
||||
// Batched dead-props: one = ANY($1) query per type, not one per
|
||||
// result (benches/DEAD-PROPS.md).
|
||||
let file_deads = files_dead_props_map(&state.webdav_dead_props, &files).await;
|
||||
let folder_deads = folders_dead_props_map(&state.webdav_dead_props, &folders).await;
|
||||
|
||||
// Files.
|
||||
for file in &files {
|
||||
let Some(subpath) = strip_home_prefix(chroot, &file.path, home_prefix) else {
|
||||
@@ -315,7 +327,7 @@ async fn handle_search(
|
||||
let href = nc_href(url_user, subpath);
|
||||
let fid = file_id_map.get(&file.id).copied();
|
||||
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
|
||||
let dead = file_dead_props(&state, file).await;
|
||||
let dead = dead_props_for(&file.id, &file_deads);
|
||||
write_file_response(
|
||||
&mut xml,
|
||||
file,
|
||||
@@ -323,7 +335,7 @@ async fn handle_search(
|
||||
(fid, oc_id.as_deref()),
|
||||
&user.username,
|
||||
&favorite_ids,
|
||||
&dead,
|
||||
dead,
|
||||
)
|
||||
.map_err(|e| AppError::internal_error(format!("XML write error: {}", e)))?;
|
||||
}
|
||||
@@ -342,7 +354,7 @@ async fn handle_search(
|
||||
let href = format!("{}/", nc_href(url_user, subpath));
|
||||
let fid = folder_id_map.get(&folder.id).copied();
|
||||
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
|
||||
let dead = folder_dead_props(&state.webdav_dead_props, folder).await;
|
||||
let dead = dead_props_for(&folder.id, &folder_deads);
|
||||
write_folder_response(
|
||||
&mut xml,
|
||||
folder,
|
||||
@@ -354,7 +366,7 @@ async fn handle_search(
|
||||
// PROPFIND on a specific collection — quota isn't
|
||||
// meaningful here (see `AppState::resolve_webdav_quota`).
|
||||
None,
|
||||
&dead,
|
||||
dead,
|
||||
)
|
||||
.map_err(|e| AppError::internal_error(format!("XML write error: {}", e)))?;
|
||||
}
|
||||
|
||||
@@ -30,7 +30,8 @@ use crate::domain::services::authorization::{Permission, Resource, Subject};
|
||||
use crate::infrastructure::services::path_resolver_service::ResolvedResource;
|
||||
use crate::infrastructure::services::webdav_dead_property_store::ResourceRef;
|
||||
use crate::interfaces::api::handlers::webdav_handler::{
|
||||
PROPFIND_BATCH_SIZE, file_dead_props, folder_dead_props, streamed_file_dead_props,
|
||||
PROPFIND_BATCH_SIZE, dead_props_for, file_dead_props, files_dead_props_map, folder_dead_props,
|
||||
folders_dead_props_map,
|
||||
};
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::range_requests::{not_modified_response, range_response};
|
||||
@@ -297,9 +298,10 @@ async fn handle_propfind(
|
||||
.map_err(|e| AppError::bad_request(format!("Failed to read body: {}", e)))?;
|
||||
|
||||
// Parse (and thereby validate) the PROPFIND body. The NC response
|
||||
// always emits the full property set, so the parsed request is not
|
||||
// consulted further — but malformed XML must still fail with 400.
|
||||
let _propfind = if body_bytes.is_empty() {
|
||||
// always emits the full property set; the parsed request is consulted
|
||||
// only to skip the quota DB round-trips when the client's explicit
|
||||
// prop list never names a quota prop. Malformed XML still fails 400.
|
||||
let propfind = if body_bytes.is_empty() {
|
||||
PropFindRequest {
|
||||
prop_find_type: crate::application::adapters::webdav_adapter::PropFindType::AllProp,
|
||||
}
|
||||
@@ -341,7 +343,13 @@ async fn handle_propfind(
|
||||
// function's username arg. Refining the owner-id usages
|
||||
// back to the canonical username is deferred to the
|
||||
// NcSession commit.
|
||||
let quota = state.resolve_webdav_quota(user.id, chroot.drive_id).await;
|
||||
// Explicit prop lists that never name a quota prop skip the
|
||||
// 2-query quota resolution (benches/QUOTA-PATH.md).
|
||||
let quota = if propfind.wants_quota() {
|
||||
state.resolve_webdav_quota(user.id, chroot.drive_id).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
Ok(build_nc_streaming_propfind(
|
||||
state.clone(),
|
||||
folder,
|
||||
@@ -1519,11 +1527,17 @@ fn build_nc_streaming_propfind(
|
||||
|
||||
// ── Children (only if Depth != 0) ────────────────────────────
|
||||
if depth != "0" {
|
||||
// Files in pages.
|
||||
let mut offset: i64 = 0;
|
||||
// Files in pages (keyset cursor — O(page) per page instead of
|
||||
// the quadratic LIMIT/OFFSET walk).
|
||||
let mut after_name: Option<String> = None;
|
||||
loop {
|
||||
let batch = file_service
|
||||
.list_files_batch_with_perms(Some(&folder.id), user_id, offset, PROPFIND_BATCH_SIZE)
|
||||
.list_files_batch_with_perms(
|
||||
Some(&folder.id),
|
||||
user_id,
|
||||
after_name.as_deref(),
|
||||
PROPFIND_BATCH_SIZE,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
if batch.is_empty() {
|
||||
@@ -1541,15 +1555,15 @@ fn build_nc_streaming_propfind(
|
||||
};
|
||||
let file_uuids: Vec<String> = batch.iter().map(|f| f.id.clone()).collect();
|
||||
let (file_id_map, _) = batch_resolve_ids(file_id_svc, &file_uuids, &[]).await;
|
||||
let mut file_deads = Vec::with_capacity(batch_len);
|
||||
for file in &batch {
|
||||
file_deads.push(streamed_file_dead_props(&state.webdav_dead_props, file).await);
|
||||
}
|
||||
// One batched dead-props query per page, not one per child
|
||||
// (benches/DEAD-PROPS.md).
|
||||
let file_deads = files_dead_props_map(&state.webdav_dead_props, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(batch_len * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for (file, dead) in batch.iter().zip(file_deads.iter()) {
|
||||
for file in batch.iter() {
|
||||
let dead = dead_props_for(&file.id, &file_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
file.name.clone()
|
||||
} else {
|
||||
@@ -1567,7 +1581,7 @@ fn build_nc_streaming_propfind(
|
||||
if (batch_len as i64) < PROPFIND_BATCH_SIZE {
|
||||
break;
|
||||
}
|
||||
offset += batch_len as i64;
|
||||
after_name = batch.last().map(|f| f.name.clone());
|
||||
}
|
||||
|
||||
// Subfolders in pages — also collections, same trailing-slash rule.
|
||||
@@ -1594,15 +1608,15 @@ fn build_nc_streaming_propfind(
|
||||
};
|
||||
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
|
||||
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
|
||||
let mut sub_deads = Vec::with_capacity(result.items.len());
|
||||
for sf in &result.items {
|
||||
sub_deads.push(folder_dead_props(&state.webdav_dead_props, sf).await);
|
||||
}
|
||||
// Batched — see benches/DEAD-PROPS.md.
|
||||
let sub_deads =
|
||||
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for (sf, dead) in result.items.iter().zip(sub_deads.iter()) {
|
||||
for sf in result.items.iter() {
|
||||
let dead = dead_props_for(&sf.id, &sub_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
sf.name.clone()
|
||||
} else {
|
||||
|
||||
@@ -46,10 +46,23 @@ pub fn create_web_routes() -> Router<Arc<AppState>> {
|
||||
let static_path = resolve_static_path(&config);
|
||||
|
||||
// SPA fallback: serve the file if it exists, else the app shell.
|
||||
let spa = ServeDir::new(&static_path).fallback(ServeFile::new(static_path.join("index.html")));
|
||||
//
|
||||
// `precompressed_*`: if the frontend build emitted a sibling `.br`/`.gz`
|
||||
// (frontend/scripts/precompress.mjs runs at build time), serve those
|
||||
// bytes directly with the right Content-Encoding instead of re-running
|
||||
// Brotli over the same immutable bundle on EVERY request — the
|
||||
// `CompressionLayer` below then skips the already-encoded response and
|
||||
// remains only the fallback for assets without a precompressed sibling
|
||||
// (benches/STATIC-PRECOMPRESSED.md).
|
||||
let spa = ServeDir::new(&static_path)
|
||||
.precompressed_br()
|
||||
.precompressed_gzip()
|
||||
.fallback(ServeFile::new(static_path.join("index.html")));
|
||||
|
||||
// Hashed, immutable assets (SvelteKit emits these under /_app/immutable).
|
||||
let app_immutable = ServeDir::new(static_path.join("_app").join("immutable"));
|
||||
let app_immutable = ServeDir::new(static_path.join("_app").join("immutable"))
|
||||
.precompressed_br()
|
||||
.precompressed_gzip();
|
||||
|
||||
Router::new()
|
||||
.nest_service(
|
||||
@@ -60,7 +73,17 @@ pub fn create_web_routes() -> Router<Arc<AppState>> {
|
||||
)),
|
||||
)
|
||||
.fallback_service(spa)
|
||||
.layer(CompressionLayer::new().br(true).gzip(true))
|
||||
// Fallback compression for assets without a precompressed sibling.
|
||||
// Quality 4, NOT the default: the default maps to Brotli q11 —
|
||||
// ~1.3 s of CPU per 700 KiB bundle per request (measured in
|
||||
// benches/STATIC-PRECOMPRESSED.md; the .br siblings above carry the
|
||||
// real q11 bytes, paid once at build time).
|
||||
.layer(
|
||||
CompressionLayer::new()
|
||||
.quality(tower_http::CompressionLevel::Precise(4))
|
||||
.br(true)
|
||||
.gzip(true),
|
||||
)
|
||||
// `if_not_present` so the immutable assets above keep their long cache;
|
||||
// the shell itself must always revalidate so a deploy can't pin a stale
|
||||
// app in browsers.
|
||||
|
||||
+12
-1
@@ -880,7 +880,18 @@ async fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
// ── file-body downloads carry Content-Disposition (see above) ──
|
||||
.and(NotForDownloads);
|
||||
|
||||
app = app.layer(CompressionLayer::new().compress_when(predicate));
|
||||
// Explicit quality: the layer's default maps to Brotli QUALITY 11
|
||||
// (async-compression Level::Default → BrotliEncoderParams::default(),
|
||||
// brotli-8.0.2 encode.rs:323) — a deploy-grade setting that cost
|
||||
// ~90 ms of CPU per 64 KiB JSON response. Level 4 emits ~15 % more
|
||||
// bytes at ~1 % of the CPU (0.9 ms) — measured in
|
||||
// benches/STATIC-PRECOMPRESSED.md. Applies to gzip too (level 4,
|
||||
// the classic dynamic-content setting).
|
||||
app = app.layer(
|
||||
CompressionLayer::new()
|
||||
.quality(tower_http::CompressionLevel::Precise(4))
|
||||
.compress_when(predicate),
|
||||
);
|
||||
}
|
||||
|
||||
// ── Security headers ─────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user