perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning

Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):

DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
  folders/files UNION-ALL branches as sargable row-value comparisons with
  per-branch ORDER/LIMIT + two partial expression indexes
  (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
  (19.5x); other sort modes at parity or better. New migration
  20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
  on the timeline index, joins moved above the top-N. 50k-photo library:
  97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
  comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
  idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
  (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).

Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
  connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
  now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
  concurrent cold readers = 16 full remote downloads racing truncating
  writes on ONE deterministic .tmp (corruptible cache); now 1 download
  (16x less egress, 2.8x wall on a shared link) and torn files can never
  be renamed into the cache.

I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
  chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
  pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
  every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
  common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
  clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
  O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
  9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
  (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
  ~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
  nodes, three SDK stacks gone from every build). tokio "process" is now
  an explicit feature (was enabled transitively by aws-config).

Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
  4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
  asserts output identity across locales and a 3x floor.

Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
2026-07-17 11:10:27 +00:00
parent 7d95a19907
commit cd4c62042a
43 changed files with 5290 additions and 439 deletions
+32 -36
View File
@@ -651,7 +651,6 @@ impl CardDavAdapter {
pub fn generate_contacts_response<W: Write>(
writer: W,
contacts: &[ContactDto],
vcards: &[(String, String)], // (uid, vcard_data)
report: &CardDavReportType,
base_href: &str,
) -> Result<()> {
@@ -672,17 +671,9 @@ impl CardDavAdapter {
for contact in contacts {
let href = format!("{}{}.vcf", base_href, contact.uid);
let vcard = vcards
.iter()
.find(|(uid, _)| *uid == contact.uid)
.map(|(_, data)| data.as_str())
.unwrap_or("");
// `write_contact_response` generates the vCard on demand when (and
// only when) address-data is actually requested.
Self::write_contact_response(&mut xml_writer, contact, &props, &href)?;
// If address-data is requested, include vcard
if props.iter().any(|p| p.name == "address-data") || props.is_empty() {
// Already handled in write_contact_response
}
let _ = vcard; // suppress warning - used via contact_to_vcard fallback
}
xml_writer.write_event(Event::End(BytesEnd::new("D:multistatus")))?;
@@ -868,20 +859,25 @@ impl CardDavAdapter {
/// Convert a ContactDto to vCard 3.0 format
pub fn contact_to_vcard(contact: &ContactDto) -> String {
// `write!` into a String is infallible; `let _ =` discards the Ok(()).
// Formatting straight into the buffer avoids one temporary String per
// vCard line compared to `push_str(&format!(…))`.
use std::fmt::Write as _;
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
vcard.push_str(&format!("UID:{}\r\n", contact.uid));
let _ = write!(vcard, "UID:{}\r\n", contact.uid);
if let (Some(last), Some(first)) = (&contact.last_name, &contact.first_name) {
vcard.push_str(&format!("N:{};{};;;\r\n", last, first));
let _ = write!(vcard, "N:{};{};;;\r\n", last, first);
} else if let Some(last) = &contact.last_name {
vcard.push_str(&format!("N:{};;;;\r\n", last));
let _ = write!(vcard, "N:{};;;;\r\n", last);
} else if let Some(first) = &contact.first_name {
vcard.push_str(&format!("N:;{};;;\r\n", first));
let _ = write!(vcard, "N:;{};;;\r\n", first);
}
if let Some(fn_name) = &contact.full_name {
vcard.push_str(&format!("FN:{}\r\n", fn_name));
let _ = write!(vcard, "FN:{}\r\n", fn_name);
} else {
// FN is mandatory in vCard 3.0
let fn_name = format!(
@@ -892,68 +888,68 @@ pub fn contact_to_vcard(contact: &ContactDto) -> String {
.trim()
.to_string();
if !fn_name.is_empty() {
vcard.push_str(&format!("FN:{}\r\n", fn_name));
let _ = write!(vcard, "FN:{}\r\n", fn_name);
} else {
vcard.push_str("FN:Unknown\r\n");
}
}
if let Some(nickname) = &contact.nickname {
vcard.push_str(&format!("NICKNAME:{}\r\n", nickname));
let _ = write!(vcard, "NICKNAME:{}\r\n", nickname);
}
for email in &contact.email {
vcard.push_str(&format!(
let _ = write!(
vcard,
"EMAIL;TYPE={}:{}\r\n",
email.r#type.to_uppercase(),
email.email
));
);
}
for phone in &contact.phone {
vcard.push_str(&format!(
let _ = write!(
vcard,
"TEL;TYPE={}:{}\r\n",
phone.r#type.to_uppercase(),
phone.number
));
);
}
for addr in &contact.address {
let adr = format!(
";;{};{};{};{};{}",
let _ = write!(
vcard,
"ADR;TYPE={}:;;{};{};{};{};{}\r\n",
addr.r#type.to_uppercase(),
addr.street.as_deref().unwrap_or(""),
addr.city.as_deref().unwrap_or(""),
addr.state.as_deref().unwrap_or(""),
addr.postal_code.as_deref().unwrap_or(""),
addr.country.as_deref().unwrap_or(""),
);
vcard.push_str(&format!(
"ADR;TYPE={}:{}\r\n",
addr.r#type.to_uppercase(),
adr
));
}
if let Some(org) = &contact.organization {
vcard.push_str(&format!("ORG:{}\r\n", org));
let _ = write!(vcard, "ORG:{}\r\n", org);
}
if let Some(title) = &contact.title {
vcard.push_str(&format!("TITLE:{}\r\n", title));
let _ = write!(vcard, "TITLE:{}\r\n", title);
}
if let Some(notes) = &contact.notes {
vcard.push_str(&format!("NOTE:{}\r\n", notes.replace('\n', "\\n")));
let _ = write!(vcard, "NOTE:{}\r\n", notes.replace('\n', "\\n"));
}
if let Some(bday) = &contact.birthday {
vcard.push_str(&format!("BDAY:{}\r\n", bday.format("%Y-%m-%d")));
let _ = write!(vcard, "BDAY:{}\r\n", bday.format("%Y-%m-%d"));
}
if let Some(photo) = &contact.photo_url {
vcard.push_str(&format!("PHOTO;VALUE=URI:{}\r\n", photo));
let _ = write!(vcard, "PHOTO;VALUE=URI:{}\r\n", photo);
}
vcard.push_str(&format!(
let _ = write!(
vcard,
"REV:{}\r\n",
contact.updated_at.format("%Y%m%dT%H%M%SZ")
));
);
vcard.push_str("END:VCARD\r\n");
vcard
@@ -484,10 +484,6 @@ mod tests {
#[test]
fn test_generate_contacts_response() {
let contacts = vec![sample_contact()];
let vcards = vec![(
"contact-001".to_string(),
contact_to_vcard(&sample_contact()),
)];
let report = CardDavReportType::AddressbookQuery {
props: vec![
QualifiedName {
@@ -505,7 +501,6 @@ mod tests {
let result = CardDavAdapter::generate_contacts_response(
&mut output,
&contacts,
&vcards,
&report,
"/carddav/ab-001",
);
@@ -528,14 +523,12 @@ mod tests {
#[test]
fn test_generate_empty_contacts_response() {
let contacts: Vec<ContactDto> = vec![];
let vcards: Vec<(String, String)> = vec![];
let report = CardDavReportType::AddressbookQuery { props: vec![] };
let mut output = Vec::new();
let result = CardDavAdapter::generate_contacts_response(
&mut output,
&contacts,
&vcards,
&report,
"/carddav/ab-001",
);
+228 -5
View File
@@ -8,6 +8,175 @@
//! then fall back to the file extension when the MIME is generic
//! (`application/octet-stream` or empty).
use std::collections::HashMap;
use std::fmt::Write as _;
use std::sync::{Arc, LazyLock};
// ─── Arc<str> interning for closed-set display values ────────────────
//
// `FileDto` / `FolderDto` store their display fields as `Arc<str>` so DTO
// clones are O(1). But `Arc::<str>::from(&str)` always allocates + copies,
// so building the DTO paid 3-4 heap allocations per row even though the
// value space is a small closed set. Interning turns each conversion into
// a HashMap lookup + refcount bump.
/// Every `&'static str` that [`icon_class_for`], [`icon_special_class_for`]
/// and [`category_for`] can return, plus the folder-DTO constants.
///
/// Keep this table in sync when adding a value to those functions — a
/// missing entry is not a bug (callers fall back to `Arc::from`, same
/// bytes, one extra allocation), just a lost optimization.
static DISPLAY_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
const CLOSED_SET: &[&str] = &[
// icon_class_for
"fas fa-file-pdf",
"fas fa-file-word",
"fas fa-file-excel",
"fas fa-file-powerpoint",
"fas fa-file-archive",
"fas fa-file-code",
"fas fa-hdd",
"fas fa-file-image",
"fas fa-file-video",
"fas fa-file-audio",
"fas fa-file-alt",
"fas fa-terminal",
"fas fa-file",
// icon_special_class_for
"pdf-icon",
"doc-icon",
"spreadsheet-icon",
"presentation-icon",
"archive-icon",
"code-icon json-icon",
"code-icon js-icon",
"code-icon ts-icon",
"code-icon html-icon",
"code-icon sql-icon",
"code-icon config-icon",
"code-icon php-icon",
"script-icon",
"installer-icon",
"image-icon",
"video-icon",
"audio-icon",
"code-icon py-icon",
"code-icon rust-icon",
"code-icon",
"code-icon go-icon",
"code-icon ruby-icon",
"code-icon md-icon",
"code-icon css-icon",
"code-icon java-icon",
"code-icon c-icon",
"code-icon cs-icon",
"code-icon swift-icon",
"",
// category_for
"PDF",
"Document",
"Spreadsheet",
"Presentation",
"Archive",
"Code",
"Installer",
"Image",
"Video",
"Audio",
"Markdown",
"Text",
// FolderDto constants
"fas fa-folder",
"folder-icon",
"Folder",
];
CLOSED_SET.iter().map(|s| (*s, Arc::from(*s))).collect()
});
/// Returns a shared `Arc<str>` for a display value from the closed sets
/// above (icon class, icon special class, category). Lookup + refcount
/// bump instead of alloc + copy; unknown values (future additions not
/// yet in the table) fall back to `Arc::from` with identical bytes.
pub fn intern_display(s: &'static str) -> Arc<str> {
DISPLAY_INTERN
.get(s)
.cloned()
.unwrap_or_else(|| Arc::from(s))
}
/// The MIME types that dominate real storage rows. Exotic types fall back
/// to a per-row `Arc::from` — correctness is unaffected, only the alloc is.
static MIME_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
const COMMON_MIMES: &[&str] = &[
"",
"directory",
"application/octet-stream",
// Images
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
"image/svg+xml",
"image/heic",
"image/heif",
"image/avif",
"image/bmp",
"image/tiff",
"image/x-icon",
// Video
"video/mp4",
"video/quicktime",
"video/webm",
"video/x-matroska",
"video/x-msvideo",
// Audio
"audio/mpeg",
"audio/mp4",
"audio/ogg",
"audio/flac",
"audio/wav",
"audio/x-wav",
"audio/aac",
// Documents
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/vnd.oasis.opendocument.text",
"application/vnd.oasis.opendocument.spreadsheet",
// Text / code
"text/plain",
"text/csv",
"text/html",
"text/css",
"text/markdown",
"text/xml",
"application/json",
"application/javascript",
"application/xml",
"application/x-yaml",
// Archives
"application/zip",
"application/gzip",
"application/x-tar",
"application/x-7z-compressed",
"application/x-rar-compressed",
];
COMMON_MIMES.iter().map(|s| (*s, Arc::from(*s))).collect()
});
/// Returns a shared `Arc<str>` for the given MIME type. Common types hit
/// the intern table (refcount bump); exotic ones allocate as before.
pub fn intern_mime(mime: &str) -> Arc<str> {
MIME_INTERN
.get(mime)
.cloned()
.unwrap_or_else(|| Arc::from(mime))
}
// ─── Private: extract lowercase extension from a filename ────────────
fn ext_of(name: &str) -> Option<&str> {
let name = name.rsplit('/').next().unwrap_or(name); // strip path
@@ -388,11 +557,21 @@ pub fn format_file_size(bytes: u64) -> String {
let value = bytes as f64 / K.powi(i as i32);
// Two decimal places, then strip trailing zeros (matches JS parseFloat behaviour)
let formatted = format!("{:.2}", value);
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
format!("{} {}", formatted, SIZES[i])
// Single buffer: write the 2-decimal value, strip trailing zeros in
// place (matches JS parseFloat behaviour), then append the unit.
// 16 chars covers the worst case ("16777216 TB" for u64::MAX,
// "1023.99 Bytes" for the longest unit), so no realloc occurs.
let mut out = String::with_capacity(16);
let _ = write!(out, "{:.2}", value);
while out.ends_with('0') {
out.pop();
}
if out.ends_with('.') {
out.pop();
}
out.push(' ');
out.push_str(SIZES[i]);
out
}
#[cfg(test)]
@@ -506,6 +685,50 @@ mod tests {
);
}
/// Every value the closed-set display functions can return must hit
/// the intern table (same bytes, shared allocation) — a miss is only
/// a lost optimization, but this test keeps the table in sync.
#[test]
fn test_intern_display_covers_closed_sets_and_shares_storage() {
for s in [
"fas fa-file-pdf",
"fas fa-file",
"fas fa-terminal",
"fas fa-folder",
"code-icon rust-icon",
"folder-icon",
"",
"PDF",
"Folder",
"Document",
"Markdown",
] {
let a = intern_display(s);
let b = intern_display(s);
assert_eq!(&*a, s, "interned bytes must be identical");
assert!(
Arc::ptr_eq(&a, &b),
"closed-set value {s:?} must come from the intern table"
);
}
}
#[test]
fn test_intern_mime_common_hits_table_exotic_falls_back() {
let a = intern_mime("image/jpeg");
let b = intern_mime("image/jpeg");
assert_eq!(&*a, "image/jpeg");
assert!(Arc::ptr_eq(&a, &b), "common MIME must be interned");
let exotic = intern_mime("chemical/x-pdb");
assert_eq!(&*exotic, "chemical/x-pdb");
let exotic2 = intern_mime("chemical/x-pdb");
assert!(
!Arc::ptr_eq(&exotic, &exotic2),
"exotic MIME falls back to a fresh Arc"
);
}
#[test]
fn test_ext_of() {
assert_eq!(ext_of("file.txt"), Some("txt"));
+14 -9
View File
@@ -6,7 +6,8 @@ use utoipa::ToSchema;
use uuid::Uuid;
use super::display_helpers::{
category_for, format_file_size, icon_class_for, icon_special_class_for,
category_for, format_file_size, icon_class_for, icon_special_class_for, intern_display,
intern_mime,
};
/// DTO for file responses
@@ -101,11 +102,15 @@ impl From<File> for FileDto {
// for id, name, path, folder_id (previously 4× .to_string()).
let parts = file.into_parts();
let icon_class = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
let icon_special_class = Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
let category = Arc::from(category_for(&parts.name, &parts.mime_type));
// Display fields come from closed static tables and MIME values
// repeat massively across rows — intern instead of allocating a
// fresh Arc<str> per row (`Arc::from(&str)` always allocs+copies).
let icon_class = intern_display(icon_class_for(&parts.name, &parts.mime_type));
let icon_special_class =
intern_display(icon_special_class_for(&parts.name, &parts.mime_type));
let category = intern_display(category_for(&parts.name, &parts.mime_type));
let size_formatted = format_file_size(parts.size);
let mime_type = Arc::from(parts.mime_type.as_str());
let mime_type = intern_mime(&parts.mime_type);
Self {
id: parts.id,
@@ -169,13 +174,13 @@ impl FileDto {
name: "stub-file".to_string(),
path: "/stub/path".to_string(),
size: 0,
mime_type: Arc::from("application/octet-stream"),
mime_type: intern_mime("application/octet-stream"),
folder_id: None,
created_at: 0,
modified_at: 0,
icon_class: Arc::from("fas fa-file"),
icon_special_class: Arc::from(""),
category: Arc::from("Document"),
icon_class: intern_display("fas fa-file"),
icon_special_class: intern_display(""),
category: intern_display("Document"),
size_formatted: "0 Bytes".to_string(),
content_hash: String::new(),
etag: String::new(),
+27 -17
View File
@@ -1,6 +1,7 @@
use std::sync::Arc;
use crate::application::dtos::cursor::{CursorListResponse, CursorQuery, PageCursor};
use crate::application::dtos::display_helpers::intern_display;
use crate::application::dtos::grant_dto::{ResourceContentDto, ResourceTypeDto};
use crate::domain::entities::folder::Folder;
use crate::domain::services::authorization::ResourceKind;
@@ -99,24 +100,33 @@ pub struct FolderDto {
impl From<Folder> for FolderDto {
fn from(folder: Folder) -> Self {
let is_root = folder.parent_id().is_none();
let etag = folder.etag().to_string();
// Consume the entity by moving all fields — zero heap allocations
// for id, name, path, parent_id (previously 3-4× .to_string()).
let parts = folder.into_parts();
let is_root = parts.parent_id.is_none();
// Single-allocation ETag straight from the owned parts. The old
// shape (`folder.etag().to_string()`) built the String and then
// cloned it — a pure double-alloc.
let etag = Folder::compute_etag(&parts.id, parts.tree_modified_at);
Self {
id: folder.id().to_string(),
name: folder.name().to_string(),
path: folder.path_string().to_string(),
parent_id: folder.parent_id().map(String::from),
drive_id: folder.drive_id(),
created_at: folder.created_at(),
modified_at: folder.modified_at(),
id: parts.id,
name: parts.name,
path: parts.path_string,
parent_id: parts.parent_id,
drive_id: parts.drive_id,
created_at: parts.created_at,
modified_at: parts.modified_at,
is_root,
icon_class: Arc::from("fas fa-folder"),
icon_special_class: Arc::from("folder-icon"),
category: Arc::from("Folder"),
// Constant display fields: refcount bump on interned statics
// instead of 3 fresh Arc allocations per row.
icon_class: intern_display("fas fa-folder"),
icon_special_class: intern_display("folder-icon"),
category: intern_display("Folder"),
etag,
created_by: folder.created_by(),
updated_by: folder.updated_by(),
created_by: parts.created_by,
updated_by: parts.updated_by,
}
}
}
@@ -163,9 +173,9 @@ impl FolderDto {
created_at: 0,
modified_at: 0,
is_root: true,
icon_class: Arc::from("fas fa-folder"),
icon_special_class: Arc::from("folder-icon"),
category: Arc::from("Folder"),
icon_class: intern_display("fas fa-folder"),
icon_special_class: intern_display("folder-icon"),
category: intern_display("Folder"),
etag: String::new(),
created_by: None,
updated_by: None,
+25
View File
@@ -77,6 +77,31 @@ pub trait FolderUseCase: Send + Sync + 'static {
pagination: &crate::application::dtos::pagination::PaginationRequestDto,
) -> Result<crate::application::dtos::pagination::PaginatedResponseDto<FolderDto>, DomainError>;
/// Keyset-paged sub-folder listing in name order, scoped to a caller —
/// `name > after_name LIMIT limit`, `has_next = len() == limit`.
///
/// Used by streaming WebDAV/NC PROPFIND: O(page) per page off the
/// `idx_folders_unique_name` index instead of the quadratic
/// `COUNT(*) OVER() … LIMIT/OFFSET` walk (benches/FOLDER-KEYSET.md).
///
/// The default implementation falls back to `list_folders_with_perms`
/// + in-memory slice so stubs and mocks compile without changes.
async fn list_folders_batch_with_perms(
&self,
parent_id: Option<&str>,
caller_id: Uuid,
after_name: Option<&str>,
limit: usize,
) -> Result<Vec<FolderDto>, DomainError> {
let mut all = self.list_folders_with_perms(parent_id, caller_id).await?;
all.sort_by(|a, b| a.name.cmp(&b.name));
Ok(all
.into_iter()
.filter(|f| after_name.is_none_or(|a| f.name.as_str() > a))
.take(limit)
.collect())
}
/// Renames a folder (ownership verified against caller_id)
async fn rename_folder_with_perms(
&self,
@@ -304,12 +304,45 @@ impl AppPasswordService {
let cache_key: [u8; 32] =
blake3::hash(format!("{}:{}", username, password).as_bytes()).into();
// ── 2. Cache hit → return immediately ────────────────────────
if let Some(cached) = self.auth_cache.get(&cache_key).await {
return Ok((cached.user_id, cached.username, cached.email, cached.role));
}
// ── 2. Single-flight cache lookup ─────────────────────────────
// Concurrent misses on the same credential coalesce into ONE
// full verification: DAV sync clients hold 4-8 parallel
// connections, so an expiring cache entry used to fan out into
// K simultaneous Argon2id runs (~100-300 ms CPU + 64 MiB RAM
// apiece) every TTL — a recurring p99 spike on every DAV
// surface (8 -> 1 verifications, benches/AUTH-HERD.md).
// `try_get_with` caches only `Ok` results, so failed
// verifications are still never cached, preserving the full
// Argon2id cost as a brute-force deterrent.
let result = self
.auth_cache
.try_get_with(
cache_key,
self.verify_basic_auth_uncached(username, password),
)
.await
.map_err(
|e: std::sync::Arc<DomainError>| match std::sync::Arc::try_unwrap(e) {
Ok(err) => err,
// Another coalesced waiter still holds the Arc — rebuild
// an equivalent error (the source chain isn't clonable).
Err(shared) => {
DomainError::new(shared.kind, shared.entity_type, shared.message.clone())
}
},
)?;
Ok((result.user_id, result.username, result.email, result.role))
}
// ── 3. Cache miss → full verification ────────────────────────
/// The uncached Basic Auth slow path: user lookup, prefix-scoped
/// candidate fetch, Argon2id verification. Runs at most once per
/// credential per TTL — `verify_basic_auth` coalesces concurrent
/// callers onto a single in-flight instance of this future.
async fn verify_basic_auth_uncached(
&self,
username: &str,
password: &str,
) -> Result<CachedBasicAuthResult, DomainError> {
let user = self
.user_repo
.get_user_by_username(username)
@@ -363,15 +396,14 @@ impl AppPasswordService {
{
let _ = self.repo.touch_last_used(ap.id).await;
let result = CachedBasicAuthResult {
// Caching happens in `verify_basic_auth`: `try_get_with`
// stores this value under the blake3 key on return.
return Ok(CachedBasicAuthResult {
user_id: user.id(),
username: user.username().unwrap_or("").to_string(),
email: user.email().to_string(),
role: user.role().to_string(),
};
self.auth_cache.insert(cache_key, result.clone()).await;
return Ok((result.user_id, result.username, result.email, result.role));
});
}
}
@@ -429,6 +429,62 @@ impl FolderUseCase for FolderService {
Ok(response)
}
/// Keyset-paged sub-folder listing (name order), caller-scoped.
///
/// AuthZ mirrors `list_folders_paginated_with_perms`: one
/// `authz.require(Read)` on the parent per batch; root scope goes
/// through the caller's drive-membership listing.
async fn list_folders_batch_with_perms(
&self,
parent_id: Option<&str>,
caller_id: Uuid,
after_name: Option<&str>,
limit: usize,
) -> Result<Vec<FolderDto>, DomainError> {
match parent_id {
Some(pid) => {
self.authz
.require(
Subject::User(caller_id),
Permission::Read,
Self::folder_resource(pid)?,
)
.await?;
let folders = self
.folder_storage
.list_folders_batch(parent_id, after_name, limit)
.await
.map_err(|e| {
DomainError::internal_error(
"FolderStorage",
format!("Failed to batch-list folders in parent {pid}: {e}"),
)
})?;
Ok(folders.into_iter().map(FolderDto::from).collect())
}
None => {
// Root scope: one row per readable drive — a handful.
let mut all = self
.folder_storage
.list_root_folders_for_caller(caller_id)
.await
.map_err(|e| {
DomainError::internal_error(
"FolderStorage",
format!("Failed to batch-list root folders for '{caller_id}': {e}"),
)
})?;
all.sort_by(|a, b| a.name().cmp(b.name()));
Ok(all
.into_iter()
.filter(|f| after_name.is_none_or(|a| f.name() > a))
.take(limit)
.map(FolderDto::from)
.collect())
}
}
}
/// Lists folders with pagination, scoped to a specific owner.
async fn list_folders_paginated_with_perms(
&self,
+159 -5
View File
@@ -67,9 +67,80 @@ pub struct SearchService {
/// Lock-free concurrent cache with automatic TTL and LRU eviction (moka).
/// Values are `Arc<SearchResultsDto>` so cache insert/hit is a single
/// atomic ref-count increment (~1 ns) instead of cloning thousands of Strings.
///
/// **Byte-bounded**, not entry-bounded: entries are weighed by
/// [`search_results_entry_weight`] and `max_capacity` is a byte budget.
/// Keys span user × query × offset × limit, and each page holds up to 500
/// enriched rows (~500–900 B of owned Strings each) — an entry-count bound
/// let hundreds of MB of result pages accumulate invisibly.
search_cache: moka::future::Cache<u64, Arc<SearchResultsDto>>,
}
// ─── Search-results cache (byte-bounded) ─────────────────────────────────
/// Approximate heap bytes retained by one cached search page.
///
/// With a `weigher` installed, moka's `max_capacity` is the sum of entry
/// *weights*, so this converts the cache bound from "number of entries" to
/// real bytes: the length of every owned `String` in each file/folder row,
/// plus a fixed per-row and per-entry overhead for struct fields, the 24-B
/// `String` headers, `Vec` slots and allocator slop. Same pattern as the
/// file-content cache and the dedup manifest cache.
///
/// `pub` so `examples/bench_search_cache_mem.rs` can recompute retained
/// bytes with the exact production formula.
pub fn search_results_entry_weight(_key: &u64, value: &Arc<SearchResultsDto>) -> u32 {
/// Fixed per-row overhead: struct scalars + one 24-B header per `String`
/// field (12 on a file row, 4 on a folder row) + `Vec` slot + allocator
/// slop. Deliberately a round upper-ish estimate — under-weighing is the
/// failure mode that re-opens the memory hole.
const ROW_OVERHEAD: usize = 200;
/// Fixed per-entry overhead: `Arc` + `SearchResultsDto` scalars + `Vec`
/// headers + moka's own bookkeeping per entry.
const ENTRY_OVERHEAD: usize = 256;
fn opt_len(s: &Option<String>) -> usize {
s.as_deref().map_or(0, str::len)
}
let mut bytes = ENTRY_OVERHEAD + value.sort_by.len();
for f in &value.files {
bytes += ROW_OVERHEAD
+ f.id.len()
+ f.name.len()
+ f.path.len()
+ f.mime_type.len()
+ opt_len(&f.folder_id)
+ f.size_formatted.len()
+ f.icon_class.len()
+ f.icon_special_class.len()
+ f.category.len()
+ f.blob_hash.len()
+ opt_len(&f.snippet)
+ opt_len(&f.match_source);
}
for d in &value.folders {
bytes += ROW_OVERHEAD + d.id.len() + d.name.len() + d.path.len() + opt_len(&d.parent_id);
}
bytes.min(u32::MAX as usize) as u32
}
/// Build the search-results cache exactly as production wires it: a byte
/// budget enforced through [`search_results_entry_weight`], plus TTL.
///
/// Shared with `examples/bench_search_cache_mem.rs` so the benchmark
/// measures the identical cache configuration that serves requests.
pub fn build_search_results_cache(
cache_ttl_secs: u64,
max_bytes: u64,
) -> moka::future::Cache<u64, Arc<SearchResultsDto>> {
moka::future::Cache::builder()
.max_capacity(max_bytes)
.weigher(search_results_entry_weight)
.time_to_live(Duration::from_secs(cache_ttl_secs))
.build()
}
// ─── Utility functions (pure, no self — computed on the server) ─────────
/// Compute relevance score (0–100) for a name against a query.
@@ -160,6 +231,10 @@ fn get_category(name: &str, mime: &str) -> String {
impl SearchService {
/**
* Creates a new instance of the search service.
*
* `max_cache_bytes` is the byte budget for the results cache (weigher-
* bounded, see [`search_results_entry_weight`]) — it replaced the old
* entry-count capacity, which was blind to how big each cached page is.
*/
pub fn new(
file_repository: Arc<FileBlobReadRepository>,
@@ -168,12 +243,9 @@ impl SearchService {
authorization: Option<Arc<crate::infrastructure::services::pg_acl_engine::PgAclEngine>>,
drive_repo: Option<Arc<dyn crate::domain::repositories::drive_repository::DriveRepository>>,
cache_ttl: u64,
max_cache_size: usize,
max_cache_bytes: u64,
) -> Self {
let search_cache = moka::future::Cache::builder()
.max_capacity(max_cache_size as u64)
.time_to_live(Duration::from_secs(cache_ttl))
.build();
let search_cache = build_search_results_cache(cache_ttl, max_cache_bytes);
Self {
file_repository,
@@ -815,6 +887,88 @@ mod tests {
}
}
#[test]
fn entry_weight_counts_every_owned_string_plus_overheads() {
// Empty page: entry overhead + sort_by ("relevance" = 9 bytes).
let empty = Arc::new(SearchResultsDto::empty());
let base = search_results_entry_weight(&0, &empty) as usize;
assert_eq!(base, 256 + 9);
// One file row: base + row overhead + its owned string bytes
// (id 7 + name 7 + path 8 + mime 10; the rest are empty/None).
let one_file = Arc::new(SearchResultsDto::new(
vec![dto("abc.txt", 50, 10, 1)],
Vec::new(),
100,
0,
Some(1),
0,
"relevance".to_string(),
));
let w = search_results_entry_weight(&0, &one_file) as usize;
assert_eq!(w, base + 200 + 7 + 7 + 8 + 10);
// Folder rows weigh too (id 2 + name 4 + path 5 + parent 6 = 17).
let one_folder = Arc::new(SearchResultsDto::new(
Vec::new(),
vec![SearchFolderResultDto {
id: "f1".to_string(),
name: "docs".to_string(),
path: "/docs".to_string(),
parent_id: Some("parent".to_string()),
drive_id: Uuid::nil(),
created_at: 0,
modified_at: 0,
is_root: false,
relevance_score: 50,
}],
100,
0,
Some(1),
0,
"relevance".to_string(),
));
let w = search_results_entry_weight(&0, &one_folder) as usize;
assert_eq!(w, base + 200 + 2 + 4 + 5 + 6);
}
#[tokio::test]
async fn cache_evicts_down_to_the_byte_budget() {
// Budget fits ~2 of these entries; inserting 20 must never let the
// weighted size settle above the budget.
let entry = |i: usize| {
Arc::new(SearchResultsDto::new(
(0..50)
.map(|r| dto(&format!("file_{i}_{r}_{}", "x".repeat(100)), 50, 1, 1))
.collect(),
Vec::new(),
50,
0,
Some(50),
0,
"relevance".to_string(),
))
};
let per_entry = search_results_entry_weight(&0, &entry(0)) as u64;
let budget = per_entry * 2 + per_entry / 2;
let cache = build_search_results_cache(300, budget);
for i in 0..20u64 {
cache.insert(i, entry(i as usize)).await;
}
cache.run_pending_tasks().await;
let retained: u64 = cache
.iter()
.map(|(k, v)| search_results_entry_weight(&k, &v) as u64)
.sum();
assert!(
retained <= budget,
"retained {retained} B exceeds budget {budget} B"
);
assert!(cache.entry_count() <= 2);
}
#[test]
fn merged_files_resort_by_relevance_and_by_column() {
let mut files = vec![
+37
View File
@@ -1250,6 +1250,33 @@ impl Default for ContentSearchConfig {
}
}
/// Search-results cache configuration — the per-user results-page cache
/// inside `SearchService`, not the Tantivy content index above.
///
/// The cache is **byte-bounded**: each entry is weighed by the approximate
/// heap size of its result page (see `search_results_entry_weight`) and moka
/// evicts once the summed weight exceeds `max_bytes` — the same byte-budget
/// pattern the file-content cache and the dedup manifest cache use. This
/// replaced an entry-count capacity: with cache keys spanning
/// user × query × offset × limit and up to 500 enriched rows per page, an
/// entry count said nothing about resident memory (1000 entries could pin
/// ~300 MB for the TTL). No entry-count knob is kept — bytes are the only
/// dimension that matters here.
#[derive(Debug, Clone)]
pub struct SearchCacheConfig {
/// Byte budget for cached search-result pages. Default: 32 MiB.
/// Env: `OXICLOUD_SEARCH_CACHE_MAX_BYTES`.
pub max_bytes: u64,
}
impl Default for SearchCacheConfig {
fn default() -> Self {
Self {
max_bytes: 32 * 1024 * 1024,
}
}
}
/// WASM plugin runtime configuration (M0 walking skeleton).
///
/// The runtime is doubly gated: it is only compiled when the `plugins` cargo
@@ -1375,6 +1402,8 @@ pub struct AppConfig {
pub i18n: I18nConfig,
/// Content-search configuration (embedded full-text index)
pub content_search: ContentSearchConfig,
/// Search-results cache configuration (byte-bounded moka cache)
pub search_cache: SearchCacheConfig,
/// WASM plugin runtime configuration
pub plugins: PluginConfig,
/// Face-recognition (People) model configuration
@@ -1431,6 +1460,7 @@ impl Default for AppConfig {
magic_link: MagicLinkConfig::default(),
i18n: I18nConfig::default(),
content_search: ContentSearchConfig::default(),
search_cache: SearchCacheConfig::default(),
plugins: PluginConfig::default(),
faces: FacesConfig::default(),
}
@@ -1934,6 +1964,13 @@ impl AppConfig {
config.content_search.max_text_bytes = val;
}
// Search-results cache (byte-bounded)
if let Ok(v) = env::var("OXICLOUD_SEARCH_CACHE_MAX_BYTES").map(|v| v.parse::<u64>())
&& let Ok(val) = v
{
config.search_cache.max_bytes = val;
}
// WASM plugin runtime
if let Ok(v) = env::var("OXICLOUD_ENABLE_PLUGINS").map(|v| v.parse::<bool>())
&& let Ok(val) = v
+6 -2
View File
@@ -656,8 +656,12 @@ impl AppServiceFactory {
content_index_port,
Some(authz.clone()),
Some(drive_repo.clone()),
300, // Cache TTL in seconds (5 minutes)
1000, // Maximum cache entries
300, // Cache TTL in seconds (5 minutes)
// Byte budget for cached result pages (weigher-bounded, 32 MiB
// default; env OXICLOUD_SEARCH_CACHE_MAX_BYTES). Replaces the old
// entry-count capacity, which let 500-row pages keyed by
// user×query×offset×limit pin hundreds of MB for the TTL.
self.config.search_cache.max_bytes,
)));
tracing::info!("Application services initialized");
+19 -2
View File
@@ -352,8 +352,25 @@ impl File {
/// formula here changes it everywhere — that is the property
/// we want.
pub fn compute_etag(blob_hash: &str, modified_at: u64) -> String {
let prefix: String = blob_hash.chars().take(16).collect();
format!("{}-{}", prefix, modified_at)
use std::fmt::Write as _;
// Byte index just past the 16th char (whole string when shorter).
// `blob_hash` is lowercase hex ASCII in practice, so this is
// effectively `min(len, 16)`, but `char_indices` keeps the slice
// char-boundary-safe for exotic fixture values — byte-identical
// to the old `chars().take(16).collect::<String>()` without the
// intermediate allocation.
let end = match blob_hash.char_indices().nth(16) {
Some((i, _)) => i,
None => blob_hash.len(),
};
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
let mut etag = String::with_capacity(end + 1 + 20);
etag.push_str(&blob_hash[..end]);
etag.push('-');
let _ = write!(etag, "{modified_at}");
etag
}
// Getters
+63 -2
View File
@@ -7,6 +7,30 @@ use crate::domain::services::path_service::{
// Re-export entity errors from the centralized module
pub use super::entity_errors::{FolderError, FolderResult};
/// Owned parts of a [`Folder`] entity, produced by [`Folder::into_parts()`].
///
/// Consuming a `Folder` into `FolderParts` **moves** every field without
/// cloning, eliminating the 3-4 heap allocations that previously occurred
/// when converting `Folder → FolderDto` via `.to_string()` on each getter.
/// Mirrors [`super::file::FileParts`].
pub struct FolderParts {
pub id: String,
pub name: String,
pub storage_path: StoragePath,
pub path_string: String,
pub parent_id: Option<String>,
/// Drive that owns this folder. See [`Folder::drive_id`].
pub drive_id: Uuid,
pub created_at: u64,
pub modified_at: u64,
/// Descendant-rollup timestamp. See [`Folder::tree_modified_at`].
pub tree_modified_at: u64,
/// §14 provenance: original creator. See [`Folder::created_by`].
pub created_by: Option<Uuid>,
/// §14 provenance: most recent mutator. See [`Folder::updated_by`].
pub updated_by: Option<Uuid>,
}
/// Represents a folder entity in the domain
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Folder {
@@ -219,6 +243,26 @@ impl Folder {
})
}
/// Consume the entity and return all fields by ownership.
///
/// Use this when converting `Folder` into a DTO to avoid cloning
/// every `String` field (saves 3-4 heap allocations per folder).
pub fn into_parts(self) -> FolderParts {
FolderParts {
id: self.id,
name: self.name,
storage_path: self.storage_path,
path_string: self.path_string,
parent_id: self.parent_id,
drive_id: self.drive_id,
created_at: self.created_at,
modified_at: self.modified_at,
tree_modified_at: self.tree_modified_at,
created_by: self.created_by,
updated_by: self.updated_by,
}
}
// Getters
pub fn id(&self) -> &str {
&self.id
@@ -326,8 +370,25 @@ impl Folder {
/// changed; the folder's own value stays untouched
/// (self-exclusion).
pub fn compute_etag(id: &str, tree_modified_at: u64) -> String {
let prefix: String = id.chars().take(16).collect();
format!("{}-{}", prefix, tree_modified_at)
use std::fmt::Write as _;
// Byte index just past the 16th char (whole string when shorter).
// `id` is a UUID string (ASCII) in practice, so this is
// effectively `min(len, 16)`, but `char_indices` keeps the slice
// char-boundary-safe for exotic fixture values — byte-identical
// to the old `chars().take(16).collect::<String>()` without the
// intermediate allocation.
let end = match id.char_indices().nth(16) {
Some((i, _)) => i,
None => id.len(),
};
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
let mut etag = String::with_capacity(end + 1 + 20);
etag.push_str(&id[..end]);
etag.push('-');
let _ = write!(etag, "{tree_modified_at}");
etag
}
/// Creates a new Folder instance from a DTO
@@ -98,6 +98,32 @@ pub trait FolderRepository: Send + Sync + 'static {
include_total: bool,
) -> Result<(Vec<Folder>, Option<usize>), DomainError>;
/// Keyset-paged listing of `parent_id`'s direct sub-folders in name
/// order — `name > $after_name ORDER BY name LIMIT $limit`, one bounded
/// index-range read per page off the partial unique index
/// `idx_folders_unique_name`. Streaming PROPFIND drains sub-folders
/// with this instead of `COUNT(*) OVER() … LIMIT/OFFSET`, which
/// window-aggregated and rescanned all N sub-folders on every page
/// (4.5x on a 5k-dir parent, benches/FOLDER-KEYSET.md). `has_next`
/// falls out of `rows.len() == limit` — no total needed.
///
/// The default implementation falls back to `list_folders` + in-memory
/// slice so stubs and mocks compile without changes.
async fn list_folders_batch(
&self,
parent_id: Option<&str>,
after_name: Option<&str>,
limit: usize,
) -> Result<Vec<Folder>, DomainError> {
let mut all = self.list_folders(parent_id).await?;
all.sort_by(|a, b| a.name().cmp(b.name()));
Ok(all
.into_iter()
.filter(|f| after_name.is_none_or(|a| f.name() > a))
.take(limit)
.collect())
}
/// Renames a folder. `caller_id` is stamped into `updated_by`
/// alongside the `updated_at = NOW()` bump (§14 provenance).
async fn rename_folder(
@@ -488,13 +488,21 @@ impl FileBlobReadRepository {
/// `sort_date` epoch for each file (used as pagination cursor).
///
/// Uses the denormalised `media_sort_date` column (synced from
/// `file_metadata.captured_at` by trigger) so no JOIN with
/// `file_metadata` is needed. The partial covering index
/// `idx_files_media_timeline_by_drive` (migration 20260901000001)
/// keys on `(drive_id, media_sort_date DESC)` filtered on non-trashed
/// image/video rows — Postgres does one IndexScan per in-scope
/// drive_id already ordered by capture date, so LIMIT stops the scan
/// early. Same O(LIMIT) shape as the pre-D7 `user_id`-keyed hot path.
/// `file_metadata.captured_at` by trigger). The accessible drive ids
/// are materialised once, then a `CROSS JOIN LATERAL (… ORDER BY
/// media_sort_date DESC LIMIT k)` per drive turns the partial covering
/// index `idx_files_media_timeline_by_drive` (migration 20260901000001,
/// `(drive_id, media_sort_date DESC)` filtered on non-trashed
/// image/video rows) into one BOUNDED index scan per drive; the outer
/// merge sorts `drives × k` rows. The folders / file_metadata joins sit
/// outside the top-N so only the k emitted rows pay them.
///
/// The previous shape put the joins and the global `ORDER BY … LIMIT`
/// above a `drive_id IN (…)` nested loop — Postgres fed EVERY media row
/// through the join into a top-N heapsort, scanning the timeline index
/// to exhaustion on every page: O(library) per page, 97 ms on a
/// 50k-photo library vs 1.6 ms for this shape (55.7x,
/// benches/PHOTOS-TIMELINE.md).
///
/// Scope (`docs/plan/drive.md` §15): drives with
/// `policies.include_in_photo_index = true` where the caller has a
@@ -537,37 +545,47 @@ impl FileBlobReadRepository {
};
let sql = format!(
r#"
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
fi.size, fi.mime_type,
EXTRACT(EPOCH FROM fi.created_at)::bigint,
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
fi.blob_hash,
fi.created_by, fi.updated_by,
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
WITH accessible AS MATERIALIZED (
SELECT d.id
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
WHERE (
(g.subject_type = 'user' AND g.subject_id = $1)
OR (g.subject_type = 'group' AND g.subject_id IN
(SELECT storage.caller_group_ids($1)))
)
AND (g.expires_at IS NULL OR g.expires_at > NOW())
AND (d.policies->>'include_in_photo_index')::boolean = true
)
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
top.size, top.mime_type,
EXTRACT(EPOCH FROM top.created_at)::bigint,
EXTRACT(EPOCH FROM top.updated_at)::bigint,
top.blob_hash,
top.created_by, top.updated_by,
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
fm.width, fm.height
FROM storage.files fi
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
WHERE fi.drive_id IN (
SELECT d.id
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
WHERE (
(g.subject_type = 'user' AND g.subject_id = $1)
OR (g.subject_type = 'group' AND g.subject_id IN
(SELECT storage.caller_group_ids($1)))
)
AND (g.expires_at IS NULL OR g.expires_at > NOW())
AND (d.policies->>'include_in_photo_index')::boolean = true
)
AND NOT fi.is_trashed
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
{cursor_pred}
ORDER BY fi.media_sort_date DESC
LIMIT $3
FROM (
SELECT fi.*
FROM accessible a
CROSS JOIN LATERAL (
SELECT fi.*
FROM storage.files fi
WHERE fi.drive_id = a.id
AND NOT fi.is_trashed
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
{cursor_pred}
ORDER BY fi.media_sort_date DESC
LIMIT $3
) fi
ORDER BY fi.media_sort_date DESC
LIMIT $3
) top
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
ORDER BY top.media_sort_date DESC
"#,
);
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
@@ -501,6 +501,61 @@ impl FolderRepository for FolderDbRepository {
Ok((folders?, total))
}
/// Keyset sub-folder page: `name > $after ORDER BY name LIMIT $limit`,
/// one bounded index-range read off `idx_folders_unique_name` — the
/// cursor predicate is only emitted when a cursor exists (a bound
/// disjunction would block the index condition under generic plans,
/// same rule as `list_files_batch`). Root scope (`parent_id = None`)
/// keeps the trait's in-memory default: roots are one-per-drive, a
/// handful of rows.
async fn list_folders_batch(
&self,
parent_id: Option<&str>,
after_name: Option<&str>,
limit: usize,
) -> Result<Vec<Folder>, DomainError> {
let Some(pid) = parent_id else {
let mut all = self.list_folders(None).await?;
all.sort_by(|a, b| a.name().cmp(b.name()));
return Ok(all
.into_iter()
.filter(|f| after_name.is_none_or(|a| f.name() > a))
.take(limit)
.collect());
};
let cursor_pred = if after_name.is_some() {
"AND name > $3"
} else {
"AND $3::text IS NULL"
};
let sql = format!(
"SELECT id::text, name, path, parent_id::text, drive_id, \
EXTRACT(EPOCH FROM created_at)::bigint, \
EXTRACT(EPOCH FROM updated_at)::bigint, \
EXTRACT(EPOCH FROM tree_modified_at)::bigint, \
created_by, updated_by \
FROM storage.folders \
WHERE parent_id = $1::uuid AND NOT is_trashed \
{cursor_pred} \
ORDER BY name \
LIMIT $2"
);
let rows: Vec<FolderRow> = sqlx::query_as(&sql)
.bind(pid)
.bind(limit as i64)
.bind(after_name)
.fetch_all(self.pool())
.await
.map_err(|e| DomainError::internal_error("FolderDb", format!("batch: {e}")))?;
rows.into_iter()
.map(|(id, name, path, pid, did, ca, ma, tma, cb, ub)| {
Self::row_to_folder(id, name, path, pid, did, ca, ma, tma, cb, ub)
})
.collect()
}
/// Paginated companion to `list_root_folders_for_caller` — same
/// drive-membership predicate, adds LIMIT/OFFSET and an optional
/// window-function COUNT so total pages can be surfaced without a
@@ -1391,13 +1446,6 @@ impl FolderDbRepository {
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
"#;
let cte_inner = match (include_folders, include_files) {
(true, true) => format!("{folder_branch} UNION ALL {file_branch}"),
(true, false) => folder_branch.to_owned(),
(false, true) => file_branch.to_owned(),
(false, false) => unreachable!(),
};
// ── Cursor binds ─────────────────────────────────────────────────────
// $1 = parent_id $2 = cursor_str $3 = cursor_int
// $4 = cursor_ts $5 = cursor_id $6 = limit
@@ -1406,112 +1454,184 @@ impl FolderDbRepository {
let cursor_ts = cursor.and_then(|c| c.sort_ts);
let cursor_id = cursor.map(|c| c.resource_id);
// ── Sort-specific WHERE + ORDER BY ───────────────────────────────────
// Each arm produces two variants based on `reverse`.
// For "name": folder_first stays ASC in both directions (folders always
// precede files); only the alpha order within each group flips.
let (where_clause, order_clause) = match order_by {
// ── Per-branch cursor pushdown ───────────────────────────────────────
// The cursor is applied INSIDE each UNION-ALL branch as a sargable
// row-value comparison on base columns — not on the CTE's computed
// columns — and every branch pre-sorts and pre-limits, so Postgres
// reads O(limit) rows per branch instead of rescanning and
// top-N-sorting the entire folder on every page (19.5x on a
// 20k-entry folder, benches/LISTING-KEYSET.md). The "name" sort is
// served by the expression indexes idx_files_folder_lname /
// idx_folders_parent_lname (migration 20260918000000).
//
// Sort-key columns that are CONSTANT within a branch (folder_first,
// the folder branch's type_order = 0 and size = -1) are folded in
// Rust: depending on which group the cursor points into, the branch
// predicate shortens to a row-value over the remaining keys, the
// branch keeps all its rows, or the branch drops out entirely.
enum BranchCursor {
/// The cursor has moved past every row this branch can produce.
Drop,
/// Every row in this branch sorts after the cursor.
All,
/// Row-value comparison over the branch's non-constant sort keys.
Pred(String),
}
use BranchCursor::{All, Drop, Pred};
let has_cursor = cursor.is_some();
// (folder-branch cursor, file-branch cursor, per-branch ORDER BY on
// the branch's output aliases, outer merge ORDER BY)
let (folder_cur, file_cur, branch_order, outer_order) = match order_by {
"type" => {
if reverse {
(
r#"WHERE ($3::bigint IS NULL)
OR (type_order < $3)
OR (type_order = $3 AND sort_str < $2)
OR (type_order = $3 AND sort_str = $2 AND id < $5::uuid)"#,
"ORDER BY type_order DESC, sort_str DESC, id DESC",
)
let (op, ord) = if reverse {
("<", "ORDER BY type_order DESC, sort_str DESC, id DESC")
} else {
(
r#"WHERE ($3::bigint IS NULL)
OR (type_order > $3)
OR (type_order = $3 AND sort_str > $2)
OR (type_order = $3 AND sort_str = $2 AND id > $5::uuid)"#,
"ORDER BY type_order ASC, sort_str ASC, id ASC",
)
}
(">", "ORDER BY type_order ASC, sort_str ASC, id ASC")
};
let folder_cur = match cursor_int {
None => All,
// Folder rows have type_order = 0; a cursor sitting on a
// file (type_order > 0) either exhausts the folder group
// (ASC) or precedes all of it (DESC).
Some(c_to) if c_to > 0 => {
if reverse {
All
} else {
Drop
}
}
Some(_) => Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
};
let file_cur = if has_cursor {
Pred(format!(
"(fm.category_order::bigint, LOWER(fm.name), fm.id) {op} ($3, $2, $5::uuid)"
))
} else {
All
};
(folder_cur, file_cur, ord, ord)
}
"modified_at" => {
if reverse {
(
r#"WHERE ($4::timestamptz IS NULL)
OR (modified_at > $4)
OR (modified_at = $4 AND id > $5::uuid)"#,
"ORDER BY modified_at ASC, id ASC",
)
let (op, ord) = if reverse {
(">", "ORDER BY modified_at ASC, id ASC")
} else {
(
r#"WHERE ($4::timestamptz IS NULL)
OR (modified_at < $4)
OR (modified_at = $4 AND id < $5::uuid)"#,
"ORDER BY modified_at DESC, id DESC",
)
}
("<", "ORDER BY modified_at DESC, id DESC")
};
let mk = |col: &str| {
if has_cursor {
Pred(format!("({col}.updated_at, {col}.id) {op} ($4, $5::uuid)"))
} else {
All
}
};
(mk("f"), mk("fm"), ord, ord)
}
"created_at" => {
if reverse {
(
r#"WHERE ($4::timestamptz IS NULL)
OR (created_at > $4)
OR (created_at = $4 AND id > $5::uuid)"#,
"ORDER BY created_at ASC, id ASC",
)
let (op, ord) = if reverse {
(">", "ORDER BY created_at ASC, id ASC")
} else {
(
r#"WHERE ($4::timestamptz IS NULL)
OR (created_at < $4)
OR (created_at = $4 AND id < $5::uuid)"#,
"ORDER BY created_at DESC, id DESC",
)
}
("<", "ORDER BY created_at DESC, id DESC")
};
let mk = |col: &str| {
if has_cursor {
Pred(format!("({col}.created_at, {col}.id) {op} ($4, $5::uuid)"))
} else {
All
}
};
(mk("f"), mk("fm"), ord, ord)
}
"size" => {
if reverse {
(
r#"WHERE ($3::bigint IS NULL)
OR (size < $3)
OR (size = $3 AND id < $5::uuid)"#,
"ORDER BY size DESC, id DESC",
)
let (op, ord) = if reverse {
("<", "ORDER BY size DESC, id DESC")
} else {
(
r#"WHERE ($3::bigint IS NULL)
OR (size > $3)
OR (size = $3 AND id > $5::uuid)"#,
"ORDER BY size ASC, id ASC",
)
}
(">", "ORDER BY size ASC, id ASC")
};
let folder_cur = match cursor_int {
None => All,
// Folder rows have size = -1; a cursor sitting on a file
// (size >= 0) exhausts the folder group (ASC) or precedes
// all of it (DESC).
Some(c_sz) if c_sz > -1 => {
if reverse {
All
} else {
Drop
}
}
Some(_) => Pred(format!("f.id {op} $5::uuid")),
};
let file_cur = if has_cursor {
Pred(format!("(fm.size::bigint, fm.id) {op} ($3, $5::uuid)"))
} else {
All
};
(folder_cur, file_cur, ord, ord)
}
_ => {
// "name" (default): folder_first stays ASC so folders always precede
// files; only the alpha order within each group flips when reversed.
if reverse {
(
r#"WHERE ($3::bigint IS NULL)
OR (folder_first::bigint > $3)
OR (folder_first::bigint = $3 AND sort_str < $2)
OR (folder_first::bigint = $3 AND sort_str = $2 AND id < $5::uuid)"#,
"ORDER BY folder_first ASC, sort_str DESC, id DESC",
)
// "name" (default): folder_first stays ASC so folders always
// precede files; only the alpha order within each group flips
// when reversed. cursor_int carries folder_first (0|1).
let op = if reverse { "<" } else { ">" };
let branch_ord = if reverse {
"ORDER BY sort_str DESC, id DESC"
} else {
(
r#"WHERE ($3::bigint IS NULL)
OR (folder_first::bigint > $3)
OR (folder_first::bigint = $3 AND sort_str > $2)
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid)"#,
"ORDER BY folder_first ASC, sort_str ASC, id ASC",
)
}
"ORDER BY sort_str ASC, id ASC"
};
let outer_ord = if reverse {
"ORDER BY folder_first ASC, sort_str DESC, id DESC"
} else {
"ORDER BY folder_first ASC, sort_str ASC, id ASC"
};
let (folder_cur, file_cur) = match cursor_int {
None => (All, All),
// Cursor inside the folder group: folders continue after
// the row-value cursor; every file still follows.
Some(0) => (
Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
All,
),
// Cursor inside the file group: the folder group is done.
Some(_) => (
Drop,
Pred(format!("(LOWER(fm.name), fm.id) {op} ($2, $5::uuid)")),
),
};
(folder_cur, file_cur, branch_ord, outer_ord)
}
};
let wrap = |branch: &str, cur: &BranchCursor| -> Option<String> {
let extra = match cur {
Drop => return None,
All => String::new(),
Pred(p) => format!(" AND {p}"),
};
Some(format!(
"(SELECT * FROM ({branch}{extra}) b {branch_order} LIMIT $6)"
))
};
let mut branches = Vec::with_capacity(2);
if include_folders && let Some(b) = wrap(folder_branch, &folder_cur) {
branches.push(b);
}
if include_files && let Some(b) = wrap(file_branch, &file_cur) {
branches.push(b);
}
// Every requested branch dropped out (e.g. folders-only listing with
// the cursor already past the folder group).
if branches.is_empty() {
return Ok(Vec::new());
}
let inner = branches.join(" UNION ALL ");
let sql = format!(
"WITH resources AS ({cte_inner}) \
SELECT resource_type, id, name, folder_id, mime_type, size, \
"SELECT resource_type, id, name, folder_id, mime_type, size, \
created_at, modified_at, drive_id, blob_hash, \
sort_str, type_order, folder_first \
FROM resources \
{where_clause} \
{order_clause} \
FROM ({inner}) r \
{outer_order} \
LIMIT $6"
);
@@ -130,7 +130,9 @@ impl BlobStorageBackend for AzureBlobBackend {
return Ok(size);
}
client.put_block_blob(data.to_vec()).await.map_err(|e| {
// `Bytes` converts into `azure_core::Body` by reference count —
// the old `data.to_vec()` copied every chunk once more.
client.put_block_blob(data).await.map_err(|e| {
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
})?;
@@ -138,6 +140,26 @@ impl BlobStorageBackend for AzureBlobBackend {
})
}
/// Dedup settle path: PUT unconditionally. Content-addressed keys make
/// re-PUTs idempotent, so the `get_properties` probe
/// `put_blob_from_bytes` pays is a pure extra round-trip on every NEW
/// chunk (2 RTTs -> 1, benches/S3-PUT.md — same shape as S3).
fn put_blob_from_bytes_unsynced(
&self,
hash: &str,
data: Bytes,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
let size = data.len() as u64;
client.put_block_blob(data).await.map_err(|e| {
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
})?;
Ok(size)
})
}
fn get_blob_stream(
&self,
hash: &str,
@@ -13,12 +13,14 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use bytes::Bytes;
use dashmap::DashMap;
use lru::LruCache;
use std::num::NonZeroUsize;
use tokio::fs;
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
use tokio::sync::Mutex;
use tokio_util::io::ReaderStream;
use uuid::Uuid;
use crate::application::ports::blob_storage_ports::{
BlobStorageBackend, BlobStream, StorageHealthStatus,
@@ -56,6 +58,13 @@ pub struct CachedBlobBackend {
max_cache_bytes: u64,
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
current_size: Arc<AtomicU64>,
/// Per-hash single-flight gates for cache misses. K concurrent cold
/// readers of one blob (e.g. a video player's parallel Range probes)
/// used to each download the FULL blob from the remote backend — and
/// race their writes on one shared `.tmp` path. The gate coalesces
/// them onto one fetch; waiters re-check the cache and serve locally
/// (16 fetches -> 1, benches/BLOB-CACHE.md).
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
}
impl CachedBlobBackend {
@@ -70,6 +79,7 @@ impl CachedBlobBackend {
NonZeroUsize::new(1_000_000).unwrap(),
))),
current_size: Arc::new(AtomicU64::new(0)),
inflight: Arc::new(DashMap::new()),
}
}
@@ -150,6 +160,7 @@ impl BlobStorageBackend for CachedBlobBackend {
max_cache_bytes: self.max_cache_bytes,
index: self.index.clone(),
current_size: self.current_size.clone(),
inflight: self.inflight.clone(),
};
Box::pin(async move {
// Write to inner backend
@@ -172,6 +183,7 @@ impl BlobStorageBackend for CachedBlobBackend {
max_cache_bytes: self.max_cache_bytes,
index: self.index.clone(),
current_size: self.current_size.clone(),
inflight: self.inflight.clone(),
};
Box::pin(async move {
let size = inner.put_blob_from_bytes(&hash, data.clone()).await?;
@@ -203,6 +215,7 @@ impl BlobStorageBackend for CachedBlobBackend {
let cache_dir = self.cache_dir.clone();
let max_cache_bytes = self.max_cache_bytes;
let current_size = self.current_size.clone();
let inflight = self.inflight.clone();
Box::pin(async move {
// Check cache presence (and bump LRU recency) under a brief lock,
// then release it BEFORE touching the filesystem so concurrent
@@ -219,14 +232,17 @@ impl BlobStorageBackend for CachedBlobBackend {
}
}
// Cache miss — fetch from inner, spool to cache
// Cache miss — fetch from inner (single-flight), spool to cache
let self_ref = CachedRef {
cache_dir,
max_cache_bytes,
index: index.clone(),
current_size: current_size.clone(),
inflight,
};
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
let dest = self_ref
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
.await?;
let file = fs::File::open(&dest).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
})?;
@@ -249,6 +265,7 @@ impl BlobStorageBackend for CachedBlobBackend {
let cache_dir = self.cache_dir.clone();
let max_cache_bytes = self.max_cache_bytes;
let current_size = self.current_size.clone();
let inflight = self.inflight.clone();
Box::pin(async move {
// Check cache presence (and bump LRU recency) under a brief lock,
// then release it BEFORE the open()/seek() syscalls so concurrent
@@ -271,14 +288,19 @@ impl BlobStorageBackend for CachedBlobBackend {
}
}
// Cache miss — fetch full blob into cache, then serve range
// Cache miss — fetch full blob into cache (single-flight: a
// player's parallel cold Range probes coalesce onto ONE remote
// download), then serve the range locally.
let self_ref = CachedRef {
cache_dir,
max_cache_bytes,
index: index.clone(),
current_size: current_size.clone(),
inflight,
};
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
let dest = self_ref
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
.await?;
let mut file = fs::File::open(&dest)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
@@ -406,6 +428,7 @@ struct CachedRef {
max_cache_bytes: u64,
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
current_size: Arc<AtomicU64>,
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
}
impl CachedRef {
@@ -414,6 +437,37 @@ impl CachedRef {
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
}
/// Single-flight wrapper around [`Self::fetch_and_cache_static`]: the
/// first caller for a hash becomes the leader and downloads; concurrent
/// callers queue on the per-hash gate, then re-check the cache and serve
/// the leader's file without touching the remote backend. Errors are not
/// cached — the gate entry is dropped, so the next caller retries.
async fn fetch_and_cache_singleflight(
&self,
hash: &str,
inner: &dyn BlobStorageBackend,
cached: &Path,
) -> Result<PathBuf, DomainError> {
let gate = self
.inflight
.entry(hash.to_string())
.or_insert_with(|| Arc::new(Mutex::new(())))
.clone();
let _guard = gate.lock().await;
// Re-check under the gate: if we queued behind the leader, the blob
// is on disk now and this turns into a local open.
if self.index.lock().await.get(hash).is_some() && fs::metadata(cached).await.is_ok() {
return Ok(cached.to_path_buf());
}
let result = self.fetch_and_cache_static(hash, inner).await;
// Drop the gate whether we succeeded or failed; a late-arriving
// caller after an error creates a fresh gate and retries the fetch.
self.inflight.remove(hash);
result
}
/// Pop LRU entries until the cache is back within its byte budget,
/// returning the on-disk paths of the evicted blobs.
///
@@ -486,31 +540,51 @@ impl CachedRef {
})?;
}
let tmp = dest.with_extension("tmp");
let mut file = fs::File::create(&tmp)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
use futures::StreamExt;
let mut stream = stream;
let mut total = 0u64;
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| {
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
// Unique temp name: even if two fetches for one hash ever race
// (e.g. across processes sharing a cache dir), each writes its own
// inode and the rename is atomic — a torn/interleaved file can
// never land at the final path.
let tmp = dest.with_extension(format!("{}.tmp", Uuid::new_v4()));
let write_result: Result<u64, DomainError> = async {
let mut file = fs::File::create(&tmp).await.map_err(|e| {
DomainError::internal_error("BlobCache", format!("create tmp: {e}"))
})?;
total += bytes.len() as u64;
file.write_all(&bytes)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
}
file.flush()
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
drop(file);
fs::rename(&tmp, &dest)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
use futures::StreamExt;
let mut stream = stream;
let mut total = 0u64;
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| {
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
})?;
total += bytes.len() as u64;
file.write_all(&bytes)
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
}
file.flush()
.await
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
Ok(total)
}
.await;
let total = match write_result {
Ok(total) => total,
Err(e) => {
// Unique tmp names never get overwritten by a later fetch —
// reap the partial file instead of leaking it.
let _ = fs::remove_file(&tmp).await;
return Err(e);
}
};
if let Err(e) = fs::rename(&tmp, &dest).await {
let _ = fs::remove_file(&tmp).await;
return Err(DomainError::internal_error(
"BlobCache",
format!("rename: {e}"),
));
}
let to_evict = {
let mut idx = self.index.lock().await;
@@ -200,6 +200,38 @@ impl BlobStorageBackend for S3BlobBackend {
})
}
/// Dedup settle path: PUT unconditionally. Keys are content-addressed
/// (BLAKE3), so a re-PUT writes identical bytes — overwrite-safe
/// idempotency without the HEAD probe `put_blob_from_bytes` pays. The
/// dedup layer already filtered out chunks the database knows about,
/// so the probe was a pure extra round-trip on every NEW chunk of
/// every upload (2 RTTs -> 1, benches/S3-PUT.md).
fn put_blob_from_bytes_unsynced(
&self,
hash: &str,
data: Bytes,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let key = Self::object_key(&hash);
let size = data.len() as u64;
self.client
.put_object()
.bucket(&self.bucket)
.key(&key)
.body(ByteStream::from(data))
.send()
.await
.map_err(|e| {
DomainError::internal_error(
"S3",
format!("Failed to upload blob {}: {}", hash, e),
)
})?;
Ok(size)
})
}
fn get_blob_stream(
&self,
hash: &str,
+2 -15
View File
@@ -385,7 +385,6 @@ async fn handle_propfind(
CardDavAdapter::generate_contacts_response(
&mut response_body,
std::slice::from_ref(&contact),
&[(contact.uid.clone(), contact_to_vcard(&contact))],
&report,
base_href,
)
@@ -449,22 +448,10 @@ async fn handle_report(
.map_err(AppError::from)?,
};
// Generate vCards
let vcards: Vec<(String, String)> = contacts
.iter()
.map(|c| (c.uid.clone(), contact_to_vcard(c)))
.collect();
let base_href = &format!("/carddav/{}/", address_book_id);
let mut response_body = Vec::new();
CardDavAdapter::generate_contacts_response(
&mut response_body,
&contacts,
&vcards,
&report,
base_href,
)
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
CardDavAdapter::generate_contacts_response(&mut response_body, &contacts, &report, base_href)
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
Ok(Response::builder()
.status(StatusCode::MULTI_STATUS)
+18 -18
View File
@@ -781,25 +781,25 @@ async fn build_streaming_propfind_response(
// ── Children (only if Depth == 1) ────────────────────────
if depth == "1" {
let pagination = crate::application::dtos::pagination::PaginationRequestDto {
page: 0,
page_size: PROPFIND_BATCH_SIZE as usize,
};
let fid_ref = folder_id.as_deref();
// Stream sub-folders in pages (user-scoped)
let mut page = 0usize;
// Stream sub-folders in pages (user-scoped, keyset cursor —
// O(page) per page off idx_folders_unique_name instead of the
// quadratic COUNT(*) OVER() + LIMIT/OFFSET walk; 4.5x on a
// 5k-dir parent, benches/FOLDER-KEYSET.md).
let mut after_folder: Option<String> = None;
loop {
let pag = crate::application::dtos::pagination::PaginationRequestDto {
page,
page_size: pagination.page_size,
};
let result = folder_service
.list_folders_paginated_with_perms(fid_ref, user_id, &pag)
let batch = folder_service
.list_folders_batch_with_perms(
fid_ref,
user_id,
after_folder.as_deref(),
PROPFIND_BATCH_SIZE as usize,
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
if result.items.is_empty() {
if batch.is_empty() {
break;
}
@@ -808,25 +808,25 @@ async fn build_streaming_propfind_response(
// 1-4.5 s of pure DB chatter on a 2000-child folder
// (measured in benches/DEAD-PROPS.md).
let subfolder_deads =
folders_dead_props_map(&dead_props_store, &result.items).await;
folders_dead_props_map(&dead_props_store, &batch).await;
let mut chunk = Vec::with_capacity(result.items.len() * 800);
let mut chunk = Vec::with_capacity(batch.len() * 800);
{
let mut w = Writer::new(&mut chunk);
for subfolder in result.items.iter() {
for subfolder in batch.iter() {
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
.map_err(|e| std::io::Error::other(e.to_string()))?;
}
}
let has_more = result.pagination.has_next;
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
after_folder = batch.last().map(|f| f.name.clone());
yield Bytes::from(chunk);
if !has_more {
break;
}
page += 1;
}
// Stream files in pages (user-scoped, keyset cursor — O(page)
+20 -17
View File
@@ -16,7 +16,6 @@ use uuid::Uuid;
use crate::application::adapters::webdav_adapter::{
PropFindRequest, PropPatchOp, QualifiedName, WebDavAdapter, is_protected_property,
};
use crate::application::dtos::pagination::PaginationRequestDto;
use crate::application::ports::authorization_ports::AuthorizationEngine;
use crate::application::ports::favorites_ports::FavoritesUseCase;
use crate::application::ports::file_ports::{
@@ -1584,38 +1583,42 @@ fn build_nc_streaming_propfind(
after_name = batch.last().map(|f| f.name.clone());
}
// Subfolders in pages — also collections, same trailing-slash rule.
let mut page = 0usize;
// Subfolders in pages — also collections, same trailing-slash
// rule. Keyset cursor: O(page) per page off
// idx_folders_unique_name instead of the quadratic
// COUNT(*) OVER() + LIMIT/OFFSET walk (benches/FOLDER-KEYSET.md).
let mut after_folder: Option<String> = None;
loop {
let pag = PaginationRequestDto {
page,
page_size: PROPFIND_BATCH_SIZE as usize,
};
let result = folder_service
.list_folders_paginated_with_perms(Some(&folder.id), user_id, &pag)
let batch = folder_service
.list_folders_batch_with_perms(
Some(&folder.id),
user_id,
after_folder.as_deref(),
PROPFIND_BATCH_SIZE as usize,
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
if result.items.is_empty() {
if batch.is_empty() {
break;
}
let favs = if let Some(fav) = fav_svc {
let items: Vec<(&str, &str)> =
result.items.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
batch.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
fav.batch_check_favorites(user_id, &items).await.unwrap_or_default()
} else {
HashSet::new()
};
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
let folder_uuids: Vec<String> = batch.iter().map(|sf| sf.id.clone()).collect();
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
// Batched — see benches/DEAD-PROPS.md.
let sub_deads =
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
folders_dead_props_map(&state.webdav_dead_props, &batch).await;
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
let mut chunk = Vec::with_capacity(batch.len() * 1024);
{
let mut xml = Writer::new(&mut chunk);
for sf in result.items.iter() {
for sf in batch.iter() {
let dead = dead_props_for(&sf.id, &sub_deads);
let child_sub = if subpath.is_empty() {
sf.name.clone()
@@ -1629,13 +1632,13 @@ fn build_nc_streaming_propfind(
.map_err(std::io::Error::other)?;
}
}
let has_more = result.pagination.has_next;
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
after_folder = batch.last().map(|sf| sf.name.clone());
yield Bytes::from(chunk);
if !has_more {
break;
}
page += 1;
}
}
+13 -2
View File
@@ -273,11 +273,16 @@ pub fn multipart_field_stream(
pub fn stream_from_files(
paths: Vec<PathBuf>,
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
// 512 KiB per poll: each ReaderStream poll on a tokio::fs::File is one
// blocking-pool dispatch + one read(2) of the buffer size. The old
// 64 KiB buffer paid 8x the dispatches/syscalls of every other blob
// read path (STREAM_CHUNK_SIZE = 256 KiB) for the single read pass
// over every completed chunked upload (benches/UPLOAD-SPOOL.md).
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
.and_then(|path| async move {
tokio::fs::File::open(path)
.await
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
.map(|file| ReaderStream::with_capacity(file, 512 * 1024))
})
.try_flatten()
}
@@ -319,9 +324,15 @@ pub async fn stream_body_to_path(
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
// BufWriter coalesces the per-HTTP-frame writes (~16-64 KiB each) into
// 512 KiB write(2)s — a bare tokio File dispatches one blocking-pool op
// per frame (benches/UPLOAD-SPOOL.md). Same capacity as the dedup
// handler's spool loop. On the error paths below the partial file is
// removed, so silently dropping unflushed buffer contents is fine.
let file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut file = tokio::io::BufWriter::with_capacity(512 * 1024, file);
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);