perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
@@ -651,7 +651,6 @@ impl CardDavAdapter {
|
||||
pub fn generate_contacts_response<W: Write>(
|
||||
writer: W,
|
||||
contacts: &[ContactDto],
|
||||
vcards: &[(String, String)], // (uid, vcard_data)
|
||||
report: &CardDavReportType,
|
||||
base_href: &str,
|
||||
) -> Result<()> {
|
||||
@@ -672,17 +671,9 @@ impl CardDavAdapter {
|
||||
|
||||
for contact in contacts {
|
||||
let href = format!("{}{}.vcf", base_href, contact.uid);
|
||||
let vcard = vcards
|
||||
.iter()
|
||||
.find(|(uid, _)| *uid == contact.uid)
|
||||
.map(|(_, data)| data.as_str())
|
||||
.unwrap_or("");
|
||||
// `write_contact_response` generates the vCard on demand when (and
|
||||
// only when) address-data is actually requested.
|
||||
Self::write_contact_response(&mut xml_writer, contact, &props, &href)?;
|
||||
// If address-data is requested, include vcard
|
||||
if props.iter().any(|p| p.name == "address-data") || props.is_empty() {
|
||||
// Already handled in write_contact_response
|
||||
}
|
||||
let _ = vcard; // suppress warning - used via contact_to_vcard fallback
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:multistatus")))?;
|
||||
@@ -868,20 +859,25 @@ impl CardDavAdapter {
|
||||
|
||||
/// Convert a ContactDto to vCard 3.0 format
|
||||
pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
// `write!` into a String is infallible; `let _ =` discards the Ok(()).
|
||||
// Formatting straight into the buffer avoids one temporary String per
|
||||
// vCard line compared to `push_str(&format!(…))`.
|
||||
use std::fmt::Write as _;
|
||||
|
||||
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
|
||||
|
||||
vcard.push_str(&format!("UID:{}\r\n", contact.uid));
|
||||
let _ = write!(vcard, "UID:{}\r\n", contact.uid);
|
||||
|
||||
if let (Some(last), Some(first)) = (&contact.last_name, &contact.first_name) {
|
||||
vcard.push_str(&format!("N:{};{};;;\r\n", last, first));
|
||||
let _ = write!(vcard, "N:{};{};;;\r\n", last, first);
|
||||
} else if let Some(last) = &contact.last_name {
|
||||
vcard.push_str(&format!("N:{};;;;\r\n", last));
|
||||
let _ = write!(vcard, "N:{};;;;\r\n", last);
|
||||
} else if let Some(first) = &contact.first_name {
|
||||
vcard.push_str(&format!("N:;{};;;\r\n", first));
|
||||
let _ = write!(vcard, "N:;{};;;\r\n", first);
|
||||
}
|
||||
|
||||
if let Some(fn_name) = &contact.full_name {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
let _ = write!(vcard, "FN:{}\r\n", fn_name);
|
||||
} else {
|
||||
// FN is mandatory in vCard 3.0
|
||||
let fn_name = format!(
|
||||
@@ -892,68 +888,68 @@ pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
.trim()
|
||||
.to_string();
|
||||
if !fn_name.is_empty() {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
let _ = write!(vcard, "FN:{}\r\n", fn_name);
|
||||
} else {
|
||||
vcard.push_str("FN:Unknown\r\n");
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(nickname) = &contact.nickname {
|
||||
vcard.push_str(&format!("NICKNAME:{}\r\n", nickname));
|
||||
let _ = write!(vcard, "NICKNAME:{}\r\n", nickname);
|
||||
}
|
||||
|
||||
for email in &contact.email {
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"EMAIL;TYPE={}:{}\r\n",
|
||||
email.r#type.to_uppercase(),
|
||||
email.email
|
||||
));
|
||||
);
|
||||
}
|
||||
|
||||
for phone in &contact.phone {
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"TEL;TYPE={}:{}\r\n",
|
||||
phone.r#type.to_uppercase(),
|
||||
phone.number
|
||||
));
|
||||
);
|
||||
}
|
||||
|
||||
for addr in &contact.address {
|
||||
let adr = format!(
|
||||
";;{};{};{};{};{}",
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"ADR;TYPE={}:;;{};{};{};{};{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
addr.street.as_deref().unwrap_or(""),
|
||||
addr.city.as_deref().unwrap_or(""),
|
||||
addr.state.as_deref().unwrap_or(""),
|
||||
addr.postal_code.as_deref().unwrap_or(""),
|
||||
addr.country.as_deref().unwrap_or(""),
|
||||
);
|
||||
vcard.push_str(&format!(
|
||||
"ADR;TYPE={}:{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
adr
|
||||
));
|
||||
}
|
||||
|
||||
if let Some(org) = &contact.organization {
|
||||
vcard.push_str(&format!("ORG:{}\r\n", org));
|
||||
let _ = write!(vcard, "ORG:{}\r\n", org);
|
||||
}
|
||||
if let Some(title) = &contact.title {
|
||||
vcard.push_str(&format!("TITLE:{}\r\n", title));
|
||||
let _ = write!(vcard, "TITLE:{}\r\n", title);
|
||||
}
|
||||
if let Some(notes) = &contact.notes {
|
||||
vcard.push_str(&format!("NOTE:{}\r\n", notes.replace('\n', "\\n")));
|
||||
let _ = write!(vcard, "NOTE:{}\r\n", notes.replace('\n', "\\n"));
|
||||
}
|
||||
if let Some(bday) = &contact.birthday {
|
||||
vcard.push_str(&format!("BDAY:{}\r\n", bday.format("%Y-%m-%d")));
|
||||
let _ = write!(vcard, "BDAY:{}\r\n", bday.format("%Y-%m-%d"));
|
||||
}
|
||||
if let Some(photo) = &contact.photo_url {
|
||||
vcard.push_str(&format!("PHOTO;VALUE=URI:{}\r\n", photo));
|
||||
let _ = write!(vcard, "PHOTO;VALUE=URI:{}\r\n", photo);
|
||||
}
|
||||
|
||||
vcard.push_str(&format!(
|
||||
let _ = write!(
|
||||
vcard,
|
||||
"REV:{}\r\n",
|
||||
contact.updated_at.format("%Y%m%dT%H%M%SZ")
|
||||
));
|
||||
);
|
||||
vcard.push_str("END:VCARD\r\n");
|
||||
|
||||
vcard
|
||||
|
||||
@@ -484,10 +484,6 @@ mod tests {
|
||||
#[test]
|
||||
fn test_generate_contacts_response() {
|
||||
let contacts = vec![sample_contact()];
|
||||
let vcards = vec![(
|
||||
"contact-001".to_string(),
|
||||
contact_to_vcard(&sample_contact()),
|
||||
)];
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: vec![
|
||||
QualifiedName {
|
||||
@@ -505,7 +501,6 @@ mod tests {
|
||||
let result = CardDavAdapter::generate_contacts_response(
|
||||
&mut output,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
"/carddav/ab-001",
|
||||
);
|
||||
@@ -528,14 +523,12 @@ mod tests {
|
||||
#[test]
|
||||
fn test_generate_empty_contacts_response() {
|
||||
let contacts: Vec<ContactDto> = vec![];
|
||||
let vcards: Vec<(String, String)> = vec![];
|
||||
let report = CardDavReportType::AddressbookQuery { props: vec![] };
|
||||
|
||||
let mut output = Vec::new();
|
||||
let result = CardDavAdapter::generate_contacts_response(
|
||||
&mut output,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
"/carddav/ab-001",
|
||||
);
|
||||
|
||||
@@ -8,6 +8,175 @@
|
||||
//! then fall back to the file extension when the MIME is generic
|
||||
//! (`application/octet-stream` or empty).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fmt::Write as _;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
// ─── Arc<str> interning for closed-set display values ────────────────
|
||||
//
|
||||
// `FileDto` / `FolderDto` store their display fields as `Arc<str>` so DTO
|
||||
// clones are O(1). But `Arc::<str>::from(&str)` always allocates + copies,
|
||||
// so building the DTO paid 3-4 heap allocations per row even though the
|
||||
// value space is a small closed set. Interning turns each conversion into
|
||||
// a HashMap lookup + refcount bump.
|
||||
|
||||
/// Every `&'static str` that [`icon_class_for`], [`icon_special_class_for`]
|
||||
/// and [`category_for`] can return, plus the folder-DTO constants.
|
||||
///
|
||||
/// Keep this table in sync when adding a value to those functions — a
|
||||
/// missing entry is not a bug (callers fall back to `Arc::from`, same
|
||||
/// bytes, one extra allocation), just a lost optimization.
|
||||
static DISPLAY_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const CLOSED_SET: &[&str] = &[
|
||||
// icon_class_for
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file-word",
|
||||
"fas fa-file-excel",
|
||||
"fas fa-file-powerpoint",
|
||||
"fas fa-file-archive",
|
||||
"fas fa-file-code",
|
||||
"fas fa-hdd",
|
||||
"fas fa-file-image",
|
||||
"fas fa-file-video",
|
||||
"fas fa-file-audio",
|
||||
"fas fa-file-alt",
|
||||
"fas fa-terminal",
|
||||
"fas fa-file",
|
||||
// icon_special_class_for
|
||||
"pdf-icon",
|
||||
"doc-icon",
|
||||
"spreadsheet-icon",
|
||||
"presentation-icon",
|
||||
"archive-icon",
|
||||
"code-icon json-icon",
|
||||
"code-icon js-icon",
|
||||
"code-icon ts-icon",
|
||||
"code-icon html-icon",
|
||||
"code-icon sql-icon",
|
||||
"code-icon config-icon",
|
||||
"code-icon php-icon",
|
||||
"script-icon",
|
||||
"installer-icon",
|
||||
"image-icon",
|
||||
"video-icon",
|
||||
"audio-icon",
|
||||
"code-icon py-icon",
|
||||
"code-icon rust-icon",
|
||||
"code-icon",
|
||||
"code-icon go-icon",
|
||||
"code-icon ruby-icon",
|
||||
"code-icon md-icon",
|
||||
"code-icon css-icon",
|
||||
"code-icon java-icon",
|
||||
"code-icon c-icon",
|
||||
"code-icon cs-icon",
|
||||
"code-icon swift-icon",
|
||||
"",
|
||||
// category_for
|
||||
"PDF",
|
||||
"Document",
|
||||
"Spreadsheet",
|
||||
"Presentation",
|
||||
"Archive",
|
||||
"Code",
|
||||
"Installer",
|
||||
"Image",
|
||||
"Video",
|
||||
"Audio",
|
||||
"Markdown",
|
||||
"Text",
|
||||
// FolderDto constants
|
||||
"fas fa-folder",
|
||||
"folder-icon",
|
||||
"Folder",
|
||||
];
|
||||
CLOSED_SET.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for a display value from the closed sets
|
||||
/// above (icon class, icon special class, category). Lookup + refcount
|
||||
/// bump instead of alloc + copy; unknown values (future additions not
|
||||
/// yet in the table) fall back to `Arc::from` with identical bytes.
|
||||
pub fn intern_display(s: &'static str) -> Arc<str> {
|
||||
DISPLAY_INTERN
|
||||
.get(s)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(s))
|
||||
}
|
||||
|
||||
/// The MIME types that dominate real storage rows. Exotic types fall back
|
||||
/// to a per-row `Arc::from` — correctness is unaffected, only the alloc is.
|
||||
static MIME_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const COMMON_MIMES: &[&str] = &[
|
||||
"",
|
||||
"directory",
|
||||
"application/octet-stream",
|
||||
// Images
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/gif",
|
||||
"image/webp",
|
||||
"image/svg+xml",
|
||||
"image/heic",
|
||||
"image/heif",
|
||||
"image/avif",
|
||||
"image/bmp",
|
||||
"image/tiff",
|
||||
"image/x-icon",
|
||||
// Video
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"video/webm",
|
||||
"video/x-matroska",
|
||||
"video/x-msvideo",
|
||||
// Audio
|
||||
"audio/mpeg",
|
||||
"audio/mp4",
|
||||
"audio/ogg",
|
||||
"audio/flac",
|
||||
"audio/wav",
|
||||
"audio/x-wav",
|
||||
"audio/aac",
|
||||
// Documents
|
||||
"application/pdf",
|
||||
"application/msword",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-powerpoint",
|
||||
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
"application/vnd.oasis.opendocument.text",
|
||||
"application/vnd.oasis.opendocument.spreadsheet",
|
||||
// Text / code
|
||||
"text/plain",
|
||||
"text/csv",
|
||||
"text/html",
|
||||
"text/css",
|
||||
"text/markdown",
|
||||
"text/xml",
|
||||
"application/json",
|
||||
"application/javascript",
|
||||
"application/xml",
|
||||
"application/x-yaml",
|
||||
// Archives
|
||||
"application/zip",
|
||||
"application/gzip",
|
||||
"application/x-tar",
|
||||
"application/x-7z-compressed",
|
||||
"application/x-rar-compressed",
|
||||
];
|
||||
COMMON_MIMES.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for the given MIME type. Common types hit
|
||||
/// the intern table (refcount bump); exotic ones allocate as before.
|
||||
pub fn intern_mime(mime: &str) -> Arc<str> {
|
||||
MIME_INTERN
|
||||
.get(mime)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(mime))
|
||||
}
|
||||
|
||||
// ─── Private: extract lowercase extension from a filename ────────────
|
||||
fn ext_of(name: &str) -> Option<&str> {
|
||||
let name = name.rsplit('/').next().unwrap_or(name); // strip path
|
||||
@@ -388,11 +557,21 @@ pub fn format_file_size(bytes: u64) -> String {
|
||||
|
||||
let value = bytes as f64 / K.powi(i as i32);
|
||||
|
||||
// Two decimal places, then strip trailing zeros (matches JS parseFloat behaviour)
|
||||
let formatted = format!("{:.2}", value);
|
||||
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
|
||||
|
||||
format!("{} {}", formatted, SIZES[i])
|
||||
// Single buffer: write the 2-decimal value, strip trailing zeros in
|
||||
// place (matches JS parseFloat behaviour), then append the unit.
|
||||
// 16 chars covers the worst case ("16777216 TB" for u64::MAX,
|
||||
// "1023.99 Bytes" for the longest unit), so no realloc occurs.
|
||||
let mut out = String::with_capacity(16);
|
||||
let _ = write!(out, "{:.2}", value);
|
||||
while out.ends_with('0') {
|
||||
out.pop();
|
||||
}
|
||||
if out.ends_with('.') {
|
||||
out.pop();
|
||||
}
|
||||
out.push(' ');
|
||||
out.push_str(SIZES[i]);
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -506,6 +685,50 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Every value the closed-set display functions can return must hit
|
||||
/// the intern table (same bytes, shared allocation) — a miss is only
|
||||
/// a lost optimization, but this test keeps the table in sync.
|
||||
#[test]
|
||||
fn test_intern_display_covers_closed_sets_and_shares_storage() {
|
||||
for s in [
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file",
|
||||
"fas fa-terminal",
|
||||
"fas fa-folder",
|
||||
"code-icon rust-icon",
|
||||
"folder-icon",
|
||||
"",
|
||||
"PDF",
|
||||
"Folder",
|
||||
"Document",
|
||||
"Markdown",
|
||||
] {
|
||||
let a = intern_display(s);
|
||||
let b = intern_display(s);
|
||||
assert_eq!(&*a, s, "interned bytes must be identical");
|
||||
assert!(
|
||||
Arc::ptr_eq(&a, &b),
|
||||
"closed-set value {s:?} must come from the intern table"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_intern_mime_common_hits_table_exotic_falls_back() {
|
||||
let a = intern_mime("image/jpeg");
|
||||
let b = intern_mime("image/jpeg");
|
||||
assert_eq!(&*a, "image/jpeg");
|
||||
assert!(Arc::ptr_eq(&a, &b), "common MIME must be interned");
|
||||
|
||||
let exotic = intern_mime("chemical/x-pdb");
|
||||
assert_eq!(&*exotic, "chemical/x-pdb");
|
||||
let exotic2 = intern_mime("chemical/x-pdb");
|
||||
assert!(
|
||||
!Arc::ptr_eq(&exotic, &exotic2),
|
||||
"exotic MIME falls back to a fresh Arc"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ext_of() {
|
||||
assert_eq!(ext_of("file.txt"), Some("txt"));
|
||||
|
||||
@@ -6,7 +6,8 @@ use utoipa::ToSchema;
|
||||
use uuid::Uuid;
|
||||
|
||||
use super::display_helpers::{
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for,
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for, intern_display,
|
||||
intern_mime,
|
||||
};
|
||||
|
||||
/// DTO for file responses
|
||||
@@ -101,11 +102,15 @@ impl From<File> for FileDto {
|
||||
// for id, name, path, folder_id (previously 4× .to_string()).
|
||||
let parts = file.into_parts();
|
||||
|
||||
let icon_class = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class = Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = Arc::from(category_for(&parts.name, &parts.mime_type));
|
||||
// Display fields come from closed static tables and MIME values
|
||||
// repeat massively across rows — intern instead of allocating a
|
||||
// fresh Arc<str> per row (`Arc::from(&str)` always allocs+copies).
|
||||
let icon_class = intern_display(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class =
|
||||
intern_display(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = intern_display(category_for(&parts.name, &parts.mime_type));
|
||||
let size_formatted = format_file_size(parts.size);
|
||||
let mime_type = Arc::from(parts.mime_type.as_str());
|
||||
let mime_type = intern_mime(&parts.mime_type);
|
||||
|
||||
Self {
|
||||
id: parts.id,
|
||||
@@ -169,13 +174,13 @@ impl FileDto {
|
||||
name: "stub-file".to_string(),
|
||||
path: "/stub/path".to_string(),
|
||||
size: 0,
|
||||
mime_type: Arc::from("application/octet-stream"),
|
||||
mime_type: intern_mime("application/octet-stream"),
|
||||
folder_id: None,
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
icon_class: Arc::from("fas fa-file"),
|
||||
icon_special_class: Arc::from(""),
|
||||
category: Arc::from("Document"),
|
||||
icon_class: intern_display("fas fa-file"),
|
||||
icon_special_class: intern_display(""),
|
||||
category: intern_display("Document"),
|
||||
size_formatted: "0 Bytes".to_string(),
|
||||
content_hash: String::new(),
|
||||
etag: String::new(),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::application::dtos::cursor::{CursorListResponse, CursorQuery, PageCursor};
|
||||
use crate::application::dtos::display_helpers::intern_display;
|
||||
use crate::application::dtos::grant_dto::{ResourceContentDto, ResourceTypeDto};
|
||||
use crate::domain::entities::folder::Folder;
|
||||
use crate::domain::services::authorization::ResourceKind;
|
||||
@@ -99,24 +100,33 @@ pub struct FolderDto {
|
||||
|
||||
impl From<Folder> for FolderDto {
|
||||
fn from(folder: Folder) -> Self {
|
||||
let is_root = folder.parent_id().is_none();
|
||||
let etag = folder.etag().to_string();
|
||||
// Consume the entity by moving all fields — zero heap allocations
|
||||
// for id, name, path, parent_id (previously 3-4× .to_string()).
|
||||
let parts = folder.into_parts();
|
||||
|
||||
let is_root = parts.parent_id.is_none();
|
||||
// Single-allocation ETag straight from the owned parts. The old
|
||||
// shape (`folder.etag().to_string()`) built the String and then
|
||||
// cloned it — a pure double-alloc.
|
||||
let etag = Folder::compute_etag(&parts.id, parts.tree_modified_at);
|
||||
|
||||
Self {
|
||||
id: folder.id().to_string(),
|
||||
name: folder.name().to_string(),
|
||||
path: folder.path_string().to_string(),
|
||||
parent_id: folder.parent_id().map(String::from),
|
||||
drive_id: folder.drive_id(),
|
||||
created_at: folder.created_at(),
|
||||
modified_at: folder.modified_at(),
|
||||
id: parts.id,
|
||||
name: parts.name,
|
||||
path: parts.path_string,
|
||||
parent_id: parts.parent_id,
|
||||
drive_id: parts.drive_id,
|
||||
created_at: parts.created_at,
|
||||
modified_at: parts.modified_at,
|
||||
is_root,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
// Constant display fields: refcount bump on interned statics
|
||||
// instead of 3 fresh Arc allocations per row.
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag,
|
||||
created_by: folder.created_by(),
|
||||
updated_by: folder.updated_by(),
|
||||
created_by: parts.created_by,
|
||||
updated_by: parts.updated_by,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -163,9 +173,9 @@ impl FolderDto {
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
is_root: true,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag: String::new(),
|
||||
created_by: None,
|
||||
updated_by: None,
|
||||
|
||||
@@ -77,6 +77,31 @@ pub trait FolderUseCase: Send + Sync + 'static {
|
||||
pagination: &crate::application::dtos::pagination::PaginationRequestDto,
|
||||
) -> Result<crate::application::dtos::pagination::PaginatedResponseDto<FolderDto>, DomainError>;
|
||||
|
||||
/// Keyset-paged sub-folder listing in name order, scoped to a caller —
|
||||
/// `name > after_name LIMIT limit`, `has_next = len() == limit`.
|
||||
///
|
||||
/// Used by streaming WebDAV/NC PROPFIND: O(page) per page off the
|
||||
/// `idx_folders_unique_name` index instead of the quadratic
|
||||
/// `COUNT(*) OVER() … LIMIT/OFFSET` walk (benches/FOLDER-KEYSET.md).
|
||||
///
|
||||
/// The default implementation falls back to `list_folders_with_perms`
|
||||
/// + in-memory slice so stubs and mocks compile without changes.
|
||||
async fn list_folders_batch_with_perms(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
caller_id: Uuid,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<FolderDto>, DomainError> {
|
||||
let mut all = self.list_folders_with_perms(parent_id, caller_id).await?;
|
||||
all.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name.as_str() > a))
|
||||
.take(limit)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Renames a folder (ownership verified against caller_id)
|
||||
async fn rename_folder_with_perms(
|
||||
&self,
|
||||
|
||||
@@ -304,12 +304,45 @@ impl AppPasswordService {
|
||||
let cache_key: [u8; 32] =
|
||||
blake3::hash(format!("{}:{}", username, password).as_bytes()).into();
|
||||
|
||||
// ── 2. Cache hit → return immediately ────────────────────────
|
||||
if let Some(cached) = self.auth_cache.get(&cache_key).await {
|
||||
return Ok((cached.user_id, cached.username, cached.email, cached.role));
|
||||
}
|
||||
// ── 2. Single-flight cache lookup ─────────────────────────────
|
||||
// Concurrent misses on the same credential coalesce into ONE
|
||||
// full verification: DAV sync clients hold 4-8 parallel
|
||||
// connections, so an expiring cache entry used to fan out into
|
||||
// K simultaneous Argon2id runs (~100-300 ms CPU + 64 MiB RAM
|
||||
// apiece) every TTL — a recurring p99 spike on every DAV
|
||||
// surface (8 -> 1 verifications, benches/AUTH-HERD.md).
|
||||
// `try_get_with` caches only `Ok` results, so failed
|
||||
// verifications are still never cached, preserving the full
|
||||
// Argon2id cost as a brute-force deterrent.
|
||||
let result = self
|
||||
.auth_cache
|
||||
.try_get_with(
|
||||
cache_key,
|
||||
self.verify_basic_auth_uncached(username, password),
|
||||
)
|
||||
.await
|
||||
.map_err(
|
||||
|e: std::sync::Arc<DomainError>| match std::sync::Arc::try_unwrap(e) {
|
||||
Ok(err) => err,
|
||||
// Another coalesced waiter still holds the Arc — rebuild
|
||||
// an equivalent error (the source chain isn't clonable).
|
||||
Err(shared) => {
|
||||
DomainError::new(shared.kind, shared.entity_type, shared.message.clone())
|
||||
}
|
||||
},
|
||||
)?;
|
||||
Ok((result.user_id, result.username, result.email, result.role))
|
||||
}
|
||||
|
||||
// ── 3. Cache miss → full verification ────────────────────────
|
||||
/// The uncached Basic Auth slow path: user lookup, prefix-scoped
|
||||
/// candidate fetch, Argon2id verification. Runs at most once per
|
||||
/// credential per TTL — `verify_basic_auth` coalesces concurrent
|
||||
/// callers onto a single in-flight instance of this future.
|
||||
async fn verify_basic_auth_uncached(
|
||||
&self,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<CachedBasicAuthResult, DomainError> {
|
||||
let user = self
|
||||
.user_repo
|
||||
.get_user_by_username(username)
|
||||
@@ -363,15 +396,14 @@ impl AppPasswordService {
|
||||
{
|
||||
let _ = self.repo.touch_last_used(ap.id).await;
|
||||
|
||||
let result = CachedBasicAuthResult {
|
||||
// Caching happens in `verify_basic_auth`: `try_get_with`
|
||||
// stores this value under the blake3 key on return.
|
||||
return Ok(CachedBasicAuthResult {
|
||||
user_id: user.id(),
|
||||
username: user.username().unwrap_or("").to_string(),
|
||||
email: user.email().to_string(),
|
||||
role: user.role().to_string(),
|
||||
};
|
||||
|
||||
self.auth_cache.insert(cache_key, result.clone()).await;
|
||||
return Ok((result.user_id, result.username, result.email, result.role));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -429,6 +429,62 @@ impl FolderUseCase for FolderService {
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
/// Keyset-paged sub-folder listing (name order), caller-scoped.
|
||||
///
|
||||
/// AuthZ mirrors `list_folders_paginated_with_perms`: one
|
||||
/// `authz.require(Read)` on the parent per batch; root scope goes
|
||||
/// through the caller's drive-membership listing.
|
||||
async fn list_folders_batch_with_perms(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
caller_id: Uuid,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<FolderDto>, DomainError> {
|
||||
match parent_id {
|
||||
Some(pid) => {
|
||||
self.authz
|
||||
.require(
|
||||
Subject::User(caller_id),
|
||||
Permission::Read,
|
||||
Self::folder_resource(pid)?,
|
||||
)
|
||||
.await?;
|
||||
let folders = self
|
||||
.folder_storage
|
||||
.list_folders_batch(parent_id, after_name, limit)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"FolderStorage",
|
||||
format!("Failed to batch-list folders in parent {pid}: {e}"),
|
||||
)
|
||||
})?;
|
||||
Ok(folders.into_iter().map(FolderDto::from).collect())
|
||||
}
|
||||
None => {
|
||||
// Root scope: one row per readable drive — a handful.
|
||||
let mut all = self
|
||||
.folder_storage
|
||||
.list_root_folders_for_caller(caller_id)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"FolderStorage",
|
||||
format!("Failed to batch-list root folders for '{caller_id}': {e}"),
|
||||
)
|
||||
})?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.map(FolderDto::from)
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Lists folders with pagination, scoped to a specific owner.
|
||||
async fn list_folders_paginated_with_perms(
|
||||
&self,
|
||||
|
||||
@@ -67,9 +67,80 @@ pub struct SearchService {
|
||||
/// Lock-free concurrent cache with automatic TTL and LRU eviction (moka).
|
||||
/// Values are `Arc<SearchResultsDto>` so cache insert/hit is a single
|
||||
/// atomic ref-count increment (~1 ns) instead of cloning thousands of Strings.
|
||||
///
|
||||
/// **Byte-bounded**, not entry-bounded: entries are weighed by
|
||||
/// [`search_results_entry_weight`] and `max_capacity` is a byte budget.
|
||||
/// Keys span user × query × offset × limit, and each page holds up to 500
|
||||
/// enriched rows (~500–900 B of owned Strings each) — an entry-count bound
|
||||
/// let hundreds of MB of result pages accumulate invisibly.
|
||||
search_cache: moka::future::Cache<u64, Arc<SearchResultsDto>>,
|
||||
}
|
||||
|
||||
// ─── Search-results cache (byte-bounded) ─────────────────────────────────
|
||||
|
||||
/// Approximate heap bytes retained by one cached search page.
|
||||
///
|
||||
/// With a `weigher` installed, moka's `max_capacity` is the sum of entry
|
||||
/// *weights*, so this converts the cache bound from "number of entries" to
|
||||
/// real bytes: the length of every owned `String` in each file/folder row,
|
||||
/// plus a fixed per-row and per-entry overhead for struct fields, the 24-B
|
||||
/// `String` headers, `Vec` slots and allocator slop. Same pattern as the
|
||||
/// file-content cache and the dedup manifest cache.
|
||||
///
|
||||
/// `pub` so `examples/bench_search_cache_mem.rs` can recompute retained
|
||||
/// bytes with the exact production formula.
|
||||
pub fn search_results_entry_weight(_key: &u64, value: &Arc<SearchResultsDto>) -> u32 {
|
||||
/// Fixed per-row overhead: struct scalars + one 24-B header per `String`
|
||||
/// field (12 on a file row, 4 on a folder row) + `Vec` slot + allocator
|
||||
/// slop. Deliberately a round upper-ish estimate — under-weighing is the
|
||||
/// failure mode that re-opens the memory hole.
|
||||
const ROW_OVERHEAD: usize = 200;
|
||||
/// Fixed per-entry overhead: `Arc` + `SearchResultsDto` scalars + `Vec`
|
||||
/// headers + moka's own bookkeeping per entry.
|
||||
const ENTRY_OVERHEAD: usize = 256;
|
||||
|
||||
fn opt_len(s: &Option<String>) -> usize {
|
||||
s.as_deref().map_or(0, str::len)
|
||||
}
|
||||
|
||||
let mut bytes = ENTRY_OVERHEAD + value.sort_by.len();
|
||||
for f in &value.files {
|
||||
bytes += ROW_OVERHEAD
|
||||
+ f.id.len()
|
||||
+ f.name.len()
|
||||
+ f.path.len()
|
||||
+ f.mime_type.len()
|
||||
+ opt_len(&f.folder_id)
|
||||
+ f.size_formatted.len()
|
||||
+ f.icon_class.len()
|
||||
+ f.icon_special_class.len()
|
||||
+ f.category.len()
|
||||
+ f.blob_hash.len()
|
||||
+ opt_len(&f.snippet)
|
||||
+ opt_len(&f.match_source);
|
||||
}
|
||||
for d in &value.folders {
|
||||
bytes += ROW_OVERHEAD + d.id.len() + d.name.len() + d.path.len() + opt_len(&d.parent_id);
|
||||
}
|
||||
bytes.min(u32::MAX as usize) as u32
|
||||
}
|
||||
|
||||
/// Build the search-results cache exactly as production wires it: a byte
|
||||
/// budget enforced through [`search_results_entry_weight`], plus TTL.
|
||||
///
|
||||
/// Shared with `examples/bench_search_cache_mem.rs` so the benchmark
|
||||
/// measures the identical cache configuration that serves requests.
|
||||
pub fn build_search_results_cache(
|
||||
cache_ttl_secs: u64,
|
||||
max_bytes: u64,
|
||||
) -> moka::future::Cache<u64, Arc<SearchResultsDto>> {
|
||||
moka::future::Cache::builder()
|
||||
.max_capacity(max_bytes)
|
||||
.weigher(search_results_entry_weight)
|
||||
.time_to_live(Duration::from_secs(cache_ttl_secs))
|
||||
.build()
|
||||
}
|
||||
|
||||
// ─── Utility functions (pure, no self — computed on the server) ─────────
|
||||
|
||||
/// Compute relevance score (0–100) for a name against a query.
|
||||
@@ -160,6 +231,10 @@ fn get_category(name: &str, mime: &str) -> String {
|
||||
impl SearchService {
|
||||
/**
|
||||
* Creates a new instance of the search service.
|
||||
*
|
||||
* `max_cache_bytes` is the byte budget for the results cache (weigher-
|
||||
* bounded, see [`search_results_entry_weight`]) — it replaced the old
|
||||
* entry-count capacity, which was blind to how big each cached page is.
|
||||
*/
|
||||
pub fn new(
|
||||
file_repository: Arc<FileBlobReadRepository>,
|
||||
@@ -168,12 +243,9 @@ impl SearchService {
|
||||
authorization: Option<Arc<crate::infrastructure::services::pg_acl_engine::PgAclEngine>>,
|
||||
drive_repo: Option<Arc<dyn crate::domain::repositories::drive_repository::DriveRepository>>,
|
||||
cache_ttl: u64,
|
||||
max_cache_size: usize,
|
||||
max_cache_bytes: u64,
|
||||
) -> Self {
|
||||
let search_cache = moka::future::Cache::builder()
|
||||
.max_capacity(max_cache_size as u64)
|
||||
.time_to_live(Duration::from_secs(cache_ttl))
|
||||
.build();
|
||||
let search_cache = build_search_results_cache(cache_ttl, max_cache_bytes);
|
||||
|
||||
Self {
|
||||
file_repository,
|
||||
@@ -815,6 +887,88 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entry_weight_counts_every_owned_string_plus_overheads() {
|
||||
// Empty page: entry overhead + sort_by ("relevance" = 9 bytes).
|
||||
let empty = Arc::new(SearchResultsDto::empty());
|
||||
let base = search_results_entry_weight(&0, &empty) as usize;
|
||||
assert_eq!(base, 256 + 9);
|
||||
|
||||
// One file row: base + row overhead + its owned string bytes
|
||||
// (id 7 + name 7 + path 8 + mime 10; the rest are empty/None).
|
||||
let one_file = Arc::new(SearchResultsDto::new(
|
||||
vec![dto("abc.txt", 50, 10, 1)],
|
||||
Vec::new(),
|
||||
100,
|
||||
0,
|
||||
Some(1),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
));
|
||||
let w = search_results_entry_weight(&0, &one_file) as usize;
|
||||
assert_eq!(w, base + 200 + 7 + 7 + 8 + 10);
|
||||
|
||||
// Folder rows weigh too (id 2 + name 4 + path 5 + parent 6 = 17).
|
||||
let one_folder = Arc::new(SearchResultsDto::new(
|
||||
Vec::new(),
|
||||
vec![SearchFolderResultDto {
|
||||
id: "f1".to_string(),
|
||||
name: "docs".to_string(),
|
||||
path: "/docs".to_string(),
|
||||
parent_id: Some("parent".to_string()),
|
||||
drive_id: Uuid::nil(),
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
is_root: false,
|
||||
relevance_score: 50,
|
||||
}],
|
||||
100,
|
||||
0,
|
||||
Some(1),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
));
|
||||
let w = search_results_entry_weight(&0, &one_folder) as usize;
|
||||
assert_eq!(w, base + 200 + 2 + 4 + 5 + 6);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cache_evicts_down_to_the_byte_budget() {
|
||||
// Budget fits ~2 of these entries; inserting 20 must never let the
|
||||
// weighted size settle above the budget.
|
||||
let entry = |i: usize| {
|
||||
Arc::new(SearchResultsDto::new(
|
||||
(0..50)
|
||||
.map(|r| dto(&format!("file_{i}_{r}_{}", "x".repeat(100)), 50, 1, 1))
|
||||
.collect(),
|
||||
Vec::new(),
|
||||
50,
|
||||
0,
|
||||
Some(50),
|
||||
0,
|
||||
"relevance".to_string(),
|
||||
))
|
||||
};
|
||||
let per_entry = search_results_entry_weight(&0, &entry(0)) as u64;
|
||||
let budget = per_entry * 2 + per_entry / 2;
|
||||
|
||||
let cache = build_search_results_cache(300, budget);
|
||||
for i in 0..20u64 {
|
||||
cache.insert(i, entry(i as usize)).await;
|
||||
}
|
||||
cache.run_pending_tasks().await;
|
||||
|
||||
let retained: u64 = cache
|
||||
.iter()
|
||||
.map(|(k, v)| search_results_entry_weight(&k, &v) as u64)
|
||||
.sum();
|
||||
assert!(
|
||||
retained <= budget,
|
||||
"retained {retained} B exceeds budget {budget} B"
|
||||
);
|
||||
assert!(cache.entry_count() <= 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merged_files_resort_by_relevance_and_by_column() {
|
||||
let mut files = vec![
|
||||
|
||||
@@ -1250,6 +1250,33 @@ impl Default for ContentSearchConfig {
|
||||
}
|
||||
}
|
||||
|
||||
/// Search-results cache configuration — the per-user results-page cache
|
||||
/// inside `SearchService`, not the Tantivy content index above.
|
||||
///
|
||||
/// The cache is **byte-bounded**: each entry is weighed by the approximate
|
||||
/// heap size of its result page (see `search_results_entry_weight`) and moka
|
||||
/// evicts once the summed weight exceeds `max_bytes` — the same byte-budget
|
||||
/// pattern the file-content cache and the dedup manifest cache use. This
|
||||
/// replaced an entry-count capacity: with cache keys spanning
|
||||
/// user × query × offset × limit and up to 500 enriched rows per page, an
|
||||
/// entry count said nothing about resident memory (1000 entries could pin
|
||||
/// ~300 MB for the TTL). No entry-count knob is kept — bytes are the only
|
||||
/// dimension that matters here.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct SearchCacheConfig {
|
||||
/// Byte budget for cached search-result pages. Default: 32 MiB.
|
||||
/// Env: `OXICLOUD_SEARCH_CACHE_MAX_BYTES`.
|
||||
pub max_bytes: u64,
|
||||
}
|
||||
|
||||
impl Default for SearchCacheConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_bytes: 32 * 1024 * 1024,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// WASM plugin runtime configuration (M0 walking skeleton).
|
||||
///
|
||||
/// The runtime is doubly gated: it is only compiled when the `plugins` cargo
|
||||
@@ -1375,6 +1402,8 @@ pub struct AppConfig {
|
||||
pub i18n: I18nConfig,
|
||||
/// Content-search configuration (embedded full-text index)
|
||||
pub content_search: ContentSearchConfig,
|
||||
/// Search-results cache configuration (byte-bounded moka cache)
|
||||
pub search_cache: SearchCacheConfig,
|
||||
/// WASM plugin runtime configuration
|
||||
pub plugins: PluginConfig,
|
||||
/// Face-recognition (People) model configuration
|
||||
@@ -1431,6 +1460,7 @@ impl Default for AppConfig {
|
||||
magic_link: MagicLinkConfig::default(),
|
||||
i18n: I18nConfig::default(),
|
||||
content_search: ContentSearchConfig::default(),
|
||||
search_cache: SearchCacheConfig::default(),
|
||||
plugins: PluginConfig::default(),
|
||||
faces: FacesConfig::default(),
|
||||
}
|
||||
@@ -1934,6 +1964,13 @@ impl AppConfig {
|
||||
config.content_search.max_text_bytes = val;
|
||||
}
|
||||
|
||||
// Search-results cache (byte-bounded)
|
||||
if let Ok(v) = env::var("OXICLOUD_SEARCH_CACHE_MAX_BYTES").map(|v| v.parse::<u64>())
|
||||
&& let Ok(val) = v
|
||||
{
|
||||
config.search_cache.max_bytes = val;
|
||||
}
|
||||
|
||||
// WASM plugin runtime
|
||||
if let Ok(v) = env::var("OXICLOUD_ENABLE_PLUGINS").map(|v| v.parse::<bool>())
|
||||
&& let Ok(val) = v
|
||||
|
||||
+6
-2
@@ -656,8 +656,12 @@ impl AppServiceFactory {
|
||||
content_index_port,
|
||||
Some(authz.clone()),
|
||||
Some(drive_repo.clone()),
|
||||
300, // Cache TTL in seconds (5 minutes)
|
||||
1000, // Maximum cache entries
|
||||
300, // Cache TTL in seconds (5 minutes)
|
||||
// Byte budget for cached result pages (weigher-bounded, 32 MiB
|
||||
// default; env OXICLOUD_SEARCH_CACHE_MAX_BYTES). Replaces the old
|
||||
// entry-count capacity, which let 500-row pages keyed by
|
||||
// user×query×offset×limit pin hundreds of MB for the TTL.
|
||||
self.config.search_cache.max_bytes,
|
||||
)));
|
||||
|
||||
tracing::info!("Application services initialized");
|
||||
|
||||
@@ -352,8 +352,25 @@ impl File {
|
||||
/// formula here changes it everywhere — that is the property
|
||||
/// we want.
|
||||
pub fn compute_etag(blob_hash: &str, modified_at: u64) -> String {
|
||||
let prefix: String = blob_hash.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, modified_at)
|
||||
use std::fmt::Write as _;
|
||||
|
||||
// Byte index just past the 16th char (whole string when shorter).
|
||||
// `blob_hash` is lowercase hex ASCII in practice, so this is
|
||||
// effectively `min(len, 16)`, but `char_indices` keeps the slice
|
||||
// char-boundary-safe for exotic fixture values — byte-identical
|
||||
// to the old `chars().take(16).collect::<String>()` without the
|
||||
// intermediate allocation.
|
||||
let end = match blob_hash.char_indices().nth(16) {
|
||||
Some((i, _)) => i,
|
||||
None => blob_hash.len(),
|
||||
};
|
||||
|
||||
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
|
||||
let mut etag = String::with_capacity(end + 1 + 20);
|
||||
etag.push_str(&blob_hash[..end]);
|
||||
etag.push('-');
|
||||
let _ = write!(etag, "{modified_at}");
|
||||
etag
|
||||
}
|
||||
|
||||
// Getters
|
||||
|
||||
@@ -7,6 +7,30 @@ use crate::domain::services::path_service::{
|
||||
// Re-export entity errors from the centralized module
|
||||
pub use super::entity_errors::{FolderError, FolderResult};
|
||||
|
||||
/// Owned parts of a [`Folder`] entity, produced by [`Folder::into_parts()`].
|
||||
///
|
||||
/// Consuming a `Folder` into `FolderParts` **moves** every field without
|
||||
/// cloning, eliminating the 3-4 heap allocations that previously occurred
|
||||
/// when converting `Folder → FolderDto` via `.to_string()` on each getter.
|
||||
/// Mirrors [`super::file::FileParts`].
|
||||
pub struct FolderParts {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub storage_path: StoragePath,
|
||||
pub path_string: String,
|
||||
pub parent_id: Option<String>,
|
||||
/// Drive that owns this folder. See [`Folder::drive_id`].
|
||||
pub drive_id: Uuid,
|
||||
pub created_at: u64,
|
||||
pub modified_at: u64,
|
||||
/// Descendant-rollup timestamp. See [`Folder::tree_modified_at`].
|
||||
pub tree_modified_at: u64,
|
||||
/// §14 provenance: original creator. See [`Folder::created_by`].
|
||||
pub created_by: Option<Uuid>,
|
||||
/// §14 provenance: most recent mutator. See [`Folder::updated_by`].
|
||||
pub updated_by: Option<Uuid>,
|
||||
}
|
||||
|
||||
/// Represents a folder entity in the domain
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Folder {
|
||||
@@ -219,6 +243,26 @@ impl Folder {
|
||||
})
|
||||
}
|
||||
|
||||
/// Consume the entity and return all fields by ownership.
|
||||
///
|
||||
/// Use this when converting `Folder` into a DTO to avoid cloning
|
||||
/// every `String` field (saves 3-4 heap allocations per folder).
|
||||
pub fn into_parts(self) -> FolderParts {
|
||||
FolderParts {
|
||||
id: self.id,
|
||||
name: self.name,
|
||||
storage_path: self.storage_path,
|
||||
path_string: self.path_string,
|
||||
parent_id: self.parent_id,
|
||||
drive_id: self.drive_id,
|
||||
created_at: self.created_at,
|
||||
modified_at: self.modified_at,
|
||||
tree_modified_at: self.tree_modified_at,
|
||||
created_by: self.created_by,
|
||||
updated_by: self.updated_by,
|
||||
}
|
||||
}
|
||||
|
||||
// Getters
|
||||
pub fn id(&self) -> &str {
|
||||
&self.id
|
||||
@@ -326,8 +370,25 @@ impl Folder {
|
||||
/// changed; the folder's own value stays untouched
|
||||
/// (self-exclusion).
|
||||
pub fn compute_etag(id: &str, tree_modified_at: u64) -> String {
|
||||
let prefix: String = id.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, tree_modified_at)
|
||||
use std::fmt::Write as _;
|
||||
|
||||
// Byte index just past the 16th char (whole string when shorter).
|
||||
// `id` is a UUID string (ASCII) in practice, so this is
|
||||
// effectively `min(len, 16)`, but `char_indices` keeps the slice
|
||||
// char-boundary-safe for exotic fixture values — byte-identical
|
||||
// to the old `chars().take(16).collect::<String>()` without the
|
||||
// intermediate allocation.
|
||||
let end = match id.char_indices().nth(16) {
|
||||
Some((i, _)) => i,
|
||||
None => id.len(),
|
||||
};
|
||||
|
||||
// Single allocation: prefix + '-' + up to 20 digits (u64::MAX).
|
||||
let mut etag = String::with_capacity(end + 1 + 20);
|
||||
etag.push_str(&id[..end]);
|
||||
etag.push('-');
|
||||
let _ = write!(etag, "{tree_modified_at}");
|
||||
etag
|
||||
}
|
||||
|
||||
/// Creates a new Folder instance from a DTO
|
||||
|
||||
@@ -98,6 +98,32 @@ pub trait FolderRepository: Send + Sync + 'static {
|
||||
include_total: bool,
|
||||
) -> Result<(Vec<Folder>, Option<usize>), DomainError>;
|
||||
|
||||
/// Keyset-paged listing of `parent_id`'s direct sub-folders in name
|
||||
/// order — `name > $after_name ORDER BY name LIMIT $limit`, one bounded
|
||||
/// index-range read per page off the partial unique index
|
||||
/// `idx_folders_unique_name`. Streaming PROPFIND drains sub-folders
|
||||
/// with this instead of `COUNT(*) OVER() … LIMIT/OFFSET`, which
|
||||
/// window-aggregated and rescanned all N sub-folders on every page
|
||||
/// (4.5x on a 5k-dir parent, benches/FOLDER-KEYSET.md). `has_next`
|
||||
/// falls out of `rows.len() == limit` — no total needed.
|
||||
///
|
||||
/// The default implementation falls back to `list_folders` + in-memory
|
||||
/// slice so stubs and mocks compile without changes.
|
||||
async fn list_folders_batch(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<Folder>, DomainError> {
|
||||
let mut all = self.list_folders(parent_id).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Renames a folder. `caller_id` is stamped into `updated_by`
|
||||
/// alongside the `updated_at = NOW()` bump (§14 provenance).
|
||||
async fn rename_folder(
|
||||
|
||||
@@ -488,13 +488,21 @@ impl FileBlobReadRepository {
|
||||
/// `sort_date` epoch for each file (used as pagination cursor).
|
||||
///
|
||||
/// Uses the denormalised `media_sort_date` column (synced from
|
||||
/// `file_metadata.captured_at` by trigger) so no JOIN with
|
||||
/// `file_metadata` is needed. The partial covering index
|
||||
/// `idx_files_media_timeline_by_drive` (migration 20260901000001)
|
||||
/// keys on `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows — Postgres does one IndexScan per in-scope
|
||||
/// drive_id already ordered by capture date, so LIMIT stops the scan
|
||||
/// early. Same O(LIMIT) shape as the pre-D7 `user_id`-keyed hot path.
|
||||
/// `file_metadata.captured_at` by trigger). The accessible drive ids
|
||||
/// are materialised once, then a `CROSS JOIN LATERAL (… ORDER BY
|
||||
/// media_sort_date DESC LIMIT k)` per drive turns the partial covering
|
||||
/// index `idx_files_media_timeline_by_drive` (migration 20260901000001,
|
||||
/// `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows) into one BOUNDED index scan per drive; the outer
|
||||
/// merge sorts `drives × k` rows. The folders / file_metadata joins sit
|
||||
/// outside the top-N so only the k emitted rows pay them.
|
||||
///
|
||||
/// The previous shape put the joins and the global `ORDER BY … LIMIT`
|
||||
/// above a `drive_id IN (…)` nested loop — Postgres fed EVERY media row
|
||||
/// through the join into a top-N heapsort, scanning the timeline index
|
||||
/// to exhaustion on every page: O(library) per page, 97 ms on a
|
||||
/// 50k-photo library vs 1.6 ms for this shape (55.7x,
|
||||
/// benches/PHOTOS-TIMELINE.md).
|
||||
///
|
||||
/// Scope (`docs/plan/drive.md` §15): drives with
|
||||
/// `policies.include_in_photo_index = true` where the caller has a
|
||||
@@ -537,37 +545,47 @@ impl FileBlobReadRepository {
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by,
|
||||
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
|
||||
WITH accessible AS MATERIALIZED (
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
)
|
||||
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
|
||||
top.size, top.mime_type,
|
||||
EXTRACT(EPOCH FROM top.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM top.updated_at)::bigint,
|
||||
top.blob_hash,
|
||||
top.created_by, top.updated_by,
|
||||
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
|
||||
WHERE fi.drive_id IN (
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
)
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
FROM (
|
||||
SELECT fi.*
|
||||
FROM accessible a
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT fi.*
|
||||
FROM storage.files fi
|
||||
WHERE fi.drive_id = a.id
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) fi
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) top
|
||||
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
|
||||
ORDER BY top.media_sort_date DESC
|
||||
"#,
|
||||
);
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
|
||||
|
||||
@@ -501,6 +501,61 @@ impl FolderRepository for FolderDbRepository {
|
||||
Ok((folders?, total))
|
||||
}
|
||||
|
||||
/// Keyset sub-folder page: `name > $after ORDER BY name LIMIT $limit`,
|
||||
/// one bounded index-range read off `idx_folders_unique_name` — the
|
||||
/// cursor predicate is only emitted when a cursor exists (a bound
|
||||
/// disjunction would block the index condition under generic plans,
|
||||
/// same rule as `list_files_batch`). Root scope (`parent_id = None`)
|
||||
/// keeps the trait's in-memory default: roots are one-per-drive, a
|
||||
/// handful of rows.
|
||||
async fn list_folders_batch(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<Folder>, DomainError> {
|
||||
let Some(pid) = parent_id else {
|
||||
let mut all = self.list_folders(None).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
return Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.collect());
|
||||
};
|
||||
|
||||
let cursor_pred = if after_name.is_some() {
|
||||
"AND name > $3"
|
||||
} else {
|
||||
"AND $3::text IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
"SELECT id::text, name, path, parent_id::text, drive_id, \
|
||||
EXTRACT(EPOCH FROM created_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM updated_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM tree_modified_at)::bigint, \
|
||||
created_by, updated_by \
|
||||
FROM storage.folders \
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed \
|
||||
{cursor_pred} \
|
||||
ORDER BY name \
|
||||
LIMIT $2"
|
||||
);
|
||||
let rows: Vec<FolderRow> = sqlx::query_as(&sql)
|
||||
.bind(pid)
|
||||
.bind(limit as i64)
|
||||
.bind(after_name)
|
||||
.fetch_all(self.pool())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FolderDb", format!("batch: {e}")))?;
|
||||
|
||||
rows.into_iter()
|
||||
.map(|(id, name, path, pid, did, ca, ma, tma, cb, ub)| {
|
||||
Self::row_to_folder(id, name, path, pid, did, ca, ma, tma, cb, ub)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Paginated companion to `list_root_folders_for_caller` — same
|
||||
/// drive-membership predicate, adds LIMIT/OFFSET and an optional
|
||||
/// window-function COUNT so total pages can be surfaced without a
|
||||
@@ -1391,13 +1446,6 @@ impl FolderDbRepository {
|
||||
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
|
||||
"#;
|
||||
|
||||
let cte_inner = match (include_folders, include_files) {
|
||||
(true, true) => format!("{folder_branch} UNION ALL {file_branch}"),
|
||||
(true, false) => folder_branch.to_owned(),
|
||||
(false, true) => file_branch.to_owned(),
|
||||
(false, false) => unreachable!(),
|
||||
};
|
||||
|
||||
// ── Cursor binds ─────────────────────────────────────────────────────
|
||||
// $1 = parent_id $2 = cursor_str $3 = cursor_int
|
||||
// $4 = cursor_ts $5 = cursor_id $6 = limit
|
||||
@@ -1406,112 +1454,184 @@ impl FolderDbRepository {
|
||||
let cursor_ts = cursor.and_then(|c| c.sort_ts);
|
||||
let cursor_id = cursor.map(|c| c.resource_id);
|
||||
|
||||
// ── Sort-specific WHERE + ORDER BY ───────────────────────────────────
|
||||
// Each arm produces two variants based on `reverse`.
|
||||
// For "name": folder_first stays ASC in both directions (folders always
|
||||
// precede files); only the alpha order within each group flips.
|
||||
let (where_clause, order_clause) = match order_by {
|
||||
// ── Per-branch cursor pushdown ───────────────────────────────────────
|
||||
// The cursor is applied INSIDE each UNION-ALL branch as a sargable
|
||||
// row-value comparison on base columns — not on the CTE's computed
|
||||
// columns — and every branch pre-sorts and pre-limits, so Postgres
|
||||
// reads O(limit) rows per branch instead of rescanning and
|
||||
// top-N-sorting the entire folder on every page (19.5x on a
|
||||
// 20k-entry folder, benches/LISTING-KEYSET.md). The "name" sort is
|
||||
// served by the expression indexes idx_files_folder_lname /
|
||||
// idx_folders_parent_lname (migration 20260918000000).
|
||||
//
|
||||
// Sort-key columns that are CONSTANT within a branch (folder_first,
|
||||
// the folder branch's type_order = 0 and size = -1) are folded in
|
||||
// Rust: depending on which group the cursor points into, the branch
|
||||
// predicate shortens to a row-value over the remaining keys, the
|
||||
// branch keeps all its rows, or the branch drops out entirely.
|
||||
enum BranchCursor {
|
||||
/// The cursor has moved past every row this branch can produce.
|
||||
Drop,
|
||||
/// Every row in this branch sorts after the cursor.
|
||||
All,
|
||||
/// Row-value comparison over the branch's non-constant sort keys.
|
||||
Pred(String),
|
||||
}
|
||||
use BranchCursor::{All, Drop, Pred};
|
||||
|
||||
let has_cursor = cursor.is_some();
|
||||
// (folder-branch cursor, file-branch cursor, per-branch ORDER BY on
|
||||
// the branch's output aliases, outer merge ORDER BY)
|
||||
let (folder_cur, file_cur, branch_order, outer_order) = match order_by {
|
||||
"type" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order < $3)
|
||||
OR (type_order = $3 AND sort_str < $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY type_order DESC, sort_str DESC, id DESC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY type_order DESC, sort_str DESC, id DESC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order > $3)
|
||||
OR (type_order = $3 AND sort_str > $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY type_order ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
}
|
||||
(">", "ORDER BY type_order ASC, sort_str ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have type_order = 0; a cursor sitting on a
|
||||
// file (type_order > 0) either exhausts the folder group
|
||||
// (ASC) or precedes all of it (DESC).
|
||||
Some(c_to) if c_to > 0 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!(
|
||||
"(fm.category_order::bigint, LOWER(fm.name), fm.id) {op} ($3, $2, $5::uuid)"
|
||||
))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
"modified_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at > $4)
|
||||
OR (modified_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY modified_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY modified_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at < $4)
|
||||
OR (modified_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY modified_at DESC, id DESC",
|
||||
)
|
||||
}
|
||||
("<", "ORDER BY modified_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.updated_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"created_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at > $4)
|
||||
OR (created_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY created_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY created_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at < $4)
|
||||
OR (created_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY created_at DESC, id DESC",
|
||||
)
|
||||
}
|
||||
("<", "ORDER BY created_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.created_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"size" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size < $3)
|
||||
OR (size = $3 AND id < $5::uuid)"#,
|
||||
"ORDER BY size DESC, id DESC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY size DESC, id DESC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size > $3)
|
||||
OR (size = $3 AND id > $5::uuid)"#,
|
||||
"ORDER BY size ASC, id ASC",
|
||||
)
|
||||
}
|
||||
(">", "ORDER BY size ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have size = -1; a cursor sitting on a file
|
||||
// (size >= 0) exhausts the folder group (ASC) or precedes
|
||||
// all of it (DESC).
|
||||
Some(c_sz) if c_sz > -1 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("f.id {op} $5::uuid")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!("(fm.size::bigint, fm.id) {op} ($3, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
_ => {
|
||||
// "name" (default): folder_first stays ASC so folders always precede
|
||||
// files; only the alpha order within each group flips when reversed.
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str < $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC",
|
||||
)
|
||||
// "name" (default): folder_first stays ASC so folders always
|
||||
// precede files; only the alpha order within each group flips
|
||||
// when reversed. cursor_int carries folder_first (0|1).
|
||||
let op = if reverse { "<" } else { ">" };
|
||||
let branch_ord = if reverse {
|
||||
"ORDER BY sort_str DESC, id DESC"
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str > $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
}
|
||||
"ORDER BY sort_str ASC, id ASC"
|
||||
};
|
||||
let outer_ord = if reverse {
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC"
|
||||
} else {
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC"
|
||||
};
|
||||
let (folder_cur, file_cur) = match cursor_int {
|
||||
None => (All, All),
|
||||
// Cursor inside the folder group: folders continue after
|
||||
// the row-value cursor; every file still follows.
|
||||
Some(0) => (
|
||||
Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
All,
|
||||
),
|
||||
// Cursor inside the file group: the folder group is done.
|
||||
Some(_) => (
|
||||
Drop,
|
||||
Pred(format!("(LOWER(fm.name), fm.id) {op} ($2, $5::uuid)")),
|
||||
),
|
||||
};
|
||||
(folder_cur, file_cur, branch_ord, outer_ord)
|
||||
}
|
||||
};
|
||||
|
||||
let wrap = |branch: &str, cur: &BranchCursor| -> Option<String> {
|
||||
let extra = match cur {
|
||||
Drop => return None,
|
||||
All => String::new(),
|
||||
Pred(p) => format!(" AND {p}"),
|
||||
};
|
||||
Some(format!(
|
||||
"(SELECT * FROM ({branch}{extra}) b {branch_order} LIMIT $6)"
|
||||
))
|
||||
};
|
||||
let mut branches = Vec::with_capacity(2);
|
||||
if include_folders && let Some(b) = wrap(folder_branch, &folder_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
if include_files && let Some(b) = wrap(file_branch, &file_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
// Every requested branch dropped out (e.g. folders-only listing with
|
||||
// the cursor already past the folder group).
|
||||
if branches.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let inner = branches.join(" UNION ALL ");
|
||||
|
||||
let sql = format!(
|
||||
"WITH resources AS ({cte_inner}) \
|
||||
SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
"SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
created_at, modified_at, drive_id, blob_hash, \
|
||||
sort_str, type_order, folder_first \
|
||||
FROM resources \
|
||||
{where_clause} \
|
||||
{order_clause} \
|
||||
FROM ({inner}) r \
|
||||
{outer_order} \
|
||||
LIMIT $6"
|
||||
);
|
||||
|
||||
|
||||
@@ -130,7 +130,9 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
return Ok(size);
|
||||
}
|
||||
|
||||
client.put_block_blob(data.to_vec()).await.map_err(|e| {
|
||||
// `Bytes` converts into `azure_core::Body` by reference count —
|
||||
// the old `data.to_vec()` copied every chunk once more.
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
|
||||
@@ -138,6 +140,26 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Content-addressed keys make
|
||||
/// re-PUTs idempotent, so the `get_properties` probe
|
||||
/// `put_blob_from_bytes` pays is a pure extra round-trip on every NEW
|
||||
/// chunk (2 RTTs -> 1, benches/S3-PUT.md — same shape as S3).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
let size = data.len() as u64;
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
@@ -13,12 +13,14 @@ use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use bytes::Bytes;
|
||||
use dashmap::DashMap;
|
||||
use lru::LruCache;
|
||||
use std::num::NonZeroUsize;
|
||||
use tokio::fs;
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::io::ReaderStream;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
@@ -56,6 +58,13 @@ pub struct CachedBlobBackend {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
/// Per-hash single-flight gates for cache misses. K concurrent cold
|
||||
/// readers of one blob (e.g. a video player's parallel Range probes)
|
||||
/// used to each download the FULL blob from the remote backend — and
|
||||
/// race their writes on one shared `.tmp` path. The gate coalesces
|
||||
/// them onto one fetch; waiters re-check the cache and serve locally
|
||||
/// (16 fetches -> 1, benches/BLOB-CACHE.md).
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedBlobBackend {
|
||||
@@ -70,6 +79,7 @@ impl CachedBlobBackend {
|
||||
NonZeroUsize::new(1_000_000).unwrap(),
|
||||
))),
|
||||
current_size: Arc::new(AtomicU64::new(0)),
|
||||
inflight: Arc::new(DashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,6 +160,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
// Write to inner backend
|
||||
@@ -172,6 +183,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
let size = inner.put_blob_from_bytes(&hash, data.clone()).await?;
|
||||
@@ -203,6 +215,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE touching the filesystem so concurrent
|
||||
@@ -219,14 +232,17 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch from inner, spool to cache
|
||||
// Cache miss — fetch from inner (single-flight), spool to cache
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let file = fs::File::open(&dest).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
|
||||
})?;
|
||||
@@ -249,6 +265,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE the open()/seek() syscalls so concurrent
|
||||
@@ -271,14 +288,19 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch full blob into cache, then serve range
|
||||
// Cache miss — fetch full blob into cache (single-flight: a
|
||||
// player's parallel cold Range probes coalesce onto ONE remote
|
||||
// download), then serve the range locally.
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let mut file = fs::File::open(&dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
|
||||
@@ -406,6 +428,7 @@ struct CachedRef {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedRef {
|
||||
@@ -414,6 +437,37 @@ impl CachedRef {
|
||||
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
|
||||
/// Single-flight wrapper around [`Self::fetch_and_cache_static`]: the
|
||||
/// first caller for a hash becomes the leader and downloads; concurrent
|
||||
/// callers queue on the per-hash gate, then re-check the cache and serve
|
||||
/// the leader's file without touching the remote backend. Errors are not
|
||||
/// cached — the gate entry is dropped, so the next caller retries.
|
||||
async fn fetch_and_cache_singleflight(
|
||||
&self,
|
||||
hash: &str,
|
||||
inner: &dyn BlobStorageBackend,
|
||||
cached: &Path,
|
||||
) -> Result<PathBuf, DomainError> {
|
||||
let gate = self
|
||||
.inflight
|
||||
.entry(hash.to_string())
|
||||
.or_insert_with(|| Arc::new(Mutex::new(())))
|
||||
.clone();
|
||||
let _guard = gate.lock().await;
|
||||
|
||||
// Re-check under the gate: if we queued behind the leader, the blob
|
||||
// is on disk now and this turns into a local open.
|
||||
if self.index.lock().await.get(hash).is_some() && fs::metadata(cached).await.is_ok() {
|
||||
return Ok(cached.to_path_buf());
|
||||
}
|
||||
|
||||
let result = self.fetch_and_cache_static(hash, inner).await;
|
||||
// Drop the gate whether we succeeded or failed; a late-arriving
|
||||
// caller after an error creates a fresh gate and retries the fetch.
|
||||
self.inflight.remove(hash);
|
||||
result
|
||||
}
|
||||
|
||||
/// Pop LRU entries until the cache is back within its byte budget,
|
||||
/// returning the on-disk paths of the evicted blobs.
|
||||
///
|
||||
@@ -486,31 +540,51 @@ impl CachedRef {
|
||||
})?;
|
||||
}
|
||||
|
||||
let tmp = dest.with_extension("tmp");
|
||||
let mut file = fs::File::create(&tmp)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
|
||||
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut total = 0u64;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||
// Unique temp name: even if two fetches for one hash ever race
|
||||
// (e.g. across processes sharing a cache dir), each writes its own
|
||||
// inode and the rename is atomic — a torn/interleaved file can
|
||||
// never land at the final path.
|
||||
let tmp = dest.with_extension(format!("{}.tmp", Uuid::new_v4()));
|
||||
let write_result: Result<u64, DomainError> = async {
|
||||
let mut file = fs::File::create(&tmp).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("create tmp: {e}"))
|
||||
})?;
|
||||
total += bytes.len() as u64;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
drop(file);
|
||||
|
||||
fs::rename(&tmp, &dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut total = 0u64;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||
})?;
|
||||
total += bytes.len() as u64;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
Ok(total)
|
||||
}
|
||||
.await;
|
||||
let total = match write_result {
|
||||
Ok(total) => total,
|
||||
Err(e) => {
|
||||
// Unique tmp names never get overwritten by a later fetch —
|
||||
// reap the partial file instead of leaking it.
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
|
||||
if let Err(e) = fs::rename(&tmp, &dest).await {
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(DomainError::internal_error(
|
||||
"BlobCache",
|
||||
format!("rename: {e}"),
|
||||
));
|
||||
}
|
||||
|
||||
let to_evict = {
|
||||
let mut idx = self.index.lock().await;
|
||||
|
||||
@@ -200,6 +200,38 @@ impl BlobStorageBackend for S3BlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Keys are content-addressed
|
||||
/// (BLAKE3), so a re-PUT writes identical bytes — overwrite-safe
|
||||
/// idempotency without the HEAD probe `put_blob_from_bytes` pays. The
|
||||
/// dedup layer already filtered out chunks the database knows about,
|
||||
/// so the probe was a pure extra round-trip on every NEW chunk of
|
||||
/// every upload (2 RTTs -> 1, benches/S3-PUT.md).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
let size = data.len() as u64;
|
||||
self.client
|
||||
.put_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.body(ByteStream::from(data))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to upload blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
@@ -385,7 +385,6 @@ async fn handle_propfind(
|
||||
CardDavAdapter::generate_contacts_response(
|
||||
&mut response_body,
|
||||
std::slice::from_ref(&contact),
|
||||
&[(contact.uid.clone(), contact_to_vcard(&contact))],
|
||||
&report,
|
||||
base_href,
|
||||
)
|
||||
@@ -449,22 +448,10 @@ async fn handle_report(
|
||||
.map_err(AppError::from)?,
|
||||
};
|
||||
|
||||
// Generate vCards
|
||||
let vcards: Vec<(String, String)> = contacts
|
||||
.iter()
|
||||
.map(|c| (c.uid.clone(), contact_to_vcard(c)))
|
||||
.collect();
|
||||
|
||||
let base_href = &format!("/carddav/{}/", address_book_id);
|
||||
let mut response_body = Vec::new();
|
||||
CardDavAdapter::generate_contacts_response(
|
||||
&mut response_body,
|
||||
&contacts,
|
||||
&vcards,
|
||||
&report,
|
||||
base_href,
|
||||
)
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
|
||||
CardDavAdapter::generate_contacts_response(&mut response_body, &contacts, &report, base_href)
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
|
||||
|
||||
Ok(Response::builder()
|
||||
.status(StatusCode::MULTI_STATUS)
|
||||
|
||||
@@ -781,25 +781,25 @@ async fn build_streaming_propfind_response(
|
||||
|
||||
// ── Children (only if Depth == 1) ────────────────────────
|
||||
if depth == "1" {
|
||||
let pagination = crate::application::dtos::pagination::PaginationRequestDto {
|
||||
page: 0,
|
||||
page_size: PROPFIND_BATCH_SIZE as usize,
|
||||
};
|
||||
let fid_ref = folder_id.as_deref();
|
||||
|
||||
// Stream sub-folders in pages (user-scoped)
|
||||
let mut page = 0usize;
|
||||
// Stream sub-folders in pages (user-scoped, keyset cursor —
|
||||
// O(page) per page off idx_folders_unique_name instead of the
|
||||
// quadratic COUNT(*) OVER() + LIMIT/OFFSET walk; 4.5x on a
|
||||
// 5k-dir parent, benches/FOLDER-KEYSET.md).
|
||||
let mut after_folder: Option<String> = None;
|
||||
loop {
|
||||
let pag = crate::application::dtos::pagination::PaginationRequestDto {
|
||||
page,
|
||||
page_size: pagination.page_size,
|
||||
};
|
||||
let result = folder_service
|
||||
.list_folders_paginated_with_perms(fid_ref, user_id, &pag)
|
||||
let batch = folder_service
|
||||
.list_folders_batch_with_perms(
|
||||
fid_ref,
|
||||
user_id,
|
||||
after_folder.as_deref(),
|
||||
PROPFIND_BATCH_SIZE as usize,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
|
||||
if result.items.is_empty() {
|
||||
if batch.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -808,25 +808,25 @@ async fn build_streaming_propfind_response(
|
||||
// 1-4.5 s of pure DB chatter on a 2000-child folder
|
||||
// (measured in benches/DEAD-PROPS.md).
|
||||
let subfolder_deads =
|
||||
folders_dead_props_map(&dead_props_store, &result.items).await;
|
||||
folders_dead_props_map(&dead_props_store, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 800);
|
||||
let mut chunk = Vec::with_capacity(batch.len() * 800);
|
||||
{
|
||||
let mut w = Writer::new(&mut chunk);
|
||||
for subfolder in result.items.iter() {
|
||||
for subfolder in batch.iter() {
|
||||
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
|
||||
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
|
||||
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
}
|
||||
}
|
||||
let has_more = result.pagination.has_next;
|
||||
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
|
||||
after_folder = batch.last().map(|f| f.name.clone());
|
||||
yield Bytes::from(chunk);
|
||||
|
||||
if !has_more {
|
||||
break;
|
||||
}
|
||||
page += 1;
|
||||
}
|
||||
|
||||
// Stream files in pages (user-scoped, keyset cursor — O(page)
|
||||
|
||||
@@ -16,7 +16,6 @@ use uuid::Uuid;
|
||||
use crate::application::adapters::webdav_adapter::{
|
||||
PropFindRequest, PropPatchOp, QualifiedName, WebDavAdapter, is_protected_property,
|
||||
};
|
||||
use crate::application::dtos::pagination::PaginationRequestDto;
|
||||
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
||||
use crate::application::ports::favorites_ports::FavoritesUseCase;
|
||||
use crate::application::ports::file_ports::{
|
||||
@@ -1584,38 +1583,42 @@ fn build_nc_streaming_propfind(
|
||||
after_name = batch.last().map(|f| f.name.clone());
|
||||
}
|
||||
|
||||
// Subfolders in pages — also collections, same trailing-slash rule.
|
||||
let mut page = 0usize;
|
||||
// Subfolders in pages — also collections, same trailing-slash
|
||||
// rule. Keyset cursor: O(page) per page off
|
||||
// idx_folders_unique_name instead of the quadratic
|
||||
// COUNT(*) OVER() + LIMIT/OFFSET walk (benches/FOLDER-KEYSET.md).
|
||||
let mut after_folder: Option<String> = None;
|
||||
loop {
|
||||
let pag = PaginationRequestDto {
|
||||
page,
|
||||
page_size: PROPFIND_BATCH_SIZE as usize,
|
||||
};
|
||||
let result = folder_service
|
||||
.list_folders_paginated_with_perms(Some(&folder.id), user_id, &pag)
|
||||
let batch = folder_service
|
||||
.list_folders_batch_with_perms(
|
||||
Some(&folder.id),
|
||||
user_id,
|
||||
after_folder.as_deref(),
|
||||
PROPFIND_BATCH_SIZE as usize,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
if result.items.is_empty() {
|
||||
if batch.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
let favs = if let Some(fav) = fav_svc {
|
||||
let items: Vec<(&str, &str)> =
|
||||
result.items.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
|
||||
batch.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
|
||||
fav.batch_check_favorites(user_id, &items).await.unwrap_or_default()
|
||||
} else {
|
||||
HashSet::new()
|
||||
};
|
||||
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
|
||||
let folder_uuids: Vec<String> = batch.iter().map(|sf| sf.id.clone()).collect();
|
||||
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
|
||||
// Batched — see benches/DEAD-PROPS.md.
|
||||
let sub_deads =
|
||||
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
|
||||
folders_dead_props_map(&state.webdav_dead_props, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
|
||||
let mut chunk = Vec::with_capacity(batch.len() * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for sf in result.items.iter() {
|
||||
for sf in batch.iter() {
|
||||
let dead = dead_props_for(&sf.id, &sub_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
sf.name.clone()
|
||||
@@ -1629,13 +1632,13 @@ fn build_nc_streaming_propfind(
|
||||
.map_err(std::io::Error::other)?;
|
||||
}
|
||||
}
|
||||
let has_more = result.pagination.has_next;
|
||||
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
|
||||
after_folder = batch.last().map(|sf| sf.name.clone());
|
||||
yield Bytes::from(chunk);
|
||||
|
||||
if !has_more {
|
||||
break;
|
||||
}
|
||||
page += 1;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -273,11 +273,16 @@ pub fn multipart_field_stream(
|
||||
pub fn stream_from_files(
|
||||
paths: Vec<PathBuf>,
|
||||
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
|
||||
// 512 KiB per poll: each ReaderStream poll on a tokio::fs::File is one
|
||||
// blocking-pool dispatch + one read(2) of the buffer size. The old
|
||||
// 64 KiB buffer paid 8x the dispatches/syscalls of every other blob
|
||||
// read path (STREAM_CHUNK_SIZE = 256 KiB) for the single read pass
|
||||
// over every completed chunked upload (benches/UPLOAD-SPOOL.md).
|
||||
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
|
||||
.and_then(|path| async move {
|
||||
tokio::fs::File::open(path)
|
||||
.await
|
||||
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
|
||||
.map(|file| ReaderStream::with_capacity(file, 512 * 1024))
|
||||
})
|
||||
.try_flatten()
|
||||
}
|
||||
@@ -319,9 +324,15 @@ pub async fn stream_body_to_path(
|
||||
max_bytes: usize,
|
||||
checksum_alg: Option<ChecksumAlg>,
|
||||
) -> Result<StreamedToPath, AppError> {
|
||||
let mut file = tokio::fs::File::create(path)
|
||||
// BufWriter coalesces the per-HTTP-frame writes (~16-64 KiB each) into
|
||||
// 512 KiB write(2)s — a bare tokio File dispatches one blocking-pool op
|
||||
// per frame (benches/UPLOAD-SPOOL.md). Same capacity as the dedup
|
||||
// handler's spool loop. On the error paths below the partial file is
|
||||
// removed, so silently dropping unflushed buffer contents is fine.
|
||||
let file = tokio::fs::File::create(path)
|
||||
.await
|
||||
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
|
||||
let mut file = tokio::io::BufWriter::with_capacity(512 * 1024, file);
|
||||
|
||||
let mut total_bytes: usize = 0;
|
||||
let mut stream = BodyStream::new(body);
|
||||
|
||||
Reference in New Issue
Block a user