perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
@@ -8,6 +8,175 @@
|
||||
//! then fall back to the file extension when the MIME is generic
|
||||
//! (`application/octet-stream` or empty).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fmt::Write as _;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
// ─── Arc<str> interning for closed-set display values ────────────────
|
||||
//
|
||||
// `FileDto` / `FolderDto` store their display fields as `Arc<str>` so DTO
|
||||
// clones are O(1). But `Arc::<str>::from(&str)` always allocates + copies,
|
||||
// so building the DTO paid 3-4 heap allocations per row even though the
|
||||
// value space is a small closed set. Interning turns each conversion into
|
||||
// a HashMap lookup + refcount bump.
|
||||
|
||||
/// Every `&'static str` that [`icon_class_for`], [`icon_special_class_for`]
|
||||
/// and [`category_for`] can return, plus the folder-DTO constants.
|
||||
///
|
||||
/// Keep this table in sync when adding a value to those functions — a
|
||||
/// missing entry is not a bug (callers fall back to `Arc::from`, same
|
||||
/// bytes, one extra allocation), just a lost optimization.
|
||||
static DISPLAY_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const CLOSED_SET: &[&str] = &[
|
||||
// icon_class_for
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file-word",
|
||||
"fas fa-file-excel",
|
||||
"fas fa-file-powerpoint",
|
||||
"fas fa-file-archive",
|
||||
"fas fa-file-code",
|
||||
"fas fa-hdd",
|
||||
"fas fa-file-image",
|
||||
"fas fa-file-video",
|
||||
"fas fa-file-audio",
|
||||
"fas fa-file-alt",
|
||||
"fas fa-terminal",
|
||||
"fas fa-file",
|
||||
// icon_special_class_for
|
||||
"pdf-icon",
|
||||
"doc-icon",
|
||||
"spreadsheet-icon",
|
||||
"presentation-icon",
|
||||
"archive-icon",
|
||||
"code-icon json-icon",
|
||||
"code-icon js-icon",
|
||||
"code-icon ts-icon",
|
||||
"code-icon html-icon",
|
||||
"code-icon sql-icon",
|
||||
"code-icon config-icon",
|
||||
"code-icon php-icon",
|
||||
"script-icon",
|
||||
"installer-icon",
|
||||
"image-icon",
|
||||
"video-icon",
|
||||
"audio-icon",
|
||||
"code-icon py-icon",
|
||||
"code-icon rust-icon",
|
||||
"code-icon",
|
||||
"code-icon go-icon",
|
||||
"code-icon ruby-icon",
|
||||
"code-icon md-icon",
|
||||
"code-icon css-icon",
|
||||
"code-icon java-icon",
|
||||
"code-icon c-icon",
|
||||
"code-icon cs-icon",
|
||||
"code-icon swift-icon",
|
||||
"",
|
||||
// category_for
|
||||
"PDF",
|
||||
"Document",
|
||||
"Spreadsheet",
|
||||
"Presentation",
|
||||
"Archive",
|
||||
"Code",
|
||||
"Installer",
|
||||
"Image",
|
||||
"Video",
|
||||
"Audio",
|
||||
"Markdown",
|
||||
"Text",
|
||||
// FolderDto constants
|
||||
"fas fa-folder",
|
||||
"folder-icon",
|
||||
"Folder",
|
||||
];
|
||||
CLOSED_SET.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for a display value from the closed sets
|
||||
/// above (icon class, icon special class, category). Lookup + refcount
|
||||
/// bump instead of alloc + copy; unknown values (future additions not
|
||||
/// yet in the table) fall back to `Arc::from` with identical bytes.
|
||||
pub fn intern_display(s: &'static str) -> Arc<str> {
|
||||
DISPLAY_INTERN
|
||||
.get(s)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(s))
|
||||
}
|
||||
|
||||
/// The MIME types that dominate real storage rows. Exotic types fall back
|
||||
/// to a per-row `Arc::from` — correctness is unaffected, only the alloc is.
|
||||
static MIME_INTERN: LazyLock<HashMap<&'static str, Arc<str>>> = LazyLock::new(|| {
|
||||
const COMMON_MIMES: &[&str] = &[
|
||||
"",
|
||||
"directory",
|
||||
"application/octet-stream",
|
||||
// Images
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/gif",
|
||||
"image/webp",
|
||||
"image/svg+xml",
|
||||
"image/heic",
|
||||
"image/heif",
|
||||
"image/avif",
|
||||
"image/bmp",
|
||||
"image/tiff",
|
||||
"image/x-icon",
|
||||
// Video
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"video/webm",
|
||||
"video/x-matroska",
|
||||
"video/x-msvideo",
|
||||
// Audio
|
||||
"audio/mpeg",
|
||||
"audio/mp4",
|
||||
"audio/ogg",
|
||||
"audio/flac",
|
||||
"audio/wav",
|
||||
"audio/x-wav",
|
||||
"audio/aac",
|
||||
// Documents
|
||||
"application/pdf",
|
||||
"application/msword",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"application/vnd.ms-excel",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.ms-powerpoint",
|
||||
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
"application/vnd.oasis.opendocument.text",
|
||||
"application/vnd.oasis.opendocument.spreadsheet",
|
||||
// Text / code
|
||||
"text/plain",
|
||||
"text/csv",
|
||||
"text/html",
|
||||
"text/css",
|
||||
"text/markdown",
|
||||
"text/xml",
|
||||
"application/json",
|
||||
"application/javascript",
|
||||
"application/xml",
|
||||
"application/x-yaml",
|
||||
// Archives
|
||||
"application/zip",
|
||||
"application/gzip",
|
||||
"application/x-tar",
|
||||
"application/x-7z-compressed",
|
||||
"application/x-rar-compressed",
|
||||
];
|
||||
COMMON_MIMES.iter().map(|s| (*s, Arc::from(*s))).collect()
|
||||
});
|
||||
|
||||
/// Returns a shared `Arc<str>` for the given MIME type. Common types hit
|
||||
/// the intern table (refcount bump); exotic ones allocate as before.
|
||||
pub fn intern_mime(mime: &str) -> Arc<str> {
|
||||
MIME_INTERN
|
||||
.get(mime)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| Arc::from(mime))
|
||||
}
|
||||
|
||||
// ─── Private: extract lowercase extension from a filename ────────────
|
||||
fn ext_of(name: &str) -> Option<&str> {
|
||||
let name = name.rsplit('/').next().unwrap_or(name); // strip path
|
||||
@@ -388,11 +557,21 @@ pub fn format_file_size(bytes: u64) -> String {
|
||||
|
||||
let value = bytes as f64 / K.powi(i as i32);
|
||||
|
||||
// Two decimal places, then strip trailing zeros (matches JS parseFloat behaviour)
|
||||
let formatted = format!("{:.2}", value);
|
||||
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
|
||||
|
||||
format!("{} {}", formatted, SIZES[i])
|
||||
// Single buffer: write the 2-decimal value, strip trailing zeros in
|
||||
// place (matches JS parseFloat behaviour), then append the unit.
|
||||
// 16 chars covers the worst case ("16777216 TB" for u64::MAX,
|
||||
// "1023.99 Bytes" for the longest unit), so no realloc occurs.
|
||||
let mut out = String::with_capacity(16);
|
||||
let _ = write!(out, "{:.2}", value);
|
||||
while out.ends_with('0') {
|
||||
out.pop();
|
||||
}
|
||||
if out.ends_with('.') {
|
||||
out.pop();
|
||||
}
|
||||
out.push(' ');
|
||||
out.push_str(SIZES[i]);
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -506,6 +685,50 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Every value the closed-set display functions can return must hit
|
||||
/// the intern table (same bytes, shared allocation) — a miss is only
|
||||
/// a lost optimization, but this test keeps the table in sync.
|
||||
#[test]
|
||||
fn test_intern_display_covers_closed_sets_and_shares_storage() {
|
||||
for s in [
|
||||
"fas fa-file-pdf",
|
||||
"fas fa-file",
|
||||
"fas fa-terminal",
|
||||
"fas fa-folder",
|
||||
"code-icon rust-icon",
|
||||
"folder-icon",
|
||||
"",
|
||||
"PDF",
|
||||
"Folder",
|
||||
"Document",
|
||||
"Markdown",
|
||||
] {
|
||||
let a = intern_display(s);
|
||||
let b = intern_display(s);
|
||||
assert_eq!(&*a, s, "interned bytes must be identical");
|
||||
assert!(
|
||||
Arc::ptr_eq(&a, &b),
|
||||
"closed-set value {s:?} must come from the intern table"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_intern_mime_common_hits_table_exotic_falls_back() {
|
||||
let a = intern_mime("image/jpeg");
|
||||
let b = intern_mime("image/jpeg");
|
||||
assert_eq!(&*a, "image/jpeg");
|
||||
assert!(Arc::ptr_eq(&a, &b), "common MIME must be interned");
|
||||
|
||||
let exotic = intern_mime("chemical/x-pdb");
|
||||
assert_eq!(&*exotic, "chemical/x-pdb");
|
||||
let exotic2 = intern_mime("chemical/x-pdb");
|
||||
assert!(
|
||||
!Arc::ptr_eq(&exotic, &exotic2),
|
||||
"exotic MIME falls back to a fresh Arc"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ext_of() {
|
||||
assert_eq!(ext_of("file.txt"), Some("txt"));
|
||||
|
||||
@@ -6,7 +6,8 @@ use utoipa::ToSchema;
|
||||
use uuid::Uuid;
|
||||
|
||||
use super::display_helpers::{
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for,
|
||||
category_for, format_file_size, icon_class_for, icon_special_class_for, intern_display,
|
||||
intern_mime,
|
||||
};
|
||||
|
||||
/// DTO for file responses
|
||||
@@ -101,11 +102,15 @@ impl From<File> for FileDto {
|
||||
// for id, name, path, folder_id (previously 4× .to_string()).
|
||||
let parts = file.into_parts();
|
||||
|
||||
let icon_class = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class = Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = Arc::from(category_for(&parts.name, &parts.mime_type));
|
||||
// Display fields come from closed static tables and MIME values
|
||||
// repeat massively across rows — intern instead of allocating a
|
||||
// fresh Arc<str> per row (`Arc::from(&str)` always allocs+copies).
|
||||
let icon_class = intern_display(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class =
|
||||
intern_display(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category = intern_display(category_for(&parts.name, &parts.mime_type));
|
||||
let size_formatted = format_file_size(parts.size);
|
||||
let mime_type = Arc::from(parts.mime_type.as_str());
|
||||
let mime_type = intern_mime(&parts.mime_type);
|
||||
|
||||
Self {
|
||||
id: parts.id,
|
||||
@@ -169,13 +174,13 @@ impl FileDto {
|
||||
name: "stub-file".to_string(),
|
||||
path: "/stub/path".to_string(),
|
||||
size: 0,
|
||||
mime_type: Arc::from("application/octet-stream"),
|
||||
mime_type: intern_mime("application/octet-stream"),
|
||||
folder_id: None,
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
icon_class: Arc::from("fas fa-file"),
|
||||
icon_special_class: Arc::from(""),
|
||||
category: Arc::from("Document"),
|
||||
icon_class: intern_display("fas fa-file"),
|
||||
icon_special_class: intern_display(""),
|
||||
category: intern_display("Document"),
|
||||
size_formatted: "0 Bytes".to_string(),
|
||||
content_hash: String::new(),
|
||||
etag: String::new(),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::application::dtos::cursor::{CursorListResponse, CursorQuery, PageCursor};
|
||||
use crate::application::dtos::display_helpers::intern_display;
|
||||
use crate::application::dtos::grant_dto::{ResourceContentDto, ResourceTypeDto};
|
||||
use crate::domain::entities::folder::Folder;
|
||||
use crate::domain::services::authorization::ResourceKind;
|
||||
@@ -99,24 +100,33 @@ pub struct FolderDto {
|
||||
|
||||
impl From<Folder> for FolderDto {
|
||||
fn from(folder: Folder) -> Self {
|
||||
let is_root = folder.parent_id().is_none();
|
||||
let etag = folder.etag().to_string();
|
||||
// Consume the entity by moving all fields — zero heap allocations
|
||||
// for id, name, path, parent_id (previously 3-4× .to_string()).
|
||||
let parts = folder.into_parts();
|
||||
|
||||
let is_root = parts.parent_id.is_none();
|
||||
// Single-allocation ETag straight from the owned parts. The old
|
||||
// shape (`folder.etag().to_string()`) built the String and then
|
||||
// cloned it — a pure double-alloc.
|
||||
let etag = Folder::compute_etag(&parts.id, parts.tree_modified_at);
|
||||
|
||||
Self {
|
||||
id: folder.id().to_string(),
|
||||
name: folder.name().to_string(),
|
||||
path: folder.path_string().to_string(),
|
||||
parent_id: folder.parent_id().map(String::from),
|
||||
drive_id: folder.drive_id(),
|
||||
created_at: folder.created_at(),
|
||||
modified_at: folder.modified_at(),
|
||||
id: parts.id,
|
||||
name: parts.name,
|
||||
path: parts.path_string,
|
||||
parent_id: parts.parent_id,
|
||||
drive_id: parts.drive_id,
|
||||
created_at: parts.created_at,
|
||||
modified_at: parts.modified_at,
|
||||
is_root,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
// Constant display fields: refcount bump on interned statics
|
||||
// instead of 3 fresh Arc allocations per row.
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag,
|
||||
created_by: folder.created_by(),
|
||||
updated_by: folder.updated_by(),
|
||||
created_by: parts.created_by,
|
||||
updated_by: parts.updated_by,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -163,9 +173,9 @@ impl FolderDto {
|
||||
created_at: 0,
|
||||
modified_at: 0,
|
||||
is_root: true,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
icon_class: intern_display("fas fa-folder"),
|
||||
icon_special_class: intern_display("folder-icon"),
|
||||
category: intern_display("Folder"),
|
||||
etag: String::new(),
|
||||
created_by: None,
|
||||
updated_by: None,
|
||||
|
||||
Reference in New Issue
Block a user