cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

perf: round 11 — StoragePath joined-only, classifier fusion, memoized bodies, query-shape pack, SPA fine-grained stars

Browse Source 
Backend (each change benchmark-gated with BEFORE replicas + equivalence
gates; see examples/bench_round11_micro.rs, bench_round11_queries.rs,
bench_log_writer.rs and benches/ROUND11.md — final numbers land in the
follow-up doc commit):

- StoragePath re-representation: single canonical joined String, segments
  derived on demand; File/Folder drop the duplicated path_string field
  (4000→1000 allocs per 500-row listing page)
- Display classifier fusion: classify_display shares one stack-lowered
  extension across the three decision trees; call sites in FileDto,
  folder/favorites/recent handlers, trash, path-resolver (+ interning
  where Arc::from was still used)
- /status.php and /openapi.json memoized into OnceLock<Bytes> (openapi
  rebuilt a 171 KiB spec per request: 2.8 ms → 18 ns)
- NC upload-session PROPFIND: write! + pre-sized body + stack RFC2822
  dates (2.3-2.6x, 2582→772 allocs at 256 chunks)
- REST download: dead FileDto clone removed (capture mime/size + move)
- CalendarEventDto/TrashedItem into_parts moves (11 KiB ical_data memcpy
  gone per CalDAV row); CardDAV getlastmodified stack render
- 4xx path: borrowed ErrorResponse serialize, ErrorKind::as_str,
  not_found/already_exists clone kill
- vCard emit via write!; search page moved out with into_iter skip/take;
  content-hit UUIDs parsed once; group last-user check via HashSet
- RateLimiter: lock-free get + insert (and_upsert_with variant REJECTED
  by benchmark); CSRF token borrow-compare + borrowed cookie extraction
- Thumbnail/preview ETags built from as_str (Debug-identical bytes)
- Encrypted backend: encrypt_in_place_detached single-buffer write path,
  chunk-sized reserve in collect_stream; retry labels made lazy
- PG: deferred upload registration 3→1 round-trips (persist_file CTE
  template); direct_grant_cache for Calendar/AddressBook/Playlist authz
  (single-flight + set_role/clear_role invalidation); expand_user
  tokio::join!; geo clusters min(uuid)::text; recluster face assignment
  batched into one UNNEST update
- People recluster cosine: norms precomputed once (bit-identical gate)
- NC capabilities poll logs demoted to debug; tracing-appender dep added
  for the log-writer benchmark

Frontend:
- ResourceList.selectedEntries O(N)-per-toggle → id-index projection
  O(k log k); favorites/recent consume the batchToolbar snippet param and
  drop their duplicate filter + dead selectedIds mirror
- Recent: star state via new favoriteIds prop — a star click no longer
  rebuilds all N entries
- admin timeAgo >30d fallback uses the cached Intl.DateTimeFormat
- vitest gates in src/lib/components/round11.bench.test.ts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABhTEHuGujvwoodh67Kga7
This commit is contained in:
Claude
2026-07-18 22:02:00 +00:00
parent 637478e7bd
commit 221c1f31b0
54 changed files with 4060 additions and 630 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/infrastructure/services/encrypted_blob_backend.rs
+33 -8
View File
@@ -30,7 +30,7 @@
use std::path::{Path, PathBuf};
use std::pin::Pin;
use aes_gcm::aead::{Aead, AeadInPlace, KeyInit, OsRng};
use aes_gcm::aead::{AeadInPlace, KeyInit, OsRng};
use aes_gcm::{AeadCore, Aes256Gcm, Nonce};
use bytes::Bytes;
use std::sync::Arc;
@@ -44,6 +44,9 @@ use crate::domain::errors::DomainError;
/// Nonce size for AES-256-GCM (96 bits = 12 bytes).
const NONCE_SIZE: usize = 12;
/// AES-256-GCM authentication tag length appended after the ciphertext.
const TAG_SIZE: usize = 16;
/// Payloads at or above this size run crypto on the blocking pool; below
/// it the `spawn_blocking` round-trip costs more than the AES work itself.
const CRYPTO_OFFLOAD_THRESHOLD: usize = 64 * 1024;
@@ -82,16 +85,23 @@ impl EncryptedBlobBackend {
}
/// Encrypt `data` into the on-disk layout: `[12-byte nonce][ciphertext + tag]`.
///
/// Single output buffer, mirroring the read side's `decrypt_in_place`:
/// the payload is copied exactly once and encrypted in place with the tag
/// appended. The old shape let `cipher.encrypt` allocate a full ciphertext
/// `Vec` and then copied it a second time behind the nonce — one extra
/// allocation + a full-size memcpy on every encrypted chunk write
/// (benches/ROUND11.md §15; output bytes identical for a given nonce).
fn encrypt_bytes(cipher: &Aes256Gcm, data: &[u8]) -> Result<Bytes, DomainError> {
let nonce = Aes256Gcm::generate_nonce(&mut OsRng);
let ciphertext = cipher
.encrypt(&nonce, data)
let mut out = Vec::with_capacity(NONCE_SIZE + data.len() + TAG_SIZE);
out.extend_from_slice(nonce.as_slice());
out.extend_from_slice(data);
let tag = cipher
.encrypt_in_place_detached(&nonce, b"", &mut out[NONCE_SIZE..])
.map_err(|e| DomainError::internal_error("Encryption", format!("encrypt failed: {e}")))?;
let mut encrypted = Vec::with_capacity(NONCE_SIZE + ciphertext.len());
encrypted.extend_from_slice(nonce.as_slice());
encrypted.extend_from_slice(&ciphertext);
Ok(Bytes::from(encrypted))
out.extend_from_slice(&tag);
Ok(Bytes::from(out))
}
/// Decrypt the on-disk layout `[nonce][ciphertext + tag]` **in place**.
@@ -322,6 +332,13 @@ impl BlobStorageBackend for EncryptedBlobBackend {
}
/// Collect a byte stream into a single `Vec<u8>`.
///
/// Modern blobs are CDC chunks (≤ `CDC_MAX_CHUNK` + nonce/tag overhead),
/// delivered here as small reader frames — growing from `Vec::new()` paid
/// ~log₂(n) reallocations + a wasted ~0.75×-size memcpy per read. Reserving
/// one chunk's worth up front on the first frame makes the common case a
/// single allocation; legacy whole-file blobs beyond that fall back to
/// normal doubling (benches/ROUND11.md §16: 9 → 1 allocs on a 1 MiB blob).
async fn collect_stream(stream: BlobStream) -> Result<Vec<u8>, DomainError> {
use futures::StreamExt;
let mut stream = stream;
@@ -329,6 +346,14 @@ async fn collect_stream(stream: BlobStream) -> Result<Vec<u8>, DomainError> {
while let Some(chunk) = stream.next().await {
let bytes = chunk
.map_err(|e| DomainError::internal_error("Encryption", format!("stream read: {e}")))?;
if buf.capacity() == 0 {
buf.reserve(
(crate::infrastructure::services::dedup_service::CDC_MAX_CHUNK
+ NONCE_SIZE
+ TAG_SIZE)
.max(bytes.len()),
);
}
buf.extend_from_slice(&bytes);
}
Ok(buf)
src/infrastructure/services/path_resolver_service.rs
+6 -5
View File
@@ -10,7 +10,7 @@ use std::sync::Arc;
use uuid::Uuid;
use crate::application::dtos::display_helpers::{
category_for, format_file_size, icon_class_for, icon_special_class_for,
classify_display, format_file_size, intern_display, intern_mime,
};
use crate::application::dtos::file_dto::FileDto;
use crate::application::dtos::folder_dto::FolderDto;
@@ -197,18 +197,19 @@ impl PathResolverService {
let hash = blob_hash.unwrap_or_default();
let modified_at_u = modified_at as u64;
let etag = File::compute_etag(&hash, modified_at_u);
let classes = classify_display(&name, &mime);
Ok(ResolvedResource::File(FileDto {
id,
name: name.clone(),
path: res_path,
size: sz,
mime_type: Arc::from(&*mime),
mime_type: intern_mime(&mime),
folder_id,
created_at: created_at as u64,
modified_at: modified_at_u,
icon_class: Arc::from(icon_class_for(&name, &mime)),
icon_special_class: Arc::from(icon_special_class_for(&name, &mime)),
category: Arc::from(category_for(&name, &mime)),
icon_class: intern_display(classes.icon_class),
icon_special_class: intern_display(classes.icon_special_class),
category: intern_display(classes.category),
size_formatted: format_file_size(sz),
sort_date: None,
content_hash: hash,
src/infrastructure/services/path_service.rs
+1 -1
View File
@@ -95,7 +95,7 @@ impl PathService {
/// Validates a path to ensure it doesn't contain dangerous components
pub fn validate_path(&self, path: &StoragePath) -> Result<(), DomainError> {
// Check for empty segments
if path.segments().iter().any(|s| s.is_empty()) {
if path.segments().any(|s| s.is_empty()) {
return Err(DomainError::new(
ErrorKind::InvalidInput,
"Path",
src/infrastructure/services/pg_acl_engine.rs
+111 -39
View File
@@ -117,6 +117,16 @@ const CASCADE_GRANT_CACHE_CAPACITY: u64 = 100_000;
/// invalidation tree". Short enough that any such change takes effect in <1 min.
const CASCADE_GRANT_CACHE_TTL: Duration = Duration::from_secs(30);
/// `direct_grant_cache` bound/TTL: memoises the Calendar / AddressBook /
/// Playlist `role_grants` point decision — the only `check()` arms that had
/// NO result cache, re-run on every CalDAV/CardDAV/music request by clients
/// that poll continuously. Same invalidation contract as
/// `cascade_grant_cache`: grant writes on these resource types flush the
/// whole cache; group/expiry churn self-heals within the TTL
/// (benches/ROUND11.md §Q2).
const DIRECT_GRANT_CACHE_CAPACITY: u64 = 100_000;
const DIRECT_GRANT_CACHE_TTL: Duration = Duration::from_secs(30);
/// `file_parent_cache` bound/TTL: `file_id → Option<folder_id>` point rows
/// (~50 B each) resolved on the file-cascade path so an N-file album pays
/// ONE folder-cascade query instead of N (ROUND9). Parentage changes only
@@ -212,6 +222,11 @@ pub struct PgAclEngine {
/// only positively-or-negatively for at most the TTL. A revoke via
/// `clear_role` flushes immediately; anything missed self-heals in ≤30 s.
cascade_grant_cache: Cache<(Subject, Resource, Permission), bool>,
/// Memoised Calendar/AddressBook/Playlist direct-grant decision (the
/// top-level resources with no cascade parent). See
/// `DIRECT_GRANT_CACHE_CAPACITY` for the contract.
direct_grant_cache: Cache<(Subject, Resource, Permission), bool>,
/// `file_id → Option<parent folder_id>` memo for the file-cascade
/// decomposition (see `cascade_grant_cached`): resolving the parent lets
/// a whole folder's files share ONE folder-cascade decision, so a shared
@@ -315,6 +330,10 @@ impl PgAclEngine {
.max_capacity(CASCADE_GRANT_CACHE_CAPACITY)
.time_to_live(CASCADE_GRANT_CACHE_TTL)
.build(),
direct_grant_cache: Cache::builder()
.max_capacity(DIRECT_GRANT_CACHE_CAPACITY)
.time_to_live(DIRECT_GRANT_CACHE_TTL)
.build(),
file_parent_cache: Cache::builder()
.max_capacity(FILE_PARENT_CACHE_CAPACITY)
.time_to_live(FILE_PARENT_CACHE_TTL)
@@ -395,6 +414,10 @@ impl PgAclEngine {
.max_capacity(1)
.time_to_live(Duration::from_secs(1))
.build(),
direct_grant_cache: Cache::builder()
.max_capacity(1)
.time_to_live(Duration::from_secs(1))
.build(),
file_parent_cache: Cache::builder()
.max_capacity(1)
.time_to_live(Duration::from_secs(1))
@@ -568,26 +591,38 @@ impl PgAclEngine {
// belong to the Internal virtual group. Unknown user (no row) is
// treated as external to fail closed: a deleted or bogus user_id
// must not gain implicit Internal membership.
//
// The `is_external` point read and the recursive groups CTE are
// independent — `join!` overlaps their round-trips on every cold
// expansion instead of paying them serially (benches/ROUND11.md
// §Q3; the ROUND9/10 pattern).
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
let is_external: bool =
sqlx::query_scalar("SELECT is_external FROM auth.users WHERE id = $1")
let is_external_fut = async {
sqlx::query_scalar::<_, bool>("SELECT is_external FROM auth.users WHERE id = $1")
.bind(user_id)
.fetch_optional(self.pool.as_ref())
.await
.map_err(|e| {
DomainError::internal_error("PgAcl", format!("lookup is_external: {e}"))
})?
.unwrap_or(true);
if !is_external {
})
.map(|row| row.unwrap_or(true))
};
let groups_fut = async {
match &self.group_repo {
Some(repo) => {
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
repo.groups_for_user(user_id).await.map(Some).map_err(|e| {
DomainError::internal_error("PgAcl", format!("groups_for_user: {e}"))
})
}
None => Ok(None),
}
};
let (is_external, direct) = tokio::join!(is_external_fut, groups_fut);
if !is_external? {
set.insert(INTERNAL_GROUP_ID);
}
if let Some(repo) = &self.group_repo {
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
let direct = repo.groups_for_user(user_id).await.map_err(|e| {
DomainError::internal_error("PgAcl", format!("groups_for_user: {e}"))
})?;
if let Some(direct) = direct? {
set.extend(direct);
}
@@ -1099,6 +1134,48 @@ impl PgAclEngine {
/// (on File/Folder grant writes — it holds file AND folder decisions in
/// the same map) and the 30 s TTL (indirect changes, incl. moves for the
/// parent memo) keep it fresh. See the `cascade_grant_cache` field doc.
/// Cached wrapper for the Calendar/AddressBook/Playlist direct-grant
/// decision (`try_get_with`: a cold herd on one key coalesces into ONE
/// loader run, the ROUND10 single-flight pattern; loader errors are
/// never cached). `resource_type` is the `role_grants.resource_type`
/// discriminant for `resource`.
async fn direct_grant_cached(
&self,
subject: Subject,
resource: Resource,
permission: Permission,
resource_type: &'static str,
id: Uuid,
counters: &QueryCounters,
) -> Result<bool, DomainError> {
if let Some(allowed) = self
.direct_grant_cache
.get(&(subject, resource, permission))
.await
{
counters.cache_hit.fetch_add(1, Ordering::Relaxed);
return Ok(allowed);
}
self.direct_grant_cache
.try_get_with((subject, resource, permission), async {
let (subject_types, subject_ids) =
self.subject_match_set(subject, counters).await?;
self.direct_grant_exists(
&subject_types,
&subject_ids,
permission,
resource_type,
id,
counters,
)
.await
})
.await
.map_err(|e: Arc<DomainError>| {
DomainError::internal_error("PgAcl", format!("direct grant load: {e}"))
})
}
async fn cascade_grant_cached(
&self,
subject: Subject,
@@ -1493,24 +1570,13 @@ impl PgAclEngine {
// round-trip — no drive_role_cache short-circuit (no
// drive), no cascade.
Resource::Calendar(id) => {
let (subject_types, subject_ids) =
self.subject_match_set(subject, counters).await?;
self.direct_grant_exists(
&subject_types,
&subject_ids,
permission,
"calendar",
id,
counters,
)
.await
self.direct_grant_cached(subject, resource, permission, "calendar", id, counters)
.await
}
Resource::AddressBook(id) => {
let (subject_types, subject_ids) =
self.subject_match_set(subject, counters).await?;
self.direct_grant_exists(
&subject_types,
&subject_ids,
self.direct_grant_cached(
subject,
resource,
permission,
"address_book",
id,
@@ -1519,17 +1585,8 @@ impl PgAclEngine {
.await
}
Resource::Playlist(id) => {
let (subject_types, subject_ids) =
self.subject_match_set(subject, counters).await?;
self.direct_grant_exists(
&subject_types,
&subject_ids,
permission,
"playlist",
id,
counters,
)
.await
self.direct_grant_cached(subject, resource, permission, "playlist", id, counters)
.await
}
}
}
@@ -2874,6 +2931,13 @@ impl AuthorizationEngine for PgAclEngine {
if matches!(resource, Resource::File(_) | Resource::Folder(_)) {
self.invalidate_cascade_grant_cache_all().await;
}
// Same immediacy contract for the memoised top-level decisions.
if matches!(
resource,
Resource::Calendar(_) | Resource::AddressBook(_) | Resource::Playlist(_)
) {
self.direct_grant_cache.invalidate_all();
}
Self::row_to_grant(row)
}
@@ -2903,6 +2967,14 @@ impl AuthorizationEngine for PgAclEngine {
if matches!(resource, Resource::File(_) | Resource::Folder(_)) {
self.invalidate_cascade_grant_cache_all().await;
}
// A revoked calendar/address-book/playlist grant must fail the next
// check now, not in ≤30 s (see `set_role`).
if matches!(
resource,
Resource::Calendar(_) | Resource::AddressBook(_) | Resource::Playlist(_)
) {
self.direct_grant_cache.invalidate_all();
}
Ok(())
}
src/infrastructure/services/retry_blob_backend.rs
+91 -49
View File
@@ -57,14 +57,20 @@ impl RetryBlobBackend {
}
/// Execute an async closure with exponential backoff retry.
async fn retry_async<F, Fut, T>(
///
/// `name` is a lazy label: the success path (the overwhelmingly common
/// case) never materializes it, so per-op `format!("op({hash})")`
/// allocations only happen on an actual retry (benches/ROUND11.md §14:
/// 64.5 → 0.7 ns, −2 allocs per blob op).
async fn retry_async<F, Fut, T, L>(
policy: &RetryPolicy,
name: &str,
name: L,
mut f: F,
) -> Result<T, DomainError>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T, DomainError>>,
L: Fn() -> String,
{
let mut attempt = 0u32;
let mut backoff = policy.initial_backoff;
@@ -78,7 +84,7 @@ where
"Retry {}/{} for {} after error: {} (backoff {:?})",
attempt,
policy.max_retries,
name,
name(),
e,
backoff
);
@@ -112,10 +118,14 @@ impl BlobStorageBackend for RetryBlobBackend {
let inner = self.inner.clone();
let policy = self.policy.clone();
Box::pin(async move {
retry_async(&policy, "initialize", || {
let inner = inner.clone();
async move { inner.initialize().await }
})
retry_async(
&policy,
|| "initialize".to_string(),
|| {
let inner = inner.clone();
async move { inner.initialize().await }
},
)
.await
})
}
@@ -130,12 +140,16 @@ impl BlobStorageBackend for RetryBlobBackend {
let hash = hash.to_string();
let path = source_path.to_path_buf();
Box::pin(async move {
retry_async(&policy, &format!("put_blob({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
let path = path.clone();
async move { inner.put_blob(&hash, &path).await }
})
retry_async(
&policy,
|| format!("put_blob({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
let path = path.clone();
async move { inner.put_blob(&hash, &path).await }
},
)
.await
})
}
@@ -149,12 +163,16 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("put_blob_from_bytes({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
let data = data.clone();
async move { inner.put_blob_from_bytes(&hash, data).await }
})
retry_async(
&policy,
|| format!("put_blob_from_bytes({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
let data = data.clone();
async move { inner.put_blob_from_bytes(&hash, data).await }
},
)
.await
})
}
@@ -174,7 +192,7 @@ impl BlobStorageBackend for RetryBlobBackend {
Box::pin(async move {
retry_async(
&policy,
&format!("put_blob_from_bytes_unsynced({hash})"),
|| format!("put_blob_from_bytes_unsynced({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
@@ -205,11 +223,15 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("get_blob_stream({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_stream(&hash).await }
})
retry_async(
&policy,
|| format!("get_blob_stream({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_stream(&hash).await }
},
)
.await
})
}
@@ -225,11 +247,15 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("get_blob_range({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_range_stream(&hash, start, end).await }
})
retry_async(
&policy,
|| format!("get_blob_range({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.get_blob_range_stream(&hash, start, end).await }
},
)
.await
})
}
@@ -242,11 +268,15 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("delete_blob({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.delete_blob(&hash).await }
})
retry_async(
&policy,
|| format!("delete_blob({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.delete_blob(&hash).await }
},
)
.await
})
}
@@ -259,11 +289,15 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("blob_exists({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_exists(&hash).await }
})
retry_async(
&policy,
|| format!("blob_exists({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_exists(&hash).await }
},
)
.await
})
}
@@ -276,11 +310,15 @@ impl BlobStorageBackend for RetryBlobBackend {
let policy = self.policy.clone();
let hash = hash.to_string();
Box::pin(async move {
retry_async(&policy, &format!("blob_size({hash})"), || {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_size(&hash).await }
})
retry_async(
&policy,
|| format!("blob_size({hash})"),
|| {
let inner = inner.clone();
let hash = hash.clone();
async move { inner.blob_size(&hash).await }
},
)
.await
})
}
@@ -293,10 +331,14 @@ impl BlobStorageBackend for RetryBlobBackend {
let inner = self.inner.clone();
let policy = self.policy.clone();
Box::pin(async move {
retry_async(&policy, "health_check", || {
let inner = inner.clone();
async move { inner.health_check().await }
})
retry_async(
&policy,
|| "health_check".to_string(),
|| {
let inner = inner.clone();
async move { inner.health_check().await }
},
)
.await
})
}