perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
@@ -0,0 +1,209 @@
|
||||
//! Basic-auth thundering-herd benchmark — K concurrent cache misses.
|
||||
//!
|
||||
//! Every WebDAV/CalDAV/CardDAV/NextCloud request authenticates through
|
||||
//! `AppPasswordService::verify_basic_auth`. The cache (TTL 300 s) used to be
|
||||
//! a plain get/insert: when a sync client holding K parallel connections hit
|
||||
//! an expired entry, all K in-flight requests missed simultaneously and each
|
||||
//! ran the full slow path — an Argon2id verification at ~64 MiB / t=3 / p=2
|
||||
//! apiece (100-300 ms CPU each). `try_get_with` now coalesces concurrent
|
||||
//! misses into ONE verification; failed verifications stay uncached.
|
||||
//!
|
||||
//! Sections:
|
||||
//! BEFORE (emulated) — K concurrent bare Argon2id verifications, the exact
|
||||
//! work the old code fanned out per herd
|
||||
//! AFTER — K concurrent verify_basic_auth on a cold cache
|
||||
//! (single-flight: 1 verification, K-1 waiters)
|
||||
//! warm-hit — p50 of the cached path
|
||||
//!
|
||||
//! Gate: AFTER's process-CPU delta must be ~1 verification (< 2x a single
|
||||
//! verify), while BEFORE burns ~K of them. All K results must be Ok and
|
||||
//! identical.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL / OXICLOUD_DB_CONNECTION_STRING
|
||||
//! from .env):
|
||||
//! cargo run --release --features bench --example bench_auth_herd
|
||||
//! Tunables: BENCH_HERD (8)
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use oxicloud::application::services::app_password_service::AppPasswordService;
|
||||
use oxicloud::infrastructure::repositories::pg::{AppPasswordPgRepository, UserPgRepository};
|
||||
use oxicloud::infrastructure::services::password_hasher::Argon2PasswordHasher;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Process CPU time (utime + stime) in seconds, from /proc/self/stat.
|
||||
fn cpu_seconds() -> f64 {
|
||||
let stat = std::fs::read_to_string("/proc/self/stat").expect("stat");
|
||||
// utime/stime are fields 14/15 (1-indexed) — index past the comm field
|
||||
// (it can contain spaces) via the closing paren.
|
||||
let rest = &stat[stat.rfind(')').unwrap() + 2..];
|
||||
let fields: Vec<&str> = rest.split_whitespace().collect();
|
||||
let utime: f64 = fields[11].parse().expect("utime");
|
||||
let stime: f64 = fields[12].parse().expect("stime");
|
||||
let hz = 100.0; // USER_HZ on all mainstream Linux configs
|
||||
(utime + stime) / hz
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL");
|
||||
let herd: usize = env_or("BENCH_HERD", 8);
|
||||
|
||||
let pool = Arc::new(
|
||||
PgPoolOptions::new()
|
||||
.max_connections(10)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect"),
|
||||
);
|
||||
|
||||
// ── Seed: user + NC-format app password (production Argon2 params) ──
|
||||
let username = format!("bench_herd_{}", std::process::id());
|
||||
let user_id: uuid::Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, password_hash, role)
|
||||
VALUES ($1, $2, '', 'user') RETURNING id",
|
||||
)
|
||||
.bind(&username)
|
||||
.bind(format!("{username}@bench.invalid"))
|
||||
.fetch_one(pool.as_ref())
|
||||
.await
|
||||
.expect("seed user");
|
||||
|
||||
// Production defaults: m=64 MiB, t=3, p=2 (config.rs auth defaults).
|
||||
let hasher = Arc::new(Argon2PasswordHasher::new(65536, 3, 2));
|
||||
let svc = Arc::new(AppPasswordService::new(
|
||||
Arc::new(AppPasswordPgRepository::new(pool.clone())),
|
||||
hasher.clone(),
|
||||
Arc::new(UserPgRepository::new(pool.clone())),
|
||||
"http://localhost".into(),
|
||||
));
|
||||
let (_ap_id, plain) = svc.create_nc(user_id, "bench").await.expect("create_nc");
|
||||
|
||||
// ── Single-verify baseline (what one Argon2id run costs here) ──────
|
||||
use oxicloud::application::ports::auth_ports::PasswordHasherPort;
|
||||
let ref_hash = hasher.hash_password("benchpw").await.expect("hash");
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
assert!(
|
||||
hasher
|
||||
.verify_password("benchpw", &ref_hash)
|
||||
.await
|
||||
.expect("verify")
|
||||
);
|
||||
let one_wall = t.elapsed().as_secs_f64();
|
||||
let one_cpu = cpu_seconds() - c;
|
||||
println!(
|
||||
"single Argon2id verify: {:.0} ms wall, {:.0} ms CPU",
|
||||
one_wall * 1000.0,
|
||||
one_cpu * 1000.0
|
||||
);
|
||||
|
||||
// ── BEFORE (emulated): K concurrent bare verifications ─────────────
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..herd {
|
||||
let h = hasher.clone();
|
||||
let rh = ref_hash.clone();
|
||||
set.spawn(async move { h.verify_password("benchpw", &rh).await.expect("verify") });
|
||||
}
|
||||
while let Some(r) = set.join_next().await {
|
||||
assert!(r.expect("join"));
|
||||
}
|
||||
let before_wall = t.elapsed().as_secs_f64();
|
||||
let before_cpu = cpu_seconds() - c;
|
||||
|
||||
// ── AFTER: K concurrent verify_basic_auth on a cold cache ──────────
|
||||
let t = Instant::now();
|
||||
let c = cpu_seconds();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..herd {
|
||||
let s = svc.clone();
|
||||
let u = username.clone();
|
||||
let p = plain.clone();
|
||||
set.spawn(async move { s.verify_basic_auth(&u, &p).await });
|
||||
}
|
||||
let mut ids = Vec::new();
|
||||
while let Some(r) = set.join_next().await {
|
||||
let (uid, uname, _, _) = r.expect("join").expect("verify_basic_auth");
|
||||
assert_eq!(uname, username);
|
||||
ids.push(uid);
|
||||
}
|
||||
assert!(ids.iter().all(|&u| u == user_id));
|
||||
let after_wall = t.elapsed().as_secs_f64();
|
||||
let after_cpu = cpu_seconds() - c;
|
||||
|
||||
// ── Warm hit p50 ────────────────────────────────────────────────────
|
||||
let mut lat = Vec::with_capacity(10_000);
|
||||
for _ in 0..10_000 {
|
||||
let t = Instant::now();
|
||||
let _ = svc
|
||||
.verify_basic_auth(&username, &plain)
|
||||
.await
|
||||
.expect("warm hit");
|
||||
lat.push(t.elapsed().as_secs_f64() * 1e6);
|
||||
}
|
||||
lat.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
let warm_p50 = lat[lat.len() / 2];
|
||||
|
||||
println!("\n# herd of {herd} concurrent Basic Auth verifications, cold cache");
|
||||
println!(
|
||||
"{:<22} {:>10} {:>10} {:>14}",
|
||||
"variant", "wall ms", "CPU ms", "verifications"
|
||||
);
|
||||
println!(
|
||||
"{:<22} {:>10.0} {:>10.0} {:>14.1}",
|
||||
"BEFORE (per-caller)",
|
||||
before_wall * 1000.0,
|
||||
before_cpu * 1000.0,
|
||||
before_cpu / one_cpu
|
||||
);
|
||||
println!(
|
||||
"{:<22} {:>10.0} {:>10.0} {:>14.1}",
|
||||
"AFTER (single-flight)",
|
||||
after_wall * 1000.0,
|
||||
after_cpu * 1000.0,
|
||||
after_cpu / one_cpu
|
||||
);
|
||||
println!("warm cache hit p50: {warm_p50:.1} us");
|
||||
|
||||
// ── Cleanup ─────────────────────────────────────────────────────────
|
||||
let _ = sqlx::query("DELETE FROM auth.app_passwords WHERE user_id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool.as_ref())
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool.as_ref())
|
||||
.await;
|
||||
|
||||
// ── Gate ────────────────────────────────────────────────────────────
|
||||
// AFTER must coalesce to ~1 verification's CPU; 2x headroom for
|
||||
// scheduler noise. BEFORE must show the herd actually fanned out.
|
||||
if after_cpu > one_cpu * 2.0 {
|
||||
eprintln!(
|
||||
"GATE FAIL: single-flight AFTER burned {:.1} verifications of CPU (expected ~1)",
|
||||
after_cpu / one_cpu
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if before_cpu < one_cpu * (herd as f64) * 0.6 {
|
||||
eprintln!(
|
||||
"GATE WARN: BEFORE emulation did not saturate ({:.1} verifs)",
|
||||
before_cpu / one_cpu
|
||||
);
|
||||
}
|
||||
println!("\nGATE PASS: cold-cache herd coalesced to ~1 Argon2id run");
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
//! CachedBlobBackend miss-stampede benchmark — duplicate remote fetches.
|
||||
//!
|
||||
//! K concurrent cold readers of ONE blob (a video player's parallel Range
|
||||
//! probes on an uncached file, N sync clients pulling the same new file)
|
||||
//! used to each download the FULL blob from the remote backend and race
|
||||
//! their writes on one shared deterministic `.tmp` path. The per-hash
|
||||
//! single-flight gate coalesces them onto one download; waiters serve the
|
||||
//! leader's cached file.
|
||||
//!
|
||||
//! The mock inner backend counts `get_blob_stream` calls and serves a
|
||||
//! 32 MiB blob with an injected 15 ms first-byte latency + paced chunks
|
||||
//! (models a remote object store).
|
||||
//!
|
||||
//! BEFORE (emulated) — K concurrent direct inner fetches, each draining
|
||||
//! the full stream (what the old miss path did)
|
||||
//! AFTER — K concurrent `CachedBlobBackend::get_blob_stream`
|
||||
//! on a cold cache
|
||||
//!
|
||||
//! Gates: AFTER's inner-fetch count == 1; the cached file must BLAKE3-match
|
||||
//! the source; K x full-drain wall reported for both.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_blob_cache
|
||||
//! Tunables: BENCH_CONCURRENCY (16), BENCH_BLOB_MB (32)
|
||||
|
||||
use std::env;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::Bytes;
|
||||
use futures::StreamExt;
|
||||
use oxicloud::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
};
|
||||
use oxicloud::domain::errors::DomainError;
|
||||
use oxicloud::infrastructure::services::cached_blob_backend::{BlobCacheConfig, CachedBlobBackend};
|
||||
|
||||
type BoxFut<'a, T> = std::pin::Pin<Box<dyn std::future::Future<Output = T> + Send + 'a>>;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Mock remote backend: one in-RAM blob, counted reads, and — crucially —
|
||||
/// SHARED aggregate bandwidth: concurrent streams split one simulated
|
||||
/// 1 GiB/s link (a real NIC/egress link doesn't hand every duplicate
|
||||
/// download its own private lane, so duplicate fetches cost real wall
|
||||
/// time, not just bytes).
|
||||
struct MockRemote {
|
||||
data: Bytes,
|
||||
fetches: AtomicU64,
|
||||
bytes_served: AtomicU64,
|
||||
/// Virtual time (µs since bench start) when the shared link frees up.
|
||||
link_busy_until_us: Arc<tokio::sync::Mutex<u64>>,
|
||||
epoch: Instant,
|
||||
}
|
||||
|
||||
const LINK_BYTES_PER_SEC: u64 = 1024 * 1024 * 1024; // 1 GiB/s aggregate
|
||||
|
||||
impl MockRemote {
|
||||
fn new(data: Bytes) -> Self {
|
||||
Self {
|
||||
data,
|
||||
fetches: AtomicU64::new(0),
|
||||
bytes_served: AtomicU64::new(0),
|
||||
link_busy_until_us: Arc::new(tokio::sync::Mutex::new(0)),
|
||||
epoch: Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
fn stream(&self) -> BlobStream {
|
||||
self.fetches.fetch_add(1, Ordering::Relaxed);
|
||||
self.bytes_served
|
||||
.fetch_add(self.data.len() as u64, Ordering::Relaxed);
|
||||
let data = self.data.clone();
|
||||
let link = self.link_busy_until_us.clone();
|
||||
let epoch = self.epoch;
|
||||
let s = async_stream::stream! {
|
||||
// First-byte latency of a remote GET.
|
||||
tokio::time::sleep(Duration::from_millis(15)).await;
|
||||
let chunk = 4 * 1024 * 1024;
|
||||
let mut off = 0usize;
|
||||
while off < data.len() {
|
||||
let end = (off + chunk).min(data.len());
|
||||
// Reserve this chunk's slot on the shared link, then sleep
|
||||
// until the slot has elapsed — bandwidth divides across
|
||||
// every in-flight stream.
|
||||
let slot_us = (end - off) as u64 * 1_000_000 / LINK_BYTES_PER_SEC;
|
||||
let wake_us = {
|
||||
let mut busy = link.lock().await;
|
||||
let now_us = epoch.elapsed().as_micros() as u64;
|
||||
let start = (*busy).max(now_us);
|
||||
*busy = start + slot_us;
|
||||
*busy
|
||||
};
|
||||
let now_us = epoch.elapsed().as_micros() as u64;
|
||||
if wake_us > now_us {
|
||||
tokio::time::sleep(Duration::from_micros(wake_us - now_us)).await;
|
||||
}
|
||||
yield Ok::<Bytes, std::io::Error>(data.slice(off..end));
|
||||
off = end;
|
||||
}
|
||||
};
|
||||
Box::pin(s)
|
||||
}
|
||||
}
|
||||
|
||||
impl BlobStorageBackend for MockRemote {
|
||||
fn initialize(&self) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
fn put_blob(&self, _hash: &str, _source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
Box::pin(async { Ok(0) })
|
||||
}
|
||||
fn put_blob_from_bytes(
|
||||
&self,
|
||||
_hash: &str,
|
||||
data: Bytes,
|
||||
) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
Box::pin(async move { Ok(data.len() as u64) })
|
||||
}
|
||||
fn get_blob_stream(&self, _hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let s = self.stream();
|
||||
Box::pin(async move { Ok(s) })
|
||||
}
|
||||
fn get_blob_range_stream(
|
||||
&self,
|
||||
_hash: &str,
|
||||
start: u64,
|
||||
end: Option<u64>,
|
||||
) -> BoxFut<'_, Result<BlobStream, DomainError>> {
|
||||
let data = self.data.clone();
|
||||
self.fetches.fetch_add(1, Ordering::Relaxed);
|
||||
Box::pin(async move {
|
||||
let end = end.unwrap_or(data.len() as u64).min(data.len() as u64);
|
||||
let s = futures::stream::once(async move {
|
||||
Ok::<Bytes, std::io::Error>(data.slice(start as usize..end as usize))
|
||||
});
|
||||
Ok(Box::pin(s) as BlobStream)
|
||||
})
|
||||
}
|
||||
fn delete_blob(&self, _hash: &str) -> BoxFut<'_, Result<(), DomainError>> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
fn blob_exists(&self, _hash: &str) -> BoxFut<'_, Result<bool, DomainError>> {
|
||||
Box::pin(async { Ok(true) })
|
||||
}
|
||||
fn blob_size(&self, _hash: &str) -> BoxFut<'_, Result<u64, DomainError>> {
|
||||
let n = self.data.len() as u64;
|
||||
Box::pin(async move { Ok(n) })
|
||||
}
|
||||
fn health_check(&self) -> BoxFut<'_, Result<StorageHealthStatus, DomainError>> {
|
||||
Box::pin(async {
|
||||
Ok(StorageHealthStatus {
|
||||
connected: true,
|
||||
backend_type: "mock".into(),
|
||||
message: "ok".into(),
|
||||
available_bytes: None,
|
||||
})
|
||||
})
|
||||
}
|
||||
fn backend_type(&self) -> &'static str {
|
||||
"mock"
|
||||
}
|
||||
fn local_blob_path(&self, _hash: &str) -> Option<PathBuf> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
async fn drain(mut s: BlobStream) -> (u64, [u8; 32]) {
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut n = 0u64;
|
||||
while let Some(chunk) = s.next().await {
|
||||
let b = chunk.expect("chunk");
|
||||
n += b.len() as u64;
|
||||
hasher.update(&b);
|
||||
}
|
||||
(n, hasher.finalize().into())
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let k: usize = env_or("BENCH_CONCURRENCY", 16);
|
||||
let blob_mb: usize = env_or("BENCH_BLOB_MB", 32);
|
||||
|
||||
let data: Bytes = (0..blob_mb * 1024 * 1024)
|
||||
.map(|i| (i * 37 % 249) as u8)
|
||||
.collect::<Vec<u8>>()
|
||||
.into();
|
||||
let ref_hash: [u8; 32] = blake3::hash(&data).into();
|
||||
let blob_len = data.len() as u64;
|
||||
let hash = "benchblobcache00000000000000000000000000000000000000000000000000";
|
||||
|
||||
// ── BEFORE (emulated): K concurrent direct inner fetches ───────────
|
||||
let remote = Arc::new(MockRemote::new(data.clone()));
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..k {
|
||||
let r = remote.clone();
|
||||
set.spawn(async move {
|
||||
let s = r.get_blob_stream(hash).await.expect("stream");
|
||||
drain(s).await
|
||||
});
|
||||
}
|
||||
while let Some(res) = set.join_next().await {
|
||||
let (n, h) = res.expect("join");
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash);
|
||||
}
|
||||
let before_wall = t.elapsed().as_secs_f64() * 1000.0;
|
||||
let before_fetches = remote.fetches.load(Ordering::Relaxed);
|
||||
let before_mb = remote.bytes_served.load(Ordering::Relaxed) / (1024 * 1024);
|
||||
|
||||
// ── AFTER: K concurrent CachedBlobBackend reads, cold cache ────────
|
||||
let remote = Arc::new(MockRemote::new(data.clone()));
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let cached = Arc::new(CachedBlobBackend::new(
|
||||
remote.clone(),
|
||||
&BlobCacheConfig {
|
||||
cache_dir: dir.path().to_path_buf(),
|
||||
max_cache_bytes: 1 << 30,
|
||||
},
|
||||
));
|
||||
cached.initialize().await.expect("init");
|
||||
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for _ in 0..k {
|
||||
let c = cached.clone();
|
||||
set.spawn(async move {
|
||||
let s = c.get_blob_stream(hash).await.expect("stream");
|
||||
drain(s).await
|
||||
});
|
||||
}
|
||||
while let Some(res) = set.join_next().await {
|
||||
let (n, h) = res.expect("join");
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash, "cached read corrupted");
|
||||
}
|
||||
let after_wall = t.elapsed().as_secs_f64() * 1000.0;
|
||||
let after_fetches = remote.fetches.load(Ordering::Relaxed);
|
||||
let after_mb = remote.bytes_served.load(Ordering::Relaxed) / (1024 * 1024);
|
||||
|
||||
// Integrity of the durable cache file itself.
|
||||
let (n, h) = drain(cached.get_blob_stream(hash).await.expect("warm")).await;
|
||||
assert_eq!(n, blob_len);
|
||||
assert_eq!(h, ref_hash, "durable cache file corrupted");
|
||||
let warm_fetches = remote.fetches.load(Ordering::Relaxed) - after_fetches;
|
||||
|
||||
println!("# {k} concurrent cold readers of one {blob_mb} MiB blob (remote: 15 ms TTFB, paced)");
|
||||
println!(
|
||||
"{:<24} {:>10} {:>14} {:>12}",
|
||||
"variant", "wall ms", "inner fetches", "remote MiB"
|
||||
);
|
||||
println!(
|
||||
"{:<24} {:>10.0} {:>14} {:>12}",
|
||||
"BEFORE (per-caller)", before_wall, before_fetches, before_mb
|
||||
);
|
||||
println!(
|
||||
"{:<24} {:>10.0} {:>14} {:>12}",
|
||||
"AFTER (single-flight)", after_wall, after_fetches, after_mb
|
||||
);
|
||||
|
||||
// ── Gates ───────────────────────────────────────────────────────────
|
||||
if after_fetches != 1 {
|
||||
eprintln!("GATE FAIL: expected exactly 1 coalesced remote fetch, got {after_fetches}");
|
||||
std::process::exit(1);
|
||||
}
|
||||
if warm_fetches != 0 {
|
||||
eprintln!("GATE FAIL: warm read hit the remote backend");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!("\nGATE PASS: {before_fetches} remote fetches -> 1, cache file verified");
|
||||
}
|
||||
@@ -0,0 +1,531 @@
|
||||
//! CardDAV REPORT generation benchmark — dead double vCard generation +
|
||||
//! O(N²) uid scan (BEFORE) vs single on-demand generation (AFTER).
|
||||
//!
|
||||
//! The old `handle_report` flow pre-generated a vCard for EVERY contact into a
|
||||
//! `Vec<(uid, vcard)>`, then `generate_contacts_response` did a linear
|
||||
//! `find(|(uid, _)| *uid == contact.uid)` per contact — O(N²) string compares
|
||||
//! — and *discarded* the result (`let _ = vcard`), because
|
||||
//! `write_contact_response` regenerates the vCard on demand anyway. The fix
|
||||
//! deletes the pre-generation and the scan, and converts `contact_to_vcard`
|
||||
//! from `push_str(&format!(…))` (one temp String per line) to
|
||||
//! `write!(&mut String, …)`.
|
||||
//!
|
||||
//! `mod before` below is a verbatim copy of the OLD code (old
|
||||
//! `contact_to_vcard`, old `generate_contacts_response` with the `vcards`
|
||||
//! parameter, and the then-current `write_contact_response`), so one binary
|
||||
//! measures both variants and byte-compares their output.
|
||||
//!
|
||||
//! Equivalence gate: BEFORE and AFTER XML must be byte-identical for every
|
||||
//! (N, prop-set) combination, and the old/new `contact_to_vcard` must agree
|
||||
//! byte-for-byte on every synthetic contact. Any mismatch exits 1 with the
|
||||
//! first differing offset.
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_carddav_report
|
||||
//! Tunables (env):
|
||||
//! BENCH_REPS (5) median reported
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use chrono::{NaiveDate, TimeZone, Utc};
|
||||
use oxicloud::application::adapters::carddav_adapter::{
|
||||
CardDavAdapter, CardDavReportType, contact_to_vcard,
|
||||
};
|
||||
use oxicloud::application::adapters::webdav_adapter::QualifiedName;
|
||||
use oxicloud::application::dtos::contact_dto::{AddressDto, ContactDto, EmailDto, PhoneDto};
|
||||
|
||||
/// Verbatim copy of the pre-fix production code (handler + adapter side),
|
||||
/// kept here so the benchmark measures the real OLD flow, not a caricature.
|
||||
mod before {
|
||||
use std::io::Write;
|
||||
|
||||
use oxicloud::application::adapters::carddav_adapter::CardDavReportType;
|
||||
use oxicloud::application::adapters::webdav_adapter::QualifiedName;
|
||||
use oxicloud::application::dtos::contact_dto::ContactDto;
|
||||
use quick_xml::Writer;
|
||||
use quick_xml::events::{BytesEnd, BytesStart, BytesText, Event};
|
||||
|
||||
/// OLD `generate_contacts_response` — takes the pre-generated `vcards`,
|
||||
/// does the O(N²) linear uid scan per contact, then throws the hit away.
|
||||
pub fn generate_contacts_response<W: Write>(
|
||||
writer: W,
|
||||
contacts: &[ContactDto],
|
||||
vcards: &[(String, String)], // (uid, vcard_data)
|
||||
report: &CardDavReportType,
|
||||
base_href: &str,
|
||||
) -> std::io::Result<()> {
|
||||
let mut xml_writer = Writer::new(writer);
|
||||
|
||||
xml_writer.write_event(Event::Start(
|
||||
BytesStart::new("D:multistatus").with_attributes([
|
||||
("xmlns:D", "DAV:"),
|
||||
("xmlns:CR", "urn:ietf:params:xml:ns:carddav"),
|
||||
]),
|
||||
))?;
|
||||
|
||||
let props = match report {
|
||||
CardDavReportType::AddressbookQuery { props } => props.clone(),
|
||||
CardDavReportType::AddressbookMultiget { props, .. } => props.clone(),
|
||||
CardDavReportType::SyncCollection { props, .. } => props.clone(),
|
||||
};
|
||||
|
||||
for contact in contacts {
|
||||
let href = format!("{}{}.vcf", base_href, contact.uid);
|
||||
let vcard = vcards
|
||||
.iter()
|
||||
.find(|(uid, _)| *uid == contact.uid)
|
||||
.map(|(_, data)| data.as_str())
|
||||
.unwrap_or("");
|
||||
write_contact_response(&mut xml_writer, contact, &props, &href)?;
|
||||
// If address-data is requested, include vcard
|
||||
if props.iter().any(|p| p.name == "address-data") || props.is_empty() {
|
||||
// Already handled in write_contact_response
|
||||
}
|
||||
let _ = vcard; // suppress warning - used via contact_to_vcard fallback
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:multistatus")))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy of the (unchanged) private `write_contact_response`, wired to the
|
||||
/// OLD `contact_to_vcard` so the BEFORE variant is fully self-contained.
|
||||
fn write_contact_response<W: Write>(
|
||||
xml_writer: &mut Writer<W>,
|
||||
contact: &ContactDto,
|
||||
props: &[QualifiedName],
|
||||
href: &str,
|
||||
) -> std::io::Result<()> {
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:response")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:href")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(href)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:href")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:propstat")))?;
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:prop")))?;
|
||||
|
||||
if props.is_empty() {
|
||||
// Return standard properties
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new("D:resourcetype")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getetag")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&format!(
|
||||
"\"{}\"",
|
||||
contact.etag
|
||||
))))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getetag")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getcontenttype")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new("text/vcard; charset=utf-8")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getcontenttype")))?;
|
||||
|
||||
// Include vCard data
|
||||
let vcard = contact_to_vcard(contact);
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("CR:address-data")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&vcard)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("CR:address-data")))?;
|
||||
} else {
|
||||
for prop in props {
|
||||
match (prop.namespace.as_str(), prop.name.as_str()) {
|
||||
("DAV:", "resourcetype") => {
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new("D:resourcetype")))?;
|
||||
}
|
||||
("DAV:", "getetag") => {
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:getetag")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&format!(
|
||||
"\"{}\"",
|
||||
contact.etag
|
||||
))))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getetag")))?;
|
||||
}
|
||||
("DAV:", "getcontenttype") => {
|
||||
xml_writer
|
||||
.write_event(Event::Start(BytesStart::new("D:getcontenttype")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(
|
||||
"text/vcard; charset=utf-8",
|
||||
)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getcontenttype")))?;
|
||||
}
|
||||
("DAV:", "getlastmodified") => {
|
||||
xml_writer
|
||||
.write_event(Event::Start(BytesStart::new("D:getlastmodified")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(
|
||||
&contact.updated_at.to_rfc2822(),
|
||||
)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:getlastmodified")))?;
|
||||
}
|
||||
("urn:ietf:params:xml:ns:carddav", "address-data") => {
|
||||
let vcard = contact_to_vcard(contact);
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("CR:address-data")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new(&vcard)))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("CR:address-data")))?;
|
||||
}
|
||||
_ => {
|
||||
let prop_name = if prop.namespace == "urn:ietf:params:xml:ns:carddav" {
|
||||
format!("CR:{}", prop.name)
|
||||
} else if prop.namespace == "DAV:" {
|
||||
format!("D:{}", prop.name)
|
||||
} else {
|
||||
prop.name.clone()
|
||||
};
|
||||
xml_writer.write_event(Event::Empty(BytesStart::new(&prop_name)))?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:prop")))?;
|
||||
|
||||
xml_writer.write_event(Event::Start(BytesStart::new("D:status")))?;
|
||||
xml_writer.write_event(Event::Text(BytesText::new("HTTP/1.1 200 OK")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:status")))?;
|
||||
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:propstat")))?;
|
||||
xml_writer.write_event(Event::End(BytesEnd::new("D:response")))?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// OLD `contact_to_vcard` — one `push_str(&format!(…))` temp String per line.
|
||||
pub fn contact_to_vcard(contact: &ContactDto) -> String {
|
||||
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
|
||||
|
||||
vcard.push_str(&format!("UID:{}\r\n", contact.uid));
|
||||
|
||||
if let (Some(last), Some(first)) = (&contact.last_name, &contact.first_name) {
|
||||
vcard.push_str(&format!("N:{};{};;;\r\n", last, first));
|
||||
} else if let Some(last) = &contact.last_name {
|
||||
vcard.push_str(&format!("N:{};;;;\r\n", last));
|
||||
} else if let Some(first) = &contact.first_name {
|
||||
vcard.push_str(&format!("N:;{};;;\r\n", first));
|
||||
}
|
||||
|
||||
if let Some(fn_name) = &contact.full_name {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
} else {
|
||||
// FN is mandatory in vCard 3.0
|
||||
let fn_name = format!(
|
||||
"{} {}",
|
||||
contact.first_name.as_deref().unwrap_or(""),
|
||||
contact.last_name.as_deref().unwrap_or(""),
|
||||
)
|
||||
.trim()
|
||||
.to_string();
|
||||
if !fn_name.is_empty() {
|
||||
vcard.push_str(&format!("FN:{}\r\n", fn_name));
|
||||
} else {
|
||||
vcard.push_str("FN:Unknown\r\n");
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(nickname) = &contact.nickname {
|
||||
vcard.push_str(&format!("NICKNAME:{}\r\n", nickname));
|
||||
}
|
||||
|
||||
for email in &contact.email {
|
||||
vcard.push_str(&format!(
|
||||
"EMAIL;TYPE={}:{}\r\n",
|
||||
email.r#type.to_uppercase(),
|
||||
email.email
|
||||
));
|
||||
}
|
||||
|
||||
for phone in &contact.phone {
|
||||
vcard.push_str(&format!(
|
||||
"TEL;TYPE={}:{}\r\n",
|
||||
phone.r#type.to_uppercase(),
|
||||
phone.number
|
||||
));
|
||||
}
|
||||
|
||||
for addr in &contact.address {
|
||||
let adr = format!(
|
||||
";;{};{};{};{};{}",
|
||||
addr.street.as_deref().unwrap_or(""),
|
||||
addr.city.as_deref().unwrap_or(""),
|
||||
addr.state.as_deref().unwrap_or(""),
|
||||
addr.postal_code.as_deref().unwrap_or(""),
|
||||
addr.country.as_deref().unwrap_or(""),
|
||||
);
|
||||
vcard.push_str(&format!(
|
||||
"ADR;TYPE={}:{}\r\n",
|
||||
addr.r#type.to_uppercase(),
|
||||
adr
|
||||
));
|
||||
}
|
||||
|
||||
if let Some(org) = &contact.organization {
|
||||
vcard.push_str(&format!("ORG:{}\r\n", org));
|
||||
}
|
||||
if let Some(title) = &contact.title {
|
||||
vcard.push_str(&format!("TITLE:{}\r\n", title));
|
||||
}
|
||||
if let Some(notes) = &contact.notes {
|
||||
vcard.push_str(&format!("NOTE:{}\r\n", notes.replace('\n', "\\n")));
|
||||
}
|
||||
if let Some(bday) = &contact.birthday {
|
||||
vcard.push_str(&format!("BDAY:{}\r\n", bday.format("%Y-%m-%d")));
|
||||
}
|
||||
if let Some(photo) = &contact.photo_url {
|
||||
vcard.push_str(&format!("PHOTO;VALUE=URI:{}\r\n", photo));
|
||||
}
|
||||
|
||||
vcard.push_str(&format!(
|
||||
"REV:{}\r\n",
|
||||
contact.updated_at.format("%Y%m%dT%H%M%SZ")
|
||||
));
|
||||
vcard.push_str("END:VCARD\r\n");
|
||||
|
||||
vcard
|
||||
}
|
||||
}
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Deterministic synthetic address book: every contact has 2 emails, 1 phone
|
||||
/// and 1 address; optional fields (nickname, notes-with-newline, birthday,
|
||||
/// photo, missing names → FN fallback) are cycled so the byte-equality gate
|
||||
/// exercises every `contact_to_vcard` branch, not just the happy path.
|
||||
fn make_contacts(n: usize) -> Vec<ContactDto> {
|
||||
let created = Utc.with_ymd_and_hms(2026, 1, 15, 9, 0, 0).unwrap();
|
||||
let updated = Utc.with_ymd_and_hms(2026, 6, 30, 18, 45, 12).unwrap();
|
||||
|
||||
(0..n)
|
||||
.map(|i| {
|
||||
let (full_name, first_name, last_name) = match i % 5 {
|
||||
0 => (
|
||||
Some(format!("Contact {i:05} Example")),
|
||||
Some(format!("Contact{i:05}")),
|
||||
Some("Example".to_string()),
|
||||
),
|
||||
1 => (
|
||||
None,
|
||||
Some(format!("Contact{i:05}")),
|
||||
Some("Example".to_string()),
|
||||
),
|
||||
2 => (None, None, Some("Example".to_string())),
|
||||
3 => (None, Some(format!("Contact{i:05}")), None),
|
||||
_ => (None, None, None), // FN:Unknown fallback
|
||||
};
|
||||
ContactDto {
|
||||
id: format!("id-{i:05}"),
|
||||
address_book_id: "bench-book".to_string(),
|
||||
uid: format!("bench-contact-{i:05}@oxicloud"),
|
||||
full_name,
|
||||
first_name,
|
||||
last_name,
|
||||
nickname: (i % 7 == 0).then(|| format!("nick{i}")),
|
||||
email: vec![
|
||||
EmailDto {
|
||||
email: format!("contact{i:05}@example.com"),
|
||||
r#type: "work".to_string(),
|
||||
is_primary: true,
|
||||
},
|
||||
EmailDto {
|
||||
email: format!("contact{i:05}@home.example.org"),
|
||||
r#type: "home".to_string(),
|
||||
is_primary: false,
|
||||
},
|
||||
],
|
||||
phone: vec![PhoneDto {
|
||||
number: format!("+1-555-{:04}", i % 10_000),
|
||||
r#type: "cell".to_string(),
|
||||
is_primary: true,
|
||||
}],
|
||||
address: vec![AddressDto {
|
||||
street: Some(format!("{} Main Street", i + 1)),
|
||||
city: Some("Springfield".to_string()),
|
||||
state: Some("IL".to_string()),
|
||||
postal_code: Some(format!("{:05}", 60_000 + (i % 1_000))),
|
||||
country: Some("USA".to_string()),
|
||||
r#type: "home".to_string(),
|
||||
is_primary: true,
|
||||
}],
|
||||
organization: Some("OxiCloud Benchmarks Inc.".to_string()),
|
||||
title: Some("Engineer".to_string()),
|
||||
notes: (i % 11 == 0).then(|| "line one\nline two & <specials>".to_string()),
|
||||
photo_url: (i % 13 == 0).then(|| format!("https://example.com/avatars/{i}.jpg")),
|
||||
birthday: (i % 3 == 0).then(|| NaiveDate::from_ymd_opt(1990, 5, 17).unwrap()),
|
||||
anniversary: None,
|
||||
created_at: created,
|
||||
updated_at: updated,
|
||||
etag: format!("etag-{i:05}"),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn dav(name: &str) -> QualifiedName {
|
||||
QualifiedName {
|
||||
namespace: "DAV:".to_string(),
|
||||
name: name.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn carddav(name: &str) -> QualifiedName {
|
||||
QualifiedName {
|
||||
namespace: "urn:ietf:params:xml:ns:carddav".to_string(),
|
||||
name: name.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// OLD handler flow: pre-generate a vCard per contact, then generate the XML
|
||||
/// (which re-generates every vCard on demand and never reads the pre-made ones).
|
||||
fn run_before(contacts: &[ContactDto], report: &CardDavReportType, base_href: &str) -> Vec<u8> {
|
||||
// Generate vCards (verbatim old handle_report pre-generation)
|
||||
let vcards: Vec<(String, String)> = contacts
|
||||
.iter()
|
||||
.map(|c| (c.uid.clone(), before::contact_to_vcard(c)))
|
||||
.collect();
|
||||
|
||||
let mut out = Vec::new();
|
||||
before::generate_contacts_response(&mut out, contacts, &vcards, report, base_href)
|
||||
.expect("BEFORE XML generation failed");
|
||||
out
|
||||
}
|
||||
|
||||
/// NEW production path.
|
||||
fn run_after(contacts: &[ContactDto], report: &CardDavReportType, base_href: &str) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
CardDavAdapter::generate_contacts_response(&mut out, contacts, report, base_href)
|
||||
.expect("AFTER XML generation failed");
|
||||
out
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
fn first_diff(a: &[u8], b: &[u8]) -> Option<usize> {
|
||||
if a == b {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
a.iter()
|
||||
.zip(b.iter())
|
||||
.position(|(x, y)| x != y)
|
||||
.unwrap_or_else(|| a.len().min(b.len())),
|
||||
)
|
||||
}
|
||||
|
||||
fn context_snippet(bytes: &[u8], at: usize) -> String {
|
||||
let start = at.saturating_sub(40);
|
||||
let end = (at + 40).min(bytes.len());
|
||||
String::from_utf8_lossy(&bytes[start..end]).into_owned()
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
let base_href = "/carddav/bench-book/";
|
||||
|
||||
let prop_sets: Vec<(&str, Vec<QualifiedName>)> = vec![
|
||||
("getetag", vec![dav("getetag")]),
|
||||
(
|
||||
"getetag + address-data",
|
||||
vec![dav("getetag"), carddav("address-data")],
|
||||
),
|
||||
// Not part of the timing table, but gated too: the empty-props
|
||||
// default path also embeds address-data.
|
||||
("(empty = allprop default)", vec![]),
|
||||
];
|
||||
let sizes = [500usize, 5_000];
|
||||
|
||||
// ── Equivalence gate ────────────────────────────────────────────────
|
||||
let gate_contacts = make_contacts(*sizes.iter().max().unwrap());
|
||||
for c in &gate_contacts {
|
||||
let old = before::contact_to_vcard(c);
|
||||
let new = contact_to_vcard(c);
|
||||
if old != new {
|
||||
let at = first_diff(old.as_bytes(), new.as_bytes()).unwrap();
|
||||
eprintln!(
|
||||
"EQUIVALENCE FAILURE: contact_to_vcard differs for uid={} at byte {}\n old: …{}…\n new: …{}…",
|
||||
c.uid,
|
||||
at,
|
||||
context_snippet(old.as_bytes(), at),
|
||||
context_snippet(new.as_bytes(), at),
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
for &n in &sizes {
|
||||
let contacts = &gate_contacts[..n];
|
||||
for (label, props) in &prop_sets {
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: props.clone(),
|
||||
};
|
||||
let old_xml = run_before(contacts, &report, base_href);
|
||||
let new_xml = run_after(contacts, &report, base_href);
|
||||
if let Some(at) = first_diff(&old_xml, &new_xml) {
|
||||
eprintln!(
|
||||
"EQUIVALENCE FAILURE: REPORT XML differs (N={}, props={}) at byte {} (before {} B, after {} B)\n before: …{}…\n after: …{}…",
|
||||
n,
|
||||
label,
|
||||
at,
|
||||
old_xml.len(),
|
||||
new_xml.len(),
|
||||
context_snippet(&old_xml, at),
|
||||
context_snippet(&new_xml, at),
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"equivalence gate: BEFORE == AFTER byte-identical for all prop sets at N = {:?} (and all {} vCards match)\n",
|
||||
sizes,
|
||||
gate_contacts.len()
|
||||
);
|
||||
|
||||
// ── Timing ──────────────────────────────────────────────────────────
|
||||
println!("| N | props | BEFORE ms | AFTER ms | speedup |");
|
||||
println!("|------:|------------------------|----------:|---------:|--------:|");
|
||||
for &n in &sizes {
|
||||
let contacts = &gate_contacts[..n];
|
||||
for (label, props) in prop_sets.iter().take(2) {
|
||||
let report = CardDavReportType::AddressbookQuery {
|
||||
props: props.clone(),
|
||||
};
|
||||
|
||||
// Warm-up (allocator, caches) — result discarded.
|
||||
let _ = run_before(contacts, &report, base_href);
|
||||
let _ = run_after(contacts, &report, base_href);
|
||||
|
||||
let mut before_ms = Vec::with_capacity(reps);
|
||||
let mut after_ms = Vec::with_capacity(reps);
|
||||
for _ in 0..reps {
|
||||
let t0 = Instant::now();
|
||||
let out = run_before(contacts, &report, base_href);
|
||||
before_ms.push(t0.elapsed().as_secs_f64() * 1_000.0);
|
||||
std::hint::black_box(&out);
|
||||
|
||||
let t1 = Instant::now();
|
||||
let out = run_after(contacts, &report, base_href);
|
||||
after_ms.push(t1.elapsed().as_secs_f64() * 1_000.0);
|
||||
std::hint::black_box(&out);
|
||||
}
|
||||
let b = median(before_ms);
|
||||
let a = median(after_ms);
|
||||
println!(
|
||||
"| {:>5} | {:<22} | {:>9.3} | {:>8.3} | {:>6.2}x |",
|
||||
n,
|
||||
label,
|
||||
b,
|
||||
a,
|
||||
b / a
|
||||
);
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"\n(median of {} reps; BEFORE includes the old handler's vCard pre-generation loop,",
|
||||
reps
|
||||
);
|
||||
println!(" which the old code then discarded — the O(N²) uid scan dominates at large N)");
|
||||
}
|
||||
@@ -0,0 +1,589 @@
|
||||
//! File/Folder entity → DTO mapping benchmark — per-row allocation churn.
|
||||
//!
|
||||
//! Isolates the variables the DTO-mapping change touches:
|
||||
//!
|
||||
//! • `Arc::<str>::from(&'static str)` for the closed-set display fields
|
||||
//! (icon class, icon special class, category) — always alloc + copy —
|
||||
//! vs interned `Arc<str>` lookups (`intern_display` / `intern_mime`).
|
||||
//! • `File::compute_etag` / `Folder::compute_etag` — `chars().take(16)
|
||||
//! .collect::<String>()` + `format!` (2 allocs) vs one sized buffer.
|
||||
//! • `format_file_size` — two `format!` calls per row vs one buffer.
|
||||
//! • `Folder → FolderDto` — per-getter `.to_string()` clones + a
|
||||
//! double-allocated etag vs `into_parts()` moves.
|
||||
//!
|
||||
//! The OLD mapping logic is copied verbatim into `mod before` so one binary
|
||||
//! reports BEFORE vs AFTER side by side, and an equivalence gate asserts the
|
||||
//! two produce byte-identical DTOs for every row (exit 1 on any diff).
|
||||
//!
|
||||
//! Sections:
|
||||
//! 1. File → FileDto wall time (p50 ns/row over BENCH_PASSES passes)
|
||||
//! 2. Folder → FolderDto wall time (same)
|
||||
//! 3. Alloc calls/row (counting global allocator wrapping System — the
|
||||
//! lib crate sets no global allocator; mimalloc lives in main.rs only,
|
||||
//! which examples do not link)
|
||||
//! 4. Equivalence gate: BEFORE output == AFTER output, field by field
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_dto_map
|
||||
//! Tunables (env):
|
||||
//! BENCH_ROWS (10000) BENCH_PASSES (100)
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::env;
|
||||
use std::hint::black_box;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use oxicloud::application::dtos::file_dto::FileDto;
|
||||
use oxicloud::application::dtos::folder_dto::FolderDto;
|
||||
use oxicloud::domain::entities::file::File;
|
||||
use oxicloud::domain::entities::folder::Folder;
|
||||
use oxicloud::domain::services::path_service::StoragePath;
|
||||
use uuid::Uuid;
|
||||
|
||||
// ─── Counting allocator (Section 3) ─────────────────────────────────────────
|
||||
|
||||
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
struct CountingAlloc;
|
||||
|
||||
unsafe impl GlobalAlloc for CountingAlloc {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc(layout) }
|
||||
}
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) }
|
||||
}
|
||||
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.realloc(ptr, layout, new_size) }
|
||||
}
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc_zeroed(layout) }
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static GLOBAL: CountingAlloc = CountingAlloc;
|
||||
|
||||
// ─── BEFORE: verbatim copy of the pre-optimization mapping logic ────────────
|
||||
|
||||
/// Pre-optimization reference implementation. Copied verbatim from the old
|
||||
/// `From<File> for FileDto` / `From<Folder> for FolderDto` bodies, the old
|
||||
/// `File::compute_etag` / `Folder::compute_etag` formulas and the old
|
||||
/// `format_file_size` — kept byte-for-byte in behaviour so the equivalence
|
||||
/// gate proves the optimized paths change nothing observable.
|
||||
#[allow(clippy::all)]
|
||||
mod before {
|
||||
use std::sync::Arc;
|
||||
|
||||
use oxicloud::application::dtos::display_helpers::{
|
||||
category_for, icon_class_for, icon_special_class_for,
|
||||
};
|
||||
use oxicloud::application::dtos::file_dto::FileDto;
|
||||
use oxicloud::application::dtos::folder_dto::FolderDto;
|
||||
use oxicloud::domain::entities::file::File;
|
||||
use oxicloud::domain::entities::folder::Folder;
|
||||
|
||||
/// Old `File::compute_etag`: intermediate `collect::<String>()` +
|
||||
/// `format!` — 2 allocations for one ~21-char string.
|
||||
fn file_compute_etag(blob_hash: &str, modified_at: u64) -> String {
|
||||
let prefix: String = blob_hash.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, modified_at)
|
||||
}
|
||||
|
||||
/// Old `Folder::compute_etag` (same shape as the file formula).
|
||||
fn folder_compute_etag(id: &str, tree_modified_at: u64) -> String {
|
||||
let prefix: String = id.chars().take(16).collect();
|
||||
format!("{}-{}", prefix, tree_modified_at)
|
||||
}
|
||||
|
||||
/// Old `format_file_size`: two `format!` calls per row.
|
||||
fn format_file_size(bytes: u64) -> String {
|
||||
if bytes == 0 {
|
||||
return "0 Bytes".to_string();
|
||||
}
|
||||
|
||||
const K: f64 = 1024.0;
|
||||
const SIZES: [&str; 5] = ["Bytes", "KB", "MB", "GB", "TB"];
|
||||
|
||||
let i = ((bytes as f64).ln() / K.ln()).floor() as usize;
|
||||
let i = i.min(SIZES.len() - 1);
|
||||
|
||||
let value = bytes as f64 / K.powi(i as i32);
|
||||
|
||||
let formatted = format!("{:.2}", value);
|
||||
let formatted = formatted.trim_end_matches('0').trim_end_matches('.');
|
||||
|
||||
format!("{} {}", formatted, SIZES[i])
|
||||
}
|
||||
|
||||
/// Old `From<File> for FileDto` body: `Arc::from(&str)` for the three
|
||||
/// display fields and the mime type (alloc + copy each), 2-alloc etag,
|
||||
/// 2-format size string.
|
||||
pub fn file_to_dto(file: File) -> FileDto {
|
||||
let etag = file_compute_etag(file.content_hash(), file.modified_at());
|
||||
let content_hash = file.content_hash().to_string();
|
||||
|
||||
let parts = file.into_parts();
|
||||
|
||||
let icon_class: Arc<str> = Arc::from(icon_class_for(&parts.name, &parts.mime_type));
|
||||
let icon_special_class: Arc<str> =
|
||||
Arc::from(icon_special_class_for(&parts.name, &parts.mime_type));
|
||||
let category: Arc<str> = Arc::from(category_for(&parts.name, &parts.mime_type));
|
||||
let size_formatted = format_file_size(parts.size);
|
||||
let mime_type: Arc<str> = Arc::from(parts.mime_type.as_str());
|
||||
|
||||
FileDto {
|
||||
id: parts.id,
|
||||
name: parts.name,
|
||||
path: parts.path_string,
|
||||
size: parts.size,
|
||||
mime_type,
|
||||
folder_id: parts.folder_id,
|
||||
created_at: parts.created_at,
|
||||
modified_at: parts.modified_at,
|
||||
icon_class,
|
||||
icon_special_class,
|
||||
category,
|
||||
size_formatted,
|
||||
sort_date: None,
|
||||
content_hash,
|
||||
etag,
|
||||
created_by: parts.created_by,
|
||||
updated_by: parts.updated_by,
|
||||
}
|
||||
}
|
||||
|
||||
/// Old `From<Folder> for FolderDto` body: per-getter `.to_string()`
|
||||
/// clones, `folder.etag().to_string()` (etag built then cloned — the
|
||||
/// verbatim double alloc) and 3 fresh `Arc::from` constants per row.
|
||||
pub fn folder_to_dto(folder: Folder) -> FolderDto {
|
||||
let is_root = folder.parent_id().is_none();
|
||||
let etag = folder_compute_etag(folder.id(), folder.tree_modified_at()).to_string();
|
||||
|
||||
FolderDto {
|
||||
id: folder.id().to_string(),
|
||||
name: folder.name().to_string(),
|
||||
path: folder.path_string().to_string(),
|
||||
parent_id: folder.parent_id().map(String::from),
|
||||
drive_id: folder.drive_id(),
|
||||
created_at: folder.created_at(),
|
||||
modified_at: folder.modified_at(),
|
||||
is_root,
|
||||
icon_class: Arc::from("fas fa-folder"),
|
||||
icon_special_class: Arc::from("folder-icon"),
|
||||
category: Arc::from("Folder"),
|
||||
etag,
|
||||
created_by: folder.created_by(),
|
||||
updated_by: folder.updated_by(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Synthetic corpus ────────────────────────────────────────────────────────
|
||||
|
||||
/// (extension, mime) matrix: interned common types, generic MIMEs that
|
||||
/// exercise the extension fallback, and exotic MIMEs that miss the intern
|
||||
/// table so the fallback `Arc::from` path is measured too.
|
||||
const KINDS: &[(&str, &str)] = &[
|
||||
("jpg", "image/jpeg"),
|
||||
("png", "image/png"),
|
||||
("heic", "image/heic"),
|
||||
("mp4", "video/mp4"),
|
||||
("mov", "video/quicktime"),
|
||||
("mp3", "audio/mpeg"),
|
||||
("flac", "audio/flac"),
|
||||
("pdf", "application/pdf"),
|
||||
(
|
||||
"docx",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
),
|
||||
(
|
||||
"xlsx",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
),
|
||||
("txt", "text/plain"),
|
||||
("md", "text/markdown"),
|
||||
("csv", "text/csv"),
|
||||
("json", "application/json"),
|
||||
("zip", "application/zip"),
|
||||
("gz", "application/gzip"),
|
||||
// Extension fallback: generic MIME, type resolved from the name.
|
||||
("rs", "application/octet-stream"),
|
||||
("py", "application/octet-stream"),
|
||||
("svelte", "application/octet-stream"),
|
||||
("dmg", "application/octet-stream"),
|
||||
("bin", "application/octet-stream"),
|
||||
// No extension + empty MIME: full-default path.
|
||||
("", ""),
|
||||
// Exotic MIMEs: miss the intern table, fall back to Arc::from.
|
||||
("pdb", "chemical/x-pdb"),
|
||||
("xyz", "application/x-very-exotic-subtype+custom"),
|
||||
];
|
||||
|
||||
const SIZES: &[u64] = &[
|
||||
0,
|
||||
137,
|
||||
500,
|
||||
1_024,
|
||||
1_536,
|
||||
65_536,
|
||||
1_048_576,
|
||||
3_423_744,
|
||||
987_654_321,
|
||||
1_073_741_824,
|
||||
5_497_558_138_880, // ~5 TB
|
||||
];
|
||||
|
||||
/// Deterministic xorshift64* — fake-but-plausible 64-char lowercase hex
|
||||
/// BLAKE3 hashes.
|
||||
fn next_seed(seed: &mut u64) -> u64 {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
seed.wrapping_mul(0x2545F4914F6CDD1D)
|
||||
}
|
||||
|
||||
fn fake_blake3(seed: &mut u64) -> String {
|
||||
format!(
|
||||
"{:016x}{:016x}{:016x}{:016x}",
|
||||
next_seed(seed),
|
||||
next_seed(seed),
|
||||
next_seed(seed),
|
||||
next_seed(seed)
|
||||
)
|
||||
}
|
||||
|
||||
fn build_files(rows: usize) -> Vec<File> {
|
||||
let mut seed = 0x9E3779B97F4A7C15u64;
|
||||
(0..rows)
|
||||
.map(|i| {
|
||||
let (ext, mime) = KINDS[i % KINDS.len()];
|
||||
let name = if ext.is_empty() {
|
||||
format!("file_{i:05}")
|
||||
} else {
|
||||
format!("file_{i:05}.{ext}")
|
||||
};
|
||||
let path = StoragePath::from_string(&format!("/bench/dir_{}/{}", i % 37, name));
|
||||
let folder_id = if i % 3 == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Uuid::from_u128(1000 + (i % 37) as u128).to_string())
|
||||
};
|
||||
let created_by = (i % 2 == 0).then(|| Uuid::from_u128(7 + (i % 5) as u128));
|
||||
let updated_by = (i % 4 == 0).then(|| Uuid::from_u128(11 + (i % 3) as u128));
|
||||
File::with_timestamps_blob_hash_and_provenance(
|
||||
Uuid::from_u128(i as u128).to_string(),
|
||||
name,
|
||||
path,
|
||||
SIZES[i % SIZES.len()],
|
||||
mime.to_string(),
|
||||
folder_id,
|
||||
1_600_000_000 + i as u64,
|
||||
1_700_000_000 + (i as u64 * 7) % 100_000,
|
||||
fake_blake3(&mut seed),
|
||||
created_by,
|
||||
updated_by,
|
||||
)
|
||||
.expect("valid synthetic file")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn build_folders(rows: usize) -> Vec<Folder> {
|
||||
(0..rows)
|
||||
.map(|i| {
|
||||
let name = format!("folder_{i:05}");
|
||||
let path = StoragePath::from_string(&format!("/bench/parent_{}/{}", i % 37, name));
|
||||
let parent_id = if i % 5 == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Uuid::from_u128(2000 + (i % 37) as u128).to_string())
|
||||
};
|
||||
let created_by = (i % 2 == 0).then(|| Uuid::from_u128(7 + (i % 5) as u128));
|
||||
let updated_by = (i % 4 == 0).then(|| Uuid::from_u128(11 + (i % 3) as u128));
|
||||
Folder::with_timestamps_tree_and_provenance(
|
||||
Uuid::from_u128(500_000 + i as u128).to_string(),
|
||||
name,
|
||||
path,
|
||||
parent_id,
|
||||
Uuid::from_u128(42 + (i % 4) as u128),
|
||||
1_600_000_000 + i as u64,
|
||||
1_700_000_000 + (i as u64 * 7) % 100_000,
|
||||
1_700_000_000 + (i as u64 * 11) % 100_000,
|
||||
created_by,
|
||||
updated_by,
|
||||
)
|
||||
.expect("valid synthetic folder")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
// ─── Measurement helpers ─────────────────────────────────────────────────────
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// p50 wall seconds per pass of `f` over `passes` passes.
|
||||
fn p50_pass_secs(passes: usize, mut f: impl FnMut()) -> f64 {
|
||||
f(); // warmup (also initializes LazyLock intern tables)
|
||||
let mut xs = Vec::with_capacity(passes);
|
||||
for _ in 0..passes {
|
||||
let t0 = Instant::now();
|
||||
f();
|
||||
xs.push(t0.elapsed().as_secs_f64());
|
||||
}
|
||||
median(xs)
|
||||
}
|
||||
|
||||
/// Allocation calls performed by one run of `f` (deterministic — the
|
||||
/// mappings do no I/O and touch no shared caches beyond the intern tables,
|
||||
/// which the warmup run already initialized).
|
||||
fn allocs_of(mut f: impl FnMut()) -> u64 {
|
||||
f(); // warmup so one-time lazy init isn't attributed to the variant
|
||||
let start = ALLOC_CALLS.load(Ordering::Relaxed);
|
||||
f();
|
||||
ALLOC_CALLS.load(Ordering::Relaxed) - start
|
||||
}
|
||||
|
||||
struct Row {
|
||||
variant: &'static str,
|
||||
ns_per_row: f64,
|
||||
allocs_per_row: f64,
|
||||
}
|
||||
|
||||
// ─── Equivalence gate (Section 4) ────────────────────────────────────────────
|
||||
|
||||
macro_rules! cmp_field {
|
||||
($diffs:expr, $i:expr, $kind:expr, $b:expr, $a:expr, $field:ident) => {
|
||||
if $b.$field != $a.$field {
|
||||
$diffs += 1;
|
||||
if $diffs <= 20 {
|
||||
println!(
|
||||
" DIFF {} row {}: {} BEFORE={:?} AFTER={:?}",
|
||||
$kind,
|
||||
$i,
|
||||
stringify!($field),
|
||||
$b.$field,
|
||||
$a.$field
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
fn diff_file(i: usize, b: &FileDto, a: &FileDto, diffs: &mut u64) {
|
||||
cmp_field!(*diffs, i, "file", b, a, id);
|
||||
cmp_field!(*diffs, i, "file", b, a, name);
|
||||
cmp_field!(*diffs, i, "file", b, a, path);
|
||||
cmp_field!(*diffs, i, "file", b, a, size);
|
||||
cmp_field!(*diffs, i, "file", b, a, mime_type);
|
||||
cmp_field!(*diffs, i, "file", b, a, folder_id);
|
||||
cmp_field!(*diffs, i, "file", b, a, created_at);
|
||||
cmp_field!(*diffs, i, "file", b, a, modified_at);
|
||||
cmp_field!(*diffs, i, "file", b, a, icon_class);
|
||||
cmp_field!(*diffs, i, "file", b, a, icon_special_class);
|
||||
cmp_field!(*diffs, i, "file", b, a, category);
|
||||
cmp_field!(*diffs, i, "file", b, a, size_formatted);
|
||||
cmp_field!(*diffs, i, "file", b, a, sort_date);
|
||||
cmp_field!(*diffs, i, "file", b, a, content_hash);
|
||||
cmp_field!(*diffs, i, "file", b, a, etag);
|
||||
cmp_field!(*diffs, i, "file", b, a, created_by);
|
||||
cmp_field!(*diffs, i, "file", b, a, updated_by);
|
||||
}
|
||||
|
||||
fn diff_folder(i: usize, b: &FolderDto, a: &FolderDto, diffs: &mut u64) {
|
||||
cmp_field!(*diffs, i, "folder", b, a, id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, name);
|
||||
cmp_field!(*diffs, i, "folder", b, a, path);
|
||||
cmp_field!(*diffs, i, "folder", b, a, parent_id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, drive_id);
|
||||
cmp_field!(*diffs, i, "folder", b, a, created_at);
|
||||
cmp_field!(*diffs, i, "folder", b, a, modified_at);
|
||||
cmp_field!(*diffs, i, "folder", b, a, is_root);
|
||||
cmp_field!(*diffs, i, "folder", b, a, icon_class);
|
||||
cmp_field!(*diffs, i, "folder", b, a, icon_special_class);
|
||||
cmp_field!(*diffs, i, "folder", b, a, category);
|
||||
cmp_field!(*diffs, i, "folder", b, a, etag);
|
||||
cmp_field!(*diffs, i, "folder", b, a, created_by);
|
||||
cmp_field!(*diffs, i, "folder", b, a, updated_by);
|
||||
}
|
||||
|
||||
// ─── Main ────────────────────────────────────────────────────────────────────
|
||||
|
||||
fn main() {
|
||||
let rows: usize = env_or("BENCH_ROWS", 10_000).max(1);
|
||||
let passes: usize = env_or("BENCH_PASSES", 100).max(1);
|
||||
|
||||
let files = build_files(rows);
|
||||
let folders = build_folders(rows);
|
||||
println!(
|
||||
"corpus: {rows} files ({} kinds x {} sizes) + {rows} folders, {passes} timed passes",
|
||||
KINDS.len(),
|
||||
SIZES.len()
|
||||
);
|
||||
println!(
|
||||
"note: each measured pass pays one entity clone per row (mapping consumes the\n\
|
||||
entity); the clone-only baseline is measured separately and subtracted.\n"
|
||||
);
|
||||
|
||||
// ── Section 1: File → FileDto wall time ─────────────────────────────
|
||||
println!("── Section 1: File → FileDto (p50 wall, net of clone) ──");
|
||||
let file_base_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(f.clone());
|
||||
}
|
||||
});
|
||||
let file_before_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(before::file_to_dto(f.clone()));
|
||||
}
|
||||
});
|
||||
let file_after_s = p50_pass_secs(passes, || {
|
||||
for f in &files {
|
||||
black_box(FileDto::from(f.clone()));
|
||||
}
|
||||
});
|
||||
let file_base_ns = file_base_s * 1e9 / rows as f64;
|
||||
let file_before_ns = (file_before_s - file_base_s) * 1e9 / rows as f64;
|
||||
let file_after_ns = (file_after_s - file_base_s) * 1e9 / rows as f64;
|
||||
println!(" clone-only baseline: {file_base_ns:8.1} ns/row");
|
||||
println!(" BEFORE mapping: {file_before_ns:8.1} ns/row");
|
||||
println!(" AFTER mapping: {file_after_ns:8.1} ns/row\n");
|
||||
|
||||
// ── Section 2: Folder → FolderDto wall time ─────────────────────────
|
||||
println!("── Section 2: Folder → FolderDto (p50 wall, net of clone) ──");
|
||||
let folder_base_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(f.clone());
|
||||
}
|
||||
});
|
||||
let folder_before_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(before::folder_to_dto(f.clone()));
|
||||
}
|
||||
});
|
||||
let folder_after_s = p50_pass_secs(passes, || {
|
||||
for f in &folders {
|
||||
black_box(FolderDto::from(f.clone()));
|
||||
}
|
||||
});
|
||||
let folder_base_ns = folder_base_s * 1e9 / rows as f64;
|
||||
let folder_before_ns = (folder_before_s - folder_base_s) * 1e9 / rows as f64;
|
||||
let folder_after_ns = (folder_after_s - folder_base_s) * 1e9 / rows as f64;
|
||||
println!(" clone-only baseline: {folder_base_ns:8.1} ns/row");
|
||||
println!(" BEFORE mapping: {folder_before_ns:8.1} ns/row");
|
||||
println!(" AFTER mapping: {folder_after_ns:8.1} ns/row\n");
|
||||
|
||||
// ── Section 3: allocation calls per row ─────────────────────────────
|
||||
println!("── Section 3: allocator calls per row (net of clone) ──");
|
||||
let file_base_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(f.clone());
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64;
|
||||
let file_before_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(before::file_to_dto(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- file_base_a;
|
||||
let file_after_a = allocs_of(|| {
|
||||
for f in &files {
|
||||
black_box(FileDto::from(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- file_base_a;
|
||||
let folder_base_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(f.clone());
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64;
|
||||
let folder_before_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(before::folder_to_dto(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- folder_base_a;
|
||||
let folder_after_a = allocs_of(|| {
|
||||
for f in &folders {
|
||||
black_box(FolderDto::from(f.clone()));
|
||||
}
|
||||
}) as f64
|
||||
/ rows as f64
|
||||
- folder_base_a;
|
||||
println!(" file clone baseline: {file_base_a:6.2} allocs/row");
|
||||
println!(" file BEFORE mapping: {file_before_a:6.2} allocs/row");
|
||||
println!(" file AFTER mapping: {file_after_a:6.2} allocs/row");
|
||||
println!(" folder clone baseline: {folder_base_a:6.2} allocs/row");
|
||||
println!(" folder BEFORE mapping: {folder_before_a:6.2} allocs/row");
|
||||
println!(" folder AFTER mapping: {folder_after_a:6.2} allocs/row\n");
|
||||
|
||||
// ── Section 4: equivalence gate ─────────────────────────────────────
|
||||
println!("── Section 4: equivalence gate (BEFORE == AFTER, field by field) ──");
|
||||
let mut diffs: u64 = 0;
|
||||
for (i, f) in files.iter().enumerate() {
|
||||
let b = before::file_to_dto(f.clone());
|
||||
let a = FileDto::from(f.clone());
|
||||
diff_file(i, &b, &a, &mut diffs);
|
||||
}
|
||||
for (i, f) in folders.iter().enumerate() {
|
||||
let b = before::folder_to_dto(f.clone());
|
||||
let a = FolderDto::from(f.clone());
|
||||
diff_folder(i, &b, &a, &mut diffs);
|
||||
}
|
||||
if diffs > 0 {
|
||||
println!(" FAILED: {diffs} field diffs between BEFORE and AFTER mappings");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!(" PASSED: {rows} files + {rows} folders map byte-identically\n");
|
||||
|
||||
// ── Markdown summary ─────────────────────────────────────────────────
|
||||
let table = [
|
||||
Row {
|
||||
variant: "File→FileDto BEFORE",
|
||||
ns_per_row: file_before_ns,
|
||||
allocs_per_row: file_before_a,
|
||||
},
|
||||
Row {
|
||||
variant: "File→FileDto AFTER",
|
||||
ns_per_row: file_after_ns,
|
||||
allocs_per_row: file_after_a,
|
||||
},
|
||||
Row {
|
||||
variant: "Folder→FolderDto BEFORE",
|
||||
ns_per_row: folder_before_ns,
|
||||
allocs_per_row: folder_before_a,
|
||||
},
|
||||
Row {
|
||||
variant: "Folder→FolderDto AFTER",
|
||||
ns_per_row: folder_after_ns,
|
||||
allocs_per_row: folder_after_a,
|
||||
},
|
||||
];
|
||||
println!("| variant | ns/row | allocs/row |");
|
||||
println!("|---|---:|---:|");
|
||||
for r in &table {
|
||||
println!(
|
||||
"| {} | {:.1} | {:.2} |",
|
||||
r.variant, r.ns_per_row, r.allocs_per_row
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
//! PROPFIND subfolder-paging benchmark — LIMIT/OFFSET + COUNT(*) OVER() vs
|
||||
//! keyset, mirroring the files-side PROPFIND-PAGING fix.
|
||||
//!
|
||||
//! The streaming PROPFIND walkers (native WebDAV + NC-DAV) page a folder's
|
||||
//! subfolders via `list_folders_paginated`, whose query is
|
||||
//! `COUNT(*) OVER() … ORDER BY name LIMIT $2 OFFSET $3` — every page
|
||||
//! window-aggregates and rescans ALL N subfolders (the total is only used
|
||||
//! for has_next), so a full walk is O(N²/page) row visits.
|
||||
//!
|
||||
//! The AFTER shape is the same keyset used for files: `name > $last ORDER BY
|
||||
//! name LIMIT k`, served by the existing UNIQUE index
|
||||
//! `idx_folders_unique_name (parent_id, name, drive_id) WHERE NOT is_trashed
|
||||
//! AND parent_id IS NOT NULL` — no migration needed. has_next falls out of
|
||||
//! `rows.len() == limit`.
|
||||
//!
|
||||
//! Equivalence gate: the drained name sequence must be identical.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_folder_keyset
|
||||
//! Tunables: BENCH_DIRS (5000), BENCH_PAGE (500), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, dirs: usize) -> (Uuid, Uuid) {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_folder_keyset', '/bench_folder_keyset', 'bench_folder_keyset', $1)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.folders (name, path, lpath, parent_id, drive_id)
|
||||
SELECT 'Dir_' || LPAD(i::text, 6, '0'),
|
||||
'/bench_folder_keyset/Dir_' || LPAD(i::text, 6, '0'),
|
||||
('bench_folder_keyset.d' || i)::ltree,
|
||||
$1, $2
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(dirs as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("dirs");
|
||||
sqlx::query("ANALYZE storage.folders")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(drive_id, folder_id)
|
||||
}
|
||||
|
||||
const COLS: &str = "id::text, name, path, parent_id::text, drive_id,
|
||||
EXTRACT(EPOCH FROM created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM updated_at)::bigint,
|
||||
EXTRACT(EPOCH FROM tree_modified_at)::bigint,
|
||||
created_by, updated_by";
|
||||
|
||||
type Row = (
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
Uuid,
|
||||
i64,
|
||||
i64,
|
||||
i64,
|
||||
Option<Uuid>,
|
||||
Option<Uuid>,
|
||||
);
|
||||
type RowWithTotal = (
|
||||
String,
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
Uuid,
|
||||
i64,
|
||||
i64,
|
||||
i64,
|
||||
Option<Uuid>,
|
||||
Option<Uuid>,
|
||||
i64,
|
||||
);
|
||||
|
||||
/// OLD: production `list_folders_paginated` shape — window total + OFFSET.
|
||||
async fn walk_offset(pool: &PgPool, parent: Uuid, page: i64) -> (Vec<String>, Vec<f64>) {
|
||||
let mut offset = 0i64;
|
||||
let mut names = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows: Vec<RowWithTotal> = sqlx::query_as(&format!(
|
||||
"SELECT {COLS}, COUNT(*) OVER() AS total_count
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed
|
||||
ORDER BY name
|
||||
LIMIT $2 OFFSET $3"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.bind(offset)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("offset page");
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
names.extend(rows.into_iter().map(|r| r.1));
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
offset += n as i64;
|
||||
}
|
||||
(names, times)
|
||||
}
|
||||
|
||||
/// NEW: keyset on the existing unique index; has_next = rows.len() == limit.
|
||||
async fn walk_keyset(pool: &PgPool, parent: Uuid, page: i64) -> (Vec<String>, Vec<f64>) {
|
||||
let mut after: Option<String> = None;
|
||||
let mut names = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows: Vec<Row> = if let Some(a) = &after {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed AND name > $3
|
||||
ORDER BY name
|
||||
LIMIT $2"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.bind(a)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
} else {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.folders
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed
|
||||
ORDER BY name
|
||||
LIMIT $2"
|
||||
))
|
||||
.bind(parent)
|
||||
.bind(page)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
}
|
||||
.expect("keyset page");
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
after = rows.last().map(|r| r.1.clone());
|
||||
names.extend(rows.into_iter().map(|r| r.1));
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
}
|
||||
(names, times)
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let dirs: usize = env_or("BENCH_DIRS", 5_000);
|
||||
let page: i64 = env_or("BENCH_PAGE", 500);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {dirs} subfolders (one-time)…");
|
||||
let (drive_id, folder_id) = seed(&pool, dirs).await;
|
||||
|
||||
let (ref_names, _) = walk_offset(&pool, folder_id, page).await;
|
||||
assert_eq!(ref_names.len(), dirs, "reference drain size");
|
||||
|
||||
println!("\n# full PROPFIND subfolder walk of a {dirs}-dir parent, {page}/page");
|
||||
println!(
|
||||
"{:<12} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
let mut base: Option<f64> = None;
|
||||
for mode in ["OFFSET", "KEYSET"] {
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut per_page: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (names, times) = if mode == "OFFSET" {
|
||||
walk_offset(&pool, folder_id, page).await
|
||||
} else {
|
||||
walk_keyset(&pool, folder_id, page).await
|
||||
};
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if names != ref_names {
|
||||
eprintln!("EQUIVALENCE FAILURE: {mode} drained a different sequence");
|
||||
failures += 1;
|
||||
}
|
||||
per_page = times;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<12} {:>11.1} {:>11.2} {:>8}",
|
||||
mode,
|
||||
ms,
|
||||
median(per_page.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,495 @@
|
||||
//! Web-UI folder listing benchmark — whole-folder rescan vs keyset pushdown.
|
||||
//!
|
||||
//! `list_resources_paged` (folder_db_repository.rs) pages the SPA files view
|
||||
//! with a UNION-ALL CTE (folders + files) and applies the keyset cursor
|
||||
//! OUTSIDE the CTE on computed columns (`sort_str = LOWER(name)`,
|
||||
//! `folder_first`). Postgres therefore scans every remaining row of the
|
||||
//! folder and top-N-sorts it on EVERY page — a 20k-file folder pays a full
|
||||
//! rescan per 200-row page.
|
||||
//!
|
||||
//! The AFTER shape pushes the cursor into each branch as a sargable
|
||||
//! row-value comparison (`(LOWER(name), id) > ($str, $id)`), gives each
|
||||
//! branch its own `ORDER BY … LIMIT`, and adds two expression indexes:
|
||||
//! idx_files_folder_lname (folder_id, LOWER(name), id) WHERE NOT is_trashed
|
||||
//! idx_folders_parent_lname (parent_id, LOWER(name), id) WHERE NOT is_trashed
|
||||
//! The outer query then merges ≤ 2·limit pre-sorted rows.
|
||||
//!
|
||||
//! Modes (full drain of the folder in default "name" order, plus a
|
||||
//! modified_at parity check):
|
||||
//! OLD/no-idx — the true BEFORE
|
||||
//! OLD/idx — new indexes alone, old query shape
|
||||
//! NEW/idx — the AFTER
|
||||
//!
|
||||
//! Equivalence gate: the drained (type, id) sequence must be identical
|
||||
//! across all modes; a mismatch aborts with exit(1).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_listing_keyset
|
||||
//! Tunables: BENCH_FILES (20000), BENCH_DIRS (300), BENCH_PAGE (200),
|
||||
//! BENCH_REPS (3)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, files: usize, dirs: usize) -> (Uuid, Uuid) {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_listing', '/bench_listing', 'bench_listing', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Mixed-case names so LOWER() actually differs from the raw column.
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.folders (name, path, lpath, parent_id, drive_id)
|
||||
SELECT 'Dir_' || LPAD(i::text, 6, '0'),
|
||||
'/bench_listing/Dir_' || LPAD(i::text, 6, '0'),
|
||||
('bench_listing.d' || i)::ltree,
|
||||
$1, $2
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(dirs as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("dirs");
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.files
|
||||
(name, folder_id, blob_hash, size, mime_type, drive_id,
|
||||
updated_at, category_order)
|
||||
SELECT 'File_' || LPAD(i::text, 8, '0') || '.JPG', $1,
|
||||
'benchlisting0000000000000000000000000000000000000000000000000000',
|
||||
1024 + i, 'image/jpeg', $2,
|
||||
NOW() - (i || ' seconds')::interval,
|
||||
3
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(files as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
sqlx::query("ANALYZE storage.files")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("ANALYZE storage.folders")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(drive_id, folder_id)
|
||||
}
|
||||
|
||||
const FOLDER_BRANCH: &str = r#"
|
||||
SELECT
|
||||
'folder'::text AS resource_type,
|
||||
f.id,
|
||||
f.name,
|
||||
f.parent_id AS folder_id,
|
||||
NULL::text AS mime_type,
|
||||
-1::bigint AS size,
|
||||
f.created_at,
|
||||
f.updated_at AS modified_at,
|
||||
f.drive_id,
|
||||
NULL::text AS blob_hash,
|
||||
LOWER(f.name) AS sort_str,
|
||||
0::bigint AS type_order,
|
||||
0::int AS folder_first
|
||||
FROM storage.folders f
|
||||
WHERE f.parent_id = $1::uuid AND NOT f.is_trashed
|
||||
"#;
|
||||
|
||||
const FILE_BRANCH: &str = r#"
|
||||
SELECT
|
||||
'file'::text AS resource_type,
|
||||
fm.id,
|
||||
fm.name,
|
||||
fm.folder_id,
|
||||
fm.mime_type,
|
||||
fm.size::bigint,
|
||||
fm.created_at,
|
||||
fm.updated_at AS modified_at,
|
||||
fm.drive_id,
|
||||
fm.blob_hash,
|
||||
LOWER(fm.name) AS sort_str,
|
||||
fm.category_order::bigint AS type_order,
|
||||
1::int AS folder_first
|
||||
FROM storage.files fm
|
||||
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
|
||||
"#;
|
||||
|
||||
const COLS: &str = "resource_type, id, name, folder_id, mime_type, size, \
|
||||
created_at, modified_at, drive_id, blob_hash, \
|
||||
sort_str, type_order, folder_first";
|
||||
|
||||
type Row = (
|
||||
String,
|
||||
Uuid,
|
||||
String,
|
||||
Option<Uuid>,
|
||||
Option<String>,
|
||||
i64,
|
||||
chrono::DateTime<chrono::Utc>,
|
||||
chrono::DateTime<chrono::Utc>,
|
||||
Uuid,
|
||||
Option<String>,
|
||||
String,
|
||||
i64,
|
||||
i32,
|
||||
);
|
||||
|
||||
/// Cursor state for the walks: (folder_first, sort_str, modified_at, id).
|
||||
#[derive(Clone)]
|
||||
struct Cur {
|
||||
ff: i64,
|
||||
sort_str: String,
|
||||
ts: chrono::DateTime<chrono::Utc>,
|
||||
id: Uuid,
|
||||
}
|
||||
|
||||
/// OLD shape, "name" order — production SQL verbatim: cursor OUTSIDE the CTE.
|
||||
async fn old_page_name(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
let sql = format!(
|
||||
"WITH resources AS ({FOLDER_BRANCH} UNION ALL {FILE_BRANCH}) \
|
||||
SELECT {COLS} FROM resources \
|
||||
WHERE ($3::bigint IS NULL) \
|
||||
OR (folder_first::bigint > $3) \
|
||||
OR (folder_first::bigint = $3 AND sort_str > $2) \
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid) \
|
||||
ORDER BY folder_first ASC, sort_str ASC, id ASC \
|
||||
LIMIT $6"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(cur.map(|c| c.sort_str.clone()))
|
||||
.bind(cur.map(|c| c.ff))
|
||||
.bind(cur.map(|c| c.ts))
|
||||
.bind(cur.map(|c| c.id))
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old name page")
|
||||
}
|
||||
|
||||
/// NEW shape, "name" order — cursor pushed into each branch as a sargable
|
||||
/// row-value comparison; each branch pre-sorts and pre-limits.
|
||||
async fn new_page_name(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
match cur {
|
||||
None => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH}) fb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (first)")
|
||||
}
|
||||
Some(c) if c.ff == 0 => {
|
||||
// Cursor sits in the folder group: folders continue after the
|
||||
// row-value cursor; ALL files still follow.
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH} \
|
||||
AND (LOWER(f.name), f.id) > ($3, $4::uuid)) fb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2) \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(&c.sort_str)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (folder cursor)")
|
||||
}
|
||||
Some(c) => {
|
||||
// Cursor sits in the file group: the folder branch is exhausted.
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
SELECT * FROM ({FILE_BRANCH} \
|
||||
AND (LOWER(fm.name), fm.id) > ($3, $4::uuid)) lb \
|
||||
ORDER BY sort_str ASC, id ASC LIMIT $2 \
|
||||
) r ORDER BY folder_first ASC, sort_str ASC, id ASC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(&c.sort_str)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new name page (file cursor)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// OLD shape, "modified_at" order (newest first) — production SQL verbatim.
|
||||
async fn old_page_modified(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
let sql = format!(
|
||||
"WITH resources AS ({FOLDER_BRANCH} UNION ALL {FILE_BRANCH}) \
|
||||
SELECT {COLS} FROM resources \
|
||||
WHERE ($4::timestamptz IS NULL) \
|
||||
OR (modified_at < $4) \
|
||||
OR (modified_at = $4 AND id < $5::uuid) \
|
||||
ORDER BY modified_at DESC, id DESC \
|
||||
LIMIT $6"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(cur.map(|c| c.sort_str.clone()))
|
||||
.bind(cur.map(|c| c.ff))
|
||||
.bind(cur.map(|c| c.ts))
|
||||
.bind(cur.map(|c| c.id))
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old modified page")
|
||||
}
|
||||
|
||||
/// NEW shape, "modified_at" order — per-branch row-value cursor + LIMIT.
|
||||
async fn new_page_modified(pool: &PgPool, parent: Uuid, cur: Option<&Cur>, limit: i64) -> Vec<Row> {
|
||||
match cur {
|
||||
None => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH}) fb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH}) lb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
) r ORDER BY modified_at DESC, id DESC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new modified page (first)")
|
||||
}
|
||||
Some(c) => {
|
||||
let sql = format!(
|
||||
"SELECT {COLS} FROM ( \
|
||||
(SELECT * FROM ({FOLDER_BRANCH} \
|
||||
AND (f.updated_at, f.id) < ($3, $4::uuid)) fb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
UNION ALL \
|
||||
(SELECT * FROM ({FILE_BRANCH} \
|
||||
AND (fm.updated_at, fm.id) < ($3, $4::uuid)) lb \
|
||||
ORDER BY modified_at DESC, id DESC LIMIT $2) \
|
||||
) r ORDER BY modified_at DESC, id DESC LIMIT $2"
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(parent)
|
||||
.bind(limit)
|
||||
.bind(c.ts)
|
||||
.bind(c.id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new modified page (cursor)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Drain the whole folder; returns ((type, id) sequence, per-page ms).
|
||||
async fn drain(
|
||||
pool: &PgPool,
|
||||
parent: Uuid,
|
||||
limit: i64,
|
||||
new_shape: bool,
|
||||
by_modified: bool,
|
||||
) -> (Vec<(String, Uuid)>, Vec<f64>) {
|
||||
let mut cur: Option<Cur> = None;
|
||||
let mut seq = Vec::new();
|
||||
let mut page_ms = Vec::new();
|
||||
loop {
|
||||
let t = Instant::now();
|
||||
let rows = match (new_shape, by_modified) {
|
||||
(false, false) => old_page_name(pool, parent, cur.as_ref(), limit).await,
|
||||
(true, false) => new_page_name(pool, parent, cur.as_ref(), limit).await,
|
||||
(false, true) => old_page_modified(pool, parent, cur.as_ref(), limit).await,
|
||||
(true, true) => new_page_modified(pool, parent, cur.as_ref(), limit).await,
|
||||
};
|
||||
page_ms.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let n = rows.len();
|
||||
if let Some(last) = rows.last() {
|
||||
cur = Some(Cur {
|
||||
ff: last.12 as i64,
|
||||
sort_str: last.10.clone(),
|
||||
ts: last.7,
|
||||
id: last.1,
|
||||
});
|
||||
}
|
||||
seq.extend(rows.into_iter().map(|r| (r.0, r.1)));
|
||||
if (n as i64) < limit {
|
||||
break;
|
||||
}
|
||||
}
|
||||
(seq, page_ms)
|
||||
}
|
||||
|
||||
async fn set_indexes(pool: &PgPool, on: bool) {
|
||||
if on {
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_files_folder_lname
|
||||
ON storage.files (folder_id, LOWER(name), id) WHERE NOT is_trashed",
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files idx");
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_folders_parent_lname
|
||||
ON storage.folders (parent_id, LOWER(name), id) WHERE NOT is_trashed",
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("folders idx");
|
||||
} else {
|
||||
sqlx::query("DROP INDEX IF EXISTS storage.idx_files_folder_lname")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("DROP INDEX IF EXISTS storage.idx_folders_parent_lname")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
}
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
fn p99(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[(xs.len() as f64 * 0.99) as usize % xs.len()]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let files: usize = env_or("BENCH_FILES", 20_000);
|
||||
let dirs: usize = env_or("BENCH_DIRS", 300);
|
||||
let page: i64 = env_or("BENCH_PAGE", 200);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {files} files + {dirs} dirs (one-time)…");
|
||||
let (drive_id, folder_id) = seed(&pool, files, dirs).await;
|
||||
let total = files + dirs;
|
||||
|
||||
// Reference sequences for the equivalence gate (computed once per mode).
|
||||
set_indexes(&pool, false).await;
|
||||
let (ref_name, _) = drain(&pool, folder_id, page, false, false).await;
|
||||
let (ref_modified, _) = drain(&pool, folder_id, page, false, true).await;
|
||||
assert_eq!(ref_name.len(), total, "name drain row count");
|
||||
assert_eq!(ref_modified.len(), total, "modified drain row count");
|
||||
|
||||
println!("\n# full SPA-listing drain of a {files}-file/{dirs}-dir folder, {page}/page");
|
||||
println!(
|
||||
"{:<28} {:>11} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "p99 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
for by_modified in [false, true] {
|
||||
let label = if by_modified { "modified_at" } else { "name" };
|
||||
let reference = if by_modified {
|
||||
&ref_modified
|
||||
} else {
|
||||
&ref_name
|
||||
};
|
||||
let mut base: Option<f64> = None;
|
||||
for (mode, new_shape, idx) in [
|
||||
("OLD/no-idx", false, false),
|
||||
("OLD/idx", false, true),
|
||||
("NEW/idx", true, true),
|
||||
] {
|
||||
set_indexes(&pool, idx).await;
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut pages: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (seq, page_ms) = drain(&pool, folder_id, page, new_shape, by_modified).await;
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if &seq != reference {
|
||||
eprintln!("EQUIVALENCE FAILURE: {label}/{mode} drained a different sequence");
|
||||
failures += 1;
|
||||
}
|
||||
pages = page_ms;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<28} {:>11.1} {:>11.2} {:>11.2} {:>8}",
|
||||
format!("{label} {mode}"),
|
||||
ms,
|
||||
median(pages.clone()),
|
||||
p99(pages.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
// Leave the new indexes in place (they are the production migration).
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
//! Photos timeline benchmark — full-library scan vs per-drive LATERAL top-N.
|
||||
//!
|
||||
//! `list_media_files` (file_blob_read_repository.rs) filters by
|
||||
//! `fi.drive_id IN (<grants subquery>)`, joins folders + file_metadata, and
|
||||
//! sorts globally by `media_sort_date DESC LIMIT k`. The doc comment claims
|
||||
//! `idx_files_media_timeline_by_drive` lets LIMIT stop the scan early, but
|
||||
//! the plan is a Nested Loop over the drive set feeding EVERY media row
|
||||
//! through a Hash Left Join into a top-N heapsort ABOVE the join — the
|
||||
//! index is drained to exhaustion on every page, so each timeline page
|
||||
//! costs O(library), not O(page).
|
||||
//!
|
||||
//! The AFTER shape materialises the accessible drive ids once, then does a
|
||||
//! `CROSS JOIN LATERAL (… ORDER BY media_sort_date DESC LIMIT k)` per drive
|
||||
//! — each LATERAL is one bounded index scan — and merges `drives × k` rows.
|
||||
//! The folders/file_metadata joins move OUTSIDE the top-N so only the k
|
||||
//! emitted rows pay them.
|
||||
//!
|
||||
//! Equivalence gate: page-by-page id sequences must be identical (the seed
|
||||
//! uses strictly distinct capture dates so ties cannot mask reordering).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_photos_timeline
|
||||
//! Tunables: BENCH_MEDIA (50000), BENCH_DRIVES (3), BENCH_PAGE (100),
|
||||
//! BENCH_PAGES (10), BENCH_REPS (3)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, media: usize, drives: usize) -> (Uuid, Vec<Uuid>) {
|
||||
let caller = Uuid::new_v4();
|
||||
let mut drive_ids = Vec::with_capacity(drives);
|
||||
for d in 0..drives {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes, policies)
|
||||
VALUES ('shared', NULL, '{\"include_in_photo_index\": true}'::jsonb)
|
||||
RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ($1, $2, $3::ltree, $4) RETURNING id",
|
||||
)
|
||||
.bind(format!("bench_photos_{d}"))
|
||||
.bind(format!("/bench_photos_{d}"))
|
||||
.bind(format!("bench_photos_{d}"))
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.role_grants
|
||||
(subject_type, subject_id, resource_type, resource_id, role, granted_by)
|
||||
VALUES ('user', $1, 'drive', $2, 'viewer', $1)",
|
||||
)
|
||||
.bind(caller)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("grant");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Strictly distinct capture dates (offset per drive) so the
|
||||
// equivalence gate cannot be masked by tie reordering.
|
||||
let per_drive = media / drives;
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.files
|
||||
(name, folder_id, blob_hash, size, mime_type, drive_id, media_sort_date)
|
||||
SELECT 'IMG_' || LPAD(i::text, 8, '0') || '.jpg', $1,
|
||||
'benchphotos00000000000000000000000000000000000000000000000000000',
|
||||
2048, 'image/jpeg', $2,
|
||||
TIMESTAMPTZ '2026-01-01 00:00:00Z' - ((i * $4 + $5) || ' seconds')::interval
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(per_drive as i32)
|
||||
.bind(drives as i32)
|
||||
.bind(d as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
drive_ids.push(drive_id);
|
||||
}
|
||||
sqlx::query("ANALYZE storage.files")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
sqlx::query("ANALYZE storage.role_grants")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(caller, drive_ids)
|
||||
}
|
||||
|
||||
type MediaRow = (
|
||||
String, // id::text
|
||||
String, // name
|
||||
Option<String>, // folder_id::text
|
||||
Option<String>, // fo.path
|
||||
i64, // size
|
||||
String, // mime_type
|
||||
i64, // created_at epoch
|
||||
i64, // updated_at epoch
|
||||
String, // blob_hash
|
||||
Option<Uuid>, // created_by
|
||||
Option<Uuid>, // updated_by
|
||||
i64, // sort_date epoch
|
||||
Option<i32>, // width
|
||||
Option<i32>, // height
|
||||
);
|
||||
|
||||
const GRANTS_SUBQ: &str = r#"
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
"#;
|
||||
|
||||
/// OLD shape — production SQL verbatim.
|
||||
async fn old_page(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
before: Option<chrono::DateTime<chrono::Utc>>,
|
||||
limit: i64,
|
||||
) -> Vec<MediaRow> {
|
||||
let cursor_pred = if before.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
fi.created_by, fi.updated_by,
|
||||
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
|
||||
WHERE fi.drive_id IN ({GRANTS_SUBQ})
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
"#
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(caller)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old page")
|
||||
}
|
||||
|
||||
/// NEW shape — accessible drives materialised once, per-drive LATERAL top-N
|
||||
/// on the timeline index, folders/metadata joined only on the emitted rows.
|
||||
async fn new_page(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
before: Option<chrono::DateTime<chrono::Utc>>,
|
||||
limit: i64,
|
||||
) -> Vec<MediaRow> {
|
||||
let cursor_pred = if before.is_some() {
|
||||
"AND fi.media_sort_date < $2"
|
||||
} else {
|
||||
"AND $2::timestamptz IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
WITH accessible AS MATERIALIZED ({GRANTS_SUBQ})
|
||||
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
|
||||
top.size, top.mime_type,
|
||||
EXTRACT(EPOCH FROM top.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM top.updated_at)::bigint,
|
||||
top.blob_hash,
|
||||
top.created_by, top.updated_by,
|
||||
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM (
|
||||
SELECT fi.*
|
||||
FROM accessible a
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT fi.*
|
||||
FROM storage.files fi
|
||||
WHERE fi.drive_id = a.id
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) fi
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) top
|
||||
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
|
||||
ORDER BY top.media_sort_date DESC
|
||||
"#
|
||||
);
|
||||
sqlx::query_as(&sql)
|
||||
.bind(caller)
|
||||
.bind(before)
|
||||
.bind(limit)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("new page")
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// Walk `pages` cursor pages; returns (id sequence, per-page ms).
|
||||
async fn walk(
|
||||
pool: &PgPool,
|
||||
caller: Uuid,
|
||||
page: i64,
|
||||
pages: usize,
|
||||
new_shape: bool,
|
||||
) -> (Vec<String>, Vec<f64>) {
|
||||
let mut before: Option<chrono::DateTime<chrono::Utc>> = None;
|
||||
let mut ids = Vec::new();
|
||||
let mut times = Vec::new();
|
||||
for _ in 0..pages {
|
||||
let t = Instant::now();
|
||||
let rows = if new_shape {
|
||||
new_page(pool, caller, before, page).await
|
||||
} else {
|
||||
old_page(pool, caller, before, page).await
|
||||
};
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if rows.is_empty() {
|
||||
break;
|
||||
}
|
||||
// Cursor semantics mirror production: whole-second epoch of the last
|
||||
// row (list_media_files hands the epoch back to the client).
|
||||
let last_epoch = rows.last().unwrap().11;
|
||||
before = chrono::DateTime::from_timestamp(last_epoch, 0);
|
||||
ids.extend(rows.into_iter().map(|r| r.0));
|
||||
}
|
||||
(ids, times)
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let media: usize = env_or("BENCH_MEDIA", 50_000);
|
||||
let drives: usize = env_or("BENCH_DRIVES", 3);
|
||||
let page: i64 = env_or("BENCH_PAGE", 100);
|
||||
let pages: usize = env_or("BENCH_PAGES", 10);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {media} media rows across {drives} drives (one-time)…");
|
||||
let (caller, drive_ids) = seed(&pool, media, drives).await;
|
||||
|
||||
let (ref_ids, _) = walk(&pool, caller, page, pages, false).await;
|
||||
assert_eq!(
|
||||
ref_ids.len(),
|
||||
(page as usize) * pages,
|
||||
"reference walk size"
|
||||
);
|
||||
|
||||
println!("\n# {pages} timeline pages of {page} over a {media}-photo library ({drives} drives)");
|
||||
println!(
|
||||
"{:<8} {:>11} {:>11} {:>8}",
|
||||
"mode", "total ms", "p50 ms/pg", "vs OLD"
|
||||
);
|
||||
|
||||
let mut failures = 0usize;
|
||||
let mut base: Option<f64> = None;
|
||||
for (mode, new_shape) in [("OLD", false), ("NEW", true)] {
|
||||
let mut totals = Vec::with_capacity(reps);
|
||||
let mut per_page: Vec<f64> = Vec::new();
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let (ids, times) = walk(&pool, caller, page, pages, new_shape).await;
|
||||
totals.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if ids != ref_ids {
|
||||
eprintln!("EQUIVALENCE FAILURE: {mode} walk drained different ids");
|
||||
failures += 1;
|
||||
}
|
||||
per_page = times;
|
||||
}
|
||||
let ms = median(totals);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<8} {:>11.1} {:>11.2} {:>8}",
|
||||
mode,
|
||||
ms,
|
||||
median(per_page.clone()),
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
for d in drive_ids {
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(d)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
}
|
||||
let _ = sqlx::query("DELETE FROM storage.role_grants WHERE subject_id = $1")
|
||||
.bind(caller)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
|
||||
if failures > 0 {
|
||||
eprintln!("\n{failures} equivalence failures — the NEW shape is NOT safe to adopt");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
//! S3 chunk-PUT benchmark — HEAD-before-PUT vs unconditional PUT.
|
||||
//!
|
||||
//! `DedupService::settle_batch` writes every NEW chunk of every upload via
|
||||
//! `put_blob_from_bytes_unsynced`. S3/Azure never overrode it, so the trait
|
||||
//! default routed it through `put_blob_from_bytes`, whose "idempotent" HEAD
|
||||
//! probe made every chunk write pay 2 request round-trips. Content-addressed
|
||||
//! keys make re-PUTs overwrite-safe, so the new override PUTs directly.
|
||||
//!
|
||||
//! The stub S3 endpoint (in-process axum, per-request latency injection)
|
||||
//! counts HEAD/PUT requests:
|
||||
//! BEFORE — put_blob_from_bytes (HEAD 404 + PUT per chunk)
|
||||
//! AFTER — put_blob_from_bytes_unsynced (PUT per chunk)
|
||||
//!
|
||||
//! Section 2 measures the removed Azure `data.to_vec()` copy in isolation.
|
||||
//!
|
||||
//! Gates: AFTER request count == chunks (vs 2x), AFTER wall < BEFORE wall.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_s3_put
|
||||
//! Tunables: BENCH_CHUNKS (500), BENCH_CHUNK_KB (256), BENCH_CONCURRENCY (8),
|
||||
//! BENCH_RTT_MS (10)
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::Bytes;
|
||||
use oxicloud::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use oxicloud::common::config::S3StorageConfig;
|
||||
use oxicloud::infrastructure::services::s3_blob_backend::S3BlobBackend;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct Counters {
|
||||
heads: Arc<AtomicU64>,
|
||||
puts: Arc<AtomicU64>,
|
||||
}
|
||||
|
||||
async fn stub_s3(latency: Duration, counters: Counters) -> String {
|
||||
use axum::http::{Method, StatusCode};
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.expect("bind");
|
||||
let addr = listener.local_addr().expect("addr");
|
||||
let app = axum::Router::new().fallback(move |req: axum::extract::Request| {
|
||||
let counters = counters.clone();
|
||||
async move {
|
||||
tokio::time::sleep(latency).await;
|
||||
match *req.method() {
|
||||
Method::HEAD => {
|
||||
counters.heads.fetch_add(1, Ordering::Relaxed);
|
||||
StatusCode::NOT_FOUND
|
||||
}
|
||||
Method::PUT => {
|
||||
// Drain the body like a real endpoint would.
|
||||
let _ = axum::body::to_bytes(req.into_body(), usize::MAX).await;
|
||||
counters.puts.fetch_add(1, Ordering::Relaxed);
|
||||
StatusCode::OK
|
||||
}
|
||||
_ => StatusCode::OK,
|
||||
}
|
||||
}
|
||||
});
|
||||
tokio::spawn(async move {
|
||||
axum::serve(listener, app).await.expect("serve");
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
async fn drive(
|
||||
backend: Arc<S3BlobBackend>,
|
||||
chunks: usize,
|
||||
chunk_kb: usize,
|
||||
concurrency: usize,
|
||||
unsynced: bool,
|
||||
) -> f64 {
|
||||
let payload = Bytes::from(vec![0x5au8; chunk_kb * 1024]);
|
||||
let sem = Arc::new(tokio::sync::Semaphore::new(concurrency));
|
||||
let t = Instant::now();
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for i in 0..chunks {
|
||||
let b = backend.clone();
|
||||
let p = payload.clone();
|
||||
let sem = sem.clone();
|
||||
set.spawn(async move {
|
||||
let _permit = sem.acquire().await.expect("sem");
|
||||
let hash = format!("{i:064x}");
|
||||
let n = if unsynced {
|
||||
b.put_blob_from_bytes_unsynced(&hash, p).await.expect("put")
|
||||
} else {
|
||||
b.put_blob_from_bytes(&hash, p).await.expect("put")
|
||||
};
|
||||
assert_eq!(n as usize, chunk_kb * 1024);
|
||||
});
|
||||
}
|
||||
while let Some(r) = set.join_next().await {
|
||||
r.expect("join");
|
||||
}
|
||||
t.elapsed().as_secs_f64() * 1000.0
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let chunks: usize = env_or("BENCH_CHUNKS", 500);
|
||||
let chunk_kb: usize = env_or("BENCH_CHUNK_KB", 256);
|
||||
let concurrency: usize = env_or("BENCH_CONCURRENCY", 8);
|
||||
let rtt_ms: u64 = env_or("BENCH_RTT_MS", 10);
|
||||
|
||||
let counters = Counters::default();
|
||||
let endpoint = stub_s3(Duration::from_millis(rtt_ms), counters.clone()).await;
|
||||
let backend = Arc::new(S3BlobBackend::new(&S3StorageConfig {
|
||||
endpoint_url: Some(endpoint),
|
||||
bucket: "bench".into(),
|
||||
region: "us-east-1".into(),
|
||||
access_key: "bench".into(),
|
||||
secret_key: "bench".into(),
|
||||
force_path_style: true,
|
||||
}));
|
||||
|
||||
println!(
|
||||
"# {chunks} x {chunk_kb} KiB chunk PUTs at concurrency {concurrency}, {rtt_ms} ms/request stub"
|
||||
);
|
||||
println!(
|
||||
"{:<26} {:>10} {:>8} {:>8} {:>8}",
|
||||
"variant", "wall ms", "HEADs", "PUTs", "vs OLD"
|
||||
);
|
||||
|
||||
// BEFORE: the trait-default route (put_blob_from_bytes = HEAD + PUT).
|
||||
let before = drive(backend.clone(), chunks, chunk_kb, concurrency, false).await;
|
||||
let before_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let before_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<26} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
"BEFORE (HEAD+PUT)", before, before_heads, before_puts, "1.0x"
|
||||
);
|
||||
|
||||
// AFTER: the unsynced override (PUT only).
|
||||
let after = drive(backend.clone(), chunks, chunk_kb, concurrency, true).await;
|
||||
let after_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let after_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<26} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
"AFTER (PUT only)",
|
||||
after,
|
||||
after_heads,
|
||||
after_puts,
|
||||
format!("{:.1}x", before / after)
|
||||
);
|
||||
|
||||
// ── Section 2: the removed Azure to_vec() copy, in isolation ───────
|
||||
let mb = 4;
|
||||
let data = Bytes::from(vec![0x77u8; mb * 1024 * 1024]);
|
||||
let reps = 200;
|
||||
let t = Instant::now();
|
||||
for _ in 0..reps {
|
||||
let v = data.to_vec();
|
||||
std::hint::black_box(&v);
|
||||
}
|
||||
let copy_ms = t.elapsed().as_secs_f64() * 1000.0 / reps as f64;
|
||||
println!(
|
||||
"\n# [2] removed Azure per-chunk copy: to_vec() of {mb} MiB = {copy_ms:.2} ms + {mb} MiB transient alloc per chunk"
|
||||
);
|
||||
|
||||
// ── Gates ───────────────────────────────────────────────────────────
|
||||
if after_heads != 0 || after_puts != chunks as u64 {
|
||||
eprintln!(
|
||||
"GATE FAIL: AFTER issued {after_heads} HEADs / {after_puts} PUTs (expected 0 / {chunks})"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if after >= before {
|
||||
eprintln!(
|
||||
"GATE FAIL: AFTER ({after:.0} ms) not faster than BEFORE ({before:.0} ms) — rollback"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!(
|
||||
"GATE PASS: {}-request walk -> {} requests, {:.1}x faster",
|
||||
before_heads + before_puts,
|
||||
after_puts,
|
||||
before / after
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,361 @@
|
||||
//! Search-results cache memory benchmark — entry-count bound vs byte bound.
|
||||
//!
|
||||
//! The search cache keys pages by user × query × offset × limit, and each
|
||||
//! page holds up to 500 enriched rows (`MAX_SEARCH_LIMIT`) of owned Strings.
|
||||
//! Bounded by ENTRY COUNT (the old scheme: `max_capacity(1000)` + TTL), a
|
||||
//! burst of keystrokes/pages/users could pin ~300 MB of invisible RSS for
|
||||
//! the 5-minute TTL. Bounded by BYTES (a `weigher` + 32 MiB budget — the
|
||||
//! same pattern as the file-content and dedup-manifest caches), retention
|
||||
//! can never exceed the budget.
|
||||
//!
|
||||
//! Two sub-phases over the same synthetic corpus (1,000 pages × 500 rows,
|
||||
//! ~150-char paths, realistic field contents):
|
||||
//! * BEFORE — a moka cache configured exactly as the old production wiring
|
||||
//! (entry-count 1000 + 300 s TTL).
|
||||
//! * AFTER — `build_search_results_cache(...)`, the *identical* function
|
||||
//! production now uses (weigher + 32 MiB + 300 s TTL).
|
||||
//!
|
||||
//! Reported per phase: entries retained, retained bytes (recomputed with the
|
||||
//! production weigher after `run_pending_tasks`), best-effort process memory
|
||||
//! (`VmHWM`/`VmRSS` from /proc/self/status), and hot-key `get()` p50 over
|
||||
//! 100k reads (proves the weigher — which only runs on insert — does not
|
||||
//! slow reads).
|
||||
//!
|
||||
//! NOTE on RSS: `VmHWM` is a monotonic high-water mark and the allocator may
|
||||
//! keep freed pages, so the AFTER phase (which runs second, after a full
|
||||
//! drop of the BEFORE cache) cannot show a peak below the BEFORE peak.
|
||||
//! Treat the RSS columns as best-effort corroboration; the authoritative
|
||||
//! metric is the weigher-recomputed retained bytes.
|
||||
//!
|
||||
//! Gates (exit code 1 on failure):
|
||||
//! * AFTER retained bytes ≤ 32 MiB budget
|
||||
//! * BEFORE retained bytes ≥ 8× the budget (measured ≈9–10×)
|
||||
//! * AFTER get() p50 within 20% of BEFORE
|
||||
//!
|
||||
//! No Postgres needed.
|
||||
//! Run: `cargo run --release --features bench --example bench_search_cache_mem`
|
||||
|
||||
use std::hint::black_box;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use oxicloud::application::dtos::search_dto::{SearchFileResultDto, SearchResultsDto};
|
||||
use oxicloud::application::services::search_service::{
|
||||
build_search_results_cache, search_results_entry_weight,
|
||||
};
|
||||
|
||||
/// Distinct cached pages inserted per phase (≈ users × queries × pages).
|
||||
const ENTRIES: u64 = 1_000;
|
||||
/// Rows per page — the handler's `MAX_SEARCH_LIMIT` clamp.
|
||||
const ROWS_PER_ENTRY: usize = 500;
|
||||
/// Production TTL (unchanged by the fix).
|
||||
const TTL_SECS: u64 = 300;
|
||||
/// The old production bound: 1000 ENTRIES, blind to entry size.
|
||||
const BEFORE_MAX_ENTRIES: u64 = 1_000;
|
||||
/// The new production bound: 32 MiB of weighed bytes.
|
||||
const AFTER_MAX_BYTES: u64 = 32 * 1024 * 1024;
|
||||
/// Hot-key reads per phase for the p50 latency comparison.
|
||||
const GETS: usize = 100_000;
|
||||
|
||||
const MIB: f64 = 1024.0 * 1024.0;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Deterministic synthetic corpus (no rand dependency)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Tiny xorshift64 PRNG — fast, deterministic, no dependency.
|
||||
fn xorshift(state: &mut u64) -> u64 {
|
||||
let mut x = *state;
|
||||
x ^= x << 13;
|
||||
x ^= x >> 7;
|
||||
x ^= x << 17;
|
||||
*state = x;
|
||||
x
|
||||
}
|
||||
|
||||
/// Lowercase-hex string of `chars` nibbles.
|
||||
fn pseudo_hex(state: &mut u64, chars: usize) -> String {
|
||||
let mut s = String::with_capacity(chars);
|
||||
while s.len() < chars {
|
||||
let block = format!("{:016x}", xorshift(state));
|
||||
let take = (chars - s.len()).min(16);
|
||||
s.push_str(&block[..take]);
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
/// 36-char UUID-shaped string (8-4-4-4-12), like the real `Uuid::to_string()`
|
||||
/// ids that populate `SearchFileResultDto::id` / `folder_id`.
|
||||
fn pseudo_uuid(state: &mut u64) -> String {
|
||||
let h = pseudo_hex(state, 32);
|
||||
format!(
|
||||
"{}-{}-{}-{}-{}",
|
||||
&h[0..8],
|
||||
&h[8..12],
|
||||
&h[12..16],
|
||||
&h[16..20],
|
||||
&h[20..32]
|
||||
)
|
||||
}
|
||||
|
||||
/// One synthetic 500-row search page with realistic field contents:
|
||||
/// UUID ids, ~30-char names, ~150-char nested drive paths, real MIME types,
|
||||
/// 64-hex BLAKE3 blob hashes, icon/category metadata, and a content-index
|
||||
/// snippet on every 8th row.
|
||||
fn synth_entry(idx: u64) -> Arc<SearchResultsDto> {
|
||||
const MIMES: [&str; 4] = [
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"image/jpeg",
|
||||
"text/markdown",
|
||||
];
|
||||
const SNIPPET: &str = "…the quarterly numbers show a steady increase in storage usage \
|
||||
across all departments, with the engineering share growing fastest and…";
|
||||
|
||||
let mut rng = idx.wrapping_mul(0x9E3779B97F4A7C15) | 1;
|
||||
let mut files = Vec::with_capacity(ROWS_PER_ENTRY);
|
||||
for row in 0..ROWS_PER_ENTRY {
|
||||
let name = format!(
|
||||
"quarterly_report_{:04}_rev{:03}.pdf",
|
||||
xorshift(&mut rng) % 10_000,
|
||||
row % 1_000
|
||||
);
|
||||
let path = format!(
|
||||
"/drives/{}/Departments/Engineering/Projects/oxicloud-benchmarks/2026/Q{}/weekly-sync-notes/attachments/{}",
|
||||
pseudo_uuid(&mut rng),
|
||||
row % 4 + 1,
|
||||
name
|
||||
);
|
||||
let content_hit = row % 8 == 0;
|
||||
let match_source = if content_hit { "content" } else { "name" };
|
||||
files.push(SearchFileResultDto {
|
||||
id: pseudo_uuid(&mut rng),
|
||||
name,
|
||||
path,
|
||||
size: 831_942,
|
||||
mime_type: MIMES[row % MIMES.len()].to_string(),
|
||||
folder_id: Some(pseudo_uuid(&mut rng)),
|
||||
created_at: 1_752_700_000,
|
||||
modified_at: 1_752_800_000,
|
||||
relevance_score: 50,
|
||||
size_formatted: "812.4 KB".to_string(),
|
||||
icon_class: "fas fa-file-pdf".to_string(),
|
||||
icon_special_class: "pdf-icon".to_string(),
|
||||
category: "document".to_string(),
|
||||
blob_hash: pseudo_hex(&mut rng, 64),
|
||||
snippet: content_hit.then(|| SNIPPET.to_string()),
|
||||
match_source: Some(match_source.to_string()),
|
||||
});
|
||||
}
|
||||
|
||||
Arc::new(SearchResultsDto::new(
|
||||
files,
|
||||
Vec::new(),
|
||||
ROWS_PER_ENTRY,
|
||||
0,
|
||||
Some(12_345),
|
||||
3,
|
||||
"relevance".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Best-effort process memory (Linux /proc; "n/a" elsewhere)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Read a kB-valued field (`VmHWM`, `VmRSS`) from /proc/self/status.
|
||||
fn status_kb(field: &str) -> Option<u64> {
|
||||
let text = std::fs::read_to_string("/proc/self/status").ok()?;
|
||||
text.lines()
|
||||
.find(|l| l.starts_with(field))
|
||||
.and_then(|l| l.split_whitespace().nth(1))
|
||||
.and_then(|kb| kb.parse().ok())
|
||||
}
|
||||
|
||||
fn fmt_kb(v: Option<u64>) -> String {
|
||||
match v {
|
||||
Some(kb) => format!("{:.1} MiB", kb as f64 / 1024.0),
|
||||
None => "n/a".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn fmt_kb_delta(start: Option<u64>, end: Option<u64>) -> String {
|
||||
match (start, end) {
|
||||
(Some(s), Some(e)) => format!("{:+.1} MiB", (e as f64 - s as f64) / 1024.0),
|
||||
_ => "n/a".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase runner
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
struct PhaseReport {
|
||||
retained_entries: u64,
|
||||
retained_bytes: u64,
|
||||
hwm_start_kb: Option<u64>,
|
||||
hwm_end_kb: Option<u64>,
|
||||
rss_start_kb: Option<u64>,
|
||||
rss_end_kb: Option<u64>,
|
||||
p50_get_ns: u64,
|
||||
}
|
||||
|
||||
/// Insert the full corpus, settle the cache, then measure retention and
|
||||
/// hot-key read latency. Identical for both variants — only the cache
|
||||
/// configuration differs.
|
||||
async fn run_phase(cache: &moka::future::Cache<u64, Arc<SearchResultsDto>>) -> PhaseReport {
|
||||
let hwm_start_kb = status_kb("VmHWM");
|
||||
let rss_start_kb = status_kb("VmRSS");
|
||||
|
||||
for i in 0..ENTRIES {
|
||||
cache.insert(i, synth_entry(i)).await;
|
||||
// Let eviction run as it would under live traffic, so evicted pages
|
||||
// are actually freed instead of piling up in moka's pending queue.
|
||||
if i % 64 == 0 {
|
||||
cache.run_pending_tasks().await;
|
||||
}
|
||||
}
|
||||
cache.run_pending_tasks().await;
|
||||
|
||||
let retained_entries = cache.entry_count();
|
||||
// Recompute retained bytes with the production weigher — for the BEFORE
|
||||
// variant this is exactly the memory its entry-count bound was blind to.
|
||||
let retained_bytes: u64 = cache
|
||||
.iter()
|
||||
.map(|(k, v)| u64::from(search_results_entry_weight(&k, &v)))
|
||||
.sum();
|
||||
|
||||
// Hot-key read latency: p50 over GETS reads of one resident key.
|
||||
let hot: u64 = *cache.iter().next().expect("cache is empty after fill").0;
|
||||
for _ in 0..1_000 {
|
||||
black_box(cache.get(&hot).await); // warmup
|
||||
}
|
||||
let mut lat_ns = Vec::with_capacity(GETS);
|
||||
for _ in 0..GETS {
|
||||
let t = Instant::now();
|
||||
let v = cache.get(&hot).await;
|
||||
lat_ns.push(t.elapsed().as_nanos() as u64);
|
||||
black_box(v);
|
||||
}
|
||||
lat_ns.sort_unstable();
|
||||
let p50_get_ns = lat_ns[lat_ns.len() / 2];
|
||||
|
||||
PhaseReport {
|
||||
retained_entries,
|
||||
retained_bytes,
|
||||
hwm_start_kb,
|
||||
hwm_end_kb: status_kb("VmHWM"),
|
||||
rss_start_kb,
|
||||
rss_end_kb: status_kb("VmRSS"),
|
||||
p50_get_ns,
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let entry_weight = u64::from(search_results_entry_weight(&0, &synth_entry(0)));
|
||||
println!("\n###########################################################");
|
||||
println!("# Search-results cache: entry-count bound vs byte bound");
|
||||
println!(
|
||||
"# corpus: {ENTRIES} pages x {ROWS_PER_ENTRY} rows, ~{:.0} KiB/page (weigher)",
|
||||
entry_weight as f64 / 1024.0
|
||||
);
|
||||
println!(
|
||||
"# BEFORE: max_capacity({BEFORE_MAX_ENTRIES}) entries + {TTL_SECS}s TTL (old di.rs wiring)"
|
||||
);
|
||||
println!(
|
||||
"# AFTER : build_search_results_cache({TTL_SECS}, {} MiB) — production fn",
|
||||
AFTER_MAX_BYTES as f64 / MIB
|
||||
);
|
||||
println!("###########################################################\n");
|
||||
|
||||
// --- Phase 1: BEFORE (entry-count bound, exactly the old wiring) ---
|
||||
let before_cache: moka::future::Cache<u64, Arc<SearchResultsDto>> =
|
||||
moka::future::Cache::builder()
|
||||
.max_capacity(BEFORE_MAX_ENTRIES)
|
||||
.time_to_live(Duration::from_secs(TTL_SECS))
|
||||
.build();
|
||||
let before = run_phase(&before_cache).await;
|
||||
// Full drop between phases so the AFTER numbers never sit on top of the
|
||||
// BEFORE cache's live memory.
|
||||
drop(before_cache);
|
||||
|
||||
// --- Phase 2: AFTER (weigher + byte budget, the production builder) ---
|
||||
let after_cache = build_search_results_cache(TTL_SECS, AFTER_MAX_BYTES);
|
||||
let after = run_phase(&after_cache).await;
|
||||
|
||||
// --- Report ---
|
||||
println!("| metric | BEFORE (1000 entries + TTL) | AFTER (weigher + 32 MiB) |");
|
||||
println!("|---|---|---|");
|
||||
println!(
|
||||
"| entries retained | {} | {} |",
|
||||
before.retained_entries, after.retained_entries
|
||||
);
|
||||
println!(
|
||||
"| retained bytes (weigher) | {:.1} MiB | {:.1} MiB |",
|
||||
before.retained_bytes as f64 / MIB,
|
||||
after.retained_bytes as f64 / MIB
|
||||
);
|
||||
println!(
|
||||
"| byte budget | n/a (entry-count bound) | {:.0} MiB |",
|
||||
AFTER_MAX_BYTES as f64 / MIB
|
||||
);
|
||||
println!(
|
||||
"| VmHWM phase delta (best-effort) | {} | {} |",
|
||||
fmt_kb_delta(before.hwm_start_kb, before.hwm_end_kb),
|
||||
fmt_kb_delta(after.hwm_start_kb, after.hwm_end_kb)
|
||||
);
|
||||
println!(
|
||||
"| VmRSS start -> end | {} -> {} | {} -> {} |",
|
||||
fmt_kb(before.rss_start_kb),
|
||||
fmt_kb(before.rss_end_kb),
|
||||
fmt_kb(after.rss_start_kb),
|
||||
fmt_kb(after.rss_end_kb)
|
||||
);
|
||||
println!(
|
||||
"| get() p50, hot key ({GETS} reads) | {} ns | {} ns |",
|
||||
before.p50_get_ns, after.p50_get_ns
|
||||
);
|
||||
println!(
|
||||
"\nRSS note: VmHWM is monotonic and the allocator may retain freed pages, \
|
||||
so the AFTER phase (running second) cannot peak below the BEFORE peak; \
|
||||
the weigher-recomputed retained bytes are the authoritative comparison."
|
||||
);
|
||||
|
||||
// --- Gates ---
|
||||
let before_ratio = before.retained_bytes as f64 / AFTER_MAX_BYTES as f64;
|
||||
let lat_ratio = after.p50_get_ns as f64 / before.p50_get_ns.max(1) as f64;
|
||||
let gate_after_bounded = after.retained_bytes <= AFTER_MAX_BYTES;
|
||||
let gate_before_unbounded = before_ratio >= 8.0;
|
||||
let gate_latency = lat_ratio <= 1.2;
|
||||
|
||||
println!("\n| gate | condition | measured | result |");
|
||||
println!("|---|---|---|---|");
|
||||
println!(
|
||||
"| AFTER bounded | retained <= 32 MiB budget | {:.1} MiB | {} |",
|
||||
after.retained_bytes as f64 / MIB,
|
||||
if gate_after_bounded { "PASS" } else { "FAIL" }
|
||||
);
|
||||
println!(
|
||||
"| BEFORE unbounded | retained >= 8x budget (~10x expected) | {before_ratio:.1}x | {} |",
|
||||
if gate_before_unbounded {
|
||||
"PASS"
|
||||
} else {
|
||||
"FAIL"
|
||||
}
|
||||
);
|
||||
println!(
|
||||
"| read parity | AFTER p50 <= 1.2x BEFORE p50 | {lat_ratio:.2}x | {} |",
|
||||
if gate_latency { "PASS" } else { "FAIL" }
|
||||
);
|
||||
|
||||
if !(gate_after_bounded && gate_before_unbounded && gate_latency) {
|
||||
eprintln!("\nbench_search_cache_mem: GATE FAILURE");
|
||||
std::process::exit(1);
|
||||
}
|
||||
println!("\nAll gates passed.");
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
//! Upload spool/assembly I/O benchmark — buffer sizing on the chunk paths.
|
||||
//!
|
||||
//! Section 1 — assembly read (`stream_from_files`): every completed chunked
|
||||
//! upload is read back once, part file by part file, through
|
||||
//! `ReaderStream::with_capacity(file, N)`. Each poll is one blocking-pool
|
||||
//! dispatch + one read(2) of N bytes; the shipped capacity was 64 KiB while
|
||||
//! every other blob read path uses 256 KiB+. Sweeps N over
|
||||
//! 64K/256K/512K/1M and reports wall time + read syscalls.
|
||||
//!
|
||||
//! Section 2 — chunk spool write (`stream_body_to_path`): the PUT handlers
|
||||
//! wrote each HTTP frame (~16-64 KiB) straight to a bare tokio File — one
|
||||
//! blocking-pool dispatch + write(2) per frame. Compares that against the
|
||||
//! adopted `BufWriter::with_capacity(512 KiB)`.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_upload_spool
|
||||
//! Tunables: BENCH_PARTS (16), BENCH_PART_MB (10), BENCH_FRAME_KB (16),
|
||||
//! BENCH_SPOOL_MB (10), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
use bytes::Bytes;
|
||||
use futures::{StreamExt, TryStreamExt, stream};
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use tokio_util::io::ReaderStream;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// (read syscalls, write syscalls) from /proc/self/io.
|
||||
fn io_counters() -> (u64, u64) {
|
||||
let s = std::fs::read_to_string("/proc/self/io").expect("io");
|
||||
let get = |k: &str| {
|
||||
s.lines()
|
||||
.find(|l| l.starts_with(k))
|
||||
.and_then(|l| l.split_whitespace().nth(1))
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(0)
|
||||
};
|
||||
(get("syscr:"), get("syscw:"))
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
/// The `stream_from_files` shape with a parameterized capacity.
|
||||
async fn drain_parts(paths: Vec<PathBuf>, cap: usize) -> (u64, [u8; 32]) {
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let mut total = 0u64;
|
||||
let s = stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
|
||||
.and_then(|path| async move {
|
||||
tokio::fs::File::open(path)
|
||||
.await
|
||||
.map(|file| ReaderStream::with_capacity(file, cap))
|
||||
})
|
||||
.try_flatten();
|
||||
let mut s = Box::pin(s);
|
||||
while let Some(chunk) = s.next().await {
|
||||
let chunk = chunk.expect("read");
|
||||
total += chunk.len() as u64;
|
||||
hasher.update(&chunk);
|
||||
}
|
||||
(total, hasher.finalize().into())
|
||||
}
|
||||
|
||||
/// The `stream_body_to_path` inner loop: frames -> file, optionally buffered.
|
||||
async fn spool_frames(frames: &[Bytes], path: &std::path::Path, buffered: bool) {
|
||||
let file = tokio::fs::File::create(path).await.expect("create");
|
||||
if buffered {
|
||||
let mut w = tokio::io::BufWriter::with_capacity(512 * 1024, file);
|
||||
for f in frames {
|
||||
w.write_all(f).await.expect("write");
|
||||
}
|
||||
w.flush().await.expect("flush");
|
||||
} else {
|
||||
let mut w = file;
|
||||
for f in frames {
|
||||
w.write_all(f).await.expect("write");
|
||||
}
|
||||
w.flush().await.expect("flush");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let parts: usize = env_or("BENCH_PARTS", 16);
|
||||
let part_mb: usize = env_or("BENCH_PART_MB", 10);
|
||||
let frame_kb: usize = env_or("BENCH_FRAME_KB", 16);
|
||||
let spool_mb: usize = env_or("BENCH_SPOOL_MB", 10);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
|
||||
// ── Section 1: assembly read capacity sweep ─────────────────────────
|
||||
println!("# [1] assembly read: {parts} x {part_mb} MiB part files, warm page cache");
|
||||
let mut paths = Vec::with_capacity(parts);
|
||||
let payload: Vec<u8> = (0..part_mb * 1024 * 1024)
|
||||
.map(|i| (i * 31 % 251) as u8)
|
||||
.collect();
|
||||
for i in 0..parts {
|
||||
let p = dir.path().join(format!("part_{i:05}"));
|
||||
tokio::fs::write(&p, &payload).await.expect("seed part");
|
||||
paths.push(p);
|
||||
}
|
||||
let expect_total = (parts * part_mb * 1024 * 1024) as u64;
|
||||
let (_, ref_hash) = drain_parts(paths.clone(), 256 * 1024).await;
|
||||
|
||||
println!(
|
||||
"{:<10} {:>10} {:>12} {:>8}",
|
||||
"capacity", "wall ms", "read sysc", "vs 64K"
|
||||
);
|
||||
let mut base: Option<f64> = None;
|
||||
for cap in [64 * 1024, 256 * 1024, 512 * 1024, 1024 * 1024] {
|
||||
let mut walls = Vec::with_capacity(reps);
|
||||
let mut syscr = 0u64;
|
||||
for _ in 0..reps {
|
||||
let (r0, _) = io_counters();
|
||||
let t = Instant::now();
|
||||
let (total, h) = drain_parts(paths.clone(), cap).await;
|
||||
walls.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let (r1, _) = io_counters();
|
||||
syscr = r1 - r0;
|
||||
assert_eq!(total, expect_total);
|
||||
assert_eq!(h, ref_hash, "content mismatch at capacity {cap}");
|
||||
}
|
||||
let ms = median(walls);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.2}x", b / ms))
|
||||
.unwrap_or_else(|| "1.00x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!(
|
||||
"{:<10} {:>10.1} {:>12} {:>8}",
|
||||
format!("{}K", cap / 1024),
|
||||
ms,
|
||||
syscr,
|
||||
speedup
|
||||
);
|
||||
}
|
||||
|
||||
// ── Section 2: chunk spool write, per-frame vs buffered ─────────────
|
||||
let frames_n = spool_mb * 1024 / frame_kb;
|
||||
println!(
|
||||
"\n# [2] chunk spool: {frames_n} x {frame_kb} KiB frames ({spool_mb} MiB), 20 files/rep"
|
||||
);
|
||||
let frame: Bytes = Bytes::from(vec![0xabu8; frame_kb * 1024]);
|
||||
let frames: Vec<Bytes> = (0..frames_n).map(|_| frame.clone()).collect();
|
||||
|
||||
println!(
|
||||
"{:<22} {:>10} {:>12} {:>8}",
|
||||
"variant", "wall ms", "write sysc", "vs bare"
|
||||
);
|
||||
let mut base: Option<f64> = None;
|
||||
for (label, buffered) in [
|
||||
("bare File (BEFORE)", false),
|
||||
("BufWriter 512K (AFTER)", true),
|
||||
] {
|
||||
let mut walls = Vec::with_capacity(reps);
|
||||
let mut syscw = 0u64;
|
||||
for r in 0..reps {
|
||||
let (_, w0) = io_counters();
|
||||
let t = Instant::now();
|
||||
for i in 0..20 {
|
||||
let p = dir.path().join(format!("spool_{r}_{i}"));
|
||||
spool_frames(&frames, &p, buffered).await;
|
||||
tokio::fs::remove_file(&p).await.ok();
|
||||
}
|
||||
walls.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
let (_, w1) = io_counters();
|
||||
syscw = w1 - w0;
|
||||
}
|
||||
let ms = median(walls);
|
||||
let speedup = base
|
||||
.map(|b| format!("{:.2}x", b / ms))
|
||||
.unwrap_or_else(|| "1.00x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!("{label:<22} {:>10.1} {:>12} {:>8}", ms, syscw, speedup);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user