perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade
Benchmark-gated round (benches/ROUND9.md): every change carries a BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from the committed harnesses on 4 cores / local PG 16. Backend: - Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced + sync_blobs — the trait default had silently reinstated HEAD-before-PUT per chunk on decorated remote stacks, undoing ROUND3 §8. Full production stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3). - NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT (bench_nc_enrich_join, injected-latency decide-by-bench). - Search enrichment consumes its DTOs and carries the interned Arc<str> display fields end-to-end (SearchFileResultDto type change, OpenAPI shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC REPORT conversion stops re-running all three classifiers per row (bench_search_enrich). - NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs), Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser> + lazy span render (11 -> 6/build) (bench_nc_session). - Storage micro-pack: atomic create_new chunk writes (2.1x fresh), stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro). - OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0 allocs/poll, byte-identical (bench_capabilities_static). - Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive (bench_drive_is_empty). - favorites/recents row-map ROUND7 port: path/name/blob_hash moved, -2.75 allocs/row (bench_resource_row_map §2). - Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page fetch, honest verdict incl. one noise-band wash documented (bench_folder_uuid_decode). - Authz: file cascade decision decomposed into memoized folder-level decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl. new direct-grant sibling isolation, revoke-flush re-verified, full integration authz suite green (bench_thumbnail_cascade_cache). Frontend (vitest gates committed beside the code): - resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x (recipients.bench.test.ts). - ResourceList selection-prune effect skips when nothing is selected (100 -> 0 Set builds per drain) and the photos timeline reads a listener-fed mobile flag instead of matchMedia per recompute (listDerives.bench.test.ts). Verification: cargo fmt + clippy --all-features --all-targets -D warnings clean; 524 unit + 554 integration (--cfg integration_tests) tests pass; frontend npm run check clean with 293 vitest tests green. Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder (maintainer sign-off), per-page batched parent resolution, JWT-claims Arc<str>, batch_operations signature widening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn
This commit is contained in:
Claude
+218
-6
@@ -13,7 +13,20 @@
|
||||
//!
|
||||
//! Section 2 measures the removed Azure `data.to_vec()` copy in isolation.
|
||||
//!
|
||||
//! Gates: AFTER request count == chunks (vs 2x), AFTER wall < BEFORE wall.
|
||||
//! Section 3 (round 9) drives the same A/B **through the decorator stacks**
|
||||
//! (`RetryBlobBackend`, `CachedBlobBackend`, and the full production
|
||||
//! Cache(Encrypted(Retry(S3))) composition). Until round 9 neither Retry nor
|
||||
//! Cached overrode `put_blob_from_bytes_unsynced`/`sync_blobs`, so the trait
|
||||
//! default silently re-routed every decorated chunk write back through the
|
||||
//! probing synced path — undoing this bench's own Section-1 win on every
|
||||
//! remote deployment with retry or cache enabled. The BEFORE arm is the
|
||||
//! still-present synced route (`put_blob_from_bytes`, byte-identical requests
|
||||
//! to what the fallthrough produced); the AFTER arm is the now-forwarded
|
||||
//! unsynced route. A write-through equivalence gate asserts the Cached stack
|
||||
//! still populates its local cache identically on both routes.
|
||||
//!
|
||||
//! Gates: AFTER request count == chunks (vs 2x), AFTER wall < BEFORE wall,
|
||||
//! per-stack AFTER HEADs == 0, cache population identical on both routes.
|
||||
//!
|
||||
//! No Postgres. Run:
|
||||
//! cargo run --release --features bench --example bench_s3_put
|
||||
@@ -28,6 +41,9 @@ use std::time::{Duration, Instant};
|
||||
use bytes::Bytes;
|
||||
use oxicloud::application::ports::blob_storage_ports::BlobStorageBackend;
|
||||
use oxicloud::common::config::S3StorageConfig;
|
||||
use oxicloud::infrastructure::services::cached_blob_backend::{BlobCacheConfig, CachedBlobBackend};
|
||||
use oxicloud::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend;
|
||||
use oxicloud::infrastructure::services::retry_blob_backend::{RetryBlobBackend, RetryPolicy};
|
||||
use oxicloud::infrastructure::services::s3_blob_backend::S3BlobBackend;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
@@ -37,6 +53,23 @@ fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Recursively count regular files under `dir` (the blob cache shards blobs
|
||||
/// into 2-hex-char prefix subdirectories).
|
||||
fn count_files(dir: &std::path::Path) -> usize {
|
||||
let mut n = 0;
|
||||
if let Ok(entries) = std::fs::read_dir(dir) {
|
||||
for entry in entries.flatten() {
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
n += count_files(&path);
|
||||
} else {
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct Counters {
|
||||
heads: Arc<AtomicU64>,
|
||||
@@ -75,11 +108,12 @@ async fn stub_s3(latency: Duration, counters: Counters) -> String {
|
||||
}
|
||||
|
||||
async fn drive(
|
||||
backend: Arc<S3BlobBackend>,
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
chunks: usize,
|
||||
chunk_kb: usize,
|
||||
concurrency: usize,
|
||||
unsynced: bool,
|
||||
hash_prefix: &str,
|
||||
) -> f64 {
|
||||
let payload = Bytes::from(vec![0x5au8; chunk_kb * 1024]);
|
||||
let sem = Arc::new(tokio::sync::Semaphore::new(concurrency));
|
||||
@@ -89,15 +123,17 @@ async fn drive(
|
||||
let b = backend.clone();
|
||||
let p = payload.clone();
|
||||
let sem = sem.clone();
|
||||
let hash = format!("{hash_prefix}{i:060x}");
|
||||
set.spawn(async move {
|
||||
let _permit = sem.acquire().await.expect("sem");
|
||||
let hash = format!("{i:064x}");
|
||||
let n = if unsynced {
|
||||
b.put_blob_from_bytes_unsynced(&hash, p).await.expect("put")
|
||||
} else {
|
||||
b.put_blob_from_bytes(&hash, p).await.expect("put")
|
||||
};
|
||||
assert_eq!(n as usize, chunk_kb * 1024);
|
||||
// Encrypted arms return the ciphertext size (plaintext + AEAD
|
||||
// framing), so gate on >= rather than == for stack generality.
|
||||
assert!(n as usize >= chunk_kb * 1024);
|
||||
});
|
||||
}
|
||||
while let Some(r) = set.join_next().await {
|
||||
@@ -106,6 +142,81 @@ async fn drive(
|
||||
t.elapsed().as_secs_f64() * 1000.0
|
||||
}
|
||||
|
||||
/// Run BEFORE (synced route == the pre-round-9 unsynced fallthrough) and
|
||||
/// AFTER (forwarded unsynced route) through one backend stack, printing the
|
||||
/// two rows and gating AFTER on zero probe requests. `prefixes` carries the
|
||||
/// (BEFORE, AFTER) hash namespaces keeping the arms' key spaces disjoint.
|
||||
async fn stack_ab(
|
||||
label: &str,
|
||||
backend: Arc<dyn BlobStorageBackend>,
|
||||
counters: &Counters,
|
||||
chunks: usize,
|
||||
chunk_kb: usize,
|
||||
concurrency: usize,
|
||||
prefixes: (&str, &str),
|
||||
) -> (f64, f64) {
|
||||
let (prefix_before, prefix_after) = prefixes;
|
||||
let before = drive(
|
||||
backend.clone(),
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
false,
|
||||
prefix_before,
|
||||
)
|
||||
.await;
|
||||
let before_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let before_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<34} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
format!("{label} BEFORE (synced route)"),
|
||||
before,
|
||||
before_heads,
|
||||
before_puts,
|
||||
"1.0x"
|
||||
);
|
||||
|
||||
let after = drive(
|
||||
backend.clone(),
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
true,
|
||||
prefix_after,
|
||||
)
|
||||
.await;
|
||||
let after_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let after_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
"{:<34} {:>10.0} {:>8} {:>8} {:>8}",
|
||||
format!("{label} AFTER (unsynced)"),
|
||||
after,
|
||||
after_heads,
|
||||
after_puts,
|
||||
format!("{:.1}x", before / after)
|
||||
);
|
||||
|
||||
if before_heads != chunks as u64 {
|
||||
eprintln!(
|
||||
"GATE FAIL [{label}]: BEFORE issued {before_heads} HEADs (expected {chunks} — the probing route must still probe)"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if after_heads != 0 || after_puts != chunks as u64 {
|
||||
eprintln!(
|
||||
"GATE FAIL [{label}]: AFTER issued {after_heads} HEADs / {after_puts} PUTs (expected 0 / {chunks})"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
if after >= before {
|
||||
eprintln!(
|
||||
"GATE FAIL [{label}]: AFTER ({after:.0} ms) not faster than BEFORE ({before:.0} ms) — rollback"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
(before, after)
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let chunks: usize = env_or("BENCH_CHUNKS", 500);
|
||||
@@ -133,7 +244,15 @@ async fn main() {
|
||||
);
|
||||
|
||||
// BEFORE: the trait-default route (put_blob_from_bytes = HEAD + PUT).
|
||||
let before = drive(backend.clone(), chunks, chunk_kb, concurrency, false).await;
|
||||
let before = drive(
|
||||
backend.clone() as Arc<dyn BlobStorageBackend>,
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
false,
|
||||
"a0a0",
|
||||
)
|
||||
.await;
|
||||
let before_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let before_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
@@ -142,7 +261,15 @@ async fn main() {
|
||||
);
|
||||
|
||||
// AFTER: the unsynced override (PUT only).
|
||||
let after = drive(backend.clone(), chunks, chunk_kb, concurrency, true).await;
|
||||
let after = drive(
|
||||
backend.clone() as Arc<dyn BlobStorageBackend>,
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
true,
|
||||
"a0a1",
|
||||
)
|
||||
.await;
|
||||
let after_heads = counters.heads.swap(0, Ordering::Relaxed);
|
||||
let after_puts = counters.puts.swap(0, Ordering::Relaxed);
|
||||
println!(
|
||||
@@ -168,6 +295,91 @@ async fn main() {
|
||||
"\n# [2] removed Azure per-chunk copy: to_vec() of {mb} MiB = {copy_ms:.2} ms + {mb} MiB transient alloc per chunk"
|
||||
);
|
||||
|
||||
// ── Section 3: the same A/B through the decorator stacks ────────────
|
||||
println!(
|
||||
"\n# [3] decorated stacks — pre-round-9 the unsynced call fell through to the synced (probing) route"
|
||||
);
|
||||
println!(
|
||||
"{:<34} {:>10} {:>8} {:>8} {:>8}",
|
||||
"variant", "wall ms", "HEADs", "PUTs", "vs OLD"
|
||||
);
|
||||
|
||||
// Retry(S3)
|
||||
let retry_stack: Arc<dyn BlobStorageBackend> = Arc::new(RetryBlobBackend::new(
|
||||
backend.clone() as Arc<dyn BlobStorageBackend>,
|
||||
RetryPolicy::default(),
|
||||
));
|
||||
stack_ab(
|
||||
"retry(s3)",
|
||||
retry_stack,
|
||||
&counters,
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
("b0b0", "b0b1"),
|
||||
)
|
||||
.await;
|
||||
|
||||
// Cache(S3) — count cache write-through population on both routes.
|
||||
let cache_dir_a = tempfile::tempdir().expect("tempdir");
|
||||
let cached_stack: Arc<dyn BlobStorageBackend> = Arc::new(CachedBlobBackend::new(
|
||||
backend.clone() as Arc<dyn BlobStorageBackend>,
|
||||
&BlobCacheConfig {
|
||||
cache_dir: cache_dir_a.path().to_path_buf(),
|
||||
max_cache_bytes: u64::MAX,
|
||||
},
|
||||
));
|
||||
stack_ab(
|
||||
"cache(s3)",
|
||||
cached_stack,
|
||||
&counters,
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
("c0c0", "c0c1"),
|
||||
)
|
||||
.await;
|
||||
// Write-through equivalence gate: BOTH routes populated the local cache
|
||||
// (the round-9 override keeps post-upload read locality intact).
|
||||
let cached_files = count_files(cache_dir_a.path());
|
||||
if cached_files != 2 * chunks {
|
||||
eprintln!(
|
||||
"GATE FAIL [cache(s3)]: cache holds {cached_files} blobs (expected {} — write-through must populate on BOTH routes)",
|
||||
2 * chunks
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
// Full production composition: Cache(Encrypted(Retry(S3))).
|
||||
let cache_dir_b = tempfile::tempdir().expect("tempdir");
|
||||
let full_stack: Arc<dyn BlobStorageBackend> = Arc::new(CachedBlobBackend::new(
|
||||
Arc::new(EncryptedBlobBackend::new(
|
||||
Arc::new(RetryBlobBackend::new(
|
||||
backend.clone() as Arc<dyn BlobStorageBackend>,
|
||||
RetryPolicy::default(),
|
||||
)),
|
||||
&[0x42u8; 32],
|
||||
)),
|
||||
&BlobCacheConfig {
|
||||
cache_dir: cache_dir_b.path().to_path_buf(),
|
||||
max_cache_bytes: u64::MAX,
|
||||
},
|
||||
));
|
||||
let (full_before, full_after) = stack_ab(
|
||||
"cache(enc(retry(s3)))",
|
||||
full_stack,
|
||||
&counters,
|
||||
chunks,
|
||||
chunk_kb,
|
||||
concurrency,
|
||||
("d0d0", "d0d1"),
|
||||
)
|
||||
.await;
|
||||
println!(
|
||||
"# full stack: a {chunks}-chunk upload sheds {} probe round-trips ({:.0} -> {:.0} ms at {rtt_ms} ms RTT)",
|
||||
chunks, full_before, full_after
|
||||
);
|
||||
|
||||
// ── Gates ───────────────────────────────────────────────────────────
|
||||
if after_heads != 0 || after_puts != chunks as u64 {
|
||||
eprintln!(
|
||||
|
||||
Reference in New Issue
Block a user