cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

perf(round26): drive-policy JSONB decode, CachedBlobBackend shard-dir pre-create, delta-upload foldhash

Browse Source 
Three benchmark-gated optimizations from the ROUND25 backlog (benches/ROUND26.md),
each with a BEFORE/AFTER gate that rolls back if AFTER does not beat BEFORE:

- P1 drive_pg_repository policy reads: decode d.policies through
  sqlx::types::Json<DrivePolicies> (one from_slice over the raw JSONB bytes)
  instead of a throwaway serde_json::Value DOM + DrivePolicies::from_value —
  6 -> 0 allocs/read, 2.77x wall. A shared policies_from_row helper preserves
  the lenient unwrap_or_default fallback (malformed bag -> all-false).
- D1 CachedBlobBackend: pre-create the 256 {00..ff} shard dirs at initialize()
  (mirroring LocalBlobBackend, reusing HEX_PREFIXES) and drop the redundant
  per-write create_dir_all on already-existing shards — ~45us + a blocking-pool
  dispatch removed per cache write on cached-remote deployments.
- G1 delta-upload have/need hash sets (distinct_hashes, authorize_chunk_download):
  SipHash -> foldhash::quality::RandomState — a fast hasher that stays
  DoS-resistant via a per-instance random seed, the required property for the
  attacker-controlled 64-hex client hashes — 2.37x wall on a 40k-hash
  negotiation. foldhash was already in the lockfile transitively (hashbrown).

Tested and REVERTED (kept as-is): moving the moka eviction unlink off the reactor
via spawn_blocking. The benchmark refuted it — on the local cache dir the
spawn_blocking dispatch (~20us) costs more than the inline unlink (~7us) it would
replace. See ROUND26.md §D2.

Adds bench_round26_{micro,diskio,hasher} (counting allocator / async wall / wall).
Verified: cargo fmt clean, cargo clippy --features bench -D warnings clean,
cargo test --lib --features bench = 529 passed / 0 failed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L8gs91AhmazoxMsDcNk3KT
This commit is contained in:
Claude
2026-07-21 01:05:12 +00:00
parent e8e4ef4b15
commit 5b2bb8f883
10 changed files with 631 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/bench_round26_diskio.rs
+84
View File
@@ -0,0 +1,84 @@
//! Round-26 disk-I/O pack (no Postgres) — async wall on a tmpfs-backed tempdir.
//!
//! [D1] `CachedBlobBackend::initialize` creates ONLY `cache_dir`, never the 256
//! `{00..ff}` shard dirs (the line-122 comment claims otherwise), so each
//! of the three cache-write sites re-runs `tokio::fs::create_dir_all(parent)`
//! on the hot path — a wasted `mkdirat(EEXIST)` + component stat + a
//! blocking-pool dispatch per chunk write on cached-remote deployments.
//! AFTER pre-creates the shard dirs at init (mirroring
//! `LocalBlobBackend::initialize`) and drops the per-write call. Gate:
//! AFTER wall (per write) strictly lower than BEFORE (the redundant
//! create_dir_all).
//!
//! [D2] TESTED AND REVERTED — see benches/ROUND26.md. Moving the moka
//! eviction-listener unlink off the reactor via `spawn_blocking` was
//! refuted by the benchmark: on the local cache dir (fast unlink ~7 µs)
//! the `spawn_blocking` dispatch (~20 µs) costs MORE on the reactor than
//! the inline `std::fs::remove_file` it replaces. The original inline
//! unlink ("a quick unlink on the inserting task's thread") is correct
//! for the fast-local-cache case; kept as-is.
//!
//! Run:
//! RUSTFLAGS="-C target-cpu=x86-64-v3" \
//! cargo run --release --features bench --example bench_round26_diskio
//! Tunables (env): D1_ITERS (20000)
use std::env;
use std::time::Instant;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn gate(tag: &str, metric: &str, before: f64, after: f64) {
if !(after < before) {
eprintln!("GATE FAIL [{tag}] {metric}: AFTER {after} !< BEFORE {before} — rollback");
std::process::exit(1);
}
}
// ── [D1] redundant create_dir_all on a warm shard vs skip ────────────────────
async fn section_d1() {
let iters: u64 = env_or("D1_ITERS", 20_000);
let dir = tempfile::tempdir().expect("tempdir");
let shard = dir.path().join("ab");
// Shard pre-created once (what AFTER's initialize does).
tokio::fs::create_dir_all(&shard).await.unwrap();
// warm
let _ = tokio::fs::create_dir_all(&shard).await;
// BEFORE: per-write create_dir_all(parent) on the already-existing shard.
let t = Instant::now();
for _ in 0..iters {
let _ = tokio::fs::create_dir_all(&shard).await;
}
let before_ns = t.elapsed().as_nanos() as f64 / iters as f64;
// AFTER: shard guaranteed present at init → the write path skips the call.
let t = Instant::now();
for _ in 0..iters {
std::hint::black_box(&shard);
}
let after_ns = t.elapsed().as_nanos() as f64 / iters as f64;
println!("## [D1] cache-write create_dir_all on a warm shard");
println!("| arm | ns/write |");
println!("| BEFORE | {before_ns:>8.1} |");
println!("| AFTER | {after_ns:>8.1} |");
println!(
"# {:.1}x — redundant create_dir_all removed per cache write\n",
before_ns / after_ns.max(0.001)
);
gate("D1", "ns/write", before_ns, after_ns);
}
#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() {
println!("# Round-26 disk-I/O pack\n");
section_d1().await;
println!("All Round-26 disk-I/O sections passed their gate.");
}
examples/bench_round26_hasher.rs
+148
View File
@@ -0,0 +1,148 @@
//! Round-26 hasher pack (no Postgres) — wall-gated, since a hasher swap changes
//! 0 allocations (the deterministic alloc counter can't score it).
//!
//! [G1] The delta-upload "have/need" negotiation builds `HashSet`s over up to
//! `max_chunk_count()` client-supplied 64-hex BLAKE3 hashes per request
//! (`distinct_hashes`, `authorize_chunk_download`'s `distinct_seen`).
//! std `HashSet` uses SipHash-1-3 (DoS-resistant but ~2-4x slower on
//! short keys). AFTER uses `foldhash::quality::RandomState` — a faster
//! non-cryptographic hash that STAYS DoS-resistant because it is
//! per-instance random-seeded (the required property for these
//! attacker-controlled inputs — not `FxHash`/fixed-seed). foldhash is
//! already in the lockfile transitively (hashbrown), so it adds no crate.
//! Gate: AFTER wall (build set + membership scan) strictly lower, AND
//! two RandomState instances must seed differently (DoS resistance kept).
//!
//! Run:
//! RUSTFLAGS="-C target-cpu=x86-64-v3" \
//! cargo run --release --features bench --example bench_round26_hasher
//! Tunables (env): G1_HASHES (40000), G1_PASSES (50)
use std::collections::HashSet;
use std::env;
use std::hint::black_box;
use std::time::Instant;
use foldhash::quality::RandomState;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn p50(mut s: Vec<f64>) -> f64 {
s.sort_by(|a, b| a.partial_cmp(b).unwrap());
s[s.len() / 2]
}
fn gate(tag: &str, metric: &str, before: f64, after: f64) {
if !(after < before) {
eprintln!("GATE FAIL [{tag}] {metric}: AFTER {after} !< BEFORE {before} — rollback");
std::process::exit(1);
}
}
/// Deterministic 64-hex "hash" strings (mirror a BLAKE3 chunk hash).
fn hashes(n: usize) -> Vec<String> {
(0..n)
.map(|i| {
let mut s = String::with_capacity(64);
for k in 0..8 {
use std::fmt::Write;
let _ = write!(
s,
"{:08x}",
(i as u64).wrapping_mul(2_654_435_761).wrapping_add(k)
);
}
s
})
.collect()
}
fn main() {
println!("# Round-26 hasher pack\n");
let n: usize = env_or("G1_HASHES", 40_000);
let passes: usize = env_or("G1_PASSES", 50);
let keys = hashes(n);
// DoS-safety: two RandomState instances must NOT hash identically (random
// per-instance seed — precomputed-collision attacks stay infeasible).
{
use std::hash::{BuildHasher, Hasher};
let (a, b) = (RandomState::default(), RandomState::default());
let mut ha = a.build_hasher();
let mut hb = b.build_hasher();
std::hash::Hash::hash(&keys[0], &mut ha);
std::hash::Hash::hash(&keys[0], &mut hb);
if ha.finish() == hb.finish() {
eprintln!("GATE FAIL [G1] two RandomState seeds produced the same hash — not DoS-safe");
std::process::exit(1);
}
}
// Equivalence: both build the same distinct set + same membership answers.
let sip: HashSet<&str> = keys.iter().map(|s| s.as_str()).collect();
let fold: HashSet<&str, RandomState> = keys.iter().map(|s| s.as_str()).collect();
assert_eq!(sip.len(), fold.len(), "G1 distinct count differs");
for k in &keys {
assert_eq!(
sip.contains(k.as_str()),
fold.contains(k.as_str()),
"G1 membership differs"
);
}
let work_sip = || {
let set: HashSet<&str> = keys.iter().map(|s| s.as_str()).collect();
let mut hits = 0usize;
for k in &keys {
if set.contains(k.as_str()) {
hits += 1;
}
}
black_box(hits)
};
let work_fold = || {
let set: HashSet<&str, RandomState> =
HashSet::with_capacity_and_hasher(keys.len(), RandomState::default());
let mut set = set;
for k in &keys {
set.insert(k.as_str());
}
let mut hits = 0usize;
for k in &keys {
if set.contains(k.as_str()) {
hits += 1;
}
}
black_box(hits)
};
black_box(work_sip());
black_box(work_fold());
let mut before = Vec::new();
let mut after = Vec::new();
for _ in 0..passes {
let t = Instant::now();
black_box(work_sip());
before.push(t.elapsed().as_secs_f64() * 1e3);
let t = Instant::now();
black_box(work_fold());
after.push(t.elapsed().as_secs_f64() * 1e3);
}
let b = p50(before);
let a = p50(after);
println!("## [G1] delta-upload hash set: SipHash vs foldhash::quality ({n} hashes)");
println!("| arm | p50 ms (build+scan) |");
println!("| BEFORE (SipHash) | {b:>10.3} |");
println!("| AFTER (foldhash) | {a:>10.3} |");
println!(
"# {:.2}x wall — DoS resistance retained (random per-instance seed)\n",
b / a.max(1e-9)
);
gate("G1", "p50 ms", b, a);
println!("Round-26 hasher section passed its gate.");
}
examples/bench_round26_micro.rs
+153
View File
@@ -0,0 +1,153 @@
//! Round-26 CPU/alloc micro-pack (no Postgres).
//!
//! Same rule as ROUND2–25: each section is BEFORE (verbatim replica of the
//! shipped-before shape) vs AFTER (replica of the shipped-after shape, which the
//! source is then made to match), with a value-equivalence gate and a
//! `GATE FAIL … rollback` `std::process::exit(1)` if the AFTER arm fails to beat
//! BEFORE — the round's roll-back rule encoded into the benchmark.
//!
//! [P1] `drive_pg_repository`'s four policy reads decode `d.policies` into a
//! throwaway `serde_json::Value` DOM and then call
//! `DrivePolicies::from_value(&raw)` (`Self::deserialize(&Value)`) — the
//! exact throwaway-DOM pattern ROUND23 §J1 removed for contacts, but left
//! on the drive-policy path (§J2 removed only the `from_value` clone). The
//! Value tree (a `Map` + boxed String key + `Value` node per policy field)
//! is walked once and dropped. AFTER decodes straight into the struct via
//! `serde_json::from_slice::<DrivePolicies>` (what `sqlx::types::Json<T>`
//! runs on the raw JSONB bytes) — no intermediate DOM. The lenient
//! `unwrap_or_default` fallback is preserved.
//!
//! Run:
//! RUSTFLAGS="-C target-cpu=x86-64-v3" \
//! cargo run --release --features bench --example bench_round26_micro
//! Tunables (env): P1_ITERS (200000)
use std::alloc::{GlobalAlloc, Layout, System};
use std::env;
use std::hint::black_box;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Instant;
use oxicloud::domain::entities::drive::DrivePolicies;
use serde::Deserialize as _;
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
static ALLOC_BYTES: AtomicU64 = AtomicU64::new(0);
struct CountingAlloc;
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
ALLOC_BYTES.fetch_add(layout.size() as u64, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
ALLOC_BYTES.fetch_add(new_size as u64, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
ALLOC_BYTES.fetch_add(layout.size() as u64, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
#[derive(Clone, Copy)]
struct Measure {
ns: f64,
allocs: f64,
bytes: f64,
}
fn measure<T>(iters: u64, mut f: impl FnMut() -> T) -> Measure {
black_box(f());
ALLOC_CALLS.store(0, Ordering::Relaxed);
ALLOC_BYTES.store(0, Ordering::Relaxed);
let start = Instant::now();
for _ in 0..iters {
black_box(f());
}
let ns = start.elapsed().as_nanos() as f64 / iters as f64;
Measure {
ns,
allocs: ALLOC_CALLS.load(Ordering::Relaxed) as f64 / iters as f64,
bytes: ALLOC_BYTES.load(Ordering::Relaxed) as f64 / iters as f64,
}
}
fn report(tag: &str, before: Measure, after: Measure) {
println!("## {tag}");
println!("| arm | ns/op | allocs/op | bytes/op |");
println!(
"| BEFORE | {:>12.1} | {:>11.2} | {:>11.0} |",
before.ns, before.allocs, before.bytes
);
println!(
"| AFTER | {:>12.1} | {:>11.2} | {:>11.0} |",
after.ns, after.allocs, after.bytes
);
println!(
"# {:.2}x wall · {:.2} fewer allocs/op · {:.0} fewer bytes/op\n",
before.ns / after.ns.max(0.0001),
before.allocs - after.allocs,
before.bytes - after.bytes
);
}
fn gate(tag: &str, metric: &str, before: f64, after: f64) {
if !(after < before) {
eprintln!("GATE FAIL [{tag}] {metric}: AFTER {after} !< BEFORE {before} — rollback");
std::process::exit(1);
}
}
// ── [P1] drive-policy JSONB decode: Value DOM + from_value vs from_slice<T> ───
fn section_p1() {
let iters: u64 = env_or("P1_ITERS", 200_000);
// A realistically-populated policies bag (several fields set); the column is
// `jsonb NOT NULL DEFAULT '{}'`, and `#[serde(default)]` fills the rest.
let json: &[u8] = br#"{"forbid_sharing":true,"forbid_public_links":true,"include_in_photo_index":true,"read_only":false,"forbid_cross_drive_move":true}"#;
// Equivalence: both arms yield the identical DrivePolicies.
let before_val: serde_json::Value = serde_json::from_slice(json).unwrap();
let before = DrivePolicies::deserialize(&before_val).unwrap_or_default();
let after = serde_json::from_slice::<DrivePolicies>(json).unwrap_or_default();
assert_eq!(before, after, "P1 decoded policies differ");
let b = measure(iters, || {
// BEFORE: raw JSONB → full serde_json::Value DOM → deserialize(&Value).
let v: serde_json::Value = serde_json::from_slice(black_box(json)).unwrap();
DrivePolicies::deserialize(&v).unwrap_or_default()
});
let a = measure(iters, || {
// AFTER: raw JSONB → from_slice::<DrivePolicies> (what sqlx Json<T> does).
serde_json::from_slice::<DrivePolicies>(black_box(json)).unwrap_or_default()
});
report(
"[P1] drive-policy JSONB decode (Value DOM vs from_slice)",
b,
a,
);
gate("P1", "allocs/op", b.allocs, a.allocs);
}
fn main() {
println!("# Round-26 micro alloc pack\n");
section_p1();
println!("All Round-26 micro sections passed their gate.");
}