perf: round 17 — dedup ingest/verify hash-clone purge, CardDAV vCard TYPE tokens

Targets the content-addressable dedup write path (the ROUND15-deferred
"dedup_service hash-String re-allocations") from both ends — the streaming
ingest loop and the delta-commit verification read — plus a CardDAV vCard
micro-cut. Every change is benchmark-gated with a hard rollback rule; no
PostgreSQL needed for any arm (benches/ROUND17.md).

Backend (counting-allocator, examples/bench_round17_micro.rs):
- D2 chunk-ingest (store_from_stream, the hottest write path — every chunk of
  every upload): the 64-char hex hash String was allocated 3x per chunk
  (to_hex + chunk_hashes clone + session_seen insert-clone, the last dropped on
  a duplicate). The intra-upload dedup set now keys on the raw 32-byte BLAKE3
  digest ([u8;32], Copy, no heap) and the manifest push is branch-split so a
  duplicate moves the hex in: 3 -> 2 allocs/new chunk, 3 -> 1/duplicate.
  Measured 214 -> 149 allocs/op (1.14x wall) on a 64-chunk 1-in-2-dup batch;
  smaller/faster set too (32B inline keys vs 64B heap Strings).
- D1 hash_chunk_sequence (delta-commit verification): took chunks by
  &[(String,u64)] and fed the backend stream with iter().cloned(), re-cloning
  every chunk hash a second time on top of the owned Vec the caller already
  built. Take the Vec by value + into_iter(): 65 -> 0 internal allocs/op, ~2.5us
  of clone work removed per verify.
- V1 vCard TYPE tokens (contact_to_vcard + generate_vcard, 5 sites): each
  EMAIL/TEL/ADR TYPE= param used ty.to_uppercase() — a throw-away String per
  token per contact. New shared fmt::push_upper writes the upper-cased chars
  straight into the buffer (byte-identical to str::to_uppercase, unit-tested):
  13 -> 5 allocs/op, 1.19x wall.

Gates: each section asserts byte/-value equivalence (D2 the ordered manifest +
sizes + write-set; D1 the removed clone is a pure copy; V1 the full vCard) and
exits non-zero if an AFTER arm fails to reduce allocations. push_upper is
unit-tested byte-equal to str::to_uppercase (fmt::tests). Verified end-to-end:
cargo fmt clean, clippy --release --all-targets --features bench -D warnings
clean, and the harness prints GATE PASS against the built release lib.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMmt7vNETYUbEG3Hc17LDx
This commit is contained in:
Claude
2026-07-19 19:33:41 +00:00
parent a09569b5c2
commit c207e66f4b
8 changed files with 608 additions and 32 deletions
+13
View File
@@ -354,6 +354,19 @@ name = "bench_micro_allocs"
path = "examples/bench_micro_allocs.rs"
required-features = ["bench"]
# Round-17 battery ────────────────────────────────────────────────────────────
# Round-17 dedup + CardDAV CPU/alloc micro-pack — `hash_chunk_sequence` by-value
# (drop the delta-commit verification's 2nd per-chunk hash clone), chunk-ingest
# `session_seen` keyed on the raw 32-byte BLAKE3 digest + branch-split manifest
# push (3 → 2/1 hash-String allocs per ingested chunk), `contact_to_vcard` TYPE
# tokens pushed upper-cased into the buffer (drop the per-token `to_uppercase`
# String). No Postgres.
[[example]]
name = "bench_round17_micro"
path = "examples/bench_round17_micro.rs"
required-features = ["bench"]
# Round-16 battery ────────────────────────────────────────────────────────────
# Round-16 CPU/alloc micro-pack — folder display constants `Arc::from` → interned
+162
View File
@@ -0,0 +1,162 @@
# Round 17 — dedup ingest/verify hash-clone purge, CardDAV vCard TYPE tokens
Benchmark-gated, same rule as ROUND2–16: every change ships with a
BEFORE/AFTER benchmark and an equivalence/safety gate; an AFTER that doesn't
beat its BEFORE is rolled back (never applied). The roll-back rule is encoded
directly into the harness as a `GATE FAIL … rollback` non-zero exit, so a
regression fails CI rather than shipping.
This round targets the **content-addressable dedup write path** — the item
carried on the ROUND15 deferred list as "`dedup_service` hash-`String`
re-allocations" — from both ends: the streaming **ingest** loop that hashes and
stores every uploaded chunk, and the delta-commit **verification** read that
re-hashes a proposed chunk sequence. Plus a CardDAV micro-cut: the vCard
emitters allocated a throw-away upper-cased `String` per `TYPE=` token.
Measured on 4 cores / 15 GiB, **no PostgreSQL needed for any Round-17 arm**
(release-profile counting-allocator example). Reproduce any row with:
```
cargo run --release --features bench --example bench_round17_micro
```
## Summary
| # | change | key metric | before → after |
|--:|---|---|---|
| **D2** | Chunk-ingest (`store_from_stream`) allocated the 64-char hex hash `String` **3× per chunk** (`to_hex().to_string()` + `chunk_hashes.push(clone)` + `session_seen.insert(clone)` — the last dropped on the spot for a duplicate). The intra-upload dedup set now keys on the raw 32-byte BLAKE3 digest (`[u8; 32]`, `Copy`, no heap), and the manifest push is split so a duplicate **moves** the hex in. | 64-chunk batch, 1-in-2 dup | **214 → 149 allocs/op (65 fewer)** · **1.14× wall** · set clone gone + dup manifest clone gone |
| **D1** | `hash_chunk_sequence` (delta-commit verification) took `chunks: &[(String,u64)]` and fed the backend stream with `chunks.iter().cloned()` — re-cloning every chunk hash a **second** time on top of the owned `Vec` the caller already built. Take the `Vec` by value and `into_iter()` it. | 64-chunk verify | **65 → 0 internal allocs/op** · **~2.5 µs of clone work removed per verify** |
| **V1** | `contact_to_vcard` / `generate_vcard` emitted every EMAIL/TEL/ADR `TYPE=` token via `ty.to_uppercase()` — one throw-away `String` per token per contact. New shared `fmt::push_upper` writes the upper-cased chars straight into the vCard buffer. | 8 tokens/op | **13 → 5 allocs/op (8 fewer)** · **1.19× wall** |
> Allocs/op is the deterministic primary gate (identical run to run); the wall
> figures are single-shot and noise-bounded (D1's AFTER arm is a near-zero-cost
> read, so its ratio swings 60–120× between runs — the stable fact is the
> ~2.5 µs / 65-alloc clone removed).
## [D2] Chunk-ingest — dedup set keyed on the raw digest
The streaming ingest loop (`DedupService::store_from_stream`) is the hottest
write path in the system: it runs for **every chunk of every upload**. Per
chunk it produced the 64-char hex hash and then allocated it three times:
```rust
let hash = blake3::hash(&data).to_hex().to_string(); // A: the hex String
chunk_hashes.push(hash.clone()); // B: manifest copy (always)
if session_seen.insert(hash.clone()) { // C: dedup-set copy (always)
pending.push((hash, Bytes::from(data))); // A moved into the write batch
}
```
`session_seen` is the **intra-upload** dedup set (has this exact chunk already
appeared in *this* stream? — repeated blocks, zero-padded regions, re-chunked
near-duplicates). It was a `HashSet<String>`, so clone **C** heap-allocated a
64-byte key for every chunk — and on a duplicate, `insert` allocated the clone
only to drop it when the key already existed. Pure waste on the case a dedup
store exists to make cheap.
A BLAKE3 digest is `[u8; 32]` — `Copy`, no heap, and the hex is a lossless
rendering of it, so keying the set on the raw digest is behaviour-identical:
```rust
let digest = blake3::hash(&data);
let hash = digest.to_hex().to_string(); // A (once)
if session_seen.insert(*digest.as_bytes()) { // Copy key — zero heap
chunk_hashes.push(hash.clone()); // B (new chunk only)
pending.push((hash, Bytes::from(data))); // A moved
} else {
chunk_hashes.push(hash); // dup: move, no clone
}
```
Clone **C** is gone for every chunk; clone **B** is gone for every *duplicate*
(it moves the hex into the manifest instead). The set also holds 32-byte inline
keys instead of 64-byte heap Strings and hashes 32 bytes per membership test.
Net per chunk: **3 → 2 allocs (new) / 3 → 1 (duplicate)** — strictly fewer on
every input, unique or duplicate.
64-chunk, 1-in-2-duplicate batch: **214 → 149 allocs/op (65 fewer), 1.14×
wall**. Gate (in-harness, replica of the exact before/after loop bodies): the
observable output is **byte-for-byte equal** — the ordered `chunk_hashes`
manifest, the `chunk_sizes`, and the distinct write-set `pending` all match;
only the private set's key representation differs — plus the `gate_allocs`
rollback exit on any AFTER that fails to reduce allocations.
## [D1] `hash_chunk_sequence` — take the chunk Vec by value
The delta-sync commit (`delta_upload_service::commit`) verifies a client's
proposed manifest by streaming the pinned chunks back out of the backend and
recomputing the whole-file BLAKE3. The caller already builds a fresh owned
`Vec<(String,u64)>` (`request.chunks.iter().map(|c| (c.h.clone(), c.s)).collect()`),
but `hash_chunk_sequence` took it by `&[(String,u64)]` and then did
`futures::stream::iter(chunks.iter().cloned())` — **re-cloning every chunk hash
a second time** to feed the stream.
Taking `chunks: Vec<(String,u64)>` by value and `into_iter()`-ing it (the caller
drops one `&`) moves those Strings straight into the stream: zero internal
clones. The streamed `(hash, size)` pairs are byte-identical, so the recomputed
hash and every per-chunk size check are unchanged.
The section isolates exactly the clone the old signature forced (the caller's
`.collect()` is identical on both shapes and excluded): **65 → 0 allocs/op** for
a 64-chunk manifest — ~2.5 µs of clone work removed per verify (the AFTER arm is
a near-zero-cost read, so the wall ratio is large but noisy: 60–120×). Gate: the
old internal clone is asserted to be a pure copy (moving changes nothing
observable), plus `gate_allocs`.
## [V1] CardDAV vCard `TYPE=` tokens — `push_upper`
Both vCard emitters — `carddav_adapter::contact_to_vcard` (the CardDAV
REPORT/GET path) and `ContactService::generate_vcard` — wrote each address /
phone / email `TYPE=` parameter with `write!(…, "{}", ty.to_uppercase())`, and
`str::to_uppercase()` heap-allocates a fresh `String` for every token. A contact
with several emails/phones/addresses pays one alloc per token, per emit, on
every address-book sync.
New shared helper `common::fmt::push_upper(buf, s)` writes the upper-cased chars
(`char::to_uppercase`, so byte-identical to `str::to_uppercase` — including
ß → SS and dotless-i) straight into the vCard buffer; the five call sites push
the fixed prefix, the upper-cased token, and the value directly. Zero
temporaries.
8-token contact: **13 → 5 allocs/op (8 fewer), 1.19× wall**. Gates:
`push_upper` is unit-tested byte-equal to `str::to_uppercase` across ASCII /
mixed / multi-char-uppercase / dotless-i inputs (`fmt::tests`), the section
asserts the full emitted vCard is byte-identical before/after, and the existing
`carddav_adapter_test::test_contact_to_vcard_full` pins the whole document.
## Not shipped — deferred to a later round
Surfaced during the Round-17 audit but not landed (each wants its own decision,
a Postgres fixture, or a streaming-I/O benchmark):
- **Storage I/O — `encrypted_blob_backend` frame size (evaluated, kept):** the
ROUND15 note floated 64 KiB → 256 KiB plaintext emit frames. `PLAINTEXT_EMIT_SIZE`
is a *deliberate* match to the 64 KiB the unencrypted backends stream, so
downstream consumers see the same backpressure shape; changing it is a
behaviour change that needs a streaming-throughput A/B (TTFB + syscalls +
peak RSS), not an alloc micro-bench. Left as-is pending that harness.
- **Storage I/O — `CachedBlobBackend` write path (needs an fs harness):**
per-write `create_dir_all` even when the shard dir exists, and inline
eviction `remove_file` on the reactor thread (carried from ROUND15).
- **Backend query-shape (needs Postgres):** `music_storage_adapter::list_public_playlists`
1 + N `COUNT(*)` fold; contact REST listings over-fetch the multi-KB `vcard`
TEXT though the `ContactDto` mappers never read it (wants a *lite* row mapper).
- **Backend CPU/alloc (no Postgres, next micro-pack):** the two WebDAV PROPFIND
surfaces still quote `d:getetag` into a fresh `String` per row and `format!`
the per-row href per child (the CalDAV reused-buffer treatment never reached
them); REST calendar-event edit re-`format!`s the whole `ical_data` body once
per changed property.
- **Frontend (vitest-benchmarkable):** `VirtualRows.offsets` prefix-sum rebuilt
in full on every photos-timeline page; the dotfile filter and
`ResourceList.itemIndexById` re-scan the whole accumulated list per page.
## Environment / methodology
- `cargo run --release --features bench --example bench_round17_micro`
— counting global allocator, no Postgres. Tunables: `BENCH_ITERS` (100000),
`BENCH_CHUNKS` (64), `BENCH_DUP_RATIO` (2).
- Each section is BEFORE (verbatim replica of the shipped-before shape) vs AFTER
(verbatim replica of the shipped-after shape) with a byte/-value equivalence
gate; the shipped source now matches each AFTER arm.
- Roll-back rule encoded per section: `std::process::exit(1)` with
`GATE FAIL … rollback` if an AFTER arm fails to reduce allocations.
+357
View File
@@ -0,0 +1,357 @@
//! Round-17 dedup + CardDAV CPU/alloc micro-pack (no Postgres).
//!
//! Same rule as ROUND2–16: each section is BEFORE (verbatim replica of the
//! shipped-before shape) vs AFTER (verbatim replica of the shipped-after shape,
//! or the shipped helper where reachable), with a byte/-value equivalence gate
//! and a `GATE FAIL … rollback` check that exits non-zero if the AFTER arm
//! fails to reduce allocations — the round's roll-back rule encoded into the
//! benchmark.
//!
//! [D1] `DedupService::hash_chunk_sequence` (delta-commit verification) took
//! `chunks: &[(String, u64)]` and fed the backend stream via
//! `chunks.iter().cloned()` — re-allocating every chunk-hash String a
//! second time, on top of the owned `Vec` the caller already built with
//! `c.h.clone()`. Taking the `Vec` by value and `into_iter()`-ing it
//! moves those Strings in: zero internal clones.
//! [D2] The chunk-ingest loop (`store_from_stream`) allocated the 64-char hex
//! hash String THREE times per chunk: `to_hex().to_string()`, then
//! `chunk_hashes.push(hash.clone())`, then `session_seen.insert(hash
//! .clone())` — the last dropped immediately on a duplicate. Keying the
//! intra-upload dedup set on the raw 32-byte BLAKE3 digest (`[u8; 32]`,
//! `Copy`, no heap) drops the set clone entirely, and moving the hex into
//! `chunk_hashes` on the duplicate branch drops the manifest clone there:
//! 3 → 2 allocs (new chunk) / 3 → 1 (duplicate), on the hottest write
//! path in the dedup system.
//! [V1] `contact_to_vcard` emitted every EMAIL/TEL/ADR `TYPE=` token via
//! `ty.to_uppercase()` — one throw-away String per token per vCard. The
//! `push_upper` helper writes the upper-cased chars straight into the
//! vCard buffer: zero temporaries.
//!
//! Run:
//! cargo run --release --features bench --example bench_round17_micro
//! Tunables (env): BENCH_ITERS (100000), BENCH_CHUNKS (64), BENCH_DUP_RATIO (2)
use std::alloc::{GlobalAlloc, Layout, System};
use std::collections::HashSet;
use std::env;
use std::fmt::Write as _;
use std::hint::black_box;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Instant;
use bytes::Bytes;
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
struct CountingAlloc;
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
struct Measured {
wall_ns_per_op: f64,
allocs_per_op: f64,
}
fn measure<F: FnMut()>(iters: usize, mut f: F) -> Measured {
let a0 = ALLOC_CALLS.load(Ordering::Relaxed);
let t = Instant::now();
for _ in 0..iters {
f();
}
let wall = t.elapsed().as_nanos() as f64 / iters as f64;
let allocs = (ALLOC_CALLS.load(Ordering::Relaxed) - a0) as f64 / iters as f64;
Measured {
wall_ns_per_op: wall,
allocs_per_op: allocs,
}
}
fn print_row(label: &str, m: &Measured) {
println!(
"| {:<46} | {:>12.1} | {:>10.2} |",
label, m.wall_ns_per_op, m.allocs_per_op
);
}
fn header_footer(name: &str, before: &Measured, after: &Measured) {
println!("| arm | ns/op | allocs/op |");
print_row(&format!("BEFORE {name}"), before);
print_row(&format!("AFTER {name}"), after);
println!(
"# {:.2}x wall, {:.2} fewer allocs/op",
before.wall_ns_per_op / after.wall_ns_per_op,
before.allocs_per_op - after.allocs_per_op
);
}
fn gate_allocs(tag: &str, before: &Measured, after: &Measured) {
if after.allocs_per_op >= before.allocs_per_op {
eprintln!("GATE FAIL [{tag}]: AFTER did not reduce allocations — rollback");
std::process::exit(1);
}
}
// ────────────────────────────────────────────────────────────────────────────
// [D1] hash_chunk_sequence — `&[..]` + iter().cloned() vs `Vec` by value
// ────────────────────────────────────────────────────────────────────────────
//
// The caller (`delta_upload_service::commit`) already owns a fresh
// `Vec<(String, u64)>` built with `c.h.clone()`; that `.collect()` is identical
// on both call shapes, so it is EXCLUDED from the comparison. The delta is what
// `hash_chunk_sequence` does INTERNALLY to feed `futures::stream::iter(..)`:
// BEFORE — `chunks.iter().cloned()` re-clones every (String, u64) → N String
// allocations (+ the collected Vec) inside the function.
// AFTER — the `Vec` is moved in and `into_iter()`- d → the Strings relocate
// with zero heap traffic; the function iterates the owned pairs.
// The streamed (hash, size) pairs are byte-identical, so the recomputed BLAKE3
// and every size check are unchanged — only the ownership differs.
fn d1_before_internal(chunks: &[(String, u64)]) -> Vec<(String, u64)> {
// Materialises `stream::iter(chunks.iter().cloned())`'s input — the same N
// element clones + one Vec the old `&[..]` signature forced. `to_vec()` is
// `iter().cloned().collect()` (identical allocations), spelled the way
// clippy prefers.
chunks.to_vec()
}
fn section_hash_chunk_sequence() {
let iters: usize = env_or("BENCH_ITERS", 100_000);
let n: usize = env_or("BENCH_CHUNKS", 64);
// A realistic manifest: N distinct 64-hex chunk hashes + declared sizes.
let base: Vec<(String, u64)> = (0..n)
.map(|i| {
let h = blake3::hash(format!("d1-chunk-{i}").as_bytes())
.to_hex()
.to_string();
(h, 1024 + i as u64)
})
.collect();
// Gate: the old internal clone is a pure copy — moving instead changes
// nothing the function observes (same pairs, same order).
assert_eq!(
d1_before_internal(&base),
base,
"d1 clone is not a pure copy"
);
let before = measure(iters, || {
// The internal re-clone the `&[..]` signature forced.
black_box(d1_before_internal(black_box(&base)));
});
let after = measure(iters, || {
// The by-value signature adds no internal copy — it consumes the moved
// pairs (modelled here as an in-order read of the same owned pairs).
for c in black_box(&base).iter() {
black_box(c);
}
});
println!("\n## [D1] hash_chunk_sequence internal clone ({n} chunks/op)");
header_footer("hash_chunk_sequence by-value", &before, &after);
gate_allocs("D1", &before, &after);
}
// ────────────────────────────────────────────────────────────────────────────
// [D2] chunk-ingest loop — 3 hash-String allocs/chunk vs 2 (new) / 1 (dup)
// ────────────────────────────────────────────────────────────────────────────
struct IngestOut {
chunk_hashes: Vec<String>,
chunk_sizes: Vec<u64>,
/// The distinct hashes that would be written to the backend, in order.
pending: Vec<String>,
}
/// BEFORE: verbatim replica of the shipped-before loop body.
fn d2_before(payloads: &[Bytes]) -> IngestOut {
let mut chunk_hashes: Vec<String> = Vec::new();
let mut chunk_sizes: Vec<u64> = Vec::new();
let mut session_seen: HashSet<String> = HashSet::new();
let mut pending: Vec<(String, Bytes)> = Vec::new();
for data in payloads {
let hash = blake3::hash(data).to_hex().to_string();
chunk_sizes.push(data.len() as u64);
chunk_hashes.push(hash.clone());
if session_seen.insert(hash.clone()) {
pending.push((hash, data.clone()));
}
}
IngestOut {
chunk_hashes,
chunk_sizes,
pending: pending.into_iter().map(|(h, _)| h).collect(),
}
}
/// AFTER: verbatim replica of the shipped-after loop body — the dedup set keys
/// on the raw 32-byte digest, and the manifest push is split across the
/// new/duplicate branches so a duplicate moves (not clones) the hex in.
fn d2_after(payloads: &[Bytes]) -> IngestOut {
let mut chunk_hashes: Vec<String> = Vec::new();
let mut chunk_sizes: Vec<u64> = Vec::new();
let mut session_seen: HashSet<[u8; 32]> = HashSet::new();
let mut pending: Vec<(String, Bytes)> = Vec::new();
for data in payloads {
let digest = blake3::hash(data);
let hash = digest.to_hex().to_string();
chunk_sizes.push(data.len() as u64);
if session_seen.insert(*digest.as_bytes()) {
chunk_hashes.push(hash.clone());
pending.push((hash, data.clone()));
} else {
chunk_hashes.push(hash);
}
}
IngestOut {
chunk_hashes,
chunk_sizes,
pending: pending.into_iter().map(|(h, _)| h).collect(),
}
}
fn section_chunk_ingest() {
let iters: usize = env_or("BENCH_ITERS", 100_000);
let n: usize = env_or("BENCH_CHUNKS", 64);
// 1-in-K chunks repeats an earlier one (models intra-file dedup: repeated
// blocks, zero-padded regions, re-chunked near-duplicates). K=2 ⇒ ~half the
// stream is duplicate, the case a dedup store exists to make cheap.
let dup_ratio: usize = env_or("BENCH_DUP_RATIO", 2).max(1);
let payloads: Vec<Bytes> = (0..n)
.map(|i| {
let key = if dup_ratio > 0 && i % dup_ratio == 0 && i >= dup_ratio {
i - dup_ratio // repeat an earlier chunk's bytes
} else {
i
};
Bytes::from(format!("d2-chunk-payload-{key}-{}", "x".repeat(256)))
})
.collect();
// Gate: identical observable output — the ordered manifest, the sizes, and
// the distinct write set are byte-for-byte equal (only the private set's key
// representation differs).
let b = d2_before(&payloads);
let a = d2_after(&payloads);
assert_eq!(b.chunk_hashes, a.chunk_hashes, "d2 manifest differs");
assert_eq!(b.chunk_sizes, a.chunk_sizes, "d2 sizes differ");
assert_eq!(b.pending, a.pending, "d2 write-set differs");
let before = measure(iters, || {
black_box(d2_before(black_box(&payloads)));
});
let after = measure(iters, || {
black_box(d2_after(black_box(&payloads)));
});
println!("\n## [D2] chunk-ingest hash allocs ({n} chunks/op, 1-in-{dup_ratio} dup)");
header_footer("chunk-ingest session_seen [u8;32]", &before, &after);
gate_allocs("D2", &before, &after);
}
// ────────────────────────────────────────────────────────────────────────────
// [V1] contact_to_vcard TYPE tokens — per-token to_uppercase() String vs push
// ────────────────────────────────────────────────────────────────────────────
/// AFTER helper: write the upper-cased form of `s` straight into `buf`.
/// Uses `char::to_uppercase`, so the bytes are identical to `s.to_uppercase()`.
fn push_upper(buf: &mut String, s: &str) {
for c in s.chars() {
for u in c.to_uppercase() {
buf.push(u);
}
}
}
/// BEFORE: verbatim replica — `write!` the `to_uppercase()` temporary.
fn v1_before(types: &[&str]) -> String {
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
for ty in types {
let _ = write!(vcard, "EMAIL;TYPE={}:x@e.test\r\n", ty.to_uppercase());
}
vcard
}
/// AFTER: verbatim replica of the shipped-after emit — push pieces + upper.
fn v1_after(types: &[&str]) -> String {
let mut vcard = String::from("BEGIN:VCARD\r\nVERSION:3.0\r\n");
for ty in types {
vcard.push_str("EMAIL;TYPE=");
push_upper(&mut vcard, ty);
vcard.push_str(":x@e.test\r\n");
}
vcard
}
fn section_vcard_types() {
let iters: usize = env_or("BENCH_ITERS", 100_000);
// A contact's worth of EMAIL/TEL/ADR type tokens (already-upper, lower,
// mixed, and an x- extension — the shapes real address books carry).
let types = [
"HOME", "work", "Cell", "voice", "fax", "x-custom", "WORK", "home",
];
// Gate: byte-identical vCard, and push_upper == str::to_uppercase per token.
for ty in types {
let mut got = String::new();
push_upper(&mut got, ty);
assert_eq!(got, ty.to_uppercase(), "push_upper differs for {ty:?}");
}
assert_eq!(v1_before(&types), v1_after(&types), "v1 vcard differs");
let before = measure(iters, || {
black_box(v1_before(black_box(&types)));
});
let after = measure(iters, || {
black_box(v1_after(black_box(&types)));
});
println!(
"\n## [V1] contact_to_vcard TYPE tokens ({} tokens/op)",
types.len()
);
header_footer("vcard TYPE push_upper", &before, &after);
gate_allocs("V1", &before, &after);
}
fn main() {
println!("#################################################################");
println!("# Round-17 dedup + CardDAV CPU/alloc micro-pack");
println!("#################################################################");
section_hash_chunk_sequence();
section_chunk_ingest();
section_vcard_types();
println!("\nGATE PASS (all sections)");
}
+13 -14
View File
@@ -970,28 +970,27 @@ pub fn contact_to_vcard(contact: &ContactDto) -> String {
}
for email in &contact.email {
let _ = write!(
vcard,
"EMAIL;TYPE={}:{}\r\n",
email.r#type.to_uppercase(),
email.email
);
vcard.push_str("EMAIL;TYPE=");
crate::common::fmt::push_upper(&mut vcard, &email.r#type);
vcard.push(':');
vcard.push_str(&email.email);
vcard.push_str("\r\n");
}
for phone in &contact.phone {
let _ = write!(
vcard,
"TEL;TYPE={}:{}\r\n",
phone.r#type.to_uppercase(),
phone.number
);
vcard.push_str("TEL;TYPE=");
crate::common::fmt::push_upper(&mut vcard, &phone.r#type);
vcard.push(':');
vcard.push_str(&phone.number);
vcard.push_str("\r\n");
}
for addr in &contact.address {
vcard.push_str("ADR;TYPE=");
crate::common::fmt::push_upper(&mut vcard, &addr.r#type);
let _ = write!(
vcard,
"ADR;TYPE={}:;;{};{};{};{};{}\r\n",
addr.r#type.to_uppercase(),
":;;{};{};{};{};{}\r\n",
addr.street.as_deref().unwrap_or(""),
addr.city.as_deref().unwrap_or(""),
addr.state.as_deref().unwrap_or(""),
+9 -9
View File
@@ -283,12 +283,11 @@ impl ContactService {
// Email addresses
for email in contact.email() {
let _ = write!(
vcard,
"EMAIL;TYPE={}:{}\r\n",
email.r#type.to_uppercase(),
email.email
);
vcard.push_str("EMAIL;TYPE=");
crate::common::fmt::push_upper(&mut vcard, &email.r#type);
vcard.push(':');
vcard.push_str(&email.email);
vcard.push_str("\r\n");
}
// Phone numbers
@@ -305,17 +304,18 @@ impl ContactService {
// Addresses
for addr in contact.address() {
let addr_type = addr.r#type.to_uppercase();
let street = addr.street.as_deref().unwrap_or_default();
let city = addr.city.as_deref().unwrap_or_default();
let state = addr.state.as_deref().unwrap_or_default();
let postal_code = addr.postal_code.as_deref().unwrap_or_default();
let country = addr.country.as_deref().unwrap_or_default();
vcard.push_str("ADR;TYPE=");
crate::common::fmt::push_upper(&mut vcard, &addr.r#type);
let _ = write!(
vcard,
"ADR;TYPE={}:;;{};{};{};{};{}\r\n",
addr_type, street, city, state, postal_code, country
":;;{};{};{};{};{}\r\n",
street, city, state, postal_code, country
);
}
@@ -398,7 +398,7 @@ impl DeltaUploadService {
let verification = self
.dedup
.hash_chunk_sequence(
&request
request
.chunks
.iter()
.map(|c| (c.h.clone(), c.s))
+30
View File
@@ -210,6 +210,22 @@ pub fn hex_lower(bytes: &[u8]) -> String {
out
}
/// Append the upper-cased form of `s` to `buf` without a temporary `String`.
///
/// Byte-identical to `buf.push_str(&s.to_uppercase())` — same
/// `char::to_uppercase` expansion (incl. ß → SS, ff → FF) — but writes straight
/// into the caller's buffer. The vCard emit path (`contact_to_vcard`,
/// `generate_vcard`) formats an `EMAIL`/`TEL`/`ADR` `TYPE=` token per line, and
/// the old `write!(…, "{}", ty.to_uppercase())` heap-allocated one throw-away
/// `String` per token per contact (benches/ROUND17.md §V1).
pub fn push_upper(buf: &mut String, s: &str) {
for c in s.chars() {
for u in c.to_uppercase() {
buf.push(u);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -232,6 +248,20 @@ mod tests {
}
}
/// `push_upper` must match `push_str(&s.to_uppercase())` byte for byte,
/// including multi-char upper-casings (ß → SS) and dotless-i.
#[test]
fn push_upper_matches_to_uppercase() {
let cases = [
"", "home", "WORK", "Cell", "voice", "x-custom", "café", "straße", "ff", "ı",
];
for s in cases {
let mut got = String::new();
push_upper(&mut got, s);
assert_eq!(got, s.to_uppercase(), "push_upper differs for {s:?}");
}
}
/// Edge-heavy corpus: epoch, single-digit day (padding!), leap day,
/// end-of-year, DST-irrelevant midsummer, far future, max in-range.
const CASES: [i64; 12] = [
+23 -8
View File
@@ -912,7 +912,7 @@ impl DedupService {
/// the declared one — the manifest's Range arithmetic depends on it.
pub async fn hash_chunk_sequence(
&self,
chunks: &[(String, u64)],
chunks: Vec<(String, u64)>,
sniff_len: usize,
) -> Result<(String, Vec<u8>), DomainError> {
let mut hasher = blake3::Hasher::new();
@@ -925,7 +925,7 @@ impl DedupService {
// strictly in manifest order: `buffered` yields in input order.
let prefetch = self.backend.read_prefetch().max(1);
let backend = self.backend.clone();
let mut opened = futures::stream::iter(chunks.iter().cloned())
let mut opened = futures::stream::iter(chunks)
.map(move |(hash, declared_size)| {
let backend = backend.clone();
async move {
@@ -1032,7 +1032,11 @@ impl DedupService {
let mut total_size: u64 = 0;
let mut chunk_hashes: Vec<String> = Vec::new();
let mut chunk_sizes: Vec<u64> = Vec::new();
let mut session_seen: HashSet<String> = HashSet::new();
// Keyed on the raw 32-byte BLAKE3 digest (`Copy`, no heap) rather than
// the 64-char hex String: the intra-upload dedup set no longer clones a
// String per chunk, holds 32-byte inline keys, and hashes 32 bytes not
// 64 on every membership test (benches/ROUND17.md §D2).
let mut session_seen: HashSet<[u8; 32]> = HashSet::new();
let mut pending: Vec<(String, Bytes)> = Vec::new();
let mut pending_bytes: usize = 0;
// Depth-1 settle pipeline: batch N settles on a spawned task while
@@ -1077,12 +1081,19 @@ impl DedupService {
// Per-chunk hashing is ≤ 1 MiB of BLAKE3 (< 1 ms) — cheaper than
// a spawn_blocking round-trip per chunk.
file_hasher.update(&data);
let hash = blake3::hash(&data).to_hex().to_string();
let digest = blake3::hash(&data);
let hash = digest.to_hex().to_string();
chunk_sizes.push(data.len() as u64);
chunk_hashes.push(hash.clone());
if session_seen.insert(hash.clone()) {
// The hex `hash` is materialised once. A genuinely new chunk needs
// it in three places — the ordered manifest, the dedup set key and
// the backend write — but the set keys on the raw digest (no clone),
// so only `chunk_hashes` is cloned before `pending` takes the
// original. A duplicate within this upload needs it only for the
// manifest: the `else` moves it in, no clone (benches/ROUND17.md §D2).
if session_seen.insert(*digest.as_bytes()) {
pending_bytes += data.len();
chunk_hashes.push(hash.clone());
pending.push((hash, Bytes::from(data)));
if pending.len() >= Self::FLUSH_MAX_CHUNKS || pending_bytes >= Self::FLUSH_MAX_BYTES
{
@@ -1111,6 +1122,10 @@ impl DedupService {
}
pending_bytes = 0;
}
} else {
// Duplicate within this upload: only the ordered manifest needs
// the hash. Move it in — no set/pending copy, zero extra allocs.
chunk_hashes.push(hash);
}
}
@@ -4000,7 +4015,7 @@ mod delta_upload_integration_tests {
.collect();
let (computed, head) = svc
.hash_chunk_sequence(&sequence, 16)
.hash_chunk_sequence(sequence.clone(), 16)
.await
.expect("verification read");
assert_eq!(computed, file_hash, "recomputed hash must match");
@@ -4015,7 +4030,7 @@ mod delta_upload_integration_tests {
let mut lying = sequence.clone();
lying[0].1 += 1;
assert!(
svc.hash_chunk_sequence(&lying, 0).await.is_err(),
svc.hash_chunk_sequence(lying, 0).await.is_err(),
"size lie must fail verification"
);