perf: round 17 — dedup ingest/verify hash-clone purge, CardDAV vCard TYPE tokens

Targets the content-addressable dedup write path (the ROUND15-deferred
"dedup_service hash-String re-allocations") from both ends — the streaming
ingest loop and the delta-commit verification read — plus a CardDAV vCard
micro-cut. Every change is benchmark-gated with a hard rollback rule; no
PostgreSQL needed for any arm (benches/ROUND17.md).

Backend (counting-allocator, examples/bench_round17_micro.rs):
- D2 chunk-ingest (store_from_stream, the hottest write path — every chunk of
  every upload): the 64-char hex hash String was allocated 3x per chunk
  (to_hex + chunk_hashes clone + session_seen insert-clone, the last dropped on
  a duplicate). The intra-upload dedup set now keys on the raw 32-byte BLAKE3
  digest ([u8;32], Copy, no heap) and the manifest push is branch-split so a
  duplicate moves the hex in: 3 -> 2 allocs/new chunk, 3 -> 1/duplicate.
  Measured 214 -> 149 allocs/op (1.14x wall) on a 64-chunk 1-in-2-dup batch;
  smaller/faster set too (32B inline keys vs 64B heap Strings).
- D1 hash_chunk_sequence (delta-commit verification): took chunks by
  &[(String,u64)] and fed the backend stream with iter().cloned(), re-cloning
  every chunk hash a second time on top of the owned Vec the caller already
  built. Take the Vec by value + into_iter(): 65 -> 0 internal allocs/op, ~2.5us
  of clone work removed per verify.
- V1 vCard TYPE tokens (contact_to_vcard + generate_vcard, 5 sites): each
  EMAIL/TEL/ADR TYPE= param used ty.to_uppercase() — a throw-away String per
  token per contact. New shared fmt::push_upper writes the upper-cased chars
  straight into the buffer (byte-identical to str::to_uppercase, unit-tested):
  13 -> 5 allocs/op, 1.19x wall.

Gates: each section asserts byte/-value equivalence (D2 the ordered manifest +
sizes + write-set; D1 the removed clone is a pure copy; V1 the full vCard) and
exits non-zero if an AFTER arm fails to reduce allocations. push_upper is
unit-tested byte-equal to str::to_uppercase (fmt::tests). Verified end-to-end:
cargo fmt clean, clippy --release --all-targets --features bench -D warnings
clean, and the harness prints GATE PASS against the built release lib.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMmt7vNETYUbEG3Hc17LDx
This commit is contained in:
Claude
2026-07-19 19:33:41 +00:00
parent a09569b5c2
commit c207e66f4b
8 changed files with 608 additions and 32 deletions
+23 -8
View File
@@ -912,7 +912,7 @@ impl DedupService {
/// the declared one — the manifest's Range arithmetic depends on it.
pub async fn hash_chunk_sequence(
&self,
chunks: &[(String, u64)],
chunks: Vec<(String, u64)>,
sniff_len: usize,
) -> Result<(String, Vec<u8>), DomainError> {
let mut hasher = blake3::Hasher::new();
@@ -925,7 +925,7 @@ impl DedupService {
// strictly in manifest order: `buffered` yields in input order.
let prefetch = self.backend.read_prefetch().max(1);
let backend = self.backend.clone();
let mut opened = futures::stream::iter(chunks.iter().cloned())
let mut opened = futures::stream::iter(chunks)
.map(move |(hash, declared_size)| {
let backend = backend.clone();
async move {
@@ -1032,7 +1032,11 @@ impl DedupService {
let mut total_size: u64 = 0;
let mut chunk_hashes: Vec<String> = Vec::new();
let mut chunk_sizes: Vec<u64> = Vec::new();
let mut session_seen: HashSet<String> = HashSet::new();
// Keyed on the raw 32-byte BLAKE3 digest (`Copy`, no heap) rather than
// the 64-char hex String: the intra-upload dedup set no longer clones a
// String per chunk, holds 32-byte inline keys, and hashes 32 bytes not
// 64 on every membership test (benches/ROUND17.md §D2).
let mut session_seen: HashSet<[u8; 32]> = HashSet::new();
let mut pending: Vec<(String, Bytes)> = Vec::new();
let mut pending_bytes: usize = 0;
// Depth-1 settle pipeline: batch N settles on a spawned task while
@@ -1077,12 +1081,19 @@ impl DedupService {
// Per-chunk hashing is ≤ 1 MiB of BLAKE3 (< 1 ms) — cheaper than
// a spawn_blocking round-trip per chunk.
file_hasher.update(&data);
let hash = blake3::hash(&data).to_hex().to_string();
let digest = blake3::hash(&data);
let hash = digest.to_hex().to_string();
chunk_sizes.push(data.len() as u64);
chunk_hashes.push(hash.clone());
if session_seen.insert(hash.clone()) {
// The hex `hash` is materialised once. A genuinely new chunk needs
// it in three places — the ordered manifest, the dedup set key and
// the backend write — but the set keys on the raw digest (no clone),
// so only `chunk_hashes` is cloned before `pending` takes the
// original. A duplicate within this upload needs it only for the
// manifest: the `else` moves it in, no clone (benches/ROUND17.md §D2).
if session_seen.insert(*digest.as_bytes()) {
pending_bytes += data.len();
chunk_hashes.push(hash.clone());
pending.push((hash, Bytes::from(data)));
if pending.len() >= Self::FLUSH_MAX_CHUNKS || pending_bytes >= Self::FLUSH_MAX_BYTES
{
@@ -1111,6 +1122,10 @@ impl DedupService {
}
pending_bytes = 0;
}
} else {
// Duplicate within this upload: only the ordered manifest needs
// the hash. Move it in — no set/pending copy, zero extra allocs.
chunk_hashes.push(hash);
}
}
@@ -4000,7 +4015,7 @@ mod delta_upload_integration_tests {
.collect();
let (computed, head) = svc
.hash_chunk_sequence(&sequence, 16)
.hash_chunk_sequence(sequence.clone(), 16)
.await
.expect("verification read");
assert_eq!(computed, file_hash, "recomputed hash must match");
@@ -4015,7 +4030,7 @@ mod delta_upload_integration_tests {
let mut lying = sequence.clone();
lying[0].1 += 1;
assert!(
svc.hash_chunk_sequence(&lying, 0).await.is_err(),
svc.hash_chunk_sequence(lying, 0).await.is_err(),
"size lie must fail verification"
);