Oxicloud/benches/ROUND17.md

Round 17 — dedup ingest/verify hash-clone purge, CardDAV vCard TYPE tokens

Benchmark-gated, same rule as ROUND2–16: every change ships with a BEFORE/AFTER benchmark and an equivalence/safety gate; an AFTER that doesn't beat its BEFORE is rolled back (never applied). The roll-back rule is encoded directly into the harness as a GATE FAIL … rollback non-zero exit, so a regression fails CI rather than shipping.

This round targets the content-addressable dedup write path — the item carried on the ROUND15 deferred list as "dedup_service hash-String re-allocations" — from both ends: the streaming ingest loop that hashes and stores every uploaded chunk, and the delta-commit verification read that re-hashes a proposed chunk sequence. Plus a CardDAV micro-cut: the vCard emitters allocated a throw-away upper-cased String per TYPE= token.

Measured on 4 cores / 15 GiB, no PostgreSQL needed for any Round-17 arm (release-profile counting-allocator example). Reproduce any row with:

cargo run --release --features bench --example bench_round17_micro

Summary

#	change	key metric	before → after
D2	Chunk-ingest (store_from_stream) allocated the 64-char hex hash String 3× per chunk (to_hex().to_string() + chunk_hashes.push(clone) + session_seen.insert(clone) — the last dropped on the spot for a duplicate). The intra-upload dedup set now keys on the raw 32-byte BLAKE3 digest ([u8; 32], Copy, no heap), and the manifest push is split so a duplicate moves the hex in.	64-chunk batch, 1-in-2 dup	214 → 149 allocs/op (65 fewer) · 1.14× wall · set clone gone + dup manifest clone gone
D1	hash_chunk_sequence (delta-commit verification) took chunks: &[(String,u64)] and fed the backend stream with chunks.iter().cloned() — re-cloning every chunk hash a second time on top of the owned Vec the caller already built. Take the Vec by value and into_iter() it.	64-chunk verify	65 → 0 internal allocs/op · ~2.5 µs of clone work removed per verify
V1	contact_to_vcard / generate_vcard emitted every EMAIL/TEL/ADR TYPE= token via ty.to_uppercase() — one throw-away String per token per contact. New shared fmt::push_upper writes the upper-cased chars straight into the vCard buffer.	8 tokens/op	13 → 5 allocs/op (8 fewer) · 1.19× wall

Allocs/op is the deterministic primary gate (identical run to run); the wall figures are single-shot and noise-bounded (D1's AFTER arm is a near-zero-cost read, so its ratio swings 60–120× between runs — the stable fact is the ~2.5 µs / 65-alloc clone removed).

[D2] Chunk-ingest — dedup set keyed on the raw digest

The streaming ingest loop (DedupService::store_from_stream) is the hottest write path in the system: it runs for every chunk of every upload. Per chunk it produced the 64-char hex hash and then allocated it three times:

let hash = blake3::hash(&data).to_hex().to_string();   // A: the hex String
chunk_hashes.push(hash.clone());                        // B: manifest copy  (always)
if session_seen.insert(hash.clone()) {                  // C: dedup-set copy  (always)
    pending.push((hash, Bytes::from(data)));            // A moved into the write batch
}

session_seen is the intra-upload dedup set (has this exact chunk already appeared in this stream? — repeated blocks, zero-padded regions, re-chunked near-duplicates). It was a HashSet<String>, so clone C heap-allocated a 64-byte key for every chunk — and on a duplicate, insert allocated the clone only to drop it when the key already existed. Pure waste on the case a dedup store exists to make cheap.

A BLAKE3 digest is [u8; 32] — Copy, no heap, and the hex is a lossless rendering of it, so keying the set on the raw digest is behaviour-identical:

let digest = blake3::hash(&data);
let hash = digest.to_hex().to_string();                 // A (once)
if session_seen.insert(*digest.as_bytes()) {            // Copy key — zero heap
    chunk_hashes.push(hash.clone());                     // B (new chunk only)
    pending.push((hash, Bytes::from(data)));             // A moved
} else {
    chunk_hashes.push(hash);                             // dup: move, no clone
}

Clone C is gone for every chunk; clone B is gone for every duplicate (it moves the hex into the manifest instead). The set also holds 32-byte inline keys instead of 64-byte heap Strings and hashes 32 bytes per membership test. Net per chunk: 3 → 2 allocs (new) / 3 → 1 (duplicate) — strictly fewer on every input, unique or duplicate.

64-chunk, 1-in-2-duplicate batch: 214 → 149 allocs/op (65 fewer), 1.14× wall. Gate (in-harness, replica of the exact before/after loop bodies): the observable output is byte-for-byte equal — the ordered chunk_hashes manifest, the chunk_sizes, and the distinct write-set pending all match; only the private set's key representation differs — plus the gate_allocs rollback exit on any AFTER that fails to reduce allocations.

[D1] hash_chunk_sequence — take the chunk Vec by value

The delta-sync commit (delta_upload_service::commit) verifies a client's proposed manifest by streaming the pinned chunks back out of the backend and recomputing the whole-file BLAKE3. The caller already builds a fresh owned Vec<(String,u64)> (request.chunks.iter().map(|c| (c.h.clone(), c.s)).collect()), but hash_chunk_sequence took it by &[(String,u64)] and then did futures::stream::iter(chunks.iter().cloned()) — re-cloning every chunk hash a second time to feed the stream.

Taking chunks: Vec<(String,u64)> by value and into_iter()-ing it (the caller drops one &) moves those Strings straight into the stream: zero internal clones. The streamed (hash, size) pairs are byte-identical, so the recomputed hash and every per-chunk size check are unchanged.

The section isolates exactly the clone the old signature forced (the caller's .collect() is identical on both shapes and excluded): 65 → 0 allocs/op for a 64-chunk manifest — ~2.5 µs of clone work removed per verify (the AFTER arm is a near-zero-cost read, so the wall ratio is large but noisy: 60–120×). Gate: the old internal clone is asserted to be a pure copy (moving changes nothing observable), plus gate_allocs.

[V1] CardDAV vCard TYPE= tokens — push_upper

Both vCard emitters — carddav_adapter::contact_to_vcard (the CardDAV REPORT/GET path) and ContactService::generate_vcard — wrote each address / phone / email TYPE= parameter with write!(…, "{}", ty.to_uppercase()), and str::to_uppercase() heap-allocates a fresh String for every token. A contact with several emails/phones/addresses pays one alloc per token, per emit, on every address-book sync.

New shared helper common::fmt::push_upper(buf, s) writes the upper-cased chars (char::to_uppercase, so byte-identical to str::to_uppercase — including ß → SS and dotless-i) straight into the vCard buffer; the five call sites push the fixed prefix, the upper-cased token, and the value directly. Zero temporaries.

8-token contact: 13 → 5 allocs/op (8 fewer), 1.19× wall. Gates: push_upper is unit-tested byte-equal to str::to_uppercase across ASCII / mixed / multi-char-uppercase / dotless-i inputs (fmt::tests), the section asserts the full emitted vCard is byte-identical before/after, and the existing carddav_adapter_test::test_contact_to_vcard_full pins the whole document.

Not shipped — deferred to a later round

Surfaced during the Round-17 audit but not landed (each wants its own decision, a Postgres fixture, or a streaming-I/O benchmark):

• Storage I/O — encrypted_blob_backend frame size (evaluated, kept): the ROUND15 note floated 64 KiB → 256 KiB plaintext emit frames. PLAINTEXT_EMIT_SIZE is a deliberate match to the 64 KiB the unencrypted backends stream, so downstream consumers see the same backpressure shape; changing it is a behaviour change that needs a streaming-throughput A/B (TTFB + syscalls + peak RSS), not an alloc micro-bench. Left as-is pending that harness.
• Storage I/O — CachedBlobBackend write path (needs an fs harness): per-write create_dir_all even when the shard dir exists, and inline eviction remove_file on the reactor thread (carried from ROUND15).
• Backend query-shape (needs Postgres): music_storage_adapter::list_public_playlists 1 + N COUNT(*) fold; contact REST listings over-fetch the multi-KB vcard TEXT though the ContactDto mappers never read it (wants a lite row mapper).
• Backend CPU/alloc (no Postgres, next micro-pack): the two WebDAV PROPFIND surfaces still quote d:getetag into a fresh String per row and format! the per-row href per child (the CalDAV reused-buffer treatment never reached them); REST calendar-event edit re-format!s the whole ical_data body once per changed property.
• Frontend (vitest-benchmarkable): VirtualRows.offsets prefix-sum rebuilt in full on every photos-timeline page; the dotfile filter and ResourceList.itemIndexById re-scan the whole accumulated list per page.

Environment / methodology

• cargo run --release --features bench --example bench_round17_micro — counting global allocator, no Postgres. Tunables: BENCH_ITERS (100000), BENCH_CHUNKS (64), BENCH_DUP_RATIO (2).
• Each section is BEFORE (verbatim replica of the shipped-before shape) vs AFTER (verbatim replica of the shipped-after shape) with a byte/-value equivalence gate; the shipped source now matches each AFTER arm.
• Roll-back rule encoded per section: std::process::exit(1) with GATE FAIL … rollback if an AFTER arm fails to reduce allocations.