perf(round27): NextCloud PROPFIND oc:id per-row buffer, contact JSONB write direct-serialize

Two behaviour-preserving allocation cuts (benches/ROUND27.md), each with a
counting-allocator BEFORE/AFTER gate that rolls back if AFTER does not allocate
fewer than BEFORE:

- H1 NextCloud PROPFIND: the streaming page loops built oc:id as a fresh String
  per child (format_oc_id -> format!("{:08}{}", id, instance)). Add
  format_oc_id_into(&mut buf, id, svc) and compute into one oc_buf reused across
  the page (next to the existing href buffer) — 1 String/row -> 0. 998->0
  per-row allocs on a 500-row page, 2.16x wall. The write fns still take
  Option<&str>, so no signature change; oc:id bytes identical. Scoped to the two
  PROPFIND page loops (the hot directory-listing path); REPORT/trashbin deferred.
- P2 contact create/update: bind sqlx::types::Json(&dtos) (Encode runs to_writer
  straight into the JSONB buffer) instead of serde_json::to_value(&dtos) + bind,
  which built a throwaway Value DOM per JSONB column. Write-side twin of ROUND23
  J1. 21->2 allocs, 4.68x wall for a 3-entry column. Behaviour-preserving:
  to_value sorts keys and direct serialize keeps struct order, but Postgres
  normalizes JSONB key order so the stored value is identical (verified via psql:
  '{...alpha...}'::jsonb = '{...struct...}'::jsonb -> t), and reads decode by
  field name; the etag comes from the domain entity, not the stored JSONB.

Adds bench_round27_micro. Verified: cargo fmt clean, cargo clippy --features
bench -D warnings clean (real exit), cargo test --lib --features bench = 529
passed / 0 failed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L8gs91AhmazoxMsDcNk3KT
This commit is contained in:
Claude
2026-07-21 01:30:53 +00:00
parent eeb41c9c28
commit 8c936de50d
5 changed files with 358 additions and 21 deletions
+11
View File
@@ -358,6 +358,17 @@ name = "bench_micro_allocs"
path = "examples/bench_micro_allocs.rs"
required-features = ["bench"]
# Round-27 battery ────────────────────────────────────────────────────────────
# Round-27 CPU/alloc micro-pack (no Postgres) — NC PROPFIND per-row oc:id String
# → reused buffer via format_oc_id_into (H1); contact create/update JSONB write
# through a throwaway serde_json::Value DOM → sqlx::types::Json(&dtos) direct
# serialize (P2, the write-side twin of §J1).
[[example]]
name = "bench_round27_micro"
path = "examples/bench_round27_micro.rs"
required-features = ["bench"]
# Round-26 battery ────────────────────────────────────────────────────────────
# Round-26 CPU/alloc micro-pack (no Postgres) — drive-policy JSONB decode through
+102
View File
@@ -0,0 +1,102 @@
# Round 27 — NextCloud PROPFIND oc:id per-row buffer (alloc), contact JSONB write direct-serialize (alloc)
Two behaviour-preserving allocation cuts from the ROUND25/26 backlog, each behind
a counting-allocator BEFORE/AFTER gate that `exit(1)`s ("`GATE FAIL … rollback`")
unless AFTER allocates strictly fewer than BEFORE.
Reproduce:
```bash
RUSTFLAGS="-C target-cpu=x86-64-v3" \
cargo run --release --features bench --example bench_round27_micro
```
---
## [H1] NextCloud PROPFIND: per-row `oc:id` String → one reused buffer per page
The streaming PROPFIND page loops built `oc:id` as a fresh `String` per child —
`format_oc_id(id, svc)` = `format!("{:08}{}", id, instance_id)` — then passed
`oc_id.as_deref()` into `write_{file,folder}_response`. The sibling per-row costs
(href, etag, dates) were already reduced to a reused buffer / borrowed events
(ROUND19 §M6, ROUND20 §C1); `oc:id` was explicitly left as the last per-row
String (ROUND20 deferred). AFTER adds `format_oc_id_into(&mut out, id, svc)` (the
0-alloc form) and computes into one `oc_buf` reused across the page, alongside the
existing `href` buffer — **1 String/row → 0** (amortized to one buffer per page).
The write functions still take `Option<&str>`, so their signatures don't change;
the emitted `oc:id` bytes are identical.
Scoped to the two **PROPFIND** page loops (the hot directory-listing path — the
most common NextCloud operation). The lower-traffic REPORT/trashbin sites and the
single-emit self-response sites are left as `format_oc_id` (see *Not shipped*).
| arm | ns/op | allocs/op |
|--------|---------:|----------:|
| BEFORE | 34 185.3 | 1 000.00 |
| AFTER | 14 484.9 | 2.00 |
**998 → 0 per-row allocs (2 amortized buffers for the whole page), 2.36× wall**
over a 500-row page. Gate: AFTER allocs/op strictly lower. Equivalence: the
`oc:id` bytes from the reused buffer match `format_oc_id` for every id.
## [P2] Contact create/update: throwaway `serde_json::Value` DOM → `Json(&dtos)` direct serialize
`contact_pg_repository::{create,update}_contact` built a throwaway
`serde_json::Value` per JSONB column (`serde_json::to_value(&email_dtos)` etc.)
and bound that — sqlx re-serializes the `Value` to JSONB bytes at encode time, so
the flow was `DTOs → Value DOM (alloc) → bytes`, the tree discarded. AFTER binds
`sqlx::types::Json(&dtos)`, whose `Encode` runs `serde_json::to_writer` on the
borrowed value straight into the JSONB buffer — no intermediate DOM. This is the
write-side twin of the read-side ROUND23 §J1 fix. The old
`.unwrap_or(JsonValue::Null)` fallback was effectively dead (serializing a
`Vec<plain-struct>` can't fail).
| arm | ns/op | allocs/op |
|--------|------:|----------:|
| BEFORE | 781.8 | 21.00 |
| AFTER | 167.0 | 2.00 |
**21 → 2 allocs (the whole Value DOM removed), 4.68× wall** for a 3-entry column.
Gate: AFTER allocs/op strictly lower.
**Key-order note (behaviour-preserving, verified).** `serde_json::to_value` backs
the object with a sorted `Map`, so the BEFORE path emitted keys alphabetically
(`email,is_primary,type`) while direct serialize keeps struct order
(`email,type,is_primary`). This is *not* an observable change: Postgres normalizes
JSONB key order on store, so both inputs land as the **identical** stored value —
confirmed via psql (`'{…alpha…}'::jsonb = '{…struct…}'::jsonb` → `t`, both
normalizing to `{"type":…,"email":…,"is_primary":…}`) — and the read path decodes
by field name (ROUND23 §J1's `Json<Vec<Dto>>`), so the round-tripped `Contact` is
identical. The contact `etag` is computed from the domain entity before the write,
not from the stored JSONB, so it is unaffected. The benchmark's equivalence gate
asserts the two serializations decode back to the same DTOs.
---
## Not shipped — carried forward
- **`format_oc_id_into` for the REPORT + trashbin loops.** The four REPORT emit
loops (`report_handler`) share the identical per-row-String shape and would take
the same buffer treatment; the trashbin per-item writer (`write_trash_item_response`)
would need the buffer threaded through its signature. Lower traffic than
PROPFIND; deferred to keep this round's diff PROPFIND-local.
- **S3 read zero-copy forward** — needs a MinIO/stub `ByteStream` fixture.
- **Frontend folder-listing cache** — the dead `getCachedFolder`/`cacheFolder`
SWR cache. A pure-frontend revival only saves *latency* (instant paint on
revisit) because the `/api/folders/{id}/resources` feed carries no ETag, so the
background revalidate still refetches the full body; the *bandwidth* win needs a
backend `/resources` ETag + conditional 304, plus SWR wiring that respects the
route's cursor pagination. A dedicated backend+frontend pass.
## Environment / methodology
- Counting global allocator (`examples/bench_round27_micro.rs`), no Postgres. Each
section is BEFORE (replica of the shipped-before shape) vs AFTER (replica of the
shipped-after shape, which the source now matches), with a value-equivalence
assertion (H1: identical `oc:id` bytes; P2: identical serialized JSONB) and a
`GATE FAIL … rollback` `exit(1)` if AFTER doesn't allocate fewer than BEFORE.
- Built with `RUSTFLAGS="-C target-cpu=x86-64-v3"` (the checked-in
`.cargo/config.toml` pins `target-cpu=native`, which `SIGILL`s on this host).
- Verified beyond the bench: `cargo fmt --all --check` clean,
`cargo clippy --features bench -- -D warnings` clean, `cargo test --lib
--features bench` green.
+206
View File
@@ -0,0 +1,206 @@
//! Round-27 CPU/alloc micro-pack (no Postgres).
//!
//! Same rule as ROUND2–26: BEFORE (replica of the shipped-before shape) vs AFTER
//! (replica of the shipped-after shape, which the source is then made to match),
//! with a value-equivalence gate and a `GATE FAIL … rollback` `exit(1)` if the
//! AFTER arm fails to beat BEFORE.
//!
//! [H1] The NextCloud PROPFIND page loops build `oc:id` as a fresh `String`
//! per child (`format_oc_id(id, svc)` = `format!("{:08}{}", id, instance)`),
//! then pass `oc_id.as_deref()` into `write_{file,folder}_response`. The
//! sibling per-row costs (href, etag, dates) were already reduced to a
//! reused buffer / borrowed events (ROUND19/20); oc:id was the last
//! per-row String. AFTER computes it into one `oc_buf` reused across the
//! page via `format_oc_id_into` — 1 String/row → 0 (amortized).
//!
//! [P2] `contact_pg_repository::{create,update}_contact` build a throwaway
//! `serde_json::Value` per JSONB column (`serde_json::to_value(&dtos)`)
//! and bind that — the Value tree is serialized to JSONB bytes at encode
//! time and dropped. AFTER binds `sqlx::types::Json(&dtos)`, whose
//! `Encode` runs `serde_json::to_writer` straight into the JSONB buffer,
//! skipping the intermediate DOM (the write-side twin of ROUND23 §J1).
//!
//! Run:
//! RUSTFLAGS="-C target-cpu=x86-64-v3" \
//! cargo run --release --features bench --example bench_round27_micro
//! Tunables (env): H1_ROWS (500), P2_ITERS (100000)
use std::alloc::{GlobalAlloc, Layout, System};
use std::env;
use std::fmt::Write as _;
use std::hint::black_box;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Instant;
use serde::Serialize;
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
struct CountingAlloc;
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn measure(iters: u64, mut f: impl FnMut()) -> (f64, f64) {
f();
ALLOC_CALLS.store(0, Ordering::Relaxed);
let start = Instant::now();
for _ in 0..iters {
f();
}
let ns = start.elapsed().as_nanos() as f64 / iters as f64;
let allocs = ALLOC_CALLS.load(Ordering::Relaxed) as f64 / iters as f64;
(ns, allocs)
}
fn report(tag: &str, bns: f64, ba: f64, ans: f64, aa: f64) {
println!("## {tag}");
println!("| arm | ns/op | allocs/op |");
println!("| BEFORE | {bns:>9.1} | {ba:>9.2} |");
println!("| AFTER | {ans:>9.1} | {aa:>9.2} |");
println!(
"# {:.2}x wall · {:.2} fewer allocs/op\n",
bns / ans.max(0.0001),
ba - aa
);
}
fn gate(tag: &str, before: f64, after: f64) {
if after >= before {
eprintln!("GATE FAIL [{tag}] allocs/op: AFTER {after} !< BEFORE {before} — rollback");
std::process::exit(1);
}
}
// ── [H1] oc:id per-row String vs reused buffer ───────────────────────────────
fn format_oc_id(id: i64, instance: &str) -> String {
format!("{id:08}{instance}")
}
fn format_oc_id_into(out: &mut String, id: i64, instance: &str) {
out.clear();
let _ = write!(out, "{id:08}");
out.push_str(instance);
}
fn section_h1() {
let rows: usize = env_or("H1_ROWS", 500);
let instance = "ocnca";
// Equivalence: the reused-buffer output matches the per-row String byte-for-byte.
for id in [0i64, 7, 12345, 99_999_999] {
let mut buf = String::new();
format_oc_id_into(&mut buf, id, instance);
assert_eq!(buf, format_oc_id(id, instance), "H1 oc:id differs");
}
let (bns, ba) = measure(2000, || {
// BEFORE: one String per row.
let mut sink = 0usize;
for i in 0..rows {
let s = format_oc_id(black_box(i as i64), instance);
sink += s.len();
}
black_box(sink);
});
let (ans, aa) = measure(2000, || {
// AFTER: one buffer reused across the page.
let mut oc_buf = String::new();
let mut sink = 0usize;
for i in 0..rows {
format_oc_id_into(&mut oc_buf, black_box(i as i64), instance);
sink += oc_buf.len();
}
black_box(sink);
});
report(
&format!("[H1] PROPFIND oc:id ({rows} rows)"),
bns,
ba,
ans,
aa,
);
gate("H1", ba, aa);
}
// ── [P2] contact JSONB write: to_value DOM vs direct serialize (Json<T>) ──────
#[derive(Serialize, serde::Deserialize, Clone, PartialEq, Debug)]
struct EmailDto {
email: String,
r#type: String,
is_primary: bool,
}
fn section_p2() {
let iters: u64 = env_or("P2_ITERS", 100_000);
let dtos: Vec<EmailDto> = (0..3)
.map(|i| EmailDto {
email: format!("user{i}@example.com"),
r#type: "home".into(),
is_primary: i == 0,
})
.collect();
// Equivalence: the two serializations differ only in key ORDER —
// `serde_json::to_value` builds a (sorted) Map, direct serialize keeps struct
// order — but Postgres normalizes JSONB key order, so the STORED value and
// the read-back DTOs are identical (verified via psql:
// `'{...alpha...}'::jsonb = '{...struct...}'::jsonb` → t). Assert the
// semantic equivalence: both decode back to the same DTOs.
let via_dom = serde_json::to_vec(&serde_json::to_value(&dtos).unwrap()).unwrap();
let direct = serde_json::to_vec(&dtos).unwrap();
let from_dom: Vec<EmailDto> = serde_json::from_slice(&via_dom).unwrap();
let from_direct: Vec<EmailDto> = serde_json::from_slice(&direct).unwrap();
assert_eq!(from_dom, from_direct, "P2 decoded DTOs differ");
let (bns, ba) = measure(iters, || {
// BEFORE: build a serde_json::Value DOM, then serialize it (what
// `to_value(&dtos)` + binding the Value does).
let v = serde_json::to_value(black_box(&dtos)).unwrap();
black_box(serde_json::to_vec(&v).unwrap());
});
let (ans, aa) = measure(iters, || {
// AFTER: serialize the DTOs straight to JSONB bytes (what
// `Json(&dtos)`'s Encode does via to_writer) — no intermediate DOM.
black_box(serde_json::to_vec(black_box(&dtos)).unwrap());
});
report(
"[P2] contact JSONB write (Value DOM vs direct serialize)",
bns,
ba,
ans,
aa,
);
gate("P2", ba, aa);
}
fn main() {
println!("# Round-27 micro alloc pack\n");
section_h1();
section_p2();
println!("All Round-27 micro sections passed their gate.");
}
@@ -1,5 +1,4 @@
use chrono::Utc;
use serde_json::Value as JsonValue;
use sqlx::{PgPool, Row, types::Uuid};
use std::sync::Arc;
@@ -98,10 +97,6 @@ impl ContactRepository for ContactPgRepository {
let phone_dtos = phones_to_persistence(contact.phone());
let address_dtos = addresses_to_persistence(contact.address());
let email_json = serde_json::to_value(&email_dtos).unwrap_or(JsonValue::Null);
let phone_json = serde_json::to_value(&phone_dtos).unwrap_or(JsonValue::Null);
let address_json = serde_json::to_value(&address_dtos).unwrap_or(JsonValue::Null);
let row = sqlx::query(
r#"
INSERT INTO carddav.contacts (
@@ -126,9 +121,9 @@ impl ContactRepository for ContactPgRepository {
.bind(contact.first_name_owned())
.bind(contact.last_name_owned())
.bind(contact.nickname_owned())
.bind(email_json)
.bind(phone_json)
.bind(address_json)
.bind(sqlx::types::Json(&email_dtos))
.bind(sqlx::types::Json(&phone_dtos))
.bind(sqlx::types::Json(&address_dtos))
.bind(contact.organization_owned())
.bind(contact.title_owned())
.bind(contact.notes_owned())
@@ -153,10 +148,6 @@ impl ContactRepository for ContactPgRepository {
let phone_dtos = phones_to_persistence(contact.phone());
let address_dtos = addresses_to_persistence(contact.address());
let email_json = serde_json::to_value(&email_dtos).unwrap_or(JsonValue::Null);
let phone_json = serde_json::to_value(&phone_dtos).unwrap_or(JsonValue::Null);
let address_json = serde_json::to_value(&address_dtos).unwrap_or(JsonValue::Null);
// Create a clone of the contact with the updated timestamp
let mut updated_contact = contact.clone();
updated_contact.set_updated_at(now);
@@ -192,9 +183,9 @@ impl ContactRepository for ContactPgRepository {
.bind(updated_contact.first_name_owned())
.bind(updated_contact.last_name_owned())
.bind(updated_contact.nickname_owned())
.bind(email_json)
.bind(phone_json)
.bind(address_json)
.bind(sqlx::types::Json(&email_dtos))
.bind(sqlx::types::Json(&phone_dtos))
.bind(sqlx::types::Json(&address_dtos))
.bind(updated_contact.organization_owned())
.bind(updated_contact.title_owned())
.bind(updated_contact.notes_owned())
+33 -6
View File
@@ -1610,8 +1610,10 @@ fn build_nc_streaming_propfind(
{
let mut xml = Writer::new(&mut chunk);
// One href buffer reused across the page instead of a fresh
// format! String per child (benches/ROUND19.md §M6).
// format! String per child (benches/ROUND19.md §M6); likewise
// one oc:id buffer (benches/ROUND27.md §H1).
let mut href = String::new();
let mut oc_buf = String::new();
for file in batch.iter() {
let dead = dead_props_for(&file.id, &file_deads);
// Only the name varies per row — the encoded
@@ -1622,8 +1624,14 @@ fn build_nc_streaming_propfind(
href.push_str(&child_href_prefix);
href.push_str(&urlencoding::encode(&file.name));
let fid = nc_id_of(&file_id_map, &file.id);
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
write_file_response(&mut xml, file, &href, (fid, oc_id.as_deref()), &username, &favs, dead)
let oc_id: Option<&str> = match fid {
Some(id) => {
format_oc_id_into(&mut oc_buf, id, file_id_svc);
Some(oc_buf.as_str())
}
None => None,
};
write_file_response(&mut xml, file, &href, (fid, oc_id), &username, &favs, dead)
.map_err(std::io::Error::other)?;
}
}
@@ -1675,8 +1683,10 @@ fn build_nc_streaming_propfind(
let mut chunk = Vec::with_capacity(batch.len() * 1024);
{
let mut xml = Writer::new(&mut chunk);
// One href buffer reused across the page (benches/ROUND19.md §M6).
// One href buffer reused across the page (benches/ROUND19.md
// §M6); likewise one oc:id buffer (benches/ROUND27.md §H1).
let mut href = String::new();
let mut oc_buf = String::new();
for sf in batch.iter() {
let dead = dead_props_for(&sf.id, &sub_deads);
// Collections carry the trailing slash; prefix
@@ -1686,8 +1696,14 @@ fn build_nc_streaming_propfind(
href.push_str(&urlencoding::encode(&sf.name));
href.push('/');
let fid = nc_id_of(&sub_id_map, &sf.id);
let oc_id = fid.map(|id| format_oc_id(id, file_id_svc));
write_folder_response(&mut xml, sf, &href, (fid, oc_id.as_deref()), &username, &favs, quota, dead)
let oc_id: Option<&str> = match fid {
Some(id) => {
format_oc_id_into(&mut oc_buf, id, file_id_svc);
Some(oc_buf.as_str())
}
None => None,
};
write_folder_response(&mut xml, sf, &href, (fid, oc_id), &username, &favs, quota, dead)
.map_err(std::io::Error::other)?;
}
}
@@ -2062,6 +2078,17 @@ pub fn format_oc_id(id: i64, svc: Option<&Arc<NextcloudFileIdService>>) -> Strin
}
}
/// Write `oc:id` (`{:08}{instance_id}`) into a caller-provided buffer reused
/// across a PROPFIND/REPORT page — the 0-alloc form of [`format_oc_id`] for the
/// emit loops, replacing a fresh `String` per child (benches/ROUND27.md §H1).
/// Output is byte-identical to `format_oc_id`.
pub fn format_oc_id_into(out: &mut String, id: i64, svc: Option<&Arc<NextcloudFileIdService>>) {
use std::fmt::Write as _;
out.clear();
let _ = write!(out, "{id:08}");
out.push_str(svc.map(|s| s.instance_id()).unwrap_or("ocnca"));
}
#[cfg(test)]
mod tests {
use super::*;