Oxicloud/benches/ROUND11.md

# Round 11 — StoragePath re-representation, classifier fusion, memoized static bodies, query-shape pack, SPA fine-grained stars

Benchmark-gated, same rule as ROUND2-10: every change ships with a
BEFORE/AFTER benchmark and an equivalence/safety gate; an AFTER that doesn't
beat its BEFORE gets rolled back or redesigned. Four candidates went through
exactly that loop this round (§Rejected below): the moka `and_upsert_with`
rate-limiter rewrite, the GET/HEAD `Last-Modified` stack-render port, the
`min(uuid)` geo-cluster cast (PostgreSQL has no such aggregate — the gate
caught it before it could ship broken), and the `tracing-appender`
non-blocking log writer (slower than sync on fast sinks, tail-loss risk on
slow ones).

Measured on 4 cores / 15 GiB, local PostgreSQL 16 (fsync off), release
profile; frontend on Node 22 / vitest 4 (jsdom). Reproduce any row with the
command in its section (`benches/ROUND11.md` §Environment).

## Summary

| # | change | key metric | before → after |
|--:|---|---|---|
| 1 | REST download: dead `FileDto` clone → capture mime/size + move | ns / allocs per download | 295.3 → 20.7 ns · 7 → 0 allocs |
| 2 | `StoragePath` → single canonical joined `String`; `File`/`Folder` drop the duplicate `path_string` field | 500-row page (depth 4) | 121.0 → 80.3 µs (**1.51x**) · 4 000 → 1 000 allocs |
| 3 | Display classifier fusion (`classify_display`, one stack-lowered ext for the three trees) | 13-row mixed corpus | 4 390 → 2 074 ns (**2.12x**) · 21 → 0 allocs |
| 4 | `/status.php` → `OnceLock<Bytes>` | per NC client poll | 836 → 28.8 ns (**29x**) · 14 → 0 allocs |
| 5 | `/openapi.json` → `OnceLock<Bytes>` (was rebuilding the 171 KiB spec per request) | per request | 2.77 ms → 18.5 ns · 12 474 → 0 allocs |
| 6 | NC upload-session PROPFIND: `write!` + pre-sized body + stack dates | 256-chunk session | 203.7 → 75.4 µs (**2.70x**) · 2 582 → 772 allocs |
| 7 | CSRF token: borrow-only compare (+ borrowed cookie extraction) | per state-changing request | 55.2 → 2.2 ns · 1 → 0 allocs |
| 8 | Thumbnail/preview ETag via `as_str` (Debug-identical bytes — cached client ETags stay valid) | per thumbnail request | 150.5 → 87.7 ns · 3 → 2 allocs |
| 9 | Recent-handler id: stack `encode_lower` | per record/remove | 52.2 → 10.7 ns · 1 → 0 allocs |
| 10 | 4xx body: borrowed `ErrorResponse` + `ErrorKind::as_str` + `not_found`/`already_exists` clone kill | per 404 | 426 → 367 ns · 11 → 8 allocs |
| 11 | vCard emit: `write!` + borrowed address fields | per contact create/update | 804 → 386 ns (**2.08x**) · 21 → 5 allocs |
| 12 | Search page: `into_iter().skip().take()` move (both branches) | 50-item page | 108.3 → 89.4 µs · −301 allocs |
| 13 | Content-hit verify: parse each UUID once | 100-hit page | 7.90 → 5.20 µs (**1.52x**) |
| 14 | Group last-user check: HashSet probe | 500×500 check | 105.3 → 17.1 µs (**6.1x**) |
| 15 | Retry op-label: lazy closure (success path never formats) | per blob op | 71.8 → 0.7 ns · 2 → 0 allocs |
| 16 | `encrypt_bytes`: in-place detached, single buffer (write now mirrors the in-place read) | 256 KiB chunk | 231.1 → 208.1 µs (**1.11x**) · 2 → 1 allocs |
| 17 | Encrypted `collect_stream`: chunk-sized reserve | 1 MiB blob, 4 KiB frames | 60.4 → 53.2 µs · 9 → 1 allocs |
| 18 | Recluster cosine: norms precomputed once (bit-identical gate over all pairs) | 200 faces × 512-dim pass | 7.70 → 7.17 ms (**1.07x**) |
| 19 | `CalendarEventDto`: `into_parts` move (the ~11 KiB `ical_data` memcpy gone) | per CalDAV event row | 497 → 283 ns (**1.76x**) · 14 → 8 allocs |
| 20 | RateLimiter: lock-free `get` (borrows the key) + `insert` | per limited request | allocs 8.0 → 6.0 · wall neutral (1 657 vs 1 695 ns, within run variance) |
| Q1 | Deferred upload registration: 3 round-trips → 1 CTE insert (the `persist_file` template) | per uploaded file (incl. cleanup DELETE) | 1.83 → 1.32 ms · gates: identical path/drive, missing-parent → not-found |
| Q2 | Calendar/AddressBook/Playlist authz `direct_grant_cache` (single-flight, invalidated on `set_role`/`clear_role`) | per DAV check | 0.197 ms → 0.2 µs on hit (**~1000x**) · revocation-flip gate OK |
| Q3 | `expand_user`: `tokio::join!` the `is_external` read + groups CTE | per cold expansion | 0.426 → 0.204 ms (**2.1x**) |
| Q5 | Recluster persistence: per-face UPDATE loop → one UNNEST batch | 200-face apply (incl. reset) | 80.0 → 8.7 ms (**9.2x**) · final column state identical |
| S1 | SPA `ResourceList.selectedEntries`: O(N)×2 per toggle → id-index O(k·log k); hosts consume the snippet param | comparisons per 51-toggle gesture (N=2 000) | 204 000 → <2 602 · identical output/order gated |
| S2 | SPA Recent: star reads the new `favoriteIds` prop — mapper no longer depends on the set | rows re-mapped per star click (N=400) | 400 → 0 · identical star states gated |
| S3 | SPA admin `timeAgo` >30d: cached `Intl.DateTimeFormat` | constructions per 1 000 formats | ≤1 · output equals `toLocaleDateString()` |

Also shipped without a dedicated row: CardDAV `getlastmodified` per-contact
stack render (ROUND10-§13 helper + chrono fallback), NC capabilities poll
logs demoted to `debug` (each poll forced a formatted line + locked stdout
write), trash `to_dto` `into_parts` move + fused/interned display fields
(trash listing and path-resolver rows now share the ROUND9 interning),
`TrashedItemDto` name/path moves.

Cross-round regression guards re-run after the StoragePath / classifier
rework — both gates PASS byte-identical, and the shipped code now beats the
numbers those rounds recorded:

- `bench_row_path` (round 4): file row 705 → 280 ns/row (2.52x), allocs
  15.75 → 6.00; folder row 684 → 365 ns (1.87x), 14.08 → 5.24.
- `bench_dto_map` (round 3): File→FileDto 839 → 606 ns/row, 10.09 → 3.08
  allocs; Folder→FolderDto 314 → 161 ns, 11.80 → 1.00 allocs.

## [1] REST download dead clone → move

```
cargo run --release --features bench --example bench_round11_micro   # §1
```

`download_file_impl` cloned the whole `FileDto` (7 owned Strings) into
`get_file_optimized_preloaded` on every authenticated download, purely to
read `mime_type`/`size` afterwards — the share path already captured+moved
(ROUND10 §3 fixed its double-*fetch*, not this clone). Now: one `Arc<str>`
bump + a `u64` copy, then move. 295.3 → 20.7 ns, 7 → 0 allocs per download.

## [2] StoragePath joined-only representation

```
cargo run --release --features bench --example bench_round11_micro   # §20
cargo run --release --features bench --example bench_row_path        # cross-round gate
```

`StoragePath` stored `segments: Vec<String>` — one heap String per path
component built on EVERY hydrated row — while the DTO path only ever
consumed the joined form, and the entities carried a second `path_string`
duplicate. The value object now stores the canonical joined `String` alone
(`"/"` or `/seg(/seg)*`); `file_name`/`parent`/`segments()`/`Display`
derive on demand; `File`/`Folder` lost the duplicate field (`path_string()`
borrows). `.segments()` had zero external callers — verified before the
rework. Equivalence gates: identical `path_string`, `file_name`, `parent`,
`Display` across the corpus, plus the round-4 harness's byte-identical
gate. 500-row page: 121.0 → 80.3 µs, 4 000 → 1 000 allocs; per-row RAM
drops by the Vec + per-segment String headers + the duplicate path.

## [3] Display classifier fusion

```
cargo run --release --features bench --example bench_round11_micro   # §21
```

Every listed file ran the full MIME classification three times
(`icon_class_for`, `icon_special_class_for`, `category_for`), each
heap-allocating its own `to_ascii_lowercase()` on the extension-fallback
path. The three decision trees are byte-for-byte preserved (they diverge
deliberately, so no merged tree); `classify_display` lowers the extension
once into a 16-byte stack buffer shared by all three. Extensions longer
than any table entry short-circuit to the same `_`-arm defaults (gated).
Call sites: `FileDto::from`, folder/favorites/recent handlers, trash
listing (×2), path-resolver — the last two also gained the ROUND9
interning they had missed (`Arc::from` per row → refcount bump).
13-row corpus: 4 390 → 2 074 ns, 21 → 0 allocs.

## [4][5] Memoized process-invariant bodies

```
cargo run --release --features bench --example bench_round11_micro   # §3, §18
```

`/status.php` rebuilt its `json!` tree per NC client poll (836 ns / 14
allocs → 28.8 ns / 0). `/openapi.json` was the extreme case: utoipa
reconstructed and re-serialized the whole 171 KiB spec on every request —
2.77 ms / 12 474 allocs → 18.5 ns / 0 via the same `OnceLock<Bytes>`
pattern as ROUND9's capabilities memoization. Byte-identical gates on both.

## [6] NC upload-session PROPFIND emit

The last hand-built XML handler: `String::new()` + `push_str(&format!(…))`
per element per chunk + chrono `to_rfc2822()` per chunk. Now pre-sized +
`write!` + `common::fmt::rfc2822_utc` (chrono fallback out-of-range).
Byte-identical gates at 16 and 256 chunks (escape distributes over
concatenation; RFC 2822 output has no XML-special chars). 16 chunks:
13.2 → 5.2 µs; 256 chunks: 203.7 → 75.4 µs, 2 582 → 772 allocs.

## [Q1] Deferred upload registration 3 → 1

```
cargo run --release --features bench --example bench_round11_queries # §1
```

The write-behind REST upload path ran parent-drive SELECT → INSERT →
parent-path SELECT, the first and third re-reading the identical
`storage.folders` row. Ported to the `WITH parent AS (…) INSERT … SELECT
… RETURNING (SELECT path FROM parent)` template `persist_file` has used
since ROUND2 — 0 rows ⇒ the same `not_found("Folder")` the old first query
produced (gated, anti-enum shape preserved). Root uploads (no parent)
keep their previous two-step shape.

## [Q2] direct_grant_cache

Calendar/AddressBook/Playlist were the only `check()` arms with no result
cache — every CalDAV/CardDAV/music request re-ran the `role_grants` point
query, and DAV clients poll continuously. Added
`direct_grant_cache: Cache<(Subject, Resource, Permission), bool>`
(30 s TTL / 100k, `try_get_with` single-flight — the ROUND8/10 pattern),
flushed on `set_role`/`clear_role` for those resource types; group/expiry
churn self-heals within the TTL exactly like `cascade_grant_cache`.
Gates: identical verdict; a revocation + flush flips the next check.
0.197 ms → 0.2 µs on hit.

## [Q3] expand_user join!

The `is_external` point read and the recursive groups CTE are independent;
serial await paid two round-trips end-to-end on every cold expansion
(per user per 30 s TTL window). `tokio::join!`: 0.426 → 0.204 ms.

## [Q5] Recluster UNNEST batch

`POST /api/people/recluster` issued one `UPDATE faces.faces SET person_id`
per face sequentially (both the unassign-small-clusters and assign loops).
Assignments now accumulate and apply as a single
`UPDATE … FROM unnest($1::uuid[], $2::uuid[])` (the ROUND10 `save_faces`
pattern) — 200-face library: 80.0 → 8.7 ms, final column state identical.
Pairs with §18's norm precomputation on the CPU side (7.70 → 7.17 ms for
the O(N²) pass, bit-identical similarity gated over every pair).

## [S1][S2][S3] SPA pack (vitest gates)

```
cd frontend && npx vitest run src/lib/components/round11.bench.test.ts
```

- **`ResourceList.selectedEntries`** (the ROUND10 flagged item): the
  component re-filtered the ENTIRE items array per selection change, and
  favorites/recent ignored the snippet param and recomputed their own
  `entries.filter(…)` shadow — two full O(N) scans per toggle, O(N²)-ish
  across a shift-range. Now an id→index Map (rebuilt only when `items`
  changes) projects the selection in O(k·log k) preserving item order;
  hosts consume the snippet param and their dead `selectedIds` mirror is
  gone (the component's prune effect already self-heals on reload).
  51-toggle gesture on 2 000 items: 204 000 → <2 602 comparisons.
- **Recent favorite star**: the entry mapper read `favoriteIds.has(id)`,
  subscribing the whole O(N) map to the SvelteSet — one star click rebuilt
  all N entries and re-rendered every visible row. `ResourceList` gained a
  `favoriteIds` prop read directly by the star widget; the mapper is
  set-independent. Star click: N → 0 rows re-mapped, star states gated
  identical.
- **admin `timeAgo`**: the >30-day fallback called `toLocaleDateString()`
  (a fresh `Intl.DateTimeFormat` per call); now the app-wide cached
  formatter — output equality gated, ≤1 construction per 1 000 formats.

## Rejected / reworked this round (the discipline working)

- **`min(fm.file_id)::text` geo-cluster cast (Q4)**: PostgreSQL has **no
  `min(uuid)` aggregate** — the "cast once per cluster" rewrite fails to
  parse (`42883`). The gate caught it before it could ship broken; the
  per-row-cast original stays (22.6 ms per 5k-row viewport, admin-shaped
  traffic), and a custom `CREATE AGGREGATE` was judged schema surface this
  query doesn't justify. The bench section now reproduces the rejection.
- **RateLimiter `entry().and_upsert_with`**: 1 657 → 2 365 ns and
  8.0 → 9.1 allocs — moka's compute-entry machinery costs more than the
  two ops it replaced. Redesigned as lock-free `get` (borrows the key, no
  alloc) + `insert`: 6.0 allocs, wall within variance; identical counter
  sequence gated. Adopted in that form.
- **GET/HEAD `Last-Modified` stack-render port**: chrono's `to_rfc2822()`
  String IS the terminal allocation the header needs (38.7 ns incl. alloc
  vs 47.0 ns stack render + the same alloc). Only body-emit sites (where
  `write!` lands in an existing buffer, 31.8 ns / 0 allocs) benefit —
  those were ported (§6 + CardDAV); header sites stay on chrono.
- **`tracing-appender` non-blocking log writer (L1)**: on a fast sink
  (stdout→/dev/null — the containerized default) sync sustains 1.41M ev/s
  vs 0.99M non-blocking, with a better tail (p999 76 vs 151 µs, max 0.8
  vs 8.3 ms — the channel hop costs more than the write). Non-blocking
  only wins on a slow sink (20 µs/line: 6.4x wall, p50 22 → 1.9 µs), but
  there the drain gate FAILED — buffered tail lines can be lost at
  shutdown, unacceptable for the audit channel. Not adopted;
  `tracing-appender` stays as a dev-dependency for the reproducible
  harness (`bench_log_writer`, 4 arms via `BENCH_LOG_ARM`/`_WRITER`).
- **First search-page model (`drain(range)`)**: −300 allocs but slower on
  wall (tail memmove). Reshaped as `into_iter().skip().take().collect()`
  — moves the page, drops the rest, wins both axes (§12 row in Summary).

## Deferred / flagged (not shipped this round)

- **NC preview 304 still runs `get_file`**: dropping the fetch on the
  revalidation path changes existence semantics (deleted file → 304
  instead of 404) — needs maintainer sign-off, same class as the standing
  CalDAV authz-before-fetch reorder (ROUND9/10 flag).
- **`CachedBlobBackend`'s `Mutex<LruCache>` index** serializes every
  cached read on remote+cache deployments; the moka byte-weigher
  migration (file-content-cache pattern) deserves its own round with a
  concurrency bench and eviction-unlink care.
- **Capture-metadata extraction reads each media file 2-3×**
  (`media_metadata_service`: kamadak full read + nom-exif path re-read +
  track fallback). Feeding nom-exif from the in-memory buffer needs its
  `MediaSource` API verified on the pinned version.
- **`CachedBlobBackend::local_blob_path` sync `stat`** (ROUND10 flag
  stands — needs an async port variant).
- **Azure SDK 0.21 stack** drags duplicate dependency trees (h2 0.3+0.4,
  three hashbrown generations, base64 0.13, getrandom 0.1) into every
  build — an SDK bump is a dedicated migration.
- **`AudioMetadataRepository::list_by_{artist,album,genre}`** are dead
  code with seq-scan `ILIKE` shapes — flag for deletion, not indexing.
- **`CachedBlobBackend::put_blob` cache population** silently fails for
  S3/Azure whole-file puts (inner backend deletes the source before the
  cache copy) — correctness note for maintainers.
- **Grouped file/grid views unvirtualized** (ROUND10 flag stands).

## Environment / methodology

- `cargo run --release --features bench --example bench_round11_micro`
  — 21 sections, counting allocator, BEFORE replicas vs shipped shapes,
  equivalence gates inline (`BENCH_ITERS`, default 100k).
- `cargo run --release --features bench --example bench_round11_queries`
  — needs Postgres; seeds and sweeps its own fixtures (`BENCH_PASSES`).
- `BENCH_LOG_ARM=sync|nonblocking [BENCH_LOG_WRITER=slow] cargo run
  --release --features bench --example bench_log_writer >/dev/null`.
- `cd frontend && npx vitest run src/lib/components/round11.bench.test.ts`.
- Cross-round guards: `bench_row_path`, `bench_dto_map`.