perf: round 15 — grouped-listing O(N²) rebucket, exif/reseed allocs, tantivy zero-hit snippet skip
Benchmark-gated, same rule as rounds 2–14: every change ships with a BEFORE/AFTER benchmark and an equivalence/safety gate; an AFTER that doesn't beat its BEFORE is rolled back. The rule is encoded per harness (GATE FAIL non-zero exit in the Rust examples, threshold expect() in vitest). F1 — Grouped listings (trash / recent / favorites / shared-with-me) re-bucketed the WHOLE accumulated list on every infinite-scroll page. ResourceSectionsBuilder (new, off the reactive graph) re-buckets only the fresh page and hands VirtualList the same rows array reference for untouched buckets. 50×50 (2 500-item) drain: 63 750 → 2 500 bucketOf calls (25.5×), 12.5 → 1.3 ms wall (9.9×); O(N²/page) → O(N). Deep-equal to the full-rebuild reference at every page for both a contiguous (date) and a non-contiguous (trash-by-drive) group-by; reference-stability + fallback gated. B1 — exif Make/Model: the display String was thrown away to allocate the trimmed copy; display_value_trimmed trims in place (drain + truncate), 2 → 1 alloc per field (8 → 4 allocs/op, 1.26×). B2 — content-index worker: text_extractor::supports (lowercases MIME + extension) was called twice per file per drain batch; classify once into a Vec<bool> and thread it through both uses. 256-file batch: 704 → 353 allocs, 34.5 → 16.7 µs (2.07×). B3 — tantivy: skip SnippetGenerator::create on a zero-hit content search (return Ok(vec![]) once top_docs.is_empty()); the per-hit loop was empty. 400-doc index: 1 575.6 → 1 237.2 ns (1.27×), widens with index size. Harnesses: examples/bench_round15_micro.rs, examples/bench_round15_tantivy.rs, frontend resourceSections.bench.test.ts; writeup in benches/ROUND15.md. Also normalizes two round14 bench examples that were committed unformatted (cargo fmt --all). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012o47jSrtL7xuNGTHXmtiYL
This commit is contained in:
+18
@@ -354,6 +354,24 @@ name = "bench_micro_allocs"
|
||||
path = "examples/bench_micro_allocs.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-15 battery ────────────────────────────────────────────────────────────
|
||||
|
||||
# Round-15 CPU/alloc micro-pack — exif Make/Model in-place trim (drop the
|
||||
# throwaway display String), content-index worker single supports() classify
|
||||
# per file (was called twice per drain batch). No Postgres.
|
||||
[[example]]
|
||||
name = "bench_round15_micro"
|
||||
path = "examples/bench_round15_micro.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-15 tantivy zero-hit snippet skip — the SnippetGenerator::create the
|
||||
# search path builds even when the query matched no documents (pure waste on a
|
||||
# no-hit content search). Builds a RAM index; no Postgres.
|
||||
[[example]]
|
||||
name = "bench_round15_tantivy"
|
||||
path = "examples/bench_round15_tantivy.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-14 battery ────────────────────────────────────────────────────────────
|
||||
|
||||
# Round-14 query-shape pack — lightbox face-box narrow projection (drop the
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# Round 15 — grouped-listing O(N²) rebucket, exif/reseed allocations, tantivy zero-hit snippet skip
|
||||
|
||||
Benchmark-gated, same rule as ROUND2–14: every change ships with a
|
||||
BEFORE/AFTER benchmark and an equivalence/safety gate; an AFTER that doesn't
|
||||
beat its BEFORE is rolled back (never applied). The roll-back rule is encoded
|
||||
directly into each harness as a `GATE FAIL … rollback` non-zero exit (Rust) or
|
||||
a threshold `expect()` (frontend), so a regression fails CI rather than
|
||||
shipping.
|
||||
|
||||
This round lands the ROUND14-deferred **flagship** — the grouped-listing
|
||||
`sections` rebuild that was the last O(N²)-per-page accumulation left in the
|
||||
SvelteKit listing surfaces — plus three backend items pulled from the same
|
||||
deferred list: two allocation cuts on the photo-ingest / content-reseed worker
|
||||
paths, and a wasted `SnippetGenerator` build removed from the zero-hit content
|
||||
search path.
|
||||
|
||||
Measured on 4 cores / 15 GiB, local PostgreSQL not needed for any Round-15 arm
|
||||
(all no-Postgres: release profile for the Rust examples; Node 22 / vitest for
|
||||
the frontend). Reproduce any row with the command in its section.
|
||||
|
||||
## Summary
|
||||
|
||||
| # | change | key metric | before → after |
|
||||
|--:|---|---|---|
|
||||
| F1 | Grouped listings (trash / recent / favorites / shared-with-me) re-bucketed the WHOLE accumulated list on every infinite-scroll page; `ResourceSectionsBuilder` re-buckets only the fresh page and reuses each untouched bucket's array reference | 50×50 (2 500-item) drain, month buckets | **63 750 → 2 500 `bucketOf` calls (25.5×)** · **12.5 → 1.3 ms wall (9.9×)** · O(N²/page) → O(N) |
|
||||
| B1 | exif `Make`/`Model` — `display_value().to_string().trim_matches('"').trim().to_string()` throws the display `String` away to allocate the trimmed copy; the in-place `drain`+`truncate` helper keeps one allocation | 4 sample values/op | **150.4 → 119.4 ns (1.26×)** · **8 → 4 allocs/op** (2 → 1 per field) |
|
||||
| B2 | Content-index worker called `text_extractor::supports` (lowercases MIME + extension) TWICE per file per drain batch; classify once into a `Vec<bool>` and thread it through both uses | 256-file reseed batch | **34.5 → 16.7 µs (2.07×)** · **704 → 353 allocs/op** |
|
||||
| B3 | Zero-hit content search still built a `SnippetGenerator` (query-compile + term weighting) though the per-hit loop was empty; return `Ok(vec![])` as soon as `top_docs.is_empty()` | no-hit query, 400-doc index | **1 575.6 → 1 237.2 ns (1.27×)** · **21 → 19 allocs/op** (widens with index size) |
|
||||
|
||||
## [F1] Grouped listings — incremental swimlane builder
|
||||
|
||||
```
|
||||
cd frontend && npx vitest run src/lib/utils/resourceSections.bench.test.ts
|
||||
```
|
||||
|
||||
Every grouped listing page (`/trash`, `/recent`, `/favorites`,
|
||||
`/shared-with-me`) loads its rows via infinite scroll (`raw = [...raw,
|
||||
...page]`), and `ResourceList`'s `sections` `$derived.by` re-bucketed the
|
||||
**whole accumulated list** on every page: Σ ≈ O(N²/page) `bucketOf` + `ctxOf`
|
||||
calls across a drain, and a brand-new rows array for *every* bucket each page
|
||||
(so `VirtualList` re-diffed every swimlane every page). This was the ROUND14
|
||||
"flagship follow-up" — the same O(N²)-per-page class ROUND6 fixed for the files
|
||||
listing, ROUND14 §F2 for favorites' `favoriteIds`, and `PhotoTimeline` for the
|
||||
photos grid.
|
||||
|
||||
`ResourceSectionsBuilder` (extracted to `$lib/utils/resourceSections`, off the
|
||||
Svelte reactive graph so it's unit/benchmark-testable) exploits the append
|
||||
invariant: a grouped listing is server-sorted by the active group's `orderBy`,
|
||||
so a fresh page only ever extends existing buckets or appends new ones. It
|
||||
detects the append (prefix-identity on the boundary object), re-buckets only
|
||||
the fresh page, and hands back the **same array reference** for every untouched
|
||||
bucket — the property `VirtualList` (which diffs its `items` prop by reference)
|
||||
relies on to skip re-rendering it — while emitting a fresh array only for
|
||||
buckets the page actually grew. Any non-append (group-by switch, deletion,
|
||||
dotfile-filter toggle) falls back to a full rebuild, so the output is always
|
||||
deep-equal to the pure `buildResourceSections` reference.
|
||||
|
||||
Correctness does **not** depend on bucket contiguity: the one non-monotonic
|
||||
group-by in the set — trash grouped **by drive** but ordered by name, so a page
|
||||
sprays items across every already-emitted drive bucket — stays byte-for-byte
|
||||
equal to the full rebuild (it just refreshes more buckets per page). Header
|
||||
labels are recomputed every sync (never cached): a group-by's `labelOf` can
|
||||
resolve asynchronously (owner / sharer names arrive after the rows), and a
|
||||
cached label would freeze the header at its fallback.
|
||||
|
||||
50×50 (2 500-item) month-bucketed drain: **63 750 → 2 500 `bucketOf` calls
|
||||
(25.5× fewer), 12.5 → 1.3 ms wall (9.9×)**. Gates: (1) equivalence — the
|
||||
incremental output is deep-equal to `buildResourceSections` at *every* page for
|
||||
both a contiguous (date) and a non-contiguous (drive) group-by; (2) reference
|
||||
stability — untouched buckets keep their exact array reference across an append
|
||||
while a grown bucket gets a fresh one; (3) correct fallback on group-by switch,
|
||||
deletion and the flat pass-through; (4) perf — `bucketOf` work is exactly O(N)
|
||||
across the drain and wall drops ≥3×.
|
||||
|
||||
## [B1]–[B2] exif / content-reseed allocation cuts
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_round15_micro
|
||||
```
|
||||
|
||||
Counting-allocator micro-bench; each section is BEFORE (the shipped-before
|
||||
shape) vs AFTER (the shipped function / shape) with a byte-identity gate.
|
||||
|
||||
- **[B1] exif `Make`/`Model` single-allocation trim.** `ExifService::extract`
|
||||
read the camera make + model as
|
||||
`field.display_value().to_string().trim_matches('"').trim().to_string()` —
|
||||
the first `to_string()` materializes the display value (unavoidable), then
|
||||
`.trim_matches('"').trim().to_string()` allocates a **second** `String` for
|
||||
the trimmed copy and drops the first. The new `display_value_trimmed` applies
|
||||
the same two-stage trim in place on the already-owned buffer (`drain` drops
|
||||
the stripped prefix, `truncate` the suffix — both reuse the allocation), so a
|
||||
quoted `"Canon"` costs one allocation instead of two. Per ingested photo (the
|
||||
Make + Model fields). 4 sample values/op: **8 → 4 allocs/op (2 → 1 per
|
||||
field), 1.26× wall**. Gate: byte-identical to the old chain across quoted /
|
||||
padded / clean shapes.
|
||||
- **[B2] Content-index worker single `supports()` classify.** `supports`
|
||||
lowercases the MIME (and, on a generic MIME, the extension) — 1–2 allocations
|
||||
— and the drain loop called it **twice per file**: once in the
|
||||
`wanted_hashes` filter, once again in the per-file records loop. The worker
|
||||
now classifies each file once into a `Vec<bool>` and threads the flag through
|
||||
both. On a full reseed that is one redundant classify (and its allocations)
|
||||
removed for *every file in the library*. 256-file batch: **704 → 353
|
||||
allocs/op, 34.5 → 16.7 µs (2.07×)**. Gate: the `(wanted, supported)` tallies
|
||||
are identical before/after.
|
||||
|
||||
## [B3] Tantivy zero-hit snippet skip
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_round15_tantivy
|
||||
```
|
||||
|
||||
`TantivyContentIndex::search_blocking` built the `SnippetGenerator` from the
|
||||
query right after the `TopDocs` search — but `SnippetGenerator::create`
|
||||
compiles the query against the index (collects query terms, looks up each
|
||||
term's document frequency, builds the weighting), and when the query matched
|
||||
**no** documents that generator is never used: the per-hit loop is empty. A
|
||||
content search for a term that isn't in any indexed document (a common miss)
|
||||
paid that build for nothing on the request path.
|
||||
|
||||
The fix returns `Ok(Vec::new())` the moment `top_docs.is_empty()`, before the
|
||||
create. The bench reproduces the exact skipped operation on a RAM index built
|
||||
with the public tantivy API (same crate + version): BEFORE = search + create,
|
||||
AFTER = search + the `is_empty()` early return; the delta is the wasted create.
|
||||
No-hit query against a 400-document index: **1 575.6 → 1 237.2 ns (1.27×), 21 →
|
||||
19 allocs/op** — the create adds ~338 ns + 2 allocs on top of the search on
|
||||
*every* zero-hit content query, and its per-term `doc_freq` lookups grow with
|
||||
the index (the RAM bench's 400 docs understate the production term dictionary).
|
||||
Gates: the miss query genuinely returns zero hits, and a control arm confirms a
|
||||
term that *does* hit still yields a snippet fragment (the skip only ever
|
||||
triggers on a true zero-hit query).
|
||||
|
||||
## Not shipped — carried forward from the ROUND14 deferred list
|
||||
|
||||
Still queued, unchanged in scope (each wants its own decision, Postgres
|
||||
fixture, or bigger refactor):
|
||||
|
||||
- **Query-shape (needs Postgres):** `music_storage_adapter::list_public_playlists`
|
||||
1 + N `COUNT(*)` fold (opt-in public-gallery path); contact REST listings
|
||||
(`search_contacts`, `get_contacts_by_address_book_paginated`,
|
||||
`get_contacts_in_group`) over-fetch the multi-KB `vcard` TEXT though every
|
||||
caller maps to a `ContactDto` with no `vcard` field (wants a *lite* row
|
||||
mapper, since the non-paginated sibling is shared with the CardDAV stream).
|
||||
- **Frontend:** `shared/+page.svelte` rebuilds the full `lanes` tree per page
|
||||
and on every grant edit; the same incremental-builder pattern F1 uses is the
|
||||
follow-up. (F1 removed the `ResourceList.sections` half of the ROUND14
|
||||
"flagship" bullet; the `lanes` half remains.)
|
||||
- **CPU/alloc (background):** REST calendar-event edit re-`format!`s the whole
|
||||
`ical_data` body once per changed property; `dedup_service` hash-`String`
|
||||
re-allocations; `exif_service` still double-allocates the GPS-ref display in
|
||||
`parse_gps_coord` (single-alloc, low-frequency — folded into B1's helper is
|
||||
possible but the ref is compared to `"S"`/`"W"` as a borrow, so it never
|
||||
needed the second alloc the Make/Model path did).
|
||||
- **Storage I/O (cached-remote class):** `CachedBlobBackend` per-write
|
||||
`create_dir_all` + inline eviction `remove_file` on the reactor thread;
|
||||
`encrypted_blob_backend` 64 KiB vs 256 KiB plaintext frames.
|
||||
|
||||
## Environment / methodology
|
||||
|
||||
- `cargo run --release --features bench --example bench_round15_micro`
|
||||
— counting allocator, no Postgres (`BENCH_ITERS`, `BENCH_BATCH`).
|
||||
- `cargo run --release --features bench --example bench_round15_tantivy`
|
||||
— builds a RAM tantivy index, no Postgres (`BENCH_ITERS`, `BENCH_DOCS`).
|
||||
- `cd frontend && npx vitest run src/lib/utils/resourceSections.bench.test.ts`.
|
||||
- Roll-back rule encoded per harness: the Rust examples `std::process::exit(1)`
|
||||
with `GATE FAIL … rollback` if an AFTER arm fails to beat its BEFORE; the
|
||||
vitest gate `expect()`s the O(N) call count and the ≥3× wall.
|
||||
@@ -64,7 +64,10 @@ unsafe impl GlobalAlloc for CountingAlloc {
|
||||
static GLOBAL: CountingAlloc = CountingAlloc;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Measured {
|
||||
@@ -80,11 +83,17 @@ fn measure<F: FnMut()>(iters: usize, mut f: F) -> Measured {
|
||||
}
|
||||
let wall = t.elapsed().as_nanos() as f64 / iters as f64;
|
||||
let allocs = (ALLOC_CALLS.load(Ordering::Relaxed) - a0) as f64 / iters as f64;
|
||||
Measured { wall_ns_per_op: wall, allocs_per_op: allocs }
|
||||
Measured {
|
||||
wall_ns_per_op: wall,
|
||||
allocs_per_op: allocs,
|
||||
}
|
||||
}
|
||||
|
||||
fn print_row(label: &str, m: &Measured) {
|
||||
println!("| {:<40} | {:>12.1} | {:>10.2} |", label, m.wall_ns_per_op, m.allocs_per_op);
|
||||
println!(
|
||||
"| {:<40} | {:>12.1} | {:>10.2} |",
|
||||
label, m.wall_ns_per_op, m.allocs_per_op
|
||||
);
|
||||
}
|
||||
|
||||
fn header_footer(name: &str, before: &Measured, after: &Measured) {
|
||||
@@ -107,11 +116,14 @@ fn section_cookie() {
|
||||
|
||||
let iters: usize = env_or("BENCH_ITERS", 200_000);
|
||||
let name = "oxicloud_access";
|
||||
let jwt = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIwMTIzNDU2Nzg5YWJjZGVmIn0.c2lnbmF0dXJlLXBsYWNlaG9sZGVy";
|
||||
let jwt =
|
||||
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIwMTIzNDU2Nzg5YWJjZGVmIn0.c2lnbmF0dXJlLXBsYWNlaG9sZGVy";
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(
|
||||
header::COOKIE,
|
||||
HeaderValue::from_str(&format!("{name}={jwt}; oxicloud_csrf=3f2504e0-4f89-41d3-9a0c-0305e82c3301"))
|
||||
HeaderValue::from_str(&format!(
|
||||
"{name}={jwt}; oxicloud_csrf=3f2504e0-4f89-41d3-9a0c-0305e82c3301"
|
||||
))
|
||||
.unwrap(),
|
||||
);
|
||||
|
||||
@@ -221,7 +233,10 @@ fn section_relevance() {
|
||||
"relevance differs for ({name:?}, {q:?})"
|
||||
);
|
||||
}
|
||||
println!("# [A2] gate: ASCII fast path matches Unicode lowercase across {} cases — OK", corpus.len());
|
||||
println!(
|
||||
"# [A2] gate: ASCII fast path matches Unicode lowercase across {} cases — OK",
|
||||
corpus.len()
|
||||
);
|
||||
|
||||
let m_before = measure(iters, || {
|
||||
for (name, q) in corpus {
|
||||
@@ -234,7 +249,10 @@ fn section_relevance() {
|
||||
}
|
||||
});
|
||||
|
||||
println!("\n## [A2] compute_relevance over a {}-row result page (per search / keystroke)", corpus.len());
|
||||
println!(
|
||||
"\n## [A2] compute_relevance over a {}-row result page (per search / keystroke)",
|
||||
corpus.len()
|
||||
);
|
||||
header_footer("relevance whole corpus", &m_before, &m_after);
|
||||
if m_after.wall_ns_per_op >= m_before.wall_ns_per_op {
|
||||
eprintln!("GATE FAIL [A2]: ASCII fast path not faster — rollback");
|
||||
@@ -252,7 +270,11 @@ fn section_sub_parse() {
|
||||
let pre_parsed = Uuid::parse_str(&sub).unwrap();
|
||||
|
||||
// Gate: the pre-parsed uuid equals a fresh parse.
|
||||
assert_eq!(Uuid::parse_str(&sub).unwrap(), pre_parsed, "uuid parse differs");
|
||||
assert_eq!(
|
||||
Uuid::parse_str(&sub).unwrap(),
|
||||
pre_parsed,
|
||||
"uuid parse differs"
|
||||
);
|
||||
println!("# [A3] gate: pre-parsed sub_id equals per-request parse — OK");
|
||||
|
||||
let m_before = measure(iters, || {
|
||||
@@ -279,18 +301,41 @@ fn section_sub_parse() {
|
||||
fn build_request_headers() -> HeaderMap {
|
||||
// A representative authed browser request.
|
||||
let mut h = HeaderMap::new();
|
||||
h.insert(header::AUTHORIZATION, HeaderValue::from_static("Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig"));
|
||||
h.insert(
|
||||
header::AUTHORIZATION,
|
||||
HeaderValue::from_static("Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig"),
|
||||
);
|
||||
h.insert(
|
||||
header::COOKIE,
|
||||
HeaderValue::from_static("oxicloud_access=eyJ.payload.sig; oxicloud_csrf=3f2504e0-4f89-41d3-9a0c-0305e82c3301"),
|
||||
HeaderValue::from_static(
|
||||
"oxicloud_access=eyJ.payload.sig; oxicloud_csrf=3f2504e0-4f89-41d3-9a0c-0305e82c3301",
|
||||
),
|
||||
);
|
||||
h.insert(header::HOST, HeaderValue::from_static("cloud.example.com"));
|
||||
h.insert(header::USER_AGENT, HeaderValue::from_static("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"));
|
||||
h.insert(header::ACCEPT, HeaderValue::from_static("application/json, text/plain, */*"));
|
||||
h.insert(header::ACCEPT_ENCODING, HeaderValue::from_static("gzip, deflate, br"));
|
||||
h.insert(header::ACCEPT_LANGUAGE, HeaderValue::from_static("en-US,en;q=0.9"));
|
||||
h.insert(header::REFERER, HeaderValue::from_static("https://cloud.example.com/files"));
|
||||
h.insert("x-csrf-token", HeaderValue::from_static("3f2504e0-4f89-41d3-9a0c-0305e82c3301"));
|
||||
h.insert(
|
||||
header::USER_AGENT,
|
||||
HeaderValue::from_static("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"),
|
||||
);
|
||||
h.insert(
|
||||
header::ACCEPT,
|
||||
HeaderValue::from_static("application/json, text/plain, */*"),
|
||||
);
|
||||
h.insert(
|
||||
header::ACCEPT_ENCODING,
|
||||
HeaderValue::from_static("gzip, deflate, br"),
|
||||
);
|
||||
h.insert(
|
||||
header::ACCEPT_LANGUAGE,
|
||||
HeaderValue::from_static("en-US,en;q=0.9"),
|
||||
);
|
||||
h.insert(
|
||||
header::REFERER,
|
||||
HeaderValue::from_static("https://cloud.example.com/files"),
|
||||
);
|
||||
h.insert(
|
||||
"x-csrf-token",
|
||||
HeaderValue::from_static("3f2504e0-4f89-41d3-9a0c-0305e82c3301"),
|
||||
);
|
||||
h.insert(header::CONNECTION, HeaderValue::from_static("keep-alive"));
|
||||
h
|
||||
}
|
||||
@@ -302,9 +347,13 @@ fn section_headermap_clone() {
|
||||
// Gate: the token extracted from a cloned map equals that from the borrowed map.
|
||||
let from_clone = {
|
||||
let c = headers.clone();
|
||||
c.get(header::AUTHORIZATION).and_then(|v| v.to_str().ok()).map(str::to_string)
|
||||
c.get(header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.map(str::to_string)
|
||||
};
|
||||
let from_borrow = headers.get(header::AUTHORIZATION).and_then(|v| v.to_str().ok());
|
||||
let from_borrow = headers
|
||||
.get(header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok());
|
||||
assert_eq!(from_clone.as_deref(), from_borrow, "authorization differs");
|
||||
println!("# [A4] gate: token from cloned map == token from borrowed map — OK");
|
||||
|
||||
@@ -358,7 +407,10 @@ fn section_caldav_rfc2822() {
|
||||
}
|
||||
println!("# [A5] gate: stack rfc2822_utc byte-identical to chrono to_rfc2822 — OK");
|
||||
|
||||
let dts: Vec<DateTime<Utc>> = secs.iter().map(|&s| DateTime::<Utc>::from_timestamp(s, 0).unwrap()).collect();
|
||||
let dts: Vec<DateTime<Utc>> = secs
|
||||
.iter()
|
||||
.map(|&s| DateTime::<Utc>::from_timestamp(s, 0).unwrap())
|
||||
.collect();
|
||||
|
||||
let m_before = measure(iters, || {
|
||||
for dt in &dts {
|
||||
@@ -372,7 +424,10 @@ fn section_caldav_rfc2822() {
|
||||
}
|
||||
});
|
||||
|
||||
println!("\n## [A5] CalDAV getlastmodified render ({} events, per REPORT/PROPFIND)", secs.len());
|
||||
println!(
|
||||
"\n## [A5] CalDAV getlastmodified render ({} events, per REPORT/PROPFIND)",
|
||||
secs.len()
|
||||
);
|
||||
header_footer("rfc2822 chrono/stack", &m_before, &m_after);
|
||||
if m_after.allocs_per_op >= m_before.allocs_per_op {
|
||||
eprintln!("GATE FAIL [A5]: stack render did not remove allocations — rollback");
|
||||
@@ -389,7 +444,12 @@ fn section_caldav_href_etag() {
|
||||
let base_href = "/caldav/alice/personal/";
|
||||
// A page of events (uid, id) like write_report_page iterates.
|
||||
let events: Vec<(String, Uuid)> = (0..40)
|
||||
.map(|i| (format!("event-uid-{i:04}-abcdef@oxicloud"), Uuid::from_u128(0x1000 + i as u128)))
|
||||
.map(|i| {
|
||||
(
|
||||
format!("event-uid-{i:04}-abcdef@oxicloud"),
|
||||
Uuid::from_u128(0x1000 + i as u128),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Gate: reused-buffer output identical to the per-event format! pair.
|
||||
@@ -426,7 +486,10 @@ fn section_caldav_href_etag() {
|
||||
}
|
||||
});
|
||||
|
||||
println!("\n## [A6] CalDAV per-event href + etag ({} events/page, per REPORT/PROPFIND)", events.len());
|
||||
println!(
|
||||
"\n## [A6] CalDAV per-event href + etag ({} events/page, per REPORT/PROPFIND)",
|
||||
events.len()
|
||||
);
|
||||
header_footer("href+etag per page", &m_before, &m_after);
|
||||
if m_after.allocs_per_op >= m_before.allocs_per_op {
|
||||
eprintln!("GATE FAIL [A6]: reused buffer did not reduce allocations — rollback");
|
||||
|
||||
@@ -57,7 +57,11 @@ fn bytes_to_embedding(b: &[u8]) -> Vec<f32> {
|
||||
/// hydrate the full 10-column row (decoding the 2 KiB embedding like
|
||||
/// `row_to_face`), then filter `user_id == caller` in Rust and keep only
|
||||
/// `(id, person_id, bbox)`.
|
||||
async fn boxes_before(pool: &PgPool, file_id: Uuid, caller: Uuid) -> Vec<(Uuid, Option<Uuid>, Vec<f32>)> {
|
||||
async fn boxes_before(
|
||||
pool: &PgPool,
|
||||
file_id: Uuid,
|
||||
caller: Uuid,
|
||||
) -> Vec<(Uuid, Option<Uuid>, Vec<f32>)> {
|
||||
let rows = sqlx::query(
|
||||
"SELECT id, file_id, user_id, person_id, bbox, det_score, quality, embedding, blob_hash, created_at
|
||||
FROM faces.faces WHERE file_id = $1",
|
||||
@@ -83,7 +87,11 @@ async fn boxes_before(pool: &PgPool, file_id: Uuid, caller: Uuid) -> Vec<(Uuid,
|
||||
}
|
||||
|
||||
/// AFTER: narrow projection, caller filter in SQL.
|
||||
async fn boxes_after(pool: &PgPool, file_id: Uuid, caller: Uuid) -> Vec<(Uuid, Option<Uuid>, Vec<f32>)> {
|
||||
async fn boxes_after(
|
||||
pool: &PgPool,
|
||||
file_id: Uuid,
|
||||
caller: Uuid,
|
||||
) -> Vec<(Uuid, Option<Uuid>, Vec<f32>)> {
|
||||
let rows = sqlx::query(
|
||||
"SELECT id, person_id, bbox FROM faces.faces WHERE file_id = $1 AND user_id = $2",
|
||||
)
|
||||
@@ -200,8 +208,12 @@ async fn section_face_boxes(pool: &PgPool) {
|
||||
let wire_after = n * (16 + 16 + 16 + 8);
|
||||
println!("\n## [Q1] Lightbox face boxes — group photo, {n} faces");
|
||||
println!("| arm | mean ms | p50 ms | p95 ms | ~bytes/req |");
|
||||
println!("| BEFORE wide row (incl. embedding) | {wm:>7.3} | {wp50:>6.3} | {wp95:>6.3} | {wire_before:>9} |");
|
||||
println!("| AFTER narrow (id,person,bbox) | {nm:>7.3} | {np50:>6.3} | {np95:>6.3} | {wire_after:>9} |");
|
||||
println!(
|
||||
"| BEFORE wide row (incl. embedding) | {wm:>7.3} | {wp50:>6.3} | {wp95:>6.3} | {wire_before:>9} |"
|
||||
);
|
||||
println!(
|
||||
"| AFTER narrow (id,person,bbox) | {nm:>7.3} | {np50:>6.3} | {np95:>6.3} | {wire_after:>9} |"
|
||||
);
|
||||
println!(
|
||||
"# {:.2}x faster; ~{} KiB embedding/columns off the wire per lightbox open (scales with face count)",
|
||||
wm / nm,
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
//! Round-15 CPU/alloc micro-pack (no Postgres).
|
||||
//!
|
||||
//! Each section is BEFORE (verbatim replica of the shipped-before shape) vs
|
||||
//! AFTER (the shipped-after shape, or the shipped function itself), with an
|
||||
//! equivalence gate and a `GATE FAIL … rollback` check that exits non-zero if
|
||||
//! the AFTER arm fails to beat its BEFORE — the round's roll-back rule encoded
|
||||
//! into the benchmark.
|
||||
//!
|
||||
//! [B1] exif Make/Model — `display_value().to_string().trim_matches('"')
|
||||
//! .trim().to_string()` allocates the display String, then throws it away
|
||||
//! to allocate the trimmed copy (2 allocs). The shipped
|
||||
//! `exif_service::display_value_trimmed` trims in place on the owned
|
||||
//! buffer (`drain` + `truncate`) — 1 alloc. Per ingested photo.
|
||||
//! [B2] content-index worker `supports()` — `text_extractor::supports`
|
||||
//! (lowercases the MIME + extension, 1–2 allocs) was called TWICE per
|
||||
//! file per drain batch: once in the wanted-hashes filter, once in the
|
||||
//! records loop. The shipped code classifies each file once into a
|
||||
//! `Vec<bool>` and threads it through both. Per reseed batch (every
|
||||
//! file in the library).
|
||||
//!
|
||||
//! Run:
|
||||
//! cargo run --release --features bench --example bench_round15_micro
|
||||
//! Tunables (env): BENCH_ITERS (200000), BENCH_BATCH (256)
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::env;
|
||||
use std::hint::black_box;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use oxicloud::infrastructure::services::search_index::text_extractor;
|
||||
|
||||
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
struct CountingAlloc;
|
||||
|
||||
unsafe impl GlobalAlloc for CountingAlloc {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc(layout) }
|
||||
}
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) }
|
||||
}
|
||||
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.realloc(ptr, layout, new_size) }
|
||||
}
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc_zeroed(layout) }
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static GLOBAL: CountingAlloc = CountingAlloc;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Measured {
|
||||
wall_ns_per_op: f64,
|
||||
allocs_per_op: f64,
|
||||
}
|
||||
|
||||
fn measure<F: FnMut()>(iters: usize, mut f: F) -> Measured {
|
||||
let a0 = ALLOC_CALLS.load(Ordering::Relaxed);
|
||||
let t = Instant::now();
|
||||
for _ in 0..iters {
|
||||
f();
|
||||
}
|
||||
let wall = t.elapsed().as_nanos() as f64 / iters as f64;
|
||||
let allocs = (ALLOC_CALLS.load(Ordering::Relaxed) - a0) as f64 / iters as f64;
|
||||
Measured {
|
||||
wall_ns_per_op: wall,
|
||||
allocs_per_op: allocs,
|
||||
}
|
||||
}
|
||||
|
||||
fn print_row(label: &str, m: &Measured) {
|
||||
println!(
|
||||
"| {:<42} | {:>12.1} | {:>10.2} |",
|
||||
label, m.wall_ns_per_op, m.allocs_per_op
|
||||
);
|
||||
}
|
||||
|
||||
fn header_footer(name: &str, before: &Measured, after: &Measured) {
|
||||
println!("| arm | ns/op | allocs/op |");
|
||||
print_row(&format!("BEFORE {name}"), before);
|
||||
print_row(&format!("AFTER {name}"), after);
|
||||
println!(
|
||||
"# {:.2}x wall, {:.2} fewer allocs/op",
|
||||
before.wall_ns_per_op / after.wall_ns_per_op,
|
||||
before.allocs_per_op - after.allocs_per_op
|
||||
);
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// [B1] exif Make/Model trim — 2 allocs (throwaway display String) vs 1 (in place)
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/// BEFORE: the shipped-before chain. `raw` stands in for the field's rendered
|
||||
/// display value; `to_string()` mirrors `display_value().to_string()` (the one
|
||||
/// unavoidable alloc), then `.trim_matches('"').trim().to_string()` allocates a
|
||||
/// second time for the trimmed copy.
|
||||
fn trim_before(raw: &str) -> String {
|
||||
raw.to_string().trim_matches('"').trim().to_string()
|
||||
}
|
||||
|
||||
/// AFTER: verbatim replica of `exif_service::display_value_trimmed` — trims in
|
||||
/// place on the already-owned buffer, so only the display String is allocated.
|
||||
fn trim_after(raw: &str) -> String {
|
||||
let mut s = raw.to_string();
|
||||
let trimmed = s.trim_matches('"').trim();
|
||||
let start = trimmed.as_ptr().addr() - s.as_ptr().addr();
|
||||
let len = trimmed.len();
|
||||
s.drain(..start);
|
||||
s.truncate(len);
|
||||
s
|
||||
}
|
||||
|
||||
fn section_exif_trim() {
|
||||
let iters: usize = env_or("BENCH_ITERS", 200_000);
|
||||
// Representative EXIF Make/Model display values: the widely-seen quoted
|
||||
// form, plus a padded one and an already-clean one.
|
||||
let samples = ["\"Canon\"", "\"NIKON CORPORATION\"", " Apple ", "SONY"];
|
||||
|
||||
// Gate: byte-identical output to the old chain across every shape.
|
||||
for s in samples {
|
||||
assert_eq!(trim_before(s), trim_after(s), "trim differs for {s:?}");
|
||||
}
|
||||
|
||||
let before = measure(iters, || {
|
||||
for s in samples {
|
||||
black_box(trim_before(black_box(s)));
|
||||
}
|
||||
});
|
||||
let after = measure(iters, || {
|
||||
for s in samples {
|
||||
black_box(trim_after(black_box(s)));
|
||||
}
|
||||
});
|
||||
|
||||
println!("\n## [B1] exif Make/Model trim (4 sample values/op)");
|
||||
header_footer("exif trim", &before, &after);
|
||||
if after.allocs_per_op >= before.allocs_per_op {
|
||||
eprintln!("GATE FAIL [B1]: in-place trim did not reduce allocations — rollback");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// [B2] content-index worker supports() — 2× per file vs 1× (memoized)
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/// One drained file row: (name, mime, size). Mirrors the worker's
|
||||
/// `FileIndexRow` projection (only the fields `supports` + the size gate read).
|
||||
struct FileRow {
|
||||
name: &'static str,
|
||||
mime: &'static str,
|
||||
size: i64,
|
||||
}
|
||||
|
||||
fn corpus(n: usize) -> Vec<FileRow> {
|
||||
// A realistic reseed mix: text/markdown/pdf/office (supported) interleaved
|
||||
// with images/video/binaries (unsupported — the fast reject).
|
||||
const MIX: &[(&str, &str, i64)] = &[
|
||||
("notes.txt", "text/plain", 4_000),
|
||||
("readme.md", "text/markdown", 8_000),
|
||||
("report.pdf", "application/pdf", 250_000),
|
||||
("photo.jpg", "image/jpeg", 3_000_000),
|
||||
(
|
||||
"sheet.xlsx",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
120_000,
|
||||
),
|
||||
("clip.mp4", "video/mp4", 40_000_000),
|
||||
("data.bin", "application/octet-stream", 1_000),
|
||||
("page.html", "text/html; charset=utf-8", 20_000),
|
||||
];
|
||||
(0..n)
|
||||
.map(|i| {
|
||||
let (name, mime, size) = MIX[i % MIX.len()];
|
||||
FileRow { name, mime, size }
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn section_supports() {
|
||||
let iters: usize = env_or("BENCH_ITERS", 200_000) / 20; // heavier op
|
||||
let batch: usize = env_or("BENCH_BATCH", 256);
|
||||
let max_bytes: u64 = 10 * 1024 * 1024;
|
||||
let files = corpus(batch);
|
||||
|
||||
// BEFORE: `supports` is evaluated in the wanted-hashes filter AND again per
|
||||
// file in the records loop — twice per file.
|
||||
let run_before = |files: &[FileRow]| -> (usize, usize) {
|
||||
let wanted = files
|
||||
.iter()
|
||||
.filter(|f| text_extractor::supports(f.name, f.mime) && f.size as u64 <= max_bytes)
|
||||
.count();
|
||||
let mut supported_files = 0;
|
||||
for f in files {
|
||||
if text_extractor::supports(f.name, f.mime) {
|
||||
supported_files += 1;
|
||||
}
|
||||
}
|
||||
(wanted, supported_files)
|
||||
};
|
||||
|
||||
// AFTER: classify each file once into a `Vec<bool>`; both the filter and the
|
||||
// records loop read the flag.
|
||||
let run_after = |files: &[FileRow]| -> (usize, usize) {
|
||||
let supported: Vec<bool> = files
|
||||
.iter()
|
||||
.map(|f| text_extractor::supports(f.name, f.mime))
|
||||
.collect();
|
||||
let wanted = files
|
||||
.iter()
|
||||
.zip(&supported)
|
||||
.filter(|&(f, s)| *s && f.size as u64 <= max_bytes)
|
||||
.count();
|
||||
let mut supported_files = 0;
|
||||
for (_, &s) in files.iter().zip(&supported) {
|
||||
if s {
|
||||
supported_files += 1;
|
||||
}
|
||||
}
|
||||
(wanted, supported_files)
|
||||
};
|
||||
|
||||
// Gate: identical (wanted, supported) tallies.
|
||||
assert_eq!(
|
||||
run_before(&files),
|
||||
run_after(&files),
|
||||
"supports tally differs"
|
||||
);
|
||||
|
||||
let before = measure(iters, || {
|
||||
black_box(run_before(black_box(&files)));
|
||||
});
|
||||
let after = measure(iters, || {
|
||||
black_box(run_after(black_box(&files)));
|
||||
});
|
||||
|
||||
println!("\n## [B2] content-index supports() ({batch} files/batch)");
|
||||
header_footer("supports/batch", &before, &after);
|
||||
if after.allocs_per_op >= before.allocs_per_op || after.wall_ns_per_op >= before.wall_ns_per_op
|
||||
{
|
||||
eprintln!("GATE FAIL [B2]: single-classify did not beat the double call — rollback");
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("#################################################################");
|
||||
println!("# Round-15 CPU/alloc micro-pack");
|
||||
println!("#################################################################");
|
||||
|
||||
section_exif_trim();
|
||||
section_supports();
|
||||
|
||||
println!("\nGATE PASS (all sections)");
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
//! Round-15 tantivy zero-hit snippet skip (no Postgres).
|
||||
//!
|
||||
//! `TantivyContentIndex::search_blocking` builds a `SnippetGenerator` from the
|
||||
//! query right after the `TopDocs` search — but a `SnippetGenerator::create`
|
||||
//! compiles the query against the index (term lookups + weight build), and when
|
||||
//! the query matched NO documents that generator is never used (the per-hit
|
||||
//! loop is empty). The shipped fix returns `Ok(Vec::new())` as soon as
|
||||
//! `top_docs.is_empty()`, before the create.
|
||||
//!
|
||||
//! This bench reproduces the exact skipped operation on a RAM index built with
|
||||
//! the public tantivy API (same crate + version the service uses):
|
||||
//! BEFORE = search (→ 0 hits) + `SnippetGenerator::create` (+ `set_max_num_chars`)
|
||||
//! AFTER = search (→ 0 hits) + `top_docs.is_empty()` early return
|
||||
//! The delta is the wasted create the fix removes from every no-hit content
|
||||
//! search. A sanity arm confirms a term that DOES hit still yields a snippet, so
|
||||
//! the skip only ever triggers on a genuine zero-hit query.
|
||||
//!
|
||||
//! Run:
|
||||
//! cargo run --release --features bench --example bench_round15_tantivy
|
||||
//! Tunables (env): BENCH_ITERS (50000), BENCH_DOCS (400)
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::env;
|
||||
use std::hint::black_box;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use tantivy::collector::TopDocs;
|
||||
use tantivy::query::QueryParser;
|
||||
use tantivy::schema::{STORED, STRING, Schema, TEXT, Value as _};
|
||||
use tantivy::snippet::SnippetGenerator;
|
||||
use tantivy::{Index, TantivyDocument, doc};
|
||||
|
||||
static ALLOC_CALLS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
struct CountingAlloc;
|
||||
|
||||
unsafe impl GlobalAlloc for CountingAlloc {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc(layout) }
|
||||
}
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) }
|
||||
}
|
||||
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.realloc(ptr, layout, new_size) }
|
||||
}
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);
|
||||
unsafe { System.alloc_zeroed(layout) }
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static GLOBAL: CountingAlloc = CountingAlloc;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
const SNIPPET_MAX_CHARS: usize = 200;
|
||||
|
||||
fn main() {
|
||||
let iters: usize = env_or("BENCH_ITERS", 50_000);
|
||||
let docs: usize = env_or("BENCH_DOCS", 400);
|
||||
|
||||
println!("#################################################################");
|
||||
println!("# Round-15 tantivy zero-hit snippet skip");
|
||||
println!("#################################################################");
|
||||
|
||||
// ── Build a RAM index: a stored content field + a name field, the shape
|
||||
// the service indexes. Fill it with realistic prose so create() has real
|
||||
// terms to weigh. ────────────────────────────────────────────────────
|
||||
let mut schema_builder = Schema::builder();
|
||||
let name = schema_builder.add_text_field("name", STRING | STORED);
|
||||
let content = schema_builder.add_text_field("content", TEXT | STORED);
|
||||
let schema = schema_builder.build();
|
||||
let index = Index::create_in_ram(schema);
|
||||
|
||||
const WORDS: &[&str] = &[
|
||||
"informe",
|
||||
"trimestral",
|
||||
"ventas",
|
||||
"region",
|
||||
"norte",
|
||||
"presupuesto",
|
||||
"reunion",
|
||||
"proyecto",
|
||||
"cliente",
|
||||
"factura",
|
||||
"contrato",
|
||||
"entrega",
|
||||
"calendario",
|
||||
"documento",
|
||||
"resumen",
|
||||
"analisis",
|
||||
"resultados",
|
||||
"equipo",
|
||||
];
|
||||
{
|
||||
let mut writer = index.writer(15_000_000).expect("writer");
|
||||
for i in 0..docs {
|
||||
let body: String = (0..40)
|
||||
.map(|j| WORDS[(i * 7 + j * 13) % WORDS.len()])
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ");
|
||||
writer
|
||||
.add_document(doc!(
|
||||
name => format!("doc-{i}.txt"),
|
||||
content => body,
|
||||
))
|
||||
.expect("add");
|
||||
}
|
||||
writer.commit().expect("commit");
|
||||
}
|
||||
let reader = index.reader().expect("reader");
|
||||
let searcher = reader.searcher();
|
||||
let parser = QueryParser::for_index(&index, vec![content]);
|
||||
|
||||
// A multi-term query of words that appear in NO document → zero hits, but
|
||||
// valid tokens (so the real code reaches the search, not the empty-token
|
||||
// guard). These are plausible-but-absent search terms.
|
||||
let miss_query = parser
|
||||
.parse_query("zzznonexistent quuxfoobar wibblewobble")
|
||||
.expect("parse");
|
||||
// A query that DOES hit — the sanity arm.
|
||||
let hit_query = parser.parse_query("informe ventas").expect("parse");
|
||||
|
||||
// ── Correctness gates ──────────────────────────────────────────────────
|
||||
let miss_hits = searcher
|
||||
.search(&miss_query, &TopDocs::with_limit(32).order_by_score())
|
||||
.expect("search");
|
||||
assert!(
|
||||
miss_hits.is_empty(),
|
||||
"miss query must return zero hits (got {})",
|
||||
miss_hits.len()
|
||||
);
|
||||
|
||||
let hit_hits = searcher
|
||||
.search(&hit_query, &TopDocs::with_limit(32).order_by_score())
|
||||
.expect("search");
|
||||
assert!(!hit_hits.is_empty(), "hit query must return hits");
|
||||
// The generator the fix keeps for real hits still produces a fragment.
|
||||
let generator = SnippetGenerator::create(&searcher, &*hit_query, content).expect("gen");
|
||||
let (_, addr) = hit_hits[0];
|
||||
let d: TantivyDocument = searcher.doc(addr).expect("doc");
|
||||
let preview = d
|
||||
.get_first(content)
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or_default()
|
||||
.to_owned();
|
||||
assert!(
|
||||
!generator.snippet(&preview).fragment().is_empty(),
|
||||
"a real hit must still yield a snippet fragment"
|
||||
);
|
||||
|
||||
// ── BEFORE: search + build the snippet generator even on zero hits. ──────
|
||||
let a0 = ALLOC_CALLS.load(Ordering::Relaxed);
|
||||
let t = Instant::now();
|
||||
for _ in 0..iters {
|
||||
let top = searcher
|
||||
.search(&miss_query, &TopDocs::with_limit(32).order_by_score())
|
||||
.expect("search");
|
||||
let sg = SnippetGenerator::create(&searcher, &*miss_query, content).map(|mut g| {
|
||||
g.set_max_num_chars(SNIPPET_MAX_CHARS);
|
||||
g
|
||||
});
|
||||
black_box((top.len(), sg.is_ok()));
|
||||
}
|
||||
let before_ns = t.elapsed().as_nanos() as f64 / iters as f64;
|
||||
let before_allocs = (ALLOC_CALLS.load(Ordering::Relaxed) - a0) as f64 / iters as f64;
|
||||
|
||||
// ── AFTER: search + the shipped early return on an empty result. ─────────
|
||||
let a1 = ALLOC_CALLS.load(Ordering::Relaxed);
|
||||
let t = Instant::now();
|
||||
for _ in 0..iters {
|
||||
let top = searcher
|
||||
.search(&miss_query, &TopDocs::with_limit(32).order_by_score())
|
||||
.expect("search");
|
||||
if top.is_empty() {
|
||||
black_box(top.len());
|
||||
continue;
|
||||
}
|
||||
// Unreached for the miss query; present so the arm is structurally the
|
||||
// shipped code, not a stripped one.
|
||||
let sg = SnippetGenerator::create(&searcher, &*miss_query, content).map(|mut g| {
|
||||
g.set_max_num_chars(SNIPPET_MAX_CHARS);
|
||||
g
|
||||
});
|
||||
black_box(sg.is_ok());
|
||||
}
|
||||
let after_ns = t.elapsed().as_nanos() as f64 / iters as f64;
|
||||
let after_allocs = (ALLOC_CALLS.load(Ordering::Relaxed) - a1) as f64 / iters as f64;
|
||||
|
||||
println!("\n## zero-hit content search ({docs} docs indexed)");
|
||||
println!("| arm | ns/op | allocs/op |");
|
||||
println!(
|
||||
"| {:<40} | {:>12.1} | {:>10.2} |",
|
||||
"BEFORE search + snippet create", before_ns, before_allocs
|
||||
);
|
||||
println!(
|
||||
"| {:<40} | {:>12.1} | {:>10.2} |",
|
||||
"AFTER search + is_empty skip", after_ns, after_allocs
|
||||
);
|
||||
println!(
|
||||
"# {:.2}x wall, {:.2} fewer allocs/op",
|
||||
before_ns / after_ns,
|
||||
before_allocs - after_allocs
|
||||
);
|
||||
|
||||
if after_ns >= before_ns || after_allocs >= before_allocs {
|
||||
eprintln!(
|
||||
"GATE FAIL [B3]: zero-hit skip did not beat building the snippet generator — rollback"
|
||||
);
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
println!("\nGATE PASS");
|
||||
}
|
||||
@@ -78,6 +78,7 @@
|
||||
import { formatBytes } from '$lib/utils/format';
|
||||
import { formatDate, iconNameFromClass, fileIconKindClass } from '$lib/utils/display';
|
||||
import { gridColumns } from '$lib/utils/grid';
|
||||
import { ResourceSectionsBuilder } from '$lib/utils/resourceSections';
|
||||
import { fileThumbnailUrl, thumbSizeForView } from '$lib/api/endpoints/files';
|
||||
import {
|
||||
canThumbnailClientSide,
|
||||
@@ -369,29 +370,24 @@
|
||||
/**
|
||||
* Partition the visible items into grouped sections when a `bucketOf` is
|
||||
* active. Server order is preserved within and across buckets (first-seen).
|
||||
*
|
||||
* `ResourceSectionsBuilder` re-buckets only the freshly-appended page rather
|
||||
* than the whole accumulated list, and hands `VirtualList` the same rows
|
||||
* array reference for every untouched bucket so it skips re-rendering it. An
|
||||
* infinite-scroll drain of a grouped listing (trash / recent / favorites /
|
||||
* shared-with-me) collapses from Σ O(N²/page) to O(N) bucketing work
|
||||
* (benches/ROUND15.md §F1). Held off the reactive graph — a plain
|
||||
* accumulator keyed by the append cursor, not $state; `sync` is idempotent,
|
||||
* so if the derive re-fires without an actual append it safely full-rebuilds
|
||||
* to the same output the pure `buildResourceSections` reference produces.
|
||||
*/
|
||||
const sections = $derived.by(
|
||||
(): Array<{ key: string; label: string; rows: Array<FileItem | FolderItem> }> => {
|
||||
const bucketOf = activeGroup?.bucketOf;
|
||||
if (!bucketOf) return [{ key: '', label: '', rows: visibleItems }];
|
||||
const order: string[] = [];
|
||||
// Transient bucketing map computed inside $derived.by — not reactive state.
|
||||
// eslint-disable-next-line svelte/prefer-svelte-reactivity
|
||||
const map = new Map<string, Array<FileItem | FolderItem>>();
|
||||
for (const item of visibleItems) {
|
||||
const k = bucketOf(item, ctxOf(item.id)) ?? '∅';
|
||||
if (!map.has(k)) {
|
||||
map.set(k, []);
|
||||
order.push(k);
|
||||
}
|
||||
map.get(k)!.push(item);
|
||||
}
|
||||
return order.map((k) => ({
|
||||
key: k,
|
||||
label: activeGroup?.labelOf?.(k) ?? k,
|
||||
rows: map.get(k)!
|
||||
}));
|
||||
}
|
||||
const sectionsBuilder = new ResourceSectionsBuilder<FileItem | FolderItem, ItemContext>();
|
||||
const sections = $derived.by(() =>
|
||||
sectionsBuilder.sync(visibleItems, {
|
||||
bucketOf: activeGroup?.bucketOf,
|
||||
labelOf: activeGroup?.labelOf,
|
||||
ctxOf: (item) => ctxOf(item.id)
|
||||
})
|
||||
);
|
||||
const grouped = $derived(!!activeGroup?.bucketOf);
|
||||
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
ResourceSectionsBuilder,
|
||||
buildResourceSections,
|
||||
type SectionGrouping
|
||||
} from './resourceSections';
|
||||
|
||||
/**
|
||||
* Benchmark gate for the incremental swimlane builder (ResourceSectionsBuilder)
|
||||
* that replaced ResourceList's `sections` `$derived.by`.
|
||||
*
|
||||
* Audit finding (ROUND14 deferred flagship): every grouped listing (trash,
|
||||
* recent, favorites, shared-with-me) pages in via `raw = [...raw, ...page]`,
|
||||
* and `sections` re-bucketed the WHOLE accumulated list on every page — Σ ≈
|
||||
* O(N²/page) `bucketOf` + `ctxOf` calls during an infinite-scroll drain, and a
|
||||
* brand-new rows array for EVERY bucket each page (so VirtualList re-diffed
|
||||
* every swimlane every page). The builder re-buckets only the fresh page and
|
||||
* hands back the same array reference for untouched buckets.
|
||||
*
|
||||
* Gates:
|
||||
* 1. Equivalence — at EVERY page of the drain, the incremental output is
|
||||
* deep-equal to the verbatim full-rebuild reference (buildResourceSections),
|
||||
* for a contiguous group-by (date, bucket aligned with order) AND a
|
||||
* non-contiguous one (trash-by-drive: name-ordered, drive-bucketed); plus
|
||||
* group-by switch, deletion and the flat pass-through fall back correctly.
|
||||
* 2. Reference stability — untouched buckets keep their exact array reference
|
||||
* across a page append (the property VirtualList relies on to skip them),
|
||||
* while a grown bucket gets a fresh one.
|
||||
* 3. Perf — bucketing work collapses from Σ O(N²/page) to O(N) across the
|
||||
* drain (deterministic `bucketOf`-call count) and wall drops ≥3x.
|
||||
*/
|
||||
|
||||
interface Item {
|
||||
id: string;
|
||||
name: string;
|
||||
driveId: string;
|
||||
/** ms epoch; descending with index (newest-first, like the server pages). */
|
||||
date: number;
|
||||
}
|
||||
|
||||
interface Ctx {
|
||||
date: number;
|
||||
driveId: string;
|
||||
}
|
||||
|
||||
const DAY = 86_400_000;
|
||||
|
||||
/** Item `i`: newest-first date, name in a fixed lexical order, round-robin drive. */
|
||||
function item(i: number): Item {
|
||||
return {
|
||||
id: `it-${i.toString().padStart(6, '0')}`,
|
||||
// Zero-padded so lexical name order is a stable, well-defined sequence.
|
||||
name: `file-${i.toString().padStart(6, '0')}`,
|
||||
driveId: `drive-${i % 4}`,
|
||||
date: 1_700_000_000_000 - i * (DAY / 2)
|
||||
};
|
||||
}
|
||||
|
||||
const contextMap = new Map<string, Ctx>();
|
||||
function ctxOf(it: Item): Ctx | undefined {
|
||||
let c = contextMap.get(it.id);
|
||||
if (!c) {
|
||||
c = { date: it.date, driveId: it.driveId };
|
||||
contextMap.set(it.id, c);
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
/** Month bucket key from a ctx date (contiguous under date order). */
|
||||
function monthKey(d: number): string {
|
||||
const dt = new Date(d);
|
||||
return `${dt.getUTCFullYear()}-${`${dt.getUTCMonth() + 1}`.padStart(2, '0')}`;
|
||||
}
|
||||
|
||||
/** Contiguous group-by: date-ordered pages, date buckets. Counts bucketOf calls. */
|
||||
function dateGrouping(counter?: { n: number }): SectionGrouping<Item, Ctx> {
|
||||
return {
|
||||
bucketOf: (_it, ctx) => {
|
||||
if (counter) counter.n++;
|
||||
return ctx ? monthKey(ctx.date) : null;
|
||||
},
|
||||
labelOf: (k) => `📅 ${k}`,
|
||||
ctxOf
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-contiguous group-by mirroring trash "by drive": pages arrive in NAME
|
||||
* order but bucket by driveId, so a fresh page sprays items across every
|
||||
* already-emitted drive bucket. Equivalence must still hold.
|
||||
*/
|
||||
function driveGrouping(counter?: { n: number }): SectionGrouping<Item, Ctx> {
|
||||
return {
|
||||
bucketOf: (_it, ctx) => {
|
||||
if (counter) counter.n++;
|
||||
return ctx ? ctx.driveId : null;
|
||||
},
|
||||
labelOf: (k) => `💾 ${k}`,
|
||||
ctxOf
|
||||
};
|
||||
}
|
||||
|
||||
const PAGE = 50;
|
||||
const PAGES = 50; // 2 500-item drain
|
||||
|
||||
describe('incremental resource sections (benchmark gate)', () => {
|
||||
for (const [name, mk] of [
|
||||
['contiguous date buckets', dateGrouping],
|
||||
['non-contiguous drive buckets', driveGrouping]
|
||||
] as const) {
|
||||
it(`stays deep-equal to the full rebuild at every page — ${name}`, () => {
|
||||
const all = Array.from({ length: PAGE * PAGES }, (_, i) => item(i));
|
||||
const builder = new ResourceSectionsBuilder<Item, Ctx>();
|
||||
// ONE stable grouping across the drain — mirrors the component, where
|
||||
// `activeGroup.bucketOf` is a fixed closure from the page's once-defined
|
||||
// `groupBys`. This is what lets the builder take its incremental path,
|
||||
// so this loop genuinely exercises it (not the rebuild fallback).
|
||||
const g = mk();
|
||||
for (let p = 1; p <= PAGES; p++) {
|
||||
const cumulative = all.slice(0, p * PAGE);
|
||||
const incremental = builder.sync(cumulative, g);
|
||||
const reference = buildResourceSections(cumulative, g);
|
||||
expect(incremental, `page ${p}`).toEqual(reference);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
it('keeps untouched bucket arrays reference-stable and refreshes grown ones', () => {
|
||||
const all = Array.from({ length: 600 }, (_, i) => item(i));
|
||||
const builder = new ResourceSectionsBuilder<Item, Ctx>();
|
||||
const g = dateGrouping();
|
||||
|
||||
const first = builder.sync(all.slice(0, 300), g);
|
||||
const refBefore = new Map(first.map((s) => [s.key, s.rows]));
|
||||
|
||||
const second = builder.sync(all.slice(0, 350), g);
|
||||
let stable = 0;
|
||||
let refreshed = 0;
|
||||
for (const s of second) {
|
||||
const prev = refBefore.get(s.key);
|
||||
if (prev === undefined) continue; // brand-new bucket
|
||||
if (prev === s.rows) stable++;
|
||||
else refreshed++;
|
||||
}
|
||||
// Date-ordered append only grows the boundary bucket(s): most earlier
|
||||
// buckets must be handed back by the SAME reference (VirtualList skips
|
||||
// them), and at least one bucket must be refreshed (it grew).
|
||||
expect(stable).toBeGreaterThan(0);
|
||||
expect(refreshed).toBeGreaterThan(0);
|
||||
expect(stable).toBeGreaterThan(refreshed);
|
||||
});
|
||||
|
||||
it('falls back to a correct full rebuild on group-by switch, deletion and flat', () => {
|
||||
const all = Array.from({ length: 600 }, (_, i) => item(i));
|
||||
const builder = new ResourceSectionsBuilder<Item, Ctx>();
|
||||
const byDate = dateGrouping();
|
||||
const byDrive = driveGrouping();
|
||||
|
||||
// Drain a few pages under date grouping, then switch to drive grouping
|
||||
// (a different bucketOf reference → rebuild).
|
||||
builder.sync(all.slice(0, 300), byDate);
|
||||
expect(builder.sync(all.slice(0, 300), byDrive)).toEqual(
|
||||
buildResourceSections(all.slice(0, 300), byDrive)
|
||||
);
|
||||
|
||||
// Deletion under the SAME grouping (list shrinks / prefix changes) →
|
||||
// rebuild via the append check, not a grouping-ref change.
|
||||
const shrunk = all.slice(0, 300).filter((_, i) => i % 7 !== 0);
|
||||
expect(builder.sync(shrunk, byDrive)).toEqual(buildResourceSections(shrunk, byDrive));
|
||||
|
||||
// Flat pass-through (no bucketOf) yields one section and doesn't wedge the
|
||||
// next grouped sync.
|
||||
const flat: SectionGrouping<Item, Ctx> = { ctxOf };
|
||||
const flatOut = builder.sync(shrunk, flat);
|
||||
expect(flatOut).toEqual([{ key: '', label: '', rows: shrunk }]);
|
||||
expect(flatOut[0].rows).toBe(shrunk); // pass-through, no copy
|
||||
expect(builder.sync(shrunk, byDate)).toEqual(buildResourceSections(shrunk, byDate));
|
||||
});
|
||||
|
||||
it('collapses bucketing work from Σ O(N²/page) to O(N) and runs ≥3x faster', () => {
|
||||
const N = PAGE * PAGES;
|
||||
const all = Array.from({ length: N }, (_, i) => item(i));
|
||||
|
||||
// AFTER: incremental — each item is bucketed exactly once across the drain.
|
||||
// ONE stable grouping (fixed bucketOf), exactly as the component supplies.
|
||||
const afterCounter = { n: 0 };
|
||||
const gAfter = dateGrouping(afterCounter);
|
||||
const builder = new ResourceSectionsBuilder<Item, Ctx>();
|
||||
const t1 = performance.now();
|
||||
for (let p = 1; p <= PAGES; p++) builder.sync(all.slice(0, p * PAGE), gAfter);
|
||||
const afterMs = performance.now() - t1;
|
||||
|
||||
// BEFORE: full rebuild per page — re-buckets the whole cumulative list.
|
||||
const beforeCounter = { n: 0 };
|
||||
const gBefore = dateGrouping(beforeCounter);
|
||||
const t0 = performance.now();
|
||||
for (let p = 1; p <= PAGES; p++) buildResourceSections(all.slice(0, p * PAGE), gBefore);
|
||||
const beforeMs = performance.now() - t0;
|
||||
|
||||
console.info(
|
||||
`resource sections ${PAGES}×${PAGE}: before ${beforeCounter.n} bucketOf calls / ${beforeMs.toFixed(1)} ms — after ${afterCounter.n} calls / ${afterMs.toFixed(1)} ms (${(beforeCounter.n / afterCounter.n).toFixed(1)}x fewer calls, ${(beforeMs / afterMs).toFixed(1)}x wall)`
|
||||
);
|
||||
|
||||
// Incremental buckets each item once: exactly N calls.
|
||||
expect(afterCounter.n).toBe(N);
|
||||
// Full rebuild is quadratic: Σ_{p=1..P} p·PAGE.
|
||||
expect(beforeCounter.n).toBe((PAGES * (PAGES + 1) * PAGE) / 2);
|
||||
expect(afterCounter.n).toBeLessThan(beforeCounter.n / 5);
|
||||
expect(afterMs).toBeLessThan(beforeMs / 3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,189 @@
|
||||
/**
|
||||
* Incremental swimlane bucketing for `ResourceList`, extracted from the
|
||||
* component so the O(N²) accumulation of its `sections` `$derived` can be
|
||||
* replaced with an append-aware builder (and unit/benchmark-tested off the
|
||||
* Svelte reactive graph).
|
||||
*
|
||||
* `ResourceList` pages its list in via infinite scroll (`raw = [...raw,
|
||||
* ...page]`), and `sections` was `$derived` over the WHOLE accumulated list —
|
||||
* so paging to item N re-buckets everything loaded so far, Σ ≈ O(N²/page)
|
||||
* main-thread work during the scroll (the same class ROUND6 fixed for the
|
||||
* files listing and ROUND14 §F2 fixed for favorites, and PhotoTimeline fixed
|
||||
* for the photos grid).
|
||||
*
|
||||
* Grouped listings sort by the active group's `orderBy`, so a fresh page only
|
||||
* ever extends existing buckets or appends new ones — it never reorders an
|
||||
* already-emitted bucket. {@link ResourceSectionsBuilder} exploits that: an
|
||||
* append re-buckets only the fresh page and hands back the SAME array
|
||||
* reference for every untouched bucket (so `VirtualList`, which diffs its
|
||||
* `items` prop by reference, skips re-rendering it) while emitting a fresh
|
||||
* array for each bucket the page actually grew.
|
||||
*
|
||||
* Correctness does not depend on bucket contiguity: even a group-by whose
|
||||
* `bucketOf` is not monotonic in server order (e.g. trash grouped by drive but
|
||||
* ordered by name) stays byte-for-byte equal to the full rebuild — it just
|
||||
* touches more buckets per page. The pure {@link buildResourceSections} is the
|
||||
* verbatim reference (what the old `sections` derive produced); the benchmark
|
||||
* gate asserts the incremental builder stays deep-equal to it at every page.
|
||||
*/
|
||||
|
||||
/** One swimlane: a bucket key, its (possibly async-resolved) header label, and its rows. */
|
||||
export interface ResourceSection<T> {
|
||||
key: string;
|
||||
label: string;
|
||||
rows: T[];
|
||||
}
|
||||
|
||||
/**
|
||||
* The grouping inputs the builder needs, mirroring `ResourceList`'s active
|
||||
* `GroupByDef` plus its per-item context accessor. `bucketOf` undefined means
|
||||
* "flat list" (a single unlabelled section). Generic over the item type `T`
|
||||
* and the per-item context envelope `C` so the module stays independent of the
|
||||
* component's concrete types.
|
||||
*/
|
||||
export interface SectionGrouping<T, C> {
|
||||
/** Map an item + its context to a bucket key; null → the `∅` catch-all bucket. */
|
||||
bucketOf?: (item: T, ctx: C | undefined) => string | null;
|
||||
/** Map a bucket key to its header label; identity when absent. */
|
||||
labelOf?: (key: string) => string;
|
||||
/** Resolve an item's context envelope (e.g. `contextMap.get(item.id)`). */
|
||||
ctxOf: (item: T) => C | undefined;
|
||||
}
|
||||
|
||||
/** The `∅` catch-all key the old derive used for a null bucket (kept byte-identical). */
|
||||
const NULL_BUCKET = '∅';
|
||||
|
||||
/**
|
||||
* Verbatim reference: the `ResourceSection[]` the old `sections` `$derived.by`
|
||||
* produced for `items` under `grouping`. Bucket order is first-appearance;
|
||||
* within a bucket, server order is preserved. The benchmark gate holds the
|
||||
* incremental builder equal to this.
|
||||
*/
|
||||
export function buildResourceSections<T, C>(
|
||||
items: T[],
|
||||
grouping: SectionGrouping<T, C>
|
||||
): ResourceSection<T>[] {
|
||||
const bucketOf = grouping.bucketOf;
|
||||
if (!bucketOf) return [{ key: '', label: '', rows: items }];
|
||||
const order: string[] = [];
|
||||
const map = new Map<string, T[]>();
|
||||
for (const item of items) {
|
||||
const k = bucketOf(item, grouping.ctxOf(item)) ?? NULL_BUCKET;
|
||||
let arr = map.get(k);
|
||||
if (arr === undefined) {
|
||||
arr = [];
|
||||
map.set(k, arr);
|
||||
order.push(k);
|
||||
}
|
||||
arr.push(item);
|
||||
}
|
||||
return order.map((k) => ({ key: k, label: grouping.labelOf?.(k) ?? k, rows: map.get(k)! }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Incremental swimlane builder. Call {@link sync} with the current (already
|
||||
* dotfile-filtered) item list and grouping on every change; it detects the
|
||||
* common case — the list grew by appending a page while the group-by is
|
||||
* unchanged — and re-buckets only the fresh items, reusing every untouched
|
||||
* bucket's array reference so `VirtualList` skips it. Any other change
|
||||
* (group-by switch, deletion, filter toggle, non-append) falls back to a full
|
||||
* rebuild, so the result is always deep-equal to {@link buildResourceSections}.
|
||||
*
|
||||
* Header labels are recomputed on every sync (never cached) because a
|
||||
* group-by's `labelOf` may resolve asynchronously — owner / sharer names
|
||||
* arrive after the rows do, and a cached label would freeze the header at its
|
||||
* fallback. Only the `rows` arrays are reference-stabilised; that is what
|
||||
* `VirtualList` diffs.
|
||||
*/
|
||||
export class ResourceSectionsBuilder<T, C> {
|
||||
/** Last synced list — the append cursor and the append-detection baseline. */
|
||||
#items: T[] = [];
|
||||
/** Bucket keys in first-appearance order. */
|
||||
#order: string[] = [];
|
||||
/** key → the bucket's rows array (a fresh reference whenever it grows). */
|
||||
#rows = new Map<string, T[]>();
|
||||
/** The `bucketOf` identity of the last grouped sync; a change forces a rebuild. */
|
||||
#bucketOf: SectionGrouping<T, C>['bucketOf'] = undefined;
|
||||
/** False until a grouped sync has populated the accumulation state. */
|
||||
#grouped = false;
|
||||
|
||||
/** Whether `next` extends `prev` (same prefix objects + strictly longer). */
|
||||
#isAppend(prev: T[], next: T[]): boolean {
|
||||
if (next.length <= prev.length) return false;
|
||||
// Prefix identity via the boundary object — O(1); the list is only ever
|
||||
// mutated by appending a page or by replacing it with a filtered copy
|
||||
// (which preserves element identity).
|
||||
return prev.length === 0 || next[prev.length - 1] === prev[prev.length - 1];
|
||||
}
|
||||
|
||||
#rebuild(items: T[], grouping: SectionGrouping<T, C>): void {
|
||||
const bucketOf = grouping.bucketOf!;
|
||||
this.#order = [];
|
||||
this.#rows = new Map();
|
||||
for (const item of items) {
|
||||
const k = bucketOf(item, grouping.ctxOf(item)) ?? NULL_BUCKET;
|
||||
let arr = this.#rows.get(k);
|
||||
if (arr === undefined) {
|
||||
arr = [];
|
||||
this.#rows.set(k, arr);
|
||||
this.#order.push(k);
|
||||
}
|
||||
arr.push(item);
|
||||
}
|
||||
this.#items = items;
|
||||
}
|
||||
|
||||
#extend(items: T[], grouping: SectionGrouping<T, C>): void {
|
||||
const bucketOf = grouping.bucketOf!;
|
||||
const fresh = items.slice(this.#items.length);
|
||||
// Collect the fresh page's items per touched bucket, preserving order and
|
||||
// first-appearance for brand-new buckets. Each touched bucket's array is
|
||||
// then rebuilt exactly once (a fresh reference so VirtualList re-renders
|
||||
// it); untouched buckets keep their existing reference untouched.
|
||||
const freshByKey = new Map<string, T[]>();
|
||||
const newKeys: string[] = [];
|
||||
for (const item of fresh) {
|
||||
const k = bucketOf(item, grouping.ctxOf(item)) ?? NULL_BUCKET;
|
||||
let arr = freshByKey.get(k);
|
||||
if (arr === undefined) {
|
||||
arr = [];
|
||||
freshByKey.set(k, arr);
|
||||
if (!this.#rows.has(k)) newKeys.push(k);
|
||||
}
|
||||
arr.push(item);
|
||||
}
|
||||
for (const [k, add] of freshByKey) {
|
||||
const existing = this.#rows.get(k);
|
||||
this.#rows.set(k, existing ? existing.concat(add) : add);
|
||||
}
|
||||
for (const k of newKeys) this.#order.push(k);
|
||||
this.#items = items;
|
||||
}
|
||||
|
||||
sync(items: T[], grouping: SectionGrouping<T, C>): ResourceSection<T>[] {
|
||||
if (!grouping.bucketOf) {
|
||||
// Flat list: a single pass-through section. Reset accumulation so a
|
||||
// later switch back to a grouped view rebuilds from scratch.
|
||||
this.#grouped = false;
|
||||
this.#bucketOf = undefined;
|
||||
this.#items = items;
|
||||
return [{ key: '', label: '', rows: items }];
|
||||
}
|
||||
if (
|
||||
this.#grouped &&
|
||||
this.#bucketOf === grouping.bucketOf &&
|
||||
this.#isAppend(this.#items, items)
|
||||
) {
|
||||
this.#extend(items, grouping);
|
||||
} else {
|
||||
this.#rebuild(items, grouping);
|
||||
}
|
||||
this.#grouped = true;
|
||||
this.#bucketOf = grouping.bucketOf;
|
||||
return this.#order.map((k) => ({
|
||||
key: k,
|
||||
label: grouping.labelOf?.(k) ?? k,
|
||||
rows: this.#rows.get(k)!
|
||||
}));
|
||||
}
|
||||
}
|
||||
@@ -61,23 +61,13 @@ impl ExifService {
|
||||
|
||||
// ── Camera info ──
|
||||
if let Some(field) = exif.get_field(Tag::Make, In::PRIMARY) {
|
||||
let val = field
|
||||
.display_value()
|
||||
.to_string()
|
||||
.trim_matches('"')
|
||||
.trim()
|
||||
.to_string();
|
||||
let val = display_value_trimmed(field);
|
||||
if !val.is_empty() {
|
||||
meta.camera_make = Some(val);
|
||||
}
|
||||
}
|
||||
if let Some(field) = exif.get_field(Tag::Model, In::PRIMARY) {
|
||||
let val = field
|
||||
.display_value()
|
||||
.to_string()
|
||||
.trim_matches('"')
|
||||
.trim()
|
||||
.to_string();
|
||||
let val = display_value_trimmed(field);
|
||||
if !val.is_empty() {
|
||||
meta.camera_model = Some(val);
|
||||
}
|
||||
@@ -115,6 +105,29 @@ impl ExifService {
|
||||
}
|
||||
}
|
||||
|
||||
/// Render an EXIF field's display value, then strip surrounding quotes and
|
||||
/// whitespace (the shape `Make`/`Model` want) in a SINGLE allocation.
|
||||
///
|
||||
/// `display_value().to_string()` is the one unavoidable allocation — the field
|
||||
/// value is materialized to text. The old `…to_string().trim_matches('"')
|
||||
/// .trim().to_string()` chain then threw that `String` away and allocated a
|
||||
/// second time for the trimmed copy. Here the same two-stage trim is applied
|
||||
/// in place on the already-owned buffer (`drain` drops the prefix, `truncate`
|
||||
/// the suffix — both reuse the allocation), so a quoted `"Canon"` costs one
|
||||
/// allocation instead of two.
|
||||
fn display_value_trimmed(field: &exif::Field) -> String {
|
||||
let mut s = field.display_value().to_string();
|
||||
// Same order the old chain used: strip `"` first, then whitespace. The
|
||||
// result is a contiguous subslice of `s`; capture its byte range before
|
||||
// mutating the owned buffer (the borrow ends at these two reads).
|
||||
let trimmed = s.trim_matches('"').trim();
|
||||
let start = trimmed.as_ptr().addr() - s.as_ptr().addr();
|
||||
let len = trimmed.len();
|
||||
s.drain(..start);
|
||||
s.truncate(len);
|
||||
s
|
||||
}
|
||||
|
||||
/// Parse EXIF datetime string "YYYY:MM:DD HH:MM:SS" into DateTime<Utc>.
|
||||
fn parse_exif_datetime(s: &str) -> Option<DateTime<Utc>> {
|
||||
// EXIF dates use ":" as separator for date parts
|
||||
|
||||
@@ -264,13 +264,21 @@ impl ContentIndexWorker {
|
||||
let found: HashSet<Uuid> = files.iter().map(|f| f.0).collect();
|
||||
deletes.extend(upsert_candidates.iter().filter(|id| !found.contains(id)));
|
||||
|
||||
// `supports` lowercases the MIME (and, on a generic MIME, the extension)
|
||||
// — 1–2 allocations per call. Classify each file ONCE here and thread the
|
||||
// flag through both the wanted-hashes filter and the per-file records
|
||||
// loop below, where it used to be re-derived a second time per file.
|
||||
let supported: Vec<bool> = files
|
||||
.iter()
|
||||
.map(|(_, _, name, _, mime, _)| text_extractor::supports(name, mime))
|
||||
.collect();
|
||||
|
||||
// Per-blob text: batch-read the extraction cache, extract misses.
|
||||
let wanted_hashes: Vec<String> = files
|
||||
.iter()
|
||||
.filter(|(_, _, name, _, mime, size)| {
|
||||
text_extractor::supports(name, mime) && *size as u64 <= self.max_extract_file_bytes
|
||||
})
|
||||
.map(|f| f.3.clone())
|
||||
.zip(&supported)
|
||||
.filter(|&(f, sup)| *sup && f.5 as u64 <= self.max_extract_file_bytes)
|
||||
.map(|(f, _)| f.3.clone())
|
||||
.collect();
|
||||
let mut text_by_hash: HashMap<String, Option<String>> = HashMap::new();
|
||||
if !wanted_hashes.is_empty() {
|
||||
@@ -287,8 +295,9 @@ impl ContentIndexWorker {
|
||||
}
|
||||
|
||||
let mut records = Vec::with_capacity(files.len());
|
||||
for (file_id, drive_id, name, blob_hash, mime, size) in files {
|
||||
let supported = text_extractor::supports(&name, &mime);
|
||||
for ((file_id, drive_id, name, blob_hash, mime, size), supported) in
|
||||
files.into_iter().zip(supported)
|
||||
{
|
||||
let content = if !supported {
|
||||
None
|
||||
} else if let Some(cached) = text_by_hash.get(&blob_hash) {
|
||||
|
||||
@@ -349,6 +349,14 @@ impl TantivyContentIndex {
|
||||
.search(&query, &TopDocs::with_limit(limit.max(1)).order_by_score())
|
||||
.map_err(|e| DomainError::internal_error("ContentIndex", format!("search: {e}")))?;
|
||||
|
||||
// No hits → no documents to highlight. `SnippetGenerator::create`
|
||||
// compiles the query against the index (term lookups + weight build);
|
||||
// for a query that matched nothing that is pure waste on the search
|
||||
// request path, and the per-hit loop below never runs. Return early.
|
||||
if top_docs.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
// Snippets highlight CONTENT matches; an empty fragment means the hit
|
||||
// came from the name (or a fuzzy variant) — no snippet then.
|
||||
let snippet_generator = SnippetGenerator::create(&searcher, &*query, fields.content)
|
||||
|
||||
Reference in New Issue
Block a user