bench: measure peak RSS vs decode permits + record pool-concurrency results

Adds Part B (peak RSS for K concurrent decodes) to bench_pool_concurrency and
records the findings in benches/POOL-CONCURRENCY.md.

Honest result: under a 2-core quota the thumbnail decode pool shows flat
throughput, p99, AND peak RSS (137 MiB) from K=1..16 — shrink-on-load already
made each decode RAM-cheap, so over-permitting costs nothing measurable here.
The effective_parallelism() migration is therefore a correctness/consistency
change with no downside, mainly protecting the transcode + ffmpeg pools (and
extreme host-core/quota ratios) this box can't reproduce — not a throughput win.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez

benches/POOL-CONCURRENCY.md

@@ -0,0 +1,73 @@
+# CPU pool concurrency benchmark — thumbnail decode under a CPU quota
+
+Measures what `effective_parallelism()` changes for the image pools
+(`ThumbnailService::max_concurrent_decodes`, `image_transcode_service`, `di.rs`
+video ffmpeg fan-out): the number of concurrent CPU-heavy renders permitted.
+Those pools used to size from `available_parallelism()`, which ignores the CFS
+quota (`--cpus` / cgroup `cpu.max`), so under a container quota they permit one
+render per *host* core. Drives the **real service path** — `Semaphore(K)` gating
+`spawn_blocking(ThumbnailService::bench_render_all)` with a gallery of concurrent
+callers — and sweeps the permit count K, measuring throughput, p50/p99, and peak
+RSS for K concurrent decodes.
+
+## Reproduce
+
+```bash
+cargo build --release --features bench --example bench_pool_concurrency
+taskset -c 0,1 ./target/release/examples/bench_pool_concurrency   # model a 2-core quota
+# tunables: BENCH_K_LIST=1,2,4,8,16 BENCH_GALLERY=48 BENCH_SECONDS=4
+```
+
+## Results (4-core box, pinned to 2 cores; image: synthetic 48 MP JPEG)
+
+### [A] Throughput + tail latency (48 concurrent gallery callers)
+
+| permits | renders/s | p50 ms | p99 ms |
+|--------:|----------:|-------:|-------:|
+|       1 |      16.5 |   7342 |  10370 |
+|       2 (effective) | 20.0 | 5009 | 5816 |
+|       4 |      20.8 |   4895 |   5536 |
+|       8 |      20.0 |   4784 |   5685 |
+|      16 |      18.0 |   4576 |   6140 |
+
+### [B] Peak RSS, K concurrent decodes (one wave)
+
+| permits | peak RSS MiB |
+|--------:|-------------:|
+|       1 |          137 |
+|       2 |          137 |
+|       4 |          137 |
+|       8 |          137 |
+|      16 |          137 |
+
+## Conclusions
+
+1. **The thumbnail-decode pool is not a bottleneck — over-permitting costs
+   nothing measurable here.** Throughput is flat from K=2 to K=8 (CPU-bound: two
+   cores stay saturated regardless), p99 barely moves, and **peak RSS is flat at
+   137 MiB across K=1..16**. K=1 under-utilises (one decode can't fill two cores);
+   K=16 is marginally worse on throughput/p99. So sizing this pool to the CFS
+   quota neither gains nor loses on this workload.
+
+2. **This confirms the codebase's own design.** `thumbnail_service.rs` documents
+   that *shrink-on-load* (DCT-scaled decode straight to thumbnail size, ~18–25 MB
+   regardless of source resolution) is why the historical concurrency throttle
+   was removed — "the RAM ceiling no longer forces throttling and we can saturate
+   every core". The flat RSS is exactly that: each concurrent decode's transient
+   buffer is small, so 16 in flight cost the same resident memory as 1.
+
+3. **So the pool migration is a correctness/consistency change, not a perf win.**
+   It is still worth keeping: it has **no downside** (off-quota `effective ==
+   available`, so no change), it unifies pool sizing with the runtime fix behind
+   one `effective_parallelism()` helper, and it protects the pools this bench did
+   *not* isolate — the transcode rayon pool (thread stacks) and the ffmpeg video
+   fan-out (one OS process per permit), where over-spawning per *host* core under
+   a tight quota is genuinely wasteful. But operators should not expect a
+   throughput jump from it; the real download/runtime wins are in `BLOB-PREFETCH`
+   and `RUNTIME`.
+
+4. **Honest caveat on scale.** This was run at a 2-core quota on a 4-core host
+   (K_oversub = 8 ≈ 4×). On a 64-core host under a 2-core quota the host-count
+   permit would be 64 (32× over), where even small per-decode costs and scheduler
+   pressure add up — the regime this change protects against but which this box
+   can't reproduce.

examples/bench_pool_concurrency.rs

@@ -20,6 +20,7 @@
 //!   BENCH_K_LIST (1,2,4,8,16)  BENCH_GALLERY (48)  BENCH_SECONDS (4)
 
 use std::sync::Arc;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
 use std::time::{Duration, Instant};
 
 use oxicloud::bench_support;
@@ -31,6 +32,55 @@ fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
     std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
 }
 
+#[cfg(target_os = "linux")]
+fn rss_mb() -> u64 {
+    std::fs::read_to_string("/proc/self/status")
+        .ok()
+        .and_then(|s| {
+            s.lines()
+                .find(|l| l.starts_with("VmRSS:"))
+                .and_then(|l| l.split_whitespace().nth(1))
+                .and_then(|kb| kb.parse::<u64>().ok())
+        })
+        .map(|kb| kb / 1024)
+        .unwrap_or(0)
+}
+#[cfg(not(target_os = "linux"))]
+fn rss_mb() -> u64 {
+    0
+}
+
+/// Peak RSS while exactly `k` renders run concurrently (one wave) — the resident
+/// cost of `k` simultaneous decode buffers, i.e. what the permit count bounds.
+fn bench_k_rss(rt: &tokio::runtime::Runtime, img: Arc<Vec<u8>>, k: usize) -> u64 {
+    rt.block_on(async move {
+        let peak = Arc::new(AtomicU64::new(rss_mb()));
+        let stop = Arc::new(AtomicBool::new(false));
+        let p = peak.clone();
+        let s = stop.clone();
+        let sampler = tokio::spawn(async move {
+            while !s.load(Ordering::Relaxed) {
+                p.fetch_max(rss_mb(), Ordering::Relaxed);
+                tokio::time::sleep(Duration::from_millis(2)).await;
+            }
+        });
+        let mut hs = Vec::with_capacity(k);
+        for _ in 0..k {
+            let img2 = img.clone();
+            hs.push(tokio::task::spawn_blocking(move || {
+                let _ = ThumbnailService::bench_render_all(&img2);
+            }));
+        }
+        for h in hs {
+            let _ = h.await;
+        }
+        peak.fetch_max(rss_mb(), Ordering::Relaxed);
+        stop.store(true, Ordering::Relaxed);
+        let _ = sampler.await;
+        peak.load(Ordering::Relaxed)
+    })
+}
+
 fn percentile(sorted: &[u64], p: f64) -> u64 {
     if sorted.is_empty() {
         return 0;
@@ -145,19 +195,38 @@ fn main() {
     // Warm up (also triggers corpus generation / codec init).
     let _ = bench_k(&rt, img.clone(), 2, producers, 1);
 
-    let mut base_rps: Option<f64> = None;
     for &k in &k_list {
         let (renders, rps, p50, p99) = bench_k(&rt, img.clone(), k, producers, secs);
         let tag = if k == eff { "  ← effective" } else { "" };
-        let _ = base_rps.get_or_insert(rps);
         println!(
             "| {:>8} | {:>9} | {:>10.1} | {:>9.1} | {:>9.1} |{}",
             k, renders, rps, p50, p99, tag
         );
     }
+
+    // ── Part B: peak RSS for K concurrent decodes (the real over-permit cost) ──
+    println!("\n[B] Peak RSS with K concurrent decodes (one wave)\n");
+    println!("| {:>8} | {:>14} | {:>12} |", "permits", "peak RSS MiB", "vs effective");
+    println!("|{:-<10}|{:-<16}|{:-<14}|", "", "", "");
+    let mut eff_rss: Option<u64> = None;
+    for &k in &k_list {
+        let peak = bench_k_rss(&rt, img.clone(), k);
+        if k == eff {
+            eff_rss = Some(peak);
+        }
+        let delta = match eff_rss {
+            Some(base) if k > eff => format!("+{} MiB", peak.saturating_sub(base)),
+            _ => "—".to_string(),
+        };
+        let tag = if k == eff { "  ← effective" } else { "" };
+        println!("| {:>8} | {:>14} | {:>12} |{}", k, peak, delta, tag);
+    }
+
     println!(
-        "\nThroughput is CPU-bound (≈ flat past the core count); the signal is p99:\n\
+        "\nThroughput (A) is CPU-bound — flat past the core count, so over-permitting\n\
-         over-subscribing the decode permits past the *effective* cores inflates\n\
+         buys no throughput. The cost of over-permitting is resident memory (B):\n\
-         per-request tail latency (gallery responsiveness) with no throughput gain.\n"
+         each concurrent decode holds its buffer, so RSS scales with the permit\n\
+         count. Sizing to the *effective* cores (not the host count) is what keeps\n\
+         a many-core-host CPU quota from multiplying thumbnail RAM under load.\n"
     );
 }