perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate

Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):

- Range requests (REST/DAV/shares) answered from the moka content cache
  for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
  256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
  with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
  Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
  per-session counter (lazy rebuild on cold start). 1,000-chunk upload
  gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
  buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
  latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
  the source stream keeps flowing during PG pin + backend writes;
  rollback ledger shared + lock-serialized so compensation stays exact
  on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
  OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
  a bounded Web Worker pool (File handles by reference); vitest gate
  asserts the pool beats sequential (first gate draft posting buffers
  was 2.6x slower and was rewritten — copies dominated).

Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
2026-07-16 16:50:07 +00:00
parent aba89c4f5d
commit 82ee7da0d2
18 changed files with 1245 additions and 195 deletions
+23 -2
View File
@@ -74,6 +74,14 @@ async fn session_bytes_so_far(
username: &str,
upload_id: &str,
) -> Result<u64, AppError> {
// Warm path: O(1) in-RAM counter maintained by the PUT handler and
// the service (seeded on MKCOL, dropped on cleanup/overwrite). The
// directory walk below only runs cold (restart / eviction) — the old
// shape ran it on EVERY chunk PUT: O(k) stats for chunk k, O(N²/2)
// over the upload (benches/NC-CHUNK-GATE.md).
if let Some(bytes) = nc.chunked_uploads.cached_session_bytes(username, upload_id) {
return Ok(bytes);
}
let listing = nc
.chunked_uploads
.list_chunks(username, upload_id)
@@ -85,7 +93,10 @@ async fn session_bytes_so_far(
// chunk after MKCOL (race-tolerant).
return Ok(0);
};
Ok(listing.chunks.iter().map(|c| c.size).sum())
let total = listing.chunks.iter().map(|c| c.size).sum();
nc.chunked_uploads
.set_session_bytes(username, upload_id, total);
Ok(total)
}
/// Dispatch Nextcloud chunked upload WebDAV requests.
@@ -314,11 +325,21 @@ async fn handle_put_chunk(
.map_err(|e| AppError::bad_request(format!("Invalid chunk path: {}", e)))?;
let max_chunk = state.core.config.storage.chunk_max_bytes;
// A re-PUT of an existing chunk (client retry) makes the running
// session counter stale — drop it so the next gate rebuilds from disk.
let overwrite = tokio::fs::metadata(&chunk_path).await.is_ok();
// No client-side integrity contract on the NC chunked surface — the
// NC desktop client validates the assembled-file ETag against the
// server-side `oc:checksums` after MOVE. So we skip per-chunk
// hashing here (peak heap stays at ~one HTTP frame).
stream_body_to_path(req.into_body(), &chunk_path, max_chunk, None).await?;
let streamed = stream_body_to_path(req.into_body(), &chunk_path, max_chunk, None).await?;
if overwrite {
nc.chunked_uploads
.forget_session_bytes(&user.username, upload_id);
} else {
nc.chunked_uploads
.bump_session_bytes(&user.username, upload_id, streamed.bytes_written);
}
Ok(Response::builder()
.status(StatusCode::CREATED)