perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change gated by a before/after in examples/bench_round2.rs — an AFTER that did not beat its BEFORE was to be rolled back; none needed it): - Range requests (REST/DAV/shares) answered from the moka content cache for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice. 256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us). - Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster). Content-Length dropped (size unknown up front). - NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM per-session counter (lazy rebuild on cold start). 1,000-chunk upload gate cost: 33.1s -> 0.09s cumulative. - Delta download + commit-verify now use the CDC path's buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open latency 440ms -> 51ms; order preserved. - CDC ingest settles batches on a spawned task (depth-1 pipeline) so the source stream keeps flowing during PG pin + backend writes; rollback ledger shared + lock-serialized so compensation stays exact on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s. OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch). - Frontend: instant-upload BLAKE3 hashing moved off the main thread to a bounded Web Worker pool (File handles by reference); vitest gate asserts the pool beats sequential (first gate draft posting buffers was 2.6x slower and was rewritten — copies dominated). Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544 integration tests green; 270 frontend tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
@@ -74,6 +74,14 @@ async fn session_bytes_so_far(
|
||||
username: &str,
|
||||
upload_id: &str,
|
||||
) -> Result<u64, AppError> {
|
||||
// Warm path: O(1) in-RAM counter maintained by the PUT handler and
|
||||
// the service (seeded on MKCOL, dropped on cleanup/overwrite). The
|
||||
// directory walk below only runs cold (restart / eviction) — the old
|
||||
// shape ran it on EVERY chunk PUT: O(k) stats for chunk k, O(N²/2)
|
||||
// over the upload (benches/NC-CHUNK-GATE.md).
|
||||
if let Some(bytes) = nc.chunked_uploads.cached_session_bytes(username, upload_id) {
|
||||
return Ok(bytes);
|
||||
}
|
||||
let listing = nc
|
||||
.chunked_uploads
|
||||
.list_chunks(username, upload_id)
|
||||
@@ -85,7 +93,10 @@ async fn session_bytes_so_far(
|
||||
// chunk after MKCOL (race-tolerant).
|
||||
return Ok(0);
|
||||
};
|
||||
Ok(listing.chunks.iter().map(|c| c.size).sum())
|
||||
let total = listing.chunks.iter().map(|c| c.size).sum();
|
||||
nc.chunked_uploads
|
||||
.set_session_bytes(username, upload_id, total);
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
/// Dispatch Nextcloud chunked upload WebDAV requests.
|
||||
@@ -314,11 +325,21 @@ async fn handle_put_chunk(
|
||||
.map_err(|e| AppError::bad_request(format!("Invalid chunk path: {}", e)))?;
|
||||
|
||||
let max_chunk = state.core.config.storage.chunk_max_bytes;
|
||||
// A re-PUT of an existing chunk (client retry) makes the running
|
||||
// session counter stale — drop it so the next gate rebuilds from disk.
|
||||
let overwrite = tokio::fs::metadata(&chunk_path).await.is_ok();
|
||||
// No client-side integrity contract on the NC chunked surface — the
|
||||
// NC desktop client validates the assembled-file ETag against the
|
||||
// server-side `oc:checksums` after MOVE. So we skip per-chunk
|
||||
// hashing here (peak heap stays at ~one HTTP frame).
|
||||
stream_body_to_path(req.into_body(), &chunk_path, max_chunk, None).await?;
|
||||
let streamed = stream_body_to_path(req.into_body(), &chunk_path, max_chunk, None).await?;
|
||||
if overwrite {
|
||||
nc.chunked_uploads
|
||||
.forget_session_bytes(&user.username, upload_id);
|
||||
} else {
|
||||
nc.chunked_uploads
|
||||
.bump_session_bytes(&user.username, upload_id, streamed.bytes_written);
|
||||
}
|
||||
|
||||
Ok(Response::builder()
|
||||
.status(StatusCode::CREATED)
|
||||
|
||||
Reference in New Issue
Block a user