fix(upload): bound delta-worker connections instead of disabling delta
Follow-up to the connection-exhaustion fix. Rather than routing large files to plain uploads (which kept STORAGE dedup but gave up delta's re-upload bandwidth savings), keep delta for every file >= 8 MB and instead cap each worker's concurrent connections so a few large files uploading at once can't blow past the browser's ~6-per-host budget and starve the small-file plain uploads. - deltaWorker.js: serialize negotiate (at most one in flight per worker) and drop chunk-PUT concurrency 2 -> 1, so each worker holds ~2 connections max. - deltaUpload.ts: revert the 64 MB threshold back to 8 MB — every large file gets sub-file dedup again. (Storage dedup was never affected: BLAKE3 + CDC + ref-counting run server-side for plain and delta uploads alike.) With main upload concurrency at 2, total in-flight upload connections stay <= ~4. npm run check: clean, 58 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -13,20 +13,12 @@ import { blake3HexOfFile } from '$lib/vendor/hashWasm';
|
||||
|
||||
/** Files smaller than this skip delta: the round-trips cost more than the bytes.
|
||||
* Also the upper bound for client-side whole-file hashing (instant by-hash
|
||||
* uploads) — we never read a file larger than this fully into memory. */
|
||||
* uploads) — we never read a file larger than this fully into memory. Files at
|
||||
* or above this run the delta worker (sub-file dedup, saves re-upload
|
||||
* bandwidth); the worker self-bounds its concurrent connections so a few
|
||||
* running at once can't exhaust the browser's ~6-per-host budget. */
|
||||
export const DELTA_UPLOAD_MIN_SIZE = 8 * 1024 * 1024;
|
||||
|
||||
/** Only files at least this large actually run the delta worker. A delta worker
|
||||
* opens SEVERAL concurrent requests (overlapping `negotiate` batches + chunk
|
||||
* PUTs); a few running at once exhaust the browser's ~6 connections-per-host
|
||||
* budget and starve plain uploads (they queue, then the upload watchdog cancels
|
||||
* them — the "stuck at N%" folder upload). Typical large files (e.g. tens of MB)
|
||||
* therefore go through a single-connection plain upload; delta is reserved for
|
||||
* genuinely huge files, where chunked, resumable transfer earns its keep and few
|
||||
* run concurrently. (Delta's real payoff — sub-file dedup — only helps on
|
||||
* re-upload anyway, not the first upload that dominates these batches.) */
|
||||
const DELTA_WORKER_MIN_SIZE = 64 * 1024 * 1024;
|
||||
|
||||
const DELTA_WORKER_URL = '/workers/deltaWorker.js';
|
||||
const DELTA_TIMEOUT_BASE_MS = 120_000;
|
||||
const DELTA_TIMEOUT_PER_GB_MS = 90_000;
|
||||
@@ -72,7 +64,7 @@ export function tryDeltaUpload(
|
||||
): Promise<DeltaUploadAnswer | null> {
|
||||
if (
|
||||
!folderId ||
|
||||
file.size < DELTA_WORKER_MIN_SIZE ||
|
||||
file.size < DELTA_UPLOAD_MIN_SIZE ||
|
||||
usable === false ||
|
||||
typeof Worker === 'undefined'
|
||||
) {
|
||||
|
||||
Reference in New Issue
Block a user