fix(upload): stop browser connection exhaustion that froze folder uploads
With WASM finally enabled, large files (e.g. 32 MB logs) started running the delta worker, which opens SEVERAL concurrent requests each (overlapping negotiate batches + chunk PUTs). A few of those running at once blew past the browser's ~6 connections-per-host limit, so plain uploads of the small files queued with zero bytes sent until the 30 s stall watchdog cancelled them — the upload "stuck at 4% / 94%" with N (pending) XHRs in the Network panel. The session-refresh request got starved too (the spurious 401s). - Raise the delta-worker threshold to 64 MB (new DELTA_WORKER_MIN_SIZE) so typical large files take a single-connection plain upload. Delta's payoff is sub-file dedup on RE-upload; on a first upload it is pure connection overhead. Client-side instant-hashing still only reads files < 8 MB into memory. - Lower upload concurrency 3 -> 2, leaving headroom under the 6-connection budget for session refresh/poll and the occasional delta worker. npm run check: clean, 58 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,9 +11,22 @@ import { getCsrfToken } from '$lib/api/csrf';
|
|||||||
import { createFileByHash, dedupCheckBatch } from '$lib/api/endpoints/files';
|
import { createFileByHash, dedupCheckBatch } from '$lib/api/endpoints/files';
|
||||||
import { blake3HexOfFile } from '$lib/vendor/hashWasm';
|
import { blake3HexOfFile } from '$lib/vendor/hashWasm';
|
||||||
|
|
||||||
/** Files smaller than this skip delta: the round-trips cost more than the bytes. */
|
/** Files smaller than this skip delta: the round-trips cost more than the bytes.
|
||||||
|
* Also the upper bound for client-side whole-file hashing (instant by-hash
|
||||||
|
* uploads) — we never read a file larger than this fully into memory. */
|
||||||
export const DELTA_UPLOAD_MIN_SIZE = 8 * 1024 * 1024;
|
export const DELTA_UPLOAD_MIN_SIZE = 8 * 1024 * 1024;
|
||||||
|
|
||||||
|
/** Only files at least this large actually run the delta worker. A delta worker
|
||||||
|
* opens SEVERAL concurrent requests (overlapping `negotiate` batches + chunk
|
||||||
|
* PUTs); a few running at once exhaust the browser's ~6 connections-per-host
|
||||||
|
* budget and starve plain uploads (they queue, then the upload watchdog cancels
|
||||||
|
* them — the "stuck at N%" folder upload). Typical large files (e.g. tens of MB)
|
||||||
|
* therefore go through a single-connection plain upload; delta is reserved for
|
||||||
|
* genuinely huge files, where chunked, resumable transfer earns its keep and few
|
||||||
|
* run concurrently. (Delta's real payoff — sub-file dedup — only helps on
|
||||||
|
* re-upload anyway, not the first upload that dominates these batches.) */
|
||||||
|
const DELTA_WORKER_MIN_SIZE = 64 * 1024 * 1024;
|
||||||
|
|
||||||
const DELTA_WORKER_URL = '/workers/deltaWorker.js';
|
const DELTA_WORKER_URL = '/workers/deltaWorker.js';
|
||||||
const DELTA_TIMEOUT_BASE_MS = 120_000;
|
const DELTA_TIMEOUT_BASE_MS = 120_000;
|
||||||
const DELTA_TIMEOUT_PER_GB_MS = 90_000;
|
const DELTA_TIMEOUT_PER_GB_MS = 90_000;
|
||||||
@@ -59,7 +72,7 @@ export function tryDeltaUpload(
|
|||||||
): Promise<DeltaUploadAnswer | null> {
|
): Promise<DeltaUploadAnswer | null> {
|
||||||
if (
|
if (
|
||||||
!folderId ||
|
!folderId ||
|
||||||
file.size < DELTA_UPLOAD_MIN_SIZE ||
|
file.size < DELTA_WORKER_MIN_SIZE ||
|
||||||
usable === false ||
|
usable === false ||
|
||||||
typeof Worker === 'undefined'
|
typeof Worker === 'undefined'
|
||||||
) {
|
) {
|
||||||
|
|||||||
@@ -289,11 +289,12 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Upload at most this many files concurrently. Bounded so one stuck file
|
// Upload at most this many files concurrently. Kept low so we stay well under
|
||||||
// blocks only its own lane (the others keep going) without overwhelming the
|
// the browser's ~6 connections-per-host budget — leaving headroom for the
|
||||||
// browser's per-host connection cap, spawning too many delta workers, or
|
// session-refresh/poll requests and (for genuinely huge files) a delta worker,
|
||||||
// over-contending the server with many large concurrent uploads.
|
// which itself opens several connections. Over-subscribing here is what made
|
||||||
const UPLOAD_CONCURRENCY = 3;
|
// small uploads queue until the watchdog cancelled them ("stuck at N%").
|
||||||
|
const UPLOAD_CONCURRENCY = 2;
|
||||||
|
|
||||||
/** Outer backstop deadline (ms). The plain-upload path already self-aborts on
|
/** Outer backstop deadline (ms). The plain-upload path already self-aborts on
|
||||||
* a stalled connection (see `uploadFileWithProgress`); this only catches a
|
* a stalled connection (see `uploadFileWithProgress`); this only catches a
|
||||||
|
|||||||
Reference in New Issue
Block a user