Claude
0fab4ce17d
Phase 0 of the delta-sync plan. Re-uploading a file the user already has (another device, a restore, a duplicate) used to transfer every byte just for the server to discard them as a dedup hit. The frontend now computes the file's BLAKE3 locally and, on a hit, registers the file with a single ~150-byte metadata call. Server — POST /api/files/by-hash: - All checks live in the application service per the AuthZ rule: Create permission on the target folder via the authorization engine, hash ownership via the existing user-scoped query (a non-owned hash returns 404 — same shape as "no such blob" — and emits an instant_upload.rejected audit event), quota on the logical size. - On success: one ref_count bump + the existing save_file_with_blob row registration (compensation included); is_new_blob=false so lifecycle hooks skip thumbnail regeneration. ~10 ms warm. - The storage-usage service is now built before the application services and injected, instead of only living on AppState. Client — WASM BLAKE3 + worker: - wasm/oxicloud-hash: the exact same blake3 crate the server uses, compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match server content addresses bit for bit. Built by scripts/build-wasm.sh; the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no npm dependencies, no wasm toolchain needed for regular builds. - static/js/workers/hashWorker.js streams the File in 8 MiB slices off the main thread (constant RAM at any file size). - features/files/instantUpload.js orchestrates: threshold (8 MiB — below it the round-trips cost more than the bytes), user-scoped /api/dedup/check, by-hash registration, and silent fallback to the normal byte upload on any miss, race or unsupported environment. Wired into both uploadFiles and uploadFolderEntries. - biome.json vendors exclusion fixed to cover nested directories (previous vendors were .mjs and never matched the *.js include). Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms warm with a 151-byte request (vs 20,971,873 bytes for the byte upload); the copy downloads byte-identical and the manifest ref_count goes 1→2; a second user probing the same hash gets exists:false and 404 plus the audit line; duplicate name → 409, malformed hash → 400; worker and wasm are served with correct MIME (application/wasm). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
80 lines
2.8 KiB
JavaScript
80 lines
2.8 KiB
JavaScript
/**
|
|
* OxiCloud — BLAKE3 hashing worker (instant-upload support).
|
|
*
|
|
* Hashes a File off the main thread, reading it in fixed-size slices so
|
|
* RAM stays constant regardless of file size. The WASM module is compiled
|
|
* from the exact same `blake3` crate the server uses, so the digest
|
|
* computed here equals the server's content address bit for bit.
|
|
*
|
|
* Protocol: receives `{ file: File }`, answers
|
|
* `{ ok: true, hash: string }` or `{ ok: false, error: string }`.
|
|
* The spawner terminates the worker after one file.
|
|
*/
|
|
|
|
// Absolute URL on purpose: vendors are served verbatim at /js/vendors/ in
|
|
// both dev and release mode (the release IIFE bundle would break a
|
|
// relative import) — same pattern as the pdf.js loader in thumbnail.js.
|
|
const WASM_GLUE_URL = '/js/vendors/hash-wasm/oxicloud_hash_wasm.js';
|
|
|
|
/**
|
|
* 8 MiB slices — large enough to amortize the per-slice Blob→ArrayBuffer
|
|
* round-trip, small enough that peak worker RAM stays flat for any size.
|
|
*/
|
|
const SLICE_BYTES = 8 * 1024 * 1024;
|
|
|
|
/**
|
|
* Typed view of the dedicated-worker global scope. The project's
|
|
* jsconfig targets the DOM lib, where `self` is a Window — cast to the
|
|
* two members this worker actually uses.
|
|
* @type {{ onmessage: ((event: MessageEvent) => void) | null,
|
|
* postMessage: (message: unknown) => void }}
|
|
*/
|
|
const workerScope = /** @type {any} */ (self);
|
|
|
|
/**
|
|
* Memoized WASM module (in-flight or settled), `default()` already run.
|
|
* Reset on failure so a later message can retry a transient load error.
|
|
* @type {Promise<any> | null}
|
|
*/
|
|
let _wasmPromise = null;
|
|
|
|
/** @returns {Promise<any>} */
|
|
function getWasm() {
|
|
if (!_wasmPromise) {
|
|
_wasmPromise = import(WASM_GLUE_URL)
|
|
.then(async (mod) => {
|
|
await mod.default();
|
|
return mod;
|
|
})
|
|
.catch((err) => {
|
|
_wasmPromise = null;
|
|
throw err;
|
|
});
|
|
}
|
|
return _wasmPromise;
|
|
}
|
|
|
|
workerScope.onmessage = async (event) => {
|
|
const file = /** @type {{ file: File }} */ (event.data).file;
|
|
try {
|
|
const wasm = await getWasm();
|
|
const hasher = new wasm.Blake3Hasher();
|
|
try {
|
|
for (let offset = 0; offset < file.size; offset += SLICE_BYTES) {
|
|
const end = Math.min(offset + SLICE_BYTES, file.size);
|
|
// eslint-disable-next-line no-await-in-loop -- sequential by design: constant RAM
|
|
const buffer = await file.slice(offset, end).arrayBuffer();
|
|
hasher.update(new Uint8Array(buffer));
|
|
}
|
|
workerScope.postMessage({ ok: true, hash: hasher.finalizeHex() });
|
|
} finally {
|
|
hasher.free();
|
|
}
|
|
} catch (err) {
|
|
workerScope.postMessage({
|
|
ok: false,
|
|
error: err instanceof Error ? err.message : String(err)
|
|
});
|
|
}
|
|
};
|