Instant upload: register already-owned content by hash, zero bytes on the wire
Phase 0 of the delta-sync plan. Re-uploading a file the user already has (another device, a restore, a duplicate) used to transfer every byte just for the server to discard them as a dedup hit. The frontend now computes the file's BLAKE3 locally and, on a hit, registers the file with a single ~150-byte metadata call. Server — POST /api/files/by-hash: - All checks live in the application service per the AuthZ rule: Create permission on the target folder via the authorization engine, hash ownership via the existing user-scoped query (a non-owned hash returns 404 — same shape as "no such blob" — and emits an instant_upload.rejected audit event), quota on the logical size. - On success: one ref_count bump + the existing save_file_with_blob row registration (compensation included); is_new_blob=false so lifecycle hooks skip thumbnail regeneration. ~10 ms warm. - The storage-usage service is now built before the application services and injected, instead of only living on AppState. Client — WASM BLAKE3 + worker: - wasm/oxicloud-hash: the exact same blake3 crate the server uses, compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match server content addresses bit for bit. Built by scripts/build-wasm.sh; the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no npm dependencies, no wasm toolchain needed for regular builds. - static/js/workers/hashWorker.js streams the File in 8 MiB slices off the main thread (constant RAM at any file size). - features/files/instantUpload.js orchestrates: threshold (8 MiB — below it the round-trips cost more than the bytes), user-scoped /api/dedup/check, by-hash registration, and silent fallback to the normal byte upload on any miss, race or unsupported environment. Wired into both uploadFiles and uploadFolderEntries. - biome.json vendors exclusion fixed to cover nested directories (previous vendors were .mjs and never matched the *.js include). Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms warm with a 151-byte request (vs 20,971,873 bytes for the byte upload); the copy downloads byte-identical and the manifest ref_count goes 1→2; a second user probing the same hash gets exists:false and 404 plus the audit line; duplicate name → 409, malformed hash → 400; worker and wasm are served with correct MIME (application/wasm). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
//! BLAKE3 for the OxiCloud web frontend.
|
||||
//!
|
||||
//! Compiled from the same `blake3` crate the server uses, so a hash
|
||||
//! computed in the browser equals the server's content address bit for
|
||||
//! bit — the property the instant-upload path depends on.
|
||||
//!
|
||||
//! The API is incremental on purpose: the worker feeds the file in
|
||||
//! slices (`Blob.slice().arrayBuffer()`), keeping RAM constant no matter
|
||||
//! how large the file is.
|
||||
|
||||
use wasm_bindgen::prelude::*;
|
||||
|
||||
/// Incremental BLAKE3 hasher.
|
||||
///
|
||||
/// ```js
|
||||
/// const h = new Blake3Hasher();
|
||||
/// h.update(chunkBytes); // repeat per slice
|
||||
/// const hex = h.finalizeHex();
|
||||
/// ```
|
||||
#[wasm_bindgen]
|
||||
pub struct Blake3Hasher {
|
||||
inner: blake3::Hasher,
|
||||
}
|
||||
|
||||
#[wasm_bindgen]
|
||||
impl Blake3Hasher {
|
||||
/// Create a fresh hasher.
|
||||
#[wasm_bindgen(constructor)]
|
||||
pub fn new() -> Blake3Hasher {
|
||||
Blake3Hasher {
|
||||
inner: blake3::Hasher::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Feed one slice of the file.
|
||||
pub fn update(&mut self, data: &[u8]) {
|
||||
self.inner.update(data);
|
||||
}
|
||||
|
||||
/// Finish and return the lowercase hex digest (64 chars). The hasher
|
||||
/// can keep receiving `update` calls afterwards (BLAKE3 finalization
|
||||
/// is non-destructive), but the frontend treats it as terminal.
|
||||
#[wasm_bindgen(js_name = finalizeHex)]
|
||||
pub fn finalize_hex(&self) -> String {
|
||||
self.inner.finalize().to_hex().to_string()
|
||||
}
|
||||
|
||||
/// Bytes hashed so far — lets the worker report progress without
|
||||
/// tracking its own counter.
|
||||
pub fn count(&self) -> f64 {
|
||||
self.inner.count() as f64
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for Blake3Hasher {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
/// One-shot convenience for small buffers.
|
||||
#[wasm_bindgen(js_name = blake3Hex)]
|
||||
pub fn blake3_hex(data: &[u8]) -> String {
|
||||
blake3::hash(data).to_hex().to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The vector the frontend smoke test uses — also proves the wasm
|
||||
/// build hashes identically to the server (same crate, same output).
|
||||
#[test]
|
||||
fn hello_world_vector() {
|
||||
let hasher = {
|
||||
let mut h = Blake3Hasher::new();
|
||||
h.update(b"Hello, ");
|
||||
h.update(b"World!");
|
||||
h
|
||||
};
|
||||
assert_eq!(
|
||||
hasher.finalize_hex(),
|
||||
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8"
|
||||
);
|
||||
assert_eq!(
|
||||
blake3_hex(b"Hello, World!"),
|
||||
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_input_vector() {
|
||||
assert_eq!(
|
||||
blake3_hex(b""),
|
||||
"af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262"
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user