Instant upload: register already-owned content by hash, zero bytes on the wire
Phase 0 of the delta-sync plan. Re-uploading a file the user already has (another device, a restore, a duplicate) used to transfer every byte just for the server to discard them as a dedup hit. The frontend now computes the file's BLAKE3 locally and, on a hit, registers the file with a single ~150-byte metadata call. Server — POST /api/files/by-hash: - All checks live in the application service per the AuthZ rule: Create permission on the target folder via the authorization engine, hash ownership via the existing user-scoped query (a non-owned hash returns 404 — same shape as "no such blob" — and emits an instant_upload.rejected audit event), quota on the logical size. - On success: one ref_count bump + the existing save_file_with_blob row registration (compensation included); is_new_blob=false so lifecycle hooks skip thumbnail regeneration. ~10 ms warm. - The storage-usage service is now built before the application services and injected, instead of only living on AppState. Client — WASM BLAKE3 + worker: - wasm/oxicloud-hash: the exact same blake3 crate the server uses, compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match server content addresses bit for bit. Built by scripts/build-wasm.sh; the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no npm dependencies, no wasm toolchain needed for regular builds. - static/js/workers/hashWorker.js streams the File in 8 MiB slices off the main thread (constant RAM at any file size). - features/files/instantUpload.js orchestrates: threshold (8 MiB — below it the round-trips cost more than the bytes), user-scoped /api/dedup/check, by-hash registration, and silent fallback to the normal byte upload on any miss, race or unsupported environment. Wired into both uploadFiles and uploadFolderEntries. - biome.json vendors exclusion fixed to cover nested directories (previous vendors were .mjs and never matched the *.js include). Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms warm with a 151-byte request (vs 20,971,873 bytes for the byte upload); the copy downloads byte-identical and the manifest ref_count goes 1→2; a second user probing the same hash gets exists:false and 404 plus the audit line; duplicate name → 409, malformed hash → 400; worker and wasm are served with correct MIME (application/wasm). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
@@ -69,6 +69,57 @@ impl FileHandler {
|
||||
}
|
||||
}
|
||||
|
||||
/// Instant upload: create a file from a blob the caller already owns.
|
||||
///
|
||||
/// Zero content bytes travel — the client proved possession of the
|
||||
/// content by hash (it computed BLAKE3 locally and confirmed via
|
||||
/// `GET /api/dedup/check/{hash}`), so the server only bumps the blob's
|
||||
/// reference count and registers the metadata row.
|
||||
///
|
||||
/// All authorization (folder Create permission, hash ownership with
|
||||
/// anti-enumeration, quota) lives in the application service.
|
||||
pub(super) async fn create_file_by_hash_impl(
|
||||
State(state): State<GlobalState>,
|
||||
auth_user: AuthUser,
|
||||
Json(request): Json<CreateFileByHashRequest>,
|
||||
) -> impl IntoResponse {
|
||||
// Hash shape check — same contract as /api/dedup/check/{hash}.
|
||||
if request.hash.len() != 64 || !request.hash.chars().all(|c| c.is_ascii_hexdigit()) {
|
||||
return AppError::bad_request(
|
||||
"Invalid hash format. Expected BLAKE3 (64 hex characters)",
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
// Basename only — same path-traversal guard as the multipart upload.
|
||||
let filename = request
|
||||
.name
|
||||
.rsplit('/')
|
||||
.next()
|
||||
.unwrap_or(&request.name)
|
||||
.rsplit('\\')
|
||||
.next()
|
||||
.unwrap_or(&request.name)
|
||||
.to_string();
|
||||
if filename.is_empty() {
|
||||
return AppError::bad_request("File name must not be empty").into_response();
|
||||
}
|
||||
|
||||
match state
|
||||
.applications
|
||||
.file_upload_service
|
||||
.create_file_from_owned_blob_with_perms(
|
||||
auth_user.id,
|
||||
filename,
|
||||
request.folder_id,
|
||||
&request.hash,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(file) => Self::created_json_response(&file).into_response(),
|
||||
Err(err) => Self::domain_error_response(err).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Core upload logic shared by [`Self::upload_file`] and
|
||||
/// [`Self::upload_file_with_thumbnails`].
|
||||
///
|
||||
@@ -1039,6 +1090,39 @@ pub async fn upload_file_with_thumbnails(
|
||||
FileHandler::upload_file_with_thumbnails_impl(state, auth_user, multipart).await
|
||||
}
|
||||
|
||||
/// Request body for the instant-upload endpoint.
|
||||
#[derive(Debug, Deserialize, ToSchema)]
|
||||
pub struct CreateFileByHashRequest {
|
||||
/// File name to create (path components are stripped).
|
||||
pub name: String,
|
||||
/// Target folder ID (the caller needs Create permission on it).
|
||||
pub folder_id: String,
|
||||
/// BLAKE3 hash (64 hex chars) of content the caller already owns.
|
||||
pub hash: String,
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/api/files/by-hash",
|
||||
request_body = CreateFileByHashRequest,
|
||||
responses(
|
||||
(status = 201, description = "File created from an already-owned blob — zero bytes transferred", body = FileDto),
|
||||
(status = 400, description = "Invalid hash format or empty name"),
|
||||
(status = 404, description = "No owned blob with this hash (anti-enumeration: same shape as unknown hash)"),
|
||||
(status = 409, description = "A file with this name already exists in the folder"),
|
||||
(status = 507, description = "Storage quota exceeded"),
|
||||
),
|
||||
security(("bearerAuth" = [])),
|
||||
tag = "files"
|
||||
)]
|
||||
pub async fn create_file_by_hash(
|
||||
state: State<GlobalState>,
|
||||
auth_user: AuthUser,
|
||||
request: Json<CreateFileByHashRequest>,
|
||||
) -> impl IntoResponse {
|
||||
FileHandler::create_file_by_hash_impl(state, auth_user, request).await
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
get,
|
||||
path = "/api/files/{id}",
|
||||
|
||||
Reference in New Issue
Block a user