Instant upload: register already-owned content by hash, zero bytes on the wire

Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.

Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
  Create permission on the target folder via the authorization engine,
  hash ownership via the existing user-scoped query (a non-owned hash
  returns 404 — same shape as "no such blob" — and emits an
  instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
  registration (compensation included); is_new_blob=false so lifecycle
  hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
  and injected, instead of only living on AppState.

Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
  compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
  server content addresses bit for bit. Built by scripts/build-wasm.sh;
  the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
  npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
  the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
  it the round-trips cost more than the bytes), user-scoped
  /api/dedup/check, by-hash registration, and silent fallback to the
  normal byte upload on any miss, race or unsupported environment.
  Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
  (previous vendors were .mjs and never matched the *.js include).

Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:54:32 +00:00
parent 944c833787
commit 0fab4ce17d
17 changed files with 1209 additions and 61 deletions
+1
View File
@@ -100,3 +100,4 @@ tests/e2e/playwright/.auth/
# Test fixtures generated on-the-fly by tests/api/run.sh # Test fixtures generated on-the-fly by tests/api/run.sh
tests/fixtures/chunk-over-cap-*.bin tests/fixtures/chunk-over-cap-*.bin
wasm/oxicloud-hash/target/
+1 -1
View File
@@ -7,7 +7,7 @@
} }
}, },
"files": { "files": {
"includes": ["static/**/*.js", "static/**/*.css", "static/**/*.json", "!static/js/vendors/"] "includes": ["static/**/*.js", "static/**/*.css", "static/**/*.json", "!static/js/vendors"]
}, },
"formatter": { "formatter": {
"enabled": true, "enabled": true,
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Rebuild the vendored BLAKE3 WASM module (static/js/vendors/hash-wasm/).
#
# The generated artifacts ARE committed — like the other vendored modules
# (pdf.js) — so regular frontend/backend builds never need the wasm
# toolchain. Re-run this script only when wasm/oxicloud-hash/ changes
# (e.g. bumping the blake3 crate, or adding FastCDC for the delta-sync
# client) and commit the regenerated files.
#
# Requirements (one-time):
# rustup target add wasm32-unknown-unknown
# cargo install wasm-bindgen-cli --locked
#
# wasm-bindgen-cli's version must match the crate's `wasm-bindgen`
# dependency; cargo prints a clear error when they drift.
set -euo pipefail
cd "$(dirname "$0")/.."
CRATE=wasm/oxicloud-hash
OUT=static/js/vendors/hash-wasm
# SIMD128 is baseline in every evergreen browser (Chrome 91+, Firefox 89+,
# Safari 16.4+) and is worth ~3-4× in hashing throughput. Browsers without
# it fail instantiation; the frontend detects that and falls back to a
# plain byte upload.
RUSTFLAGS="-C target-feature=+simd128" \
cargo build \
--manifest-path "$CRATE/Cargo.toml" \
--target wasm32-unknown-unknown \
--release
wasm-bindgen \
--target web \
--no-typescript \
--out-dir "$OUT" \
"$CRATE/target/wasm32-unknown-unknown/release/oxicloud_hash_wasm.wasm"
echo "Vendored: $(ls -la "$OUT" | tail -n +2 | awk '{print $9, "("$5" bytes)"}' | xargs)"
+141 -1
View File
@@ -1,14 +1,19 @@
use std::sync::Arc; use std::sync::Arc;
use uuid::Uuid;
use crate::application::dtos::file_dto::FileDto; use crate::application::dtos::file_dto::FileDto;
use crate::application::ports::authorization_ports::AuthorizationEngine;
use crate::application::ports::file_lifecycle::FileLifecycleHook; use crate::application::ports::file_lifecycle::FileLifecycleHook;
use crate::application::ports::file_ports::{FileUploadUseCase, StoredBlob}; use crate::application::ports::file_ports::{FileUploadUseCase, StoredBlob};
use crate::application::ports::storage_ports::{FileReadPort, FileWritePort}; use crate::application::ports::storage_ports::{FileReadPort, FileWritePort, StorageUsagePort};
use crate::application::services::storage_usage_service::StorageUsageService; use crate::application::services::storage_usage_service::StorageUsageService;
use crate::common::errors::DomainError; use crate::common::errors::DomainError;
use crate::domain::services::authorization::{Permission, Resource, Subject};
use crate::infrastructure::repositories::pg::FileBlobReadRepository; use crate::infrastructure::repositories::pg::FileBlobReadRepository;
use crate::infrastructure::repositories::pg::FileBlobWriteRepository; use crate::infrastructure::repositories::pg::FileBlobWriteRepository;
use crate::infrastructure::services::dedup_service::DedupService;
use crate::infrastructure::services::file_content_cache::FileContentCache; use crate::infrastructure::services::file_content_cache::FileContentCache;
use crate::infrastructure::services::pg_acl_engine::PgAclEngine;
use tracing::{debug, info, warn}; use tracing::{debug, info, warn};
/// Helper function to extract username from folder path string. /// Helper function to extract username from folder path string.
@@ -49,6 +54,18 @@ pub struct FileUploadService {
content_cache: Option<Arc<FileContentCache>>, content_cache: Option<Arc<FileContentCache>>,
/// Single lifecycle dispatcher — fires on_file_created / on_file_updated. /// Single lifecycle dispatcher — fires on_file_created / on_file_updated.
file_lifecycle_hook: Option<Arc<dyn FileLifecycleHook>>, file_lifecycle_hook: Option<Arc<dyn FileLifecycleHook>>,
/// Dependencies of the instant-upload path
/// (`create_file_from_owned_blob_with_perms`); `None` in minimal test
/// wiring.
instant_upload: Option<InstantUploadDeps>,
}
/// Everything the instant-upload path needs beyond the upload service's own
/// ports: permission checks, the dedup index, and quota enforcement.
struct InstantUploadDeps {
authz: Arc<PgAclEngine>,
dedup: Arc<DedupService>,
quota: Arc<StorageUsageService>,
} }
impl FileUploadService { impl FileUploadService {
@@ -60,6 +77,7 @@ impl FileUploadService {
storage_usage_service: None, storage_usage_service: None,
content_cache: None, content_cache: None,
file_lifecycle_hook: None, file_lifecycle_hook: None,
instant_upload: None,
} }
} }
@@ -74,9 +92,26 @@ impl FileUploadService {
storage_usage_service: None, storage_usage_service: None,
content_cache: None, content_cache: None,
file_lifecycle_hook: None, file_lifecycle_hook: None,
instant_upload: None,
} }
} }
/// Wires the authorization engine, dedup index and quota service that
/// power the instant-upload path.
pub fn with_instant_upload(
mut self,
authz: Arc<PgAclEngine>,
dedup: Arc<DedupService>,
quota: Arc<StorageUsageService>,
) -> Self {
self.instant_upload = Some(InstantUploadDeps {
authz,
dedup,
quota,
});
self
}
/// Configures the content cache for invalidation on file updates. /// Configures the content cache for invalidation on file updates.
pub fn with_content_cache(mut self, cache: Arc<FileContentCache>) -> Self { pub fn with_content_cache(mut self, cache: Arc<FileContentCache>) -> Self {
self.content_cache = Some(cache); self.content_cache = Some(cache);
@@ -98,6 +133,111 @@ impl FileUploadService {
self self
} }
// ── Instant upload (zero content bytes) ──────────────────────
/// Register a new file row pointing at a blob the caller **already
/// owns** — the instant-upload path: the client proved it has the
/// content by hash, so no bytes travel and no chunk is written. Pure
/// metadata: one ref_count bump + one row INSERT.
///
/// Security model (mirrors `GET /api/dedup/check/{hash}`):
/// - The caller must have `Create` permission on the target folder.
/// - The hash is only claimable when the caller owns at least one
/// non-trashed file referencing it — never a global content oracle.
/// A non-owned hash returns `NotFound` (anti-enumeration: same shape
/// as "no such blob") and emits an `instant_upload.rejected` audit
/// event with the real reason.
/// - Quota is enforced on the logical size, exactly like a byte upload.
pub async fn create_file_from_owned_blob_with_perms(
&self,
caller_id: Uuid,
name: String,
folder_id: String,
hash: &str,
) -> Result<FileDto, DomainError> {
let Some(InstantUploadDeps {
authz,
dedup,
quota,
}) = &self.instant_upload
else {
return Err(DomainError::internal_error(
"FileUpload",
"instant upload is not wired (authz/dedup/quota missing)",
));
};
// ── AuthZ: Create on the target folder ───────────────────
let folder_uuid = Uuid::parse_str(&folder_id)
.map_err(|_| DomainError::not_found("Folder", folder_id.clone()))?;
authz
.require(
Subject::User(caller_id),
Permission::Create,
Resource::Folder(folder_uuid),
)
.await?;
// ── Ownership: only blobs the caller can already read ────
if !dedup
.user_owns_blob_reference(hash, &caller_id.to_string())
.await
{
tracing::info!(
target: "audit",
event = "instant_upload.rejected",
reason = "hash_not_owned",
caller_id = %caller_id,
blob_hash = %hash,
"👮🏻‍♂️ Instant upload rejected: caller owns no file referencing the claimed hash",
);
return Err(DomainError::not_found("Blob", hash));
}
let Some(metadata) = dedup.get_blob_metadata(hash).await else {
// Lost a race with the last-reference delete — same shape as
// "never existed".
return Err(DomainError::not_found("Blob", hash));
};
// ── Quota on the logical size, before taking any reference ──
quota.check_storage_quota(caller_id, metadata.size).await?;
// The manifest knows the original content type; fall back to the
// new name's extension when the stored one is generic.
let claimed = metadata.content_type.as_deref().unwrap_or("");
let content_type =
match crate::common::mime_detect::refine_content_type(&[], &name, claimed) {
ct if ct.is_empty() => "application/octet-stream".to_string(),
ct => ct,
};
// Take the reference the row registration will consume (it releases
// it again on any failure). A concurrent GC between the ownership
// check and this bump surfaces as NotFound — the client falls back
// to a normal byte upload.
dedup.add_reference(hash).await?;
let dto = self
.upload_file_streaming(
name,
Some(folder_id),
content_type,
StoredBlob {
hash: hash.to_string(),
size: metadata.size,
is_new_blob: false,
},
)
.await?;
info!(
"⚡ INSTANT UPLOAD: {} ({} bytes, 0 transferred, ID: {})",
dto.name, metadata.size, dto.id
);
Ok(dto)
}
// ── private helpers ────────────────────────────────────────── // ── private helpers ──────────────────────────────────────────
/// Optionally update storage usage after a successful upload. /// Optionally update storage usage after a successful upload.
+20 -5
View File
@@ -439,6 +439,7 @@ impl AppServiceFactory {
repos: &RepositoryServices, repos: &RepositoryServices,
trash_service: Option<Arc<TrashService>>, trash_service: Option<Arc<TrashService>>,
authz: &Arc<PgAclEngine>, authz: &Arc<PgAclEngine>,
storage_usage: &Arc<StorageUsageService>,
) -> ApplicationServices { ) -> ApplicationServices {
// Main services // Main services
let folder_service = Arc::new(FolderService::new( let folder_service = Arc::new(FolderService::new(
@@ -452,7 +453,12 @@ impl AppServiceFactory {
repos.file_read_repository.clone(), repos.file_read_repository.clone(),
) )
.with_content_cache(core.file_content_cache.clone()) .with_content_cache(core.file_content_cache.clone())
.with_file_lifecycle_hook(core.file_lifecycle.clone()), .with_file_lifecycle_hook(core.file_lifecycle.clone())
.with_instant_upload(
authz.clone(),
core.dedup_service.clone(),
storage_usage.clone(),
),
); );
let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache( let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache(
@@ -733,9 +739,19 @@ impl AppServiceFactory {
.create_trash_service(&repos, &core, &authorization) .create_trash_service(&repos, &core, &authorization)
.await; .await;
// 3c. Storage usage / quota service (needed by the instant-upload
// path inside the application services, and re-exposed on AppState
// for the handler-side quota checks of the byte-upload paths).
let storage_usage = self.create_storage_usage_service(&repos, &pool, &maintenance_pool);
// 4. Application services (with trash + authz already wired) // 4. Application services (with trash + authz already wired)
let mut apps = let mut apps = self.create_application_services(
self.create_application_services(&core, &repos, trash_service.clone(), &authorization); &core,
&repos,
trash_service.clone(),
&authorization,
&storage_usage,
);
// 5. Share service // 5. Share service
let share_service = self.create_share_service(&repos, &pool, &authorization); let share_service = self.create_share_service(&repos, &pool, &authorization);
@@ -775,8 +791,7 @@ impl AppServiceFactory {
recent_service = Some(recent.clone()); recent_service = Some(recent.clone());
apps.recent_service = Some(recent); apps.recent_service = Some(recent);
storage_usage_service = storage_usage_service = Some(storage_usage.clone());
Some(self.create_storage_usage_service(&repos, &pool, &maintenance_pool));
self.start_tree_etag_flush_job(&maintenance_pool); self.start_tree_etag_flush_job(&maintenance_pool);
@@ -69,6 +69,57 @@ impl FileHandler {
} }
} }
/// Instant upload: create a file from a blob the caller already owns.
///
/// Zero content bytes travel — the client proved possession of the
/// content by hash (it computed BLAKE3 locally and confirmed via
/// `GET /api/dedup/check/{hash}`), so the server only bumps the blob's
/// reference count and registers the metadata row.
///
/// All authorization (folder Create permission, hash ownership with
/// anti-enumeration, quota) lives in the application service.
pub(super) async fn create_file_by_hash_impl(
State(state): State<GlobalState>,
auth_user: AuthUser,
Json(request): Json<CreateFileByHashRequest>,
) -> impl IntoResponse {
// Hash shape check — same contract as /api/dedup/check/{hash}.
if request.hash.len() != 64 || !request.hash.chars().all(|c| c.is_ascii_hexdigit()) {
return AppError::bad_request(
"Invalid hash format. Expected BLAKE3 (64 hex characters)",
)
.into_response();
}
// Basename only — same path-traversal guard as the multipart upload.
let filename = request
.name
.rsplit('/')
.next()
.unwrap_or(&request.name)
.rsplit('\\')
.next()
.unwrap_or(&request.name)
.to_string();
if filename.is_empty() {
return AppError::bad_request("File name must not be empty").into_response();
}
match state
.applications
.file_upload_service
.create_file_from_owned_blob_with_perms(
auth_user.id,
filename,
request.folder_id,
&request.hash,
)
.await
{
Ok(file) => Self::created_json_response(&file).into_response(),
Err(err) => Self::domain_error_response(err).into_response(),
}
}
/// Core upload logic shared by [`Self::upload_file`] and /// Core upload logic shared by [`Self::upload_file`] and
/// [`Self::upload_file_with_thumbnails`]. /// [`Self::upload_file_with_thumbnails`].
/// ///
@@ -1039,6 +1090,39 @@ pub async fn upload_file_with_thumbnails(
FileHandler::upload_file_with_thumbnails_impl(state, auth_user, multipart).await FileHandler::upload_file_with_thumbnails_impl(state, auth_user, multipart).await
} }
/// Request body for the instant-upload endpoint.
#[derive(Debug, Deserialize, ToSchema)]
pub struct CreateFileByHashRequest {
/// File name to create (path components are stripped).
pub name: String,
/// Target folder ID (the caller needs Create permission on it).
pub folder_id: String,
/// BLAKE3 hash (64 hex chars) of content the caller already owns.
pub hash: String,
}
#[utoipa::path(
post,
path = "/api/files/by-hash",
request_body = CreateFileByHashRequest,
responses(
(status = 201, description = "File created from an already-owned blob — zero bytes transferred", body = FileDto),
(status = 400, description = "Invalid hash format or empty name"),
(status = 404, description = "No owned blob with this hash (anti-enumeration: same shape as unknown hash)"),
(status = 409, description = "A file with this name already exists in the folder"),
(status = 507, description = "Storage quota exceeded"),
),
security(("bearerAuth" = [])),
tag = "files"
)]
pub async fn create_file_by_hash(
state: State<GlobalState>,
auth_user: AuthUser,
request: Json<CreateFileByHashRequest>,
) -> impl IntoResponse {
FileHandler::create_file_by_hash_impl(state, auth_user, request).await
}
#[utoipa::path( #[utoipa::path(
get, get,
path = "/api/files/{id}", path = "/api/files/{id}",
+1
View File
@@ -80,6 +80,7 @@ use crate::interfaces::api::handlers::file_handler::MoveFilePayload;
// File handlers (free functions — see file_handler.rs for why) // File handlers (free functions — see file_handler.rs for why)
handlers::file_handler::list_files_query, handlers::file_handler::list_files_query,
handlers::file_handler::upload_file_with_thumbnails, handlers::file_handler::upload_file_with_thumbnails,
handlers::file_handler::create_file_by_hash,
handlers::file_handler::download_file, handlers::file_handler::download_file,
handlers::file_handler::get_thumbnail, handlers::file_handler::get_thumbnail,
handlers::file_handler::upload_thumbnail, handlers::file_handler::upload_thumbnail,
+3 -2
View File
@@ -55,8 +55,8 @@ use crate::interfaces::api::handlers::chunked_upload_handler::{
cancel_upload, complete_upload, create_upload, get_upload_status, upload_chunk, cancel_upload, complete_upload, create_upload, get_upload_status, upload_chunk,
}; };
use crate::interfaces::api::handlers::file_handler::{ use crate::interfaces::api::handlers::file_handler::{
delete_file, download_file, get_file_metadata, get_thumbnail, list_files_query, create_file_by_hash, delete_file, download_file, get_file_metadata, get_thumbnail,
move_file_simple, rename_file, upload_file_with_thumbnails, upload_thumbnail, list_files_query, move_file_simple, rename_file, upload_file_with_thumbnails, upload_thumbnail,
}; };
#[allow(deprecated)] #[allow(deprecated)]
use crate::interfaces::api::handlers::folder_handler::{ use crate::interfaces::api::handlers::folder_handler::{
@@ -229,6 +229,7 @@ pub fn create_api_routes(app_state: &Arc<AppState>) -> Router<Arc<AppState>> {
let basic_file_router = Router::new() let basic_file_router = Router::new()
.route("/", get(list_files_query)) .route("/", get(list_files_query))
.route("/upload", post(upload_file_with_thumbnails)) .route("/upload", post(upload_file_with_thumbnails))
.route("/by-hash", post(create_file_by_hash))
.route("/{id}", get(download_file)) .route("/{id}", get(download_file))
.route( .route(
"/{id}/thumbnail/{size}", "/{id}/thumbnail/{size}",
+23
View File
@@ -518,3 +518,26 @@
* @typedef {{kind: 'user', id: string} | {kind: 'group', id: string}} GroupMemberItem * @typedef {{kind: 'user', id: string} | {kind: 'group', id: string}} GroupMemberItem
*/ */
// ------------------- Instant upload (dedup)
/**
* Response from `GET /api/dedup/check/{hash}` — user-scoped: `exists` only
* reflects content the CALLER already owns, never global existence.
* Mirrors `HashCheckResponse` on the server (`dedup_handler.rs`).
* @typedef {Object} HashCheckAnswer
* @property {boolean} exists
* @property {string} hash BLAKE3 echoed back (64 hex chars)
* @property {number} [existing_size] size in bytes, present when `exists`
*/
/**
* Request body for `POST /api/files/by-hash` (instant upload — registers a
* file from an already-owned blob, zero content bytes on the wire).
* Mirrors `CreateFileByHashRequest` on the server (`file_handler.rs`).
* The 201 response body is a {@link FileItem}.
* @typedef {Object} CreateFileByHash
* @property {string} name file name to create (basename only)
* @property {string} folder_id target folder (caller needs Create on it)
* @property {string} hash BLAKE3 of the owned content (64 hex chars)
*/
+67 -52
View File
@@ -12,6 +12,7 @@ import { i18n } from '../../core/i18n.js';
import { notifications } from '../../core/notifications.js'; import { notifications } from '../../core/notifications.js';
import { invalidateFolderMeta } from '../../model/filesModel.js'; import { invalidateFolderMeta } from '../../model/filesModel.js';
import { triggerBrowserDownload } from '../../utils/download.js'; import { triggerBrowserDownload } from '../../utils/download.js';
import { tryInstantUpload } from './instantUpload.js';
/** /**
* @typedef {Object} BatchResult * @typedef {Object} BatchResult
@@ -404,20 +405,28 @@ const fileOps = {
if (quotaStop) return; if (quotaStop) return;
const file = readableFiles[idx]; const file = readableFiles[idx];
const formData = new FormData(); // ── Instant upload: when the server already has this exact
if (targetFolderId) formData.append('folder_id', targetFolderId); // content for this user, register it by hash — zero bytes
formData.append('file', file); // on the wire. Any miss/failure falls back to a byte upload.
/** @type {UploadAnswer | null} */
let result = await tryInstantUpload(file, targetFolderId);
if (result) {
if (batchId) {
try {
notifications.updateFile(batchId, file.name, 100, result.ok ? 'done' : 'error');
} catch (_) {}
}
} else {
const formData = new FormData();
if (targetFolderId) formData.append('folder_id', targetFolderId);
formData.append('file', file);
console.log(`Uploading file to folder: ${targetFolderId || 'root'}`, { // Scale stall timeout with file size:
file: file.name, // base 120s + 60s per GB, so a 7 GB file gets ~540s stall limit
size: file.size const sizeGB = file.size / (1024 * 1024 * 1024);
}); const dynamicTimeout = Math.max(120000, 120000 + Math.ceil(sizeGB) * 60000);
result = await this._uploadFileXHR(formData, batchId, file.name, dynamicTimeout);
// Scale stall timeout with file size: }
// base 120s + 60s per GB, so a 7 GB file gets ~540s stall limit
const sizeGB = file.size / (1024 * 1024 * 1024);
const dynamicTimeout = Math.max(120000, 120000 + Math.ceil(sizeGB) * 60000);
const result = await this._uploadFileXHR(formData, batchId, file.name, dynamicTimeout);
uploadedCount++; uploadedCount++;
@@ -663,48 +672,54 @@ const fileOps = {
const parentPath = parts.slice(0, -1).join('/'); const parentPath = parts.slice(0, -1).join('/');
const targetFolderId = folderMap.get(parentPath) || currentFolderId; const targetFolderId = folderMap.get(parentPath) || currentFolderId;
// ── FIFO/pipe guard (0-byte files only) ── // ── Instant upload (zero bytes on the wire) ──
// Named pipes (runit supervise/control) report size=0 // Same fallback contract as uploadFiles: a null result
// but block on open(). Pre-read only 0-byte files into // means "do the byte upload". The shared accounting
// memory; files with size>0 are always regular files and // after this try block handles both outcomes.
// go straight to FormData (zero extra memory copy). const instant = await tryInstantUpload(file, targetFolderId);
/** @type {Blob} */ if (instant) {
let uploadFile = file; // default: use original File result = instant;
if (file.size === 0) { } else {
try { // ── FIFO/pipe guard (0-byte files only) ──
const buf = await Promise.race([ // Named pipes (runit supervise/control) report size=0
file.arrayBuffer(), // but block on open(). Pre-read only 0-byte files into
new Promise((_, rej) => setTimeout(() => rej(new Error('read-timeout')), 2000)) // memory; files with size>0 are always regular files and
]); // go straight to FormData (zero extra memory copy).
uploadFile = new Blob([buf], { /** @type {Blob} */
type: file.type || 'application/octet-stream' let uploadFile = file; // default: use original File
}); if (file.size === 0) {
} catch { try {
console.warn(`[SKIP] #${idx} ${rel} — cannot read 0-byte file (FIFO/pipe?), skipping`); const buf = await Promise.race([
uploadedCount++; file.arrayBuffer(),
successCount++; new Promise((_, rej) => setTimeout(() => rej(new Error('read-timeout')), 2000))
if (batchId) { ]);
try { uploadFile = new Blob([buf], {
notifications.fileCompleted(batchId, true); type: file.type || 'application/octet-stream'
} catch (_) {} });
} catch {
console.warn(`[SKIP] #${idx} ${rel} — cannot read 0-byte file (FIFO/pipe?), skipping`);
uploadedCount++;
successCount++;
if (batchId) {
try {
notifications.fileCompleted(batchId, true);
} catch (_) {}
}
return;
} }
return;
} }
const formData = new FormData();
formData.append('folder_id', targetFolderId);
formData.append('file', uploadFile, file.name);
const thisTimeout =
file.size === 0
? TIMEOUT_MS_ZERO
: Math.max(TIMEOUT_MIN_MS, TIMEOUT_BASE_MS + Math.ceil(file.size / (1024 * 1024)) * TIMEOUT_PER_MB_MS);
result = await this._uploadFileFetch(formData, thisTimeout);
} }
const formData = new FormData();
formData.append('folder_id', targetFolderId);
formData.append('file', uploadFile, file.name);
const thisTimeout =
file.size === 0
? TIMEOUT_MS_ZERO
: Math.max(TIMEOUT_MIN_MS, TIMEOUT_BASE_MS + Math.ceil(file.size / (1024 * 1024)) * TIMEOUT_PER_MB_MS);
console.log(`[UPLOAD START] #${idx} ${rel} (${file.size} bytes, timeout=${thisTimeout}ms)`);
result = await this._uploadFileFetch(formData, thisTimeout);
console.log(`[UPLOAD END] #${idx} ${rel} ok=${result.ok}${result.errorMsg ? ` err=${result.errorMsg}` : ''}`);
} catch (e) { } catch (e) {
result = { result = {
ok: false, ok: false,
+177
View File
@@ -0,0 +1,177 @@
/**
* OxiCloud - Instant upload (zero-byte dedup upload)
*
* Before transferring a file's bytes, compute its BLAKE3 locally (in a
* worker, off the main thread) and ask the server whether the caller
* already owns that exact content (`GET /api/dedup/check/{hash}` — the
* check is user-scoped, never a global content oracle). On a hit, a
* single metadata call (`POST /api/files/by-hash`) registers the file
* with ZERO content bytes on the wire.
*
* Performance posture:
* - Hashing runs in a dedicated worker with WASM SIMD128 — the UI thread
* never blocks, RAM stays constant (8 MiB slices).
* - Files below {@link INSTANT_UPLOAD_MIN_SIZE} skip the whole dance:
* two extra round-trips cost more than just uploading them.
* - Any failure (no WASM support, worker error, server miss, races)
* falls back silently to the normal byte upload — instant upload is
* an optimization, never a gate.
*/
import { getCsrfHeaders } from '../../core/csrf.js';
/**
* Files smaller than this upload normally: hashing + two round-trips
* outweigh the transfer. 8 MiB matches the chunked-upload threshold's
* order of magnitude.
*/
export const INSTANT_UPLOAD_MIN_SIZE = 8 * 1024 * 1024;
// Absolute URL on purpose — works in dev and in the release IIFE bundle
// (same pattern as the pdf.js loader in thumbnail.js).
const HASH_WORKER_URL = '/js/workers/hashWorker.js';
/** Hashing budget: 60 s base + 30 s per GB (WASM SIMD does ~0.5-1 GB/s). */
const HASH_TIMEOUT_BASE_MS = 60000;
const HASH_TIMEOUT_PER_GB_MS = 30000;
/**
* `false` once the environment proved unable to run the worker/WASM
* (old browser, blocked worker) — later files skip straight to the byte
* upload instead of failing the same way again. `null` = not yet known.
* @type {boolean | null}
*/
let _instantUploadUsable = null;
/**
* Hash a file in a one-shot worker. Resolves `null` on any failure —
* the caller falls back to a normal upload.
* @param {File} file
* @returns {Promise<string | null>}
*/
function hashFileInWorker(file) {
return new Promise((resolve) => {
/** @type {Worker} */
let worker;
try {
worker = new Worker(HASH_WORKER_URL, { type: 'module' });
} catch (_) {
_instantUploadUsable = false;
resolve(null);
return;
}
const sizeGB = file.size / (1024 * 1024 * 1024);
const timeoutMs = HASH_TIMEOUT_BASE_MS + Math.ceil(sizeGB) * HASH_TIMEOUT_PER_GB_MS;
/** @param {string | null} hash */
const settle = (hash) => {
clearTimeout(timer);
worker.terminate();
resolve(hash);
};
const timer = setTimeout(() => settle(null), timeoutMs);
worker.onmessage = (event) => {
const data = /** @type {{ ok: boolean, hash?: string, error?: string }} */ (event.data);
if (!data.ok) {
// The worker ran but WASM failed (e.g. no SIMD128 support):
// a permanent environment property, don't retry per file.
_instantUploadUsable = false;
}
settle(data.ok && data.hash ? data.hash : null);
};
worker.onerror = () => {
// Worker script failed to load/parse — permanent.
_instantUploadUsable = false;
settle(null);
};
worker.postMessage({ file });
});
}
/**
* Ask the server whether the caller already owns content with this hash.
* @param {string} hash
* @returns {Promise<boolean>}
*/
async function callerOwnsHash(hash) {
try {
const response = await fetch(`/api/dedup/check/${hash}`, {
headers: { 'Cache-Control': 'no-cache, no-store, must-revalidate' }
});
if (!response.ok) return false;
const body = /** @type {import('../../core/types.js').HashCheckAnswer} */ (await response.json());
return body.exists === true;
} catch (_) {
return false;
}
}
/**
* Try to register `file` as a zero-byte instant upload.
*
* Returns `null` whenever the byte upload should proceed (file too
* small, environment unusable, hash miss, lost race, transient errors).
* Returns an upload-result object compatible with the uploaders'
* `UploadAnswer` shape when the attempt is conclusive — success, quota
* exceeded, or name conflict (a byte upload would fail identically).
*
* @param {File} file
* @param {string | null | undefined} folderId
* @returns {Promise<{ ok: boolean, data?: any, errorMsg?: string, isQuotaError?: boolean } | null>}
*/
export async function tryInstantUpload(file, folderId) {
if (!folderId || file.size < INSTANT_UPLOAD_MIN_SIZE || _instantUploadUsable === false || typeof Worker === 'undefined') {
return null;
}
const hash = await hashFileInWorker(file);
if (!hash) return null;
if (!(await callerOwnsHash(hash))) return null;
try {
const response = await fetch('/api/files/by-hash', {
method: 'POST',
headers: {
...getCsrfHeaders(),
'Content-Type': 'application/json',
'Cache-Control': 'no-cache, no-store, must-revalidate'
},
body: JSON.stringify(
/** @type {import('../../core/types.js').CreateFileByHash} */ ({
name: file.name,
folder_id: folderId,
hash
})
)
});
if (response.status === 201) {
return { ok: true, data: await response.json() };
}
/** @type {string} */
let errorMsg = `Instant upload failed (HTTP ${response.status})`;
try {
const body = await response.json();
errorMsg = body.message || body.error || errorMsg;
} catch (_) {}
if (response.status === 507) {
return { ok: false, isQuotaError: true, errorMsg };
}
if (response.status === 409) {
// Duplicate name in the folder — a byte upload would hit the
// exact same conflict; surface it without transferring.
return { ok: false, errorMsg };
}
// 404 (ownership race with a delete+GC), 4xx/5xx: fall back to the
// byte upload — the server dedups it on write anyway.
return null;
} catch (_) {
return null;
}
}
+261
View File
@@ -0,0 +1,261 @@
/**
* Incremental BLAKE3 hasher.
*
* ```js
* const h = new Blake3Hasher();
* h.update(chunkBytes); // repeat per slice
* const hex = h.finalizeHex();
* ```
*/
export class Blake3Hasher {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
Blake3HasherFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_blake3hasher_free(ptr, 0);
}
/**
* Bytes hashed so far — lets the worker report progress without
* tracking its own counter.
* @returns {number}
*/
count() {
const ret = wasm.blake3hasher_count(this.__wbg_ptr);
return ret;
}
/**
* Finish and return the lowercase hex digest (64 chars). The hasher
* can keep receiving `update` calls afterwards (BLAKE3 finalization
* is non-destructive), but the frontend treats it as terminal.
* @returns {string}
*/
finalizeHex() {
let deferred1_0;
let deferred1_1;
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
wasm.blake3hasher_finalizeHex(retptr, this.__wbg_ptr);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
deferred1_0 = r0;
deferred1_1 = r1;
return getStringFromWasm0(r0, r1);
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
wasm.__wbindgen_export2(deferred1_0, deferred1_1, 1);
}
}
/**
* Create a fresh hasher.
*/
constructor() {
const ret = wasm.blake3hasher_new();
this.__wbg_ptr = ret;
Blake3HasherFinalization.register(this, this.__wbg_ptr, this);
return this;
}
/**
* Feed one slice of the file.
* @param {Uint8Array} data
*/
update(data) {
const ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_export);
const len0 = WASM_VECTOR_LEN;
wasm.blake3hasher_update(this.__wbg_ptr, ptr0, len0);
}
}
if (Symbol.dispose) Blake3Hasher.prototype[Symbol.dispose] = Blake3Hasher.prototype.free;
/**
* One-shot convenience for small buffers.
* @param {Uint8Array} data
* @returns {string}
*/
export function blake3Hex(data) {
let deferred2_0;
let deferred2_1;
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArray8ToWasm0(data, wasm.__wbindgen_export);
const len0 = WASM_VECTOR_LEN;
wasm.blake3Hex(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
deferred2_0 = r0;
deferred2_1 = r1;
return getStringFromWasm0(r0, r1);
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
wasm.__wbindgen_export2(deferred2_0, deferred2_1, 1);
}
}
function __wbg_get_imports() {
const import0 = {
__proto__: null,
__wbg___wbindgen_throw_bbadd78c1bac3a77: (arg0, arg1) => {
throw new Error(getStringFromWasm0(arg0, arg1));
}
};
return {
__proto__: null,
'./oxicloud_hash_wasm_bg.js': import0
};
}
const Blake3HasherFinalization =
typeof FinalizationRegistry === 'undefined'
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry((ptr) => wasm.__wbg_blake3hasher_free(ptr, 1));
let cachedDataViewMemory0 = null;
function getDataViewMemory0() {
if (
cachedDataViewMemory0 === null ||
cachedDataViewMemory0.buffer.detached === true ||
(cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)
) {
cachedDataViewMemory0 = new DataView(wasm.memory.buffer);
}
return cachedDataViewMemory0;
}
function getStringFromWasm0(ptr, len) {
return decodeText(ptr >>> 0, len);
}
let cachedUint8ArrayMemory0 = null;
function getUint8ArrayMemory0() {
if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) {
cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer);
}
return cachedUint8ArrayMemory0;
}
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1, 1) >>> 0;
getUint8ArrayMemory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
const MAX_SAFARI_DECODE_BYTES = 2146435072;
let numBytesDecoded = 0;
function decodeText(ptr, len) {
numBytesDecoded += len;
if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) {
cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
numBytesDecoded = len;
}
return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len));
}
let WASM_VECTOR_LEN = 0;
let wasmModule, wasmInstance, wasm;
function __wbg_finalize_init(instance, module) {
wasmInstance = instance;
wasm = instance.exports;
wasmModule = module;
cachedDataViewMemory0 = null;
cachedUint8ArrayMemory0 = null;
return wasm;
}
async function __wbg_load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
const validResponse = module.ok && expectedResponseType(module.type);
if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') {
console.warn(
'`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n',
e
);
} else {
throw e;
}
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
function expectedResponseType(type) {
switch (type) {
case 'basic':
case 'cors':
case 'default':
return true;
}
return false;
}
}
function initSync(module) {
if (wasm !== undefined) return wasm;
if (module !== undefined) {
if (Object.getPrototypeOf(module) === Object.prototype) {
({ module } = module);
} else {
console.warn('using deprecated parameters for `initSync()`; pass a single object instead');
}
}
const imports = __wbg_get_imports();
if (!(module instanceof WebAssembly.Module)) {
module = new WebAssembly.Module(module);
}
const instance = new WebAssembly.Instance(module, imports);
return __wbg_finalize_init(instance, module);
}
async function __wbg_init(module_or_path) {
if (wasm !== undefined) return wasm;
if (module_or_path !== undefined) {
if (Object.getPrototypeOf(module_or_path) === Object.prototype) {
({ module_or_path } = module_or_path);
} else {
console.warn('using deprecated parameters for the initialization function; pass a single object instead');
}
}
if (module_or_path === undefined) {
module_or_path = new URL('oxicloud_hash_wasm_bg.wasm', import.meta.url);
}
const imports = __wbg_get_imports();
if (
typeof module_or_path === 'string' ||
(typeof Request === 'function' && module_or_path instanceof Request) ||
(typeof URL === 'function' && module_or_path instanceof URL)
) {
module_or_path = fetch(module_or_path);
}
const { instance, module } = await __wbg_load(await module_or_path, imports);
return __wbg_finalize_init(instance, module);
}
export { __wbg_init as default, initSync };
Binary file not shown.
+79
View File
@@ -0,0 +1,79 @@
/**
* OxiCloud — BLAKE3 hashing worker (instant-upload support).
*
* Hashes a File off the main thread, reading it in fixed-size slices so
* RAM stays constant regardless of file size. The WASM module is compiled
* from the exact same `blake3` crate the server uses, so the digest
* computed here equals the server's content address bit for bit.
*
* Protocol: receives `{ file: File }`, answers
* `{ ok: true, hash: string }` or `{ ok: false, error: string }`.
* The spawner terminates the worker after one file.
*/
// Absolute URL on purpose: vendors are served verbatim at /js/vendors/ in
// both dev and release mode (the release IIFE bundle would break a
// relative import) — same pattern as the pdf.js loader in thumbnail.js.
const WASM_GLUE_URL = '/js/vendors/hash-wasm/oxicloud_hash_wasm.js';
/**
* 8 MiB slices — large enough to amortize the per-slice Blob→ArrayBuffer
* round-trip, small enough that peak worker RAM stays flat for any size.
*/
const SLICE_BYTES = 8 * 1024 * 1024;
/**
* Typed view of the dedicated-worker global scope. The project's
* jsconfig targets the DOM lib, where `self` is a Window — cast to the
* two members this worker actually uses.
* @type {{ onmessage: ((event: MessageEvent) => void) | null,
* postMessage: (message: unknown) => void }}
*/
const workerScope = /** @type {any} */ (self);
/**
* Memoized WASM module (in-flight or settled), `default()` already run.
* Reset on failure so a later message can retry a transient load error.
* @type {Promise<any> | null}
*/
let _wasmPromise = null;
/** @returns {Promise<any>} */
function getWasm() {
if (!_wasmPromise) {
_wasmPromise = import(WASM_GLUE_URL)
.then(async (mod) => {
await mod.default();
return mod;
})
.catch((err) => {
_wasmPromise = null;
throw err;
});
}
return _wasmPromise;
}
workerScope.onmessage = async (event) => {
const file = /** @type {{ file: File }} */ (event.data).file;
try {
const wasm = await getWasm();
const hasher = new wasm.Blake3Hasher();
try {
for (let offset = 0; offset < file.size; offset += SLICE_BYTES) {
const end = Math.min(offset + SLICE_BYTES, file.size);
// eslint-disable-next-line no-await-in-loop -- sequential by design: constant RAM
const buffer = await file.slice(offset, end).arrayBuffer();
hasher.update(new Uint8Array(buffer));
}
workerScope.postMessage({ ok: true, hash: hasher.finalizeHex() });
} finally {
hasher.free();
}
} catch (err) {
workerScope.postMessage({
ok: false,
error: err instanceof Error ? err.message : String(err)
});
}
};
+184
View File
@@ -0,0 +1,184 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "arrayref"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb"
[[package]]
name = "arrayvec"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "blake3"
version = "1.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce"
dependencies = [
"arrayref",
"arrayvec",
"cc",
"cfg-if",
"constant_time_eq",
"cpufeatures",
]
[[package]]
name = "bumpalo"
version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "cc"
version = "1.2.63"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "constant_time_eq"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "libc"
version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "oxicloud-hash-wasm"
version = "0.1.0"
dependencies = [
"blake3",
"wasm-bindgen",
]
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "rustversion"
version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "wasm-bindgen"
version = "0.2.123"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a254a4b10c19a76f09a27640e7ffbf9bc30bf67e16a3bf28aaefa4920fe81563"
dependencies = [
"cfg-if",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.123"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24a40fc75b0ec6f3746ceb10d36f53a93dcd68a93b11b6445983945d79eba0dc"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.123"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "908f34bd9b9ce3d4caf07b72dfab63d61504d156856c6bd3cd87fa350cf3985b"
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.123"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7acbf7616c27b194bbb550bf77ed0c2c3e5b7fd1260a93082b95fb7f47959b92"
dependencies = [
"unicode-ident",
]
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "oxicloud-hash-wasm"
version = "0.1.0"
edition = "2021"
description = "BLAKE3 hashing for the OxiCloud web frontend — compiled from the exact same crate the server uses, so client-side hashes match server-side content addressing bit for bit."
publish = false
# Standalone workspace root: this crate is built only by
# scripts/build-wasm.sh (wasm32 target) and must not join the server
# workspace — `cargo test --workspace` / clippy on the host have nothing
# useful to do with a wasm cdylib.
[workspace]
[lib]
crate-type = ["cdylib"]
[dependencies]
# Same major as the server's Cargo.toml. `wasm32_simd` enables the WASM
# SIMD128 kernels (~3-4× over the portable implementation); every
# evergreen browser since 2023 supports it, and the frontend falls back
# to a plain byte upload when instantiation fails.
blake3 = { version = "1.8.4", default-features = false, features = ["wasm32_simd"] }
wasm-bindgen = "0.2"
[profile.release]
# Hashing throughput is the whole point of this module.
opt-level = 3
lto = "fat"
codegen-units = 1
strip = true
+98
View File
@@ -0,0 +1,98 @@
//! BLAKE3 for the OxiCloud web frontend.
//!
//! Compiled from the same `blake3` crate the server uses, so a hash
//! computed in the browser equals the server's content address bit for
//! bit — the property the instant-upload path depends on.
//!
//! The API is incremental on purpose: the worker feeds the file in
//! slices (`Blob.slice().arrayBuffer()`), keeping RAM constant no matter
//! how large the file is.
use wasm_bindgen::prelude::*;
/// Incremental BLAKE3 hasher.
///
/// ```js
/// const h = new Blake3Hasher();
/// h.update(chunkBytes); // repeat per slice
/// const hex = h.finalizeHex();
/// ```
#[wasm_bindgen]
pub struct Blake3Hasher {
inner: blake3::Hasher,
}
#[wasm_bindgen]
impl Blake3Hasher {
/// Create a fresh hasher.
#[wasm_bindgen(constructor)]
pub fn new() -> Blake3Hasher {
Blake3Hasher {
inner: blake3::Hasher::new(),
}
}
/// Feed one slice of the file.
pub fn update(&mut self, data: &[u8]) {
self.inner.update(data);
}
/// Finish and return the lowercase hex digest (64 chars). The hasher
/// can keep receiving `update` calls afterwards (BLAKE3 finalization
/// is non-destructive), but the frontend treats it as terminal.
#[wasm_bindgen(js_name = finalizeHex)]
pub fn finalize_hex(&self) -> String {
self.inner.finalize().to_hex().to_string()
}
/// Bytes hashed so far — lets the worker report progress without
/// tracking its own counter.
pub fn count(&self) -> f64 {
self.inner.count() as f64
}
}
impl Default for Blake3Hasher {
fn default() -> Self {
Self::new()
}
}
/// One-shot convenience for small buffers.
#[wasm_bindgen(js_name = blake3Hex)]
pub fn blake3_hex(data: &[u8]) -> String {
blake3::hash(data).to_hex().to_string()
}
#[cfg(test)]
mod tests {
use super::*;
/// The vector the frontend smoke test uses — also proves the wasm
/// build hashes identically to the server (same crate, same output).
#[test]
fn hello_world_vector() {
let hasher = {
let mut h = Blake3Hasher::new();
h.update(b"Hello, ");
h.update(b"World!");
h
};
assert_eq!(
hasher.finalize_hex(),
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8"
);
assert_eq!(
blake3_hex(b"Hello, World!"),
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8"
);
}
#[test]
fn empty_input_vector() {
assert_eq!(
blake3_hex(b""),
"af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262"
);
}
}