Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
+8
-59
@@ -24,6 +24,7 @@ use crate::application::dtos::search_dto::{
|
||||
};
|
||||
use crate::application::ports::file_ports::{
|
||||
FileManagementUseCase, FileRetrievalUseCase, FileUploadUseCase, OptimizedFileContent,
|
||||
StoredBlob,
|
||||
};
|
||||
use crate::application::ports::folder_ports::FolderUseCase;
|
||||
|
||||
@@ -149,14 +150,13 @@ impl FileReadPort for StubFileReadPort {
|
||||
pub struct StubFileWritePort;
|
||||
|
||||
impl FileWritePort for StubFileWritePort {
|
||||
async fn save_file_from_temp(
|
||||
async fn save_file_with_blob(
|
||||
&self,
|
||||
_name: String,
|
||||
_folder_id: Option<String>,
|
||||
_content_type: String,
|
||||
_temp_path: &Path,
|
||||
_blob_hash: &str,
|
||||
_size: u64,
|
||||
_pre_computed_hash: Option<String>,
|
||||
) -> Result<File, DomainError> {
|
||||
Ok(File::default())
|
||||
}
|
||||
@@ -185,13 +185,11 @@ impl FileWritePort for StubFileWritePort {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn update_file_content_from_temp(
|
||||
async fn update_file_content_with_blob(
|
||||
&self,
|
||||
_file_id: &str,
|
||||
_temp_path: &Path,
|
||||
_blob_hash: &str,
|
||||
_size: u64,
|
||||
_content_type: Option<String>,
|
||||
_pre_computed_hash: Option<String>,
|
||||
_modified_at: Option<i64>,
|
||||
) -> Result<(String, i64), DomainError> {
|
||||
Ok((String::new(), 0))
|
||||
@@ -478,40 +476,7 @@ impl FileUploadUseCase for StubFileUploadUseCase {
|
||||
_name: String,
|
||||
_folder_id: Option<String>,
|
||||
_content_type: String,
|
||||
_temp_path: &Path,
|
||||
_size: u64,
|
||||
_pre_computed_hash: Option<String>,
|
||||
) -> Result<FileDto, DomainError> {
|
||||
Ok(FileDto::default())
|
||||
}
|
||||
|
||||
async fn upload_file_from_path(
|
||||
&self,
|
||||
_name: String,
|
||||
_folder_id: Option<String>,
|
||||
_content_type: String,
|
||||
_file_path: &Path,
|
||||
_pre_computed_hash: Option<String>,
|
||||
) -> Result<FileDto, DomainError> {
|
||||
Ok(FileDto::default())
|
||||
}
|
||||
|
||||
async fn create_file(
|
||||
&self,
|
||||
_parent_path: &str,
|
||||
_filename: &str,
|
||||
_content: &[u8],
|
||||
_content_type: &str,
|
||||
) -> Result<FileDto, DomainError> {
|
||||
Ok(FileDto::default())
|
||||
}
|
||||
|
||||
async fn update_file(
|
||||
&self,
|
||||
_path: &str,
|
||||
_content: &[u8],
|
||||
_content_type: &str,
|
||||
_modified_at: Option<i64>,
|
||||
_blob: StoredBlob,
|
||||
) -> Result<FileDto, DomainError> {
|
||||
Ok(FileDto::default())
|
||||
}
|
||||
@@ -519,10 +484,8 @@ impl FileUploadUseCase for StubFileUploadUseCase {
|
||||
async fn update_file_streaming(
|
||||
&self,
|
||||
_path: &str,
|
||||
_temp_path: &Path,
|
||||
_size: u64,
|
||||
_blob: StoredBlob,
|
||||
_content_type: &str,
|
||||
_pre_computed_hash: Option<String>,
|
||||
_modified_at: Option<i64>,
|
||||
) -> Result<FileDto, DomainError> {
|
||||
Ok(FileDto::default())
|
||||
@@ -756,25 +719,11 @@ impl SearchUseCase for StubSearchUseCase {
|
||||
// DedupPort
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use crate::application::ports::dedup_ports::{
|
||||
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
|
||||
};
|
||||
use crate::application::ports::dedup_ports::{BlobMetadataDto, DedupPort, DedupStatsDto};
|
||||
|
||||
pub struct StubDedupPort;
|
||||
|
||||
impl DedupPort for StubDedupPort {
|
||||
async fn store_from_file(
|
||||
&self,
|
||||
_source_path: &Path,
|
||||
_content_type: Option<String>,
|
||||
_pre_computed_hash: Option<String>,
|
||||
) -> Result<DedupResultDto, DomainError> {
|
||||
Err(DomainError::internal_error(
|
||||
"DedupService",
|
||||
"DedupService not initialized",
|
||||
))
|
||||
}
|
||||
|
||||
async fn blob_exists(&self, _hash: &str) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user