feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip

- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
  (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
  Phase 0: batch-check existing chunks (single PG query)
  Phase 1: selective disk read (skip existing chunks entirely)
  Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
This commit is contained in:
Diocrafts
2026-04-14 23:17:39 +02:00
parent cd3733b459
commit 761d159a92
15 changed files with 1988 additions and 530 deletions
@@ -54,6 +54,12 @@ pub trait BlobStorageBackend: Send + Sync + 'static {
/// without overwriting. Returns the number of bytes stored.
fn put_blob(&self, hash: &str, source_path: &Path) -> BoxFut<'_, Result<u64, DomainError>>;
/// Store a blob from in-memory bytes (used by CDC chunk storage).
///
/// Must be **idempotent**: if the blob already exists the call succeeds
/// without overwriting. Returns the number of bytes stored.
fn put_blob_from_bytes(&self, hash: &str, data: Bytes) -> BoxFut<'_, Result<u64, DomainError>>;
/// Stream the full blob content in chunks.
fn get_blob_stream(&self, hash: &str) -> BoxFut<'_, Result<BlobStream, DomainError>>;
+1 -13
View File
@@ -28,16 +28,11 @@ pub struct BlobMetadataDto {
#[derive(Debug, Clone)]
pub enum DedupResultDto {
/// New content was stored (first occurrence).
NewBlob {
hash: String,
size: u64,
blob_path: PathBuf,
},
NewBlob { hash: String, size: u64 },
/// Content already existed; a reference was added instead.
ExistingBlob {
hash: String,
size: u64,
blob_path: PathBuf,
saved_bytes: u64,
},
}
@@ -57,13 +52,6 @@ impl DedupResultDto {
}
}
pub fn blob_path(&self) -> &Path {
match self {
DedupResultDto::NewBlob { blob_path, .. } => blob_path,
DedupResultDto::ExistingBlob { blob_path, .. } => blob_path,
}
}
pub fn was_deduplicated(&self) -> bool {
matches!(self, DedupResultDto::ExistingBlob { .. })
}