feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing - Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[]) - Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends) - 3-phase store_chunks pipeline: Phase 0: batch-check existing chunks (single PG query) Phase 1: selective disk read (skip existing chunks entirely) Phase 2: parallel upload with buffer_unordered(8) - CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback - Transactional manifest + chunk ref-count cascade on remove_reference - 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.) - Update deduplication.md to reflect new architecture
This commit is contained in:
@@ -517,7 +517,8 @@ impl ChunkedUploadService {
|
||||
let data_clone = data.clone(); // Bytes::clone is O(1) — just an Arc increment
|
||||
let actual_checksum = tokio::task::spawn_blocking(move || {
|
||||
use md5::{Digest, Md5};
|
||||
format!("{:x}", Md5::digest(&data_clone))
|
||||
let hash = Md5::digest(&data_clone);
|
||||
hash.iter().map(|b| format!("{b:02x}")).collect::<String>()
|
||||
})
|
||||
.await
|
||||
.map_err(|e| format!("MD5 checksum task failed: {e}"))?;
|
||||
|
||||
Reference in New Issue
Block a user