Diocrafts
761d159a92
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing - Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[]) - Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends) - 3-phase store_chunks pipeline: Phase 0: batch-check existing chunks (single PG query) Phase 1: selective disk read (skip existing chunks entirely) Phase 2: parallel upload with buffer_unordered(8) - CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback - Transactional manifest + chunk ref-count cascade on remove_reference - 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.) - Update deduplication.md to reflect new architecture
25 lines
1023 B
SQL
25 lines
1023 B
SQL
-- Content-Defined Chunking (CDC) manifests for sub-file deduplication.
|
|
--
|
|
-- Each file uploaded via CDC is split into variable-size chunks (FastCDC).
|
|
-- The manifest records the ordered list of chunk hashes that compose the file.
|
|
-- Individual chunks are stored in storage.blobs (shared across manifests).
|
|
--
|
|
-- Legacy whole-file blobs (pre-CDC) remain in storage.blobs and are
|
|
-- accessed directly when no matching manifest row exists.
|
|
|
|
CREATE TABLE IF NOT EXISTS storage.chunk_manifests (
|
|
file_hash VARCHAR(64) PRIMARY KEY,
|
|
chunk_hashes TEXT[] NOT NULL,
|
|
chunk_sizes BIGINT[] NOT NULL,
|
|
total_size BIGINT NOT NULL,
|
|
chunk_count INTEGER NOT NULL,
|
|
content_type TEXT,
|
|
ref_count INTEGER NOT NULL DEFAULT 1 CHECK (ref_count >= 0),
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
);
|
|
|
|
-- Index for GC: find manifests with no references.
|
|
CREATE INDEX IF NOT EXISTS idx_chunk_manifests_ref_count_zero
|
|
ON storage.chunk_manifests (file_hash)
|
|
WHERE ref_count = 0;
|