47 lines
1.6 KiB
Markdown
47 lines
1.6 KiB
Markdown
|
# File Deduplication
|
||
|
|
|
||
|
|
OxiCloud uses **SHA-256 content-addressable storage** to avoid storing duplicate files. If two users upload the same file, only one copy is stored on disk.
|
||
|
|
|
||
|
|
## How It Works
|
||
|
|
|
||
|
|
1. When a file is uploaded, its SHA-256 hash is computed
|
||
|
|
2. The hash is checked against the blob store (`.blobs/{prefix}/{hash}.blob`)
|
||
|
|
3. If a blob with that hash already exists, the file metadata points to the existing blob (no extra disk usage)
|
||
|
|
4. If not, the content is saved as a new blob
|
||
|
|
5. A reference counter tracks how many files point to each blob
|
||
|
|
|
||
|
|
## Automatic Cleanup
|
||
|
|
|
||
|
|
When a file is permanently deleted:
|
||
|
|
|
||
|
|
1. The blob's reference count is decremented
|
||
|
|
2. If the reference count reaches zero, the blob is removed from disk
|
||
|
|
|
||
|
|
This means disk space is only freed when the **last** reference to a blob is removed.
|
||
|
|
|
||
|
|
## Storage Layout
|
||
|
|
|
||
|
|
```
|
||
|
|
storage/
|
||
|
|
├── .blobs/
|
||
|
|
│ ├── a1/
|
||
|
|
│ │ └── a1b2c3d4...sha256.blob
|
||
|
|
│ ├── f8/
|
||
|
|
│ │ └── f8e7d6c5...sha256.blob
|
||
|
|
│ └── ...
|
||
|
|
```
|
||
|
|
|
||
|
|
The first two hex characters of the hash are used as a directory prefix to avoid having millions of files in a single directory.
|
||
|
|
|
||
|
|
## Benefits
|
||
|
|
|
||
|
|
- **Disk savings** — identical files across users consume storage only once
|
||
|
|
- **Instant uploads** — if the blob already exists, the upload completes immediately
|
||
|
|
- **Integrity** — SHA-256 ensures bit-for-bit correctness
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
- Deduplication is based on exact content match (byte-identical files)
|
||
|
|
- Near-duplicate files (e.g., a JPEG re-saved at slightly different quality) are stored separately
|
||
|
|
- Encryption at rest would require per-user keys, which breaks deduplication (planned as opt-in)
|