feat: thumbnail dedup — store thumbnails by blob_hash instead of file_id (#233)

Thumbnails are now keyed by blob_hash on disk so identical files share
a single set of thumbnails (icon/preview/large). For 4000 duplicate
files with the same content, this reduces thumbnail storage from 12,000
files to just 3.

Changes:
- get_thumbnail_path() keys by blob_hash instead of file_id
- get_thumbnail(), get_cached_thumbnail(), generate_all_sizes_background()
  accept blob_hash parameter for disk dedup
- generate_all_sizes_background() fast path: if blob-hash thumbnails
  already exist on disk, skip image processing entirely and just
  populate moka cache for the new file_id
- delete_thumbnails() only invalidates moka cache (shared disk
  thumbnails must not be deleted when one file is removed)
- delete_blob_thumbnails() added for GC; garbage_collect() now cleans
  up orphaned thumbnail files alongside blob files
- External thumbnails (video frames) stored as ext-{file_id}.jpg
  since they are client-generated and not dedup-able
- ThumbnailPort trait updated with blob_hash parameters
- All handler call sites updated (file_handler, preview_handler)
- Tests updated for new signatures
This commit is contained in:
Diocrafts
2026-04-12 00:50:10 +02:00
parent e91dcee0ea
commit 2dde4da5cf
6 changed files with 164 additions and 41 deletions
+20 -6
View File
@@ -68,18 +68,27 @@ pub trait ThumbnailPort: Send + Sync + 'static {
/// Get a thumbnail, generating it on-demand if needed.
///
/// Returns the thumbnail bytes in WebP format.
/// `blob_hash` is the content hash used as the disk storage key
/// (dedup: identical blobs share one set of thumbnails).
async fn get_thumbnail(
&self,
file_id: &str,
blob_hash: &str,
size: ThumbnailSize,
original_path: &Path,
) -> Result<Bytes, DomainError>;
/// Generate all thumbnail sizes for a file in the background.
///
/// Called after file upload to pre-generate thumbnails.
fn generate_all_sizes_background(self: Arc<Self>, file_id: String, original_path: PathBuf);
/// `blob_hash` is the content hash used as the disk storage key.
/// If thumbnails already exist for this hash, only the moka cache
/// is populated (zero CPU for image processing).
fn generate_all_sizes_background(
self: Arc<Self>,
file_id: String,
blob_hash: String,
original_path: PathBuf,
);
/// Delete all thumbnails for a file.
async fn delete_thumbnails(&self, file_id: &str) -> Result<(), DomainError>;
@@ -87,9 +96,14 @@ pub trait ThumbnailPort: Send + Sync + 'static {
/// Try to get a cached thumbnail without generating one.
///
/// Returns `None` if no cached thumbnail exists on disk or in memory.
/// Used for non-image file types (videos) where thumbnails are
/// generated client-side and uploaded.
async fn get_cached_thumbnail(&self, file_id: &str, size: ThumbnailSize) -> Option<Bytes>;
/// `blob_hash` is used to locate the file on disk. If `None`, only
/// the in-memory moka cache is checked.
async fn get_cached_thumbnail(
&self,
file_id: &str,
blob_hash: Option<&str>,
size: ThumbnailSize,
) -> Option<Bytes>;
/// Store an externally-generated thumbnail (e.g. client-side video frame).
///