feat(thumbnails): also store derived thumbnails as blobs

Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.

That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.

DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:

  * writes the bytes through the normal CDC path, so derived blobs
    inherit the backend, encryption, migration and key rotation that
    source content already gets;
  * records (source_hash, kind, variant) -> blob_hash;
  * releases the reference store_from_stream took IF the mapping
    already existed. Two instances racing to render the same thumbnail
    must leave ref_count at 1, not 2 — otherwise every re-render
    inflates it and pins the blob forever.

ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.

The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.

Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.

fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-08-23 21:22:26 +02:00
parent a7938344dd
commit 1c488b7df5
2 changed files with 117 additions and 11 deletions
@@ -1115,7 +1115,7 @@ impl ThumbnailService {
}
};
self.render_and_persist_all_webp(&file_id, &blob_hash, original_data)
self.render_and_persist_all_webp(&file_id, &blob_hash, original_data, Some(&dedup))
.await;
tracing::info!("✅ Background thumbnail generation complete: {}", file_id);
@@ -1126,7 +1126,21 @@ impl ThumbnailService {
/// blob_hash (disk `{hash}.webp` + moka). Shared by the image upload path and
/// the video path (which passes the extracted frame as the source), so both
/// produce identical, dedup-able, content-negotiable thumbnails.
async fn render_and_persist_all_webp(&self, file_id: &str, blob_hash: &str, source: Bytes) {
/// `dedup` is `Some` on every path that has a handle, which is every
/// eager background path. When present each rendered size is ALSO stored
/// as a derived blob and recorded in `storage.content_derived_blobs`.
///
/// The sidecar write is deliberately kept: this slice fills the table
/// while reads still come from disk, so a rollback at any point leaves
/// working thumbnails and the table can be inspected against real data
/// before anything depends on it. See `docs/plan/derived-blobs.md`.
async fn render_and_persist_all_webp(
&self,
file_id: &str,
blob_hash: &str,
source: Bytes,
dedup: Option<&DedupService>,
) {
let results = tokio::task::spawn_blocking(move || {
Self::render_all_thumbnails_from_data(source.as_ref(), ThumbnailFormat::Webp)
})
@@ -1151,15 +1165,39 @@ impl ThumbnailService {
}
if let Err(e) = fs::write(&thumb_path, &bytes).await {
tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e);
} else {
let cache_key = ThumbnailCacheKey {
file_id: file_id.to_string(),
size,
format: ThumbnailFormat::Webp,
};
self.cache.insert(cache_key, bytes).await;
tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size);
continue;
}
// Tier-3 copy. Best-effort and logged: a failure here must not
// cost the user their thumbnail, which is already on disk and in
// the cache. `derived_import` sweeps anything missed.
if let Some(dedup) = dedup
&& let Err(e) = dedup
.store_derived_blob(
blob_hash,
"thumbnail",
size.dir_name(),
"image/webp",
bytes.clone(),
)
.await
{
tracing::warn!(
target: "oxicloud::dedup",
error = %e,
"failed to record derived blob for {} {:?}",
file_id,
size,
);
}
let cache_key = ThumbnailCacheKey {
file_id: file_id.to_string(),
size,
format: ThumbnailFormat::Webp,
};
self.cache.insert(cache_key, bytes).await;
tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size);
}
}
@@ -1251,7 +1289,7 @@ impl ThumbnailService {
Ok(p) => p,
Err(_) => return,
};
self.render_and_persist_all_webp(&file_id, &blob_hash, frame)
self.render_and_persist_all_webp(&file_id, &blob_hash, frame, Some(&dedup))
.await;
tracing::info!("✅ Video thumbnail generation complete: {}", file_id);
});