Files

T

DioCrafts 5722481c4a feat(thumbnails): server-side video thumbnails via ffmpeg

Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).

- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
  (infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
  extract one representative frame as PNG, bounded by its own semaphore + a
  per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
  videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
  generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
  to a size- and time-bounded temp file on the data volume, extract a frame, and
  reuse the shared render_and_persist_all_webp helper — so video thumbnails are
  WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
  Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
  at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
  path; the server is now the source of truth.

Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.

Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-21 23:23:04 +02:00

4.4 KiB

Raw Permalink Blame History

Video thumbnails — server-side ffmpeg (Option B)

Video thumbnails are now generated server-side on upload: a lifecycle hook streams the (decrypted) blob to a temp file, ffmpeg extracts one representative frame, and that frame goes through the same WebP pipeline as photos — so video thumbnails are WebP, blob-hash keyed (dedup'd), and content-negotiated, exactly like images.

This replaces the old browser path, which only generated a thumbnail when the Photos grid first rendered a video tile, the <img> 404'd, and the browser re-downloaded the video to seek a frame and PUT 3 JPEGs back.

What it buys

Coverage incl. HEVC/iPhone. The browser <video> element cannot decode HEVC/H.265 (.mov from iPhones), ProRes, many mkv/avi — so the old path produced no thumbnail for them. ffmpeg decodes all of them.
No client re-download. The frame is taken server-side from the blob the server already has — the browser never pulls the video back.
Eager. Thumbnails are ready before the gallery asks; tiles paint immediately instead of waiting on a failed <img> + re-download cascade.

Reproduce

cargo run --release --features bench --example bench_video_thumbnails

Requires ffmpeg on PATH (with libx264/libx265/libvpx-vp9 to synthesize the test corpus — written to benches/corpus/, git-ignored). The corpus is a testsrc pattern at several codecs/resolutions, incl. an HEVC .mov.

Results (14 cores, ffmpeg 8.1)

case	video KB	extract ms	frame KB	icon	preview	large (WebP B)	ok
h264 720p	46.9	49.2	44.1	1818	3836	6844	✅
h264 1080p	76.1	64.9	37.9	1844	3888	6870	✅
HEVC 1080p .mov	38.6	69.5	51.9	1858	3928	6976	✅
VP9 720p .webm	187.8	67.8	14.6	1824	3814	6650	✅

Coverage: 4/4 codecs, including HEVC/.mov — the browser path produced 0 for HEVC. Going from "no thumbnail" to "a thumbnail" is the real headline for iPhone footage.
Extraction: ~50–70 ms/frame for 720p–1080p. Paid once, in a background task, per unique blob — never on the request path. Bounded by a per-process timeout and a dedicated concurrency semaphore.
Served bytes per tile: ~3.8 KB (preview WebP) — same compact WebP as photos.

Transfer per first view (the bandwidth win)

OLD (browser re-downloads the video, worst case): 349 KB for 4 tiles  +  3 JPEG PUTs/video
NEW (fetch the server WebP preview):               15.1 KB             +  0 client decode
→ ~23× less data on this corpus.

⚠️ The test clips are tiny (3 s testsrc, 38–188 KB), which understates the win enormously. Real phone videos are 10–100+ MB; the old path re-downloaded a large fraction of that per first view, vs ~4 KB now — i.e. thousands-fold less for a 50 MB clip, plus it works for HEVC at all.

How it's wired

application/ports/video_frame_ports.rs — VideoFramePort (extract one PNG frame from a video file).
infrastructure/services/ffmpeg_video_frame_service.rs — shells out to the system ffmpeg (no compile-time libav dep); NoopVideoFrameService when ffmpeg is absent/disabled, so videos degrade to "no thumbnail" gracefully.
ThumbnailRefreshHook::on_file_created routes video/* to generate_video_thumbnails_background, which streams the blob to a temp file (capped, decrypting), extracts a frame, and reuses render_and_persist_all_webp.
GET /api/files/{id}/thumbnail/{size} serves the blob-hash WebP for videos too; a miss returns 204 (generation in flight / unavailable).

Config

Env	Default	Meaning
`OXICLOUD_ENABLE_VIDEO_THUMBNAILS`	`true`	Master switch (also needs ffmpeg present).
`OXICLOUD_FFMPEG_PATH`	`ffmpeg`	Path to the ffmpeg binary.
`OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY`	`cpus/2`	Max concurrent ffmpeg processes.
`OXICLOUD_VIDEO_THUMBNAIL_TIMEOUT_SECS`	`30`	Per-extraction wall-clock cap.
`OXICLOUD_VIDEO_THUMBNAIL_MAX_MB`	`2048`	Skip videos larger than this (no temp materialise).

The Docker runtime image installs ffmpeg. Existing videos uploaded before this change get a thumbnail the next time their blob is (re)created; a backfill task is a possible follow-up.

4.4 KiB Raw Permalink Blame History Unescape Escape