DioCrafts
5722481c4a
Videos now get a thumbnail generated eagerly server-side on upload, through the same WebP/blob-hash pipeline as photos — instead of the old browser path that only ran when the Photos grid first rendered a video tile, re-downloaded the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at all for HEVC/.mov, which a browser <video> cannot decode). - New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService (infrastructure): shell out to the system ffmpeg (no compile-time libav dep), extract one representative frame as PNG, bounded by its own semaphore + a per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so videos degrade gracefully to no thumbnail. - ThumbnailRefreshHook.on_file_created routes video/* to generate_video_thumbnails_background: stream the (decrypted, reassembled) blob to a size- and time-bounded temp file on the data volume, extract a frame, and reuse the shared render_and_persist_all_webp helper — so video thumbnails are WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos. - GET thumbnail serves the video's WebP to every client (byte-sniffed Content-Type); a genuine miss returns 204. - Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB. - Dockerfile installs ffmpeg in the runtime image. - Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download path; the server is now the source of truth. Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl. HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on the test corpus (far more on real multi-MB clips). Methodology in benches/VIDEO-THUMB.md. Hardening from an adversarial review: video render holds the decode_semaphore like the image path; the ffmpeg scale filter bounds both dimensions; the blob stream has a timeout; the temp file lives on the data volume; the size cap uses saturating_mul. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
89 lines
4.4 KiB
Markdown
89 lines
4.4 KiB
Markdown
# Video thumbnails — server-side ffmpeg (Option B)
|
||
|
||
Video thumbnails are now generated **server-side on upload**: a lifecycle hook
|
||
streams the (decrypted) blob to a temp file, `ffmpeg` extracts one representative
|
||
frame, and that frame goes through the **same WebP pipeline as photos** — so
|
||
video thumbnails are WebP, blob-hash keyed (dedup'd), and content-negotiated,
|
||
exactly like images.
|
||
|
||
This replaces the old browser path, which only generated a thumbnail when the
|
||
Photos grid first rendered a video tile, the `<img>` 404'd, and the browser
|
||
**re-downloaded the video** to seek a frame and PUT 3 JPEGs back.
|
||
|
||
## What it buys
|
||
|
||
1. **Coverage incl. HEVC/iPhone.** The browser `<video>` element cannot decode
|
||
HEVC/H.265 (`.mov` from iPhones), ProRes, many mkv/avi — so the old path
|
||
produced **no** thumbnail for them. ffmpeg decodes all of them.
|
||
2. **No client re-download.** The frame is taken server-side from the blob the
|
||
server already has — the browser never pulls the video back.
|
||
3. **Eager.** Thumbnails are ready before the gallery asks; tiles paint
|
||
immediately instead of waiting on a failed `<img>` + re-download cascade.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
cargo run --release --features bench --example bench_video_thumbnails
|
||
```
|
||
|
||
Requires `ffmpeg` on PATH (with libx264/libx265/libvpx-vp9 to synthesize the
|
||
test corpus — written to `benches/corpus/`, git-ignored). The corpus is a
|
||
`testsrc` pattern at several codecs/resolutions, incl. an HEVC `.mov`.
|
||
|
||
## Results (14 cores, ffmpeg 8.1)
|
||
|
||
| case | video KB | extract ms | frame KB | icon | preview | large (WebP B) | ok |
|
||
|-----------------|---------:|-----------:|---------:|-----:|--------:|---------------:|----|
|
||
| h264 720p | 46.9 | 49.2 | 44.1 | 1818 | 3836 | 6844 | ✅ |
|
||
| h264 1080p | 76.1 | 64.9 | 37.9 | 1844 | 3888 | 6870 | ✅ |
|
||
| HEVC 1080p .mov | 38.6 | 69.5 | 51.9 | 1858 | 3928 | 6976 | ✅ |
|
||
| VP9 720p .webm | 187.8 | 67.8 | 14.6 | 1824 | 3814 | 6650 | ✅ |
|
||
|
||
- **Coverage: 4/4 codecs, including HEVC/.mov** — the browser path produced 0 for
|
||
HEVC. Going from "no thumbnail" to "a thumbnail" is the real headline for
|
||
iPhone footage.
|
||
- **Extraction: ~50–70 ms/frame** for 720p–1080p. Paid once, in a background task,
|
||
per unique blob — never on the request path. Bounded by a per-process timeout
|
||
and a dedicated concurrency semaphore.
|
||
- **Served bytes per tile: ~3.8 KB (preview WebP)** — same compact WebP as photos.
|
||
|
||
### Transfer per first view (the bandwidth win)
|
||
|
||
```
|
||
OLD (browser re-downloads the video, worst case): 349 KB for 4 tiles + 3 JPEG PUTs/video
|
||
NEW (fetch the server WebP preview): 15.1 KB + 0 client decode
|
||
→ ~23× less data on this corpus.
|
||
```
|
||
|
||
> ⚠️ The test clips are tiny (3 s `testsrc`, 38–188 KB), which **understates** the
|
||
> win enormously. Real phone videos are 10–100+ MB; the old path re-downloaded a
|
||
> large fraction of that per first view, vs ~4 KB now — i.e. thousands-fold less
|
||
> for a 50 MB clip, plus it works for HEVC at all.
|
||
|
||
## How it's wired
|
||
|
||
- `application/ports/video_frame_ports.rs` — `VideoFramePort` (extract one PNG
|
||
frame from a video file).
|
||
- `infrastructure/services/ffmpeg_video_frame_service.rs` — shells out to the
|
||
system `ffmpeg` (no compile-time libav dep); `NoopVideoFrameService` when
|
||
ffmpeg is absent/disabled, so videos degrade to "no thumbnail" gracefully.
|
||
- `ThumbnailRefreshHook::on_file_created` routes `video/*` to
|
||
`generate_video_thumbnails_background`, which streams the blob to a temp file
|
||
(capped, decrypting), extracts a frame, and reuses `render_and_persist_all_webp`.
|
||
- GET `/api/files/{id}/thumbnail/{size}` serves the blob-hash WebP for videos
|
||
too; a miss returns 204 (generation in flight / unavailable).
|
||
|
||
## Config
|
||
|
||
| Env | Default | Meaning |
|
||
|---|---|---|
|
||
| `OXICLOUD_ENABLE_VIDEO_THUMBNAILS` | `true` | Master switch (also needs ffmpeg present). |
|
||
| `OXICLOUD_FFMPEG_PATH` | `ffmpeg` | Path to the ffmpeg binary. |
|
||
| `OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY` | `cpus/2` | Max concurrent ffmpeg processes. |
|
||
| `OXICLOUD_VIDEO_THUMBNAIL_TIMEOUT_SECS` | `30` | Per-extraction wall-clock cap. |
|
||
| `OXICLOUD_VIDEO_THUMBNAIL_MAX_MB` | `2048` | Skip videos larger than this (no temp materialise). |
|
||
|
||
The Docker runtime image installs `ffmpeg`. Existing videos uploaded before this
|
||
change get a thumbnail the next time their blob is (re)created; a backfill task
|
||
is a possible follow-up.
|