feat(thumbnails): server-side video thumbnails via ffmpeg
Videos now get a thumbnail generated eagerly server-side on upload, through the same WebP/blob-hash pipeline as photos — instead of the old browser path that only ran when the Photos grid first rendered a video tile, re-downloaded the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at all for HEVC/.mov, which a browser <video> cannot decode). - New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService (infrastructure): shell out to the system ffmpeg (no compile-time libav dep), extract one representative frame as PNG, bounded by its own semaphore + a per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so videos degrade gracefully to no thumbnail. - ThumbnailRefreshHook.on_file_created routes video/* to generate_video_thumbnails_background: stream the (decrypted, reassembled) blob to a size- and time-bounded temp file on the data volume, extract a frame, and reuse the shared render_and_persist_all_webp helper — so video thumbnails are WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos. - GET thumbnail serves the video's WebP to every client (byte-sniffed Content-Type); a genuine miss returns 204. - Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB. - Dockerfile installs ffmpeg in the runtime image. - Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download path; the server is now the source of truth. Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl. HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on the test corpus (far more on real multi-MB clips). Methodology in benches/VIDEO-THUMB.md. Hardening from an adversarial review: video render holds the decode_semaphore like the image path; the ffmpeg scale filter bounds both dimensions; the blob stream has a timeout; the temp file lives on the data volume; the size cap uses saturating_mul. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
DioCrafts
@@ -155,6 +155,13 @@ name = "bench_thumbnails_mem"
|
||||
path = "examples/bench_thumbnails_mem.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Video thumbnail benchmark — Option B (server-side ffmpeg frame → WebP). Needs
|
||||
# `ffmpeg` on PATH (libx264/libx265/libvpx-vp9 to synthesize the test corpus).
|
||||
[[example]]
|
||||
name = "bench_video_thumbnails"
|
||||
path = "examples/bench_video_thumbnails.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# DB connection-pool tail-latency benchmark (needs the dev Postgres up).
|
||||
[[example]]
|
||||
name = "bench_db_pool"
|
||||
|
||||
+3
-1
@@ -67,9 +67,11 @@ LABEL org.opencontainers.image.title="OxiCloud" \
|
||||
|
||||
# Install only necessary runtime dependencies and update packages
|
||||
# su-exec is needed by the entrypoint to drop privileges after fixing volume permissions.
|
||||
# ffmpeg powers server-side video thumbnail extraction (one frame → WebP pipeline);
|
||||
# without it videos simply have no thumbnail (OXICLOUD_ENABLE_VIDEO_THUMBNAILS).
|
||||
# No libpq: the pure-Rust sqlx postgres driver never links it.
|
||||
RUN apk --no-cache upgrade && \
|
||||
apk add --no-cache libgcc ca-certificates tzdata su-exec && \
|
||||
apk add --no-cache libgcc ca-certificates tzdata su-exec ffmpeg && \
|
||||
addgroup -g 1001 -S oxicloud && \
|
||||
adduser -u 1001 -S oxicloud -G oxicloud
|
||||
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Video thumbnails — server-side ffmpeg (Option B)
|
||||
|
||||
Video thumbnails are now generated **server-side on upload**: a lifecycle hook
|
||||
streams the (decrypted) blob to a temp file, `ffmpeg` extracts one representative
|
||||
frame, and that frame goes through the **same WebP pipeline as photos** — so
|
||||
video thumbnails are WebP, blob-hash keyed (dedup'd), and content-negotiated,
|
||||
exactly like images.
|
||||
|
||||
This replaces the old browser path, which only generated a thumbnail when the
|
||||
Photos grid first rendered a video tile, the `<img>` 404'd, and the browser
|
||||
**re-downloaded the video** to seek a frame and PUT 3 JPEGs back.
|
||||
|
||||
## What it buys
|
||||
|
||||
1. **Coverage incl. HEVC/iPhone.** The browser `<video>` element cannot decode
|
||||
HEVC/H.265 (`.mov` from iPhones), ProRes, many mkv/avi — so the old path
|
||||
produced **no** thumbnail for them. ffmpeg decodes all of them.
|
||||
2. **No client re-download.** The frame is taken server-side from the blob the
|
||||
server already has — the browser never pulls the video back.
|
||||
3. **Eager.** Thumbnails are ready before the gallery asks; tiles paint
|
||||
immediately instead of waiting on a failed `<img>` + re-download cascade.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
cargo run --release --features bench --example bench_video_thumbnails
|
||||
```
|
||||
|
||||
Requires `ffmpeg` on PATH (with libx264/libx265/libvpx-vp9 to synthesize the
|
||||
test corpus — written to `benches/corpus/`, git-ignored). The corpus is a
|
||||
`testsrc` pattern at several codecs/resolutions, incl. an HEVC `.mov`.
|
||||
|
||||
## Results (14 cores, ffmpeg 8.1)
|
||||
|
||||
| case | video KB | extract ms | frame KB | icon | preview | large (WebP B) | ok |
|
||||
|-----------------|---------:|-----------:|---------:|-----:|--------:|---------------:|----|
|
||||
| h264 720p | 46.9 | 49.2 | 44.1 | 1818 | 3836 | 6844 | ✅ |
|
||||
| h264 1080p | 76.1 | 64.9 | 37.9 | 1844 | 3888 | 6870 | ✅ |
|
||||
| HEVC 1080p .mov | 38.6 | 69.5 | 51.9 | 1858 | 3928 | 6976 | ✅ |
|
||||
| VP9 720p .webm | 187.8 | 67.8 | 14.6 | 1824 | 3814 | 6650 | ✅ |
|
||||
|
||||
- **Coverage: 4/4 codecs, including HEVC/.mov** — the browser path produced 0 for
|
||||
HEVC. Going from "no thumbnail" to "a thumbnail" is the real headline for
|
||||
iPhone footage.
|
||||
- **Extraction: ~50–70 ms/frame** for 720p–1080p. Paid once, in a background task,
|
||||
per unique blob — never on the request path. Bounded by a per-process timeout
|
||||
and a dedicated concurrency semaphore.
|
||||
- **Served bytes per tile: ~3.8 KB (preview WebP)** — same compact WebP as photos.
|
||||
|
||||
### Transfer per first view (the bandwidth win)
|
||||
|
||||
```
|
||||
OLD (browser re-downloads the video, worst case): 349 KB for 4 tiles + 3 JPEG PUTs/video
|
||||
NEW (fetch the server WebP preview): 15.1 KB + 0 client decode
|
||||
→ ~23× less data on this corpus.
|
||||
```
|
||||
|
||||
> ⚠️ The test clips are tiny (3 s `testsrc`, 38–188 KB), which **understates** the
|
||||
> win enormously. Real phone videos are 10–100+ MB; the old path re-downloaded a
|
||||
> large fraction of that per first view, vs ~4 KB now — i.e. thousands-fold less
|
||||
> for a 50 MB clip, plus it works for HEVC at all.
|
||||
|
||||
## How it's wired
|
||||
|
||||
- `application/ports/video_frame_ports.rs` — `VideoFramePort` (extract one PNG
|
||||
frame from a video file).
|
||||
- `infrastructure/services/ffmpeg_video_frame_service.rs` — shells out to the
|
||||
system `ffmpeg` (no compile-time libav dep); `NoopVideoFrameService` when
|
||||
ffmpeg is absent/disabled, so videos degrade to "no thumbnail" gracefully.
|
||||
- `ThumbnailRefreshHook::on_file_created` routes `video/*` to
|
||||
`generate_video_thumbnails_background`, which streams the blob to a temp file
|
||||
(capped, decrypting), extracts a frame, and reuses `render_and_persist_all_webp`.
|
||||
- GET `/api/files/{id}/thumbnail/{size}` serves the blob-hash WebP for videos
|
||||
too; a miss returns 204 (generation in flight / unavailable).
|
||||
|
||||
## Config
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `OXICLOUD_ENABLE_VIDEO_THUMBNAILS` | `true` | Master switch (also needs ffmpeg present). |
|
||||
| `OXICLOUD_FFMPEG_PATH` | `ffmpeg` | Path to the ffmpeg binary. |
|
||||
| `OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY` | `cpus/2` | Max concurrent ffmpeg processes. |
|
||||
| `OXICLOUD_VIDEO_THUMBNAIL_TIMEOUT_SECS` | `30` | Per-extraction wall-clock cap. |
|
||||
| `OXICLOUD_VIDEO_THUMBNAIL_MAX_MB` | `2048` | Skip videos larger than this (no temp materialise). |
|
||||
|
||||
The Docker runtime image installs `ffmpeg`. Existing videos uploaded before this
|
||||
change get a thumbnail the next time their blob is (re)created; a backfill task
|
||||
is a possible follow-up.
|
||||
@@ -0,0 +1,232 @@
|
||||
//! Video thumbnail benchmark — Option B (server-side ffmpeg frame → WebP).
|
||||
//!
|
||||
//! Measures the "after" of moving video thumbnail generation off the browser
|
||||
//! and onto the server:
|
||||
//! * extraction time per codec/resolution (the new server cost),
|
||||
//! * the WebP thumbnail bytes actually served per tile (the new transfer),
|
||||
//! * codec coverage incl. HEVC/MOV — the iPhone case a browser `<video>`
|
||||
//! cannot decode, so the old client-side path produced *no* thumbnail.
|
||||
//!
|
||||
//! vs the OLD client-side path, whose first view of each video tile re-downloaded
|
||||
//! the video from the server (metadata + byte-ranges, up to the whole file) and
|
||||
//! PUT 3 JPEGs back.
|
||||
//!
|
||||
//! Requires `ffmpeg` on PATH (with libx264/libx265/libvpx-vp9 to generate the
|
||||
//! corpus). Run: `cargo run --release --features bench --example bench_video_thumbnails`.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use oxicloud::application::ports::thumbnail_ports::ThumbnailFormat;
|
||||
use oxicloud::application::ports::video_frame_ports::VideoFramePort;
|
||||
use oxicloud::infrastructure::services::ffmpeg_video_frame_service::FfmpegVideoFrameService;
|
||||
use oxicloud::infrastructure::services::thumbnail_service::{ThumbnailService, ThumbnailSize};
|
||||
|
||||
/// One synthetic test video: a label, output filename, and the ffmpeg encode
|
||||
/// args (a `testsrc` pattern keeps it license-free and deterministic enough).
|
||||
struct VideoSpec {
|
||||
name: &'static str,
|
||||
filename: &'static str,
|
||||
encode_args: &'static [&'static str],
|
||||
}
|
||||
|
||||
const SPECS: &[VideoSpec] = &[
|
||||
VideoSpec {
|
||||
name: "h264 720p",
|
||||
filename: "video_h264_720p.mp4",
|
||||
encode_args: &[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc=duration=3:size=1280x720:rate=30",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
},
|
||||
VideoSpec {
|
||||
name: "h264 1080p",
|
||||
filename: "video_h264_1080p.mp4",
|
||||
encode_args: &[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc=duration=3:size=1920x1080:rate=30",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
},
|
||||
VideoSpec {
|
||||
// The iPhone case: HEVC/H.265 in a QuickTime .mov — undecodable by a
|
||||
// browser <video>, so the old client path produced nothing for these.
|
||||
name: "HEVC 1080p .mov",
|
||||
filename: "video_hevc_1080p.mov",
|
||||
encode_args: &[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc=duration=3:size=1920x1080:rate=30",
|
||||
"-c:v",
|
||||
"libx265",
|
||||
"-tag:v",
|
||||
"hvc1",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
},
|
||||
VideoSpec {
|
||||
name: "VP9 720p .webm",
|
||||
filename: "video_vp9_720p.webm",
|
||||
encode_args: &[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc=duration=3:size=1280x720:rate=30",
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-b:v",
|
||||
"1M",
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
fn corpus_dir() -> PathBuf {
|
||||
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("benches")
|
||||
.join("corpus")
|
||||
}
|
||||
|
||||
/// Generate a test video with ffmpeg if it isn't already on disk.
|
||||
fn ensure_video(ffmpeg: &str, spec: &VideoSpec, path: &Path) {
|
||||
if path.exists() {
|
||||
return;
|
||||
}
|
||||
let mut cmd = std::process::Command::new(ffmpeg);
|
||||
cmd.arg("-y")
|
||||
.arg("-hide_banner")
|
||||
.arg("-loglevel")
|
||||
.arg("error");
|
||||
cmd.args(spec.encode_args);
|
||||
cmd.arg(path);
|
||||
match cmd.status() {
|
||||
Ok(s) if s.success() => {}
|
||||
Ok(s) => eprintln!("ffmpeg gen {} exited {s}", spec.name),
|
||||
Err(e) => eprintln!("ffmpeg gen {} failed: {e}", spec.name),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let ffmpeg = std::env::var("OXICLOUD_FFMPEG_PATH").unwrap_or_else(|_| "ffmpeg".to_string());
|
||||
if !FfmpegVideoFrameService::is_available(&ffmpeg) {
|
||||
eprintln!("ffmpeg not found (set OXICLOUD_FFMPEG_PATH) — cannot run video bench");
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
let dir = corpus_dir();
|
||||
let _ = std::fs::create_dir_all(&dir);
|
||||
for spec in SPECS {
|
||||
ensure_video(&ffmpeg, spec, &dir.join(spec.filename));
|
||||
}
|
||||
|
||||
let svc = FfmpegVideoFrameService::new(ffmpeg.clone(), 4, Duration::from_secs(60));
|
||||
|
||||
println!("== Video thumbnails: server-side ffmpeg frame → WebP pipeline (Option B) ==");
|
||||
println!(
|
||||
"| {:<16} | {:>9} | {:>10} | {:>9} | {:>6} | {:>8} | {:>7} | {:<4} |",
|
||||
"case", "video KB", "extract ms", "frame KB", "icon B", "prev B", "large B", "ok"
|
||||
);
|
||||
println!(
|
||||
"|{:-<18}|{:-<11}|{:-<12}|{:-<11}|{:-<8}|{:-<10}|{:-<9}|{:-<6}|",
|
||||
"", "", "", "", "", "", "", ""
|
||||
);
|
||||
|
||||
let mut old_transfer_kb = 0f64; // re-download the video on first view (worst case)
|
||||
let mut new_transfer_b = 0u64; // fetch the preview WebP thumbnail
|
||||
let mut covered = 0usize;
|
||||
|
||||
for spec in SPECS {
|
||||
let path = dir.join(spec.filename);
|
||||
let video_kb = std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0) as f64 / 1024.0;
|
||||
|
||||
// Best-of-3 extraction time.
|
||||
let mut best = Duration::MAX;
|
||||
let mut frame = bytes::Bytes::new();
|
||||
let mut ok = true;
|
||||
for _ in 0..3 {
|
||||
let t = Instant::now();
|
||||
match svc.extract_frame(&path).await {
|
||||
Ok(f) => {
|
||||
best = best.min(t.elapsed());
|
||||
frame = f;
|
||||
}
|
||||
Err(e) => {
|
||||
println!("| {:<16} | extract failed: {e}", spec.name);
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok || frame.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let thumbs = match ThumbnailService::bench_render_all_fmt(&frame, ThumbnailFormat::Webp) {
|
||||
Ok(t) => t,
|
||||
Err(e) => {
|
||||
println!("| {:<16} | webp render failed: {e}", spec.name);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let sz = |want: ThumbnailSize| {
|
||||
thumbs
|
||||
.iter()
|
||||
.find(|(s, _)| *s == want)
|
||||
.map(|(_, b)| *b)
|
||||
.unwrap_or(0)
|
||||
};
|
||||
let (icon, preview, large) = (
|
||||
sz(ThumbnailSize::Icon),
|
||||
sz(ThumbnailSize::Preview),
|
||||
sz(ThumbnailSize::Large),
|
||||
);
|
||||
|
||||
covered += 1;
|
||||
old_transfer_kb += video_kb;
|
||||
new_transfer_b += preview as u64;
|
||||
|
||||
println!(
|
||||
"| {:<16} | {:>9.1} | {:>10.1} | {:>9.1} | {:>6} | {:>8} | {:>7} | {:<4} |",
|
||||
spec.name,
|
||||
video_kb,
|
||||
best.as_secs_f64() * 1000.0,
|
||||
frame.len() as f64 / 1024.0,
|
||||
icon,
|
||||
preview,
|
||||
large,
|
||||
"yes"
|
||||
);
|
||||
}
|
||||
|
||||
println!(
|
||||
"\n Coverage: {}/{} codecs produced a thumbnail server-side (incl. HEVC/.mov — the \
|
||||
browser <video> path produced 0 for HEVC).",
|
||||
covered,
|
||||
SPECS.len()
|
||||
);
|
||||
let new_kb = new_transfer_b as f64 / 1024.0;
|
||||
println!(
|
||||
" Per-first-view transfer to show {} video tiles:\n OLD (client re-downloads the video, worst case): {:.0} KB + 3 JPEG PUTs/video\n NEW (fetch the server WebP preview): {:.1} KB + 0 client decode\n → up to {:.0}× less data, and it is eager (ready before the gallery asks).",
|
||||
covered,
|
||||
old_transfer_kb,
|
||||
new_kb,
|
||||
if new_kb > 0.0 {
|
||||
old_transfer_kb / new_kb
|
||||
} else {
|
||||
0.0
|
||||
}
|
||||
);
|
||||
}
|
||||
@@ -114,21 +114,3 @@ export async function batchTrash(fileIds: string[]): Promise<Set<string>> {
|
||||
}
|
||||
return trashed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload a generated thumbnail blob for a file at a given size. Used by the
|
||||
* photos grid to persist client-generated video frames server-side.
|
||||
*/
|
||||
export async function uploadThumbnail(
|
||||
fileId: string,
|
||||
size: 'icon' | 'preview' | 'large',
|
||||
blob: Blob,
|
||||
contentType = 'image/jpeg'
|
||||
): Promise<void> {
|
||||
await apiFetch(`/api/files/${fileId}/thumbnail/${size}`, {
|
||||
method: 'PUT',
|
||||
credentials: 'same-origin',
|
||||
headers: { ...getCsrfHeaders(), 'Content-Type': contentType },
|
||||
body: blob
|
||||
});
|
||||
}
|
||||
|
||||
@@ -6,12 +6,7 @@
|
||||
import { useSelection } from '$lib/composables/useSelection.svelte';
|
||||
import { errorToast } from '$lib/utils/errors';
|
||||
import { onMount } from 'svelte';
|
||||
import {
|
||||
batchTrash,
|
||||
fetchPhotos,
|
||||
uploadThumbnail,
|
||||
type PhotoItem
|
||||
} from '$lib/api/endpoints/photos';
|
||||
import { batchTrash, fetchPhotos, type PhotoItem } from '$lib/api/endpoints/photos';
|
||||
import { peopleEnabled } from '$lib/api/endpoints/people';
|
||||
import { fileDownloadUrl, fileThumbnailUrl } from '$lib/api/endpoints/files';
|
||||
import Icon from '$lib/icons/Icon.svelte';
|
||||
@@ -55,9 +50,6 @@
|
||||
else if (tab === 'people') void peopleView.load();
|
||||
});
|
||||
|
||||
/** Client-generated video frame thumbnails (file id → data/URL). */
|
||||
let videoThumbs = $state<Record<string, string>>({});
|
||||
|
||||
/** EXIF-aware timestamp (seconds → ms), matching the OLD grouping logic. */
|
||||
function bucketKey(d: Date): string {
|
||||
const y = d.getFullYear();
|
||||
@@ -290,95 +282,17 @@
|
||||
}
|
||||
|
||||
/**
|
||||
* A tile thumbnail failed to load (no server thumbnail — e.g. SVGs, which the
|
||||
* backend can't rasterise — or a transient error). Hide the broken <img> so
|
||||
* the always-present placeholder shows through, and for videos kick off
|
||||
* client-side frame extraction (which, on success, re-renders the tile from
|
||||
* `videoThumbs`).
|
||||
* A tile thumbnail failed to load — no server thumbnail yet (SVGs the backend
|
||||
* can't rasterise; a video whose server-side frame extraction is still running
|
||||
* or unavailable) or a transient error. Hide the broken <img> so the
|
||||
* always-present placeholder (and, for videos, the play badge) shows through.
|
||||
*
|
||||
* Video thumbnails are now produced server-side (ffmpeg) on upload through the
|
||||
* same WebP pipeline as photos — the browser no longer re-downloads the video
|
||||
* to extract a frame.
|
||||
*/
|
||||
function onThumbError(e: Event, photo: PhotoItem) {
|
||||
function onThumbError(e: Event) {
|
||||
(e.currentTarget as HTMLImageElement).style.display = 'none';
|
||||
if (isVideo(photo)) void generateVideoThumb(photo);
|
||||
}
|
||||
|
||||
// ── Client-side video thumbnail generation ──────────────────────────────
|
||||
// When the server has no thumbnail for a video tile the <img> errors; we
|
||||
// then extract a frame with the browser's native decoder and upload it.
|
||||
|
||||
async function generateVideoThumb(file: PhotoItem) {
|
||||
if (videoThumbs[file.id]) return;
|
||||
try {
|
||||
const bitmap = await frameFromVideo(`/api/files/${file.id}?inline=true`);
|
||||
const SIZES: Array<['icon' | 'preview' | 'large', number, number]> = [
|
||||
['icon', 150, 150],
|
||||
['preview', 400, 400],
|
||||
['large', 800, 800]
|
||||
];
|
||||
let previewData = '';
|
||||
// Render the blobs and push all three sizes in parallel; `previewData`
|
||||
// is captured before its upload so the local preview shows even if that
|
||||
// upload fails (allSettled swallows per-size failures, as before).
|
||||
await Promise.allSettled(
|
||||
SIZES.map(async ([size, w, h]) => {
|
||||
const blob = await bitmapToBlob(bitmap, w, h);
|
||||
if (size === 'preview') previewData = await blobToDataUrl(blob);
|
||||
await uploadThumbnail(file.id, size, blob);
|
||||
})
|
||||
);
|
||||
if (previewData) videoThumbs = { ...videoThumbs, [file.id]: previewData };
|
||||
} catch {
|
||||
// Keep the generic play badge on failure.
|
||||
}
|
||||
}
|
||||
|
||||
function frameFromVideo(src: string): Promise<ImageBitmap> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const video = document.createElement('video');
|
||||
video.src = src;
|
||||
video.muted = true;
|
||||
video.preload = 'metadata';
|
||||
video.onloadedmetadata = () => {
|
||||
video.currentTime = (video.duration || 3) / 3;
|
||||
};
|
||||
video.onseeked = async () => {
|
||||
try {
|
||||
const bitmap = await createImageBitmap(video);
|
||||
video.removeAttribute('src');
|
||||
video.load();
|
||||
resolve(bitmap);
|
||||
} catch (e) {
|
||||
reject(e instanceof Error ? e : new Error(String(e)));
|
||||
}
|
||||
};
|
||||
video.onerror = () => reject(new Error('video frame extraction failed'));
|
||||
});
|
||||
}
|
||||
|
||||
async function bitmapToBlob(bitmap: ImageBitmap, tw: number, th: number): Promise<Blob> {
|
||||
const ratio = bitmap.width / bitmap.height;
|
||||
const target = tw / th;
|
||||
const w = ratio > target ? tw : Math.round(th * ratio);
|
||||
const h = ratio > target ? Math.round(tw / ratio) : th;
|
||||
const canvas = document.createElement('canvas');
|
||||
canvas.width = w;
|
||||
canvas.height = h;
|
||||
canvas.getContext('2d')?.drawImage(bitmap, 0, 0, w, h);
|
||||
return new Promise<Blob>((resolve, reject) => {
|
||||
canvas.toBlob(
|
||||
(b) => (b ? resolve(b) : reject(new Error('canvas toBlob failed'))),
|
||||
'image/jpeg',
|
||||
0.8
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
function blobToDataUrl(blob: Blob): Promise<string> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => resolve(String(reader.result));
|
||||
reader.onerror = () => reject(new Error('blob read failed'));
|
||||
reader.readAsDataURL(blob);
|
||||
});
|
||||
}
|
||||
|
||||
onMount(() => {
|
||||
@@ -540,9 +454,6 @@
|
||||
it can't load (no server thumbnail, e.g. SVG), hides itself to reveal
|
||||
this default rather than the browser's broken-image glyph. -->
|
||||
<span class="photo-tile__placeholder" aria-hidden="true"><Icon name="file-image" /></span>
|
||||
{#if videoThumbs[photo.id]}
|
||||
<img src={videoThumbs[photo.id]} alt={photo.name} loading="lazy" decoding="async" />
|
||||
{:else}
|
||||
<img
|
||||
src={fileThumbnailUrl(photo.id, 'preview')}
|
||||
srcset={`${fileThumbnailUrl(photo.id, 'icon')} 150w, ${fileThumbnailUrl(photo.id, 'preview')} 400w, ${fileThumbnailUrl(photo.id, 'large')} 800w`}
|
||||
@@ -550,9 +461,8 @@
|
||||
alt={photo.name}
|
||||
loading="lazy"
|
||||
decoding="async"
|
||||
onerror={(e) => onThumbError(e, photo)}
|
||||
onerror={onThumbError}
|
||||
/>
|
||||
{/if}
|
||||
{#if isVideo(photo)}
|
||||
<span class="photo-tile__video-badge" aria-hidden="true"><Icon name="play" /></span>
|
||||
{/if}
|
||||
|
||||
@@ -26,4 +26,5 @@ pub mod thumbnail_ports;
|
||||
pub mod transcode_ports;
|
||||
pub mod trash_ports;
|
||||
pub mod user_lifecycle;
|
||||
pub mod video_frame_ports;
|
||||
pub mod zip_ports;
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
//! Video frame extraction port.
|
||||
//!
|
||||
//! Pulls a single representative still frame out of a video so the existing
|
||||
//! image thumbnail pipeline (shrink-on-load → SIMD resize → WebP → blob-hash
|
||||
//! storage → HTTP content negotiation) can treat videos exactly like photos —
|
||||
//! eagerly, server-side, on upload. Keeping this behind a port lets the
|
||||
//! composition root swap in a no-op when `ffmpeg` is absent or the feature is
|
||||
//! off, so video uploads degrade gracefully (no thumbnail) instead of erroring.
|
||||
|
||||
use crate::common::errors::DomainError;
|
||||
use async_trait::async_trait;
|
||||
use bytes::Bytes;
|
||||
use std::path::Path;
|
||||
|
||||
/// Extracts a representative still frame from a video file.
|
||||
///
|
||||
/// Implementations:
|
||||
/// - `FfmpegVideoFrameService` — shells out to the system `ffmpeg`, covering
|
||||
/// every container/codec (incl. HEVC/MOV, which a browser `<video>` cannot
|
||||
/// decode). Bounds its own process concurrency and per-call timeout.
|
||||
/// - `NoopVideoFrameService` — registered when `ffmpeg` is unavailable or the
|
||||
/// feature is disabled; `is_supported_video` returns false so the lifecycle
|
||||
/// hook never attempts video thumbnails.
|
||||
#[async_trait]
|
||||
pub trait VideoFramePort: Send + Sync + 'static {
|
||||
/// Whether `mime_type` is a video this extractor will attempt to thumbnail.
|
||||
/// The no-op implementation always returns false.
|
||||
fn is_supported_video(&self, mime_type: &str) -> bool;
|
||||
|
||||
/// Extract one representative frame from the video file at `path`, returning
|
||||
/// encoded **PNG** bytes ready to feed into the image thumbnail renderer.
|
||||
/// `path` must point at the decoded (decrypted, reassembled) video on disk.
|
||||
async fn extract_frame(&self, path: &Path) -> Result<Bytes, DomainError>;
|
||||
}
|
||||
@@ -901,6 +901,11 @@ pub struct FeaturesConfig {
|
||||
/// Expose other OxiCloud users as a read-only "system" address book
|
||||
/// at GET /api/address-books. Set to false to hide the user directory.
|
||||
pub expose_system_users: bool,
|
||||
/// Generate video thumbnails server-side via `ffmpeg` on upload. When true
|
||||
/// (and ffmpeg is detected at startup) videos get a representative-frame
|
||||
/// thumbnail through the same WebP pipeline as photos; otherwise videos have
|
||||
/// no thumbnail. Env: `OXICLOUD_ENABLE_VIDEO_THUMBNAILS`.
|
||||
pub enable_video_thumbnails: bool,
|
||||
}
|
||||
|
||||
impl Default for FeaturesConfig {
|
||||
@@ -915,6 +920,7 @@ impl Default for FeaturesConfig {
|
||||
enable_places: true, // Photo map (GET /api/photos/geo + Places tab)
|
||||
enable_faces: false, // People/faces (biometric) — opt-in, off by default
|
||||
expose_system_users: true, // Expose OxiCloud users as address book by default
|
||||
enable_video_thumbnails: true, // Video thumbs via ffmpeg (if detected)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1469,6 +1475,13 @@ impl AppConfig {
|
||||
config.features.enable_places = val;
|
||||
}
|
||||
|
||||
if let Ok(enable_video_thumbnails) =
|
||||
env::var("OXICLOUD_ENABLE_VIDEO_THUMBNAILS").map(|v| v.parse::<bool>())
|
||||
&& let Ok(val) = enable_video_thumbnails
|
||||
{
|
||||
config.features.enable_video_thumbnails = val;
|
||||
}
|
||||
|
||||
if let Ok(enable_faces) = env::var("OXICLOUD_ENABLE_FACES").map(|v| v.parse::<bool>())
|
||||
&& let Ok(val) = enable_faces
|
||||
{
|
||||
|
||||
@@ -46,6 +46,7 @@ use crate::infrastructure::services::search_index::content_index_worker::Content
|
||||
use crate::infrastructure::services::search_index::tantivy_content_index::TantivyContentIndex;
|
||||
use crate::infrastructure::services::trash_cleanup_service::TrashCleanupService;
|
||||
|
||||
use crate::application::ports::video_frame_ports::VideoFramePort;
|
||||
use crate::application::services::app_password_service::AppPasswordService;
|
||||
use crate::application::services::blob_lifecycle_service::BlobLifecycleService;
|
||||
use crate::application::services::calendar_service::CalendarService;
|
||||
@@ -66,6 +67,9 @@ use crate::infrastructure::repositories::pg::{
|
||||
use crate::infrastructure::services::audio_metadata_service::AudioMetadataService;
|
||||
use crate::infrastructure::services::chunked_upload_service::ChunkedUploadService;
|
||||
use crate::infrastructure::services::dedup_service::DedupService;
|
||||
use crate::infrastructure::services::ffmpeg_video_frame_service::{
|
||||
FfmpegVideoFrameService, NoopVideoFrameService,
|
||||
};
|
||||
use crate::infrastructure::services::image_transcode_service::ImageTranscodeService;
|
||||
use crate::infrastructure::services::jwt_service::JwtTokenService;
|
||||
use crate::infrastructure::services::media_metadata_service::MediaMetadataService;
|
||||
@@ -350,9 +354,64 @@ impl AppServiceFactory {
|
||||
// ThumbnailRefreshHook: handles FileLifecycleHook events (create/update/delete).
|
||||
// Implemented on ThumbnailRefreshHook (not ThumbnailService) to avoid circular Arc:
|
||||
// DedupService → BlobLifecycleService → ThumbnailRefreshHook → DedupService.
|
||||
// Video frame extractor for thumbnails. Detect ffmpeg once at startup so
|
||||
// the choice (real extractor vs. no-op) is logged here instead of failing
|
||||
// per upload.
|
||||
let video_frame: Arc<dyn VideoFramePort> = {
|
||||
let ffmpeg_path =
|
||||
std::env::var("OXICLOUD_FFMPEG_PATH").unwrap_or_else(|_| "ffmpeg".to_string());
|
||||
if self.config.features.enable_video_thumbnails
|
||||
&& FfmpegVideoFrameService::is_available(&ffmpeg_path)
|
||||
{
|
||||
let cpus = std::thread::available_parallelism()
|
||||
.map(|n| n.get())
|
||||
.unwrap_or(4);
|
||||
let concurrency = std::env::var("OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<usize>().ok())
|
||||
.unwrap_or((cpus / 2).max(1));
|
||||
let timeout = std::time::Duration::from_secs(
|
||||
std::env::var("OXICLOUD_VIDEO_THUMBNAIL_TIMEOUT_SECS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(30),
|
||||
);
|
||||
tracing::info!(
|
||||
"🎬 Video thumbnails enabled (ffmpeg '{}', concurrency {})",
|
||||
ffmpeg_path,
|
||||
concurrency
|
||||
);
|
||||
Arc::new(FfmpegVideoFrameService::new(
|
||||
ffmpeg_path,
|
||||
concurrency,
|
||||
timeout,
|
||||
))
|
||||
} else {
|
||||
if self.config.features.enable_video_thumbnails {
|
||||
tracing::warn!(
|
||||
"🎬 Video thumbnails enabled but ffmpeg not found at '{}' \
|
||||
(set OXICLOUD_FFMPEG_PATH) — videos will have no thumbnail",
|
||||
ffmpeg_path
|
||||
);
|
||||
} else {
|
||||
tracing::info!("🎬 Video thumbnails disabled");
|
||||
}
|
||||
Arc::new(NoopVideoFrameService)
|
||||
}
|
||||
};
|
||||
// Cap on bytes streamed to a temp file for frame extraction (default 2 GB).
|
||||
// saturating_mul so an absurd MB value can't silently wrap to a tiny cap.
|
||||
let video_max_bytes: u64 = std::env::var("OXICLOUD_VIDEO_THUMBNAIL_MAX_MB")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.unwrap_or(2048)
|
||||
.saturating_mul(1024 * 1024);
|
||||
|
||||
let thumbnail_refresh_hook = Arc::new(ThumbnailRefreshHook::new(
|
||||
thumbnail_service.clone(),
|
||||
dedup_service.clone(),
|
||||
video_frame,
|
||||
video_max_bytes,
|
||||
));
|
||||
|
||||
// Build the unified FileLifecycleService dispatcher.
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
//! ffmpeg-backed video frame extractor (and a no-op fallback).
|
||||
//!
|
||||
//! Shells out to the system `ffmpeg` binary rather than linking libav*: no
|
||||
//! compile-time dependency, no binary bloat, and it decodes every container the
|
||||
//! browser `<video>` element cannot (HEVC/MOV, ProRes, mkv/avi/wmv…). The
|
||||
//! extracted still frame is handed to the existing image thumbnail pipeline, so
|
||||
//! video thumbnails become first-class: WebP, blob-hash keyed (dedup'd), and
|
||||
//! served through the same content negotiation as photos.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use bytes::Bytes;
|
||||
use std::path::Path;
|
||||
use std::process::Stdio;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::process::Command;
|
||||
use tokio::sync::Semaphore;
|
||||
use tokio::time::timeout;
|
||||
|
||||
use crate::application::ports::video_frame_ports::VideoFramePort;
|
||||
use crate::common::errors::DomainError;
|
||||
|
||||
/// PNG file signature — guards against feeding a non-image (or empty) ffmpeg
|
||||
/// output into the renderer.
|
||||
const PNG_MAGIC: &[u8; 8] = b"\x89PNG\r\n\x1a\n";
|
||||
|
||||
/// Extracts a representative frame by invoking `ffmpeg`.
|
||||
pub struct FfmpegVideoFrameService {
|
||||
ffmpeg_path: String,
|
||||
/// Bounds concurrent ffmpeg processes — video decode is CPU-heavy and runs
|
||||
/// outside the async runtime, so it gets its own (smaller) limit rather than
|
||||
/// sharing the image decode semaphore.
|
||||
semaphore: Arc<Semaphore>,
|
||||
/// Per-extraction wall-clock cap; the child is killed on overrun.
|
||||
timeout: Duration,
|
||||
}
|
||||
|
||||
impl FfmpegVideoFrameService {
|
||||
pub fn new(ffmpeg_path: String, concurrency: usize, timeout: Duration) -> Self {
|
||||
Self {
|
||||
ffmpeg_path,
|
||||
semaphore: Arc::new(Semaphore::new(concurrency.max(1))),
|
||||
timeout,
|
||||
}
|
||||
}
|
||||
|
||||
/// Best-effort startup probe: true if `<ffmpeg_path> -version` runs and exits
|
||||
/// 0. Synchronous so the composition root can decide — register the real
|
||||
/// extractor or fall back to [`NoopVideoFrameService`] — without an async
|
||||
/// context, and so a misconfigured path is logged once at boot instead of
|
||||
/// failing per upload.
|
||||
pub fn is_available(ffmpeg_path: &str) -> bool {
|
||||
std::process::Command::new(ffmpeg_path)
|
||||
.arg("-version")
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.status()
|
||||
.map(|s| s.success())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl VideoFramePort for FfmpegVideoFrameService {
|
||||
fn is_supported_video(&self, mime_type: &str) -> bool {
|
||||
// ffmpeg is the arbiter of what actually decodes; gate broadly on video/*
|
||||
// and let extraction fail gracefully for the rare unsupported container.
|
||||
mime_type.starts_with("video/")
|
||||
}
|
||||
|
||||
async fn extract_frame(&self, path: &Path) -> Result<Bytes, DomainError> {
|
||||
let _permit =
|
||||
self.semaphore.acquire().await.map_err(|_| {
|
||||
DomainError::internal_error("VideoFrame", "extractor semaphore closed")
|
||||
})?;
|
||||
|
||||
// One representative still → PNG on stdout. The `thumbnail` filter scans a
|
||||
// window of frames and picks the most representative one (skipping black
|
||||
// intros) without needing a separate duration probe. Fit within a
|
||||
// 1024×1024 box (preserving aspect) — enough for the 800px `large`
|
||||
// thumbnail, and bounding BOTH dimensions caps the emitted PNG size so a
|
||||
// hostile/extreme geometry can't balloon the buffered output. Arguments
|
||||
// are passed individually (never through a shell), so a hostile file name
|
||||
// cannot inject anything.
|
||||
let run = Command::new(&self.ffmpeg_path)
|
||||
.arg("-nostdin")
|
||||
.arg("-loglevel")
|
||||
.arg("error")
|
||||
.arg("-i")
|
||||
.arg(path)
|
||||
.arg("-vf")
|
||||
.arg("thumbnail,scale=w='min(1024,iw)':h='min(1024,ih)':force_original_aspect_ratio=decrease")
|
||||
.arg("-frames:v")
|
||||
.arg("1")
|
||||
.arg("-f")
|
||||
.arg("image2pipe")
|
||||
.arg("-vcodec")
|
||||
.arg("png")
|
||||
.arg("pipe:1")
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.output();
|
||||
|
||||
let output = timeout(self.timeout, run)
|
||||
.await
|
||||
.map_err(|_| DomainError::internal_error("VideoFrame", "ffmpeg timed out"))?
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("VideoFrame", format!("ffmpeg spawn failed: {e}"))
|
||||
})?;
|
||||
|
||||
if !output.status.success() {
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
return Err(DomainError::internal_error(
|
||||
"VideoFrame",
|
||||
format!("ffmpeg exited {}: {}", output.status, stderr.trim()),
|
||||
));
|
||||
}
|
||||
|
||||
let png = output.stdout;
|
||||
if png.len() < PNG_MAGIC.len() || &png[..PNG_MAGIC.len()] != PNG_MAGIC {
|
||||
return Err(DomainError::internal_error(
|
||||
"VideoFrame",
|
||||
"ffmpeg produced no decodable frame",
|
||||
));
|
||||
}
|
||||
Ok(Bytes::from(png))
|
||||
}
|
||||
}
|
||||
|
||||
/// No-op extractor used when `ffmpeg` is unavailable or video thumbnails are
|
||||
/// disabled. `is_supported_video` returns false so the lifecycle hook never
|
||||
/// attempts generation; videos simply have no thumbnail (the prior behaviour).
|
||||
pub struct NoopVideoFrameService;
|
||||
|
||||
#[async_trait]
|
||||
impl VideoFramePort for NoopVideoFrameService {
|
||||
fn is_supported_video(&self, _mime_type: &str) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
async fn extract_frame(&self, _path: &Path) -> Result<Bytes, DomainError> {
|
||||
Err(DomainError::internal_error(
|
||||
"VideoFrame",
|
||||
"video thumbnail extraction is disabled (no ffmpeg)",
|
||||
))
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ pub mod encrypted_blob_backend;
|
||||
pub mod exif_service;
|
||||
pub mod face_geometry;
|
||||
pub mod face_indexing_service;
|
||||
pub mod ffmpeg_video_frame_service;
|
||||
pub mod file_content_cache;
|
||||
pub mod file_system_i18n_service;
|
||||
pub mod image_transcode_service;
|
||||
|
||||
@@ -25,6 +25,7 @@ use tokio::time::timeout;
|
||||
use crate::application::ports::thumbnail_ports::{
|
||||
ThumbnailFormat, ThumbnailPort, ThumbnailSize as PortThumbnailSize, ThumbnailStatsDto,
|
||||
};
|
||||
use crate::application::ports::video_frame_ports::VideoFramePort;
|
||||
use crate::domain::errors::{DomainError, ErrorKind};
|
||||
use crate::infrastructure::services::dedup_service::DedupService;
|
||||
|
||||
@@ -94,6 +95,11 @@ const WEBP_QUALITY: f32 = 82.0;
|
||||
/// Environment override for the decode-concurrency cap (ops tuning).
|
||||
const DECODE_CONCURRENCY_ENV: &str = "OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY";
|
||||
|
||||
/// Wall-clock cap for streaming a video blob to a temp file before frame
|
||||
/// extraction. Bounds a stalled remote object-store read so the background task
|
||||
/// (and its temp file) can't hang forever.
|
||||
const STREAM_TO_TEMP_TIMEOUT: Duration = Duration::from_secs(120);
|
||||
|
||||
/// Compute max concurrent thumbnail decode operations at runtime.
|
||||
///
|
||||
/// Uses all available CPUs (min 2). Before shrink-on-load each decode
|
||||
@@ -1094,8 +1100,20 @@ impl ThumbnailService {
|
||||
}
|
||||
};
|
||||
|
||||
self.render_and_persist_all_webp(&file_id, &blob_hash, original_data)
|
||||
.await;
|
||||
|
||||
tracing::info!("✅ Background thumbnail generation complete: {}", file_id);
|
||||
});
|
||||
}
|
||||
|
||||
/// Render every size as WebP from a decoded source image and persist them by
|
||||
/// blob_hash (disk `{hash}.webp` + moka). Shared by the image upload path and
|
||||
/// the video path (which passes the extracted frame as the source), so both
|
||||
/// produce identical, dedup-able, content-negotiable thumbnails.
|
||||
async fn render_and_persist_all_webp(&self, file_id: &str, blob_hash: &str, source: Bytes) {
|
||||
let results = tokio::task::spawn_blocking(move || {
|
||||
Self::render_all_thumbnails_from_data(original_data.as_ref(), ThumbnailFormat::Webp)
|
||||
Self::render_all_thumbnails_from_data(source.as_ref(), ThumbnailFormat::Webp)
|
||||
})
|
||||
.await;
|
||||
|
||||
@@ -1112,7 +1130,7 @@ impl ThumbnailService {
|
||||
};
|
||||
|
||||
for (size, bytes) in thumbnails {
|
||||
let thumb_path = self.get_thumbnail_path(&blob_hash, size, ThumbnailFormat::Webp);
|
||||
let thumb_path = self.get_thumbnail_path(blob_hash, size, ThumbnailFormat::Webp);
|
||||
if let Some(parent) = thumb_path.parent() {
|
||||
let _ = fs::create_dir_all(parent).await;
|
||||
}
|
||||
@@ -1120,7 +1138,7 @@ impl ThumbnailService {
|
||||
tracing::warn!("Failed to save thumbnail {} {:?}: {}", file_id, size, e);
|
||||
} else {
|
||||
let cache_key = ThumbnailCacheKey {
|
||||
file_id: file_id.clone(),
|
||||
file_id: file_id.to_string(),
|
||||
size,
|
||||
format: ThumbnailFormat::Webp,
|
||||
};
|
||||
@@ -1128,11 +1146,149 @@ impl ThumbnailService {
|
||||
tracing::debug!("✅ Generated thumbnail: {} {:?}", file_id, size);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tracing::info!("✅ Background thumbnail generation complete: {}", file_id);
|
||||
/// Eagerly generate WebP thumbnails for a freshly-uploaded video.
|
||||
///
|
||||
/// Streams the (decrypted, reassembled) blob to a temp file — bounded by
|
||||
/// `max_bytes` so a giant upload never materialises in full — extracts one
|
||||
/// representative frame via `video_frame`, then runs it through the same
|
||||
/// WebP/blob-hash pipeline as photos. Any miss or failure leaves the video
|
||||
/// without a thumbnail (the prior behaviour), never an error to the user.
|
||||
pub fn generate_video_thumbnails_background(
|
||||
self: Arc<Self>,
|
||||
file_id: String,
|
||||
blob_hash: String,
|
||||
dedup: Arc<DedupService>,
|
||||
video_frame: Arc<dyn VideoFramePort>,
|
||||
max_bytes: u64,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
// Blob deleted before this task ran → nothing to do.
|
||||
if !dedup.blob_exists(&blob_hash).await {
|
||||
return;
|
||||
}
|
||||
|
||||
// Dedup hit: thumbnails for this content already exist on disk — just
|
||||
// warm the moka cache (zero CPU), mirroring the image path.
|
||||
let all_exist = {
|
||||
let mut ok = true;
|
||||
for size in ThumbnailSize::all() {
|
||||
let p = self.get_thumbnail_path(&blob_hash, *size, ThumbnailFormat::Webp);
|
||||
if fs::metadata(&p).await.is_err() {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
ok
|
||||
};
|
||||
if all_exist {
|
||||
for size in ThumbnailSize::all() {
|
||||
let p = self.get_thumbnail_path(&blob_hash, *size, ThumbnailFormat::Webp);
|
||||
if let Ok(data) = fs::read(&p).await {
|
||||
let key = ThumbnailCacheKey {
|
||||
file_id: file_id.clone(),
|
||||
size: *size,
|
||||
format: ThumbnailFormat::Webp,
|
||||
};
|
||||
self.cache.insert(key, Bytes::from(data)).await;
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// ffmpeg needs a seekable file, and the blob may be encrypted/chunked
|
||||
// on disk — stream through the normal decrypting read path into a
|
||||
// bounded temp file (on the data volume) rather than reading the raw
|
||||
// blob path. Bounded by size (max_bytes) AND time, so a stalled remote
|
||||
// backend can't hang the task or leak its temp file forever.
|
||||
let tmp = match tokio::time::timeout(
|
||||
STREAM_TO_TEMP_TIMEOUT,
|
||||
Self::stream_blob_to_temp(&dedup, &blob_hash, max_bytes, &self.thumbnails_root),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(Ok(t)) => t,
|
||||
Ok(Err(e)) => {
|
||||
tracing::warn!("🎬 video thumb: stream {} failed: {}", file_id, e);
|
||||
return;
|
||||
}
|
||||
Err(_) => {
|
||||
tracing::warn!("🎬 video thumb: stream {} timed out", file_id);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let frame = match video_frame.extract_frame(tmp.path()).await {
|
||||
Ok(f) => f,
|
||||
Err(e) => {
|
||||
tracing::info!("🎬 video thumb: extract {} skipped: {}", file_id, e);
|
||||
return;
|
||||
}
|
||||
};
|
||||
drop(tmp); // remove the temp video promptly; we have the frame bytes
|
||||
|
||||
// Bound the CPU-bound decode+resize with the same decode_semaphore the
|
||||
// image path uses (the ffmpeg extractor's own permit covered only
|
||||
// extraction and is already released), so concurrent video uploads
|
||||
// can't spawn unbounded render jobs.
|
||||
let _permit = match self.decode_semaphore.acquire().await {
|
||||
Ok(p) => p,
|
||||
Err(_) => return,
|
||||
};
|
||||
self.render_and_persist_all_webp(&file_id, &blob_hash, frame)
|
||||
.await;
|
||||
tracing::info!("✅ Video thumbnail generation complete: {}", file_id);
|
||||
});
|
||||
}
|
||||
|
||||
/// Stream a blob through the decrypting/CDC-aware read path into a temp file,
|
||||
/// aborting if it exceeds `max_bytes`. The returned handle deletes the file
|
||||
/// on drop. Bounded memory: chunks are written straight to disk, never
|
||||
/// buffered whole.
|
||||
async fn stream_blob_to_temp(
|
||||
dedup: &DedupService,
|
||||
blob_hash: &str,
|
||||
max_bytes: u64,
|
||||
temp_dir: &Path,
|
||||
) -> Result<tempfile::NamedTempFile, DomainError> {
|
||||
use futures::StreamExt;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
let mut stream = dedup.read_blob_stream(blob_hash).await?;
|
||||
// Colocate the temp video with the data volume (sized for blobs) instead
|
||||
// of the OS temp dir, which can be small or a tmpfs in containers.
|
||||
let tmp = tempfile::Builder::new()
|
||||
.prefix("oxithumb-")
|
||||
.tempfile_in(temp_dir)
|
||||
.map_err(|e| DomainError::internal_error("VideoFrame", format!("temp file: {e}")))?;
|
||||
let mut out = tokio::fs::File::create(tmp.path())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("VideoFrame", format!("temp open: {e}")))?;
|
||||
|
||||
let mut written: u64 = 0;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk.map_err(|e| {
|
||||
DomainError::internal_error("VideoFrame", format!("blob read: {e}"))
|
||||
})?;
|
||||
written += chunk.len() as u64;
|
||||
if written > max_bytes {
|
||||
return Err(DomainError::internal_error(
|
||||
"VideoFrame",
|
||||
format!("video exceeds {max_bytes}-byte thumbnail cap"),
|
||||
));
|
||||
}
|
||||
out.write_all(&chunk).await.map_err(|e| {
|
||||
DomainError::internal_error("VideoFrame", format!("temp write: {e}"))
|
||||
})?;
|
||||
}
|
||||
out.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("VideoFrame", format!("temp flush: {e}")))?;
|
||||
drop(out); // close our write handle before ffmpeg opens the path
|
||||
Ok(tmp)
|
||||
}
|
||||
|
||||
/// Delete thumbnails for a file.
|
||||
///
|
||||
/// Only invalidates the in-memory moka cache (keyed by file_id).
|
||||
@@ -1205,11 +1361,26 @@ impl ThumbnailService {
|
||||
pub struct ThumbnailRefreshHook {
|
||||
thumbnail: Arc<ThumbnailService>,
|
||||
dedup: Arc<DedupService>,
|
||||
/// Video frame extractor (ffmpeg, or a no-op when unavailable/disabled).
|
||||
video_frame: Arc<dyn VideoFramePort>,
|
||||
/// Max bytes streamed to a temp file for video frame extraction; larger
|
||||
/// videos are skipped (no thumbnail) rather than materialised in full.
|
||||
video_max_bytes: u64,
|
||||
}
|
||||
|
||||
impl ThumbnailRefreshHook {
|
||||
pub fn new(thumbnail: Arc<ThumbnailService>, dedup: Arc<DedupService>) -> Self {
|
||||
Self { thumbnail, dedup }
|
||||
pub fn new(
|
||||
thumbnail: Arc<ThumbnailService>,
|
||||
dedup: Arc<DedupService>,
|
||||
video_frame: Arc<dyn VideoFramePort>,
|
||||
video_max_bytes: u64,
|
||||
) -> Self {
|
||||
Self {
|
||||
thumbnail,
|
||||
dedup,
|
||||
video_frame,
|
||||
video_max_bytes,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1222,9 +1393,10 @@ impl crate::application::ports::file_lifecycle::FileLifecycleHook for ThumbnailR
|
||||
is_new_blob: bool,
|
||||
) {
|
||||
// Blob-hash thumbnail already exists on disk when is_new_blob=false — skip.
|
||||
if !is_new_blob || !ThumbnailService::is_supported_image(content_type) {
|
||||
if !is_new_blob {
|
||||
return;
|
||||
}
|
||||
if ThumbnailService::is_supported_image(content_type) {
|
||||
self.thumbnail
|
||||
.clone()
|
||||
.generate_all_sizes_background_from_blob(
|
||||
@@ -1232,6 +1404,17 @@ impl crate::application::ports::file_lifecycle::FileLifecycleHook for ThumbnailR
|
||||
blob_hash.to_string(),
|
||||
self.dedup.clone(),
|
||||
);
|
||||
} else if self.video_frame.is_supported_video(content_type) {
|
||||
// Videos: extract a frame server-side (ffmpeg) and run it through the
|
||||
// same WebP/blob-hash pipeline — eager, off the request path.
|
||||
self.thumbnail.clone().generate_video_thumbnails_background(
|
||||
file_id.to_string(),
|
||||
blob_hash.to_string(),
|
||||
self.dedup.clone(),
|
||||
self.video_frame.clone(),
|
||||
self.video_max_bytes,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn on_file_copied(
|
||||
|
||||
@@ -429,8 +429,11 @@ impl FileHandler {
|
||||
}
|
||||
};
|
||||
|
||||
// Non-image (video, etc.) with no cached thumbnail → 204
|
||||
if !thumbnail_service.is_supported_image(&file.mime_type) {
|
||||
// Images and videos both store blob-hash thumbnails (videos via an
|
||||
// eagerly-extracted frame); anything else has nothing to thumbnail → 204.
|
||||
let is_image = thumbnail_service.is_supported_image(&file.mime_type);
|
||||
let is_video = file.mime_type.starts_with("video/");
|
||||
if !is_image && !is_video {
|
||||
return Response::builder()
|
||||
.status(StatusCode::NO_CONTENT)
|
||||
.header(header::CACHE_CONTROL, "no-store")
|
||||
@@ -470,6 +473,44 @@ impl FileHandler {
|
||||
.into_response();
|
||||
}
|
||||
|
||||
// Videos: thumbnails are produced eagerly server-side (ffmpeg) on upload,
|
||||
// and persisted WebP-only. Serve that WebP regardless of the negotiated
|
||||
// format — a JPEG/`*/*`/no-Accept client still gets it, correctly labelled
|
||||
// via byte-sniffing — otherwise non-WebP clients would 204 forever despite
|
||||
// a valid thumbnail on disk. We never image-decode a video, so a genuine
|
||||
// miss (generation in flight or unavailable) returns 204.
|
||||
if is_video {
|
||||
if let Some(data) = thumbnail_service
|
||||
.get_cached_thumbnail(
|
||||
&id,
|
||||
Some(&blob_hash),
|
||||
thumb_size.into(),
|
||||
ThumbnailFormat::Webp,
|
||||
)
|
||||
.await
|
||||
{
|
||||
return Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header(
|
||||
header::CONTENT_TYPE,
|
||||
crate::common::mime_detect::thumbnail_content_type(&data),
|
||||
)
|
||||
.header(header::CONTENT_LENGTH, data.len())
|
||||
.header(header::CACHE_CONTROL, "public, max-age=31536000, immutable")
|
||||
.header(header::ETAG, &etag)
|
||||
.header(header::VARY, header::ACCEPT.as_str())
|
||||
.body(Body::from(data))
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
return Response::builder()
|
||||
.status(StatusCode::NO_CONTENT)
|
||||
.header(header::CACHE_CONTROL, "no-store")
|
||||
.body(Body::empty())
|
||||
.unwrap()
|
||||
.into_response();
|
||||
}
|
||||
|
||||
match thumbnail_service
|
||||
.get_thumbnail_from_blob(
|
||||
&id,
|
||||
|
||||
Reference in New Issue
Block a user