diff --git a/docs/config/env.md b/docs/config/env.md index 7ddee4f7..0d1e02a8 100644 --- a/docs/config/env.md +++ b/docs/config/env.md @@ -18,7 +18,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator | `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Direct (non-chunked) uploads stream straight into the blob store and need no spool directory. Placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). | | `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. | | `OXICLOUD_METRICS_LISTEN` | (unset) | Prometheus `/metrics` listener address (e.g. `127.0.0.1:9090`, IPv6 allowed as `[::1]:9090`). **Unset = disabled**: no `/metrics` endpoint is bound and no metrics recorder is installed (zero runtime cost). When set, a separate HTTP listener on this address serves the text-format scrape. **Deliberately NOT merged into the main API** — no auth, CSRF, or DPoP layer in front. Bind to loopback or a private interface unless you intend to expose metrics publicly. Starter counters: `oxicloud_dpop_verify_failed_total{reason}`, `oxicloud_dpop_proof_missing_total`, `oxicloud_dpop_header_missing_on_bound_session_total`, `oxicloud_dpop_replay_detected_total`, `oxicloud_dpop_nonce_challenges_issued_total`. | -| `OXICLOUD_STARTUP_JOBS` | `thumb_derived_import?repair=true,thumb_attached_import?repair=true` | Background jobs dispatched once at boot, comma-separated, each `name` or `name?flag=true` using the same syntax as `POST /api/admin/jobs/{name}/trigger`. Flags: `force`, `deep`, `repair`, `storage`. **The default migrates thumbnails out of the legacy `.thumbnails/` directory and deletes the originals**, so the migration completes without anyone triggering it from the admin panel; each sidecar is read back through the normal stack before it is unlinked, and every deletion is audited. An explicit value **replaces** the default; set it empty (`OXICLOUD_STARTUP_JOBS=`) to disable startup jobs, or to `thumb_derived_import,thumb_attached_import` to import without deleting. **Non-blocking** — readiness never waits on a job; entries run sequentially in the background. **Fail-fast** — an unknown job name or flag panics at boot, because a silently-dropped entry means a migration that never runs. A run interrupted by a restart resumes from its cursor on the next boot, so a long migration finishes across restarts. Safe to leave at the default: the jobs are idempotent, and once drained a run does nothing. See [Thumbnail Migration](./thumbnail-migration.md) for the upgrade runbook. | +| `OXICLOUD_STARTUP_JOBS` | `thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true` | Background jobs dispatched once at boot, comma-separated, each `name` or `name?flag=true` using the same syntax as `POST /api/admin/jobs/{name}/trigger`. Flags: `force`, `deep`, `repair`, `storage`. **The default migrates thumbnails out of the legacy `.thumbnails/` directory and deletes the originals**, so the migration completes without anyone triggering it from the admin panel; each sidecar is read back through the normal stack before it is unlinked, and every deletion is audited. An explicit value **replaces** the default; set it empty (`OXICLOUD_STARTUP_JOBS=`) to disable startup jobs, or to `thumb_derived_import,thumb_attached_import` to import without deleting. **Non-blocking** — readiness never waits on a job; entries run sequentially in the background. **Fail-fast** — an unknown job name or flag panics at boot, because a silently-dropped entry means a migration that never runs. A run interrupted by a restart resumes from its cursor on the next boot, so a long migration finishes across restarts. Safe to leave at the default: the jobs are idempotent, and once drained a run does nothing. See [Thumbnail Migration](./thumbnail-migration.md) for the upgrade runbook. | ## Database diff --git a/docs/plan/job-registry.md b/docs/plan/job-registry.md index b9845953..dda8a884 100644 --- a/docs/plan/job-registry.md +++ b/docs/plan/job-registry.md @@ -746,7 +746,7 @@ optionally with the same query syntax the admin trigger URL uses. **The default is both migration jobs, in repair mode:** ``` -OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true +OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true ``` An explicit value replaces that list; an empty value disables startup diff --git a/example.env b/example.env index 9fa80904..69de5230 100644 --- a/example.env +++ b/example.env @@ -62,7 +62,7 @@ OXICLOUD_SERVER_HOST=0.0.0.0 # storage). # # DEFAULT (applied when this variable is unset): -# thumb_derived_import?repair=true,thumb_attached_import?repair=true +# thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true # # Those two migrate thumbnails out of the legacy .thumbnails/ directory # into blob storage and then delete the originals, so the migration diff --git a/src/common/config.rs b/src/common/config.rs index 31f4ac94..67001e03 100644 --- a/src/common/config.rs +++ b/src/common/config.rs @@ -2401,8 +2401,15 @@ fn parse_startup_job(raw: &str) -> Result { /// Set `OXICLOUD_STARTUP_JOBS=` (empty) to disable startup jobs /// entirely; any explicit value replaces this list rather than adding /// to it. -const DEFAULT_STARTUP_JOBS: &str = - "thumb_derived_import?repair=true,thumb_attached_import?repair=true"; +/// `transcode_import` joins them for the same reason and on the same +/// terms. Its artifacts are the most disposable of the three — a +/// transcode is a pure function of its source, so anything deleted in +/// error is recomputed on the next request — and its `.skip` markers +/// collapse to one row per distinct content, which is the saving that +/// only happens once the import runs. +const DEFAULT_STARTUP_JOBS: &str = "thumb_derived_import?repair=true,\ + thumb_attached_import?repair=true,\ + transcode_import?repair=true"; /// Parse the whole `OXICLOUD_STARTUP_JOBS` value. Empty → no startup /// jobs (an explicit opt-out); unset → [`DEFAULT_STARTUP_JOBS`]. @@ -3999,7 +4006,14 @@ mod tests { fn default_startup_jobs_drain_both_thumbnail_tiers() { let jobs = AppConfig::default().startup_jobs; let names: Vec<&str> = jobs.iter().map(|j| j.name.as_str()).collect(); - assert_eq!(names, ["thumb_derived_import", "thumb_attached_import"]); + assert_eq!( + names, + [ + "thumb_derived_import", + "thumb_attached_import", + "transcode_import" + ] + ); assert!(jobs.iter().all(|j| j.args.repair)); assert!(jobs.iter().all(|j| !j.args.deep && !j.args.force)); } diff --git a/src/infrastructure/services/image_transcode_service.rs b/src/infrastructure/services/image_transcode_service.rs index 1c99732f..12cc6300 100644 --- a/src/infrastructure/services/image_transcode_service.rs +++ b/src/infrastructure/services/image_transcode_service.rs @@ -16,7 +16,7 @@ use bytes::Bytes; use image::ImageFormat; use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, OnceLock}; use tokio::fs; @@ -173,6 +173,15 @@ pub struct ImageTranscodeService { /// `OnceLock` rather than a `Mutex`: set exactly once at boot, read on /// every request, never replaced. dedup: OnceLock>, + /// Whether `.transcoded/` still exists, probed once by + /// [`Self::initialize`]. `false` short-circuits the local-cache reads + /// without a syscall. + /// + /// Starts `true` so a service constructed without `initialize` (tests) + /// behaves as before. Failing open is the safe direction: the wrong + /// value costs syscalls, the opposite would hide cached entries that + /// are still there. + legacy_cache: AtomicBool, } impl ImageTranscodeService { @@ -203,6 +212,7 @@ impl ImageTranscodeService { memory_cache, stats: Arc::new(AtomicTranscodeStats::default()), dedup: OnceLock::new(), + legacy_cache: AtomicBool::new(true), } } @@ -234,16 +244,52 @@ impl ImageTranscodeService { /// absence that `transcode_import` works toward unreachable, which is /// exactly the bug that kept `.thumbnails/` alive across restarts. pub async fn initialize(&self) -> std::io::Result<()> { - fs::create_dir_all(&self.cache_dir).await?; - fs::create_dir_all(self.cache_dir.join("webp")).await?; + // Probes, does NOT create. + // + // Creating the tree at boot is what kept `.thumbnails/` alive across + // restarts: the import removed it, the next boot put it back, and + // the absence the read path gates on was unreachable by + // construction. The write path below already calls `create_dir_all` + // on the parent before writing, so nothing needs it created eagerly + // — the only thing eager creation achieved was defeating the drain. + // + // One `stat` on the root, cached for the process lifetime. It can + // only be stale in the harmless direction: a drain completing + // mid-life leaves the flag true until restart, costing the same + // failed opens as before. It never goes false while entries remain, + // because only `transcode_import` removes the tree and it removes + // the whole thing at once. + let present = fs::metadata(&self.cache_dir).await.is_ok(); + self.legacy_cache.store(present, Ordering::Relaxed); + tracing::info!( - "🖼️ Image transcode service initialized (rayon pool: {} threads, cache dir: {:?})", + "🖼️ Image transcode service initialized (rayon pool: {} threads)", transcode_thread_count(), - self.cache_dir ); + if present { + tracing::info!( + target: "oxicloud::transcode", + event = "transcode.legacy_cache_present", + path = ?self.cache_dir, + "legacy transcode cache present — reads fall back to it. Run \ + transcode_import with ?repair=true to drain it." + ); + } Ok(()) } + /// Whether the legacy local cache is worth touching. + /// + /// Unlike the thumbnail tiers this may legitimately never reach `false`: + /// callers with no content hash (external mounts) cannot use the + /// content-keyed tier at all, so they still read and write here. On an + /// install without such mounts the directory drains once and stays + /// gone; on one with them it persists, and that is correct rather than + /// a stalled migration. + fn legacy_cache_active(&self) -> bool { + self.legacy_cache.load(Ordering::Relaxed) + } + /// Check if a mime type can be transcoded. /// /// JPEG is deliberately excluded: the `image` crate's WebP encoder is @@ -402,7 +448,7 @@ impl ImageTranscodeService { // // Drained by `transcode_import`; kept as a fallback until it is gone. let cache_path = self.get_cache_path(file_id, target_format); - if tokio::fs::try_exists(&cache_path).await.unwrap_or(false) { + if self.legacy_cache_active() && tokio::fs::try_exists(&cache_path).await.unwrap_or(false) { match fs::read(&cache_path).await { Ok(data) => { self.stats.disk_hits.fetch_add(1, Ordering::Relaxed); @@ -417,7 +463,8 @@ impl ImageTranscodeService { // ── Negative verdict persisted on disk (survives restarts) ── let skip_marker = self.get_skip_marker_path(file_id, target_format); - if tokio::fs::try_exists(&skip_marker).await.unwrap_or(false) { + if self.legacy_cache_active() && tokio::fs::try_exists(&skip_marker).await.unwrap_or(false) + { self.stats.disk_hits.fetch_add(1, Ordering::Relaxed); tracing::debug!("💾 Transcode negative disk marker HIT: {}", file_id); return Ok(Bytes::new());