Oxicloud/docs/architecture/jobs.md

Background Jobs

OxiCloud runs periodic maintenance work through a central JobRegistry scheduler. This document is for implementors adding a new tenant or debugging an existing one.

The design rationale (why a registry vs. per-service tokio::spawn loops, the two-engine split, plugin future) lives in the plan doc: docs/plan/job-registry.md. This page is the "how to plug in" reference.

---

When does a background loop belong here?

Single decision question:

"Would an operator plausibly POST /api/admin/jobs/{name}/trigger to make it run right now?"

If yes, register it with JobRegistry. You get:

• Uniform admin surface (list, trigger, last-outcome).
• Uniform oxicloud::scheduler log line per run with elapsed_ms.
• Panic containment (a handler that panics doesn't kill the scheduler).
• Exclusivity: only one in-flight run per job name (a tick that fires while the previous run is still going is skipped, with a warning).
• Optional wall-clock timeout.
• Optional acceleration parameter (?force=true → JobRunArgs.force).

If no — the loop is a queue drainer (tree_etag_flush_job), a continuous event reactor (content_index_worker), or a stats printer (db_pool_monitor) — keep it as its own dedicated tokio::spawn loop. Wedging it into JobRegistry adds framework overhead for no operator benefit.

---

Current tenants

Job name	Cadence	Force semantic	Service
trash_cleanup	24 h (hardcoded in DI, no env var yet)	ignored	trash_cleanup_service.rs
storage_reconcile	OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default 600s, min 30s)	ignored	storage_usage_service.rs
dedup_gc	on-demand only (trash cleanup runs it inline as its tail step)	force=true → garbage_collect_force() (skip orphan grace)	dedup_service.rs
grant_cleanup	OXICLOUD_GRANT_CLEANUP_INTERVAL_HOURS (default 24h) — feature-gated by OXICLOUD_GRANT_CLEANUP_ENABLED	force=true → purge(Some(0)) (grace_days=0)	grant_cleanup_service.rs

---

Adding a new job — recipe

1. Implement JobHandler on your service

use std::sync::Arc;
use async_trait::async_trait;

use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};

pub const MY_JOB_NAME: &str = "my_job";  // stable snake_case

#[async_trait]
impl JobHandler for MyService {
    fn name(&self) -> &str {
        MY_JOB_NAME
    }

    async fn run(&self, args: &JobRunArgs) -> JobOutcome {
        // …do work…
        match self.do_the_work().await {
            Ok(count) => JobOutcome::ok_with(
                count,
                serde_json::json!({
                    // Anything useful for `GET /api/admin/jobs` or logs.
                    "bytes_processed": total_bytes,
                    "forced": args.force,
                }),
            ),
            Err(e) => JobOutcome::err(format!("my_job failed: {e}")),
        }
    }
}

Rules:

• name() must be stable — it appears in log lines, admin URLs, and admin.background_runs.job_name (once Part 2 lands). Renaming is a breaking change to operator scripts and log dashboards.
• args.force semantics are per-job. If your job has no acceleration mode (e.g. reconciliation, which is always idempotent), ignore it. If it does (e.g. skip a grace window), document the behaviour on the handler docstring.
• Return JobOutcome::ok_with(count, extra). count is the primary scalar operators read (rows swept, blobs reclaimed). extra is a free-form JSON blob surfaced in the log line and the admin listing.
• Return JobOutcome::err(msg) on failure. The supervisor logs outcome=err, cause=handler and continues to the next tick. Don't catch panics inside run() — the supervisor does it, and hiding one loses the cause=panicked diagnostic.

2. Add a register() method to your service

Every service uses the chainable self-registration shape:

impl MyService {
    /// Register self with the periodic-job scheduler and return the
    /// same `Arc<Self>` for DI-style chaining.
    pub async fn register(self: Arc<Self>, registry: &JobRegistry) -> Arc<Self> {
        let interval = /* Some(Duration::from_secs(...)) or None for on-demand */;
        registry.register(self.clone(), interval, None /* no timeout */).await;
        self
    }
}

• Interval Some(dur) → scheduled. The supervisor fires the handler every dur.
• Interval None → on-demand only. Never fires periodically; only reachable via POST /api/admin/jobs/{name}/trigger or a programmatic call to registry.trigger(name, args). Use for jobs whose periodic work happens elsewhere (dedup GC piggybacks on trash cleanup) but which still benefit from a uniform admin trigger.
• Timeout Some(dur) → the supervisor wraps the handler in tokio::time::timeout. Timeout trip is logged as outcome=err, cause=timeout. Use sparingly — most native jobs don't need it. Note: aborting a task mid-run is best-effort; a handler that ignores await points may still finish in the background.

3. Wire in common/di.rs

One statement per service:

let my_service = Arc::new(MyService::new(deps...))
    .register(&core.job_registry)
    .await;

register() panics on error (duplicate job name = DI wiring bug; boot must fail loud) and emits a job.registered log line on success. No if let Err(e) scaffolding needed at the call site.

core.job_registry is populated inside create_core_services and lives on CoreServices. SchedulerEngine spawns the supervisor task at the end of build_app_state after every service has registered.

4. That's it

• GET /api/admin/jobs immediately lists the new job.
• POST /api/admin/jobs/my_job/trigger runs one dispatch off-schedule.
• POST /api/admin/jobs/my_job/trigger?force=true runs one dispatch with JobRunArgs { force: true }.
• If the job is scheduled, the supervisor fires it at the configured cadence.

No handler registration in the router, no admin trigger endpoint to add — the framework already covers those uniformly.

---

Admin surface

Production admin endpoints, always on, audit-logged. Admin-only via the standard /api/admin/* middleware (no dedicated feature flag).

GET  /api/admin/jobs
     → [{ name, interval_ms?, next_run_at?, last_run_at?, last_outcome?, running }]

POST /api/admin/jobs/{name}/trigger[?force=<bool>]
     → 200 { ok: true, outcome: { outcome: "ok" | "err", count, extra?, message? } }
     → 404 { error: "job not registered", name }

• interval_ms / next_run_at are absent for on-demand jobs (skip_serializing_if=None).
• last_run_at / last_outcome are absent until the first run completes.
• running is true iff the in-flight permit is currently held.
• Every trigger call emits a target: "audit" log line (event = "job.trigger") before dispatch.

---

Log lines

Uniform target: oxicloud::scheduler.

Event	Fields	When
scheduler.started	—	Supervisor loop starts
scheduler.ready	registered = N	All services have registered
job.registered	job, cadence ("every 24 h" / "on-demand")	Each register() call
job.run	job, outcome (ok|err), cause? (handler|timeout|panicked), count, elapsed_ms, extra?, error?	Every dispatch
job.tick_skipped	job, interval_ms, running_for_ms	Tick fires while the previous run is still going
job.trigger	job, force (audit channel)	Admin POST /trigger

elapsed_ms is the raw scalar in the structured field; the human message renders it as 12ms / 1.4s / 4m30s so tail -f operators see the duration inline.

---

Testing

The scheduler's own tests live in src/infrastructure/scheduler/ (registry.rs::tests, engine.rs::tests) and use dummy handlers. No integration test infrastructure is needed to add a new service — your service's normal unit tests cover the run() logic, and the Hurl suite tests/api/admin_jobs.hurl covers the admin surface generically.

If your service has a test that needs to construct the type WITHOUT registering with a scheduler (e.g. isolated unit tests), skip the .register(&reg).await chain and use the bare Arc::new(Service::new(...)).

---

Non-goals

• Cross-job dependencies. No depends_on — each job runs independently. If you find yourself needing "job B runs after job A completes", route the completion signal through a lifecycle hook (FileLifecycleHook, BlobLifecycleHook), not through the scheduler.
• Distributed scheduling. Single-process only. If OxiCloud ever runs multi-node, the pattern is SELECT … FOR UPDATE SKIP LOCKED on a lease table — not this design.
• Cron expressions. Fixed intervals only. Real cron (day-of-week/month, arbitrary times) can layer on top later via a next_run: Box<dyn NextRun> trait; nothing needs it today.
• Backfill on startup. If the process was down when a scheduled tick was due, the missed tick is NOT caught up — the next tick fires at its normal interval.

---

Related

• Plan doc: docs/plan/job-registry.md
• Long-running / resumable jobs (Part 2, not yet built): docs/plan/job-registry.md#part-2--recoverable-run-engine
• Consistency checks (a future Part 2 consumer): docs/plan/consistency-check.md