feat(job-registry): wire /api/admin/jobs/*
This commit is contained in:
+79
-26
@@ -53,8 +53,8 @@ duplicating them between parts.
|
|||||||
Not every background loop belongs in JobRegistry. The single question
|
Not every background loop belongs in JobRegistry. The single question
|
||||||
that decides:
|
that decides:
|
||||||
|
|
||||||
> **"Would an operator plausibly `POST /trigger-job/{name}` to make
|
> **"Would an operator plausibly `POST /api/admin/jobs/{name}/trigger`
|
||||||
> it run right now?"**
|
> to make it run right now?"**
|
||||||
|
|
||||||
**Yes → migrate.** The whole payoff of JobRegistry is a uniform
|
**Yes → migrate.** The whole payoff of JobRegistry is a uniform
|
||||||
*operator surface* — list, trigger, last-outcome, log line, config
|
*operator surface* — list, trigger, last-outcome, log line, config
|
||||||
@@ -144,7 +144,26 @@ pub trait JobHandler: Send + Sync {
|
|||||||
/// extra }` on success — the count is the primary scalar the job
|
/// extra }` on success — the count is the primary scalar the job
|
||||||
/// reports (rows swept, ETags flushed, blobs GC'd). Return
|
/// reports (rows swept, ETags flushed, blobs GC'd). Return
|
||||||
/// `Err(msg)` on failure; the supervisor logs it and continues.
|
/// `Err(msg)` on failure; the supervisor logs it and continues.
|
||||||
async fn run(&self) -> JobOutcome;
|
///
|
||||||
|
/// `args` carries per-dispatch parameters. Periodic ticks pass
|
||||||
|
/// `JobRunArgs::default()`; admin triggers can set `force: true`
|
||||||
|
/// to request acceleration semantics (e.g. dedup GC skips its
|
||||||
|
/// orphan grace window, grant cleanup uses grace = 0). Handlers
|
||||||
|
/// that don't understand a given arg silently ignore it — no
|
||||||
|
/// return-error path just because a caller set an unused flag.
|
||||||
|
async fn run(&self, args: &JobRunArgs) -> JobOutcome;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Per-dispatch parameters. Grows over time; today it carries only
|
||||||
|
/// `force`. Kept as a struct (not `bool`) so we don't have to change
|
||||||
|
/// signatures the next time a job needs another knob.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct JobRunArgs {
|
||||||
|
/// Request acceleration semantics. Semantics are per-job:
|
||||||
|
/// - `dedup_gc`: skip the orphan grace window (grace = 0).
|
||||||
|
/// - `grant_cleanup`: grace = 0.
|
||||||
|
/// - Others: silently ignored.
|
||||||
|
pub force: bool,
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -294,19 +313,22 @@ registry.register(
|
|||||||
- `Some(dur)` — supervisor fires the job every `dur`. Also admin-triggerable.
|
- `Some(dur)` — supervisor fires the job every `dur`. Also admin-triggerable.
|
||||||
- `None` — supervisor never fires the job. Admin-triggerable only. Dispatch still routes through the same `JobRegistry::trigger(name)` path so the job gets the same panic-containment, timeout, exclusivity, and log-line treatment as scheduled ones.
|
- `None` — supervisor never fires the job. Admin-triggerable only. Dispatch still routes through the same `JobRegistry::trigger(name)` path so the job gets the same panic-containment, timeout, exclusivity, and log-line treatment as scheduled ones.
|
||||||
|
|
||||||
### Manual dispatch — `JobRegistry::trigger(name)`
|
### Manual dispatch — `JobRegistry::trigger(name, args)`
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
pub async fn trigger(&self, name: &str) -> Option<JobOutcome>;
|
pub async fn trigger(&self, name: &str, args: &JobRunArgs) -> Option<JobOutcome>;
|
||||||
```
|
```
|
||||||
|
|
||||||
The single entry point for running a registered job outside the
|
The single entry point for running a registered job outside the
|
||||||
scheduler's tick loop. Called by:
|
scheduler's tick loop. Called by:
|
||||||
- The admin endpoint (`POST /api/admin/internal/trigger-job/{name}`).
|
- The admin endpoint (`POST /api/admin/jobs/{name}/trigger?force=<bool>`).
|
||||||
- Any service that wants a scheduler-uniform dispatch of a peer job
|
- Any service that wants a scheduler-uniform dispatch of a peer job
|
||||||
(e.g. an inline call from trash cleanup to `trigger("dedup_gc")`,
|
(e.g. an inline call from trash cleanup to `trigger("dedup_gc", &args)`,
|
||||||
if we later route the piggyback through the registry).
|
if we later route the piggyback through the registry).
|
||||||
|
|
||||||
|
The supervisor's periodic ticks invoke the same underlying dispatch
|
||||||
|
with `JobRunArgs::default()` — periodic runs never force.
|
||||||
|
|
||||||
Returns `None` when the name doesn't exist. Returns `Some(JobOutcome)`
|
Returns `None` when the name doesn't exist. Returns `Some(JobOutcome)`
|
||||||
otherwise — even when exclusivity kicks the trigger out (that maps
|
otherwise — even when exclusivity kicks the trigger out (that maps
|
||||||
to `Ok { count: 0, extra: {"skipped": "already_running"} }`, not
|
to `Ok { count: 0, extra: {"skipped": "already_running"} }`, not
|
||||||
@@ -384,11 +406,11 @@ into the scheduler.
|
|||||||
2. **Boot**: start server; expect `scheduler started, N job(s) registered`.
|
2. **Boot**: start server; expect `scheduler started, N job(s) registered`.
|
||||||
3. **Admin listing**:
|
3. **Admin listing**:
|
||||||
```
|
```
|
||||||
curl -s http://localhost:8086/api/admin/internal/jobs -H "Authorization: Bearer $TOKEN"
|
curl -s http://localhost:8086/api/admin/jobs -H "Authorization: Bearer $TOKEN"
|
||||||
```
|
```
|
||||||
returns a JSON array with each registered job, its `interval_ms`,
|
returns a JSON array with each registered job, its `interval_ms`,
|
||||||
`next_run_at`, and `last_outcome` (null until first tick).
|
`next_run_at`, and `last_outcome` (null until first tick).
|
||||||
4. **Trigger**: `POST /api/admin/internal/trigger-job/trash_cleanup`
|
4. **Trigger**: `POST /api/admin/jobs/trash_cleanup/trigger`
|
||||||
invokes the handler immediately, records the outcome.
|
invokes the handler immediately, records the outcome.
|
||||||
5. **Panic containment**: unit test a handler that panics; `last_outcome`
|
5. **Panic containment**: unit test a handler that panics; `last_outcome`
|
||||||
records `Err(...)` with `cause = "panicked"` in the log; the scheduler
|
records `Err(...)` with `cause = "panicked"` in the log; the scheduler
|
||||||
@@ -665,16 +687,17 @@ into this general one.
|
|||||||
|
|
||||||
### Admin surface (recoverable runs)
|
### Admin surface (recoverable runs)
|
||||||
|
|
||||||
Same URL taxonomy as Part 1, extended for run identity:
|
Same URL taxonomy as Part 1 — resource-first, action second, all
|
||||||
|
under `/api/admin/jobs/{name}/*`. Extended for run identity:
|
||||||
|
|
||||||
```
|
```
|
||||||
POST /api/admin/internal/trigger-job/{name}
|
POST /api/admin/jobs/{name}/trigger
|
||||||
→ { run_id, status } # starts or resumes; idempotent
|
→ { run_id, status } # starts or resumes; idempotent
|
||||||
POST /api/admin/internal/trigger-job/{name}/cancel
|
POST /api/admin/jobs/{name}/cancel
|
||||||
→ { run_id, status: "CancelRequested" }
|
→ { run_id, status: "CancelRequested" }
|
||||||
GET /api/admin/internal/jobs/{name}/runs
|
GET /api/admin/jobs/{name}/runs
|
||||||
→ [{ run_id, status, started_at, last_progress_at, stats, ... }]
|
→ [{ run_id, status, started_at, last_progress_at, stats, ... }]
|
||||||
GET /api/admin/internal/jobs/{name}/runs/{id}
|
GET /api/admin/jobs/{name}/runs/{id}
|
||||||
→ { run_id, status, cursor_hex, stats, params, error_message, ... }
|
→ { run_id, status, cursor_hex, stats, params, error_message, ... }
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -696,13 +719,13 @@ GET /api/admin/internal/jobs/{name}/runs/{id}
|
|||||||
### Verification (Part 2)
|
### Verification (Part 2)
|
||||||
|
|
||||||
1. **Compile + schema-migration idempotence.**
|
1. **Compile + schema-migration idempotence.**
|
||||||
2. **Fresh run:** `POST /trigger-job/storage_migration` → new row with
|
2. **Fresh run:** `POST /api/admin/jobs/storage_migration/trigger` → new row with
|
||||||
`status='Running'`, `cursor=NULL`.
|
`status='Running'`, `cursor=NULL`.
|
||||||
3. **Concurrent trigger:** second `POST` while the first is running
|
3. **Concurrent trigger:** second `POST` while the first is running
|
||||||
returns the SAME `run_id` (idempotent, DB unique index enforces).
|
returns the SAME `run_id` (idempotent, DB unique index enforces).
|
||||||
4. **Cancel + resume round-trip:** `trigger-job/…/cancel` flips to
|
4. **Cancel + resume round-trip:** `/api/admin/jobs/…/cancel` flips to
|
||||||
`CancelRequested`; handler polls, returns `Paused { cursor }`;
|
`CancelRequested`; handler polls, returns `Paused { cursor }`;
|
||||||
engine writes `Paused`. `POST /trigger-job/…` again resumes; cursor
|
engine writes `Paused`. `POST /api/admin/jobs/…/trigger` again resumes; cursor
|
||||||
picks up where left off; `stats.count` continues accumulating.
|
picks up where left off; `stats.count` continues accumulating.
|
||||||
5. **Crash recovery:** stop the server mid-run; restart; boot sweep
|
5. **Crash recovery:** stop the server mid-run; restart; boot sweep
|
||||||
flips the row to `Paused` with `error_message = 'server restart mid-run'`;
|
flips the row to `Paused` with `error_message = 'server restart mid-run'`;
|
||||||
@@ -721,16 +744,46 @@ GET /api/admin/internal/jobs/{name}/runs/{id}
|
|||||||
|
|
||||||
### Admin URL taxonomy
|
### Admin URL taxonomy
|
||||||
|
|
||||||
All under `/api/admin/internal/*`, gated by the existing
|
All scheduler endpoints live on the **production admin surface**:
|
||||||
`OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS` env var — reuses the same
|
`/api/admin/jobs/*`. Always on, audit-logged, no feature-flag gate —
|
||||||
admin-guard middleware and the same "disabled → 404" contract as
|
these are the operational levers you actually want ops to reach in
|
||||||
today's per-service triggers.
|
prod. See `project_admin_url_taxonomy` for the `/admin` vs
|
||||||
|
`/admin/internal` split we're honouring here.
|
||||||
|
|
||||||
**Existing per-service shims** (`trigger-sweep`, `trigger-gc`,
|
**Resource-first URL taxonomy** for every scheduler-owned endpoint:
|
||||||
`trigger-grant-cleanup`) stay as thin forwards to `trigger-job/{name}`
|
|
||||||
during migration so the existing Hurl suites keep working.
|
```
|
||||||
Deprecation surfaces via a `Deprecation: true` response header
|
GET /api/admin/jobs # list all
|
||||||
operators can grep for.
|
POST /api/admin/jobs/{name}/trigger # one dispatch (Part 1 + 2)
|
||||||
|
POST /api/admin/jobs/{name}/cancel # cooperative pause (Part 2)
|
||||||
|
GET /api/admin/jobs/{name}/runs # run history (Part 2)
|
||||||
|
GET /api/admin/jobs/{name}/runs/{id} # single run detail (Part 2)
|
||||||
|
```
|
||||||
|
|
||||||
|
`{name}` is the stable `JobHandler::name()` identifier. `trigger`
|
||||||
|
accepts an optional `?force=<bool>` query param that maps to
|
||||||
|
`JobRunArgs.force`.
|
||||||
|
|
||||||
|
**Audit logging.** Every `POST` to `/api/admin/jobs/*` emits a
|
||||||
|
`target: "audit"` line before invoking the registry — bulk-effect
|
||||||
|
mutations belong on the audit stream. Success/failure outcome fires
|
||||||
|
its own `oxicloud::scheduler` line via the existing supervisor path.
|
||||||
|
|
||||||
|
**Legacy shim retirement** (Stage 2 — follow-up PR after this one):
|
||||||
|
|
||||||
|
The three existing internal endpoints map 1:1 to the new surface:
|
||||||
|
|
||||||
|
| Legacy | Replacement |
|
||||||
|
|---|---|
|
||||||
|
| `POST /admin/internal/trigger-sweep` | `POST /admin/jobs/storage_reconcile/trigger` |
|
||||||
|
| `POST /admin/internal/trigger-gc?force=X` | `POST /admin/jobs/dedup_gc/trigger?force=X` |
|
||||||
|
| `POST /admin/internal/trigger-grant-cleanup?force=X` | `POST /admin/jobs/grant_cleanup/trigger?force=X` |
|
||||||
|
|
||||||
|
Rewritten as thin forwards to the new endpoints with a `Deprecation:
|
||||||
|
true` response header while Hurl suites migrate to the new paths. Once
|
||||||
|
all callers cut over, the shims are deleted AND the
|
||||||
|
`OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS` env var is removed — its
|
||||||
|
sole purpose was gating those shims.
|
||||||
|
|
||||||
### Config surface — env vars
|
### Config surface — env vars
|
||||||
|
|
||||||
|
|||||||
@@ -578,7 +578,7 @@ impl StorageUsageService {
|
|||||||
|
|
||||||
pub const STORAGE_RECONCILE_JOB_NAME: &str = "storage_reconcile";
|
pub const STORAGE_RECONCILE_JOB_NAME: &str = "storage_reconcile";
|
||||||
|
|
||||||
use crate::infrastructure::scheduler::{JobHandler, JobOutcome};
|
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRunArgs};
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
|
||||||
#[async_trait]
|
#[async_trait]
|
||||||
@@ -601,7 +601,11 @@ impl JobHandler for StorageUsageService {
|
|||||||
/// Failure of one sub-sweep short-circuits the tick to `Err`;
|
/// Failure of one sub-sweep short-circuits the tick to `Err`;
|
||||||
/// operators see `outcome=err, cause=handler` in the scheduler
|
/// operators see `outcome=err, cause=handler` in the scheduler
|
||||||
/// log and the individual sweep's own `error!` line above it.
|
/// log and the individual sweep's own `error!` line above it.
|
||||||
async fn run(&self) -> JobOutcome {
|
///
|
||||||
|
/// `args.force` is ignored — reconciliation is idempotent and has
|
||||||
|
/// no acceleration semantics; every run does the same set-based
|
||||||
|
/// UPDATE regardless.
|
||||||
|
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
|
||||||
let drives = match self.update_all_drives_storage_usage().await {
|
let drives = match self.update_all_drives_storage_usage().await {
|
||||||
Ok(n) => n,
|
Ok(n) => n,
|
||||||
Err(e) => return JobOutcome::Err(format!("drive reconciliation failed: {e}")),
|
Err(e) => return JobOutcome::Err(format!("drive reconciliation failed: {e}")),
|
||||||
|
|||||||
+1
-1
@@ -1281,7 +1281,7 @@ impl AppServiceFactory {
|
|||||||
// sweep already runs GC as its tail step, so a periodic dedup
|
// sweep already runs GC as its tail step, so a periodic dedup
|
||||||
// schedule would double the work. Registering with `interval =
|
// schedule would double the work. Registering with `interval =
|
||||||
// None` keeps it admin-triggerable through the uniform scheduler
|
// None` keeps it admin-triggerable through the uniform scheduler
|
||||||
// surface (`POST /api/admin/internal/trigger-job/dedup_gc`).
|
// surface (`POST /api/admin/jobs/dedup_gc/trigger`).
|
||||||
if let Err(e) = core
|
if let Err(e) = core
|
||||||
.job_registry
|
.job_registry
|
||||||
.register(
|
.register(
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ use chrono::Utc;
|
|||||||
use tokio::task::JoinHandle;
|
use tokio::task::JoinHandle;
|
||||||
|
|
||||||
use super::registry::{JobEntry, JobRegistry};
|
use super::registry::{JobEntry, JobRegistry};
|
||||||
use super::types::{ErrCause, JobOutcome};
|
use super::types::{ErrCause, JobOutcome, JobRunArgs};
|
||||||
|
|
||||||
/// Public handle to the running supervisor.
|
/// Public handle to the running supervisor.
|
||||||
///
|
///
|
||||||
@@ -93,8 +93,9 @@ async fn run(registry: Arc<JobRegistry>) {
|
|||||||
|
|
||||||
// Fire and forget from the supervisor's perspective — we
|
// Fire and forget from the supervisor's perspective — we
|
||||||
// don't care about the outcome, `dispatch` records it on the
|
// don't care about the outcome, `dispatch` records it on the
|
||||||
// entry and emits the log line itself.
|
// entry and emits the log line itself. Periodic ticks never
|
||||||
let _ = dispatch(&name, entry).await;
|
// force — that's an admin-trigger-only affordance.
|
||||||
|
let _ = dispatch(&name, entry, &JobRunArgs::default()).await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -112,7 +113,15 @@ async fn run(registry: Arc<JobRegistry>) {
|
|||||||
///
|
///
|
||||||
/// Non-panicking; every failure path resolves to a `JobOutcome::Err`
|
/// Non-panicking; every failure path resolves to a `JobOutcome::Err`
|
||||||
/// with a `cause` log field.
|
/// with a `cause` log field.
|
||||||
pub(super) async fn dispatch(name: &str, entry: Arc<JobEntry>) -> JobOutcome {
|
///
|
||||||
|
/// `args` is passed through to `JobHandler::run`. The supervisor's
|
||||||
|
/// periodic ticks pass `JobRunArgs::default()`; the admin trigger
|
||||||
|
/// endpoint forwards parsed query params such as `?force=true`.
|
||||||
|
pub(super) async fn dispatch(
|
||||||
|
name: &str,
|
||||||
|
entry: Arc<JobEntry>,
|
||||||
|
args: &JobRunArgs,
|
||||||
|
) -> JobOutcome {
|
||||||
// Try to acquire the single-permit gate. `try_acquire` is
|
// Try to acquire the single-permit gate. `try_acquire` is
|
||||||
// non-blocking — if held, we know the previous run is still
|
// non-blocking — if held, we know the previous run is still
|
||||||
// executing and skip this tick.
|
// executing and skip this tick.
|
||||||
@@ -157,9 +166,11 @@ pub(super) async fn dispatch(name: &str, entry: Arc<JobEntry>) -> JobOutcome {
|
|||||||
let start_instant = Instant::now();
|
let start_instant = Instant::now();
|
||||||
|
|
||||||
// Spawn so panics land as `JoinError::is_panic()` instead of
|
// Spawn so panics land as `JoinError::is_panic()` instead of
|
||||||
// unwinding into the supervisor loop.
|
// unwinding into the supervisor loop. Args cloned into the spawn
|
||||||
|
// scope so the borrow doesn't outlive the caller.
|
||||||
let handler = entry.handler.clone();
|
let handler = entry.handler.clone();
|
||||||
let join = tokio::spawn(async move { handler.run().await });
|
let args_owned = args.clone();
|
||||||
|
let join = tokio::spawn(async move { handler.run(&args_owned).await });
|
||||||
|
|
||||||
let (outcome, cause) = match entry.timeout {
|
let (outcome, cause) = match entry.timeout {
|
||||||
Some(dur) => match tokio::time::timeout(dur, join).await {
|
Some(dur) => match tokio::time::timeout(dur, join).await {
|
||||||
@@ -305,7 +316,7 @@ mod tests {
|
|||||||
fn name(&self) -> &str {
|
fn name(&self) -> &str {
|
||||||
&self.name
|
&self.name
|
||||||
}
|
}
|
||||||
async fn run(&self) -> JobOutcome {
|
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
|
||||||
self.calls.fetch_add(1, Ordering::SeqCst);
|
self.calls.fetch_add(1, Ordering::SeqCst);
|
||||||
if !self.sleep.is_zero() {
|
if !self.sleep.is_zero() {
|
||||||
tokio::time::sleep(self.sleep).await;
|
tokio::time::sleep(self.sleep).await;
|
||||||
@@ -321,7 +332,7 @@ mod tests {
|
|||||||
fn name(&self) -> &str {
|
fn name(&self) -> &str {
|
||||||
"panicker"
|
"panicker"
|
||||||
}
|
}
|
||||||
async fn run(&self) -> JobOutcome {
|
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
|
||||||
panic!("intentional test panic");
|
panic!("intentional test panic");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -331,7 +342,7 @@ mod tests {
|
|||||||
// Directly exercise translate_join with a spawned panic — the
|
// Directly exercise translate_join with a spawned panic — the
|
||||||
// supervisor loop's dispatch path uses this same helper.
|
// supervisor loop's dispatch path uses this same helper.
|
||||||
let handler = Arc::new(PanickingHandler);
|
let handler = Arc::new(PanickingHandler);
|
||||||
let join = tokio::spawn(async move { handler.run().await });
|
let join = tokio::spawn(async move { handler.run(&JobRunArgs::default()).await });
|
||||||
let (outcome, cause) = translate_join(join.await);
|
let (outcome, cause) = translate_join(join.await);
|
||||||
assert!(!outcome.is_ok());
|
assert!(!outcome.is_ok());
|
||||||
assert_eq!(cause, Some(ErrCause::Panicked));
|
assert_eq!(cause, Some(ErrCause::Panicked));
|
||||||
@@ -361,13 +372,15 @@ mod tests {
|
|||||||
// Kick off dispatch 1 in the background — it holds the permit
|
// Kick off dispatch 1 in the background — it holds the permit
|
||||||
// for ~200 ms.
|
// for ~200 ms.
|
||||||
let entry_bg = entry.clone();
|
let entry_bg = entry.clone();
|
||||||
let bg = tokio::spawn(async move { dispatch("overrun", entry_bg).await });
|
let bg = tokio::spawn(async move {
|
||||||
|
dispatch("overrun", entry_bg, &JobRunArgs::default()).await
|
||||||
|
});
|
||||||
|
|
||||||
// Give dispatch 1 time to grab the permit.
|
// Give dispatch 1 time to grab the permit.
|
||||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||||
|
|
||||||
// Dispatch 2 should observe the permit taken and skip.
|
// Dispatch 2 should observe the permit taken and skip.
|
||||||
dispatch("overrun", entry.clone()).await;
|
dispatch("overrun", entry.clone(), &JobRunArgs::default()).await;
|
||||||
|
|
||||||
// Only dispatch 1's handler should have actually run so far.
|
// Only dispatch 1's handler should have actually run so far.
|
||||||
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
assert_eq!(calls.load(Ordering::SeqCst), 1);
|
||||||
@@ -396,7 +409,7 @@ mod tests {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
let entry = registry.get("slow").await.unwrap();
|
let entry = registry.get("slow").await.unwrap();
|
||||||
|
|
||||||
dispatch("slow", entry.clone()).await;
|
dispatch("slow", entry.clone(), &JobRunArgs::default()).await;
|
||||||
|
|
||||||
// The timeout fired; last_outcome must be Err.
|
// The timeout fired; last_outcome must be Err.
|
||||||
let state = entry.state.lock().unwrap();
|
let state = entry.state.lock().unwrap();
|
||||||
|
|||||||
@@ -3,12 +3,13 @@
|
|||||||
//! Everything a native service needs to write to plug into the periodic
|
//! Everything a native service needs to write to plug into the periodic
|
||||||
//! scheduler is on this page. See `docs/plan/job-registry.md` Part 1
|
//! scheduler is on this page. See `docs/plan/job-registry.md` Part 1
|
||||||
//! for the design rationale and migration criterion (the "operator
|
//! for the design rationale and migration criterion (the "operator
|
||||||
//! trigger" question — if an operator would never `POST /trigger-job`
|
//! trigger" question — if an operator would never
|
||||||
//! for this loop, it doesn't belong here; keep it as a core worker).
|
//! `POST /api/admin/jobs/{name}/trigger` for this loop, it doesn't
|
||||||
|
//! belong here; keep it as a core worker).
|
||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
|
||||||
use super::types::JobOutcome;
|
use super::types::{JobOutcome, JobRunArgs};
|
||||||
|
|
||||||
/// Implemented by every service that wants to run on a fixed interval
|
/// Implemented by every service that wants to run on a fixed interval
|
||||||
/// through the periodic scheduler.
|
/// through the periodic scheduler.
|
||||||
@@ -31,7 +32,7 @@ use super::types::JobOutcome;
|
|||||||
///
|
///
|
||||||
/// Return a stable, unique snake_case identifier. Log lines
|
/// Return a stable, unique snake_case identifier. Log lines
|
||||||
/// (`job = %name`), admin listing, admin trigger URLs
|
/// (`job = %name`), admin listing, admin trigger URLs
|
||||||
/// (`POST /api/admin/internal/trigger-job/{name}`) and env vars
|
/// (`POST /api/admin/jobs/{name}/trigger`) and env vars
|
||||||
/// (`OXICLOUD_JOB_<NAME>_INTERVAL_HOURS`) all key on this. Renaming
|
/// (`OXICLOUD_JOB_<NAME>_INTERVAL_HOURS`) all key on this. Renaming
|
||||||
/// after release is a breaking change to operator scripts and log
|
/// after release is a breaking change to operator scripts and log
|
||||||
/// dashboards.
|
/// dashboards.
|
||||||
@@ -64,7 +65,15 @@ pub trait JobHandler: Send + Sync {
|
|||||||
fn name(&self) -> &str;
|
fn name(&self) -> &str;
|
||||||
|
|
||||||
/// One execution. Called at the registered interval and (optionally)
|
/// One execution. Called at the registered interval and (optionally)
|
||||||
/// on admin trigger. See trait-level docs for guidance on when to
|
/// on admin trigger.
|
||||||
/// return Ok vs Err.
|
///
|
||||||
async fn run(&self) -> JobOutcome;
|
/// `args` carries per-dispatch parameters (`force: bool` today).
|
||||||
|
/// Periodic ticks pass [`JobRunArgs::default()`]; admin triggers
|
||||||
|
/// forward query params such as `?force=true`. Handlers that don't
|
||||||
|
/// understand a given arg silently ignore it — the arg exists to
|
||||||
|
/// give per-job acceleration semantics without spreading per-job
|
||||||
|
/// knowledge into every caller.
|
||||||
|
///
|
||||||
|
/// See trait-level docs for guidance on when to return Ok vs Err.
|
||||||
|
async fn run(&self, args: &JobRunArgs) -> JobOutcome;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -29,5 +29,5 @@ mod types;
|
|||||||
|
|
||||||
pub use engine::SchedulerEngine;
|
pub use engine::SchedulerEngine;
|
||||||
pub use handler::JobHandler;
|
pub use handler::JobHandler;
|
||||||
pub use registry::{JobEntry, JobRegistry, RegisterError};
|
pub use registry::{JobEntry, JobRegistry, JobSummary, RegisterError};
|
||||||
pub use types::{ErrCause, JobOutcome};
|
pub use types::{ErrCause, JobOutcome, JobRunArgs};
|
||||||
|
|||||||
@@ -16,10 +16,11 @@ use std::sync::{Arc, Mutex};
|
|||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
|
use serde::Serialize;
|
||||||
use tokio::sync::{RwLock, Semaphore};
|
use tokio::sync::{RwLock, Semaphore};
|
||||||
|
|
||||||
use super::handler::JobHandler;
|
use super::handler::JobHandler;
|
||||||
use super::types::JobOutcome;
|
use super::types::{JobOutcome, JobRunArgs};
|
||||||
|
|
||||||
/// A registered job plus its runtime state. Held as `Arc<JobEntry>`
|
/// A registered job plus its runtime state. Held as `Arc<JobEntry>`
|
||||||
/// inside the registry so the engine can hold a snapshot across an
|
/// inside the registry so the engine can hold a snapshot across an
|
||||||
@@ -158,6 +159,32 @@ impl JobRegistry {
|
|||||||
guard.iter().map(|(k, v)| (k.clone(), v.clone())).collect()
|
guard.iter().map(|(k, v)| (k.clone(), v.clone())).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Serialisable snapshot for `GET /api/admin/jobs`. Each entry
|
||||||
|
/// captures the operator-visible state: interval (null for on-
|
||||||
|
/// demand), next scheduled dispatch (null for on-demand), when
|
||||||
|
/// the last run started, and its outcome.
|
||||||
|
pub async fn snapshot(&self) -> Vec<JobSummary> {
|
||||||
|
let entries = self.snapshot_all().await;
|
||||||
|
entries
|
||||||
|
.into_iter()
|
||||||
|
.map(|(name, entry)| {
|
||||||
|
let state = entry.state.lock().expect("JobState mutex poisoned");
|
||||||
|
let (last_run_at, last_outcome) = match &state.last_outcome {
|
||||||
|
Some((at, outcome)) => (Some(*at), Some(outcome.clone())),
|
||||||
|
None => (None, None),
|
||||||
|
};
|
||||||
|
JobSummary {
|
||||||
|
name,
|
||||||
|
interval_ms: entry.interval.map(|d| d.as_millis() as u64),
|
||||||
|
next_run_at: state.next_run_at,
|
||||||
|
last_run_at,
|
||||||
|
last_outcome,
|
||||||
|
running: state.current_run_start.is_some(),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
/// Count of registered jobs — used for the startup log line.
|
/// Count of registered jobs — used for the startup log line.
|
||||||
pub async fn len(&self) -> usize {
|
pub async fn len(&self) -> usize {
|
||||||
self.entries.read().await.len()
|
self.entries.read().await.len()
|
||||||
@@ -171,7 +198,7 @@ impl JobRegistry {
|
|||||||
/// Manual dispatch — the single entry point for running a
|
/// Manual dispatch — the single entry point for running a
|
||||||
/// registered job outside the scheduler's tick loop. Called by:
|
/// registered job outside the scheduler's tick loop. Called by:
|
||||||
///
|
///
|
||||||
/// - The admin endpoint `POST /api/admin/internal/trigger-job/{name}`.
|
/// - The admin endpoint `POST /api/admin/jobs/{name}/trigger`.
|
||||||
/// - Any service that wants a scheduler-uniform dispatch of a
|
/// - Any service that wants a scheduler-uniform dispatch of a
|
||||||
/// peer job (uniform log line, exclusivity, panic containment,
|
/// peer job (uniform log line, exclusivity, panic containment,
|
||||||
/// timeout enforcement).
|
/// timeout enforcement).
|
||||||
@@ -185,9 +212,17 @@ impl JobRegistry {
|
|||||||
///
|
///
|
||||||
/// Works for BOTH scheduled and on-demand jobs — for on-demand
|
/// Works for BOTH scheduled and on-demand jobs — for on-demand
|
||||||
/// jobs this is the only way they ever run.
|
/// jobs this is the only way they ever run.
|
||||||
pub async fn trigger(self: &Arc<Self>, name: &str) -> Option<JobOutcome> {
|
///
|
||||||
|
/// `args` is forwarded to `JobHandler::run`. Admin trigger routes
|
||||||
|
/// use `JobRunArgs { force: query.force }`; programmatic callers
|
||||||
|
/// that just want a plain run pass `JobRunArgs::default()`.
|
||||||
|
pub async fn trigger(
|
||||||
|
self: &Arc<Self>,
|
||||||
|
name: &str,
|
||||||
|
args: &JobRunArgs,
|
||||||
|
) -> Option<JobOutcome> {
|
||||||
let entry = self.get(name).await?;
|
let entry = self.get(name).await?;
|
||||||
Some(super::engine::dispatch(name, entry).await)
|
Some(super::engine::dispatch(name, entry, args).await)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -203,6 +238,29 @@ pub enum RegisterError {
|
|||||||
DuplicateName(String),
|
DuplicateName(String),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Per-job row in the `GET /api/admin/jobs` response.
|
||||||
|
///
|
||||||
|
/// - `interval_ms` — periodic cadence; `null` for on-demand jobs.
|
||||||
|
/// - `next_run_at` — next scheduled dispatch; `null` for on-demand.
|
||||||
|
/// - `last_run_at` / `last_outcome` — most recent completed run;
|
||||||
|
/// `null` until the first run finishes.
|
||||||
|
/// - `running` — true iff the in-flight permit is currently held
|
||||||
|
/// (either the supervisor tick is in progress or an admin trigger
|
||||||
|
/// raced in).
|
||||||
|
#[derive(Debug, Clone, Serialize)]
|
||||||
|
pub struct JobSummary {
|
||||||
|
pub name: String,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub interval_ms: Option<u64>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub next_run_at: Option<DateTime<Utc>>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub last_run_at: Option<DateTime<Utc>>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub last_outcome: Option<JobOutcome>,
|
||||||
|
pub running: bool,
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
@@ -217,7 +275,7 @@ mod tests {
|
|||||||
fn name(&self) -> &str {
|
fn name(&self) -> &str {
|
||||||
&self.name
|
&self.name
|
||||||
}
|
}
|
||||||
async fn run(&self) -> JobOutcome {
|
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
|
||||||
JobOutcome::ok(0)
|
JobOutcome::ok(0)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -299,13 +357,20 @@ mod tests {
|
|||||||
let reg = Arc::new(JobRegistry::new());
|
let reg = Arc::new(JobRegistry::new());
|
||||||
reg.register(handler("gc"), None, None).await.unwrap();
|
reg.register(handler("gc"), None, None).await.unwrap();
|
||||||
|
|
||||||
let outcome = reg.trigger("gc").await.expect("job exists");
|
let outcome = reg
|
||||||
|
.trigger("gc", &JobRunArgs::default())
|
||||||
|
.await
|
||||||
|
.expect("job exists");
|
||||||
assert!(outcome.is_ok());
|
assert!(outcome.is_ok());
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn trigger_returns_none_for_unknown_job() {
|
async fn trigger_returns_none_for_unknown_job() {
|
||||||
let reg = Arc::new(JobRegistry::new());
|
let reg = Arc::new(JobRegistry::new());
|
||||||
assert!(reg.trigger("nope").await.is_none());
|
assert!(
|
||||||
|
reg.trigger("nope", &JobRunArgs::default())
|
||||||
|
.await
|
||||||
|
.is_none()
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,6 +9,27 @@ use std::fmt;
|
|||||||
|
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
/// Per-dispatch parameters passed from the caller (scheduler tick or
|
||||||
|
/// admin trigger) into [`JobHandler::run`](super::handler::JobHandler::run).
|
||||||
|
///
|
||||||
|
/// Deliberately a struct — not a bare `bool` — so we don't churn every
|
||||||
|
/// handler signature the next time a job needs another knob. Grows by
|
||||||
|
/// addition; renaming a field is a breaking change to admin scripts
|
||||||
|
/// that pass query params, so treat like SQL columns.
|
||||||
|
///
|
||||||
|
/// **Handlers that don't understand a given arg silently ignore it.**
|
||||||
|
/// No error path just because a caller set an unused flag — that would
|
||||||
|
/// leak per-job semantics into callers who don't need to know.
|
||||||
|
///
|
||||||
|
/// Semantics of `force`, per job:
|
||||||
|
/// - `dedup_gc` — skip the orphan grace window (grace = 0).
|
||||||
|
/// - `grant_cleanup` — grace = 0.
|
||||||
|
/// - Others (trash_cleanup, storage_reconcile, …) — ignored.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct JobRunArgs {
|
||||||
|
pub force: bool,
|
||||||
|
}
|
||||||
|
|
||||||
/// Uniform outcome the supervisor logs and stores for every job dispatch.
|
/// Uniform outcome the supervisor logs and stores for every job dispatch.
|
||||||
///
|
///
|
||||||
/// Two variants, deliberately. Distinguishing *why* a job failed
|
/// Two variants, deliberately. Distinguishing *why* a job failed
|
||||||
|
|||||||
@@ -3124,7 +3124,7 @@ impl DedupPort for DedupService {
|
|||||||
|
|
||||||
/// Registered name for the dedup GC job. Stable identifier used in
|
/// Registered name for the dedup GC job. Stable identifier used in
|
||||||
/// log lines, `admin.background_runs.job_name` (when Part 2 lands),
|
/// log lines, `admin.background_runs.job_name` (when Part 2 lands),
|
||||||
/// and admin URLs (`POST /api/admin/internal/trigger-job/dedup_gc`).
|
/// and admin URLs (`POST /api/admin/jobs/dedup_gc/trigger`).
|
||||||
pub const DEDUP_GC_JOB_NAME: &str = "dedup_gc";
|
pub const DEDUP_GC_JOB_NAME: &str = "dedup_gc";
|
||||||
|
|
||||||
#[async_trait::async_trait]
|
#[async_trait::async_trait]
|
||||||
@@ -3136,7 +3136,7 @@ impl crate::infrastructure::scheduler::JobHandler for DedupService {
|
|||||||
/// Runs one `garbage_collect` sweep — the same reclamation that
|
/// Runs one `garbage_collect` sweep — the same reclamation that
|
||||||
/// `TrashCleanupService` invokes inline as its tail step, exposed
|
/// `TrashCleanupService` invokes inline as its tail step, exposed
|
||||||
/// through the scheduler so operators can trigger it uniformly via
|
/// through the scheduler so operators can trigger it uniformly via
|
||||||
/// `POST /api/admin/internal/trigger-job/dedup_gc`.
|
/// `POST /api/admin/jobs/dedup_gc/trigger`.
|
||||||
///
|
///
|
||||||
/// Registered with `interval = None` (on-demand only): the periodic
|
/// Registered with `interval = None` (on-demand only): the periodic
|
||||||
/// tick belongs to trash cleanup, whose sweep already runs GC as
|
/// tick belongs to trash cleanup, whose sweep already runs GC as
|
||||||
@@ -3148,12 +3148,28 @@ impl crate::infrastructure::scheduler::JobHandler for DedupService {
|
|||||||
/// `count` reports blobs reclaimed; `extra.bytes_reclaimed` reports
|
/// `count` reports blobs reclaimed; `extra.bytes_reclaimed` reports
|
||||||
/// the freed disk. GC returning `(0, 0)` is normal — it means trash
|
/// the freed disk. GC returning `(0, 0)` is normal — it means trash
|
||||||
/// cleanup already reaped everything.
|
/// cleanup already reaped everything.
|
||||||
async fn run(&self) -> crate::infrastructure::scheduler::JobOutcome {
|
///
|
||||||
|
/// `args.force = true` skips the orphan grace window
|
||||||
|
/// (`garbage_collect_force` — grace_secs = 0), matching the legacy
|
||||||
|
/// `POST /admin/internal/trigger-gc?force=true` semantics. Unsafe
|
||||||
|
/// under concurrent uploads: only reachable through the admin
|
||||||
|
/// endpoint and only intentionally used by tests + operator
|
||||||
|
/// diagnostic sessions.
|
||||||
|
async fn run(
|
||||||
|
&self,
|
||||||
|
args: &crate::infrastructure::scheduler::JobRunArgs,
|
||||||
|
) -> crate::infrastructure::scheduler::JobOutcome {
|
||||||
use crate::infrastructure::scheduler::JobOutcome;
|
use crate::infrastructure::scheduler::JobOutcome;
|
||||||
match self.garbage_collect().await {
|
let result = if args.force {
|
||||||
Ok((items, bytes)) => {
|
self.garbage_collect_force().await
|
||||||
JobOutcome::ok_with(items, serde_json::json!({ "bytes_reclaimed": bytes }))
|
} else {
|
||||||
}
|
self.garbage_collect().await
|
||||||
|
};
|
||||||
|
match result {
|
||||||
|
Ok((items, bytes)) => JobOutcome::ok_with(
|
||||||
|
items,
|
||||||
|
serde_json::json!({ "bytes_reclaimed": bytes, "forced": args.force }),
|
||||||
|
),
|
||||||
Err(e) => JobOutcome::Err(format!("dedup GC failed: {e}")),
|
Err(e) => JobOutcome::Err(format!("dedup GC failed: {e}")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ use tracing::{error, info};
|
|||||||
|
|
||||||
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
use crate::application::ports::authorization_ports::AuthorizationEngine;
|
||||||
use crate::common::errors::DomainError;
|
use crate::common::errors::DomainError;
|
||||||
use crate::infrastructure::scheduler::{JobHandler, JobOutcome};
|
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRunArgs};
|
||||||
use crate::infrastructure::services::pg_acl_engine::PgAclEngine;
|
use crate::infrastructure::services::pg_acl_engine::PgAclEngine;
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
|
||||||
@@ -112,19 +112,26 @@ impl JobHandler for GrantCleanupService {
|
|||||||
GRANT_CLEANUP_JOB_NAME
|
GRANT_CLEANUP_JOB_NAME
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Runs one purge with the configured grace window. `count` on the
|
/// Runs one purge. `count` on the returned `JobOutcome::Ok` is
|
||||||
/// returned `JobOutcome::Ok` is the number of `role_grants` rows
|
/// the number of `role_grants` rows physically deleted;
|
||||||
/// physically deleted; `extra.grace_days` records which grace was
|
/// `extra.grace_days` records which grace was applied so admin
|
||||||
/// applied so admin listings can see it without a second lookup.
|
/// listings can see it without a second lookup.
|
||||||
///
|
///
|
||||||
/// Admin `?force=true` (grace = 0) does NOT come through here —
|
/// `args.force = true` collapses the grace window to zero for
|
||||||
/// that path calls `purge(Some(0))` directly on the shared
|
/// this run only — matches the legacy
|
||||||
/// `Arc<GrantCleanupService>` from the handler.
|
/// `POST /admin/internal/trigger-grant-cleanup?force=true` shape.
|
||||||
async fn run(&self) -> JobOutcome {
|
/// The configured `self.grace_days` is not mutated.
|
||||||
match self.purge(None).await {
|
async fn run(&self, args: &JobRunArgs) -> JobOutcome {
|
||||||
Ok(count) => {
|
let grace_override = if args.force { Some(0) } else { None };
|
||||||
JobOutcome::ok_with(count, serde_json::json!({ "grace_days": self.grace_days }))
|
let effective_grace = grace_override.unwrap_or(self.grace_days);
|
||||||
}
|
match self.purge(grace_override).await {
|
||||||
|
Ok(count) => JobOutcome::ok_with(
|
||||||
|
count,
|
||||||
|
serde_json::json!({
|
||||||
|
"grace_days": effective_grace,
|
||||||
|
"forced": args.force,
|
||||||
|
}),
|
||||||
|
),
|
||||||
Err(e) => JobOutcome::Err(format!("grant cleanup failed: {e}")),
|
Err(e) => JobOutcome::Err(format!("grant cleanup failed: {e}")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ use tracing::{debug, error, info, instrument};
|
|||||||
use crate::common::errors::Result;
|
use crate::common::errors::Result;
|
||||||
use crate::domain::repositories::trash_repository::TrashRepository;
|
use crate::domain::repositories::trash_repository::TrashRepository;
|
||||||
use crate::infrastructure::repositories::pg::trash_db_repository::TrashDbRepository;
|
use crate::infrastructure::repositories::pg::trash_db_repository::TrashDbRepository;
|
||||||
use crate::infrastructure::scheduler::{JobHandler, JobOutcome};
|
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRunArgs};
|
||||||
use crate::infrastructure::services::dedup_service::DedupService;
|
use crate::infrastructure::services::dedup_service::DedupService;
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
|
||||||
@@ -165,7 +165,11 @@ impl JobHandler for TrashCleanupService {
|
|||||||
///
|
///
|
||||||
/// Failure of the trash sweep itself → `Err`. GC failure alone is
|
/// Failure of the trash sweep itself → `Err`. GC failure alone is
|
||||||
/// non-fatal and stays logged only.
|
/// non-fatal and stays logged only.
|
||||||
async fn run(&self) -> JobOutcome {
|
///
|
||||||
|
/// `args.force` is ignored — trash cleanup has no acceleration
|
||||||
|
/// concept (retention windows are per-item metadata, not a runtime
|
||||||
|
/// knob).
|
||||||
|
async fn run(&self, _args: &JobRunArgs) -> JobOutcome {
|
||||||
match self.run_once().await {
|
match self.run_once().await {
|
||||||
Ok(stats) => {
|
Ok(stats) => {
|
||||||
let removed = stats.files_purged + stats.folders_purged;
|
let removed = stats.files_purged + stats.folders_purged;
|
||||||
|
|||||||
@@ -155,6 +155,12 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
|
|||||||
"/internal/trigger-grant-cleanup",
|
"/internal/trigger-grant-cleanup",
|
||||||
post(internal_trigger_grant_cleanup),
|
post(internal_trigger_grant_cleanup),
|
||||||
)
|
)
|
||||||
|
// JobRegistry admin surface — production, always-on,
|
||||||
|
// audit-logged. See `docs/plan/job-registry.md` §Cross-cutting.
|
||||||
|
// The `/internal/trigger-*` shims above will be retired in a
|
||||||
|
// follow-up PR (deprecated forwards to these endpoints).
|
||||||
|
.route("/jobs", get(list_jobs))
|
||||||
|
.route("/jobs/{name}/trigger", post(trigger_job))
|
||||||
// Drives — admin-wide view (distinct from `/api/drives` which
|
// Drives — admin-wide view (distinct from `/api/drives` which
|
||||||
// is filtered to the caller's role grants).
|
// is filtered to the caller's role grants).
|
||||||
.route("/drives", get(list_all_drives))
|
.route("/drives", get(list_all_drives))
|
||||||
@@ -2298,3 +2304,98 @@ pub async fn internal_trigger_grant_cleanup(
|
|||||||
)
|
)
|
||||||
.into_response()
|
.into_response()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
// JobRegistry admin surface (`/api/admin/jobs/*`)
|
||||||
|
// ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// `GET /api/admin/jobs` — enumerate every registered job with its
|
||||||
|
/// interval, next-run/last-run timestamps, and last outcome.
|
||||||
|
///
|
||||||
|
/// Production endpoint (no `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS`
|
||||||
|
/// gate). Read-only, so no audit line — the standard admin-middleware
|
||||||
|
/// auth check is enough.
|
||||||
|
#[utoipa::path(
|
||||||
|
get,
|
||||||
|
path = "/api/admin/jobs",
|
||||||
|
responses(
|
||||||
|
(status = 200, description = "Jobs listed"),
|
||||||
|
(status = 401, description = "Unauthorized"),
|
||||||
|
(status = 403, description = "Admin required"),
|
||||||
|
),
|
||||||
|
security(("bearerAuth" = [])),
|
||||||
|
tag = "admin"
|
||||||
|
)]
|
||||||
|
pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse {
|
||||||
|
let summary = state.core.job_registry.snapshot().await;
|
||||||
|
(StatusCode::OK, Json(summary)).into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query parameters for `POST /api/admin/jobs/{name}/trigger`.
|
||||||
|
///
|
||||||
|
/// `force=true` requests acceleration semantics from handlers that
|
||||||
|
/// support it (dedup_gc → grace = 0, grant_cleanup → grace = 0).
|
||||||
|
/// Silently ignored by handlers that don't (trash_cleanup,
|
||||||
|
/// storage_reconcile).
|
||||||
|
#[derive(serde::Deserialize)]
|
||||||
|
pub struct TriggerJobQuery {
|
||||||
|
#[serde(default)]
|
||||||
|
pub force: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `POST /api/admin/jobs/{name}/trigger` — dispatch one run off-schedule.
|
||||||
|
///
|
||||||
|
/// Returns the job's `JobOutcome` inline. Idempotent under exclusivity:
|
||||||
|
/// if the previous run is still in flight, the handler returns
|
||||||
|
/// `Ok { count: 0, extra: { "skipped": "already_running" } }` rather
|
||||||
|
/// than spawning a parallel dispatch.
|
||||||
|
///
|
||||||
|
/// Emits an audit line before dispatch — bulk-mutation side effects on
|
||||||
|
/// operator command belong on the audit stream.
|
||||||
|
#[utoipa::path(
|
||||||
|
post,
|
||||||
|
path = "/api/admin/jobs/{name}/trigger",
|
||||||
|
params(("name" = String, Path, description = "Registered job name")),
|
||||||
|
responses(
|
||||||
|
(status = 200, description = "Dispatched; outcome inline"),
|
||||||
|
(status = 401, description = "Unauthorized"),
|
||||||
|
(status = 403, description = "Admin required"),
|
||||||
|
(status = 404, description = "Job not registered"),
|
||||||
|
),
|
||||||
|
security(("bearerAuth" = [])),
|
||||||
|
tag = "admin"
|
||||||
|
)]
|
||||||
|
pub async fn trigger_job(
|
||||||
|
State(state): State<Arc<AppState>>,
|
||||||
|
axum::extract::Path(name): axum::extract::Path<String>,
|
||||||
|
axum::extract::Query(query): axum::extract::Query<TriggerJobQuery>,
|
||||||
|
) -> impl IntoResponse {
|
||||||
|
use crate::infrastructure::scheduler::JobRunArgs;
|
||||||
|
// Audit line BEFORE dispatch so an operator triggering something
|
||||||
|
// that then hangs still leaves a trail.
|
||||||
|
tracing::info!(
|
||||||
|
target: "audit",
|
||||||
|
event = "job.trigger",
|
||||||
|
job = %name,
|
||||||
|
force = query.force,
|
||||||
|
"👮🏻♂️ Admin triggered job {} (force={})",
|
||||||
|
name,
|
||||||
|
query.force,
|
||||||
|
);
|
||||||
|
let args = JobRunArgs { force: query.force };
|
||||||
|
match state.core.job_registry.trigger(&name, &args).await {
|
||||||
|
Some(outcome) => (
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(serde_json::json!({ "ok": true, "outcome": outcome })),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
None => (
|
||||||
|
StatusCode::NOT_FOUND,
|
||||||
|
Json(serde_json::json!({
|
||||||
|
"error": "job not registered",
|
||||||
|
"name": name,
|
||||||
|
})),
|
||||||
|
)
|
||||||
|
.into_response(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user