cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(admin): separate a job's run STATE from its OUTCOME

Browse Source 
Ed's diagnosis, and it is the root of three symptoms I had been patching
one at a time: a job has two independent statuses, and the panel was
collapsing them into one column.

  * STATE — where the run is in its lifecycle: running, paused,
    cancelled, completed, failed.
  * OUTCOME — how the work turned out: ok, issues, notices, err.

They are orthogonal. A paused run has no outcome yet. A completed run's
outcome may still be "issues". Conflating them produced, in order:

  1. a paused migration rendering as a green "ok" — the outcome was
     genuinely ok, the STATE was Paused, and only the outcome was shown;
  2. my first fix, which put "blocked" into the OUTCOME column — a
     category error, encoding lifecycle into the result axis;
  3. a cancelled job still reading "blocked", because that outcome was
     cached in memory while the cancel had flipped the row in SQL.

The layout already had both columns. State just never rendered anything
but "running" or "—", so the status axis had no home and the information
leaked into Outcome.

Now:

  * State renders `last_run_status`, sourced from the run ROW. Memory
    cannot answer this — it is empty after a restart and stale after a
    cancel, both of which the row gets right. The retryable reason, when
    there is one, is the pill's tooltip.
  * Outcome goes back to describing only the work: ok / issues /
    notices / err. No lifecycle in it.

`JobSummary` gains `last_run_status`, and `last_run_at` falls back to
the row's `started_at` when memory has none — a restart left the column
reading "never" for a job whose last run was hours earlier.

"never" is now reserved for jobs that genuinely never ran. With a run
row present but no cached outcome the cell reads "—": the honest "no
outcome recorded", rather than a claim the run history immediately
contradicts.

The enrichment query generalises rather than multiplying — it already
fetched Paused rows for the Resume button, so it now takes the latest
row per job via `DISTINCT ON` and derives state, timestamp and paused
brief from it. Sound as "the current run" because the
`one_active_run_per_job` partial unique index permits one non-terminal
row per job and a resume reuses it, so a non-terminal row is always
newest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-09-07 21:48:21 +02:00
parent e054987c65
commit 0cdb2bb0a9
4 changed files with 140 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/infrastructure/scheduler/registry.rs
+14 -1
View File
@@ -249,11 +249,12 @@ impl JobRegistry {
last_outcome,
running: state.current_run_start.is_some(),
recoverable: entry.handler.is_recoverable(),
// Both populated in the `list_jobs` handler — one
// All populated in the `list_jobs` handler — two
// from a DB round-trip, one from AppConfig. Kept
// out of the registry snapshot so the in-memory
// scheduler state pulls in neither dependency.
paused_run: None,
last_run_status: None,
startup: None,
}
})
@@ -349,6 +350,18 @@ pub struct JobSummary {
/// job, most of which the job ignored with no way to tell.
#[serde(skip_serializing_if = "<[_]>::is_empty")]
pub parameters: &'static [JobParam],
/// Status of this job's most recent run row, for recoverable jobs.
///
/// Populated by the `list_jobs` handler from the DB, and it exists
/// because [`Self::last_outcome`] cannot answer this: that field is
/// in-memory, written when a dispatch completes through the engine,
/// so anything changing a run row without running the handler leaves
/// it stale. Cancelling a Paused run is exactly that — a direct SQL
/// flip — and the panel went on showing the pause's outcome.
///
/// Prefer this over `last_outcome` wherever the two could disagree.
#[serde(skip_serializing_if = "Option::is_none")]
pub last_run_status: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub interval_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
src/interfaces/api/handlers/admin_handler.rs
+68 -15
View File
@@ -2570,41 +2570,94 @@ pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse
// failures fall back to the pre-enrichment shape so the endpoint
// stays useful when the jobs DB is temporarily unreachable.
if let Some(pool) = state.db_pool.as_ref() {
let paused_rows: Vec<(String, uuid::Uuid, Option<i64>, Option<i64>)> = sqlx::query_as(
// The LATEST run per job, whatever its status — not just the
// paused ones.
//
// `last_outcome` is in-memory, written when a dispatch finishes
// through the engine. Anything that changes a run row WITHOUT
// running the handler leaves it stale: cancelling a Paused run
// is a direct SQL flip to `Cancelled`, so the panel kept
// rendering the outcome of the run that pause belonged to — a
// cancelled job still showing "blocked".
//
// `DISTINCT ON` is safe as "the current run": the
// `one_active_run_per_job` partial unique index allows only one
// non-terminal row per job, and a resume reuses it rather than
// starting a new one, so a non-terminal row is always the newest.
/// `(job_name, status, run_id, started_at, scanned, total)` — the
/// enrichment row shape, named so the query's type stays legible.
type LatestRunRow = (
String,
String,
uuid::Uuid,
chrono::DateTime<chrono::Utc>,
Option<i64>,
Option<i64>,
);
let latest_rows: Vec<LatestRunRow> = sqlx::query_as(
r#"
SELECT
SELECT DISTINCT ON (job_name)
job_name,
status::TEXT,
id,
started_at,
(stats ->> 'scanned_count')::BIGINT AS scanned,
(params ->> 'total_rows')::BIGINT AS total
FROM jobs.recoverable_runs
WHERE status = 'Paused'
ORDER BY job_name, started_at DESC
"#,
)
.fetch_all(pool.as_ref())
.await
.unwrap_or_default();
let by_name: std::collections::HashMap<String, PausedRunBrief> = paused_rows
type LatestRun = (String, chrono::DateTime<chrono::Utc>, PausedRunBrief);
let by_name: std::collections::HashMap<String, LatestRun> = latest_rows
.into_iter()
.map(|(name, id, scanned, total)| {
.map(|(name, status, id, started_at, scanned, total)| {
(
name,
PausedRunBrief {
id,
scanned: scanned.unwrap_or(0).max(0) as u64,
total: total.filter(|t| *t > 0).map(|t| t as u64),
},
(
status,
started_at,
PausedRunBrief {
id,
scanned: scanned.unwrap_or(0).max(0) as u64,
total: total.filter(|t| *t > 0).map(|t| t as u64),
},
),
)
})
.collect();
for job in summary.iter_mut() {
if job.recoverable
&& !job.running
&& let Some(paused) = by_name.get(&job.name)
{
job.paused_run = Some(paused.clone());
if !job.recoverable {
continue;
}
let Some((status, started_at, brief)) = by_name.get(&job.name) else {
continue;
};
// Always reported, so the panel can prefer the row's truth
// over the in-memory outcome rather than guessing which is
// fresher.
job.last_run_status = Some(status.clone());
// Fill the timestamp too when memory has none.
//
// `last_outcome` and `last_run_at` are both in-memory, so a
// restart empties them and the row read "never" for a job
// with real runs in the DB — the opposite failure to the
// stale-outcome one, and just as misleading. The row is
// authoritative for "did this ever run"; memory only adds
// the richer outcome detail when it happens to be warm.
//
// Only when absent: a warm `last_run_at` describes the last
// DISPATCH, which for a non-recoverable tick is finer-grained
// than any run row.
if job.last_run_at.is_none() {
job.last_run_at = Some(*started_at);
}
if !job.running && status == "Paused" {
job.paused_run = Some(brief.clone());
}
}
}