Oxicloud/docs/plan/sessions.md

# Session Liveness Tracking — `last_seen_at` + Prometheus

Track per-session and per-user "currently active" signals cheaply, and
expose them as Prometheus gauges so an operator (demo instance,
production) can plot concurrency over time. Also unlocks future
features that need "when was this session last used" (idle-timeout
enforcement, per-user session-limit quotas, admin dashboard freshness).

Companion doc for the design decisions covered here — narrative on the
overall session model lives in
[docs/architecture/auth-model.md](../architecture/auth-model.md).

## Purpose — what we want to see

Two distinct signals, deliberately separate:

1. **Online sessions** (`oxicloud_sessions_online`) — count of
   non-revoked `auth.sessions` rows that had a request within the last
   N minutes. One user with three devices (browser + phone + Nextcloud
   desktop) contributes **three** to this count. Useful for
   provisioning ("how many concurrent connections do I need to
   support?") and load-shape planning.

2. **Online users** (`oxicloud_sessions_online_users`) — count of
   DISTINCT `user_id` values behind those online sessions. Same
   three-device user contributes **one** to this count. Useful for
   billing shape ("how many humans are actually using the system?")
   and for the demo landing page's "N users online right now" widget.

The gap between the two IS the multi-device factor. A healthy system
where users routinely have web + desktop client should show `sessions
≈ 2 × users`. A sudden `sessions >> users × 3` is a signal — an app
that opens fresh sessions instead of reusing them, or a
credential-stuffing pattern.

### Terminology — "online" vs "active"

The admin sessions panel already has a lifecycle filter
`Active | Expired | Revoked` — where **active** means
`!revoked && !expired` (row is still usable). That's orthogonal to
"had a request lately", so both concepts fighting for the same word
was going to confuse admins reading the panel.

**Decision (Ed, 2026-08-18)** — "online" is the *presence* signal
throughout the stack:

- **UI**: green-dot badge next to each row when
  `SessionSummaryDto::is_online == true`; grey dot + "last seen X ago"
  otherwise. Lifecycle filter stays `Active | Expired | Revoked`
  unchanged.
- **DTO**: `is_online: bool` on `SessionSummaryDto`, computed
  server-side (avoids the SPA doing clock math and drifting from the
  server view). Guaranteed `false` on revoked / expired rows so an
  admin never sees "Online" on a row they just revoked.
- **Metrics**: `oxicloud_sessions_online` / `_online_users`.
- **Threshold**: single `pub const ONLINE_WINDOW` in
  `application/dtos/session_dto.rs` — DTO derivation AND gauge query
  read from the same constant so the per-row badge count and the
  gauge aggregate stay consistent by construction.

## Why the existing signals don't answer this — DECIDED

`auth.sessions.created_at` is the closest existing proxy. But it
moves on **session rotation**, not per-request:

- Sessions rotate on every silent refresh (`apiFetch`'s 401 → refresh
  path). Rotation cadence = `access_token_expiry_secs` (default
  3600, i.e. 1 h).
- So `WHERE created_at > NOW() - INTERVAL '1 hour'` catches everyone
  who refreshed in the last cycle — but a user who's actively clicking
  around for 45 min hasn't rotated yet, so their `created_at` is 45
  min old. Threshold `< 30 min` false-negatives them.
- Resolution is capped at the access-token TTL. At the recommended
  prod value of 15 min, `created_at` gives 15-min granularity. At the
  test value of 60 s it's near-real-time — but no operator wants
  to force 60 s token TTL just for observability.

So `created_at` is an OK first-pass proxy but bad enough that we
should add a dedicated column that moves per-request.

## Schema — `last_seen_at`

Migration `<TS>_sessions_last_seen_at.sql`:

```sql
ALTER TABLE auth.sessions
    ADD COLUMN last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW();

-- Partial index — the "active in the last N min" query is the only
-- reason to scan this column, and it always filters on revoked = FALSE.
-- Full index would double the write cost for zero read benefit.
CREATE INDEX idx_sessions_last_seen_at ON auth.sessions(last_seen_at)
    WHERE revoked = FALSE;
```

Default `NOW()` — existing rows on migration land at "just seen"
which is slightly optimistic but the alternative (NULL / epoch) makes
every historic session count as long-idle for the first N min after
deploy. `NOW()` matches "assume everyone's active" and the metric
converges to reality within one N-min bucket.

**Not in scope for this schema**: adding `last_seen_at` to
`auth.app_passwords`. Nextcloud-desktop clients authenticate through
that table; separate concern, separate PR if we want desktop-client
liveness.

## Coalescing writes — DECIDED: in-process DashMap + periodic flush

Naive shape (one `UPDATE` per authenticated request) would multiply
the DB write rate by every non-mutating request the SPA fires
(listing pages, thumbnails, delta-upload chunk PUTs). Untenable.

The pattern that scales:

- Middleware stamps `(session_id, Instant::now())` into a shared
  `DashMap<Uuid, DateTime<Utc>>`. **O(1)** per request, no I/O.
- Background task drains the map every N seconds (default 30) and
  emits ONE batched `UPDATE ... FROM UNNEST(...)` covering every
  distinct session seen in the window.
- The map data structure IS the dedup: same `session_id` → same key
  → last write wins. A session seen 100× in the window contributes
  ONE row to the batched update, with the latest timestamp.

The `UPDATE` uses `greatest()` to be idempotent under any
retry/race/clock-skew:

```sql
UPDATE auth.sessions AS s
SET last_seen_at = greatest(s.last_seen_at, t.seen_at)
FROM UNNEST($1::uuid[], $2::timestamptz[]) AS t(id, seen_at)
WHERE s.id = t.id;
```

**Restart durability**: up to one flush interval of activity lost on
hard crash. On graceful shutdown (SIGTERM) run one final flush
synchronously before exit — zero loss on planned rolling restarts.

**Failure durability**: if the batched UPDATE fails (PG blip),
DON'T clear the DashMap; next tick retries with the accumulated set
overlaid on any new activity.

### Why not `Arc<Mutex<HashMap>>`

Every authenticated request writes. Under any concurrency (delta
uploads, thumbnail bursts, folder listing paginations firing in
parallel) a single mutex becomes the bottleneck. `DashMap`'s
per-shard locking (16-32 shards by default) parallelises writes
across distinct keys — different session_ids don't contend.

### Why not `Arc<RwLock<HashMap>>`

The workload is write-heavy (every auth'd request writes, reads only
fire in the flusher). RwLock would still serialize the writes for no
benefit.

### Why not PG `NOTIFY` / `LISTEN`

Considered. Trade-offs:

- **Pro**: cross-instance coalescing — multiple OxiCloud processes
  behind a load balancer push to one channel, single flusher owns
  writes.
- **Con**: every request pays a PG round-trip (`SELECT
  pg_notify(...)`) — ~1 ms per request vs ~50 ns for DashMap insert.
  On hot endpoints (delta chunk PUTs, thumbnails) this is
  measurable.
- **Con**: adds a persistent LISTEN connection to the pool.
- **Con**: OxiCloud is single-instance today. The multi-instance win
  doesn't apply.

**Deferred**: if OxiCloud ever grows a multi-instance deployment
story (Kubernetes, active-active behind a load balancer), migrate
the flusher to `NOTIFY`-based ingest — schema stays identical, only
the tracker implementation swaps. Document the migration path in
[Future — multi-instance](#future--multi-instance) below.

## Metric surface — Prometheus

Exposed via the existing `/metrics` endpoint (see
`src/interfaces/metrics.rs`; gated on `OXICLOUD_METRICS_LISTEN`).

### Gauges (polled every 30 s from a background task)

```
# HELP oxicloud_sessions_online Non-revoked sessions seen in the last N min.
# TYPE oxicloud_sessions_online gauge
oxicloud_sessions_online <value>

# HELP oxicloud_sessions_online_users Distinct users behind online sessions.
# TYPE oxicloud_sessions_online_users gauge
oxicloud_sessions_online_users <value>

# HELP oxicloud_sessions_total_non_revoked Total non-revoked sessions
# regardless of activity — the long tail (mobile clients still holding
# refresh tokens they haven't used in weeks).
# TYPE oxicloud_sessions_total_non_revoked gauge
oxicloud_sessions_total_non_revoked <value>
```

Queries powering each:

```sql
-- oxicloud_sessions_online
SELECT COUNT(*) FROM auth.sessions
WHERE revoked = FALSE
  AND last_seen_at > NOW() - $1::interval;   -- $1 = ONLINE_WINDOW

-- oxicloud_sessions_online_users
SELECT COUNT(DISTINCT user_id) FROM auth.sessions
WHERE revoked = FALSE
  AND last_seen_at > NOW() - $1::interval;

-- oxicloud_sessions_total_non_revoked
SELECT COUNT(*) FROM auth.sessions WHERE revoked = FALSE;
```

All three run on the maintenance pool (background polling shouldn't
compete with request-serving connections). Three lightweight
`COUNT(*)` reads every 30 s; measured cost negligible even on
tens-of-thousands-of-rows tables thanks to the partial index.

### Counters (already in-place shape)

`oxicloud_sessions_created_total` and
`oxicloud_sessions_revoked_total{reason}` — extend the existing
counter surface in the auth service (`session.created` audit line
sites) to also `metrics::counter!(...)`. Not strictly needed for the
"how many active" question but useful sanity signal on the
dashboard: rate of creation vs rate of revocation should be
approximately balanced at steady state.

## Config surface

**No new env var.** Ed's call (2026-08-18): tuning the online window
is a deployment-shape question we haven't had to answer in practice,
and adding an env knob invites premature customization. The three
knobs stay hardcoded:

- **Online window** — 5 min. Feels responsive for a demo
  landing page without over-fluctuating with tab-open-then-close
  blips. Lives at `pub const ONLINE_WINDOW` in
  `src/application/dtos/session_dto.rs`; the gauges module in
  `src/infrastructure/services/session_liveness_gauges.rs` reads
  from that constant so the DTO badge and the gauge aggregate
  can't drift.
- **Flush interval** — 30 s. Balances DB write load against gauge
  freshness (typical Prometheus scrape at 15 s sees the value
  refreshed after at most two scrapes). Lives at `FLUSH_INTERVAL`
  in `src/infrastructure/services/last_seen_tracker.rs`.
- **DashMap shard count** — crate default (16). Only worth
  surfacing when profiling shows shard contention.

## Middleware wiring

Auth extractor (`CurrentUserId`) already loads the session by
refresh-token cookie / bearer-token subject. Extend the post-load
path:

```rust
// After successful session lookup + auth checks:
state.last_seen_tracker.stamp(session.id);
```

`LastSeenTracker` shape:

```rust
pub struct LastSeenTracker {
    seen: Arc<DashMap<Uuid, DateTime<Utc>>>,
    pool: Arc<PgPool>,
}

impl LastSeenTracker {
    pub fn new(pool: Arc<PgPool>) -> Arc<Self> {
        let seen = Arc::new(DashMap::new());
        let this = Arc::new(Self { seen: seen.clone(), pool: pool.clone() });
        tokio::spawn(this.clone().flush_loop());
        this
    }

    /// Called from the auth middleware on every authenticated request.
    /// O(1); no I/O; no round-trip.
    pub fn stamp(&self, session_id: Uuid) {
        self.seen.insert(session_id, Utc::now());
    }

    /// Called from the graceful-shutdown handler.
    pub async fn flush_now(&self) -> Result<(), sqlx::Error> {
        /* drain + one batched UPDATE, same as the loop body */
    }

    async fn flush_loop(self: Arc<Self>) {
        let mut ticker = tokio::time::interval(Duration::from_secs(30));
        loop {
            ticker.tick().await;
            let _ = self.flush_now().await;   // errors logged, not propagated
        }
    }
}
```

Wired in `common/di.rs`; injected into `AppState` and referenced by
the auth extractor.

## Graceful shutdown

Hook into the existing SIGTERM handler in `main.rs` to call
`tracker.flush_now().await` before the runtime exits. Ensures rolling
restarts / container replacements don't lose the last 30 s of
liveness data.

## Admin dashboard integration (deferred, sibling PR)

Once `last_seen_at` exists, the admin sessions panel can render a
"last seen X min ago" column instead of only "created X ago". Small
follow-up — not part of this plan's scope, but the schema addition
unlocks it.

## Testing

Two hermetic units (no DB):

1. `DashMap` dedup test — insert same key 10× with different
   timestamps, drain, assert one entry with the latest timestamp.
2. `flush_now()` UPDATE-shape test — mock a `PgExecutor`, assert the
   batched UNNEST binds match the drained set. Uses `sqlx-mock` or
   equivalent.

One integration (real PG, gated on `integration_tests` cfg):

3. End-to-end — insert a session row, `stamp()` it, call `flush_now`,
   assert `last_seen_at > created_at`. Covers the whole write path
   including the `greatest()` guard.

## Phasing

1. **Migration** — add column + partial index. Ships alone; zero
   application-layer impact. Reversible via `DROP COLUMN`.  ✅ **2026-08-18**
   (`migrations/20261014000000_sessions_last_seen_at.sql`).
2. **`LastSeenTracker` service** — DashMap + flusher task. Wire into
   `AppState`. Middleware calls `stamp()`. Now `last_seen_at` moves
   in real time.  ✅ **2026-08-18** (`src/infrastructure/services/last_seen_tracker.rs`).
3. **JWT `sid` claim** — token minters carry the fresh session's
   id; auth middleware reads it and stamps with no DB round trip.
   New tokens carry it, old tokens still validate (Option → no-op).
   ✅ **2026-08-18** (extends `TokenClaims::sid` on
   `application/ports/auth_ports.rs`).
4. **Prometheus gauges** — background poller updates the three
   gauges every 30 s using the queries above. Gated on
   `OXICLOUD_METRICS_LISTEN` (no recorder → no periodic PG hits).
   ✅ **2026-08-18** (`src/infrastructure/services/session_liveness_gauges.rs`).
5. **Graceful-shutdown flush** — hook into SIGTERM handler.
   ✅ **2026-08-18** (added `shutdown_signal` +
   `with_graceful_shutdown` in `main.rs`).
6. **Session DTO exposes `last_seen_at` + `is_online`** —
   `GET /api/admin/sessions` returns both so the admin table + external
   monitors can read them. `is_online` is the server-side derivation
   against `ONLINE_WINDOW` (see terminology decision above). ✅
   **2026-08-18** (`application/dtos/session_dto.rs`).
7. **Admin dashboard "Online" column** (deferred to a sibling PR) —
   render `is_online` as a green/grey dot next to each row plus
   "last seen X ago" from `last_seen_at`.

## Future — multi-instance

If OxiCloud grows a multi-instance deployment story (K8s replicaset,
active-active behind a load balancer), the in-process DashMap becomes
insufficient — each process holds its own map, N flushers race
UPDATEs, coalescing across instances doesn't happen.

Migration path when that becomes real:

1. Keep the schema (`last_seen_at` column + partial index).
2. Replace the `LastSeenTracker::stamp()` in-process insert with a
   `SELECT pg_notify('oxicloud_session_seen', $session_id)` call.
3. Move the flusher into a **single elected worker** (leader election
   via advisory lock in PG). That worker `LISTEN`s the channel,
   accumulates into an in-process HashMap, flushes on the same 30 s
   ticker.
4. Every OxiCloud instance publishes; one instance consumes.

Trade-off: NOTIFY costs a PG round-trip per request (~1 ms) vs the
current ~50 ns DashMap insert. Only pay that when multi-instance
coalescing actually matters. Schema and gauge queries stay identical;
only the tracker implementation swaps.

## Open questions

1. ~~Definition of "active"~~ — DECIDED 2026-08-18: hardcoded to
   5 min, no env var. See [Config surface](#config-surface).
2. **Should the gauge query drop long-inactive sessions?** — a
   Nextcloud desktop client checking in every 6 h qualifies as
   "active" if the window is 6 h. Probably want two separate metrics
   (web-active < 5 min AND dav-active < 6 h) but scope creep for the
   initial ship.
3. **Multi-tab dedup on the UI side** — three browser tabs of the
   same user share ONE session (same cookies, same refresh token,
   same row). Naturally deduped at the DB layer — no action needed.
4. **App-password rows** — Nextcloud desktop / mobile clients use
   `auth.app_passwords` on top of the session model. Whether they
   should get their own `last_seen_at` (and a companion metric) is
   deferred; separate concern, separate PR.