fix(msg-bus): prevent race on reconnect
and update plan
This commit is contained in:
+159
-35
@@ -16,6 +16,78 @@ editing is one consumer on top; folder-live updates, notifications,
|
|||||||
job progress, presence, and sync-client push invalidation follow with
|
job progress, presence, and sync-client push invalidation follow with
|
||||||
almost no extra scaffolding.
|
almost no extra scaffolding.
|
||||||
|
|
||||||
|
## Status — 2026-09-11
|
||||||
|
|
||||||
|
The `feat/message-bus` branch delivers **D + F + follow-ups shipped
|
||||||
|
end-to-end** on the FE and BE, verified by S1–S11 in the api-test
|
||||||
|
smoke suite plus manual multi-user E2E. Live today:
|
||||||
|
|
||||||
|
- **Bus core** — `MessageBus` port + `InProcessMessageBus` +
|
||||||
|
`NoopReplicator`. `📤 bus publish` trace under
|
||||||
|
`RUST_LOG=oxicloud::message_bus=debug`.
|
||||||
|
- **WS handler** (`/api/rt/ws`) — JSON-RPC 2.0, `rt.subscribe /
|
||||||
|
unsubscribe / event / revoked / ping / pong`, server-initiated RFC
|
||||||
|
6455 keepalive Ping.
|
||||||
|
- **Auth for the WS upgrade** — **F ticket flow shipped**.
|
||||||
|
`POST /api/rt/ticket` mints a one-shot 30 s ticket under the full
|
||||||
|
auth+DPoP+CSRF chain; the browser passes it via
|
||||||
|
`Sec-WebSocket-Protocol: oxi.ticket.<uuid>`. Also accepts
|
||||||
|
`Authorization: Bearer <jwt>` for programmatic clients
|
||||||
|
(`rt-hurl-helper`). Route mounted OUTSIDE `protected_api` so the
|
||||||
|
standard DPoP-required middleware doesn't 401 browsers that can't
|
||||||
|
attach a `DPoP:` header to `new WebSocket()`. See
|
||||||
|
`handlers/rt_ws.rs` module doc.
|
||||||
|
- **Events firing end-to-end** — every `MessageBusEvent` variant
|
||||||
|
except CRDT-flavoured ones:
|
||||||
|
- `FileCreated / Renamed / Moved / Deleted` (via
|
||||||
|
`FileUploadService` + `FileManagementService`)
|
||||||
|
- `FolderCreated / Renamed / Moved / Deleted` (via `FolderService`
|
||||||
|
for direct paths; `TrashService` publishes `FolderDeleted` on the
|
||||||
|
trash-first path — the FE hits that path via
|
||||||
|
`DELETE /api/folders/{id}`)
|
||||||
|
- `AuthzChanged` (via `ShareService::revoke_grant`) drives the
|
||||||
|
grant-revocation eviction cascade.
|
||||||
|
- **Grant-revocation eviction (Slice C)** — WS handler
|
||||||
|
auto-subscribes each session to `user:{caller}:authz`; on
|
||||||
|
`AuthzChanged` the reader translates to `SessionOut::EvictFolders`
|
||||||
|
and emits `rt.revoked` per evicted topic. Scope is per-topic; the
|
||||||
|
session itself and unrelated subscriptions survive.
|
||||||
|
- **FE composables** — `useTopic` (generic), `useFolderTopic`
|
||||||
|
(folder-view sugar with per-verb + `onRevoked` + `onReconnect`
|
||||||
|
handlers), `useReconnect` (session-level, fires after 2nd+ open).
|
||||||
|
Types generated from AsyncAPI via `@asyncapi/modelina` in
|
||||||
|
`frontend/src/lib/generated/message-bus/`; `check-message-bus-spec`
|
||||||
|
CI + `pre-pull-request` block on drift.
|
||||||
|
- **Folder-view live refresh** — `+page.svelte` wires every
|
||||||
|
`onFile*`/`onFolder*` handler to `scheduleLiveReload`, 100 ms
|
||||||
|
coalesce. Revocation → toast + `goto('/files')`. Reconnect →
|
||||||
|
refetch via `onReconnect` (bridges the "events lost during outage
|
||||||
|
window" gap; see `project_message_bus_reconnect_gap` memory).
|
||||||
|
Actor-echo skip was REMOVED for multi-tab correctness — refetch is
|
||||||
|
idempotent, ~30 ms per self-mutation.
|
||||||
|
- **Client-side resilience** — jittered exponential backoff
|
||||||
|
(250 ms → 30 s cap), circuit breaker at 20 consecutive failures
|
||||||
|
(~5 minutes of retry — covers a cargo-release restart), `untrack`
|
||||||
|
in every mutation entry point so `$state` reads don't leak into
|
||||||
|
caller `$effect` deps.
|
||||||
|
- **AuthZ tested** — S3 (folder no_read), S4 (nonexistent folder =
|
||||||
|
anti-enum parity), S9 (cross-user identity topic → `topic_forbidden`).
|
||||||
|
- **Ticket tested** — S10 (happy path), S11 (single-use replay
|
||||||
|
rejected).
|
||||||
|
|
||||||
|
Deferred and still open — see the Roadmap section and the
|
||||||
|
`project_message_bus_reconnect_gap` memory:
|
||||||
|
|
||||||
|
- **Notifications table + bell** (E) — topic + producer + auto-sub
|
||||||
|
land here. Same pattern as `:authz`.
|
||||||
|
- **Presence** (Phase B) — `folder:{id}:presence` topic + awareness
|
||||||
|
frames.
|
||||||
|
- **Yjs collab** — `docs/plan/markdown-collab.md`, depends on the
|
||||||
|
binary-frame routing this plan sketches but doesn't ship.
|
||||||
|
- **Broker replicator** (Postgres LISTEN/NOTIFY or Redis) — for
|
||||||
|
multi-instance and durable event log. `BusReplicator` port
|
||||||
|
declared, `NoopReplicator` wired today.
|
||||||
|
|
||||||
## Non-goals
|
## Non-goals
|
||||||
|
|
||||||
- Persistent event log with "you missed these" replay. Durable state
|
- Persistent event log with "you missed these" replay. Durable state
|
||||||
@@ -352,25 +424,61 @@ everywhere.
|
|||||||
|
|
||||||
## Frontend components
|
## Frontend components
|
||||||
|
|
||||||
### 1. Singleton client (`lib/stores/message-bus.svelte.ts`)
|
### 1. Singleton client (`lib/message-bus/client.svelte.ts`)
|
||||||
|
|
||||||
- Fetches a ticket via `POST /api/rt/ticket` (through `apiFetch`, so
|
Location is `lib/message-bus/` (subsystem dir, mirrors `lib/auth/` and
|
||||||
DPoP is applied).
|
`lib/upload/` — see `frontend/AGENTS.md`), NOT `lib/stores/` — the
|
||||||
- Opens `wss:///api/rt/ws?ticket=…`.
|
client is subsystem-scoped plumbing, not global reactive state that
|
||||||
|
routes read from.
|
||||||
|
|
||||||
|
- **Fetches a ticket** via `POST /api/rt/ticket` (through `apiFetch`,
|
||||||
|
so DPoP is applied; `getCsrfHeaders()` merged in for the state-
|
||||||
|
changing POST). Ticket then passes on the WS upgrade via
|
||||||
|
`Sec-WebSocket-Protocol: oxi.ticket.<uuid>` (NOT a query param —
|
||||||
|
keeps the token off access logs and out of Referer / URL bar).
|
||||||
|
- Opens `wss://<same-origin>/api/rt/ws` with the subprotocol.
|
||||||
- **Refcounted subscriptions**:
|
- **Refcounted subscriptions**:
|
||||||
`subs: Map<TopicKey, { count, listeners: Set<Handler> }>`.
|
`#subs: Map<TopicKey, { count, handlers, revokedHandlers, acked }>`.
|
||||||
- On subscribe by first component: send frame; on last unsubscribe:
|
- On first refcount of a topic: send `rt.subscribe`; on last drop:
|
||||||
send frame.
|
send `rt.unsubscribe`.
|
||||||
- On reconnect: reissue ticket, re-establish WS, re-send `subscribe`
|
- **On reconnect**: replay every already-known topic (client-side
|
||||||
for every live topic — components don't care.
|
state survives the disconnect); fire `onReconnect` handlers so
|
||||||
- Backoff: exponential (250 ms → 30 s), full-jitter.
|
consumers refetch and catch up on events dropped during the
|
||||||
- Health: `$state({ connected, latencyMs, subscribedTopics })`
|
outage window.
|
||||||
exposed for a debug indicator.
|
- **Backoff**: exponential (250 ms → 30 s), full-jitter. Circuit
|
||||||
|
breaker at 20 consecutive failures (~5 minutes of retry —
|
||||||
|
comfortably covers a cargo-release restart); trip logs one `error`
|
||||||
|
line and stops until `messageBus.reconnect()` is called or the
|
||||||
|
page reloads.
|
||||||
|
- **Reactive-safety rule**: every mutation entry point
|
||||||
|
(`subscribe`, `onReconnect`, `reconnect`, `close`, `#call`) wraps
|
||||||
|
its `$state` reads in `untrack(() => …)`. Without this a caller's
|
||||||
|
`$effect` inherits a hidden dep on `state`, and each transition
|
||||||
|
(idle → connecting → connected → disconnected → …) re-fires the
|
||||||
|
effect — an observed 1000+/s loop on server-down. See the
|
||||||
|
`feedback-svelte5-untrack-mutation-methods` memory for the
|
||||||
|
general rule and the docstring on `subscribe` for the concrete
|
||||||
|
case.
|
||||||
|
- **Health**: `state = $state<ConnectionState>` +
|
||||||
|
`latencyMs = $state<number | null>` exposed for a future debug
|
||||||
|
indicator (no UI consumes them yet — silent MVP).
|
||||||
|
|
||||||
### 2. Composable (`lib/composables/useTopic.ts`)
|
### 2. Composables
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
useTopic(`folder:${folderId}`, (evt) => { /* mutate local $state */ });
|
// Generic — subscribe to any topic.
|
||||||
|
useTopic(topic, onEvent, onRevoked?)
|
||||||
|
|
||||||
|
// Folder-view sugar — per-verb handlers + reconnect hook.
|
||||||
|
useFolderTopic(() => folderId, {
|
||||||
|
onFileCreated, onFileRenamed, onFileMoved, onFileDeleted,
|
||||||
|
onFolderCreated, onFolderRenamed, onFolderMoved, onFolderDeleted,
|
||||||
|
onRevoked, // grant revoked, subscription evicted server-side
|
||||||
|
onReconnect, // WS came back; consumers refetch to catch up
|
||||||
|
})
|
||||||
|
|
||||||
|
// Session-level reconnect (fires on 2nd+ open, never initial).
|
||||||
|
useReconnect(cb)
|
||||||
```
|
```
|
||||||
|
|
||||||
Handles `$effect` lifecycle (subscribe on mount, unsubscribe on
|
Handles `$effect` lifecycle (subscribe on mount, unsubscribe on
|
||||||
@@ -1047,28 +1155,44 @@ this baseline once the baseline is green.
|
|||||||
|
|
||||||
Ships the infrastructure and the two most visible consumers together.
|
Ships the infrastructure and the two most visible consumers together.
|
||||||
|
|
||||||
- Bus port + `InProcessMessageBus` + `NoopReplicator` + WS handler
|
- **✅ Bus port** + `InProcessMessageBus` + `NoopReplicator` + WS
|
||||||
+ ticket endpoint.
|
handler + **ticket endpoint (F)**.
|
||||||
- Frontend singleton + `useTopic` composable.
|
- **✅ Frontend singleton** + `useTopic` + `useFolderTopic` +
|
||||||
- Topics live: `folder:{id}`, `user:{u}:notifications`, `job:{id}`,
|
`useReconnect` composables. `oxi:message-bus` logger namespace.
|
||||||
`collab:{file_id}`, `collab:{file_id}:awareness`.
|
- **Topics live today**: `folder:{id}`, `user:{u}:authz`.
|
||||||
- **Folder-live updates**: `FolderService` / `FileManagementService`
|
- **Topics reserved but not producing**: `user:{u}:notifications`,
|
||||||
publish `file.created` / `file.deleted` / `file.renamed` /
|
`job:{id}`, `collab:{file_id}`, `collab:{file_id}:awareness` —
|
||||||
`file.moved` after commit; FE folder view subscribes and mutates
|
land with their consumers below.
|
||||||
local state — no manual refresh.
|
- **✅ Folder-live updates**: `FolderService` / `FileUploadService` /
|
||||||
- **Job dashboard live**: `JobRegistry` publishes step progress and
|
`FileManagementService` / `TrashService` publish `file_created /
|
||||||
terminal state; FE job dashboard subscribes and replaces the
|
renamed / moved / deleted` and `folder_created / renamed / moved /
|
||||||
current polling.
|
deleted` after commit; FE folder view refetches on receipt (100 ms
|
||||||
- **Notifications table + bell**: new `notifications` table +
|
coalesce, idempotent). Multi-user + multi-tab verified.
|
||||||
`NotificationService` port; initial ingesters for `share-granted`,
|
- **✅ Grant-revocation eviction** (Slice C): `AuthzChanged` →
|
||||||
`new-login-from-new-device`, `job-completed-for-you`,
|
per-topic `rt.revoked`; folder view toasts + navigates to
|
||||||
`storage-quota-threshold`. FE bell with unread count, slide-out
|
`/files`. Session survives; unrelated subs unaffected.
|
||||||
panel, toast pop on receive.
|
- **✅ Refetch-on-reconnect**: `messageBus.onReconnect(cb)` →
|
||||||
- **MD collab editor**: see companion plan
|
`useReconnect` composable → folder view refetches after WS comes
|
||||||
`docs/plan/markdown-collab.md` — depends on this phase's WS
|
back. Bridges the in-memory-bus "events lost during outage" gap
|
||||||
handler + binary frame routing.
|
(see `project_message_bus_reconnect_gap` memory).
|
||||||
|
- **Job dashboard live** — TODO. `JobRegistry` publishes step
|
||||||
|
progress and terminal state; FE job dashboard subscribes and
|
||||||
|
replaces polling.
|
||||||
|
- **Notifications table + bell** — TODO (Slice E). New
|
||||||
|
`notifications` table + `NotificationService` port; initial
|
||||||
|
ingesters for `share-granted`, `new-login-from-new-device`,
|
||||||
|
`job-completed-for-you`, `storage-quota-threshold`. FE bell with
|
||||||
|
unread count, slide-out panel, toast pop on receive. Auto-subscribe
|
||||||
|
to `user:{u}:notifications` server-side, same pattern as
|
||||||
|
`user:{u}:authz` today.
|
||||||
|
- **MD collab editor** — TODO. See companion plan
|
||||||
|
`docs/plan/markdown-collab.md`. Depends on binary-frame routing
|
||||||
|
which this plan sketches but doesn't ship (`rt_ws.rs` today drops
|
||||||
|
binary frames with a debug log).
|
||||||
|
|
||||||
Deliverables sized ~4 weeks end-to-end.
|
Deliverables sized ~4 weeks end-to-end. Slice D (folder-live) and
|
||||||
|
Slice F (ticket flow) landed 2026-09-11. Slices E + collab are the
|
||||||
|
open work in Phase A.
|
||||||
|
|
||||||
### Phase B — Presence + comments
|
### Phase B — Presence + comments
|
||||||
|
|
||||||
|
|||||||
@@ -104,10 +104,11 @@ const RECONNECT_MAX_MS = 30_000;
|
|||||||
* give up and stay `disconnected` until the caller explicitly asks
|
* give up and stay `disconnected` until the caller explicitly asks
|
||||||
* to `reconnect()`. Prevents an unrecoverable auth state (revoked
|
* to `reconnect()`. Prevents an unrecoverable auth state (revoked
|
||||||
* session, wrong CSRF cookie, missing DPoP nonce) from flooding
|
* session, wrong CSRF cookie, missing DPoP nonce) from flooding
|
||||||
* logs. Ten attempts × exponential-backoff-with-jitter is roughly a
|
* logs. Twenty attempts × exponential-backoff-with-jitter caps
|
||||||
* minute of trying — long enough for a transient blip, short enough
|
* around 5 minutes of retrying — comfortably covers a cargo-release
|
||||||
* to stop before it's noise. */
|
* server restart on a hot machine while still short-circuiting a
|
||||||
const MAX_CONSECUTIVE_FAILURES = 10;
|
* genuine permanent failure before it becomes noise. */
|
||||||
|
const MAX_CONSECUTIVE_FAILURES = 20;
|
||||||
|
|
||||||
interface SubEntry {
|
interface SubEntry {
|
||||||
count: number;
|
count: number;
|
||||||
|
|||||||
@@ -438,23 +438,27 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Live folder updates (message bus) ────────────────────────────
|
// ── Live folder updates (message bus) ────────────────────────────
|
||||||
// Subscribe to `folder:{currentId}` and refresh when another tab —
|
// Subscribe to `folder:{currentId}` and refresh when THIS session's
|
||||||
// or another user with a share — mutates something in this folder.
|
// tabs, another tab of the same user, or another user with a share
|
||||||
// The refresh call is coalesced through `#reloadScheduled` so a
|
// mutates something in this folder. Refetch is coalesced through
|
||||||
// burst of events (e.g. a multi-file upload) collapses to a single
|
// `reloadScheduled` so a burst of events (multi-file upload) collapses
|
||||||
// fetch. Local mutations trigger `reload()` themselves, so events
|
// to a single fetch.
|
||||||
// authored by this same user are dropped as echo (the `actor` on
|
//
|
||||||
// the event is the caller UUID from the server).
|
// Actor-echo skip was REMOVED: previously we skipped events whose
|
||||||
|
// `actor` equalled `session.user.id`, on the assumption "this tab
|
||||||
|
// already updated its state via the local mutation path". That is
|
||||||
|
// true for the ACTIVE tab, but it also silenced updates from OTHER
|
||||||
|
// TABS of the same user. Since `reload()` is idempotent (replaces
|
||||||
|
// `listing.files` with the same server state) the extra fetch on
|
||||||
|
// self-authored events costs one round-trip (~30 ms locally, never
|
||||||
|
// visible) and gains multi-tab correctness. The `reloadScheduled`
|
||||||
|
// coalescer already prevents redundant work when the local mutation
|
||||||
|
// path and the bus event race.
|
||||||
//
|
//
|
||||||
// See `docs/plan/message-bus.md § D` and the `useFolderTopic`
|
// See `docs/plan/message-bus.md § D` and the `useFolderTopic`
|
||||||
// composable for the wiring.
|
// composable for the wiring.
|
||||||
let reloadScheduled = false;
|
let reloadScheduled = false;
|
||||||
function scheduleLiveReload(actor: string): void {
|
function scheduleLiveReload(_actor: string): void {
|
||||||
// Actor echo: this same session's mutations already updated the
|
|
||||||
// listing through their own success path, so a re-fetch would
|
|
||||||
// only cost a round-trip. Other tabs of the same user still see
|
|
||||||
// the change (they render from their own state, not this one).
|
|
||||||
if (session.user?.id && actor === session.user.id) return;
|
|
||||||
if (reloadScheduled) return;
|
if (reloadScheduled) return;
|
||||||
reloadScheduled = true;
|
reloadScheduled = true;
|
||||||
// Coalesce a burst; 100 ms is enough for the tail of a multi-
|
// Coalesce a burst; 100 ms is enough for the tail of a multi-
|
||||||
@@ -493,12 +497,9 @@
|
|||||||
// through the same `scheduleLiveReload` coalescer as event-
|
// through the same `scheduleLiveReload` coalescer as event-
|
||||||
// driven refreshes so a burst of reconnects (rare, but the
|
// driven refreshes so a burst of reconnects (rare, but the
|
||||||
// circuit breaker can produce one) collapses to a single
|
// circuit breaker can produce one) collapses to a single
|
||||||
// fetch. Passing an actor of `null`-equivalent — use an
|
// fetch. See `project_message_bus_reconnect_gap` memory.
|
||||||
// empty string so the echo-skip's `actor === user.id`
|
|
||||||
// check never matches. See
|
|
||||||
// `project_message_bus_reconnect_gap` memory.
|
|
||||||
busLog.warn('reconnected — refetching folder');
|
busLog.warn('reconnected — refetching folder');
|
||||||
scheduleLiveReload('');
|
scheduleLiveReload('reconnect');
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user