Compare commits
10 Commits
c4dfa9ccf2
...
4d067b3fc6
| Author | SHA1 | Date | |
|---|---|---|---|
| 4d067b3fc6 | |||
| fd9e1e669f | |||
| b5648b82f9 | |||
| a95a6b106c | |||
| 0b9e8bfe23 | |||
| 03f2956457 | |||
| 47346b9309 | |||
| 4840930767 | |||
| 69fc88982e | |||
| ca85ac7307 |
@@ -2,6 +2,72 @@
|
||||
|
||||
This file provides guidance to coding agents (Claude Code, Codex, Cursor, Aider, …) working with this repository. Claude Code reads it via `@AGENTS.md` in `CLAUDE.md`.
|
||||
|
||||
# Purpose — what OxiCloud is, and what it is not
|
||||
|
||||
Read this before designing anything. Most "should we…" questions are answered by
|
||||
the scale target rather than by taste.
|
||||
|
||||
- **Open source, and it stays that way.** MIT (`LICENSE`). This is a constraint on
|
||||
what you may add, not a footnote: dependencies must be license-compatible — a
|
||||
GPL/AGPL crate or npm package would force the whole project to relicense, and
|
||||
that is not on the table. No feature may be gated behind a licence key or an
|
||||
"enterprise edition", and nothing core may hard-depend on a proprietary service
|
||||
or SDK. Vendored frontend assets (`frontend/static/vendors/`) carry the same
|
||||
rule; record the licence when vendoring.
|
||||
- **Self-hosted**, for an individual or an enterprise. The operator is not an SRE
|
||||
team: defaults must be safe, failures loud, and nothing may silently depend on a
|
||||
cloud service.
|
||||
- **Target scale is up to ~10k users.** Design against that number in both
|
||||
directions. Do not build million-user machinery (sharding, eventual consistency,
|
||||
service decomposition) for a load that will never arrive; equally, never ship
|
||||
anything O(users) per request, or a table scan that is fine at 50 users and
|
||||
fatal at 10k.
|
||||
- **Not a mass hoster.** OxiCloud does not claim to serve millions of users on one
|
||||
deployment, and trade-offs should not pretend otherwise.
|
||||
- **Decentralised by intent.** Many instances federating beats one large instance —
|
||||
OpenCloudMesh is one route. Prefer designs that survive "this is one of many
|
||||
instances" over ones assuming a single authoritative deployment.
|
||||
|
||||
Targets:
|
||||
|
||||
- **Feature ambition: Google Workspace / Office 365.** Breadth of capability is a
|
||||
goal, not scope creep.
|
||||
- **Collaboration is the main feature axis.** OxiCloud is not a personal backup
|
||||
drive that happens to have sharing bolted on — sharing, shared drives, grants,
|
||||
co-editing (WOPI) and live updates are the product. When choosing what to build
|
||||
or how to build it, the multi-user case is the primary one, not the case to
|
||||
generalise to later. A feature that works only for a single owner is unfinished.
|
||||
- **Customer target: NextCloud users.** Hence the NextCloud-compatible API surface
|
||||
(`/remote.php`, `/ocs`, `/status.php`) — compatibility is a feature, and breaking
|
||||
it costs adopters.
|
||||
|
||||
## Design axes
|
||||
|
||||
Four things decide an open design question. **Security and resilience are
|
||||
absolute** — they are not traded against anything. Performance is measured against
|
||||
the 10k target. Privacy is a direction with a stated endpoint.
|
||||
|
||||
- **Resilience.** This is a storage product: **no data loss, no data corruption,
|
||||
ever.** Anything that can silently drop or alter bytes is a top-severity defect,
|
||||
not a trade-off. In practice that means: a job that skips work must never report
|
||||
success (pause at a cursor instead — `docs/plan/jobs-handling-recoverable-error.md`);
|
||||
a read failure is never proof that data is absent; content-addressing and
|
||||
ref-counting are load-bearing, not decoration; and consistency checks are
|
||||
discovery-only unless repair is explicitly requested.
|
||||
- **Security.** Prefer deny-by-default over assert-later; a guarantee enforced by
|
||||
the type system or the router beats one a reviewer must remember. AuthZ lives in
|
||||
the service layer, never in handlers. See `src/AGENTS.md` § AuthZ enforcement
|
||||
points.
|
||||
- **Performance.** Measure against 10k users, not a dev instance. The hot paths are
|
||||
listing, thumbnails and auth — a per-row query or an extra round trip there is a
|
||||
real regression even when it looks harmless.
|
||||
- **Privacy.** When the backend belongs to a third party (S3, Azure), encryption at
|
||||
rest is a *should-have*; **end-to-end encryption is the target.** Designs that
|
||||
assume the server can always read plaintext will have to be undone — the `Vault`
|
||||
drive kind is reserved for the E2E case.
|
||||
|
||||
Where two conflict, resilience and security win, and the cost is documented.
|
||||
|
||||
# Architecture
|
||||
|
||||
This project is split into two parts:
|
||||
|
||||
@@ -176,6 +176,10 @@ export default defineConfig({
|
||||
{ text: "Authentication model", link: "/architecture/auth-model" },
|
||||
{ text: "Magic-link auth", link: "/architecture/magic-link-auth" },
|
||||
{ text: "Background jobs", link: "/architecture/jobs" },
|
||||
{
|
||||
text: "Message bus & notifications",
|
||||
link: "/architecture/message-bus-and-notifications",
|
||||
},
|
||||
{ text: "UI diagnostics", link: "/architecture/ui-diagnostics" },
|
||||
],
|
||||
},
|
||||
|
||||
@@ -77,3 +77,4 @@ src/
|
||||
- [Backend Storage →](/architecture/backend-storage)
|
||||
- [Derived and Attached Blobs →](/architecture/derived-and-attached-blobs) — thumbnails, transcodes and uploaded previews: why content-keyed and file-keyed artifacts need separate tables
|
||||
- [Background Jobs →](/architecture/jobs)
|
||||
- [Message Bus & Notifications →](/architecture/message-bus-and-notifications) — real-time WebSocket bus (topics, AuthZ scopes, tab-visibility grace-close), persistent notifications (bell), AsyncAPI vs OpenAPI schema ownership
|
||||
|
||||
@@ -0,0 +1,645 @@
|
||||
# Message Bus & Persistent Notifications
|
||||
|
||||
OxiCloud has two coupled subsystems that together power its real-time
|
||||
UX — a **live message bus** over WebSocket for "something just
|
||||
happened, refresh your view", and a **persistent notifications
|
||||
table** for "you need to know about this even if you weren't
|
||||
online." This document explains how both work, how they authenticate
|
||||
and authorize subscribers, and why the frontend deliberately drops
|
||||
the WebSocket while a tab is hidden.
|
||||
|
||||
Design docs the shipped code implements: [`docs/plan/message-bus.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md)
|
||||
+ [`docs/plan/templated-messages.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md).
|
||||
|
||||
---
|
||||
|
||||
## The two channels
|
||||
|
||||
| | **Message bus (WebSocket)** | **Persistent notifications (REST + DB)** |
|
||||
|---|---|---|
|
||||
| Purpose | "Something changed, refresh your view" | "You need to know about this — later is fine" |
|
||||
| Transport | JSON-RPC 2.0 over `/api/rt/ws` | `GET/POST/DELETE /api/notifications/*` + `notif.notifications` table |
|
||||
| Delivery | Best-effort, in-memory, no replay | Durable, per-user rows, survive reboot / offline |
|
||||
| Payload | Thin "poke" facts (id + verb) | Full per-kind DTO with all render data |
|
||||
| Loss on disconnect | Yes (events during outage window are dropped) | No (rows are the source of truth) |
|
||||
| Schema owner | AsyncAPI (`resources/gen/asyncapi.json`) | OpenAPI (`resources/gen/openapi.json`) |
|
||||
|
||||
The two work together: an ingester that wants to notify a user
|
||||
writes **both** — the DB row (for durability + the bell's history)
|
||||
AND publishes a bus event on `user:{u}:notifications` (so online
|
||||
sessions refetch instantly instead of waiting for the next
|
||||
mount). The wire event on that topic is a **pure poke** — empty
|
||||
`data: {}`. The row's real content only ever crosses the REST wire.
|
||||
|
||||
---
|
||||
|
||||
## Schema ownership — AsyncAPI vs OpenAPI
|
||||
|
||||
The bus and the REST endpoints have separate wire specs. The rule
|
||||
the codebase adopts to keep them from drifting:
|
||||
|
||||
> **AsyncAPI defines the envelope + transport for clients.
|
||||
> OpenAPI defines the payload.**
|
||||
|
||||
Concretely:
|
||||
|
||||
| Type | Home | How it stays in sync with Rust |
|
||||
|---|---|---|
|
||||
| **Bus events** (`MessageBusEvent`, subscribe / unsubscribe frames, revoked notifications, envelope shape) | **AsyncAPI** — `resources/gen/asyncapi.json` | Hand-written in `src/bin/generate-asyncapi.rs` via `json!` macros; kept in lockstep with `MessageBusEvent`'s serde shape. Small drift risk — Rust is truth. |
|
||||
| **REST DTOs** (response bodies, request bodies, per-kind notification payloads) | **OpenAPI** — `resources/gen/openapi.json` | `#[derive(utoipa::ToSchema)]` on the Rust struct. Utoipa walks `#[utoipa::path(...)]` handlers + registered schemas. **No drift possible** — projection is derived from Rust. |
|
||||
| **Types on both wires** (rare; none today) | Would live as one Rust struct with both derives, or wait for single-source codegen | — |
|
||||
|
||||
### Why this split, not one unified spec
|
||||
|
||||
The instinct is to put the notification payload schema in AsyncAPI
|
||||
alongside the bus event that triggers a refetch. It looks cleaner
|
||||
until you realize the payload **never travels on the bus wire** —
|
||||
the bus event is `NotificationReceived` with empty `data: {}`, a
|
||||
pure cache-invalidation poke. The FE fetches the payload from
|
||||
`GET /api/notifications`, which is REST → OpenAPI's territory.
|
||||
Putting the payload schema in AsyncAPI would mean "documenting
|
||||
this shape on a transport it doesn't travel on" — a conceptual
|
||||
stretch that adds a drift risk for zero gain.
|
||||
|
||||
### Rejected alternatives
|
||||
|
||||
- **Dual-spec (same type declared in both AsyncAPI + OpenAPI).**
|
||||
Guaranteed drift unless both come from a single codegen. Nothing
|
||||
in the tooling today produces both, so we'd hand-maintain two
|
||||
copies of every shared type. Bug factory.
|
||||
- **Cross-spec `$ref`** — AsyncAPI 3.0 allows `"$ref":
|
||||
"openapi.json#/components/schemas/Foo"`, and utoipa's
|
||||
`components(schemas(...))` can publish orphan types (no
|
||||
`#[utoipa::path]` reference) so OpenAPI advertises "internal"
|
||||
schemas. Technically workable but: Modelina + Swagger UI + Redoc
|
||||
handle external refs inconsistently, OpenAPI stops being "the
|
||||
REST contract" and becomes "a general schema registry",
|
||||
reviewers get confused. Legal, fragile, avoided.
|
||||
- **Bus event carries the full payload** (revert the pure-poke
|
||||
design). Would put per-kind payload schemas in AsyncAPI as
|
||||
`MessageBusEvent::NotificationReceived { granter_id, resource_id,
|
||||
… }`. Rejected because the FE has to REST-fetch anyway (bell
|
||||
reads from DB for history + persistence), so the fields on the
|
||||
wire are dead weight — same-content overlap between the two
|
||||
specs, no consumer benefit.
|
||||
- **Session-resume tokens** (`rt.subscribe { since: N }` +
|
||||
server-side ring buffer). Would let the bus deliver missed rows
|
||||
directly on reconnect, saving one REST round-trip. Rejected for
|
||||
**backward compatibility with `OXICLOUD_MESSAGEBUS_ENABLE=false`**:
|
||||
ops who disable the WS rely on the bell falling back to REST;
|
||||
bus-only replay would leave those deployments with no catch-up
|
||||
path. The REST `?after=` cursor works in every mode (bus on, bus
|
||||
off, network gap); the bus stays purely "instant-poke".
|
||||
|
||||
### What this looks like in the tree
|
||||
|
||||
- `src/application/ports/message_bus_ports.rs` — `MessageBusEvent`
|
||||
enum (Rust source of truth for bus wire shapes).
|
||||
- `src/bin/generate-asyncapi.rs` — projects those Rust variants
|
||||
into `resources/gen/asyncapi.json`.
|
||||
- `src/domain/entities/notification.rs` — `SharegrantedPayload`
|
||||
and its siblings, `#[derive(ToSchema)]`, source of truth for
|
||||
REST payload shapes.
|
||||
- `src/interfaces/api/mod.rs` — utoipa `#[openapi(components(schemas(SharegrantedPayload, ...)))]`
|
||||
registers the payload in OpenAPI even though `NotificationDto.payload`
|
||||
stays `serde_json::Value` on the response type. (FE type-narrows
|
||||
on `row.kind` and casts to the right shape.)
|
||||
- Nothing lives in both specs today.
|
||||
|
||||
### Adding a new bus event
|
||||
|
||||
1. Add a variant to `MessageBusEvent`.
|
||||
2. Add the variant to `generate-asyncapi.rs`'s `event_kind` enum
|
||||
and (if the variant has payload fields) a schema function.
|
||||
3. Regenerate AsyncAPI + FE DTOs via `just asyncapi` +
|
||||
`npm run gen:message-bus`.
|
||||
4. **Do not** add the variant to OpenAPI. Bus events don't
|
||||
travel on REST.
|
||||
|
||||
### Adding a new notification kind's payload
|
||||
|
||||
1. Add a Rust struct in `domain/entities/notification.rs` with
|
||||
`#[derive(Serialize, Deserialize, ToSchema)]`.
|
||||
2. Register it in `src/interfaces/api/mod.rs`'s
|
||||
`components(schemas(...))` list.
|
||||
3. Regenerate OpenAPI via `just openapi`.
|
||||
4. **Do not** add the struct to AsyncAPI. Notification payloads
|
||||
only cross the REST wire.
|
||||
|
||||
---
|
||||
|
||||
## Message bus
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SERVICE LAYER │
|
||||
│ │
|
||||
│ ShareService.grant() ── after commit ──▶ bus.publish(…) │
|
||||
│ FileMgmtService.…() ── after commit ──▶ bus.publish(…) │
|
||||
│ NotificationService ── after commit ──▶ bus.publish(…) │
|
||||
│ Scheduler engine ── on run start/end ──▶ bus.publish │
|
||||
└──────────────────────────┬──────────────────────────────────┘
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ MessageBus port (application/ports/message_bus_ports.rs) │
|
||||
│ │
|
||||
│ InProcessMessageBus │
|
||||
│ DashMap<Topic, tokio::broadcast::Sender<Event>> │
|
||||
└──────────────────────────┬──────────────────────────────────┘
|
||||
│
|
||||
│ (optional replicator seam)
|
||||
▼
|
||||
NoopReplicator (v1)
|
||||
PgListenReplicator (deferred)
|
||||
BrokerReplicator (RabbitMQ / NATS, deferred)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ WS handler (interfaces/api/handlers/rt_ws.rs) │
|
||||
│ │
|
||||
│ One session per socket: │
|
||||
│ HashMap<wire_key, Sub> + outbound mpsc │
|
||||
│ │
|
||||
│ rt.subscribe / rt.unsubscribe frames │
|
||||
│ rt.event / rt.revoked / rt.pong notifications │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Topics — a typed enum, not a string
|
||||
|
||||
```rust
|
||||
pub enum Topic {
|
||||
Folder(Uuid), // "folder:{uuid}"
|
||||
UserAuthz(Uuid), // "user:{uuid}:authz"
|
||||
UserNotifications(Uuid), // "user:{uuid}:notifications"
|
||||
Job(String), // "job:{name}"
|
||||
}
|
||||
```
|
||||
|
||||
Defined in `application/ports/message_bus_ports.rs`. Encoded to a
|
||||
stable dotted wire form; parsed back with strict validation. The
|
||||
wire form doubles as a routing key for future broker replicators
|
||||
(RabbitMQ topic exchanges, NATS subjects).
|
||||
|
||||
### Events
|
||||
|
||||
`MessageBusEvent` (same module) is the discriminated union of every
|
||||
payload a publisher can produce — `FileCreated`, `FolderMoved`,
|
||||
`AuthzChanged`, `JobRunStarted / Progress / Ended`,
|
||||
`NotificationReceived`, etc. Serde tags with `#[serde(tag = "event",
|
||||
rename_all = "snake_case")]`, so the wire is
|
||||
`{"event": "file_created", "file_id": "...", "actor": "..."}`.
|
||||
|
||||
Payloads are **thin facts**: the ID of the changed resource + the
|
||||
actor + the verb. Clients refetch details via REST if they need
|
||||
them. Keeps the AuthZ surface small (thin payloads can't leak
|
||||
fields the caller couldn't already read via REST) and keeps events
|
||||
well under any future broker's message-size cap.
|
||||
|
||||
### Wire protocol
|
||||
|
||||
JSON-RPC 2.0 over text frames. Full protocol in [`docs/plan/message-bus.md § Wire protocol`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md).
|
||||
|
||||
```jsonc
|
||||
// Client → server
|
||||
{"jsonrpc":"2.0","id":1,"method":"rt.subscribe","params":{"topic":"folder:abc-…"}}
|
||||
|
||||
// Server → client (ack)
|
||||
{"jsonrpc":"2.0","id":1,"result":{"subscribed":"folder:abc-…"}}
|
||||
|
||||
// Server → client (push, id-less notification)
|
||||
{"jsonrpc":"2.0","method":"rt.event","params":{
|
||||
"topic":"folder:abc-…",
|
||||
"event":"file_created",
|
||||
"data":{"file_id":"…","name":"…","parent_id":"…","actor":"…"}
|
||||
}}
|
||||
```
|
||||
|
||||
### Authentication for the WebSocket upgrade
|
||||
|
||||
Two paths, both accepted by the same handler:
|
||||
|
||||
| Client kind | Path | Why |
|
||||
|---|---|---|
|
||||
| **Programmatic** (CLI, test helper) | `Authorization: Bearer <jwt>` on the upgrade | The `new WebSocket()` API in browsers can attach `Sec-WebSocket-Protocol` but NOT arbitrary headers, so browsers can't do this. |
|
||||
| **Browser** | `POST /api/rt/ticket` (with the full DPoP + CSRF middleware chain) mints a one-shot 30-second opaque UUID; the browser opens the WS with `Sec-WebSocket-Protocol: oxi.ticket.<uuid>` | DPoP-bound sessions cannot attach a `DPoP:` header to `new WebSocket()`. The ticket flow moves the DPoP check to a normal POST that DOES support headers, and the WS upgrade just redeems the opaque token. |
|
||||
|
||||
Tickets are single-use, TTL 30 s, stored in a `RtTicketStore`
|
||||
(in-memory). Redemption removes the entry — replay is impossible.
|
||||
|
||||
The WS route is deliberately mounted **outside** the
|
||||
`protected_api` middleware stack — otherwise the DPoP-required
|
||||
layer would 401 every browser on upgrade before the ticket flow
|
||||
could kick in.
|
||||
|
||||
---
|
||||
|
||||
## The three AuthZ scopes
|
||||
|
||||
`Topic::required_perm(&self) -> AuthzCheck` dispatches every
|
||||
subscribe attempt into exactly one of three classes. This is the
|
||||
authoritative diagram of what the WS handler enforces:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Topic::required_perm() │
|
||||
└─────┬─────────────┬─────────────┬────────┘
|
||||
│ │ │
|
||||
ResourceRead │ IdentityMatch │ RoleAdmin
|
||||
▼ ▼ ▼
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Class 1 │ │ Class 2 │ │ Class 3 │
|
||||
│ per-resource│ │ per-user │ │ per-session │
|
||||
│ │ │ │ │ (role) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
### Class 1 — Per-resource (`AuthzCheck::ResourceRead`)
|
||||
|
||||
**Topics:** `folder:{id}`, and (future) `file:{id}`,
|
||||
`drive:{id}`, `calendar:{id}`, `addressbook:{id}`.
|
||||
|
||||
**Rule:** the caller must hold **`Read`** on the resource via the
|
||||
same `AuthorizationEngine` that guards every REST endpoint. Owner
|
||||
short-circuits pass; direct grants pass; group-mediated grants
|
||||
pass; drive-membership cascades pass. Everything else is denied.
|
||||
|
||||
**Wire response on denial:** JSON-RPC error object with
|
||||
`code = -32001`, `message = "no_read"`. Same shape whether the
|
||||
resource doesn't exist OR the caller lacks the grant — **anti-
|
||||
enumeration invariant**. Audit reason (`no_read` /
|
||||
`no_such_resource`) distinguishes internally.
|
||||
|
||||
**On grant revocation:** the WS handler auto-subscribes each
|
||||
session to `user:{caller}:authz` (Class 2 below). When a
|
||||
`MessageBusEvent::AuthzChanged { affected_folders }` fires, the
|
||||
session's reader translates it to an internal `EvictFolders`
|
||||
signal → the main loop walks the sub set and drops any Class-1
|
||||
subscription whose resource was affected, emitting a client-visible
|
||||
`rt.revoked` notification per evicted topic. Same pattern applies
|
||||
to any Class-1 topic when the AuthZ model widens beyond folders.
|
||||
|
||||
### Class 2 — Per-user, strict privacy (`AuthzCheck::IdentityMatch`)
|
||||
|
||||
**Topics:** `user:{u}:authz`, `user:{u}:notifications`.
|
||||
|
||||
**Rule:** direct UUID equality — `caller_id == u`. **No admin
|
||||
bypass, no group indirection, no owner short-circuit.** Admins
|
||||
cannot subscribe to other users' `:authz` or `:notifications`
|
||||
streams; that's a privacy invariant, not a mere policy choice.
|
||||
|
||||
**Wire response on mismatch:** `topic_forbidden` — the **same
|
||||
wire shape as an unknown topic**. An attacker probing
|
||||
`user:{someone_else_uuid}:authz` cannot distinguish "user exists
|
||||
but not me" from "no such user."
|
||||
|
||||
**Auto-subscription:** the WS handler auto-subscribes every session
|
||||
to its own `user:{caller}:authz` AND `user:{caller}:notifications`
|
||||
at session open. No `rt.subscribe` frame is needed from the client
|
||||
for these — they're always active for the caller's own UUID.
|
||||
|
||||
### Class 3 — Per-session role (`AuthzCheck::RoleAdmin`)
|
||||
|
||||
**Topics:** `job:{name}` today. Future `admin:*` topics land here.
|
||||
|
||||
**Rule:** the session's snapshotted role at open time must be
|
||||
`admin`. The handler resolves `caller_role` once during session
|
||||
setup via `resolve_live_role` and stores it on the session state —
|
||||
no per-subscribe DB round-trip.
|
||||
|
||||
**Wire response on non-admin:** `topic_forbidden` — same anti-enum
|
||||
shape as Class 2. A non-admin probing job topics cannot enumerate
|
||||
which jobs are registered.
|
||||
|
||||
**Why snapshot at session open, not per subscribe:** admin role
|
||||
loss is rare + trivially recoverable (the user closes the tab and
|
||||
reopens, hitting the fresh role check). Per-subscribe checks would
|
||||
be an extra DB round-trip on every frame with no meaningful
|
||||
security gain — the WS session itself was authenticated at upgrade
|
||||
time under the current role.
|
||||
|
||||
### Adding a new topic
|
||||
|
||||
Every new topic variant must decide which class it belongs to at
|
||||
`Topic::required_perm`. The compiler enforces exhaustiveness — a
|
||||
new variant with no branch fails to build, which is deliberate. New
|
||||
topics get audited before shipping precisely because the
|
||||
`required_perm` match forces the author to state the class
|
||||
explicitly.
|
||||
|
||||
---
|
||||
|
||||
## Tab-visibility grace-close — reducing idle connections
|
||||
|
||||
Every open browser tab holds one WebSocket to the server. A user
|
||||
with five tabs open holds five sockets. A user who leaves a tab
|
||||
open all day but only uses one holds five sockets, four of them
|
||||
serving nothing.
|
||||
|
||||
The frontend closes the WebSocket **after 60 seconds of tab
|
||||
hidden** and reopens it when the tab becomes visible again. The
|
||||
subscription state is preserved locally through the outage — every
|
||||
subscriber's release handle stays live, the reactive store still
|
||||
holds the last-known list — but the wire is silent while the tab
|
||||
is hidden.
|
||||
|
||||
### Implementation
|
||||
|
||||
Frontend `MessageBusClient` (`frontend/src/lib/message-bus/client.svelte.ts`)
|
||||
attaches a `visibilitychange` listener on construction:
|
||||
|
||||
- **Tab hidden** → starts a 60-second timer.
|
||||
- **Timer fires while still hidden** → close the WS via a
|
||||
`#closeForHidden` path that sets state to `disconnected` but
|
||||
preserves `#subs` for later replay. The `#onClose` handler is
|
||||
guarded by `#tabIsHidden()` — an auto-reconnect won't fire while
|
||||
the tab remains hidden.
|
||||
- **Tab visible again** → cancels the timer if it hadn't fired
|
||||
yet; if the WS was closed, kicks off a normal reconnect.
|
||||
- **On reopen**, the WS handler auto-subscribes to `:authz` and
|
||||
`:notifications` again, and the client replays every entry in
|
||||
`#subs` as `rt.subscribe` frames. From the user's POV, the state
|
||||
is identical to what they left behind.
|
||||
|
||||
### What this trades
|
||||
|
||||
- **Saved**: N-1 idle sockets per user with N tabs open, over the
|
||||
hidden-tab window. Meaningful at scale (100 users × 5 tabs × 8
|
||||
idle hours = 4000 idle-tab-hours of connection state to keep
|
||||
alive per day).
|
||||
- **Lost**: bus events published during the 60-second delay
|
||||
(transient) + the whole grace-close window (indefinite while
|
||||
hidden) are dropped for that session. **Recovery**: on reopen,
|
||||
every consumer that cares refetches. See "Reconnect catch-up"
|
||||
below.
|
||||
|
||||
### Why 60 seconds
|
||||
|
||||
Short enough that leaving a tab for a coffee break doesn't burn
|
||||
the connection. Long enough that the momentary focus-shifts
|
||||
users do all day (Cmd-Tab to another app, back within seconds)
|
||||
don't churn the socket. Not tunable per-user — the value is
|
||||
hard-coded in `HIDDEN_GRACE_MS`.
|
||||
|
||||
### Reconnect catch-up — the `?after=` cursor
|
||||
|
||||
For consumers whose state can't be reconstructed by refetching a
|
||||
current listing (specifically: **notifications**, whose bell must
|
||||
show rows that landed during the outage), the FE issues a delta
|
||||
fetch:
|
||||
|
||||
- Store tracks `#lastReceivedAt` — the newest `created_at` seen
|
||||
before disconnect.
|
||||
- On `messageBus.onReconnect(...)` fire, calls
|
||||
`GET /api/notifications?after=<lastReceivedAt>&limit=100`.
|
||||
- Merges the returned rows into local state via `mergeById` —
|
||||
duplicates are resolved with **incoming wins** (server value
|
||||
overrides local, so a `read_at` flip on another device shows
|
||||
up correctly).
|
||||
|
||||
The server-side `?after=` predicate is **strict `>`** — a row at
|
||||
exactly `lastReceivedAt` is excluded. This makes the WS push (which
|
||||
delivers a row at time T) and the delta fetch (which asks for
|
||||
"anything after T") non-overlapping by construction. `mergeById`
|
||||
handles the case where the two paths race and both deliver the
|
||||
same row.
|
||||
|
||||
For consumers whose state IS a current listing (folder view: the
|
||||
files/subfolders in a folder), reconnect just refetches the listing
|
||||
via the normal REST endpoint. `useReconnect` composable exposes
|
||||
`onReconnect(cb)` as a one-liner for that pattern.
|
||||
|
||||
---
|
||||
|
||||
## Persistent notifications (the bell)
|
||||
|
||||
### Data model
|
||||
|
||||
```sql
|
||||
notif.notifications (
|
||||
id UUID PRIMARY KEY,
|
||||
user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL, -- 'share_granted' | 'new_login_from_new_device' | …
|
||||
payload JSONB NOT NULL, -- per-kind shape (see below)
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
read_at TIMESTAMPTZ -- NULL = unread
|
||||
)
|
||||
```
|
||||
|
||||
Two indexes:
|
||||
|
||||
- `(user_id, created_at DESC) INCLUDE (read_at, kind)` covers the
|
||||
bell's list query + the mark-all-read filter.
|
||||
- `(read_at) WHERE read_at IS NOT NULL` — partial, tiny on healthy
|
||||
DBs; feeds the retention job's DELETE.
|
||||
|
||||
### Ingesters
|
||||
|
||||
An ingester is a code path that calls
|
||||
`NotificationApplicationService::create(NewNotification)`. The
|
||||
service atomically:
|
||||
|
||||
1. `INSERT INTO notif.notifications RETURNING …` — durable row.
|
||||
2. `bus.publish(Topic::UserNotifications(user_id), NotificationReceived)` — the fast-path poke.
|
||||
|
||||
Publish happens **after** the DB write succeeds, never inside a
|
||||
transaction — a rolled-back INSERT would otherwise fan out a lie.
|
||||
|
||||
**Currently shipped ingester:** `share_granted` in
|
||||
`interfaces/api/handlers/grant_handler.rs::create_grant`. Fires
|
||||
after `authz.set_role(...)` succeeds. Fans out to every resolved
|
||||
recipient user:
|
||||
|
||||
- `Subject::User(id)` → one row for that user.
|
||||
- `Subject::Group(id)` → one row per transitive member (via
|
||||
`SubjectGroupService::list_transitive_users`).
|
||||
- `Subject::Token(_)` → no row (anonymous share links have no
|
||||
target user).
|
||||
|
||||
Self-shares (owner grants themselves via a group they belong to)
|
||||
skip. Failure to write is best-effort — a warn log; the grant row
|
||||
stays durable in `role_grants`, the recipient can still discover
|
||||
the share via `/api/grants/shared-with-me`.
|
||||
|
||||
**Planned but not wired** (each needs a prerequisite subsystem
|
||||
listed in [`docs/plan/message-bus.md § Deferred`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md)):
|
||||
|
||||
| Kind | Prerequisite |
|
||||
|---|---|
|
||||
| `new_login_from_new_device` | Device-fingerprint tracking table |
|
||||
| `job_completed_for_you` | Scheduler engine threading the trigger caller's `caller_id` through `dispatch()` |
|
||||
| `storage_quota_threshold` | Per-user usage/quota comparator with threshold-crossing detection |
|
||||
|
||||
### Payload shape — typed per kind
|
||||
|
||||
The `payload` column is JSONB (schema-free at the DB layer). Each
|
||||
kind's Rust shape lives in `domain/entities/notification.rs` with
|
||||
`#[derive(Serialize, Deserialize, ToSchema)]`. OpenAPI picks up the
|
||||
struct automatically. Adding a new field is additive on JSONB — no
|
||||
migration.
|
||||
|
||||
Example — `share_granted`:
|
||||
|
||||
```rust
|
||||
pub struct SharegrantedPayload {
|
||||
pub granter_id: Uuid,
|
||||
pub resource_type: String, // 'folder' | 'file' | 'drive' | …
|
||||
pub resource_id: Uuid,
|
||||
pub resource_name: Option<String>, // snapshot at grant time
|
||||
pub resource_path: Option<String>, // storage path snapshot
|
||||
pub navigate_folder_id: Option<Uuid>, // FE routing target for drives
|
||||
pub role: String,
|
||||
pub expires_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
```
|
||||
|
||||
The name/path fields are **snapshotted at grant time**. If the
|
||||
folder is later renamed or moved, the notification still reflects
|
||||
what it was called when the share happened. Same principle as
|
||||
email invitations or activity feeds: the record is a fact about
|
||||
what was true at the moment, not a live pointer.
|
||||
|
||||
### Schema-ownership rule — AsyncAPI vs OpenAPI
|
||||
|
||||
> **AsyncAPI defines the envelope + transport for clients.
|
||||
> OpenAPI defines the payload.**
|
||||
|
||||
The bus event `MessageBusEvent::NotificationReceived` is a **unit
|
||||
variant** — it serializes to `{"event":"notification_received",
|
||||
"data":{}}` with no fields on the wire. The topic identifies the
|
||||
semantic; the FE responds by refetching from REST.
|
||||
|
||||
The payload's shape lives in OpenAPI via `ToSchema` on
|
||||
`SharegrantedPayload` (and its future siblings), auto-derived from
|
||||
Rust. AsyncAPI never sees these types — payloads don't travel on
|
||||
the bus wire.
|
||||
|
||||
**Why this split** — it eliminates schema drift between the two
|
||||
specs. A payload edit changes Rust → OpenAPI updates on regen
|
||||
(mechanical). AsyncAPI stays stable (hand-written, but it never
|
||||
touches payloads). Same rule applies to any future bus consumer
|
||||
that also has a REST DTO — Rust is the source of truth; each spec
|
||||
projects the parts of Rust that travel on its transport.
|
||||
|
||||
Design rationale + rejected alternatives (dual-spec, cross-`$ref`,
|
||||
per-kind Svelte components) in [`docs/plan/templated-messages.md § Schema ownership`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md).
|
||||
|
||||
### REST surface — `/api/notifications`
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/notifications` | List newest-first. Query params: `unread` (bool), `before` / `after` (cursors), `limit` |
|
||||
| `GET` | `/api/notifications/unread` | Badge-only fast path (returns just `unread_count`) |
|
||||
| `POST` | `/api/notifications/{id}/read` | Mark one row read (idempotent, always 204) |
|
||||
| `POST` | `/api/notifications/read-all` | Bulk mark-all-read (returns rows updated) |
|
||||
| `DELETE` | `/api/notifications/{id}` | Hard-delete one row (idempotent, always 204) |
|
||||
|
||||
**Anti-enumeration:** mark-read and delete always respond 204 —
|
||||
whether the row existed and belonged to the caller, or didn't
|
||||
exist at all, or belonged to someone else. Every mutating endpoint
|
||||
scopes on `caller_id` at the SQL layer; the response shape is
|
||||
identical across the three outcomes.
|
||||
|
||||
### Retention
|
||||
|
||||
The `notifications_cleanup` scheduled job (daily, same tier as
|
||||
`trash_cleanup`) DELETEs read rows older than
|
||||
`OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` (default 30). **Unread
|
||||
rows are preserved unconditionally** — a user offline for a month
|
||||
still sees the share-granted notice when they log back in.
|
||||
|
||||
Runtime override via the job's `retention_days` parameter on the
|
||||
admin panel's trigger — the env default seeds it, the panel
|
||||
overrides at trigger time.
|
||||
|
||||
### Frontend rendering
|
||||
|
||||
`frontend/src/lib/composables/useNotifications.svelte.ts` owns the
|
||||
module-scoped store — one instance per SPA session. Exposes:
|
||||
|
||||
- `notifications.items` — reactive list (newest first)
|
||||
- `notifications.unread` — reactive badge count
|
||||
- `notifications.refresh()` / `refreshDelta()` / `markRead(id)` /
|
||||
`markAllRead()` / `delete(id)`
|
||||
|
||||
`NotificationRow.svelte` handles the actual rendering — one file,
|
||||
one `switch` on `row.kind`, one rich template per shipped kind
|
||||
(`share_granted` today; the others fall back to a generic string).
|
||||
Extraction into per-kind components is deferred until a single
|
||||
kind's block exceeds ~30 lines or two kinds start needing the same
|
||||
sub-component (see [`docs/plan/templated-messages.md § Rendering`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md)).
|
||||
|
||||
### Notification click routing
|
||||
|
||||
| `resource_type` | Route | Data used |
|
||||
|---|---|---|
|
||||
| `folder` | `/files/{resource_id}` | `resource_id` |
|
||||
| `file` | `/shared-with-me?file={resource_id}` | `resource_id` — the `/files/{uuid}` route requires a **folder** id, and a file-scoped grant may not include parent-folder access. `/shared-with-me` is the guaranteed-accessible home for every recipient of a `share_granted`, and its `?file=` deep link opens the inline `FileViewer`. |
|
||||
| `drive` | `/files/{navigate_folder_id}` | Drives have no browsable URL of their own; `navigate_folder_id` is the drive's `root_folder_id`, enriched at ingest via `DriveRepository::get_by_id`. |
|
||||
| `calendar` / `address_book` / `playlist` | no link (bold text) | Not addressable via `/files/*`. |
|
||||
|
||||
The bell also fires a **transient toast** (via the existing
|
||||
`ui.notify(...)` mechanism) on every fresh row that arrives via
|
||||
delta — the toast fades out after ~4 s while the persistent row
|
||||
stays in the bell's history section. Same bell icon, same badge
|
||||
count, no duplicate UX.
|
||||
|
||||
---
|
||||
|
||||
## Feature flags & config
|
||||
|
||||
| Env var | Default | Effect |
|
||||
|---|---|---|
|
||||
| `OXICLOUD_MESSAGEBUS_ENABLE` | `true` | Master switch. When `false`, the `/api/rt/ws` and `/api/rt/ticket` routes are **not registered** at boot (Axum returns 404), and the FE `useNotifications` composable skips the WS setup entirely. The bell falls back to REST-only mode — polling on mount, delta on manual refresh. Zero client-side error spam. |
|
||||
| `OXICLOUD_MESSAGEBUS_KEEPALIVE_SECONDS` | `30` | Server-initiated RFC 6455 Ping interval on each WS connection. Prevents intermediate proxies (nginx, Traefik, Cloudflare) from reaping the TCP session as idle. |
|
||||
| `OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` | `30` | Retention window for read notifications. Unread rows are always preserved. The `notifications_cleanup` job clamps to a minimum of 1 day. |
|
||||
|
||||
Client discovers all of these via `GET /api/config` — no in-band
|
||||
"does the server support the bus?" probe needed. The FE reads
|
||||
`serverConfig.features.message_bus` at boot and skips WS setup
|
||||
entirely when it's false.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Scenario | Behavior |
|
||||
|---|---|
|
||||
| Bus is disabled server-side (`OXICLOUD_MESSAGEBUS_ENABLE=false`) | `/api/rt/ws` returns 404. `useTopic` in the FE returns early. Bell works via REST only. |
|
||||
| Network drops mid-session | Client-side jittered exponential backoff (250 ms → 30 s cap, 20-failure circuit breaker). On reconnect, WS handler re-auto-subscribes to `:authz` + `:notifications`; `useReconnect` composable fires `onReconnect` callbacks so views refetch. |
|
||||
| Server restart | Same as network drop — the WS breaks, client backs off, reconnects when server is back. Events published during the outage are lost (no persistent event log by design); consumers refetch. |
|
||||
| Tab hidden > 60 s | WS closed via `visibilitychange` grace-close. State preserved locally. On visibility return, reconnect + replay subscriptions. |
|
||||
| Publish before commit | Not allowed. Every publish site is documented as "after commit". A publish inside a transaction that rolls back would fan out a lie. |
|
||||
| Broker replicator failure (future) | The local `InProcessMessageBus` publishes still succeed — the replicator is beside the bus, not in front. Broker-hop failures affect multi-instance fanout but never local delivery. |
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
The api-test suite exercises the full stack end-to-end via `rt-hurl-helper` (a small Rust binary gated on `test_utils`) — Hurl alone can't drive a WebSocket. Sixteen scenarios in `tests/api/rt_bus_check.sh`:
|
||||
|
||||
- Positive delivery, topic isolation
|
||||
- AuthZ denial (Class 1 folder), unknown-topic anti-enum
|
||||
- Server keepalive, delete emits, move fan-out
|
||||
- Grant-revoke eviction (`AuthzChanged` → `rt.revoked`)
|
||||
- Cross-user identity gate (Class 2)
|
||||
- Ticket happy path + single-use replay refused
|
||||
- Admin-only job topic (Class 3)
|
||||
- `notification_received` wire push, DB row via `GET /api/notifications`
|
||||
- Cross-user notifications identity gate (Class 2, notifications topic)
|
||||
- `?after=` cursor with strict-`>` boundary invariant
|
||||
|
||||
`mergeById` — the FE's WS/reconnect race dedup — has its own
|
||||
Vitest with 5 covered cases (empty, non-overlap, exact-dup,
|
||||
stale-read-at overwrite, mixed overlap).
|
||||
|
||||
---
|
||||
|
||||
## Further reading
|
||||
|
||||
- Plan doc: [`docs/plan/message-bus.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md) — full design rationale, roadmap, and deferred slices (Yjs collab, broker replicator, SharedWorker, Web Push).
|
||||
- Plan doc: [`docs/plan/templated-messages.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md) — schema-ownership rule, rendering shape, notification routing decision.
|
||||
- [ReBAC Authorization](/architecture/rebac-authorization) — the engine every Class-1 topic gate calls.
|
||||
- [Background jobs](/architecture/jobs) — `notifications_cleanup` is one of them; `Topic::Job` publishes on job lifecycle.
|
||||
@@ -123,6 +123,70 @@ rather than as a visible error.
|
||||
| `OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` | `30` | Retention window for **read** notification rows (`notif.notifications`). The `notifications_cleanup` scheduled job runs daily and deletes rows where `read_at IS NOT NULL` and `read_at < now() - retention_days`. Unread rows are preserved unconditionally — the whole point of the durable table is that a user offline for a month still sees the share-granted notice on next login. Clamped to a minimum of 1 (0 would purge every read row on every tick). Adjust down for compliance-sensitive deployments where "cleared once seen" matters; adjust up when operators expect users to reference old notifications for support. |
|
||||
| `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX` | `@drive` | Native WebDAV URL segment that renders the caller's drive list. Sanitized by trimming leading/trailing `/`. Three shapes: (1) default `@drive` — `/webdav/…` addresses the caller's default personal drive (back-compat), `/webdav/@drive/` returns the drive listing, `/webdav/@drive/<uuid\|name>/…` targets a specific drive. (2) empty string `""` — `/webdav/` IS the drive listing, `/webdav/<uuid\|name>/…` targets a specific drive, no default-drive shortcut. (3) any other string (e.g. `drives`) — same shape as `@drive` with that segment substituted. Only drives the caller has Read on via `role_grants` resolve. |
|
||||
|
||||
## Boot-time refuse-to-boot checks
|
||||
|
||||
Some upgrades add invariants the running database must satisfy
|
||||
BEFORE the new binary can serve traffic. These are enforced by
|
||||
read-only checks that run after `sqlx::migrate!()` and before the
|
||||
server binds a listen socket. If a check fails, the server exits
|
||||
with a FATAL message spelling out the exact CLI command to run.
|
||||
|
||||
The server **never silently mutates data** at boot — every fix is
|
||||
an explicit `oxicloud migrate <name>` invocation. Follows the
|
||||
"discovery-only by default, mutation opt-in" rule that also
|
||||
governs the consistency-check jobs.
|
||||
|
||||
### `lowercase-usernames`
|
||||
|
||||
Three outcomes at boot, only one of which stops the server:
|
||||
|
||||
- **All lowercase (or `NULL`)** — the check is a no-op, boot
|
||||
proceeds unchanged.
|
||||
- **Mixed-case rows exist, no `LOWER(username)` collision** —
|
||||
boot **auto-lowercases** them in one atomic transaction, emits
|
||||
a structured audit line per rename
|
||||
(`user.username_lowercased_on_boot`, INFO) plus an INFO summary
|
||||
(`user.usernames_lowercased_on_boot_summary` with `renamed=N`),
|
||||
and continues. Silent action is confined to the case with
|
||||
exactly one correct move: `Alice` (with no `alice` row) becomes
|
||||
`alice`.
|
||||
- **`LOWER(username)` collision** — two or more active rows share
|
||||
the same lowercase form (e.g. `Alice` + `alice`). Boot
|
||||
**refuses to start** with a FATAL message spelling out every
|
||||
collision group and the exact CLI command to resolve it.
|
||||
Tiebreak needs a human.
|
||||
|
||||
Soft-deleted / disabled accounts (`active = false`) and `NULL`
|
||||
usernames (OPAQUE-migrated) are skipped in every case.
|
||||
|
||||
**Refusal message pattern:** `FATAL: cannot start — N colliding
|
||||
username group(s) (M affected account(s) in total)`, followed by
|
||||
each group's canonical form and its members with `id` +
|
||||
`last_login`. Up to 10 groups shown; the `--dry-run` CLI reveals
|
||||
the full list.
|
||||
|
||||
**Fix (only required when the server refused):**
|
||||
```
|
||||
oxicloud migrate lowercase-usernames --dry-run # preview the tiebreak
|
||||
oxicloud migrate lowercase-usernames # apply
|
||||
```
|
||||
|
||||
**What the migration does:** lowercases every mixed-case
|
||||
username. On collision (`Alice` + `alice` both exist), the
|
||||
tiebreak `(last_login_at DESC NULLS LAST, created_at ASC)` picks
|
||||
a winner; losers get `alice-2`, `-3`, … as a suffix. Sessions
|
||||
and grants survive the rename — both key on the user's UUID.
|
||||
|
||||
**Client compat:** NextCloud clients that cached URLs like
|
||||
`/remote.php/dav/files/Alice/…` continue to work indefinitely —
|
||||
the Basic Auth middleware and URL parser both lowercase on
|
||||
decode. NC desktop clients will prompt a one-time re-sync on
|
||||
first PROPFIND after upgrade; DAVX5 and NC mobile handle it
|
||||
silently. See `docs/install/binary.md § Upgrading from a
|
||||
case-sensitive-usernames release` for the full upgrade flow.
|
||||
|
||||
**Design:** `docs/plan/username-lowercase.md`.
|
||||
|
||||
## Storage Entries (multi-entry, recommended)
|
||||
|
||||
Declare one or more **named** storage backends. The one the app runs on is picked from the DB (`admin_settings.storage.active_backend_name`); the admin panel's storage tab flips the pointer, and cross-backend migration is a recoverable job that copies blobs between two entries with a read-only safety window. See [Admin Settings — Storage & Migration](/config/admin-settings) for the operator flow and the [multi-entry design doc](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/storage-multi-entry.md) for the full model.
|
||||
|
||||
@@ -217,6 +217,75 @@ supported by sqlx's migration model; if you need to roll back, stop
|
||||
the server, roll back your Postgres data directory to a snapshot, and
|
||||
install the previous binary.
|
||||
|
||||
### Upgrading from a case-sensitive-usernames release
|
||||
|
||||
Releases that predate the case-insensitive-usernames change stored
|
||||
`Alice`, `alice`, and `ALICE` as three separate accounts. The
|
||||
current release treats usernames as case-insensitive (canonical
|
||||
lowercase in the database). What happens on the first boot after
|
||||
upgrade depends on your data:
|
||||
|
||||
**No mixed-case usernames.** The check is a no-op; the server
|
||||
starts normally. Nothing to do.
|
||||
|
||||
**Mixed-case usernames with no collision.** The server
|
||||
**auto-lowercases** them in one atomic transaction at boot and
|
||||
continues. Each rename is recorded in the audit log
|
||||
(`user.username_lowercased_on_boot`) and a WARN summary line
|
||||
names the total count. `Alice` (with no `alice` row alongside)
|
||||
becomes `alice`; no ops action needed. This covers the vast
|
||||
majority of single-admin self-hosted deployments.
|
||||
|
||||
**Mixed-case usernames WITH a collision** (`Alice` + `alice`
|
||||
both exist as active accounts). The server **refuses to boot**
|
||||
— tiebreak requires a human. Run the migration:
|
||||
|
||||
```
|
||||
# Preview the tiebreak — no writes.
|
||||
sudo -u oxicloud DATABASE_URL="postgres://..." \
|
||||
/usr/local/bin/oxicloud migrate lowercase-usernames --dry-run
|
||||
|
||||
# Apply. Renames run in a single transaction; safe to re-run if aborted.
|
||||
sudo -u oxicloud DATABASE_URL="postgres://..." \
|
||||
/usr/local/bin/oxicloud migrate lowercase-usernames
|
||||
```
|
||||
|
||||
On collision, the migration picks a winner by `(last_login_at
|
||||
DESC NULLS LAST, created_at ASC)` — most recently active keeps
|
||||
the canonical lowercase name; the loser gets `alice-2`, `-3`, …
|
||||
as a suffix. **Sessions and grants survive the rename** — both
|
||||
key on the user's UUID, not the username.
|
||||
|
||||
Skipped in every path above: soft-deleted / disabled accounts
|
||||
and OPAQUE-migrated users whose `username` column is NULL.
|
||||
Neither blocks boot.
|
||||
|
||||
After the migration completes, restart the service:
|
||||
|
||||
```
|
||||
sudo systemctl start oxicloud
|
||||
```
|
||||
|
||||
**Nextcloud desktop clients** will prompt a one-time re-sync on
|
||||
their first PROPFIND after upgrade — the account URL case
|
||||
changed. Data is safe (files re-verify via ETag, not re-uploaded).
|
||||
DAVX5 (calendars, contacts) and NC mobile handle the URL case
|
||||
change silently. **No client upgrade or reconfiguration is
|
||||
required** — the server accepts uppercase URL segments (`Alice`
|
||||
in `/remote.php/dav/files/Alice/...`) indefinitely.
|
||||
|
||||
The refusal message printed by the server on boot (collision
|
||||
path only) includes the exact CLI command above, so you can't
|
||||
miss it. Full plan and rationale in
|
||||
`docs/plan/username-lowercase.md`.
|
||||
|
||||
**Explicit-preview path.** If you'd rather run the migration
|
||||
before the binary swap — to review renames on your own schedule
|
||||
or to gate a backup step — run `oxicloud migrate
|
||||
lowercase-usernames --dry-run` against the OLD binary's DB
|
||||
first, then apply. On next boot the new binary sees a
|
||||
lowercase-clean DB and the auto-rename path is a no-op.
|
||||
|
||||
## Installing via `cargo binstall`
|
||||
|
||||
If you already have the Rust toolchain and just want the binary
|
||||
|
||||
@@ -0,0 +1,581 @@
|
||||
# Plan — Case-insensitive usernames (lowercase-on-ingest)
|
||||
|
||||
## Context
|
||||
|
||||
Feature ask: [issue #691](https://github.com/AtalayaLabs/OxiCloud/issues/691).
|
||||
Usernames are currently case-sensitive, so `Alice`, `alice`, and `ALICE`
|
||||
refer to three different accounts. Users hit this as a login friction —
|
||||
they type their name with different capitalization on different clients
|
||||
and get "invalid credentials" instead of a successful login.
|
||||
|
||||
## Why this is simpler than it looks in this codebase specifically
|
||||
|
||||
- `validate_username` in `src/domain/entities/user.rs:884` already
|
||||
restricts usernames to ASCII-only `[a-zA-Z0-9._-]{2,64}` with no `@`.
|
||||
The Unicode case-folding minefield (Turkish dotted-I, German ß, Greek
|
||||
final sigma, NFC vs NFD) does not apply — ASCII case-folding is
|
||||
trivial (`to_ascii_lowercase`), deterministic, and locale-independent.
|
||||
- OIDC identity binds via `(iss, sub)` in
|
||||
`get_user_by_federation_subject` at
|
||||
`src/infrastructure/repositories/pg/user_pg_repository.rs:1257-1303` —
|
||||
case-sensitivity of the local username is orthogonal to OIDC identity
|
||||
matching. No OIDC breakage risk.
|
||||
- Password verification runs through Argon2's `verify_password`
|
||||
(constant-time by construction). Not affected.
|
||||
- `@`-forbidden rule in usernames is the disjoint namespace with email
|
||||
lookup (`dispatch_login` at `auth_application_service.rs:1018`).
|
||||
Case-insensitive usernames align semantics with email addresses
|
||||
(already case-insensitive in practice), so any future
|
||||
`groupname@domain` composition stays consistent.
|
||||
- NextCloud URL `/remote.php/dav/files/{user}/…` uses `{user}` as an
|
||||
informational / consistency-check marker, not a security boundary —
|
||||
the chroot ACL is the real authz. Handling case in the URL segment is
|
||||
a small local change (documented in `session.rs:33-34`).
|
||||
|
||||
## Design decisions
|
||||
|
||||
1. **Silently lowercase on ingest** (registration, admin-create, OIDC
|
||||
provisioning, rename). Never reject uppercase input from clients —
|
||||
accept liberally, store strictly (Postel's Law).
|
||||
2. **Explicit migration** (`oxicloud migrate lowercase-usernames
|
||||
[--dry-run]`). The server never mutates `auth.users` at boot. Ops
|
||||
MUST run the migration explicitly. Follows the
|
||||
[[feedback_no_silent_auto_repair]] rule: consistency tenants are
|
||||
discovery-only by default; mutation is opt-in.
|
||||
3. **Refuse-to-boot** if any active-user username is not already
|
||||
lowercase. Boot error message shows the exact CLI command to run.
|
||||
Boot performs a read-only verification only.
|
||||
4. **Collision tiebreak** on migration: `(last_login_at DESC NULLS
|
||||
LAST, created_at ASC)`. Winner keeps the canonical lowercased name.
|
||||
Losers get `-2`, `-3`, … suffix (increment until free), matching the
|
||||
pattern in `oxicloud migrate nfc-filenames`.
|
||||
5. **Active accounts only** in the boot check + migration. Soft-deleted
|
||||
/ disabled rows are skipped (they don't block usable logins). NULL
|
||||
usernames (OPAQUE-migrated accounts) are skipped in every layer —
|
||||
the boot verifier, the migration UPDATE, the CLI report. The
|
||||
`WHERE username <> LOWER(username)` predicate is already NULL-safe
|
||||
by SQL semantics (NULL comparisons yield NULL, filtered out); state
|
||||
it explicitly so a reviewer isn't left wondering.
|
||||
6. **Rename-only** on collision resolution — sessions are not
|
||||
invalidated. Sessions key on `user_id` so they survive the rename.
|
||||
7. **Un-soft-delete of a mixed-case account uses the SAME suffix
|
||||
scheme.** If `Alice` is soft-deleted (skipped by migration) and
|
||||
later un-soft-deleted while `alice` already exists, the un-soft-
|
||||
delete path re-normalizes via `set_username` and, on collision,
|
||||
assigns `alice-2` / `alice-3` / … — the same helper the migration
|
||||
CLI calls. Both callers reach for a shared
|
||||
`find_free_username_suffix(pool, base) -> String` in
|
||||
`src/common/username_migration.rs` so migration + un-soft-delete
|
||||
agree by construction. Without this, an un-soft-delete could
|
||||
create a fresh collision that the next boot's auto-rename
|
||||
couldn't resolve (auto-rename handles singletons only) — the
|
||||
server would then refuse-to-boot until an admin resolves the
|
||||
tiebreak.
|
||||
8. **Boot-time behaviour has three outcomes, not two.** The
|
||||
verifier categorises the DB into: (a) clean — nothing to do;
|
||||
(b) mixed-case rows with no `LOWER(username)` collision — the
|
||||
server **auto-lowercases them in one atomic transaction and
|
||||
continues**, emitting an audit line per rename; (c) at least
|
||||
one `LOWER(username)` collision — the server **refuses to
|
||||
boot** because tiebreak requires human judgement. Silent
|
||||
action is bounded to (b), where there is exactly one correct
|
||||
move. This is a narrower reading of
|
||||
[[feedback_no_silent_auto_repair]] than "no silent action
|
||||
ever": the rule targets consistency-check jobs where drift is
|
||||
a bug signal; a schema-adjacent boot invariant with a
|
||||
unique-correct-fix is a different situation. Making the
|
||||
trivial-case common path a no-op massively lowers upgrade
|
||||
friction for the 90% self-hosted deployment.
|
||||
|
||||
## Not in scope
|
||||
|
||||
- Unicode case-folding (usernames are ASCII-only by validation).
|
||||
- `display_name` split (usernames were already just identifiers;
|
||||
free-form display is a separate future feature if a user asks for
|
||||
it — deferred pending real demand signal).
|
||||
- OIDC provisioning behaviour change beyond the ingest-normalize point.
|
||||
- Case-insensitivity for emails (already achieved in practice; not
|
||||
touched).
|
||||
- Any change to `validate_username`'s character-class rules.
|
||||
- Any change to the WebDAV URL shape `/dav/files/{user}/…` (client
|
||||
compat; drop deferred separately per [[project_nc_multidrive_poc]]).
|
||||
- Group names (`SubjectGroup`). Lowercase by convention today; no
|
||||
runtime enforcement, no migration. If group-name case-insensitivity
|
||||
becomes a real ask, it lands as a sibling plan doc with the same
|
||||
shape.
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 1. Ingest normalization
|
||||
|
||||
Change `validate_username` to return the canonical form instead of
|
||||
`()`:
|
||||
|
||||
```rust
|
||||
// src/domain/entities/user.rs — new signature
|
||||
fn validate_username(username: &str) -> UserResult<String> {
|
||||
let normalized = username.trim().to_ascii_lowercase();
|
||||
// ... existing length + charset + boundary checks apply to `normalized` ...
|
||||
Ok(normalized)
|
||||
}
|
||||
```
|
||||
|
||||
Every caller that today does `Self::validate_username(u)?;` becomes
|
||||
`let u = Self::validate_username(&u)?;` — the returned canonical form
|
||||
is what gets stored. Because the return type changes from `Result<()>`
|
||||
to `Result<String>`, any caller that ignores the result now becomes a
|
||||
compile error — the type system forces every write path through the
|
||||
normalizer.
|
||||
|
||||
Write sites all funnel through `User::new` (`src/domain/entities/user.rs:313`)
|
||||
or `User::set_username` (`:811`), so the signature change catches the
|
||||
entity-write path automatically. Callers to touch:
|
||||
|
||||
- Application services calling `User::new`:
|
||||
- `auth_application_service.rs:840` — `register()` public signup
|
||||
- `auth_application_service.rs:944` — `setup_create_admin()`
|
||||
first-boot admin
|
||||
- `auth_application_service.rs:3520`, `:3532` — `admin_create_user()`
|
||||
external + internal branches
|
||||
- `auth_application_service.rs:4692` — OIDC JIT provisioning
|
||||
- `magic_link_invite_service.rs:233` — magic-link external invite
|
||||
|
||||
- User-driven rename (calls `User::set_username`):
|
||||
- `auth_application_service.rs:2756-2801` — `update_profile()`
|
||||
|
||||
- Repository-write compile-error catches:
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs:281`
|
||||
(`create_user` INSERT) and `:740` (`update_user` UPDATE) — these
|
||||
bind `user_clone.username()`, which is now guaranteed lowercase by
|
||||
the entity constructor.
|
||||
|
||||
### 2. OIDC JIT derivation
|
||||
|
||||
`auth_application_service.rs:4649-4690` derives a local username from
|
||||
the OIDC `preferred_username` / `name` / `sub` claims, filters to
|
||||
`[a-zA-Z0-9._-]`, and truncates. **It does not currently lowercase.**
|
||||
Add `to_ascii_lowercase()` on the derived string before passing to
|
||||
`User::new`. This is beyond what the entity signature change catches —
|
||||
explicit fix required.
|
||||
|
||||
### 3. Lookup normalization
|
||||
|
||||
Repository `find_by_username`-style methods internally lowercase the
|
||||
input before the SQL query, so callers don't have to remember. One-line
|
||||
change per method:
|
||||
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs:487`
|
||||
(`get_user_by_username`) — add `let username = username.trim().
|
||||
to_ascii_lowercase();` before the `.bind(&username)` at line 488.
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1043`
|
||||
(`search_users`) — `ILIKE` is already case-insensitive by
|
||||
construction; verify nothing regresses.
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1504`
|
||||
(`search_usernames`) — same as above.
|
||||
- `src/application/services/storage_usage_service.rs:145-149`
|
||||
(`update_user_storage_usage_by_username`) — raw SQL bind; normalize
|
||||
before `.bind()`.
|
||||
- `src/cli/opaque.rs:125`, `:202` — `opaque reset` CLI identifier
|
||||
dispatch on `@`; lowercase the username branch input.
|
||||
|
||||
Post-migration, the DB is fully lowercase so `WHERE username = 'alice'`
|
||||
matches. The mixed-case-DB-during-transition state cannot serve
|
||||
traffic because the boot flow either (a) auto-renames the singleton
|
||||
rows before `AppState` assembles, or (b) refuses to boot on collision
|
||||
groups.
|
||||
|
||||
### 4. NextCloud DAV surface
|
||||
|
||||
Two coordinated changes on the NC surface:
|
||||
|
||||
- `src/interfaces/nextcloud/basic_auth_middleware.rs:94-134` — decoded
|
||||
`raw_username` from the Basic Auth header, lowercase the whole
|
||||
string. Safe for the `user~drive_uuid` multi-drive format because
|
||||
UUID hex is `[0-9a-f-]` which lowercases to itself.
|
||||
- `src/interfaces/nextcloud/basic_auth_middleware.rs:307-323`
|
||||
(`parse_basic_auth` helper) — lowercase the username portion before
|
||||
returning.
|
||||
- `src/interfaces/nextcloud/session.rs:90-111`
|
||||
(`extract_url_user`) — lowercase the returned `Cow<'_, str>` value
|
||||
from URL decode. The cross-check comparison at `session.rs:157-161`
|
||||
(`url_user != session.raw_username`) then compares normalized vs
|
||||
normalized — no change needed at the comparison site itself.
|
||||
|
||||
Downstream `session.raw_username` consumers (WebDAV / OCS href
|
||||
builders, MOVE Destination parsers, avatar / trashbin handlers) all
|
||||
pass through and emit lowercase automatically — no per-site change
|
||||
needed.
|
||||
|
||||
**Client compatibility:** NC / DAVX5 clients that cached URLs like
|
||||
`/remote.php/dav/files/Alice/…` continue to work through the migration
|
||||
because the server accepts uppercase URL segments **indefinitely**
|
||||
(the Basic Auth middleware + `extract_url_user` both lowercase on
|
||||
decode). No forced client upgrade or reconfiguration. PROPFIND
|
||||
response bodies emit lowercase hrefs (from canonical
|
||||
`session.raw_username`), which well-behaved clients update on next
|
||||
sync.
|
||||
|
||||
Expected per-client behavior on first PROPFIND after upgrade:
|
||||
|
||||
- **Nextcloud desktop** — prompts a one-time re-sync notification
|
||||
when it notices the account URL case changed. Files re-verify
|
||||
via ETag, so no re-upload; the re-sync completes in
|
||||
seconds-to-minutes depending on file count. Users click through
|
||||
the reconnect dialog.
|
||||
- **DAVX5** (calendars, contacts) — silent update of the internal
|
||||
`principal-URL`; user sees no dialog.
|
||||
- **NC mobile app** — silent refresh of the account tile.
|
||||
- **Older / misbehaving clients** — may create a duplicate account
|
||||
profile (rare, cosmetic, not destructive).
|
||||
|
||||
**Zero data risk in every path.** The chroot ACL keys on
|
||||
`user_id`, not username, so files, calendars, contacts, and
|
||||
grants all follow the user across the rename. The blast radius
|
||||
is a one-time UX notification, not lost bytes.
|
||||
|
||||
**Power-user pre-emption** (worth documenting in CHANGELOG): ops
|
||||
who want to avoid the re-sync prompt entirely can, before
|
||||
upgrading, log into each NC desktop client and manually update
|
||||
the account URL from `.../USERNAME` to lowercase. Cheap
|
||||
prophylactic for organizations rolling out to non-technical
|
||||
users.
|
||||
|
||||
### 5. Chunked-upload directory rename
|
||||
|
||||
`src/infrastructure/services/nextcloud_chunked_upload_service.rs:99-103`
|
||||
uses `user.username` as an on-disk directory name AND as an in-memory
|
||||
cache key. Post-migration, `user.username` becomes lowercase; any
|
||||
in-flight upload for `Alice` at migration time strands the on-disk
|
||||
`base_dir/Alice/upload_xxx/` directory and orphans its cache entry.
|
||||
|
||||
The migration command SHOULD walk `base_dir/*/` and rename any
|
||||
mixed-case subdirectory to its lowercase form. Collision handling
|
||||
(both `Alice/` and `alice/` present) → merge contents; else simple
|
||||
rename. In practice this is likely a no-op — chunked-upload state
|
||||
is ephemeral, and simultaneous mixed-case uploads by the same user
|
||||
are rare.
|
||||
|
||||
**Implementation status:** deferred. Chunked-upload state is
|
||||
ephemeral: any in-flight upload that gets stranded is retryable
|
||||
by the client (the upload session's timeout eventually purges the
|
||||
stale dir; the client retries with a fresh `upload_id`, this time
|
||||
under the lowercase username). Wiring the dir-walk into the CLI
|
||||
adds ~40 lines of async filesystem code (walk, collision merge,
|
||||
mtime-preserving move) and a new `--chunk-dir <path>` arg — the
|
||||
CLI otherwise doesn't need to know about the storage-path
|
||||
config layer. Not worth it for a rare no-op; add if user reports
|
||||
show a real problem.
|
||||
|
||||
**Ops manual step** — if a migration is run WHILE an upload is
|
||||
in flight, ops can either restart the affected client (the
|
||||
upload session is stateful across a `create → chunks → complete`
|
||||
cycle, so the client will retry from scratch) or manually
|
||||
`mv base_dir/Alice base_dir/alice` after the DB migration
|
||||
completes.
|
||||
|
||||
### 6. Boot-time verification
|
||||
|
||||
New module `src/common/username_migration.rs` exposing:
|
||||
|
||||
```rust
|
||||
pub async fn verify_all_usernames_lowercase(pool: &PgPool) -> Result<(), String>
|
||||
```
|
||||
|
||||
Runs after `sqlx::migrate!()` completes, before `AppState` is
|
||||
assembled. Query:
|
||||
|
||||
```sql
|
||||
SELECT id, username, created_at, last_login_at
|
||||
FROM auth.users
|
||||
WHERE username <> LOWER(username)
|
||||
-- NULL usernames (OPAQUE-migrated accounts) are already filtered
|
||||
-- out by SQL semantics: NULL <> anything yields NULL, which
|
||||
-- WHERE excludes. Explicit for the reviewer's benefit.
|
||||
-- add is_deleted / disabled filter if such a flag exists
|
||||
ORDER BY LOWER(username),
|
||||
(last_login_at IS NULL),
|
||||
last_login_at DESC NULLS LAST,
|
||||
created_at ASC
|
||||
LIMIT 200; -- soft cap on error-message size
|
||||
```
|
||||
|
||||
If empty → boot proceeds. If non-empty → format the FATAL error and
|
||||
return `Err(String)`. `main.rs` propagates via `?` to a non-zero
|
||||
process exit.
|
||||
|
||||
Boot only READS `auth.users`; never WRITES. This is the "explicit
|
||||
migration required" enforcement layer.
|
||||
|
||||
**Error message format** (self-sufficient — no docs required at 3 AM):
|
||||
|
||||
```
|
||||
FATAL: cannot start — <N> user account(s) have non-lowercase usernames.
|
||||
|
||||
Before this version can boot, run the migration:
|
||||
|
||||
oxicloud migrate lowercase-usernames --dry-run # preview
|
||||
oxicloud migrate lowercase-usernames # apply
|
||||
|
||||
Affected accounts (up to 20 shown; full list via the dry-run):
|
||||
|
||||
Alice (id: a1b2c3d4-... last_login: 2026-08-01)
|
||||
BOB (id: 9abc0000-... last_login: never)
|
||||
...
|
||||
|
||||
The migration handles case-collisions (Alice + alice → alice keeps
|
||||
the name based on most recent login; the other gets alice-2 suffix).
|
||||
Sessions and grants survive the rename (they key on user_id).
|
||||
```
|
||||
|
||||
### 7. Migration CLI
|
||||
|
||||
New action under `oxicloud migrate`:
|
||||
|
||||
```rust
|
||||
// src/cli/migrate.rs — extend the Action enum
|
||||
Action::LowercaseUsernames { dry_run: bool }
|
||||
```
|
||||
|
||||
Following the shape of `run_nfc_filenames`:
|
||||
|
||||
- Load all active users (skip soft-deleted / disabled AND rows
|
||||
where `username IS NULL` — OPAQUE-migrated accounts have no
|
||||
username string to normalize)
|
||||
- Group by `LOWER(username)`
|
||||
- For each group:
|
||||
- Single-member group with mixed-case name → UPDATE to lowercase
|
||||
- Multi-member group (collision) → apply tiebreak
|
||||
`(last_login_at DESC NULLS LAST, created_at ASC)`, winner UPDATEs
|
||||
to lowercase, losers UPDATE to `<lowercase>-2`, `-3`, … (increment
|
||||
until free)
|
||||
- Per-row `println!` log:
|
||||
`NORMALIZE user=<uuid> '<before>' ({}B) → '<after>' ({}B)`
|
||||
- Summary at end: scanned / already-lowercase / normalized /
|
||||
collision-resolved / renamed-to-suffix
|
||||
- `--dry-run` guards all UPDATEs
|
||||
|
||||
After the DB pass, the chunked-upload directory rename step (see
|
||||
Deliverable 5) is deferred; run manually only if in-flight uploads
|
||||
were live at migration time.
|
||||
|
||||
Suffix search reuses the pattern from
|
||||
`find_free_folder_duplicate_name` in the existing NFC migration —
|
||||
increment-until-free loop, starting at `-2`, probing until an
|
||||
unused suffix is found. Robust against pre-existing rows like
|
||||
`alice-2` already being taken (the probe just steps past them
|
||||
to `-3`, `-4`, …).
|
||||
|
||||
Extracted into a shared public helper in
|
||||
`src/common/username_migration.rs`:
|
||||
|
||||
```rust
|
||||
pub async fn find_free_username_suffix(pool: &PgPool, base: &str) -> Result<String, sqlx::Error>
|
||||
```
|
||||
|
||||
Both the migration CLI AND the un-soft-delete API (Design decision
|
||||
7) call this helper — same collision-resolution behavior by
|
||||
construction, no drift risk between the two paths.
|
||||
|
||||
Bounded at 10,000 as a safety cap. The probability of reaching
|
||||
that in a real deployment is negligible — it would require ~10 K
|
||||
distinct accounts all originally cased differently but sharing
|
||||
the same lowercase form (a normal collision is 2-3 accounts, not
|
||||
10 K). If the cap ever fires, something is very wrong with the
|
||||
account universe and the migration ABORTs with a loud error
|
||||
rather than silently truncating — the loud abort IS the
|
||||
detection mechanism.
|
||||
|
||||
### 8. Test seed audit
|
||||
|
||||
Sweep-verified: existing test seeds all produce lowercase or NULL
|
||||
usernames. Worth one more grep pass to ensure no test fixture INSERTs
|
||||
`INSERT INTO auth.users … 'AliceTest'` — if any exist, lowercase them
|
||||
in the same commit to avoid CI refuse-to-boot regressions.
|
||||
|
||||
Files verified (all safe):
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1786`
|
||||
- `src/infrastructure/repositories/pg/opaque_pg_repository.rs:339`
|
||||
(NULL)
|
||||
- `src/application/services/auth_application_service.rs:4982` (NULL)
|
||||
- `src/application/services/subject_group_service.rs:796` (NULL)
|
||||
- `src/bin/load-seed.rs:414`, `:446` (`load_user_XXXX` — lowercase)
|
||||
- `src/mount_it_support.rs:61` (`make_user(name)` — verify callers)
|
||||
- `tests/common/init-test-schema.sh:40` (`ci-admin` — lowercase)
|
||||
|
||||
### 9. Cosmetic side-effects (worth noting in CHANGELOG, non-blocking)
|
||||
|
||||
- `src/interfaces/nextcloud/avatar_handler.rs:283` — `pick_color`
|
||||
derives a deterministic tile color from username bytes. Users whose
|
||||
canonical username had uppercase letters will get a different
|
||||
fallback-avatar tile color after the migration. One-time cosmetic
|
||||
change.
|
||||
- **NC desktop may perform a one-time re-sync** — see Deliverable 4.
|
||||
|
||||
### 10. Documentation
|
||||
|
||||
Release notes / CHANGELOG entry is NOT part of this PR — the
|
||||
canonical repo's maintainer handles release notes at version-bump
|
||||
time. This PR just leaves the notes-worthy items enumerated here
|
||||
so the maintainer has the bullets to pick from when the next
|
||||
version ships:
|
||||
|
||||
- Server auto-lowercases non-colliding mixed-case usernames at
|
||||
first boot. No ops action needed for the common case.
|
||||
- On `LOWER(username)` collision (`Alice` + `alice` both active),
|
||||
the server refuses to boot; ops runs `oxicloud migrate
|
||||
lowercase-usernames`. Exact CLI command shown in the refusal.
|
||||
- Nextcloud desktop clients will prompt for a one-time re-sync on
|
||||
first PROPFIND after upgrade. Files are ETag-verified, not
|
||||
re-uploaded. DAVX5 and NC mobile handle the URL case change
|
||||
silently. **No forced client upgrade or reconfiguration** —
|
||||
server accepts uppercase URL segments indefinitely.
|
||||
- Optional pre-emption for non-technical users: ops can manually
|
||||
update the account URL to lowercase in each NC desktop client
|
||||
before upgrading, avoiding the re-sync prompt entirely.
|
||||
- Usernames become lowercase in ALL UI display surfaces (share
|
||||
dialogs, activity feeds, admin panels, PROPFIND response
|
||||
bodies, notification bell). Login identity unchanged from the
|
||||
user's POV (they can still type any case at the login form).
|
||||
- Avatar fallback color may change for users with previously-
|
||||
uppercase usernames.
|
||||
- Note: `display_name` is a possible follow-up if users miss
|
||||
capitalisation for display — deferred pending demand signal, no
|
||||
compat cost to adding later.
|
||||
|
||||
The two docs that DO ship with this PR:
|
||||
|
||||
- `docs/config/env.md` — note the boot-time check + migration command.
|
||||
- `docs/install/binary.md` — upgrade-from-case-sensitive section.
|
||||
|
||||
### 11. Test coverage
|
||||
|
||||
- **Unit**: `validate_username("Alice")` returns `Ok("alice")`;
|
||||
`validate_username("alice-")` returns `Err(...)` unchanged;
|
||||
`validate_username(" Alice ")` returns `Ok("alice")`.
|
||||
- **Unit**: `format_refusal_message_collisions` — 1 group renders
|
||||
canonical + members + CLI; > 10 groups renders overflow tail;
|
||||
total-affected-count sums across groups.
|
||||
- **Hurl** (`tests/api/lowercase_usernames.hurl`, new): register a
|
||||
user with `MixedCase`, assert DB stores `mixedcase`; log in with
|
||||
`MIXEDCASE` and `mixedcase` — both succeed; rename to `NewName`,
|
||||
assert `newname` stored; NC Basic Auth accepts `MixedCase:pass`,
|
||||
`MIXEDCASE:pass`, `mixedcase:pass`.
|
||||
- **Manual** (against dev DB, not CI):
|
||||
- Auto-rename path: `UPDATE auth.users SET username='Alice' WHERE
|
||||
username='alice'` (no collision); boot server → verify audit log
|
||||
line + WARN summary, row is `alice` after boot, service starts.
|
||||
- Collision path: `INSERT INTO auth.users … 'Alice'` on top of
|
||||
existing `alice`; boot server → verify refusal message names both
|
||||
rows + exact CLI shown, server exits non-zero.
|
||||
- `oxicloud migrate lowercase-usernames --dry-run` → verify report
|
||||
of collision + tiebreak decision
|
||||
- `oxicloud migrate lowercase-usernames` → verify apply, one row
|
||||
keeps `alice`, other gets `alice-2`
|
||||
- Boot again → succeeds (Clean outcome)
|
||||
- `curl -u ALICE:pass https://oxicloud/remote.php/dav/files/ALICE/…`
|
||||
→ succeeds (accepts uppercase input, resolves to lowercase user)
|
||||
|
||||
## Cache-and-consistency observations (informational)
|
||||
|
||||
- `src/infrastructure/services/login_lockout_service.rs:33,68-89` —
|
||||
already lowercases the key at line 58. No code change; comment
|
||||
becomes factual not incidental.
|
||||
- `src/application/services/app_password_service.rs:89,317-323` —
|
||||
BLAKE3-keyed cache using the raw wire username. Post-normalization,
|
||||
both sides normalize consistently → cache stays coherent. 300 s TTL
|
||||
self-heals any transitional window.
|
||||
- `NC_CHROOT_CACHE` in `basic_auth_middleware.rs:34-40` — keyed on
|
||||
`Uuid`, not username. Unaffected.
|
||||
|
||||
## Critical files
|
||||
|
||||
Full enumeration in the Deliverables sections above. Grouped summary:
|
||||
|
||||
**Ingest normalizer:**
|
||||
- `src/domain/entities/user.rs` (signature change + callers)
|
||||
|
||||
**Application services (write callers):**
|
||||
- `src/application/services/auth_application_service.rs`
|
||||
- `src/application/services/magic_link_invite_service.rs`
|
||||
|
||||
**Repositories (lookup normalization):**
|
||||
- `src/infrastructure/repositories/pg/user_pg_repository.rs`
|
||||
- `src/application/services/storage_usage_service.rs`
|
||||
- `src/cli/opaque.rs`
|
||||
|
||||
**NextCloud DAV surface:**
|
||||
- `src/interfaces/nextcloud/basic_auth_middleware.rs`
|
||||
- `src/interfaces/nextcloud/session.rs`
|
||||
- `src/infrastructure/services/nextcloud_chunked_upload_service.rs`
|
||||
|
||||
**New files:**
|
||||
- `src/common/username_migration.rs`
|
||||
- `tests/api/lowercase_usernames.hurl`
|
||||
|
||||
**Main entry:**
|
||||
- `src/main.rs` (call verifier after `sqlx::migrate!()`)
|
||||
|
||||
**Migration CLI:**
|
||||
- `src/cli/migrate.rs`
|
||||
|
||||
**Docs:**
|
||||
- `CHANGELOG.md`
|
||||
- `docs/config/env.md`
|
||||
- `docs/install/binary.md`
|
||||
|
||||
## Delivery order
|
||||
|
||||
1. Change `validate_username` signature to return `Result<String>` —
|
||||
one file.
|
||||
2. Fix OIDC JIT derivation
|
||||
(`auth_application_service.rs:4649-4690`) to lowercase before
|
||||
passing to `User::new` — explicit change beyond the entity
|
||||
normalizer's compile-time catches.
|
||||
3. Iterate on compile errors — the return-type change catches every
|
||||
downstream write-site.
|
||||
4. Update repository lookup methods (`user_pg_repository.rs`,
|
||||
`storage_usage_service.rs`, `cli/opaque.rs`) to internally
|
||||
lowercase input before `.bind()`.
|
||||
5. Update NC `basic_auth_middleware.rs` (lowercase `raw_username` at
|
||||
decode) + `session.rs::extract_url_user` (lowercase return).
|
||||
6. Add the boot-time verification helper (`src/common/username_migration.rs`)
|
||||
+ wire into `main.rs`.
|
||||
7. Extend `oxicloud migrate` with `lowercase-usernames [--dry-run]` —
|
||||
DB pass + chunked-upload directory rename.
|
||||
8. Test seed audit (grep pass).
|
||||
9. Add hurl coverage.
|
||||
10. Admin docs update (`docs/config/env.md` boot-check subsection +
|
||||
`docs/install/binary.md` upgrade section). CHANGELOG is Dio's
|
||||
job at version-bump time — not part of this PR.
|
||||
11. Manual smoke test against dev DB.
|
||||
12. PR to canonical.
|
||||
|
||||
## Total scope estimate
|
||||
|
||||
~6-8 hours of careful work. Larger than the initial estimate because
|
||||
of these sweep-surfaced additions:
|
||||
|
||||
- OIDC JIT explicit fix (small).
|
||||
- Chunked-upload directory rename step in the migration (~30 min).
|
||||
- Test-seed audit (~15 min).
|
||||
- More lookup callsites than initially thought.
|
||||
|
||||
The shape is uniform (`to_ascii_lowercase()` at every touchpoint) and
|
||||
the compiler catches missed entity-write sites via the
|
||||
`Result<String>` signature change. The parts NOT caught by the
|
||||
compiler (OIDC JIT, lookup normalizers, NC URL segment, chunked-upload
|
||||
directory) are the ones needing careful review — enumerated above.
|
||||
|
||||
## References
|
||||
|
||||
- Issue: [#691](https://github.com/AtalayaLabs/OxiCloud/issues/691)
|
||||
- Related feature restrictions today:
|
||||
- `validate_username` at
|
||||
`src/domain/entities/user.rs:884-916`
|
||||
- `@`-disjoint dispatch at
|
||||
`src/application/services/auth_application_service.rs:1018`
|
||||
- Related project docs:
|
||||
- `docs/plan/auth-simplification.md` — the broader auth surface this
|
||||
fits within
|
||||
- Prior similar migration:
|
||||
`oxicloud migrate nfc-filenames` in `src/cli/migrate.rs`
|
||||
@@ -15,6 +15,15 @@ Non-obvious rules that trip up new code. Terse on purpose.
|
||||
- Any new endpoint that mints or consumes credentials/tokens must consult one of the `is_*_login_allowed()` helpers, not the raw allowlist.
|
||||
- Any new "policy-disabled" refusal must emit an `audit`-target line before returning — matches `auth.login_rejected`, `magic_link.redemption_rejected` conventions.
|
||||
|
||||
## AuthZ enforcement points
|
||||
|
||||
- **The extractors are NOT a choke point.** Four paths authenticate without ever touching `AuthUser` / `CurrentUserId`: `middleware/admin.rs::require_authenticated` (re-parses the JWT from header *or* cookie itself), the three DAV handlers' hand-rolled `extract_user` (`webdav_handler.rs:154`, `caldav_handler.rs:421`, `carddav_handler.rs:173`), `POST /api/auth/refresh` (mounted outside `auth_middleware`, `main.rs:775`), and `GET /api/rt/ws` (self-auths from a raw Bearer and never reads `claims.role`, `rt_ws.rs:240-255`). A rule added to an extractor does not hold until it is added to these too.
|
||||
- **Never hand-roll principal extraction.** `req.extensions().get::<Arc<CurrentUser>>()` inside a handler is exactly the anti-pattern above — it bypasses every `FromRequestParts` guard. Take the extractor, or call the shared assertion.
|
||||
- **A method check is not an authorization check.** GETs that mint credentials or write exist: `GET /api/wopi/editor-url` returns a WOPI token usable for `POST /wopi/files/{id}/contents`; `GET /api/s/{token}` writes via `register_shared_link_access`; `GET /api/auth/device/verify` is an oracle on live device codes; `GET /api/batch/download` builds an arbitrary ZIP from a querystring. Never gate on verb alone.
|
||||
- **An auth helper's `_ =>` arm must DENY.** Two fail-open gates exist and are bugs, not patterns to copy: `require_internal_user` (`middleware/user.rs:64-71`) admits the caller on *any* `get_user_flags` error, and `decide_live_role` (`:169-176`) resurrects the claim role on a transient DB error. The first is the only middleware guarding all three DAV surfaces.
|
||||
- **Prefer deny-by-default over assert-in-handler.** A restriction enforced inside the extractor covers ~200 call sites with no edits; the same restriction as "handler takes an optional principal and asserts" is one forgotten call away from silently accepting. `OptionalUserId` (`middleware/auth.rs:85-98`) is the cautionary tale — it exists, it is dead code, and nothing ever used it.
|
||||
- Anonymous-session direction (share links as a principal): `docs/plan/rationalize-publicshare.md`.
|
||||
|
||||
## Storage backend access
|
||||
|
||||
- **Read blob content through `Arc<DedupService>`.** It's the ONE canonical read abstraction — CDC-manifest-aware (`file.blob_hash` may reference a chunk manifest, not a blob), backend-agnostic (Local/S3/Azure), wrapper-transparent (encryption/retry/cache). Never take `Arc<dyn BlobStorageBackend>` directly in a service that reads content; you'll silently break on any file ≥ 64 KiB (`CDC_MIN_CHUNK`). Follow `thumbnail_service`, `audio_metadata_service`, `media_metadata_service`, `face_indexing_service`, `search_index::content_index_worker` as reference impls.
|
||||
@@ -30,3 +39,9 @@ Non-obvious rules that trip up new code. Terse on purpose.
|
||||
- **After adding: `cargo run --bin generate-openapi`** to regenerate `resources/gen/openapi.json`, then `git diff resources/gen/openapi.json` — the new path + its request/response schemas must be present. Zero-diff means you missed the registration.
|
||||
- Sanity check for the whole surface: `diff <(grep -oE 'path = "/api[^"]+"' src/interfaces/api/handlers/*.rs | grep -oE '/api[^"]+' | sort -u) <(jq -r '.paths | keys | .[]' resources/gen/openapi.json | sort -u)` — should always be empty. Non-empty diff = drift.
|
||||
- Handlers referenced by the `paths(...)` list MUST be `pub` (module-visible from the paths list). Private `async fn` compiles at the router mount but breaks the paths list with a visibility error — see `get_smtp_info`, `send_smtp_test`, `get_user_profile` for the retrofit.
|
||||
|
||||
### Security / scope in the spec
|
||||
|
||||
- **`security(("bearerAuth" = []))` — the empty array is the SCOPES list**, not decoration. Every route currently declares the same thing, so the spec claims "a session is required" for `GET /api/version` and `PUT /api/admin/users/{id}/role` alike: true, and useless. If a route's gate differs from the default, declare it there.
|
||||
- OpenAPI has **no field for a minimum role** — OAuth2 has no role concept, so the spec has nowhere to put one. Use a pseudo-scope (`["role:admin"]`) rather than a vendor extension no tooling renders.
|
||||
- **Declaring is not enforcing.** utoipa's `security` wires nothing, so it drifts from the real gate silently. Any scope worth declaring is worth a test cross-checking it against the actual mount — otherwise the spec becomes a parallel description of the authorization boundary rather than a picture of it.
|
||||
|
||||
@@ -3425,34 +3425,37 @@ impl AuthApplicationService {
|
||||
&self,
|
||||
dto: crate::application::dtos::settings_dto::AdminCreateUserDto,
|
||||
) -> Result<FullUserDto, DomainError> {
|
||||
// Validate username length
|
||||
if dto.username.len() < 3 || dto.username.len() > 254 {
|
||||
return Err(DomainError::new(
|
||||
ErrorKind::InvalidInput,
|
||||
"User",
|
||||
"Username must be between 3 and 254 characters".to_string(),
|
||||
));
|
||||
}
|
||||
// Normalise the username up-front — trim + lowercase — and use
|
||||
// the canonical form for every downstream check + generated
|
||||
// value below. `User::new` also normalises internally, but the
|
||||
// placeholder-email fallback and the duplicate-check error
|
||||
// message live above that call, so they'd otherwise capture the
|
||||
// raw wire input (e.g. `UpperCase@oxicloud.local`).
|
||||
// See docs/plan/username-lowercase.md § Design decision 5.
|
||||
let username = User::validate_username(&dto.username).map_err(|e| {
|
||||
DomainError::new(ErrorKind::InvalidInput, "User", format!("Username: {e}"))
|
||||
})?;
|
||||
|
||||
// Check for duplicate username
|
||||
if self
|
||||
.user_storage
|
||||
.get_user_by_username(&dto.username)
|
||||
.get_user_by_username(&username)
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
return Err(DomainError::new(
|
||||
ErrorKind::AlreadyExists,
|
||||
"User",
|
||||
format!("User '{}' already exists", dto.username),
|
||||
format!("User '{username}' already exists"),
|
||||
));
|
||||
}
|
||||
|
||||
// Email: use provided or generate placeholder
|
||||
// Email: use provided or generate placeholder from the
|
||||
// canonical (lowercase) username.
|
||||
let email = dto
|
||||
.email
|
||||
.filter(|e| !e.trim().is_empty())
|
||||
.unwrap_or_else(|| format!("{}@oxicloud.local", dto.username));
|
||||
.unwrap_or_else(|| format!("{username}@oxicloud.local"));
|
||||
|
||||
// Check email uniqueness
|
||||
if self.user_storage.get_user_by_email(&email).await.is_ok() {
|
||||
@@ -3519,7 +3522,7 @@ impl AuthApplicationService {
|
||||
let user = if is_external {
|
||||
User::new(
|
||||
email,
|
||||
Some(dto.username.clone()),
|
||||
Some(username.clone()),
|
||||
Some(password_hash),
|
||||
None, // federation_kind: admin-created external, no federation link yet
|
||||
None, // federation_issuer
|
||||
@@ -3531,7 +3534,7 @@ impl AuthApplicationService {
|
||||
} else {
|
||||
User::new(
|
||||
email,
|
||||
Some(dto.username.clone()),
|
||||
Some(username.clone()),
|
||||
Some(password_hash),
|
||||
None, // federation_kind: admin-created local user
|
||||
None, // federation_issuer
|
||||
@@ -3750,8 +3753,33 @@ impl AuthApplicationService {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Activate or deactivate a user (admin only)
|
||||
/// Activate or deactivate a user (admin only).
|
||||
///
|
||||
/// **Reactivation collision handling** — when a deactivated account
|
||||
/// holds a mixed-case username from before the lowercase-usernames
|
||||
/// migration (its row was skipped by that migration precisely
|
||||
/// because it was deactivated), reactivating it can produce a
|
||||
/// username collision if `LOWER(other.username) == LOWER(this.username)`
|
||||
/// for an active row. We resolve the collision by:
|
||||
///
|
||||
/// 1. Re-normalising via `User::set_username` (returns the
|
||||
/// canonical lowercase form on success).
|
||||
/// 2. If the canonical form is already taken by another active
|
||||
/// row, probe `<lower>-2`, `-3`, … via the shared
|
||||
/// `find_free_username_suffix` helper.
|
||||
/// 3. Persist the resolved name BEFORE flipping `active = true`
|
||||
/// so no time window has two-active-users with the same
|
||||
/// lowercase form.
|
||||
///
|
||||
/// See `docs/plan/username-lowercase.md § Design decision 7`.
|
||||
/// Without this, un-soft-deleting the only pre-migration
|
||||
/// mixed-case survivor would refuse-to-boot on the next restart.
|
||||
pub async fn set_user_active(&self, user_id: Uuid, active: bool) -> Result<(), DomainError> {
|
||||
// Only the activate direction needs the collision-resolution
|
||||
// dance — deactivation just flips a bit.
|
||||
if active {
|
||||
self.resolve_reactivation_collision(user_id).await?;
|
||||
}
|
||||
self.user_storage
|
||||
.set_user_active_status(user_id, active)
|
||||
.await?;
|
||||
@@ -3759,6 +3787,93 @@ impl AuthApplicationService {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Pre-flight for `set_user_active(active = true)`: ensures the
|
||||
/// target user's username is canonical (lowercase) AND unique
|
||||
/// against currently-active accounts. Renames the target row if
|
||||
/// either invariant would break.
|
||||
///
|
||||
/// NULL usernames (OPAQUE-migrated accounts) are a no-op — nothing
|
||||
/// to normalise, nothing to collide.
|
||||
async fn resolve_reactivation_collision(&self, user_id: Uuid) -> Result<(), DomainError> {
|
||||
let target = self.user_storage.get_user_by_id(user_id).await?;
|
||||
let Some(current) = target.username().map(str::to_string) else {
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
let canonical = current.to_ascii_lowercase();
|
||||
|
||||
// Look for another ACTIVE user holding the canonical form.
|
||||
// The migration CLI's `find_free_username_suffix` probes
|
||||
// directly via a pool; here we don't have the pool
|
||||
// (`AuthApplicationService` holds a `dyn UserRepository`
|
||||
// trait object). Use `get_user_by_username` — the repo
|
||||
// normalises input to lowercase before bind, so this
|
||||
// resolves against the canonical row.
|
||||
let collision = self
|
||||
.user_storage
|
||||
.get_user_by_username(&canonical)
|
||||
.await
|
||||
.ok()
|
||||
.filter(|other| other.id() != user_id && other.is_active());
|
||||
|
||||
let chosen_name = match collision {
|
||||
None => canonical,
|
||||
Some(_) => {
|
||||
// Collision — probe `<canonical>-2`, `-3`, … via
|
||||
// repository lookups. Same shape as the migration
|
||||
// CLI's `find_free_username_suffix`, just against
|
||||
// the repo trait instead of a raw pool. Both paths
|
||||
// agree by construction on the numbering scheme.
|
||||
//
|
||||
// Cap at 10_000 (matches the shared helper's cap —
|
||||
// see `docs/plan/username-lowercase.md § 3. Suffix-
|
||||
// collision robustness`). Reaching the cap means
|
||||
// the account universe has an anomaly worth
|
||||
// investigating; loud abort beats silent truncation.
|
||||
const SUFFIX_PROBE_CAP: i32 = 10_000;
|
||||
let mut chosen: Option<String> = None;
|
||||
for n in 2..=SUFFIX_PROBE_CAP {
|
||||
let candidate = format!("{canonical}-{n}");
|
||||
match self.user_storage.get_user_by_username(&candidate).await {
|
||||
Ok(_) => continue,
|
||||
Err(_) => {
|
||||
chosen = Some(candidate);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
let suffixed = chosen.ok_or_else(|| {
|
||||
DomainError::internal_error(
|
||||
"User",
|
||||
format!(
|
||||
"reactivation-collision suffix probe exhausted \
|
||||
{SUFFIX_PROBE_CAP} candidates for base '{canonical}'"
|
||||
),
|
||||
)
|
||||
})?;
|
||||
tracing::info!(
|
||||
target: "audit",
|
||||
event = "user.reactivation_renamed",
|
||||
reason = "collision_with_active",
|
||||
target_id = %user_id,
|
||||
from = %current,
|
||||
to = %suffixed,
|
||||
"🔄 user reactivation renamed to avoid username collision",
|
||||
);
|
||||
suffixed
|
||||
}
|
||||
};
|
||||
|
||||
if target.username() != Some(chosen_name.as_str()) {
|
||||
let mut renamed = target;
|
||||
renamed
|
||||
.set_username(chosen_name)
|
||||
.map_err(|e| DomainError::internal_error("User", format!("set_username: {e}")))?;
|
||||
self.user_storage.update_user(renamed).await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Change user role (admin only).
|
||||
///
|
||||
/// Refuses `role = "admin"` when the target is external (grant-only).
|
||||
@@ -4435,10 +4550,14 @@ impl AuthApplicationService {
|
||||
.clone()
|
||||
.or(claims.name.clone())
|
||||
.unwrap_or_else(|| format!("oidc_{}", &claims.sub[..8.min(claims.sub.len())]));
|
||||
// Placeholder-email fallback when the IdP omits `email` from
|
||||
// the claim set. Lowercase the local-part so the fake address
|
||||
// matches the storage convention for other placeholder-email
|
||||
// paths (see `admin_create_user`'s `<username>@oxicloud.local`).
|
||||
let oidc_email = claims
|
||||
.email
|
||||
.clone()
|
||||
.unwrap_or_else(|| format!("{}@oidc.local", oidc_username));
|
||||
.unwrap_or_else(|| format!("{}@oidc.local", oidc_username.to_ascii_lowercase()));
|
||||
|
||||
// 5. Look up existing user by OIDC subject.
|
||||
//
|
||||
@@ -4652,16 +4771,30 @@ impl AuthApplicationService {
|
||||
&oidc_username
|
||||
};
|
||||
|
||||
// Filter to valid username characters only, then truncate to 32 chars
|
||||
// Lowercase at JIT derivation. `validate_username` in
|
||||
// `User::new` would lowercase too, but the collision
|
||||
// check below (`get_user_by_username`) needs the
|
||||
// canonical form BEFORE `User::new` is called —
|
||||
// otherwise `Alice` from an IdP claim would look
|
||||
// "free" against an `alice` row on the first pass
|
||||
// and fail the DB unique constraint at INSERT time.
|
||||
// See `docs/plan/username-lowercase.md § 2. OIDC JIT
|
||||
// derivation`.
|
||||
//
|
||||
// ASCII-only by the char-filter below, so
|
||||
// `to_ascii_lowercase()` is deterministic and
|
||||
// locale-safe.
|
||||
let mut username = base_username
|
||||
.chars()
|
||||
.filter(|c| {
|
||||
c.is_ascii_alphanumeric() || *c == '-' || *c == '_' || *c == '.'
|
||||
})
|
||||
.take(32)
|
||||
.collect::<String>();
|
||||
.collect::<String>()
|
||||
.to_ascii_lowercase();
|
||||
|
||||
// Filter helper: removes any chars that are not valid in a username
|
||||
// Filter helper: removes any chars that are not valid in a username.
|
||||
// Lowercases too so the collision-suffix path below writes canonical form.
|
||||
let filter_username_chars = |s: &str| {
|
||||
s.chars()
|
||||
.filter(|c| {
|
||||
@@ -4669,6 +4802,7 @@ impl AuthApplicationService {
|
||||
})
|
||||
.take(32)
|
||||
.collect::<String>()
|
||||
.to_ascii_lowercase()
|
||||
};
|
||||
|
||||
// Ensure minimum length (the padding suffix must also be filtered)
|
||||
|
||||
@@ -125,10 +125,16 @@ impl StorageUsageService {
|
||||
}
|
||||
|
||||
/// Same as [`Self::update_user_storage_usage`], keyed by username.
|
||||
///
|
||||
/// Lowercases input before bind — same rule as
|
||||
/// `UserRepository::get_user_by_username`. Usernames are canonical
|
||||
/// (lowercase) in the DB post-migration; callers may pass any case.
|
||||
/// See `docs/plan/username-lowercase.md`.
|
||||
pub async fn update_user_storage_usage_by_username(
|
||||
&self,
|
||||
username: &str,
|
||||
) -> Result<i64, DomainError> {
|
||||
let username = username.trim().to_ascii_lowercase();
|
||||
let total_usage: Option<i64> = sqlx::query_scalar(
|
||||
r#"
|
||||
UPDATE auth.users u
|
||||
@@ -146,7 +152,7 @@ impl StorageUsageService {
|
||||
RETURNING u.storage_used_bytes
|
||||
"#,
|
||||
)
|
||||
.bind(username)
|
||||
.bind(&username)
|
||||
.fetch_optional(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
|
||||
@@ -82,11 +82,35 @@ pub enum Action {
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
},
|
||||
|
||||
/// Lowercase every active user's username in `auth.users`.
|
||||
///
|
||||
/// Enforcement-companion for the case-insensitive-usernames
|
||||
/// migration (see `docs/plan/username-lowercase.md`). The server
|
||||
/// refuses to boot after upgrade until this has run. Data touched
|
||||
/// is `auth.users.username` only.
|
||||
///
|
||||
/// Collision handling: when `Alice` and `alice` both exist,
|
||||
/// tiebreak `(last_login_at DESC NULLS LAST, created_at ASC)` —
|
||||
/// the winner keeps the canonical lowercased name, losers get
|
||||
/// `<lowercase>-2`, `-3`, … via the shared
|
||||
/// [`common::username_migration::find_free_username_suffix`]
|
||||
/// probe. Sessions and grants survive the rename (they key on
|
||||
/// `user_id`).
|
||||
///
|
||||
/// Skipped: soft-deleted / disabled rows and rows where
|
||||
/// `username IS NULL` (OPAQUE-migrated accounts).
|
||||
LowercaseUsernames {
|
||||
/// Print what would change without touching the DB.
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
},
|
||||
}
|
||||
|
||||
pub async fn run(action: Action) -> u8 {
|
||||
match action {
|
||||
Action::NfcFilenames { dry_run } => run_nfc_filenames(dry_run).await,
|
||||
Action::LowercaseUsernames { dry_run } => run_lowercase_usernames(dry_run).await,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -758,3 +782,293 @@ async fn run_folders(pool: &PgPool, dry_run: bool, stats: &mut Stats) -> Result<
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ════════════════════════════════════════════════════════════════════════════
|
||||
// lowercase-usernames
|
||||
// ════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
#[derive(Default)]
|
||||
struct UsernameStats {
|
||||
scanned: u64,
|
||||
already_lowercase: u64,
|
||||
normalized_in_place: u64,
|
||||
/// Multi-member `LOWER(username)` group where the tiebreak
|
||||
/// winner kept the canonical name.
|
||||
collision_winners: u64,
|
||||
/// Multi-member losers renamed to `<lowercase>-N`.
|
||||
renamed_to_suffix: u64,
|
||||
/// Rows the scan touched but the loop declined to modify. Today
|
||||
/// this is inactive rows (soft-deleted / admin-disabled). NULL
|
||||
/// usernames never enter the scan so they don't contribute here.
|
||||
skipped: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct UsernameRow {
|
||||
id: Uuid,
|
||||
username: String,
|
||||
/// Inactive rows (soft-deleted / admin-disabled) are read but not
|
||||
/// modified — normalising a name we can't reach anyway risks
|
||||
/// creating a `<lower>-N` conflict with a future re-activation of
|
||||
/// the same handle. The loop uses this flag to skip and count.
|
||||
active: bool,
|
||||
// Kept for the SQL row-shape roundtrip (the SELECT ordering
|
||||
// depends on them) even though the Rust-side grouping only
|
||||
// reads `id` and `username`. Marked `#[allow(dead_code)]`
|
||||
// so clippy doesn't nag; renaming to `_last_login_at` would
|
||||
// work too but the SQL column names are load-bearing for the
|
||||
// sqlx `Row::get` calls below.
|
||||
#[allow(dead_code)]
|
||||
last_login_at: Option<DateTime<Utc>>,
|
||||
#[allow(dead_code)]
|
||||
created_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
async fn run_lowercase_usernames(dry_run: bool) -> u8 {
|
||||
let database_url = match env::var("DATABASE_URL") {
|
||||
Ok(v) => v,
|
||||
Err(_) => {
|
||||
eprintln!("migrate lowercase-usernames: DATABASE_URL not set");
|
||||
return 2;
|
||||
}
|
||||
};
|
||||
|
||||
let pool = match PgPool::connect(&database_url).await {
|
||||
Ok(p) => p,
|
||||
Err(e) => {
|
||||
eprintln!("migrate lowercase-usernames: failed to connect: {e}");
|
||||
return 2;
|
||||
}
|
||||
};
|
||||
|
||||
if dry_run {
|
||||
println!("migrate lowercase-usernames: DRY RUN (no writes)");
|
||||
} else {
|
||||
println!("migrate lowercase-usernames: applying changes");
|
||||
}
|
||||
|
||||
// Load every user with a non-NULL username, active and inactive
|
||||
// alike. Inactive rows are surfaced (not filtered at scan-time)
|
||||
// so the `skipped (inactive)` counter can report them honestly —
|
||||
// an operator reading the summary sees "10 rows scanned, 2
|
||||
// inactive were passed on" instead of a phantom 0.
|
||||
//
|
||||
// NULL usernames stay out of the scan: there's nothing to
|
||||
// normalise for OPAQUE-migrated rows, and pulling them would
|
||||
// inflate `scanned` with rows the migration has no verb for.
|
||||
//
|
||||
// Ordering: alphabetic by `LOWER(username)` groups collisions
|
||||
// together, then the intra-group order is the tiebreak
|
||||
// (`last_login_at DESC NULLS LAST, created_at ASC` — most
|
||||
// recently active wins the canonical name).
|
||||
let rows: Vec<UsernameRow> = match sqlx::query(
|
||||
r#"
|
||||
SELECT id, username, active, last_login_at, created_at
|
||||
FROM auth.users
|
||||
WHERE username IS NOT NULL
|
||||
ORDER BY LOWER(username),
|
||||
(last_login_at IS NULL),
|
||||
last_login_at DESC NULLS LAST,
|
||||
created_at ASC
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
{
|
||||
Ok(rs) => rs
|
||||
.into_iter()
|
||||
.map(|r| UsernameRow {
|
||||
id: r.get::<Uuid, _>("id"),
|
||||
username: r.get::<String, _>("username"),
|
||||
active: r.get::<bool, _>("active"),
|
||||
last_login_at: r.try_get::<DateTime<Utc>, _>("last_login_at").ok(),
|
||||
created_at: r.get::<DateTime<Utc>, _>("created_at"),
|
||||
})
|
||||
.collect(),
|
||||
Err(e) => {
|
||||
eprintln!("migrate lowercase-usernames: initial scan failed: {e}");
|
||||
return 2;
|
||||
}
|
||||
};
|
||||
|
||||
let mut stats = UsernameStats::default();
|
||||
|
||||
// Group by `LOWER(username)`. Order preserved from the SQL query
|
||||
// → within a group, the FIRST row is the tiebreak winner.
|
||||
//
|
||||
// Inactive rows are filtered OUT of the grouping (not just
|
||||
// skipped inside the loop) so they can't create a phantom
|
||||
// collision with an active row sharing their lowercase form.
|
||||
// Example: inactive `Alice` + active `alice` would otherwise
|
||||
// look like a two-member group; filtering inactive first leaves
|
||||
// `alice` as a clean singleton no-op. The count goes to
|
||||
// `stats.skipped`, surfaced in the summary as `skipped (inactive)`.
|
||||
let mut groups: Vec<(String, Vec<UsernameRow>)> = Vec::new();
|
||||
for row in rows {
|
||||
stats.scanned += 1;
|
||||
if !row.active {
|
||||
stats.skipped += 1;
|
||||
continue;
|
||||
}
|
||||
let key = row.username.to_ascii_lowercase();
|
||||
match groups.last_mut() {
|
||||
Some((k, v)) if k == &key => v.push(row),
|
||||
_ => groups.push((key, vec![row])),
|
||||
}
|
||||
}
|
||||
|
||||
for (lower, members) in groups {
|
||||
if members.len() == 1 {
|
||||
let row = &members[0];
|
||||
if row.username == lower {
|
||||
stats.already_lowercase += 1;
|
||||
continue;
|
||||
}
|
||||
// Single-member group with a mixed-case name → straight
|
||||
// rename to the lowercase form. No collision.
|
||||
if !dry_run && let Err(e) = update_username(&pool, row.id, &lower).await {
|
||||
eprintln!(
|
||||
"migrate lowercase-usernames: UPDATE failed for {}: {e}",
|
||||
row.id
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
println!(
|
||||
"NORMALIZE user={} '{}' → '{}'",
|
||||
row.id, row.username, lower
|
||||
);
|
||||
stats.normalized_in_place += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Multi-member group → collision. Members are already ordered
|
||||
// by the tiebreak. Winner takes the canonical lowercase name,
|
||||
// losers get `<lower>-2`, `-3`, … via the shared suffix helper.
|
||||
//
|
||||
// ORDER MATTERS: losers must be renamed FIRST. If we renamed
|
||||
// the winner to `<lower>` while a loser still holds that
|
||||
// exact name (the common case where the winner is mixed-case
|
||||
// and the loser is already-lowercase), the UNIQUE constraint
|
||||
// `users_username_key` fires. Freeing the canonical form by
|
||||
// suffixing every non-winner member first eliminates the
|
||||
// race entirely.
|
||||
let (winner, losers) = members.split_first().expect("non-empty by construction");
|
||||
|
||||
// Suffixes assigned inside this group during this run. Used
|
||||
// to keep dry-run consistent (no DB writes → the shared
|
||||
// suffix helper would hand the same probe back for every
|
||||
// loser). During apply, the DB itself deduplicates, but
|
||||
// tracking here keeps the two modes structurally identical.
|
||||
let mut reserved_this_group: Vec<String> = Vec::new();
|
||||
for loser in losers {
|
||||
let suffixed =
|
||||
match pick_free_suffix_avoiding(&pool, &lower, &reserved_this_group).await {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
eprintln!(
|
||||
"migrate lowercase-usernames: suffix probe failed for {}: {e}",
|
||||
loser.id
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
};
|
||||
if !dry_run && let Err(e) = update_username(&pool, loser.id, &suffixed).await {
|
||||
eprintln!(
|
||||
"migrate lowercase-usernames: loser UPDATE failed for {}: {e}",
|
||||
loser.id
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
println!(
|
||||
"RENAME user={} '{}' → '{}' (collision suffix)",
|
||||
loser.id, loser.username, suffixed
|
||||
);
|
||||
stats.renamed_to_suffix += 1;
|
||||
reserved_this_group.push(suffixed);
|
||||
}
|
||||
|
||||
if winner.username == lower {
|
||||
// Winner already holds the canonical name (a lowercase
|
||||
// row happened to be the most recently active; other
|
||||
// members are the ones needing renames).
|
||||
stats.already_lowercase += 1;
|
||||
} else {
|
||||
if !dry_run && let Err(e) = update_username(&pool, winner.id, &lower).await {
|
||||
eprintln!(
|
||||
"migrate lowercase-usernames: winner UPDATE failed for {}: {e}",
|
||||
winner.id
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
println!(
|
||||
"NORMALIZE user={} '{}' → '{}' (collision winner)",
|
||||
winner.id, winner.username, lower
|
||||
);
|
||||
stats.collision_winners += 1;
|
||||
}
|
||||
}
|
||||
|
||||
println!();
|
||||
println!("Summary:");
|
||||
println!(" scanned: {}", stats.scanned);
|
||||
println!(" already-lowercase: {}", stats.already_lowercase);
|
||||
println!(" normalized in place: {}", stats.normalized_in_place);
|
||||
println!(" collision winners: {}", stats.collision_winners);
|
||||
println!(" renamed to suffix: {}", stats.renamed_to_suffix);
|
||||
println!(" skipped (inactive): {}", stats.skipped);
|
||||
if dry_run
|
||||
&& (stats.normalized_in_place + stats.collision_winners + stats.renamed_to_suffix) > 0
|
||||
{
|
||||
println!();
|
||||
println!("(dry-run — re-run without --dry-run to apply)");
|
||||
}
|
||||
0
|
||||
}
|
||||
|
||||
async fn update_username(pool: &PgPool, id: Uuid, new_name: &str) -> Result<(), sqlx::Error> {
|
||||
sqlx::query("UPDATE auth.users SET username = $1, updated_at = NOW() WHERE id = $2")
|
||||
.bind(new_name)
|
||||
.bind(id)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Pick the next free `<base>-<N>` suffix, skipping any suffix already
|
||||
/// reserved earlier in this migration run.
|
||||
///
|
||||
/// Wraps [`crate::common::username_migration::find_free_username_suffix`]
|
||||
/// with an additional local guard: in dry-run mode, no UPDATEs land so
|
||||
/// the DB probe would return the same suffix for every loser in a
|
||||
/// multi-loser group. The `reserved` slice lets the caller feed back
|
||||
/// the suffixes it has already announced, and the probe steps past
|
||||
/// them. In apply mode the DB probe alone would be enough (each real
|
||||
/// UPDATE moves the state forward), but the same code path keeps the
|
||||
/// two modes structurally identical.
|
||||
async fn pick_free_suffix_avoiding(
|
||||
pool: &PgPool,
|
||||
base: &str,
|
||||
reserved: &[String],
|
||||
) -> Result<String, sqlx::Error> {
|
||||
// Try the shared helper's default candidate first; if it collides
|
||||
// with a same-run reservation, increment past it and probe again.
|
||||
let mut n = 2;
|
||||
loop {
|
||||
let candidate = format!("{base}-{n}");
|
||||
let db_taken: (bool,) =
|
||||
sqlx::query_as("SELECT EXISTS(SELECT 1 FROM auth.users WHERE username = $1)")
|
||||
.bind(&candidate)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
let locally_taken = reserved.iter().any(|s| s == &candidate);
|
||||
if !db_taken.0 && !locally_taken {
|
||||
return Ok(candidate);
|
||||
}
|
||||
n += 1;
|
||||
if n > 10_000 {
|
||||
// Same cap as the shared helper — a loud panic beats a
|
||||
// silent truncation for an anomaly this rare.
|
||||
panic!("pick_free_suffix_avoiding: exhausted 10000 suffix probes for base '{base}'",);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+17
-4
@@ -128,8 +128,15 @@ async fn run_reset(user: Option<String>, all: bool, dry_run: bool) -> u8 {
|
||||
let rows_result = if all {
|
||||
sqlx::query(select_sql).fetch_all(&pool).await
|
||||
} else {
|
||||
let ident = user.as_deref().unwrap();
|
||||
sqlx::query(select_sql).bind(ident).fetch_all(&pool).await
|
||||
// Normalise before bind so the CLI accepts any case for
|
||||
// the username branch (email is already case-insensitive
|
||||
// via a functional index on LOWER(email); lowercasing here
|
||||
// for both branches is harmless — emails are lowercase
|
||||
// ASCII in `auth.users.email` too).
|
||||
// See `docs/plan/username-lowercase.md § 3. Lookup normalization`.
|
||||
let ident_raw = user.as_deref().unwrap();
|
||||
let ident = ident_raw.trim().to_ascii_lowercase();
|
||||
sqlx::query(select_sql).bind(&ident).fetch_all(&pool).await
|
||||
};
|
||||
let rows = match rows_result {
|
||||
Ok(r) => r,
|
||||
@@ -204,8 +211,14 @@ async fn run_reset(user: Option<String>, all: bool, dry_run: bool) -> u8 {
|
||||
let write_result = if all {
|
||||
sqlx::query(update_sql_all).execute(&pool).await
|
||||
} else {
|
||||
let ident = user.as_deref().unwrap();
|
||||
sqlx::query(update_sql_one).bind(ident).execute(&pool).await
|
||||
// Same normalisation as the read path above — usernames are
|
||||
// canonical lowercase in the DB. See docs/plan/username-lowercase.md.
|
||||
let ident_raw = user.as_deref().unwrap();
|
||||
let ident = ident_raw.trim().to_ascii_lowercase();
|
||||
sqlx::query(update_sql_one)
|
||||
.bind(&ident)
|
||||
.execute(&pool)
|
||||
.await
|
||||
};
|
||||
let affected = match write_result {
|
||||
Ok(r) => r.rows_affected(),
|
||||
|
||||
@@ -8,3 +8,4 @@ pub mod mime_detect;
|
||||
pub mod runtime;
|
||||
pub mod stubs;
|
||||
pub mod text;
|
||||
pub mod username_migration;
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
//! Username-lowercase boot flow: verifier, auto-rename, shared collision helper.
|
||||
//!
|
||||
//! The plan (`docs/plan/username-lowercase.md`) makes usernames
|
||||
//! case-insensitive by canonicalising to lowercase on ingest. Three
|
||||
//! pieces of infrastructure live here:
|
||||
//!
|
||||
//! 1. [`verify_all_usernames_lowercase`] — a **read-only** check that
|
||||
//! runs after `sqlx::migrate!()` at boot. Classifies every active
|
||||
//! mixed-case row into one of three outcomes:
|
||||
//!
|
||||
//! - [`UsernameCaseCheck::Clean`] — nothing to do.
|
||||
//! - [`UsernameCaseCheck::AutoRenamable`] — mixed-case rows exist
|
||||
//! but each `LOWER(username)` form is unique in the active-user
|
||||
//! set. Safe to lowercase in one atomic transaction; boot proceeds.
|
||||
//! - [`UsernameCaseCheck::Collisions`] — at least one group has
|
||||
//! two or more active rows sharing a `LOWER(username)` (e.g.
|
||||
//! `Alice` + `alice`). Tiebreak requires human judgement; the
|
||||
//! server refuses to start and prints the CLI command.
|
||||
//!
|
||||
//! Follows [[feedback_no_silent_auto_repair]] in spirit: silent
|
||||
//! action is limited to cases where there is exactly one correct
|
||||
//! move (rename the sole mixed-case row to its lowercase form).
|
||||
//! Anywhere ambiguity exists (which of `Alice` and `alice` keeps
|
||||
//! the canonical name?), boot refuses and defers to `oxicloud
|
||||
//! migrate lowercase-usernames`.
|
||||
//!
|
||||
//! 2. [`apply_auto_renames`] — the one-transaction UPDATE loop that
|
||||
//! performs the auto-rename path. Emits a structured audit line
|
||||
//! per row (`user.username_lowercased_on_boot`). All-or-nothing:
|
||||
//! a mid-tx failure aborts the transaction and boot fails, so the
|
||||
//! DB is never left in a half-renamed state.
|
||||
//!
|
||||
//! 3. [`find_free_username_suffix`] — the shared collision-resolution
|
||||
//! helper. Called by the migration CLI when it lowercases a name
|
||||
//! that would clash with an existing row, AND by the un-soft-delete
|
||||
//! API when it re-normalises a mixed-case account whose lowercase
|
||||
//! form is now taken by someone else.
|
||||
//!
|
||||
//! `NULL` usernames (OPAQUE-migrated accounts) are always skipped — the
|
||||
//! SQL `WHERE username <> LOWER(username)` predicate is NULL-safe by
|
||||
//! semantics (`NULL <> anything` yields `NULL`, which `WHERE` excludes).
|
||||
//! Soft-deleted / disabled accounts (`active = false`) are also skipped:
|
||||
//! they can't serve traffic anyway.
|
||||
|
||||
use sqlx::{PgPool, Row};
|
||||
|
||||
/// One mixed-case account row. Used both for the auto-rename list and
|
||||
/// for reporting collision-group members.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct MixedCaseAccount {
|
||||
pub id: uuid::Uuid,
|
||||
pub username: String,
|
||||
pub last_login_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
}
|
||||
|
||||
/// A `LOWER(username)` group with two or more active members. At least
|
||||
/// one member is mixed-case (that's what made the group visible to the
|
||||
/// verifier); the other member(s) may be already-lowercase (e.g.
|
||||
/// `Alice` + `alice`) or also mixed-case (`Alice` + `ALICE`).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CollisionGroup {
|
||||
/// The lowercase form shared by every member.
|
||||
pub canonical: String,
|
||||
/// Members, ordered by the tiebreak that the migration CLI
|
||||
/// applies: `last_login_at DESC NULLS LAST, created_at ASC`.
|
||||
pub members: Vec<MixedCaseAccount>,
|
||||
}
|
||||
|
||||
/// The three outcomes of the boot-time verifier.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum UsernameCaseCheck {
|
||||
/// Every active username is already lowercase (or `NULL`). Boot
|
||||
/// proceeds unmodified.
|
||||
Clean,
|
||||
/// Mixed-case rows exist, but each `LOWER(username)` form is
|
||||
/// unique among active users. Safe to lowercase atomically at
|
||||
/// boot; the caller runs [`apply_auto_renames`].
|
||||
AutoRenamable(Vec<MixedCaseAccount>),
|
||||
/// At least one `LOWER(username)` group has two or more active
|
||||
/// members. Tiebreak requires human judgement; the caller formats
|
||||
/// a refusal message via [`format_refusal_message_collisions`] and
|
||||
/// aborts boot.
|
||||
Collisions(Vec<CollisionGroup>),
|
||||
}
|
||||
|
||||
/// Boot-time verifier. Runs AFTER `sqlx::migrate!()` and BEFORE
|
||||
/// `AppState` is assembled. Read-only: never mutates `auth.users`.
|
||||
///
|
||||
/// Returns [`UsernameCaseCheck`] describing what (if anything) the
|
||||
/// caller should do. Errors are limited to DB failures — semantic
|
||||
/// outcomes are all `Ok(_)` variants.
|
||||
pub async fn verify_all_usernames_lowercase(pool: &PgPool) -> Result<UsernameCaseCheck, String> {
|
||||
// First pass: mixed-case rows that have NO other active row
|
||||
// sharing their LOWER form. These are safe to auto-rename.
|
||||
let auto_rows = sqlx::query(
|
||||
r#"
|
||||
SELECT u.id, u.username, u.last_login_at
|
||||
FROM auth.users u
|
||||
WHERE u.active = true
|
||||
AND u.username <> LOWER(u.username)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM auth.users u2
|
||||
WHERE u2.active = true
|
||||
AND u2.id <> u.id
|
||||
AND LOWER(u2.username) = LOWER(u.username)
|
||||
)
|
||||
ORDER BY LOWER(u.username)
|
||||
"#,
|
||||
)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.map_err(|e| format!("username lowercase verifier: singleton query failed: {e}"))?;
|
||||
|
||||
// Second pass: every active row that belongs to a colliding
|
||||
// group — a `LOWER(username)` shared by two or more active rows
|
||||
// where at least one member is mixed-case. Result includes
|
||||
// already-lowercase members so the refusal report shows the full
|
||||
// context of each collision.
|
||||
let collision_rows = sqlx::query(
|
||||
r#"
|
||||
WITH colliding_lowers AS (
|
||||
SELECT LOWER(username) AS canonical
|
||||
FROM auth.users
|
||||
WHERE active = true
|
||||
GROUP BY LOWER(username)
|
||||
HAVING COUNT(*) > 1
|
||||
AND SUM(CASE WHEN username <> LOWER(username) THEN 1 ELSE 0 END) >= 1
|
||||
)
|
||||
SELECT id, username, last_login_at, LOWER(username) AS canonical
|
||||
FROM auth.users
|
||||
WHERE active = true
|
||||
AND LOWER(username) IN (SELECT canonical FROM colliding_lowers)
|
||||
ORDER BY LOWER(username),
|
||||
(last_login_at IS NULL),
|
||||
last_login_at DESC NULLS LAST,
|
||||
created_at ASC
|
||||
"#,
|
||||
)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.map_err(|e| format!("username lowercase verifier: collision query failed: {e}"))?;
|
||||
|
||||
if !collision_rows.is_empty() {
|
||||
// Group by canonical. Rows are already ordered by canonical
|
||||
// then by tiebreak, so a fold is enough.
|
||||
let mut groups: Vec<CollisionGroup> = Vec::new();
|
||||
for r in collision_rows {
|
||||
let canonical: String = r.get("canonical");
|
||||
let member = MixedCaseAccount {
|
||||
id: r.get::<uuid::Uuid, _>("id"),
|
||||
username: r.get::<String, _>("username"),
|
||||
last_login_at: r
|
||||
.try_get::<chrono::DateTime<chrono::Utc>, _>("last_login_at")
|
||||
.ok(),
|
||||
};
|
||||
match groups.last_mut() {
|
||||
Some(g) if g.canonical == canonical => g.members.push(member),
|
||||
_ => groups.push(CollisionGroup {
|
||||
canonical,
|
||||
members: vec![member],
|
||||
}),
|
||||
}
|
||||
}
|
||||
return Ok(UsernameCaseCheck::Collisions(groups));
|
||||
}
|
||||
|
||||
if auto_rows.is_empty() {
|
||||
return Ok(UsernameCaseCheck::Clean);
|
||||
}
|
||||
|
||||
let accounts = auto_rows
|
||||
.into_iter()
|
||||
.map(|r| MixedCaseAccount {
|
||||
id: r.get::<uuid::Uuid, _>("id"),
|
||||
username: r.get::<String, _>("username"),
|
||||
last_login_at: r
|
||||
.try_get::<chrono::DateTime<chrono::Utc>, _>("last_login_at")
|
||||
.ok(),
|
||||
})
|
||||
.collect();
|
||||
Ok(UsernameCaseCheck::AutoRenamable(accounts))
|
||||
}
|
||||
|
||||
/// Apply the atomic auto-rename transaction. All UPDATEs succeed
|
||||
/// together or all roll back — the DB is never left in a half-renamed
|
||||
/// state. Each successful rename emits a structured audit line.
|
||||
///
|
||||
/// The `WHERE id = $1 AND username = $3` guard defends against a
|
||||
/// concurrent rename between the SELECT and this UPDATE. If some
|
||||
/// other process renamed the row in that window, the UPDATE affects
|
||||
/// zero rows and we log a warning but do not fail the transaction —
|
||||
/// the row is already lowercase (that's why the guard didn't match),
|
||||
/// so the invariant still holds.
|
||||
pub async fn apply_auto_renames(
|
||||
pool: &PgPool,
|
||||
accounts: &[MixedCaseAccount],
|
||||
) -> Result<(), sqlx::Error> {
|
||||
let mut tx = pool.begin().await?;
|
||||
for acc in accounts {
|
||||
let new_username = acc.username.to_ascii_lowercase();
|
||||
let res = sqlx::query(
|
||||
r#"
|
||||
UPDATE auth.users
|
||||
SET username = $2
|
||||
WHERE id = $1
|
||||
AND username = $3
|
||||
"#,
|
||||
)
|
||||
.bind(acc.id)
|
||||
.bind(&new_username)
|
||||
.bind(&acc.username)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
if res.rows_affected() == 0 {
|
||||
tracing::warn!(
|
||||
target: "audit",
|
||||
event = "user.username_lowercase_skipped_on_boot",
|
||||
reason = "row_changed_between_verify_and_apply",
|
||||
user_id = %acc.id,
|
||||
expected_username = %acc.username,
|
||||
"👮🏻♂️ skipped auto-lowercase: row was modified after verifier ran",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
target: "audit",
|
||||
event = "user.username_lowercased_on_boot",
|
||||
reason = "unique_lowercase_group",
|
||||
user_id = %acc.id,
|
||||
old_username = %acc.username,
|
||||
new_username = %new_username,
|
||||
"👮🏻♂️ auto-lowercased username at boot",
|
||||
);
|
||||
}
|
||||
tx.commit().await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Format the FATAL error string shown when boot refuses to proceed
|
||||
/// because at least one `LOWER(username)` group has multiple active
|
||||
/// members. Self-sufficient — an operator at 3 AM shouldn't need to
|
||||
/// consult docs to know what to do.
|
||||
pub fn format_refusal_message_collisions(groups: &[CollisionGroup]) -> String {
|
||||
use std::fmt::Write;
|
||||
let total_members: usize = groups.iter().map(|g| g.members.len()).sum();
|
||||
let mut out = String::new();
|
||||
let _ = write!(
|
||||
&mut out,
|
||||
"\nFATAL: cannot start — {} colliding username group(s) \
|
||||
({} affected account(s) in total).\n\n\
|
||||
Non-colliding mixed-case rows are auto-renamed at boot. \
|
||||
These groups can't be resolved automatically because two or \
|
||||
more active accounts share the same lowercase form, and only \
|
||||
a human can decide who keeps the canonical name.\n\n\
|
||||
Run the migration:\n\n \
|
||||
oxicloud migrate lowercase-usernames --dry-run # preview the tiebreak\n \
|
||||
oxicloud migrate lowercase-usernames # apply\n\n\
|
||||
The tiebreak rule is `last_login_at DESC NULLS LAST, \
|
||||
created_at ASC` — the most recently active member keeps the \
|
||||
canonical lowercase name; the losers get `-2`, `-3`, … as a \
|
||||
suffix. Sessions and grants survive the rename (they key on \
|
||||
user_id, not username).\n\n\
|
||||
Collision groups (up to 10 shown):\n",
|
||||
groups.len(),
|
||||
total_members
|
||||
);
|
||||
for g in groups.iter().take(10) {
|
||||
let _ = writeln!(&mut out, "\n Canonical form: {}", g.canonical);
|
||||
for m in &g.members {
|
||||
let last = m
|
||||
.last_login_at
|
||||
.map(|t| t.format("%Y-%m-%d").to_string())
|
||||
.unwrap_or_else(|| "never".to_string());
|
||||
let _ = writeln!(
|
||||
&mut out,
|
||||
" {} (id: {} last_login: {})",
|
||||
m.username, m.id, last
|
||||
);
|
||||
}
|
||||
}
|
||||
if groups.len() > 10 {
|
||||
let _ = writeln!(
|
||||
&mut out,
|
||||
"\n ... and {} more group(s). Run --dry-run for the full list.",
|
||||
groups.len() - 10
|
||||
);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Cap on the suffix-probe loop. If we ever need `<base>-10000` there's
|
||||
/// something very wrong with the account universe — collisions in the
|
||||
/// wild are 2-3 accounts, not 10 K. The loud abort IS the detection.
|
||||
/// See [`docs/plan/username-lowercase.md § 3. Suffix-collision robustness`].
|
||||
const SUFFIX_PROBE_CAP: i32 = 10_000;
|
||||
|
||||
/// Find the next free `<base>-<N>` suffix for a colliding username.
|
||||
///
|
||||
/// Starts at `<base>-2` and increments until an unused suffix is
|
||||
/// found. Robust against pre-existing rows already occupying some
|
||||
/// suffixes (the probe steps past them).
|
||||
///
|
||||
/// Called by:
|
||||
/// - The migration CLI when a `LOWER(username)` group has multiple
|
||||
/// members and the tiebreak winner keeps the canonical name; the
|
||||
/// losers get `<base>-2`, `-3`, … from this helper.
|
||||
/// - The un-soft-delete API when re-normalising a mixed-case
|
||||
/// account whose lowercase form is now taken by an active row.
|
||||
///
|
||||
/// Both callers reach for this single function so the two paths
|
||||
/// agree by construction — no drift risk between the migration and
|
||||
/// runtime un-soft-delete.
|
||||
pub async fn find_free_username_suffix(pool: &PgPool, base: &str) -> Result<String, sqlx::Error> {
|
||||
for n in 2..=SUFFIX_PROBE_CAP {
|
||||
let candidate = format!("{base}-{n}");
|
||||
let exists: (bool,) =
|
||||
sqlx::query_as("SELECT EXISTS(SELECT 1 FROM auth.users WHERE username = $1)")
|
||||
.bind(&candidate)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
if !exists.0 {
|
||||
return Ok(candidate);
|
||||
}
|
||||
}
|
||||
// If we get here, something is very wrong. Loud panic beats
|
||||
// silent truncation to whatever the caller's fallback is.
|
||||
panic!(
|
||||
"find_free_username_suffix: exhausted {SUFFIX_PROBE_CAP} suffix probes for base '{base}'; \
|
||||
the account universe likely has an anomaly worth investigating"
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn acc(name: &str) -> MixedCaseAccount {
|
||||
MixedCaseAccount {
|
||||
id: uuid::Uuid::nil(),
|
||||
username: name.into(),
|
||||
last_login_at: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn refusal_message_lists_groups_and_cli() {
|
||||
let groups = vec![CollisionGroup {
|
||||
canonical: "alice".into(),
|
||||
members: vec![acc("Alice"), acc("alice")],
|
||||
}];
|
||||
let msg = format_refusal_message_collisions(&groups);
|
||||
assert!(msg.contains("1 colliding username group(s)"));
|
||||
assert!(msg.contains("2 affected account(s)"));
|
||||
assert!(msg.contains("oxicloud migrate lowercase-usernames"));
|
||||
assert!(msg.contains("Canonical form: alice"));
|
||||
assert!(msg.contains("Alice"));
|
||||
assert!(msg.contains("last_login: never"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn refusal_message_caps_group_display_and_notes_overflow() {
|
||||
let groups: Vec<_> = (0..15)
|
||||
.map(|i| CollisionGroup {
|
||||
canonical: format!("user{i:02}"),
|
||||
members: vec![acc(&format!("User{i:02}")), acc(&format!("user{i:02}"))],
|
||||
})
|
||||
.collect();
|
||||
let msg = format_refusal_message_collisions(&groups);
|
||||
// First 10 groups shown by canonical name.
|
||||
assert!(msg.contains("Canonical form: user00"));
|
||||
assert!(msg.contains("Canonical form: user09"));
|
||||
// Overflow tail names how many are hidden.
|
||||
assert!(msg.contains("and 5 more group(s)"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn refusal_message_reports_total_across_all_groups() {
|
||||
// Two groups with different sizes — 2 + 3 = 5 members total.
|
||||
let groups = vec![
|
||||
CollisionGroup {
|
||||
canonical: "alice".into(),
|
||||
members: vec![acc("Alice"), acc("alice")],
|
||||
},
|
||||
CollisionGroup {
|
||||
canonical: "bob".into(),
|
||||
members: vec![acc("Bob"), acc("BOB"), acc("bob")],
|
||||
},
|
||||
];
|
||||
let msg = format_refusal_message_collisions(&groups);
|
||||
assert!(msg.contains("2 colliding username group(s)"));
|
||||
assert!(msg.contains("5 affected account(s)"));
|
||||
}
|
||||
}
|
||||
+93
-14
@@ -322,9 +322,15 @@ impl User {
|
||||
is_external: bool,
|
||||
) -> UserResult<Self> {
|
||||
Self::validate_email(&email)?;
|
||||
if let Some(ref u) = username {
|
||||
Self::validate_username(u)?;
|
||||
}
|
||||
// Shadow `username` with the canonical (trimmed, lowercased)
|
||||
// form returned by `validate_username`. Every downstream write
|
||||
// consumes the shadowed binding, so the row that lands in the
|
||||
// DB is always the normalised value. See
|
||||
// `docs/plan/username-lowercase.md`.
|
||||
let username = match username {
|
||||
Some(u) => Some(Self::validate_username(&u)?),
|
||||
None => None,
|
||||
};
|
||||
if let Some(ref h) = password_hash
|
||||
&& h.is_empty()
|
||||
{
|
||||
@@ -809,8 +815,10 @@ impl User {
|
||||
/// renamed: it was display text at creation; the folder is owned
|
||||
/// by `user_id`.
|
||||
pub fn set_username(&mut self, new_username: String) -> UserResult<()> {
|
||||
Self::validate_username(&new_username)?;
|
||||
self.username = Some(new_username);
|
||||
// Canonical form (trim + lowercase) — see
|
||||
// `validate_username`. Callers can pass any case; we store
|
||||
// the normalised value.
|
||||
self.username = Some(Self::validate_username(&new_username)?);
|
||||
self.updated_at = Utc::now();
|
||||
Ok(())
|
||||
}
|
||||
@@ -881,20 +889,41 @@ impl User {
|
||||
/// a handle that shadows another user's email). No leading/trailing
|
||||
/// dot or hyphen. The character set also prevents XSS payloads from
|
||||
/// being stored as usernames.
|
||||
fn validate_username(username: &str) -> UserResult<()> {
|
||||
let len = username.chars().count();
|
||||
/// Validate AND canonicalise a username.
|
||||
///
|
||||
/// Two normalisations run first, before every check:
|
||||
/// - `trim()` — strip whitespace clients may have added.
|
||||
/// - `to_ascii_lowercase()` — usernames are case-insensitive
|
||||
/// identifiers. Users type `Alice`, `ALICE`, `alice` on
|
||||
/// different clients; all three refer to the same account.
|
||||
/// ASCII-only by construction (charset check below), so
|
||||
/// `to_ascii_lowercase` is deterministic and locale-safe —
|
||||
/// no Unicode case-folding surprises (Turkish dotted-I,
|
||||
/// German ß, Greek final sigma, NFC vs NFD).
|
||||
///
|
||||
/// Returns the canonical form on success. Every entity write
|
||||
/// site consumes the returned string — because the signature
|
||||
/// changed from `Result<()>` to `Result<String>`, any caller
|
||||
/// that ignored the result is now a compile error. That's
|
||||
/// what forces every write path through the normaliser.
|
||||
///
|
||||
/// See `docs/plan/username-lowercase.md` for the full design.
|
||||
pub fn validate_username(username: &str) -> UserResult<String> {
|
||||
let normalized = username.trim().to_ascii_lowercase();
|
||||
|
||||
let len = normalized.chars().count();
|
||||
if !(2..=64).contains(&len) {
|
||||
return Err(UserError::InvalidUsername(
|
||||
"Username must be between 2 and 64 characters".to_string(),
|
||||
));
|
||||
}
|
||||
if username.contains('@') {
|
||||
if normalized.contains('@') {
|
||||
return Err(UserError::InvalidUsername(
|
||||
"Username must not contain '@' — use the email field for email addresses"
|
||||
.to_string(),
|
||||
));
|
||||
}
|
||||
if !username
|
||||
if !normalized
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == '.')
|
||||
{
|
||||
@@ -903,16 +932,16 @@ impl User {
|
||||
.to_string(),
|
||||
));
|
||||
}
|
||||
if username.starts_with('.')
|
||||
|| username.starts_with('-')
|
||||
|| username.ends_with('.')
|
||||
|| username.ends_with('-')
|
||||
if normalized.starts_with('.')
|
||||
|| normalized.starts_with('-')
|
||||
|| normalized.ends_with('.')
|
||||
|| normalized.ends_with('-')
|
||||
{
|
||||
return Err(UserError::InvalidUsername(
|
||||
"Username must not start or end with a dot or hyphen".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
Ok(normalized)
|
||||
}
|
||||
|
||||
/// Basic but meaningful email validation:
|
||||
@@ -1046,4 +1075,54 @@ mod tests {
|
||||
assert_eq!(u.display_full(true), "solo@x.com");
|
||||
assert_eq!(u.display_full(false), "solo@x.com");
|
||||
}
|
||||
|
||||
// ── validate_username: normalization + rules ─────────────────────────────
|
||||
//
|
||||
// Post-lowercase-migration `validate_username` returns the canonical
|
||||
// (trimmed, lowercased) form on success. Every write-site consumes
|
||||
// that returned string via the shadow in `User::new` /
|
||||
// `set_username`, so the invariant "usernames in `auth.users` are
|
||||
// always canonical" is enforced at the domain boundary.
|
||||
//
|
||||
// The rules that DON'T change (charset, length, no leading/trailing
|
||||
// dot or hyphen, no `@`) get their coverage here too so a future
|
||||
// rewrite of `validate_username` can't regress them silently.
|
||||
|
||||
#[test]
|
||||
fn validate_username_lowercases_and_trims() {
|
||||
// Uppercase in the middle → canonical form is lowercase.
|
||||
assert_eq!(User::validate_username("Alice").unwrap(), "alice");
|
||||
// All-uppercase.
|
||||
assert_eq!(User::validate_username("ALICE").unwrap(), "alice");
|
||||
// Whitespace around a mixed-case name → both stripped.
|
||||
assert_eq!(User::validate_username(" Alice ").unwrap(), "alice");
|
||||
// Already-canonical passes through unchanged.
|
||||
assert_eq!(User::validate_username("alice").unwrap(), "alice");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_username_charset_and_boundary_rules_survive_normalization() {
|
||||
// Trailing hyphen — still rejected after the case-fold.
|
||||
assert!(User::validate_username("alice-").is_err());
|
||||
// Leading dot.
|
||||
assert!(User::validate_username(".alice").is_err());
|
||||
// Whitespace INSIDE the name (not just around it) — the
|
||||
// charset check rejects space characters.
|
||||
assert!(User::validate_username("Al ice").is_err());
|
||||
// Non-ASCII letter — usernames are ASCII-only.
|
||||
assert!(User::validate_username("Álice").is_err());
|
||||
// `@` is forbidden (disjoint namespace with email lookup).
|
||||
assert!(User::validate_username("alice@example").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_username_length_bounds_apply_after_trim() {
|
||||
// Two-char minimum satisfied AFTER trim.
|
||||
assert_eq!(User::validate_username(" ab ").unwrap(), "ab");
|
||||
// Below the minimum after trim.
|
||||
assert!(User::validate_username(" a ").is_err());
|
||||
// Above the maximum after trim.
|
||||
let too_long = "a".repeat(65);
|
||||
assert!(User::validate_username(&too_long).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,6 +61,46 @@ pub async fn create_database_pools(config: &AppConfig) -> Result<DbPools> {
|
||||
}
|
||||
tracing::info!("Database migrations complete");
|
||||
|
||||
// Username-lowercase verifier — three outcomes:
|
||||
// * Clean → nothing to do.
|
||||
// * AutoRenamable → non-colliding mixed-case rows exist; lowercase
|
||||
// them in one transaction and continue. Silent
|
||||
// action is bounded to the case where there is
|
||||
// exactly one correct move ([[feedback_no_silent_auto_repair]]
|
||||
// in spirit — ambiguity → refusal, unique fix → apply).
|
||||
// Each rename emits an audit line.
|
||||
// * Collisions → two or more active rows share a LOWER(username)
|
||||
// form (e.g. `Alice` + `alice`); tiebreak needs a
|
||||
// human, refuse to boot and print the CLI command.
|
||||
// See `common::username_migration::verify_all_usernames_lowercase`.
|
||||
use crate::common::username_migration::{
|
||||
UsernameCaseCheck, apply_auto_renames, format_refusal_message_collisions,
|
||||
verify_all_usernames_lowercase,
|
||||
};
|
||||
match verify_all_usernames_lowercase(&primary).await {
|
||||
Ok(UsernameCaseCheck::Clean) => {}
|
||||
Ok(UsernameCaseCheck::AutoRenamable(accounts)) => {
|
||||
let count = accounts.len();
|
||||
if let Err(e) = apply_auto_renames(&primary, &accounts).await {
|
||||
return Err(DbError(format!(
|
||||
"username lowercase auto-rename failed at boot: {e}. \
|
||||
Run `oxicloud migrate lowercase-usernames --dry-run` to \
|
||||
inspect the current state, then apply manually."
|
||||
)));
|
||||
}
|
||||
tracing::info!(
|
||||
target: "audit",
|
||||
event = "user.usernames_lowercased_on_boot_summary",
|
||||
renamed = count,
|
||||
"auto-lowercased {count} non-colliding mixed-case username(s) at boot",
|
||||
);
|
||||
}
|
||||
Ok(UsernameCaseCheck::Collisions(groups)) => {
|
||||
return Err(DbError(format_refusal_message_collisions(&groups)));
|
||||
}
|
||||
Err(msg) => return Err(DbError(msg)),
|
||||
}
|
||||
|
||||
// --- maintenance pool ---
|
||||
let maintenance = create_pool_with_retries(
|
||||
&config.database.connection_string,
|
||||
|
||||
@@ -472,8 +472,15 @@ impl UserRepository for UserPgRepository {
|
||||
Ok((user, flags))
|
||||
}
|
||||
|
||||
/// Gets a user by username
|
||||
/// Gets a user by username.
|
||||
///
|
||||
/// Lowercases the input before binding: usernames are stored in
|
||||
/// canonical (lowercase, trimmed) form by `validate_username`
|
||||
/// (see `docs/plan/username-lowercase.md`), and callers may pass
|
||||
/// whatever case the user typed at the login form. Normalising
|
||||
/// here means every caller doesn't have to remember.
|
||||
async fn get_user_by_username(&self, username: &str) -> UserRepositoryResult<User> {
|
||||
let username = username.trim().to_ascii_lowercase();
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT
|
||||
@@ -487,7 +494,7 @@ impl UserRepository for UserPgRepository {
|
||||
WHERE username = $1
|
||||
"#,
|
||||
)
|
||||
.bind(username)
|
||||
.bind(&username)
|
||||
.fetch_one(&*self.pool)
|
||||
.await
|
||||
.map_err(Self::map_sqlx_error)?;
|
||||
|
||||
@@ -72,6 +72,86 @@ pub const CDC_AVG_CHUNK: usize = 262_144;
|
||||
/// Maximum CDC chunk size (1 MB).
|
||||
pub const CDC_MAX_CHUNK: usize = 1_048_576;
|
||||
|
||||
// ════════════════════════════════════════════════════════════════════════════
|
||||
// ref_count audit log — attribution trail for ref_count changes
|
||||
//
|
||||
// Emitted at every semantic ref-count mutation site so an unexplained drift
|
||||
// (`manifests_consistency` / `blobs_consistency` finding) can be traced back
|
||||
// to its calling function within one log query. See
|
||||
// `docs/plan/refcount-audit.md` for design rationale, retention model, and
|
||||
// upgrade path to a DB-backed table if log retention proves insufficient.
|
||||
//
|
||||
// Enable at runtime with `RUST_LOG=oxicloud::refcount=info`. Off by default;
|
||||
// info-level so a healthy prod deployment doesn't spam the log stream.
|
||||
// ════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
/// Table names the audit stream uses. Constants (not free strings) so grep
|
||||
/// across a log stream matches a fixed vocabulary and a typo in a call site
|
||||
/// fails to compile instead of silently drifting.
|
||||
mod refcount_audit_table {
|
||||
pub const CHUNK_MANIFESTS: &str = "chunk_manifests";
|
||||
pub const BLOBS: &str = "blobs";
|
||||
}
|
||||
|
||||
/// Source labels for the audit stream — one stable string per Rust function
|
||||
/// that mutates a ref_count. Kept as a module so `grep source=` on the log
|
||||
/// stream shows a fixed enumeration; a rename here is intentional, a rename
|
||||
/// at a call site alone doesn't compile.
|
||||
///
|
||||
/// See `docs/plan/refcount-audit.md § Callsites to instrument` for the full
|
||||
/// list. New callers add a new constant here; adding one string at the call
|
||||
/// site alone is discouraged (breaks the "closed vocabulary" property).
|
||||
mod refcount_audit_source {
|
||||
pub const STORE_FROM_STREAM_NEW_MANIFEST: &str = "store_from_stream.new_manifest";
|
||||
pub const BUMP_MANIFEST_IF_EXISTS: &str = "bump_manifest_if_exists";
|
||||
pub const ADD_REFERENCE_MANIFEST: &str = "add_reference.manifest";
|
||||
pub const ADD_REFERENCE_LEGACY: &str = "add_reference.legacy";
|
||||
pub const REMOVE_MANIFEST_REFERENCE_DECREMENT: &str = "remove_manifest_reference.decrement";
|
||||
pub const REMOVE_MANIFEST_REFERENCE_DELETE: &str = "remove_manifest_reference.delete";
|
||||
pub const REMOVE_LEGACY_REFERENCE: &str = "remove_legacy_reference";
|
||||
pub const STORE_ATTACHED_BLOB_SAME_CONTENT_BALANCE: &str =
|
||||
"store_attached_blob.same_content_balance";
|
||||
pub const STORE_ATTACHED_BLOB_REPLACE_RELEASE: &str = "store_attached_blob.replace_release";
|
||||
}
|
||||
|
||||
/// Outcome of [`DedupService::store_attached_blob_if_absent`]. Split
|
||||
/// so the caller (`thumb_attached_import_service`) can bump its
|
||||
/// `imported` vs `already` counters without a second query.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum AttachedBlobInsertOutcome {
|
||||
/// We won the atomic INSERT — the row now points at `hash`.
|
||||
Inserted { hash: String },
|
||||
/// A row already existed when the atomic INSERT ran (someone else
|
||||
/// won, or the migration was re-triggered). `existing_hash` is
|
||||
/// the row's current `blob_hash` as read moments after the DO
|
||||
/// NOTHING resolved — useful for the import service's
|
||||
/// verify-and-unlink readback.
|
||||
AlreadyPresent { existing_hash: String },
|
||||
}
|
||||
|
||||
/// Emit a single audit line for a ref_count change. Called AFTER the SQL
|
||||
/// UPDATE / INSERT / DELETE returns Ok, so a rolled-back transaction won't
|
||||
/// leave a phantom log line (the SQL error path returns before this call).
|
||||
///
|
||||
/// `delta` is the signed change (`+1` on increment, `-1` on decrement,
|
||||
/// `-old_count` when the row is deleted at its last reference — the
|
||||
/// convention is "resulting ref_count is 0 for reads afterward").
|
||||
///
|
||||
/// The tracing span inherits request-scope context (request_id, caller_id
|
||||
/// from auth middleware, job run id from scheduler) automatically, so
|
||||
/// no explicit correlation-id plumbing is needed here.
|
||||
#[inline]
|
||||
fn audit_ref_count(table: &'static str, hash: &str, delta: i32, source: &'static str) {
|
||||
tracing::info!(
|
||||
target: "oxicloud::refcount",
|
||||
table,
|
||||
hash = %hash,
|
||||
delta,
|
||||
source,
|
||||
"ref_count {}", if delta >= 0 { "+" } else { "-" }
|
||||
);
|
||||
}
|
||||
|
||||
// ── CDC helper types ─────────────────────────────────────────────────────────
|
||||
|
||||
/// Everything a streaming chunk ingest learned about its byte stream.
|
||||
@@ -751,24 +831,146 @@ impl DedupService {
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?;
|
||||
|
||||
// A replaced row's old blob loses its only reference from here. Not
|
||||
// releasing it would pin those bytes forever — nothing else points at
|
||||
// a superseded preview.
|
||||
if let Some((old_hash,)) = previous
|
||||
&& old_hash != attached_hash
|
||||
&& let Err(e) = self.remove_reference(&old_hash).await
|
||||
{
|
||||
tracing::warn!(
|
||||
target: "oxicloud::dedup",
|
||||
error = %e,
|
||||
"failed to release replaced attached-blob reference for {}",
|
||||
&old_hash[..old_hash.len().min(12)],
|
||||
);
|
||||
// Two shapes to balance depending on whether the UPSERT was a
|
||||
// real content replacement or a same-content re-store:
|
||||
//
|
||||
// - `previous == Some(old) && old != attached_hash` — different
|
||||
// content overwritten in the row. Release the old blob's ref
|
||||
// (its file_attached row is gone; would leak forever otherwise).
|
||||
//
|
||||
// - `previous == Some(old) && old == attached_hash` — SAME-content
|
||||
// re-store. `store_from_stream` above incremented the manifest
|
||||
// unconditionally, but the row's blob_hash didn't change so no
|
||||
// logical reference was added. Cancel the phantom increment
|
||||
// here, or it accumulates one +1 leak per same-content call.
|
||||
// Mirrors the pattern `store_derived_blob` uses on its
|
||||
// `ON CONFLICT DO NOTHING` `inserted == 0` branch.
|
||||
// See `docs/plan/refcount-audit.md` for how the audit stream
|
||||
// would surface this class of drift if it reappears.
|
||||
//
|
||||
// - `previous == None` — brand new (file_id, kind, variant) row.
|
||||
// `store_from_stream`'s +1 pairs with the new row's implicit
|
||||
// reference; nothing to release.
|
||||
if let Some((old_hash,)) = previous {
|
||||
let source = if old_hash == attached_hash {
|
||||
refcount_audit_source::STORE_ATTACHED_BLOB_SAME_CONTENT_BALANCE
|
||||
} else {
|
||||
refcount_audit_source::STORE_ATTACHED_BLOB_REPLACE_RELEASE
|
||||
};
|
||||
if let Err(e) = self.remove_reference(&old_hash).await {
|
||||
tracing::warn!(
|
||||
target: "oxicloud::dedup",
|
||||
error = %e,
|
||||
kind = source,
|
||||
"failed to balance attached-blob reference for {}",
|
||||
&old_hash[..old_hash.len().min(12)],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(attached_hash)
|
||||
}
|
||||
|
||||
/// Atomic never-overwrite variant of [`Self::store_attached_blob`],
|
||||
/// for migration/import paths whose semantic is "write if absent,
|
||||
/// leave alone if present" — mirrors the shape
|
||||
/// [`Self::store_derived_blob`] already uses.
|
||||
///
|
||||
/// The plain `store_attached_blob` reads `previous`, then upserts,
|
||||
/// then decrements — non-transactional. That's correct for the
|
||||
/// user-driven PUT thumbnail path (a real replacement should
|
||||
/// release the superseded blob), but it opens a check-then-act
|
||||
/// race with concurrent writers when the caller's intent is
|
||||
/// "only import if this file hasn't already got a preview".
|
||||
/// `thumb_attached_import_service` is exactly that caller.
|
||||
///
|
||||
/// This variant uses a single-statement `INSERT ... ON CONFLICT
|
||||
/// DO NOTHING` — race-free by construction. If the row already
|
||||
/// exists (any content), the atomic INSERT is a no-op and we
|
||||
/// release the reference `store_from_stream` just took. If we
|
||||
/// won the insert, the reference is legitimately held by our
|
||||
/// new row.
|
||||
///
|
||||
/// Return value discriminates the two cases so the caller can
|
||||
/// track its own `imported` vs `already` counters:
|
||||
/// - [`AttachedBlobInsertOutcome::Inserted`] — we wrote the row.
|
||||
/// - [`AttachedBlobInsertOutcome::AlreadyPresent`] — a row was
|
||||
/// there when we arrived; we made no change and released our
|
||||
/// ref. `existing_hash` is the concurrent winner's blob hash,
|
||||
/// returned via a follow-up SELECT (so it's not strictly
|
||||
/// atomic with the INSERT, but that's fine — the row's shape
|
||||
/// is stable now that a concurrent writer can no longer
|
||||
/// collide with us here; any later change goes through the
|
||||
/// full `store_attached_blob` UPSERT path, which is out of
|
||||
/// scope for this method's "if absent" contract).
|
||||
pub async fn store_attached_blob_if_absent(
|
||||
&self,
|
||||
file_id: &str,
|
||||
kind: &str,
|
||||
variant: &str,
|
||||
content_type: &str,
|
||||
bytes: Bytes,
|
||||
uploaded_by: uuid::Uuid,
|
||||
) -> Result<AttachedBlobInsertOutcome, DomainError> {
|
||||
let stored = self
|
||||
.store_from_stream(
|
||||
stream::once(async move { Ok::<Bytes, std::io::Error>(bytes) }),
|
||||
Some(content_type.to_string()),
|
||||
)
|
||||
.await?;
|
||||
let attached_hash = stored.hash().to_string();
|
||||
|
||||
// Single-statement atomic INSERT. `ON CONFLICT DO NOTHING`
|
||||
// means: if another writer got there first, we silently
|
||||
// yield. Same primitive `store_derived_blob` uses.
|
||||
let inserted = sqlx::query(
|
||||
"INSERT INTO storage.file_attached_blobs
|
||||
(file_id, kind, variant, blob_hash, content_type, uploaded_by)
|
||||
VALUES ($1::uuid, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT (file_id, kind, variant) DO NOTHING",
|
||||
)
|
||||
.bind(file_id)
|
||||
.bind(kind)
|
||||
.bind(variant)
|
||||
.bind(&attached_hash)
|
||||
.bind(content_type)
|
||||
.bind(uploaded_by)
|
||||
.execute(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?
|
||||
.rows_affected();
|
||||
|
||||
if inserted == 0 {
|
||||
// Row already existed when we arrived. Release the ref
|
||||
// `store_from_stream` above took — the row that would
|
||||
// justify it is not ours. Best-effort: leaving a
|
||||
// dangling ref is worse than a warn log line.
|
||||
if let Err(e) = self.remove_reference(&attached_hash).await {
|
||||
tracing::warn!(
|
||||
target: "oxicloud::dedup",
|
||||
error = %e,
|
||||
"failed to release duplicate attached-blob reference for {}",
|
||||
&attached_hash[..attached_hash.len().min(12)],
|
||||
);
|
||||
}
|
||||
|
||||
// Fetch the concurrent winner's hash so the import
|
||||
// service can readback-verify against its sidecar. The
|
||||
// window between DO NOTHING and this SELECT is narrow;
|
||||
// if the row gets updated in it, the sidecar delete
|
||||
// path fails its verify and keeps the sidecar — the
|
||||
// conservative fallback.
|
||||
let existing = self.find_attached_blob(file_id, kind, variant).await;
|
||||
return Ok(AttachedBlobInsertOutcome::AlreadyPresent {
|
||||
existing_hash: existing.map(|r| r.blob_hash).unwrap_or_default(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(AttachedBlobInsertOutcome::Inserted {
|
||||
hash: attached_hash,
|
||||
})
|
||||
}
|
||||
|
||||
/// Look up bytes attached to a file. File-keyed counterpart of
|
||||
/// [`Self::find_derived_blob`].
|
||||
pub async fn find_attached_blob(
|
||||
@@ -1316,6 +1518,17 @@ impl DedupService {
|
||||
total_size,
|
||||
chunk_hashes.len(),
|
||||
);
|
||||
// The manifest INSERT above set `ref_count = 1` — that's
|
||||
// the initial reference held by whatever callsite drove
|
||||
// this ingest (a file's body, a preview attachment, a
|
||||
// derivation). Audit-log the +1 so drift investigations
|
||||
// can find where a manifest first came into existence.
|
||||
audit_ref_count(
|
||||
refcount_audit_table::CHUNK_MANIFESTS,
|
||||
file_hash,
|
||||
1,
|
||||
refcount_audit_source::STORE_FROM_STREAM_NEW_MANIFEST,
|
||||
);
|
||||
self.fire_blob_creation_hooks(file_hash, content_type.as_deref());
|
||||
return Ok(DedupResultDto::NewBlob {
|
||||
hash: file_hash.to_string(),
|
||||
@@ -1747,7 +1960,7 @@ impl DedupService {
|
||||
/// Bump a manifest's ref_count if it exists; returns its total_size.
|
||||
/// Single statement — no window between the existence check and the bump.
|
||||
async fn bump_manifest_if_exists(&self, file_hash: &str) -> Result<Option<i64>, DomainError> {
|
||||
sqlx::query_scalar::<_, i64>(
|
||||
let bumped = sqlx::query_scalar::<_, i64>(
|
||||
"UPDATE storage.chunk_manifests SET ref_count = ref_count + 1
|
||||
WHERE file_hash = $1
|
||||
RETURNING total_size",
|
||||
@@ -1757,7 +1970,17 @@ impl DedupService {
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("Dedup", format!("Failed to bump manifest ref_count: {e}"))
|
||||
})
|
||||
})?;
|
||||
|
||||
if bumped.is_some() {
|
||||
audit_ref_count(
|
||||
refcount_audit_table::CHUNK_MANIFESTS,
|
||||
file_hash,
|
||||
1,
|
||||
refcount_audit_source::BUMP_MANIFEST_IF_EXISTS,
|
||||
);
|
||||
}
|
||||
Ok(bumped)
|
||||
}
|
||||
|
||||
/// Stream → chunk store, WITHOUT creating a manifest.
|
||||
@@ -2219,6 +2442,12 @@ impl DedupService {
|
||||
.rows_affected();
|
||||
|
||||
if manifest_affected > 0 {
|
||||
audit_ref_count(
|
||||
refcount_audit_table::CHUNK_MANIFESTS,
|
||||
hash,
|
||||
1,
|
||||
refcount_audit_source::ADD_REFERENCE_MANIFEST,
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
@@ -2246,6 +2475,12 @@ impl DedupService {
|
||||
));
|
||||
}
|
||||
|
||||
audit_ref_count(
|
||||
refcount_audit_table::BLOBS,
|
||||
hash,
|
||||
1,
|
||||
refcount_audit_source::ADD_REFERENCE_LEGACY,
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -2355,6 +2590,17 @@ impl DedupService {
|
||||
&file_hash[..12],
|
||||
chunk_hashes.len()
|
||||
);
|
||||
// Emit AFTER the commit so a rolled-back TX doesn't leave a
|
||||
// phantom audit line — the "delta = -current_rc" reflects
|
||||
// "the manifest is gone, effective ref_count is 0". Convention
|
||||
// for the audit stream: use the delta that would produce a
|
||||
// read-back of 0.
|
||||
audit_ref_count(
|
||||
refcount_audit_table::CHUNK_MANIFESTS,
|
||||
file_hash,
|
||||
-current_rc,
|
||||
refcount_audit_source::REMOVE_MANIFEST_REFERENCE_DELETE,
|
||||
);
|
||||
Ok(true)
|
||||
} else {
|
||||
// Still has references — just decrement
|
||||
@@ -2373,6 +2619,12 @@ impl DedupService {
|
||||
.map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?;
|
||||
|
||||
tracing::debug!("Reference removed from manifest {}", &file_hash[..12]);
|
||||
audit_ref_count(
|
||||
refcount_audit_table::CHUNK_MANIFESTS,
|
||||
file_hash,
|
||||
-1,
|
||||
refcount_audit_source::REMOVE_MANIFEST_REFERENCE_DECREMENT,
|
||||
);
|
||||
Ok(false)
|
||||
}
|
||||
}
|
||||
@@ -2430,6 +2682,12 @@ impl DedupService {
|
||||
self.reap_blob(hash).await;
|
||||
|
||||
tracing::info!("BLOB DELETED: {} (no more references)", &hash[..12]);
|
||||
audit_ref_count(
|
||||
refcount_audit_table::BLOBS,
|
||||
hash,
|
||||
-ref_count,
|
||||
refcount_audit_source::REMOVE_LEGACY_REFERENCE,
|
||||
);
|
||||
Ok(true)
|
||||
} else {
|
||||
// Still has references — just decrement
|
||||
@@ -2450,6 +2708,12 @@ impl DedupService {
|
||||
})?;
|
||||
|
||||
tracing::debug!("Reference removed from blob {}", &hash[..12]);
|
||||
audit_ref_count(
|
||||
refcount_audit_table::BLOBS,
|
||||
hash,
|
||||
-1,
|
||||
refcount_audit_source::REMOVE_LEGACY_REFERENCE,
|
||||
);
|
||||
Ok(false)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -274,49 +274,31 @@ impl RecoverableJobHandler for ThumbAttachedImport {
|
||||
};
|
||||
let file_id_str = file_id.to_string();
|
||||
|
||||
// Already mapped. Checked BEFORE storing, because
|
||||
// `store_attached_blob` is ON CONFLICT DO UPDATE and would
|
||||
// release then retake the reference on every run.
|
||||
if let Some(existing) = self
|
||||
.dedup
|
||||
.find_attached_blob(&file_id_str, "preview", &dir_name)
|
||||
.await
|
||||
{
|
||||
already += 1;
|
||||
// Drains on a later run too: importing first and enabling
|
||||
// deletion afterwards is the expected operator sequence,
|
||||
// so reaching here is the common path rather than an edge
|
||||
// case.
|
||||
if delete_imported {
|
||||
let path = self.thumbnails_root.join(&dir_name).join(&name);
|
||||
if ThumbDerivedImport::verify_and_unlink(
|
||||
&self.dedup,
|
||||
THUMB_ATTACHED_IMPORT_JOB_NAME,
|
||||
&file_id_str,
|
||||
&existing.blob_hash,
|
||||
&path,
|
||||
)
|
||||
.await
|
||||
{
|
||||
deleted += 1;
|
||||
} else {
|
||||
unverified += 1;
|
||||
record_or_log(
|
||||
store,
|
||||
THUMB_ATTACHED_IMPORT_JOB_NAME,
|
||||
"sidecar_delete_unverified",
|
||||
"anomaly",
|
||||
None,
|
||||
serde_json::json!({
|
||||
"path": position,
|
||||
"file_id": file_id_str,
|
||||
"note": "attached blob did not read back; sidecar kept",
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
} else if !self.file_exists(file_id).await {
|
||||
// Orphan check first — no atomic-insert exists for a
|
||||
// file_id whose FK would reject. Same rationale as before;
|
||||
// the race window between this check and the INSERT is
|
||||
// narrow AND covered by the FK constraint if the file is
|
||||
// deleted after we look — the atomic INSERT would then
|
||||
// fail loudly instead of silently drift.
|
||||
//
|
||||
// Everything else — "row present" and "row absent" —
|
||||
// used to be split across two branches with a
|
||||
// non-transactional `find_attached_blob` between the
|
||||
// check and the write. That opened a check-then-act
|
||||
// race window: a concurrent thumbnail writer could
|
||||
// INSERT the row after the check returned None, and the
|
||||
// subsequent `store_attached_blob` UPSERT-UPDATE would
|
||||
// fire with same-or-different content. In the
|
||||
// same-content case that leaked +1 on the manifest ref
|
||||
// (pre-fix; guard branch now cancels).
|
||||
//
|
||||
// Merged into ONE atomic call
|
||||
// `store_attached_blob_if_absent`: single-statement
|
||||
// `INSERT ... ON CONFLICT DO NOTHING`, race-free by
|
||||
// construction. The outcome enum distinguishes the two
|
||||
// paths so `imported` and `already` counters stay
|
||||
// accurate.
|
||||
if !self.file_exists(file_id).await {
|
||||
// The file is gone, so this sidecar is unimportable: the
|
||||
// FK on `file_id` would reject the row. Mirrors the
|
||||
// dead-source case in thumb_derived_import.
|
||||
@@ -389,9 +371,10 @@ impl RecoverableJobHandler for ThumbAttachedImport {
|
||||
let path = self.thumbnails_root.join(&dir_name).join(&name);
|
||||
match fs::read(&path).await {
|
||||
Ok(data) => {
|
||||
use crate::infrastructure::services::dedup_service::AttachedBlobInsertOutcome;
|
||||
match self
|
||||
.dedup
|
||||
.store_attached_blob(
|
||||
.store_attached_blob_if_absent(
|
||||
&file_id_str,
|
||||
"preview",
|
||||
&dir_name,
|
||||
@@ -404,14 +387,39 @@ impl RecoverableJobHandler for ThumbAttachedImport {
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(attached_hash) => {
|
||||
imported += 1;
|
||||
if delete_imported {
|
||||
Ok(outcome) => {
|
||||
// Bump the right counter AND pick the
|
||||
// hash we'll verify-and-unlink against:
|
||||
// Inserted — our new blob
|
||||
// AlreadyPresent — the concurrent
|
||||
// winner's blob
|
||||
// Both drain the sidecar identically
|
||||
// (verify-then-unlink on repair mode).
|
||||
let verify_hash = match outcome {
|
||||
AttachedBlobInsertOutcome::Inserted { hash } => {
|
||||
imported += 1;
|
||||
hash
|
||||
}
|
||||
AttachedBlobInsertOutcome::AlreadyPresent {
|
||||
existing_hash,
|
||||
} => {
|
||||
already += 1;
|
||||
existing_hash
|
||||
}
|
||||
};
|
||||
// Empty existing_hash only happens if
|
||||
// the AlreadyPresent path's follow-up
|
||||
// SELECT was overtaken by another
|
||||
// writer. verify_and_unlink would
|
||||
// refuse the sidecar delete in that
|
||||
// case anyway, but skipping the call
|
||||
// saves the pointless readback.
|
||||
if delete_imported && !verify_hash.is_empty() {
|
||||
if ThumbDerivedImport::verify_and_unlink(
|
||||
&self.dedup,
|
||||
THUMB_ATTACHED_IMPORT_JOB_NAME,
|
||||
&file_id_str,
|
||||
&attached_hash,
|
||||
&verify_hash,
|
||||
&path,
|
||||
)
|
||||
.await
|
||||
|
||||
@@ -94,6 +94,19 @@ pub async fn basic_auth_middleware(
|
||||
let (raw_username, password) =
|
||||
parse_basic_auth(auth_header).ok_or(NextcloudAuthError::Unauthorized)?;
|
||||
|
||||
// Canonicalise the whole Basic-Auth username to lowercase.
|
||||
//
|
||||
// Usernames are canonical (lowercase) in the DB post-migration
|
||||
// (`docs/plan/username-lowercase.md`), and NC / DAVX5 clients that
|
||||
// cached URLs from before the migration keep sending `Alice:pass`
|
||||
// — the server continues to accept that indefinitely by
|
||||
// lowercasing here. Safe for the multi-drive `user~drive_uuid`
|
||||
// composite because UUID hex `[0-9a-f-]` lowercases to itself.
|
||||
//
|
||||
// ASCII-only by `validate_username`'s charset check, so
|
||||
// `to_ascii_lowercase` is deterministic and locale-safe.
|
||||
let raw_username = raw_username.to_ascii_lowercase();
|
||||
|
||||
// ── Multi-drive composite-username parse ────────────────────────
|
||||
// POC wire shape: `{username}~{drive_marker}` may appear in the
|
||||
// Basic Auth header. `~` was chosen because it needs no URL
|
||||
@@ -319,7 +332,13 @@ pub fn parse_basic_auth(header_value: &str) -> Option<(String, String)> {
|
||||
let decoded = String::from_utf8(decoded).ok()?;
|
||||
let (user, pass) = decoded.split_once(':')?;
|
||||
|
||||
Some((user.to_string(), pass.to_string()))
|
||||
// Canonicalise the username to lowercase here too, so any caller
|
||||
// that reaches for `parse_basic_auth` directly (bypassing the
|
||||
// middleware wrapper) also sees the canonical form. Redundant with
|
||||
// the middleware's explicit `to_ascii_lowercase` on `raw_username`
|
||||
// — belt-and-braces to keep the invariant local to the parser too.
|
||||
// See `docs/plan/username-lowercase.md § 4. NextCloud DAV surface`.
|
||||
Some((user.to_ascii_lowercase(), pass.to_string()))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -107,7 +107,21 @@ fn extract_url_user(path: &str) -> Option<std::borrow::Cow<'_, str>> {
|
||||
// common path allocates nothing; only a percent-encoded username owns. The
|
||||
// old `.into_owned()` forced a `String` on EVERY path-scoped NC DAV request
|
||||
// (benches/ROUND19.md §M7). The caller compares by slice.
|
||||
urlencoding::decode(user_seg).ok()
|
||||
//
|
||||
// Lowercase before returning so cached client URLs like
|
||||
// `/dav/files/Alice/...` compare equal to the canonical
|
||||
// (lowercase) `session.raw_username`. See
|
||||
// `docs/plan/username-lowercase.md § 4. NextCloud DAV surface`.
|
||||
//
|
||||
// The lowercase transform always allocates (`to_ascii_lowercase`
|
||||
// on a `str` returns `String`). Trades the "Cow::Borrowed common
|
||||
// path" of the ROUND19 optimisation for correctness of the case-
|
||||
// insensitive comparison at line 157 — a `&str` compare with a
|
||||
// borrowed segment against a lowercase `session.raw_username`
|
||||
// would silently mismatch for `Alice`. The alloc is one small
|
||||
// String per NC DAV request; the correctness win is worth it.
|
||||
let decoded = urlencoding::decode(user_seg).ok()?;
|
||||
Some(std::borrow::Cow::Owned(decoded.to_ascii_lowercase()))
|
||||
}
|
||||
|
||||
/// Axum extractor: the shared handle to the request's [`NcSession`].
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
# =============================================================
|
||||
# OxiCloud — usernames are case-insensitive (silent lowercase on ingest)
|
||||
# =============================================================
|
||||
# Pin for `docs/plan/username-lowercase.md`. The plan makes usernames
|
||||
# case-insensitive by canonicalising to lowercase in
|
||||
# `User::validate_username`. This file covers the wire surface:
|
||||
#
|
||||
# 1. Registration with a mixed-case username lands as lowercase.
|
||||
# 2. Login by the ORIGINAL mixed-case string succeeds (server
|
||||
# normalises on lookup).
|
||||
# 3. Login by ALL-CAPS of the same name also succeeds.
|
||||
# 4. Login by the canonical lowercase form succeeds.
|
||||
# 5. Profile rename to a mixed-case name lands as lowercase.
|
||||
# 6. NC Basic Auth accepts every case variant of the same account.
|
||||
#
|
||||
# Character-set + boundary rules (no leading dot, no `@`, etc.) are
|
||||
# NOT re-tested here — that's the Rust unit-test surface. This file
|
||||
# only pins the end-to-end normalization behaviour.
|
||||
#
|
||||
# Runs after `setup.hurl` (admin exists). Uses a self-contained user
|
||||
# to avoid interfering with other scenarios.
|
||||
# =============================================================
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 1 — Register with a mixed-case username. Expect the server
|
||||
# to silently store the lowercase form.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/auth/register
|
||||
Content-Type: application/json
|
||||
{
|
||||
"username": "MixedCaseUser",
|
||||
"email": "mixedcase@example.com",
|
||||
"password": "MixedCasePassword1!"
|
||||
}
|
||||
|
||||
HTTP 200
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 2 — Log in with the ORIGINAL mixed-case string. Server
|
||||
# should normalise on lookup and accept.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{ "username": "MixedCaseUser", "password": "MixedCasePassword1!" }
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
jsonpath "$.access_token" isString
|
||||
# The `/auth/me` response inside the login reply exposes the canonical
|
||||
# stored username. Post-migration, it MUST be lowercase regardless of
|
||||
# what the caller typed at registration.
|
||||
jsonpath "$.user.full.user.username" == "mixedcaseuser"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 3 — Log in with ALL-CAPS. Same account, different casing.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{ "username": "MIXEDCASEUSER", "password": "MixedCasePassword1!" }
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
jsonpath "$.user.full.user.username" == "mixedcaseuser"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 4 — Log in with the canonical lowercase form. Same account.
|
||||
# Capture the token for the profile-rename step below.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{ "username": "mixedcaseuser", "password": "MixedCasePassword1!" }
|
||||
|
||||
HTTP 200
|
||||
[Captures]
|
||||
mixed_token: jsonpath "$.access_token"
|
||||
[Asserts]
|
||||
jsonpath "$.user.full.user.username" == "mixedcaseuser"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 5 — Rename via profile PATCH. New name is mixed-case; server
|
||||
# must store it as lowercase. Same rule as registration, applied on
|
||||
# the mutation path.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PATCH {{base_url}}/api/auth/me
|
||||
Authorization: Bearer {{mixed_token}}
|
||||
Content-Type: application/json
|
||||
{ "username": "RenamedTarget" }
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
# The response echoes the stored (canonical) form.
|
||||
jsonpath "$.full.user.username" == "renamedtarget"
|
||||
|
||||
|
||||
# Old (pre-rename) username no longer resolves — login fails 401.
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{ "username": "MixedCaseUser", "password": "MixedCasePassword1!" }
|
||||
|
||||
HTTP 401
|
||||
|
||||
|
||||
# New (post-rename) mixed-case login succeeds.
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{ "username": "RENAMEDTARGET", "password": "MixedCasePassword1!" }
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
jsonpath "$.user.full.user.username" == "renamedtarget"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 6 — NextCloud Basic Auth accepts every case variant.
|
||||
# The middleware lowercases the decoded username on the auth path,
|
||||
# and `extract_url_user` lowercases the URL segment. A cached
|
||||
# client URL like `.../dav/files/RenamedTarget` continues to work
|
||||
# indefinitely across the migration.
|
||||
#
|
||||
# Uses PROPFIND `Depth: 0` on `/remote.php/dav/files/{user}/` — a
|
||||
# well-formed request that touches Basic Auth + URL parse + chroot
|
||||
# resolve in one hop. 207 Multi-Status is the expected success shape.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PROPFIND {{base_url}}/remote.php/dav/files/renamedtarget/
|
||||
Depth: 0
|
||||
[BasicAuth]
|
||||
renamedtarget: MixedCasePassword1!
|
||||
|
||||
HTTP 207
|
||||
|
||||
|
||||
PROPFIND {{base_url}}/remote.php/dav/files/RenamedTarget/
|
||||
Depth: 0
|
||||
[BasicAuth]
|
||||
RenamedTarget: MixedCasePassword1!
|
||||
|
||||
HTTP 207
|
||||
|
||||
|
||||
PROPFIND {{base_url}}/remote.php/dav/files/RENAMEDTARGET/
|
||||
Depth: 0
|
||||
[BasicAuth]
|
||||
RENAMEDTARGET: MixedCasePassword1!
|
||||
|
||||
HTTP 207
|
||||
|
||||
|
||||
# Mixed case in URL, lowercase in Basic Auth — still works because
|
||||
# both surfaces normalise before comparison.
|
||||
PROPFIND {{base_url}}/remote.php/dav/files/RenamedTarget/
|
||||
Depth: 0
|
||||
[BasicAuth]
|
||||
renamedtarget: MixedCasePassword1!
|
||||
|
||||
HTTP 207
|
||||
@@ -254,6 +254,71 @@ ATTACHED_2=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id
|
||||
[[ "$ATTACHED_2" == "$ATTACHED_AFTER" ]] || fail "re-run duplicated attached rows"
|
||||
log "re-run is a no-op: rows and refcounts unchanged."
|
||||
|
||||
# ── 5c. store_attached_blob same-content guard (regression test) ─────────────
|
||||
#
|
||||
# The PUT /api/files/{id}/thumbnail/{size} endpoint calls
|
||||
# `dedup_service::store_attached_blob` directly — no pre-check like
|
||||
# `thumb_attached_import_service` does. A same-content re-PUT is the
|
||||
# ONLY current public surface that exercises the ref-balance branch
|
||||
# added to `store_attached_blob` after the Sept-2026 manifest-drift
|
||||
# investigation:
|
||||
#
|
||||
# if let Some((old_hash,)) = previous {
|
||||
# // same-content: cancel `store_from_stream`'s spurious +1
|
||||
# // different-hash: release the superseded blob's ref
|
||||
# remove_reference(&old_hash)
|
||||
# }
|
||||
#
|
||||
# Before the fix, same-content re-PUT would leak +1 on the manifest's
|
||||
# ref_count every time (store_from_stream incremented, guard skipped
|
||||
# the decrement when old == new). This test PUTs the same thumbnail
|
||||
# bytes twice and asserts the manifest's ref_count is unchanged.
|
||||
#
|
||||
# Uses variant "icon" so we don't collide with the row the import
|
||||
# job populated above (variant "preview"), keeping the two flows
|
||||
# independent. Server re-encodes to JPEG deterministically, so both
|
||||
# PUTs produce byte-identical manifest content.
|
||||
|
||||
log "5c. Same-content re-PUT via API does not churn refcount"
|
||||
|
||||
# Round 1: first PUT populates the row (INSERT — previous=None, guard
|
||||
# doesn't fire). This is the fresh-ingest path; ref_count becomes 1.
|
||||
curl -sf -X PUT \
|
||||
-H "$AUTH" \
|
||||
-H "Content-Type: image/jpeg" \
|
||||
--data-binary "@$UPLOADED_THUMB" \
|
||||
"$base_url/api/files/$FILE_ID/thumbnail/icon" \
|
||||
>/dev/null || fail "5c: first PUT of thumbnail (variant=icon) failed"
|
||||
|
||||
GUARD_HASH=$(sql "SELECT blob_hash FROM storage.file_attached_blobs \
|
||||
WHERE file_id='$FILE_ID' AND kind='preview' AND variant='icon' \
|
||||
LIMIT 1;")
|
||||
[[ -n "$GUARD_HASH" ]] || fail "5c: first PUT did not land a file_attached_blobs row"
|
||||
|
||||
GUARD_REFS_BEFORE=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$GUARD_HASH';")
|
||||
# Legacy blobs (pre-CDC) don't have a chunk_manifests row — skip
|
||||
# the test in that case rather than fail on an unrelated path.
|
||||
if [[ -z "$GUARD_REFS_BEFORE" ]]; then
|
||||
log "5c: attached blob is on the legacy path (no manifest) — guard test skipped (targets CDC path)"
|
||||
else
|
||||
# Round 2: second PUT with the SAME bytes. UPSERT-UPDATE fires,
|
||||
# previous.blob_hash == new attached_hash, and the guard MUST
|
||||
# cancel `store_from_stream`'s +1. Without the fix, refcount
|
||||
# would go from 1 → 2 here.
|
||||
curl -sf -X PUT \
|
||||
-H "$AUTH" \
|
||||
-H "Content-Type: image/jpeg" \
|
||||
--data-binary "@$UPLOADED_THUMB" \
|
||||
"$base_url/api/files/$FILE_ID/thumbnail/icon" \
|
||||
>/dev/null || fail "5c: same-content re-PUT of thumbnail failed"
|
||||
|
||||
GUARD_REFS_AFTER=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$GUARD_HASH';")
|
||||
[[ "$GUARD_REFS_AFTER" == "$GUARD_REFS_BEFORE" ]] \
|
||||
|| fail "5c: same-content re-PUT churned refcount: $GUARD_REFS_BEFORE → $GUARD_REFS_AFTER (guard regressed?)"
|
||||
|
||||
log "5c: same-content re-PUT stable, refcount=$GUARD_REFS_AFTER"
|
||||
fi
|
||||
|
||||
# ── 5b. Deletion: the destructive half, and the only one that can lose data
|
||||
#
|
||||
# Everything above is additive and recoverable. This unlinks files after a
|
||||
|
||||
Reference in New Issue
Block a user