Compare commits

...

10 Commits

Author SHA1 Message Date
Dionisio Pozo 4d067b3fc6 Merge pull request #727 from EdouardVanbelle/chore/ambition
Docker Publish (release, main, dry-run) / Pre-publish Tests (push) Has been cancelled
Docker Publish (release, main, dry-run) / Build & Push Multi-Arch (push) Has been cancelled
CI / changes (push) Has been cancelled
CI / Build (push) Has been cancelled
Docker Build and Test / Build and Test Docker Image (push) Has been cancelled
CI / Frontend — svelte-check, ESLint, Stylelint, Prettier (push) Has been cancelled
CI / Message-bus spec — AsyncAPI + TypeScript DTO drift (push) Has been cancelled
CI / Migration ordering (new migrations postdate target branch) (push) Has been cancelled
CI / Rustfmt (push) Has been cancelled
CI / Clippy (push) Has been cancelled
CI / Wasm — fmt + clippy (push) Has been cancelled
CI / Wasm — release tests (push) Has been cancelled
CI / Plugins — fixtures + runtime tests (push) Has been cancelled
CI / Server Unit and Functionnal Tests (push) Has been cancelled
CI / Security Audit (push) Has been cancelled
CI / API, WebDAV & OIDC tests (push) Has been cancelled
CI / Bundled-assets binary — embed + SPA-serve integration (push) Has been cancelled
CI / WebDAV RFC 4918 — litmus (59/59) (push) Has been cancelled
CI / CalDAV + CardDAV — python-caldav (push) Has been cancelled
CI / Frontend end-to-end tests (via Playwright) (push) Has been cancelled
2026-09-14 10:23:33 +08:00
Dionisio Pozo fd9e1e669f Merge pull request #728 from EdouardVanbelle/feat/username-lowercase 2026-09-14 10:23:20 +08:00
Dionisio Pozo b5648b82f9 Merge pull request #726 from EdouardVanbelle/docs/message-bus 2026-09-14 02:24:24 +08:00
Edouard Vanbelle a95a6b106c feat(username): normalize username into lowercase
- normalize username into lowercase (this is already ASCII only)
- permit users to login with their username with insensitive case
- if a disabled account is reactivated and got a collision, it will normalize it too
- server will stop on collision (ex: 2 entries with `Alice` and `alice`)
  in a such case admin can run:

```
oxicloud migrate lowercase-usernames --dry-run
```
then
```
oxicloud migrate lowercase-usernames
```
2026-09-13 19:53:09 +02:00
Edouard Vanbelle 0b9e8bfe23 docs(plan): case-insensitive usernames
Records the design for issue #691 (make usernames case-insensitive):
silently lowercase on ingest, explicit `oxicloud migrate
lowercase-usernames [--dry-run]`, refuse-to-boot until DB is fully
lowercase. Includes the chunked-upload directory rename step and the
OIDC JIT lowercase fix surfaced during the design sweep.

Design deferred pieces (display_name split, WebDAV URL redesign) are
listed under "Not in scope" so the boundary is explicit.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-09-13 17:00:36 +02:00
Edouard Vanbelle 03f2956457 docs(message-bus + notifications): add arch doc 2026-09-13 14:12:47 +02:00
Edouard Vanbelle 47346b9309 chore: ambition and target 2026-09-13 13:21:05 +02:00
Dionisio Pozo 4840930767 Merge pull request #724 from EdouardVanbelle/fix/ref_count 2026-09-13 16:29:34 +08:00
Dionisio Pozo 69fc88982e Merge pull request #723 from EdouardVanbelle/feat/notifications 2026-09-13 16:29:14 +08:00
Edouard Vanbelle ca85ac7307 fix(file_attached): doe not increment ref_count if new attachement has same hash
- do not increment ref_count if new attachement to a file with same data
- add audit log to help identifying other future issue in ref_count
- prevent race condition while attaching a blob
2026-09-13 02:00:32 +02:00
23 changed files with 3070 additions and 105 deletions
+66
View File
@@ -2,6 +2,72 @@
This file provides guidance to coding agents (Claude Code, Codex, Cursor, Aider, …) working with this repository. Claude Code reads it via `@AGENTS.md` in `CLAUDE.md`. This file provides guidance to coding agents (Claude Code, Codex, Cursor, Aider, …) working with this repository. Claude Code reads it via `@AGENTS.md` in `CLAUDE.md`.
# Purpose — what OxiCloud is, and what it is not
Read this before designing anything. Most "should we…" questions are answered by
the scale target rather than by taste.
- **Open source, and it stays that way.** MIT (`LICENSE`). This is a constraint on
what you may add, not a footnote: dependencies must be license-compatible — a
GPL/AGPL crate or npm package would force the whole project to relicense, and
that is not on the table. No feature may be gated behind a licence key or an
"enterprise edition", and nothing core may hard-depend on a proprietary service
or SDK. Vendored frontend assets (`frontend/static/vendors/`) carry the same
rule; record the licence when vendoring.
- **Self-hosted**, for an individual or an enterprise. The operator is not an SRE
team: defaults must be safe, failures loud, and nothing may silently depend on a
cloud service.
- **Target scale is up to ~10k users.** Design against that number in both
directions. Do not build million-user machinery (sharding, eventual consistency,
service decomposition) for a load that will never arrive; equally, never ship
anything O(users) per request, or a table scan that is fine at 50 users and
fatal at 10k.
- **Not a mass hoster.** OxiCloud does not claim to serve millions of users on one
deployment, and trade-offs should not pretend otherwise.
- **Decentralised by intent.** Many instances federating beats one large instance —
OpenCloudMesh is one route. Prefer designs that survive "this is one of many
instances" over ones assuming a single authoritative deployment.
Targets:
- **Feature ambition: Google Workspace / Office 365.** Breadth of capability is a
goal, not scope creep.
- **Collaboration is the main feature axis.** OxiCloud is not a personal backup
drive that happens to have sharing bolted on — sharing, shared drives, grants,
co-editing (WOPI) and live updates are the product. When choosing what to build
or how to build it, the multi-user case is the primary one, not the case to
generalise to later. A feature that works only for a single owner is unfinished.
- **Customer target: NextCloud users.** Hence the NextCloud-compatible API surface
(`/remote.php`, `/ocs`, `/status.php`) — compatibility is a feature, and breaking
it costs adopters.
## Design axes
Four things decide an open design question. **Security and resilience are
absolute** — they are not traded against anything. Performance is measured against
the 10k target. Privacy is a direction with a stated endpoint.
- **Resilience.** This is a storage product: **no data loss, no data corruption,
ever.** Anything that can silently drop or alter bytes is a top-severity defect,
not a trade-off. In practice that means: a job that skips work must never report
success (pause at a cursor instead — `docs/plan/jobs-handling-recoverable-error.md`);
a read failure is never proof that data is absent; content-addressing and
ref-counting are load-bearing, not decoration; and consistency checks are
discovery-only unless repair is explicitly requested.
- **Security.** Prefer deny-by-default over assert-later; a guarantee enforced by
the type system or the router beats one a reviewer must remember. AuthZ lives in
the service layer, never in handlers. See `src/AGENTS.md` § AuthZ enforcement
points.
- **Performance.** Measure against 10k users, not a dev instance. The hot paths are
listing, thumbnails and auth — a per-row query or an extra round trip there is a
real regression even when it looks harmless.
- **Privacy.** When the backend belongs to a third party (S3, Azure), encryption at
rest is a *should-have*; **end-to-end encryption is the target.** Designs that
assume the server can always read plaintext will have to be undone — the `Vault`
drive kind is reserved for the E2E case.
Where two conflict, resilience and security win, and the cost is documented.
# Architecture # Architecture
This project is split into two parts: This project is split into two parts:
+4
View File
@@ -176,6 +176,10 @@ export default defineConfig({
{ text: "Authentication model", link: "/architecture/auth-model" }, { text: "Authentication model", link: "/architecture/auth-model" },
{ text: "Magic-link auth", link: "/architecture/magic-link-auth" }, { text: "Magic-link auth", link: "/architecture/magic-link-auth" },
{ text: "Background jobs", link: "/architecture/jobs" }, { text: "Background jobs", link: "/architecture/jobs" },
{
text: "Message bus & notifications",
link: "/architecture/message-bus-and-notifications",
},
{ text: "UI diagnostics", link: "/architecture/ui-diagnostics" }, { text: "UI diagnostics", link: "/architecture/ui-diagnostics" },
], ],
}, },
+1
View File
@@ -77,3 +77,4 @@ src/
- [Backend Storage →](/architecture/backend-storage) - [Backend Storage →](/architecture/backend-storage)
- [Derived and Attached Blobs →](/architecture/derived-and-attached-blobs) — thumbnails, transcodes and uploaded previews: why content-keyed and file-keyed artifacts need separate tables - [Derived and Attached Blobs →](/architecture/derived-and-attached-blobs) — thumbnails, transcodes and uploaded previews: why content-keyed and file-keyed artifacts need separate tables
- [Background Jobs →](/architecture/jobs) - [Background Jobs →](/architecture/jobs)
- [Message Bus & Notifications →](/architecture/message-bus-and-notifications) — real-time WebSocket bus (topics, AuthZ scopes, tab-visibility grace-close), persistent notifications (bell), AsyncAPI vs OpenAPI schema ownership
@@ -0,0 +1,645 @@
# Message Bus & Persistent Notifications
OxiCloud has two coupled subsystems that together power its real-time
UX — a **live message bus** over WebSocket for "something just
happened, refresh your view", and a **persistent notifications
table** for "you need to know about this even if you weren't
online." This document explains how both work, how they authenticate
and authorize subscribers, and why the frontend deliberately drops
the WebSocket while a tab is hidden.
Design docs the shipped code implements: [`docs/plan/message-bus.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md)
+ [`docs/plan/templated-messages.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md).
---
## The two channels
| | **Message bus (WebSocket)** | **Persistent notifications (REST + DB)** |
|---|---|---|
| Purpose | "Something changed, refresh your view" | "You need to know about this — later is fine" |
| Transport | JSON-RPC 2.0 over `/api/rt/ws` | `GET/POST/DELETE /api/notifications/*` + `notif.notifications` table |
| Delivery | Best-effort, in-memory, no replay | Durable, per-user rows, survive reboot / offline |
| Payload | Thin "poke" facts (id + verb) | Full per-kind DTO with all render data |
| Loss on disconnect | Yes (events during outage window are dropped) | No (rows are the source of truth) |
| Schema owner | AsyncAPI (`resources/gen/asyncapi.json`) | OpenAPI (`resources/gen/openapi.json`) |
The two work together: an ingester that wants to notify a user
writes **both** — the DB row (for durability + the bell's history)
AND publishes a bus event on `user:{u}:notifications` (so online
sessions refetch instantly instead of waiting for the next
mount). The wire event on that topic is a **pure poke** — empty
`data: {}`. The row's real content only ever crosses the REST wire.
---
## Schema ownership — AsyncAPI vs OpenAPI
The bus and the REST endpoints have separate wire specs. The rule
the codebase adopts to keep them from drifting:
> **AsyncAPI defines the envelope + transport for clients.
> OpenAPI defines the payload.**
Concretely:
| Type | Home | How it stays in sync with Rust |
|---|---|---|
| **Bus events** (`MessageBusEvent`, subscribe / unsubscribe frames, revoked notifications, envelope shape) | **AsyncAPI** — `resources/gen/asyncapi.json` | Hand-written in `src/bin/generate-asyncapi.rs` via `json!` macros; kept in lockstep with `MessageBusEvent`'s serde shape. Small drift risk — Rust is truth. |
| **REST DTOs** (response bodies, request bodies, per-kind notification payloads) | **OpenAPI** — `resources/gen/openapi.json` | `#[derive(utoipa::ToSchema)]` on the Rust struct. Utoipa walks `#[utoipa::path(...)]` handlers + registered schemas. **No drift possible** — projection is derived from Rust. |
| **Types on both wires** (rare; none today) | Would live as one Rust struct with both derives, or wait for single-source codegen | — |
### Why this split, not one unified spec
The instinct is to put the notification payload schema in AsyncAPI
alongside the bus event that triggers a refetch. It looks cleaner
until you realize the payload **never travels on the bus wire** —
the bus event is `NotificationReceived` with empty `data: {}`, a
pure cache-invalidation poke. The FE fetches the payload from
`GET /api/notifications`, which is REST → OpenAPI's territory.
Putting the payload schema in AsyncAPI would mean "documenting
this shape on a transport it doesn't travel on" — a conceptual
stretch that adds a drift risk for zero gain.
### Rejected alternatives
- **Dual-spec (same type declared in both AsyncAPI + OpenAPI).**
Guaranteed drift unless both come from a single codegen. Nothing
in the tooling today produces both, so we'd hand-maintain two
copies of every shared type. Bug factory.
- **Cross-spec `$ref`** — AsyncAPI 3.0 allows `"$ref":
"openapi.json#/components/schemas/Foo"`, and utoipa's
`components(schemas(...))` can publish orphan types (no
`#[utoipa::path]` reference) so OpenAPI advertises "internal"
schemas. Technically workable but: Modelina + Swagger UI + Redoc
handle external refs inconsistently, OpenAPI stops being "the
REST contract" and becomes "a general schema registry",
reviewers get confused. Legal, fragile, avoided.
- **Bus event carries the full payload** (revert the pure-poke
design). Would put per-kind payload schemas in AsyncAPI as
`MessageBusEvent::NotificationReceived { granter_id, resource_id,
… }`. Rejected because the FE has to REST-fetch anyway (bell
reads from DB for history + persistence), so the fields on the
wire are dead weight — same-content overlap between the two
specs, no consumer benefit.
- **Session-resume tokens** (`rt.subscribe { since: N }` +
server-side ring buffer). Would let the bus deliver missed rows
directly on reconnect, saving one REST round-trip. Rejected for
**backward compatibility with `OXICLOUD_MESSAGEBUS_ENABLE=false`**:
ops who disable the WS rely on the bell falling back to REST;
bus-only replay would leave those deployments with no catch-up
path. The REST `?after=` cursor works in every mode (bus on, bus
off, network gap); the bus stays purely "instant-poke".
### What this looks like in the tree
- `src/application/ports/message_bus_ports.rs` — `MessageBusEvent`
enum (Rust source of truth for bus wire shapes).
- `src/bin/generate-asyncapi.rs` — projects those Rust variants
into `resources/gen/asyncapi.json`.
- `src/domain/entities/notification.rs` — `SharegrantedPayload`
and its siblings, `#[derive(ToSchema)]`, source of truth for
REST payload shapes.
- `src/interfaces/api/mod.rs` — utoipa `#[openapi(components(schemas(SharegrantedPayload, ...)))]`
registers the payload in OpenAPI even though `NotificationDto.payload`
stays `serde_json::Value` on the response type. (FE type-narrows
on `row.kind` and casts to the right shape.)
- Nothing lives in both specs today.
### Adding a new bus event
1. Add a variant to `MessageBusEvent`.
2. Add the variant to `generate-asyncapi.rs`'s `event_kind` enum
and (if the variant has payload fields) a schema function.
3. Regenerate AsyncAPI + FE DTOs via `just asyncapi` +
`npm run gen:message-bus`.
4. **Do not** add the variant to OpenAPI. Bus events don't
travel on REST.
### Adding a new notification kind's payload
1. Add a Rust struct in `domain/entities/notification.rs` with
`#[derive(Serialize, Deserialize, ToSchema)]`.
2. Register it in `src/interfaces/api/mod.rs`'s
`components(schemas(...))` list.
3. Regenerate OpenAPI via `just openapi`.
4. **Do not** add the struct to AsyncAPI. Notification payloads
only cross the REST wire.
---
## Message bus
### Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ SERVICE LAYER │
│ │
│ ShareService.grant() ── after commit ──▶ bus.publish(…) │
│ FileMgmtService.…() ── after commit ──▶ bus.publish(…) │
│ NotificationService ── after commit ──▶ bus.publish(…) │
│ Scheduler engine ── on run start/end ──▶ bus.publish │
└──────────────────────────┬──────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ MessageBus port (application/ports/message_bus_ports.rs) │
│ │
│ InProcessMessageBus │
│ DashMap<Topic, tokio::broadcast::Sender<Event>> │
└──────────────────────────┬──────────────────────────────────┘
│
│ (optional replicator seam)
▼
NoopReplicator (v1)
PgListenReplicator (deferred)
BrokerReplicator (RabbitMQ / NATS, deferred)
│
▼
┌─────────────────────────────────────────────────────────────┐
│ WS handler (interfaces/api/handlers/rt_ws.rs) │
│ │
│ One session per socket: │
│ HashMap<wire_key, Sub> + outbound mpsc │
│ │
│ rt.subscribe / rt.unsubscribe frames │
│ rt.event / rt.revoked / rt.pong notifications │
└─────────────────────────────────────────────────────────────┘
```
### Topics — a typed enum, not a string
```rust
pub enum Topic {
Folder(Uuid), // "folder:{uuid}"
UserAuthz(Uuid), // "user:{uuid}:authz"
UserNotifications(Uuid), // "user:{uuid}:notifications"
Job(String), // "job:{name}"
}
```
Defined in `application/ports/message_bus_ports.rs`. Encoded to a
stable dotted wire form; parsed back with strict validation. The
wire form doubles as a routing key for future broker replicators
(RabbitMQ topic exchanges, NATS subjects).
### Events
`MessageBusEvent` (same module) is the discriminated union of every
payload a publisher can produce — `FileCreated`, `FolderMoved`,
`AuthzChanged`, `JobRunStarted / Progress / Ended`,
`NotificationReceived`, etc. Serde tags with `#[serde(tag = "event",
rename_all = "snake_case")]`, so the wire is
`{"event": "file_created", "file_id": "...", "actor": "..."}`.
Payloads are **thin facts**: the ID of the changed resource + the
actor + the verb. Clients refetch details via REST if they need
them. Keeps the AuthZ surface small (thin payloads can't leak
fields the caller couldn't already read via REST) and keeps events
well under any future broker's message-size cap.
### Wire protocol
JSON-RPC 2.0 over text frames. Full protocol in [`docs/plan/message-bus.md § Wire protocol`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md).
```jsonc
// Client → server
{"jsonrpc":"2.0","id":1,"method":"rt.subscribe","params":{"topic":"folder:abc-…"}}
// Server → client (ack)
{"jsonrpc":"2.0","id":1,"result":{"subscribed":"folder:abc-…"}}
// Server → client (push, id-less notification)
{"jsonrpc":"2.0","method":"rt.event","params":{
"topic":"folder:abc-…",
"event":"file_created",
"data":{"file_id":"…","name":"…","parent_id":"…","actor":"…"}
}}
```
### Authentication for the WebSocket upgrade
Two paths, both accepted by the same handler:
| Client kind | Path | Why |
|---|---|---|
| **Programmatic** (CLI, test helper) | `Authorization: Bearer <jwt>` on the upgrade | The `new WebSocket()` API in browsers can attach `Sec-WebSocket-Protocol` but NOT arbitrary headers, so browsers can't do this. |
| **Browser** | `POST /api/rt/ticket` (with the full DPoP + CSRF middleware chain) mints a one-shot 30-second opaque UUID; the browser opens the WS with `Sec-WebSocket-Protocol: oxi.ticket.<uuid>` | DPoP-bound sessions cannot attach a `DPoP:` header to `new WebSocket()`. The ticket flow moves the DPoP check to a normal POST that DOES support headers, and the WS upgrade just redeems the opaque token. |
Tickets are single-use, TTL 30 s, stored in a `RtTicketStore`
(in-memory). Redemption removes the entry — replay is impossible.
The WS route is deliberately mounted **outside** the
`protected_api` middleware stack — otherwise the DPoP-required
layer would 401 every browser on upgrade before the ticket flow
could kick in.
---
## The three AuthZ scopes
`Topic::required_perm(&self) -> AuthzCheck` dispatches every
subscribe attempt into exactly one of three classes. This is the
authoritative diagram of what the WS handler enforces:
```
┌──────────────────────────────────────────┐
│ Topic::required_perm() │
└─────┬─────────────┬─────────────┬────────┘
│ │ │
ResourceRead │ IdentityMatch │ RoleAdmin
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Class 1 │ │ Class 2 │ │ Class 3 │
│ per-resource│ │ per-user │ │ per-session │
│ │ │ │ │ (role) │
└─────────────┘ └─────────────┘ └─────────────┘
```
### Class 1 — Per-resource (`AuthzCheck::ResourceRead`)
**Topics:** `folder:{id}`, and (future) `file:{id}`,
`drive:{id}`, `calendar:{id}`, `addressbook:{id}`.
**Rule:** the caller must hold **`Read`** on the resource via the
same `AuthorizationEngine` that guards every REST endpoint. Owner
short-circuits pass; direct grants pass; group-mediated grants
pass; drive-membership cascades pass. Everything else is denied.
**Wire response on denial:** JSON-RPC error object with
`code = -32001`, `message = "no_read"`. Same shape whether the
resource doesn't exist OR the caller lacks the grant — **anti-
enumeration invariant**. Audit reason (`no_read` /
`no_such_resource`) distinguishes internally.
**On grant revocation:** the WS handler auto-subscribes each
session to `user:{caller}:authz` (Class 2 below). When a
`MessageBusEvent::AuthzChanged { affected_folders }` fires, the
session's reader translates it to an internal `EvictFolders`
signal → the main loop walks the sub set and drops any Class-1
subscription whose resource was affected, emitting a client-visible
`rt.revoked` notification per evicted topic. Same pattern applies
to any Class-1 topic when the AuthZ model widens beyond folders.
### Class 2 — Per-user, strict privacy (`AuthzCheck::IdentityMatch`)
**Topics:** `user:{u}:authz`, `user:{u}:notifications`.
**Rule:** direct UUID equality — `caller_id == u`. **No admin
bypass, no group indirection, no owner short-circuit.** Admins
cannot subscribe to other users' `:authz` or `:notifications`
streams; that's a privacy invariant, not a mere policy choice.
**Wire response on mismatch:** `topic_forbidden` — the **same
wire shape as an unknown topic**. An attacker probing
`user:{someone_else_uuid}:authz` cannot distinguish "user exists
but not me" from "no such user."
**Auto-subscription:** the WS handler auto-subscribes every session
to its own `user:{caller}:authz` AND `user:{caller}:notifications`
at session open. No `rt.subscribe` frame is needed from the client
for these — they're always active for the caller's own UUID.
### Class 3 — Per-session role (`AuthzCheck::RoleAdmin`)
**Topics:** `job:{name}` today. Future `admin:*` topics land here.
**Rule:** the session's snapshotted role at open time must be
`admin`. The handler resolves `caller_role` once during session
setup via `resolve_live_role` and stores it on the session state —
no per-subscribe DB round-trip.
**Wire response on non-admin:** `topic_forbidden` — same anti-enum
shape as Class 2. A non-admin probing job topics cannot enumerate
which jobs are registered.
**Why snapshot at session open, not per subscribe:** admin role
loss is rare + trivially recoverable (the user closes the tab and
reopens, hitting the fresh role check). Per-subscribe checks would
be an extra DB round-trip on every frame with no meaningful
security gain — the WS session itself was authenticated at upgrade
time under the current role.
### Adding a new topic
Every new topic variant must decide which class it belongs to at
`Topic::required_perm`. The compiler enforces exhaustiveness — a
new variant with no branch fails to build, which is deliberate. New
topics get audited before shipping precisely because the
`required_perm` match forces the author to state the class
explicitly.
---
## Tab-visibility grace-close — reducing idle connections
Every open browser tab holds one WebSocket to the server. A user
with five tabs open holds five sockets. A user who leaves a tab
open all day but only uses one holds five sockets, four of them
serving nothing.
The frontend closes the WebSocket **after 60 seconds of tab
hidden** and reopens it when the tab becomes visible again. The
subscription state is preserved locally through the outage — every
subscriber's release handle stays live, the reactive store still
holds the last-known list — but the wire is silent while the tab
is hidden.
### Implementation
Frontend `MessageBusClient` (`frontend/src/lib/message-bus/client.svelte.ts`)
attaches a `visibilitychange` listener on construction:
- **Tab hidden** → starts a 60-second timer.
- **Timer fires while still hidden** → close the WS via a
`#closeForHidden` path that sets state to `disconnected` but
preserves `#subs` for later replay. The `#onClose` handler is
guarded by `#tabIsHidden()` — an auto-reconnect won't fire while
the tab remains hidden.
- **Tab visible again** → cancels the timer if it hadn't fired
yet; if the WS was closed, kicks off a normal reconnect.
- **On reopen**, the WS handler auto-subscribes to `:authz` and
`:notifications` again, and the client replays every entry in
`#subs` as `rt.subscribe` frames. From the user's POV, the state
is identical to what they left behind.
### What this trades
- **Saved**: N-1 idle sockets per user with N tabs open, over the
hidden-tab window. Meaningful at scale (100 users × 5 tabs × 8
idle hours = 4000 idle-tab-hours of connection state to keep
alive per day).
- **Lost**: bus events published during the 60-second delay
(transient) + the whole grace-close window (indefinite while
hidden) are dropped for that session. **Recovery**: on reopen,
every consumer that cares refetches. See "Reconnect catch-up"
below.
### Why 60 seconds
Short enough that leaving a tab for a coffee break doesn't burn
the connection. Long enough that the momentary focus-shifts
users do all day (Cmd-Tab to another app, back within seconds)
don't churn the socket. Not tunable per-user — the value is
hard-coded in `HIDDEN_GRACE_MS`.
### Reconnect catch-up — the `?after=` cursor
For consumers whose state can't be reconstructed by refetching a
current listing (specifically: **notifications**, whose bell must
show rows that landed during the outage), the FE issues a delta
fetch:
- Store tracks `#lastReceivedAt` — the newest `created_at` seen
before disconnect.
- On `messageBus.onReconnect(...)` fire, calls
`GET /api/notifications?after=<lastReceivedAt>&limit=100`.
- Merges the returned rows into local state via `mergeById` —
duplicates are resolved with **incoming wins** (server value
overrides local, so a `read_at` flip on another device shows
up correctly).
The server-side `?after=` predicate is **strict `>`** — a row at
exactly `lastReceivedAt` is excluded. This makes the WS push (which
delivers a row at time T) and the delta fetch (which asks for
"anything after T") non-overlapping by construction. `mergeById`
handles the case where the two paths race and both deliver the
same row.
For consumers whose state IS a current listing (folder view: the
files/subfolders in a folder), reconnect just refetches the listing
via the normal REST endpoint. `useReconnect` composable exposes
`onReconnect(cb)` as a one-liner for that pattern.
---
## Persistent notifications (the bell)
### Data model
```sql
notif.notifications (
id UUID PRIMARY KEY,
user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
kind TEXT NOT NULL, -- 'share_granted' | 'new_login_from_new_device' | …
payload JSONB NOT NULL, -- per-kind shape (see below)
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
read_at TIMESTAMPTZ -- NULL = unread
)
```
Two indexes:
- `(user_id, created_at DESC) INCLUDE (read_at, kind)` covers the
bell's list query + the mark-all-read filter.
- `(read_at) WHERE read_at IS NOT NULL` — partial, tiny on healthy
DBs; feeds the retention job's DELETE.
### Ingesters
An ingester is a code path that calls
`NotificationApplicationService::create(NewNotification)`. The
service atomically:
1. `INSERT INTO notif.notifications RETURNING …` — durable row.
2. `bus.publish(Topic::UserNotifications(user_id), NotificationReceived)` — the fast-path poke.
Publish happens **after** the DB write succeeds, never inside a
transaction — a rolled-back INSERT would otherwise fan out a lie.
**Currently shipped ingester:** `share_granted` in
`interfaces/api/handlers/grant_handler.rs::create_grant`. Fires
after `authz.set_role(...)` succeeds. Fans out to every resolved
recipient user:
- `Subject::User(id)` → one row for that user.
- `Subject::Group(id)` → one row per transitive member (via
`SubjectGroupService::list_transitive_users`).
- `Subject::Token(_)` → no row (anonymous share links have no
target user).
Self-shares (owner grants themselves via a group they belong to)
skip. Failure to write is best-effort — a warn log; the grant row
stays durable in `role_grants`, the recipient can still discover
the share via `/api/grants/shared-with-me`.
**Planned but not wired** (each needs a prerequisite subsystem
listed in [`docs/plan/message-bus.md § Deferred`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md)):
| Kind | Prerequisite |
|---|---|
| `new_login_from_new_device` | Device-fingerprint tracking table |
| `job_completed_for_you` | Scheduler engine threading the trigger caller's `caller_id` through `dispatch()` |
| `storage_quota_threshold` | Per-user usage/quota comparator with threshold-crossing detection |
### Payload shape — typed per kind
The `payload` column is JSONB (schema-free at the DB layer). Each
kind's Rust shape lives in `domain/entities/notification.rs` with
`#[derive(Serialize, Deserialize, ToSchema)]`. OpenAPI picks up the
struct automatically. Adding a new field is additive on JSONB — no
migration.
Example — `share_granted`:
```rust
pub struct SharegrantedPayload {
pub granter_id: Uuid,
pub resource_type: String, // 'folder' | 'file' | 'drive' | …
pub resource_id: Uuid,
pub resource_name: Option<String>, // snapshot at grant time
pub resource_path: Option<String>, // storage path snapshot
pub navigate_folder_id: Option<Uuid>, // FE routing target for drives
pub role: String,
pub expires_at: Option<DateTime<Utc>>,
}
```
The name/path fields are **snapshotted at grant time**. If the
folder is later renamed or moved, the notification still reflects
what it was called when the share happened. Same principle as
email invitations or activity feeds: the record is a fact about
what was true at the moment, not a live pointer.
### Schema-ownership rule — AsyncAPI vs OpenAPI
> **AsyncAPI defines the envelope + transport for clients.
> OpenAPI defines the payload.**
The bus event `MessageBusEvent::NotificationReceived` is a **unit
variant** — it serializes to `{"event":"notification_received",
"data":{}}` with no fields on the wire. The topic identifies the
semantic; the FE responds by refetching from REST.
The payload's shape lives in OpenAPI via `ToSchema` on
`SharegrantedPayload` (and its future siblings), auto-derived from
Rust. AsyncAPI never sees these types — payloads don't travel on
the bus wire.
**Why this split** — it eliminates schema drift between the two
specs. A payload edit changes Rust → OpenAPI updates on regen
(mechanical). AsyncAPI stays stable (hand-written, but it never
touches payloads). Same rule applies to any future bus consumer
that also has a REST DTO — Rust is the source of truth; each spec
projects the parts of Rust that travel on its transport.
Design rationale + rejected alternatives (dual-spec, cross-`$ref`,
per-kind Svelte components) in [`docs/plan/templated-messages.md § Schema ownership`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md).
### REST surface — `/api/notifications`
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/notifications` | List newest-first. Query params: `unread` (bool), `before` / `after` (cursors), `limit` |
| `GET` | `/api/notifications/unread` | Badge-only fast path (returns just `unread_count`) |
| `POST` | `/api/notifications/{id}/read` | Mark one row read (idempotent, always 204) |
| `POST` | `/api/notifications/read-all` | Bulk mark-all-read (returns rows updated) |
| `DELETE` | `/api/notifications/{id}` | Hard-delete one row (idempotent, always 204) |
**Anti-enumeration:** mark-read and delete always respond 204 —
whether the row existed and belonged to the caller, or didn't
exist at all, or belonged to someone else. Every mutating endpoint
scopes on `caller_id` at the SQL layer; the response shape is
identical across the three outcomes.
### Retention
The `notifications_cleanup` scheduled job (daily, same tier as
`trash_cleanup`) DELETEs read rows older than
`OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` (default 30). **Unread
rows are preserved unconditionally** — a user offline for a month
still sees the share-granted notice when they log back in.
Runtime override via the job's `retention_days` parameter on the
admin panel's trigger — the env default seeds it, the panel
overrides at trigger time.
### Frontend rendering
`frontend/src/lib/composables/useNotifications.svelte.ts` owns the
module-scoped store — one instance per SPA session. Exposes:
- `notifications.items` — reactive list (newest first)
- `notifications.unread` — reactive badge count
- `notifications.refresh()` / `refreshDelta()` / `markRead(id)` /
`markAllRead()` / `delete(id)`
`NotificationRow.svelte` handles the actual rendering — one file,
one `switch` on `row.kind`, one rich template per shipped kind
(`share_granted` today; the others fall back to a generic string).
Extraction into per-kind components is deferred until a single
kind's block exceeds ~30 lines or two kinds start needing the same
sub-component (see [`docs/plan/templated-messages.md § Rendering`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md)).
### Notification click routing
| `resource_type` | Route | Data used |
|---|---|---|
| `folder` | `/files/{resource_id}` | `resource_id` |
| `file` | `/shared-with-me?file={resource_id}` | `resource_id` — the `/files/{uuid}` route requires a **folder** id, and a file-scoped grant may not include parent-folder access. `/shared-with-me` is the guaranteed-accessible home for every recipient of a `share_granted`, and its `?file=` deep link opens the inline `FileViewer`. |
| `drive` | `/files/{navigate_folder_id}` | Drives have no browsable URL of their own; `navigate_folder_id` is the drive's `root_folder_id`, enriched at ingest via `DriveRepository::get_by_id`. |
| `calendar` / `address_book` / `playlist` | no link (bold text) | Not addressable via `/files/*`. |
The bell also fires a **transient toast** (via the existing
`ui.notify(...)` mechanism) on every fresh row that arrives via
delta — the toast fades out after ~4 s while the persistent row
stays in the bell's history section. Same bell icon, same badge
count, no duplicate UX.
---
## Feature flags & config
| Env var | Default | Effect |
|---|---|---|
| `OXICLOUD_MESSAGEBUS_ENABLE` | `true` | Master switch. When `false`, the `/api/rt/ws` and `/api/rt/ticket` routes are **not registered** at boot (Axum returns 404), and the FE `useNotifications` composable skips the WS setup entirely. The bell falls back to REST-only mode — polling on mount, delta on manual refresh. Zero client-side error spam. |
| `OXICLOUD_MESSAGEBUS_KEEPALIVE_SECONDS` | `30` | Server-initiated RFC 6455 Ping interval on each WS connection. Prevents intermediate proxies (nginx, Traefik, Cloudflare) from reaping the TCP session as idle. |
| `OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` | `30` | Retention window for read notifications. Unread rows are always preserved. The `notifications_cleanup` job clamps to a minimum of 1 day. |
Client discovers all of these via `GET /api/config` — no in-band
"does the server support the bus?" probe needed. The FE reads
`serverConfig.features.message_bus` at boot and skips WS setup
entirely when it's false.
---
## Failure modes
| Scenario | Behavior |
|---|---|
| Bus is disabled server-side (`OXICLOUD_MESSAGEBUS_ENABLE=false`) | `/api/rt/ws` returns 404. `useTopic` in the FE returns early. Bell works via REST only. |
| Network drops mid-session | Client-side jittered exponential backoff (250 ms → 30 s cap, 20-failure circuit breaker). On reconnect, WS handler re-auto-subscribes to `:authz` + `:notifications`; `useReconnect` composable fires `onReconnect` callbacks so views refetch. |
| Server restart | Same as network drop — the WS breaks, client backs off, reconnects when server is back. Events published during the outage are lost (no persistent event log by design); consumers refetch. |
| Tab hidden > 60 s | WS closed via `visibilitychange` grace-close. State preserved locally. On visibility return, reconnect + replay subscriptions. |
| Publish before commit | Not allowed. Every publish site is documented as "after commit". A publish inside a transaction that rolls back would fan out a lie. |
| Broker replicator failure (future) | The local `InProcessMessageBus` publishes still succeed — the replicator is beside the bus, not in front. Broker-hop failures affect multi-instance fanout but never local delivery. |
---
## Testing
The api-test suite exercises the full stack end-to-end via `rt-hurl-helper` (a small Rust binary gated on `test_utils`) — Hurl alone can't drive a WebSocket. Sixteen scenarios in `tests/api/rt_bus_check.sh`:
- Positive delivery, topic isolation
- AuthZ denial (Class 1 folder), unknown-topic anti-enum
- Server keepalive, delete emits, move fan-out
- Grant-revoke eviction (`AuthzChanged` → `rt.revoked`)
- Cross-user identity gate (Class 2)
- Ticket happy path + single-use replay refused
- Admin-only job topic (Class 3)
- `notification_received` wire push, DB row via `GET /api/notifications`
- Cross-user notifications identity gate (Class 2, notifications topic)
- `?after=` cursor with strict-`>` boundary invariant
`mergeById` — the FE's WS/reconnect race dedup — has its own
Vitest with 5 covered cases (empty, non-overlap, exact-dup,
stale-read-at overwrite, mixed overlap).
---
## Further reading
- Plan doc: [`docs/plan/message-bus.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/message-bus.md) — full design rationale, roadmap, and deferred slices (Yjs collab, broker replicator, SharedWorker, Web Push).
- Plan doc: [`docs/plan/templated-messages.md`](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/templated-messages.md) — schema-ownership rule, rendering shape, notification routing decision.
- [ReBAC Authorization](/architecture/rebac-authorization) — the engine every Class-1 topic gate calls.
- [Background jobs](/architecture/jobs) — `notifications_cleanup` is one of them; `Topic::Job` publishes on job lifecycle.
+64
View File
@@ -123,6 +123,70 @@ rather than as a visible error.
| `OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` | `30` | Retention window for **read** notification rows (`notif.notifications`). The `notifications_cleanup` scheduled job runs daily and deletes rows where `read_at IS NOT NULL` and `read_at < now() - retention_days`. Unread rows are preserved unconditionally — the whole point of the durable table is that a user offline for a month still sees the share-granted notice on next login. Clamped to a minimum of 1 (0 would purge every read row on every tick). Adjust down for compliance-sensitive deployments where "cleared once seen" matters; adjust up when operators expect users to reference old notifications for support. | | `OXICLOUD_NOTIFICATIONS_RETENTION_DAYS` | `30` | Retention window for **read** notification rows (`notif.notifications`). The `notifications_cleanup` scheduled job runs daily and deletes rows where `read_at IS NOT NULL` and `read_at < now() - retention_days`. Unread rows are preserved unconditionally — the whole point of the durable table is that a user offline for a month still sees the share-granted notice on next login. Clamped to a minimum of 1 (0 would purge every read row on every tick). Adjust down for compliance-sensitive deployments where "cleared once seen" matters; adjust up when operators expect users to reference old notifications for support. |
| `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX` | `@drive` | Native WebDAV URL segment that renders the caller's drive list. Sanitized by trimming leading/trailing `/`. Three shapes: (1) default `@drive` — `/webdav/…` addresses the caller's default personal drive (back-compat), `/webdav/@drive/` returns the drive listing, `/webdav/@drive/<uuid\|name>/…` targets a specific drive. (2) empty string `""` — `/webdav/` IS the drive listing, `/webdav/<uuid\|name>/…` targets a specific drive, no default-drive shortcut. (3) any other string (e.g. `drives`) — same shape as `@drive` with that segment substituted. Only drives the caller has Read on via `role_grants` resolve. | | `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX` | `@drive` | Native WebDAV URL segment that renders the caller's drive list. Sanitized by trimming leading/trailing `/`. Three shapes: (1) default `@drive` — `/webdav/…` addresses the caller's default personal drive (back-compat), `/webdav/@drive/` returns the drive listing, `/webdav/@drive/<uuid\|name>/…` targets a specific drive. (2) empty string `""` — `/webdav/` IS the drive listing, `/webdav/<uuid\|name>/…` targets a specific drive, no default-drive shortcut. (3) any other string (e.g. `drives`) — same shape as `@drive` with that segment substituted. Only drives the caller has Read on via `role_grants` resolve. |
## Boot-time refuse-to-boot checks
Some upgrades add invariants the running database must satisfy
BEFORE the new binary can serve traffic. These are enforced by
read-only checks that run after `sqlx::migrate!()` and before the
server binds a listen socket. If a check fails, the server exits
with a FATAL message spelling out the exact CLI command to run.
The server **never silently mutates data** at boot — every fix is
an explicit `oxicloud migrate <name>` invocation. Follows the
"discovery-only by default, mutation opt-in" rule that also
governs the consistency-check jobs.
### `lowercase-usernames`
Three outcomes at boot, only one of which stops the server:
- **All lowercase (or `NULL`)** — the check is a no-op, boot
proceeds unchanged.
- **Mixed-case rows exist, no `LOWER(username)` collision** —
boot **auto-lowercases** them in one atomic transaction, emits
a structured audit line per rename
(`user.username_lowercased_on_boot`, INFO) plus an INFO summary
(`user.usernames_lowercased_on_boot_summary` with `renamed=N`),
and continues. Silent action is confined to the case with
exactly one correct move: `Alice` (with no `alice` row) becomes
`alice`.
- **`LOWER(username)` collision** — two or more active rows share
the same lowercase form (e.g. `Alice` + `alice`). Boot
**refuses to start** with a FATAL message spelling out every
collision group and the exact CLI command to resolve it.
Tiebreak needs a human.
Soft-deleted / disabled accounts (`active = false`) and `NULL`
usernames (OPAQUE-migrated) are skipped in every case.
**Refusal message pattern:** `FATAL: cannot start — N colliding
username group(s) (M affected account(s) in total)`, followed by
each group's canonical form and its members with `id` +
`last_login`. Up to 10 groups shown; the `--dry-run` CLI reveals
the full list.
**Fix (only required when the server refused):**
```
oxicloud migrate lowercase-usernames --dry-run # preview the tiebreak
oxicloud migrate lowercase-usernames # apply
```
**What the migration does:** lowercases every mixed-case
username. On collision (`Alice` + `alice` both exist), the
tiebreak `(last_login_at DESC NULLS LAST, created_at ASC)` picks
a winner; losers get `alice-2`, `-3`, … as a suffix. Sessions
and grants survive the rename — both key on the user's UUID.
**Client compat:** NextCloud clients that cached URLs like
`/remote.php/dav/files/Alice/…` continue to work indefinitely —
the Basic Auth middleware and URL parser both lowercase on
decode. NC desktop clients will prompt a one-time re-sync on
first PROPFIND after upgrade; DAVX5 and NC mobile handle it
silently. See `docs/install/binary.md § Upgrading from a
case-sensitive-usernames release` for the full upgrade flow.
**Design:** `docs/plan/username-lowercase.md`.
## Storage Entries (multi-entry, recommended) ## Storage Entries (multi-entry, recommended)
Declare one or more **named** storage backends. The one the app runs on is picked from the DB (`admin_settings.storage.active_backend_name`); the admin panel's storage tab flips the pointer, and cross-backend migration is a recoverable job that copies blobs between two entries with a read-only safety window. See [Admin Settings — Storage & Migration](/config/admin-settings) for the operator flow and the [multi-entry design doc](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/storage-multi-entry.md) for the full model. Declare one or more **named** storage backends. The one the app runs on is picked from the DB (`admin_settings.storage.active_backend_name`); the admin panel's storage tab flips the pointer, and cross-backend migration is a recoverable job that copies blobs between two entries with a read-only safety window. See [Admin Settings — Storage & Migration](/config/admin-settings) for the operator flow and the [multi-entry design doc](https://github.com/oxicloud/oxicloud/blob/main/docs/plan/storage-multi-entry.md) for the full model.
+69
View File
@@ -217,6 +217,75 @@ supported by sqlx's migration model; if you need to roll back, stop
the server, roll back your Postgres data directory to a snapshot, and the server, roll back your Postgres data directory to a snapshot, and
install the previous binary. install the previous binary.
### Upgrading from a case-sensitive-usernames release
Releases that predate the case-insensitive-usernames change stored
`Alice`, `alice`, and `ALICE` as three separate accounts. The
current release treats usernames as case-insensitive (canonical
lowercase in the database). What happens on the first boot after
upgrade depends on your data:
**No mixed-case usernames.** The check is a no-op; the server
starts normally. Nothing to do.
**Mixed-case usernames with no collision.** The server
**auto-lowercases** them in one atomic transaction at boot and
continues. Each rename is recorded in the audit log
(`user.username_lowercased_on_boot`) and a WARN summary line
names the total count. `Alice` (with no `alice` row alongside)
becomes `alice`; no ops action needed. This covers the vast
majority of single-admin self-hosted deployments.
**Mixed-case usernames WITH a collision** (`Alice` + `alice`
both exist as active accounts). The server **refuses to boot**
— tiebreak requires a human. Run the migration:
```
# Preview the tiebreak — no writes.
sudo -u oxicloud DATABASE_URL="postgres://..." \
/usr/local/bin/oxicloud migrate lowercase-usernames --dry-run
# Apply. Renames run in a single transaction; safe to re-run if aborted.
sudo -u oxicloud DATABASE_URL="postgres://..." \
/usr/local/bin/oxicloud migrate lowercase-usernames
```
On collision, the migration picks a winner by `(last_login_at
DESC NULLS LAST, created_at ASC)` — most recently active keeps
the canonical lowercase name; the loser gets `alice-2`, `-3`, …
as a suffix. **Sessions and grants survive the rename** — both
key on the user's UUID, not the username.
Skipped in every path above: soft-deleted / disabled accounts
and OPAQUE-migrated users whose `username` column is NULL.
Neither blocks boot.
After the migration completes, restart the service:
```
sudo systemctl start oxicloud
```
**Nextcloud desktop clients** will prompt a one-time re-sync on
their first PROPFIND after upgrade — the account URL case
changed. Data is safe (files re-verify via ETag, not re-uploaded).
DAVX5 (calendars, contacts) and NC mobile handle the URL case
change silently. **No client upgrade or reconfiguration is
required** — the server accepts uppercase URL segments (`Alice`
in `/remote.php/dav/files/Alice/...`) indefinitely.
The refusal message printed by the server on boot (collision
path only) includes the exact CLI command above, so you can't
miss it. Full plan and rationale in
`docs/plan/username-lowercase.md`.
**Explicit-preview path.** If you'd rather run the migration
before the binary swap — to review renames on your own schedule
or to gate a backup step — run `oxicloud migrate
lowercase-usernames --dry-run` against the OLD binary's DB
first, then apply. On next boot the new binary sees a
lowercase-clean DB and the auto-rename path is a no-op.
## Installing via `cargo binstall` ## Installing via `cargo binstall`
If you already have the Rust toolchain and just want the binary If you already have the Rust toolchain and just want the binary
+581
View File
@@ -0,0 +1,581 @@
# Plan — Case-insensitive usernames (lowercase-on-ingest)
## Context
Feature ask: [issue #691](https://github.com/AtalayaLabs/OxiCloud/issues/691).
Usernames are currently case-sensitive, so `Alice`, `alice`, and `ALICE`
refer to three different accounts. Users hit this as a login friction —
they type their name with different capitalization on different clients
and get "invalid credentials" instead of a successful login.
## Why this is simpler than it looks in this codebase specifically
- `validate_username` in `src/domain/entities/user.rs:884` already
restricts usernames to ASCII-only `[a-zA-Z0-9._-]{2,64}` with no `@`.
The Unicode case-folding minefield (Turkish dotted-I, German ß, Greek
final sigma, NFC vs NFD) does not apply — ASCII case-folding is
trivial (`to_ascii_lowercase`), deterministic, and locale-independent.
- OIDC identity binds via `(iss, sub)` in
`get_user_by_federation_subject` at
`src/infrastructure/repositories/pg/user_pg_repository.rs:1257-1303` —
case-sensitivity of the local username is orthogonal to OIDC identity
matching. No OIDC breakage risk.
- Password verification runs through Argon2's `verify_password`
(constant-time by construction). Not affected.
- `@`-forbidden rule in usernames is the disjoint namespace with email
lookup (`dispatch_login` at `auth_application_service.rs:1018`).
Case-insensitive usernames align semantics with email addresses
(already case-insensitive in practice), so any future
`groupname@domain` composition stays consistent.
- NextCloud URL `/remote.php/dav/files/{user}/…` uses `{user}` as an
informational / consistency-check marker, not a security boundary —
the chroot ACL is the real authz. Handling case in the URL segment is
a small local change (documented in `session.rs:33-34`).
## Design decisions
1. **Silently lowercase on ingest** (registration, admin-create, OIDC
provisioning, rename). Never reject uppercase input from clients —
accept liberally, store strictly (Postel's Law).
2. **Explicit migration** (`oxicloud migrate lowercase-usernames
[--dry-run]`). The server never mutates `auth.users` at boot. Ops
MUST run the migration explicitly. Follows the
[[feedback_no_silent_auto_repair]] rule: consistency tenants are
discovery-only by default; mutation is opt-in.
3. **Refuse-to-boot** if any active-user username is not already
lowercase. Boot error message shows the exact CLI command to run.
Boot performs a read-only verification only.
4. **Collision tiebreak** on migration: `(last_login_at DESC NULLS
LAST, created_at ASC)`. Winner keeps the canonical lowercased name.
Losers get `-2`, `-3`, … suffix (increment until free), matching the
pattern in `oxicloud migrate nfc-filenames`.
5. **Active accounts only** in the boot check + migration. Soft-deleted
/ disabled rows are skipped (they don't block usable logins). NULL
usernames (OPAQUE-migrated accounts) are skipped in every layer —
the boot verifier, the migration UPDATE, the CLI report. The
`WHERE username <> LOWER(username)` predicate is already NULL-safe
by SQL semantics (NULL comparisons yield NULL, filtered out); state
it explicitly so a reviewer isn't left wondering.
6. **Rename-only** on collision resolution — sessions are not
invalidated. Sessions key on `user_id` so they survive the rename.
7. **Un-soft-delete of a mixed-case account uses the SAME suffix
scheme.** If `Alice` is soft-deleted (skipped by migration) and
later un-soft-deleted while `alice` already exists, the un-soft-
delete path re-normalizes via `set_username` and, on collision,
assigns `alice-2` / `alice-3` / … — the same helper the migration
CLI calls. Both callers reach for a shared
`find_free_username_suffix(pool, base) -> String` in
`src/common/username_migration.rs` so migration + un-soft-delete
agree by construction. Without this, an un-soft-delete could
create a fresh collision that the next boot's auto-rename
couldn't resolve (auto-rename handles singletons only) — the
server would then refuse-to-boot until an admin resolves the
tiebreak.
8. **Boot-time behaviour has three outcomes, not two.** The
verifier categorises the DB into: (a) clean — nothing to do;
(b) mixed-case rows with no `LOWER(username)` collision — the
server **auto-lowercases them in one atomic transaction and
continues**, emitting an audit line per rename; (c) at least
one `LOWER(username)` collision — the server **refuses to
boot** because tiebreak requires human judgement. Silent
action is bounded to (b), where there is exactly one correct
move. This is a narrower reading of
[[feedback_no_silent_auto_repair]] than "no silent action
ever": the rule targets consistency-check jobs where drift is
a bug signal; a schema-adjacent boot invariant with a
unique-correct-fix is a different situation. Making the
trivial-case common path a no-op massively lowers upgrade
friction for the 90% self-hosted deployment.
## Not in scope
- Unicode case-folding (usernames are ASCII-only by validation).
- `display_name` split (usernames were already just identifiers;
free-form display is a separate future feature if a user asks for
it — deferred pending real demand signal).
- OIDC provisioning behaviour change beyond the ingest-normalize point.
- Case-insensitivity for emails (already achieved in practice; not
touched).
- Any change to `validate_username`'s character-class rules.
- Any change to the WebDAV URL shape `/dav/files/{user}/…` (client
compat; drop deferred separately per [[project_nc_multidrive_poc]]).
- Group names (`SubjectGroup`). Lowercase by convention today; no
runtime enforcement, no migration. If group-name case-insensitivity
becomes a real ask, it lands as a sibling plan doc with the same
shape.
## Deliverables
### 1. Ingest normalization
Change `validate_username` to return the canonical form instead of
`()`:
```rust
// src/domain/entities/user.rs — new signature
fn validate_username(username: &str) -> UserResult<String> {
let normalized = username.trim().to_ascii_lowercase();
// ... existing length + charset + boundary checks apply to `normalized` ...
Ok(normalized)
}
```
Every caller that today does `Self::validate_username(u)?;` becomes
`let u = Self::validate_username(&u)?;` — the returned canonical form
is what gets stored. Because the return type changes from `Result<()>`
to `Result<String>`, any caller that ignores the result now becomes a
compile error — the type system forces every write path through the
normalizer.
Write sites all funnel through `User::new` (`src/domain/entities/user.rs:313`)
or `User::set_username` (`:811`), so the signature change catches the
entity-write path automatically. Callers to touch:
- Application services calling `User::new`:
- `auth_application_service.rs:840` — `register()` public signup
- `auth_application_service.rs:944` — `setup_create_admin()`
first-boot admin
- `auth_application_service.rs:3520`, `:3532` — `admin_create_user()`
external + internal branches
- `auth_application_service.rs:4692` — OIDC JIT provisioning
- `magic_link_invite_service.rs:233` — magic-link external invite
- User-driven rename (calls `User::set_username`):
- `auth_application_service.rs:2756-2801` — `update_profile()`
- Repository-write compile-error catches:
- `src/infrastructure/repositories/pg/user_pg_repository.rs:281`
(`create_user` INSERT) and `:740` (`update_user` UPDATE) — these
bind `user_clone.username()`, which is now guaranteed lowercase by
the entity constructor.
### 2. OIDC JIT derivation
`auth_application_service.rs:4649-4690` derives a local username from
the OIDC `preferred_username` / `name` / `sub` claims, filters to
`[a-zA-Z0-9._-]`, and truncates. **It does not currently lowercase.**
Add `to_ascii_lowercase()` on the derived string before passing to
`User::new`. This is beyond what the entity signature change catches —
explicit fix required.
### 3. Lookup normalization
Repository `find_by_username`-style methods internally lowercase the
input before the SQL query, so callers don't have to remember. One-line
change per method:
- `src/infrastructure/repositories/pg/user_pg_repository.rs:487`
(`get_user_by_username`) — add `let username = username.trim().
to_ascii_lowercase();` before the `.bind(&username)` at line 488.
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1043`
(`search_users`) — `ILIKE` is already case-insensitive by
construction; verify nothing regresses.
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1504`
(`search_usernames`) — same as above.
- `src/application/services/storage_usage_service.rs:145-149`
(`update_user_storage_usage_by_username`) — raw SQL bind; normalize
before `.bind()`.
- `src/cli/opaque.rs:125`, `:202` — `opaque reset` CLI identifier
dispatch on `@`; lowercase the username branch input.
Post-migration, the DB is fully lowercase so `WHERE username = 'alice'`
matches. The mixed-case-DB-during-transition state cannot serve
traffic because the boot flow either (a) auto-renames the singleton
rows before `AppState` assembles, or (b) refuses to boot on collision
groups.
### 4. NextCloud DAV surface
Two coordinated changes on the NC surface:
- `src/interfaces/nextcloud/basic_auth_middleware.rs:94-134` — decoded
`raw_username` from the Basic Auth header, lowercase the whole
string. Safe for the `user~drive_uuid` multi-drive format because
UUID hex is `[0-9a-f-]` which lowercases to itself.
- `src/interfaces/nextcloud/basic_auth_middleware.rs:307-323`
(`parse_basic_auth` helper) — lowercase the username portion before
returning.
- `src/interfaces/nextcloud/session.rs:90-111`
(`extract_url_user`) — lowercase the returned `Cow<'_, str>` value
from URL decode. The cross-check comparison at `session.rs:157-161`
(`url_user != session.raw_username`) then compares normalized vs
normalized — no change needed at the comparison site itself.
Downstream `session.raw_username` consumers (WebDAV / OCS href
builders, MOVE Destination parsers, avatar / trashbin handlers) all
pass through and emit lowercase automatically — no per-site change
needed.
**Client compatibility:** NC / DAVX5 clients that cached URLs like
`/remote.php/dav/files/Alice/…` continue to work through the migration
because the server accepts uppercase URL segments **indefinitely**
(the Basic Auth middleware + `extract_url_user` both lowercase on
decode). No forced client upgrade or reconfiguration. PROPFIND
response bodies emit lowercase hrefs (from canonical
`session.raw_username`), which well-behaved clients update on next
sync.
Expected per-client behavior on first PROPFIND after upgrade:
- **Nextcloud desktop** — prompts a one-time re-sync notification
when it notices the account URL case changed. Files re-verify
via ETag, so no re-upload; the re-sync completes in
seconds-to-minutes depending on file count. Users click through
the reconnect dialog.
- **DAVX5** (calendars, contacts) — silent update of the internal
`principal-URL`; user sees no dialog.
- **NC mobile app** — silent refresh of the account tile.
- **Older / misbehaving clients** — may create a duplicate account
profile (rare, cosmetic, not destructive).
**Zero data risk in every path.** The chroot ACL keys on
`user_id`, not username, so files, calendars, contacts, and
grants all follow the user across the rename. The blast radius
is a one-time UX notification, not lost bytes.
**Power-user pre-emption** (worth documenting in CHANGELOG): ops
who want to avoid the re-sync prompt entirely can, before
upgrading, log into each NC desktop client and manually update
the account URL from `.../USERNAME` to lowercase. Cheap
prophylactic for organizations rolling out to non-technical
users.
### 5. Chunked-upload directory rename
`src/infrastructure/services/nextcloud_chunked_upload_service.rs:99-103`
uses `user.username` as an on-disk directory name AND as an in-memory
cache key. Post-migration, `user.username` becomes lowercase; any
in-flight upload for `Alice` at migration time strands the on-disk
`base_dir/Alice/upload_xxx/` directory and orphans its cache entry.
The migration command SHOULD walk `base_dir/*/` and rename any
mixed-case subdirectory to its lowercase form. Collision handling
(both `Alice/` and `alice/` present) → merge contents; else simple
rename. In practice this is likely a no-op — chunked-upload state
is ephemeral, and simultaneous mixed-case uploads by the same user
are rare.
**Implementation status:** deferred. Chunked-upload state is
ephemeral: any in-flight upload that gets stranded is retryable
by the client (the upload session's timeout eventually purges the
stale dir; the client retries with a fresh `upload_id`, this time
under the lowercase username). Wiring the dir-walk into the CLI
adds ~40 lines of async filesystem code (walk, collision merge,
mtime-preserving move) and a new `--chunk-dir <path>` arg — the
CLI otherwise doesn't need to know about the storage-path
config layer. Not worth it for a rare no-op; add if user reports
show a real problem.
**Ops manual step** — if a migration is run WHILE an upload is
in flight, ops can either restart the affected client (the
upload session is stateful across a `create → chunks → complete`
cycle, so the client will retry from scratch) or manually
`mv base_dir/Alice base_dir/alice` after the DB migration
completes.
### 6. Boot-time verification
New module `src/common/username_migration.rs` exposing:
```rust
pub async fn verify_all_usernames_lowercase(pool: &PgPool) -> Result<(), String>
```
Runs after `sqlx::migrate!()` completes, before `AppState` is
assembled. Query:
```sql
SELECT id, username, created_at, last_login_at
FROM auth.users
WHERE username <> LOWER(username)
-- NULL usernames (OPAQUE-migrated accounts) are already filtered
-- out by SQL semantics: NULL <> anything yields NULL, which
-- WHERE excludes. Explicit for the reviewer's benefit.
-- add is_deleted / disabled filter if such a flag exists
ORDER BY LOWER(username),
(last_login_at IS NULL),
last_login_at DESC NULLS LAST,
created_at ASC
LIMIT 200; -- soft cap on error-message size
```
If empty → boot proceeds. If non-empty → format the FATAL error and
return `Err(String)`. `main.rs` propagates via `?` to a non-zero
process exit.
Boot only READS `auth.users`; never WRITES. This is the "explicit
migration required" enforcement layer.
**Error message format** (self-sufficient — no docs required at 3 AM):
```
FATAL: cannot start — <N> user account(s) have non-lowercase usernames.
Before this version can boot, run the migration:
oxicloud migrate lowercase-usernames --dry-run # preview
oxicloud migrate lowercase-usernames # apply
Affected accounts (up to 20 shown; full list via the dry-run):
Alice (id: a1b2c3d4-... last_login: 2026-08-01)
BOB (id: 9abc0000-... last_login: never)
...
The migration handles case-collisions (Alice + alice → alice keeps
the name based on most recent login; the other gets alice-2 suffix).
Sessions and grants survive the rename (they key on user_id).
```
### 7. Migration CLI
New action under `oxicloud migrate`:
```rust
// src/cli/migrate.rs — extend the Action enum
Action::LowercaseUsernames { dry_run: bool }
```
Following the shape of `run_nfc_filenames`:
- Load all active users (skip soft-deleted / disabled AND rows
where `username IS NULL` — OPAQUE-migrated accounts have no
username string to normalize)
- Group by `LOWER(username)`
- For each group:
- Single-member group with mixed-case name → UPDATE to lowercase
- Multi-member group (collision) → apply tiebreak
`(last_login_at DESC NULLS LAST, created_at ASC)`, winner UPDATEs
to lowercase, losers UPDATE to `<lowercase>-2`, `-3`, … (increment
until free)
- Per-row `println!` log:
`NORMALIZE user=<uuid> '<before>' ({}B) → '<after>' ({}B)`
- Summary at end: scanned / already-lowercase / normalized /
collision-resolved / renamed-to-suffix
- `--dry-run` guards all UPDATEs
After the DB pass, the chunked-upload directory rename step (see
Deliverable 5) is deferred; run manually only if in-flight uploads
were live at migration time.
Suffix search reuses the pattern from
`find_free_folder_duplicate_name` in the existing NFC migration —
increment-until-free loop, starting at `-2`, probing until an
unused suffix is found. Robust against pre-existing rows like
`alice-2` already being taken (the probe just steps past them
to `-3`, `-4`, …).
Extracted into a shared public helper in
`src/common/username_migration.rs`:
```rust
pub async fn find_free_username_suffix(pool: &PgPool, base: &str) -> Result<String, sqlx::Error>
```
Both the migration CLI AND the un-soft-delete API (Design decision
7) call this helper — same collision-resolution behavior by
construction, no drift risk between the two paths.
Bounded at 10,000 as a safety cap. The probability of reaching
that in a real deployment is negligible — it would require ~10 K
distinct accounts all originally cased differently but sharing
the same lowercase form (a normal collision is 2-3 accounts, not
10 K). If the cap ever fires, something is very wrong with the
account universe and the migration ABORTs with a loud error
rather than silently truncating — the loud abort IS the
detection mechanism.
### 8. Test seed audit
Sweep-verified: existing test seeds all produce lowercase or NULL
usernames. Worth one more grep pass to ensure no test fixture INSERTs
`INSERT INTO auth.users … 'AliceTest'` — if any exist, lowercase them
in the same commit to avoid CI refuse-to-boot regressions.
Files verified (all safe):
- `src/infrastructure/repositories/pg/user_pg_repository.rs:1786`
- `src/infrastructure/repositories/pg/opaque_pg_repository.rs:339`
(NULL)
- `src/application/services/auth_application_service.rs:4982` (NULL)
- `src/application/services/subject_group_service.rs:796` (NULL)
- `src/bin/load-seed.rs:414`, `:446` (`load_user_XXXX` — lowercase)
- `src/mount_it_support.rs:61` (`make_user(name)` — verify callers)
- `tests/common/init-test-schema.sh:40` (`ci-admin` — lowercase)
### 9. Cosmetic side-effects (worth noting in CHANGELOG, non-blocking)
- `src/interfaces/nextcloud/avatar_handler.rs:283` — `pick_color`
derives a deterministic tile color from username bytes. Users whose
canonical username had uppercase letters will get a different
fallback-avatar tile color after the migration. One-time cosmetic
change.
- **NC desktop may perform a one-time re-sync** — see Deliverable 4.
### 10. Documentation
Release notes / CHANGELOG entry is NOT part of this PR — the
canonical repo's maintainer handles release notes at version-bump
time. This PR just leaves the notes-worthy items enumerated here
so the maintainer has the bullets to pick from when the next
version ships:
- Server auto-lowercases non-colliding mixed-case usernames at
first boot. No ops action needed for the common case.
- On `LOWER(username)` collision (`Alice` + `alice` both active),
the server refuses to boot; ops runs `oxicloud migrate
lowercase-usernames`. Exact CLI command shown in the refusal.
- Nextcloud desktop clients will prompt for a one-time re-sync on
first PROPFIND after upgrade. Files are ETag-verified, not
re-uploaded. DAVX5 and NC mobile handle the URL case change
silently. **No forced client upgrade or reconfiguration** —
server accepts uppercase URL segments indefinitely.
- Optional pre-emption for non-technical users: ops can manually
update the account URL to lowercase in each NC desktop client
before upgrading, avoiding the re-sync prompt entirely.
- Usernames become lowercase in ALL UI display surfaces (share
dialogs, activity feeds, admin panels, PROPFIND response
bodies, notification bell). Login identity unchanged from the
user's POV (they can still type any case at the login form).
- Avatar fallback color may change for users with previously-
uppercase usernames.
- Note: `display_name` is a possible follow-up if users miss
capitalisation for display — deferred pending demand signal, no
compat cost to adding later.
The two docs that DO ship with this PR:
- `docs/config/env.md` — note the boot-time check + migration command.
- `docs/install/binary.md` — upgrade-from-case-sensitive section.
### 11. Test coverage
- **Unit**: `validate_username("Alice")` returns `Ok("alice")`;
`validate_username("alice-")` returns `Err(...)` unchanged;
`validate_username(" Alice ")` returns `Ok("alice")`.
- **Unit**: `format_refusal_message_collisions` — 1 group renders
canonical + members + CLI; > 10 groups renders overflow tail;
total-affected-count sums across groups.
- **Hurl** (`tests/api/lowercase_usernames.hurl`, new): register a
user with `MixedCase`, assert DB stores `mixedcase`; log in with
`MIXEDCASE` and `mixedcase` — both succeed; rename to `NewName`,
assert `newname` stored; NC Basic Auth accepts `MixedCase:pass`,
`MIXEDCASE:pass`, `mixedcase:pass`.
- **Manual** (against dev DB, not CI):
- Auto-rename path: `UPDATE auth.users SET username='Alice' WHERE
username='alice'` (no collision); boot server → verify audit log
line + WARN summary, row is `alice` after boot, service starts.
- Collision path: `INSERT INTO auth.users … 'Alice'` on top of
existing `alice`; boot server → verify refusal message names both
rows + exact CLI shown, server exits non-zero.
- `oxicloud migrate lowercase-usernames --dry-run` → verify report
of collision + tiebreak decision
- `oxicloud migrate lowercase-usernames` → verify apply, one row
keeps `alice`, other gets `alice-2`
- Boot again → succeeds (Clean outcome)
- `curl -u ALICE:pass https://oxicloud/remote.php/dav/files/ALICE/…`
→ succeeds (accepts uppercase input, resolves to lowercase user)
## Cache-and-consistency observations (informational)
- `src/infrastructure/services/login_lockout_service.rs:33,68-89` —
already lowercases the key at line 58. No code change; comment
becomes factual not incidental.
- `src/application/services/app_password_service.rs:89,317-323` —
BLAKE3-keyed cache using the raw wire username. Post-normalization,
both sides normalize consistently → cache stays coherent. 300 s TTL
self-heals any transitional window.
- `NC_CHROOT_CACHE` in `basic_auth_middleware.rs:34-40` — keyed on
`Uuid`, not username. Unaffected.
## Critical files
Full enumeration in the Deliverables sections above. Grouped summary:
**Ingest normalizer:**
- `src/domain/entities/user.rs` (signature change + callers)
**Application services (write callers):**
- `src/application/services/auth_application_service.rs`
- `src/application/services/magic_link_invite_service.rs`
**Repositories (lookup normalization):**
- `src/infrastructure/repositories/pg/user_pg_repository.rs`
- `src/application/services/storage_usage_service.rs`
- `src/cli/opaque.rs`
**NextCloud DAV surface:**
- `src/interfaces/nextcloud/basic_auth_middleware.rs`
- `src/interfaces/nextcloud/session.rs`
- `src/infrastructure/services/nextcloud_chunked_upload_service.rs`
**New files:**
- `src/common/username_migration.rs`
- `tests/api/lowercase_usernames.hurl`
**Main entry:**
- `src/main.rs` (call verifier after `sqlx::migrate!()`)
**Migration CLI:**
- `src/cli/migrate.rs`
**Docs:**
- `CHANGELOG.md`
- `docs/config/env.md`
- `docs/install/binary.md`
## Delivery order
1. Change `validate_username` signature to return `Result<String>` —
one file.
2. Fix OIDC JIT derivation
(`auth_application_service.rs:4649-4690`) to lowercase before
passing to `User::new` — explicit change beyond the entity
normalizer's compile-time catches.
3. Iterate on compile errors — the return-type change catches every
downstream write-site.
4. Update repository lookup methods (`user_pg_repository.rs`,
`storage_usage_service.rs`, `cli/opaque.rs`) to internally
lowercase input before `.bind()`.
5. Update NC `basic_auth_middleware.rs` (lowercase `raw_username` at
decode) + `session.rs::extract_url_user` (lowercase return).
6. Add the boot-time verification helper (`src/common/username_migration.rs`)
+ wire into `main.rs`.
7. Extend `oxicloud migrate` with `lowercase-usernames [--dry-run]` —
DB pass + chunked-upload directory rename.
8. Test seed audit (grep pass).
9. Add hurl coverage.
10. Admin docs update (`docs/config/env.md` boot-check subsection +
`docs/install/binary.md` upgrade section). CHANGELOG is Dio's
job at version-bump time — not part of this PR.
11. Manual smoke test against dev DB.
12. PR to canonical.
## Total scope estimate
~6-8 hours of careful work. Larger than the initial estimate because
of these sweep-surfaced additions:
- OIDC JIT explicit fix (small).
- Chunked-upload directory rename step in the migration (~30 min).
- Test-seed audit (~15 min).
- More lookup callsites than initially thought.
The shape is uniform (`to_ascii_lowercase()` at every touchpoint) and
the compiler catches missed entity-write sites via the
`Result<String>` signature change. The parts NOT caught by the
compiler (OIDC JIT, lookup normalizers, NC URL segment, chunked-upload
directory) are the ones needing careful review — enumerated above.
## References
- Issue: [#691](https://github.com/AtalayaLabs/OxiCloud/issues/691)
- Related feature restrictions today:
- `validate_username` at
`src/domain/entities/user.rs:884-916`
- `@`-disjoint dispatch at
`src/application/services/auth_application_service.rs:1018`
- Related project docs:
- `docs/plan/auth-simplification.md` — the broader auth surface this
fits within
- Prior similar migration:
`oxicloud migrate nfc-filenames` in `src/cli/migrate.rs`
+15
View File
@@ -15,6 +15,15 @@ Non-obvious rules that trip up new code. Terse on purpose.
- Any new endpoint that mints or consumes credentials/tokens must consult one of the `is_*_login_allowed()` helpers, not the raw allowlist. - Any new endpoint that mints or consumes credentials/tokens must consult one of the `is_*_login_allowed()` helpers, not the raw allowlist.
- Any new "policy-disabled" refusal must emit an `audit`-target line before returning — matches `auth.login_rejected`, `magic_link.redemption_rejected` conventions. - Any new "policy-disabled" refusal must emit an `audit`-target line before returning — matches `auth.login_rejected`, `magic_link.redemption_rejected` conventions.
## AuthZ enforcement points
- **The extractors are NOT a choke point.** Four paths authenticate without ever touching `AuthUser` / `CurrentUserId`: `middleware/admin.rs::require_authenticated` (re-parses the JWT from header *or* cookie itself), the three DAV handlers' hand-rolled `extract_user` (`webdav_handler.rs:154`, `caldav_handler.rs:421`, `carddav_handler.rs:173`), `POST /api/auth/refresh` (mounted outside `auth_middleware`, `main.rs:775`), and `GET /api/rt/ws` (self-auths from a raw Bearer and never reads `claims.role`, `rt_ws.rs:240-255`). A rule added to an extractor does not hold until it is added to these too.
- **Never hand-roll principal extraction.** `req.extensions().get::<Arc<CurrentUser>>()` inside a handler is exactly the anti-pattern above — it bypasses every `FromRequestParts` guard. Take the extractor, or call the shared assertion.
- **A method check is not an authorization check.** GETs that mint credentials or write exist: `GET /api/wopi/editor-url` returns a WOPI token usable for `POST /wopi/files/{id}/contents`; `GET /api/s/{token}` writes via `register_shared_link_access`; `GET /api/auth/device/verify` is an oracle on live device codes; `GET /api/batch/download` builds an arbitrary ZIP from a querystring. Never gate on verb alone.
- **An auth helper's `_ =>` arm must DENY.** Two fail-open gates exist and are bugs, not patterns to copy: `require_internal_user` (`middleware/user.rs:64-71`) admits the caller on *any* `get_user_flags` error, and `decide_live_role` (`:169-176`) resurrects the claim role on a transient DB error. The first is the only middleware guarding all three DAV surfaces.
- **Prefer deny-by-default over assert-in-handler.** A restriction enforced inside the extractor covers ~200 call sites with no edits; the same restriction as "handler takes an optional principal and asserts" is one forgotten call away from silently accepting. `OptionalUserId` (`middleware/auth.rs:85-98`) is the cautionary tale — it exists, it is dead code, and nothing ever used it.
- Anonymous-session direction (share links as a principal): `docs/plan/rationalize-publicshare.md`.
## Storage backend access ## Storage backend access
- **Read blob content through `Arc<DedupService>`.** It's the ONE canonical read abstraction — CDC-manifest-aware (`file.blob_hash` may reference a chunk manifest, not a blob), backend-agnostic (Local/S3/Azure), wrapper-transparent (encryption/retry/cache). Never take `Arc<dyn BlobStorageBackend>` directly in a service that reads content; you'll silently break on any file ≥ 64 KiB (`CDC_MIN_CHUNK`). Follow `thumbnail_service`, `audio_metadata_service`, `media_metadata_service`, `face_indexing_service`, `search_index::content_index_worker` as reference impls. - **Read blob content through `Arc<DedupService>`.** It's the ONE canonical read abstraction — CDC-manifest-aware (`file.blob_hash` may reference a chunk manifest, not a blob), backend-agnostic (Local/S3/Azure), wrapper-transparent (encryption/retry/cache). Never take `Arc<dyn BlobStorageBackend>` directly in a service that reads content; you'll silently break on any file ≥ 64 KiB (`CDC_MIN_CHUNK`). Follow `thumbnail_service`, `audio_metadata_service`, `media_metadata_service`, `face_indexing_service`, `search_index::content_index_worker` as reference impls.
@@ -30,3 +39,9 @@ Non-obvious rules that trip up new code. Terse on purpose.
- **After adding: `cargo run --bin generate-openapi`** to regenerate `resources/gen/openapi.json`, then `git diff resources/gen/openapi.json` — the new path + its request/response schemas must be present. Zero-diff means you missed the registration. - **After adding: `cargo run --bin generate-openapi`** to regenerate `resources/gen/openapi.json`, then `git diff resources/gen/openapi.json` — the new path + its request/response schemas must be present. Zero-diff means you missed the registration.
- Sanity check for the whole surface: `diff <(grep -oE 'path = "/api[^"]+"' src/interfaces/api/handlers/*.rs | grep -oE '/api[^"]+' | sort -u) <(jq -r '.paths | keys | .[]' resources/gen/openapi.json | sort -u)` — should always be empty. Non-empty diff = drift. - Sanity check for the whole surface: `diff <(grep -oE 'path = "/api[^"]+"' src/interfaces/api/handlers/*.rs | grep -oE '/api[^"]+' | sort -u) <(jq -r '.paths | keys | .[]' resources/gen/openapi.json | sort -u)` — should always be empty. Non-empty diff = drift.
- Handlers referenced by the `paths(...)` list MUST be `pub` (module-visible from the paths list). Private `async fn` compiles at the router mount but breaks the paths list with a visibility error — see `get_smtp_info`, `send_smtp_test`, `get_user_profile` for the retrofit. - Handlers referenced by the `paths(...)` list MUST be `pub` (module-visible from the paths list). Private `async fn` compiles at the router mount but breaks the paths list with a visibility error — see `get_smtp_info`, `send_smtp_test`, `get_user_profile` for the retrofit.
### Security / scope in the spec
- **`security(("bearerAuth" = []))` — the empty array is the SCOPES list**, not decoration. Every route currently declares the same thing, so the spec claims "a session is required" for `GET /api/version` and `PUT /api/admin/users/{id}/role` alike: true, and useless. If a route's gate differs from the default, declare it there.
- OpenAPI has **no field for a minimum role** — OAuth2 has no role concept, so the spec has nowhere to put one. Use a pseudo-scope (`["role:admin"]`) rather than a vendor extension no tooling renders.
- **Declaring is not enforcing.** utoipa's `security` wires nothing, so it drifts from the real gate silently. Any scope worth declaring is worth a test cross-checking it against the actual mount — otherwise the spec becomes a parallel description of the authorization boundary rather than a picture of it.
@@ -3425,34 +3425,37 @@ impl AuthApplicationService {
&self, &self,
dto: crate::application::dtos::settings_dto::AdminCreateUserDto, dto: crate::application::dtos::settings_dto::AdminCreateUserDto,
) -> Result<FullUserDto, DomainError> { ) -> Result<FullUserDto, DomainError> {
// Validate username length // Normalise the username up-front — trim + lowercase — and use
if dto.username.len() < 3 || dto.username.len() > 254 { // the canonical form for every downstream check + generated
return Err(DomainError::new( // value below. `User::new` also normalises internally, but the
ErrorKind::InvalidInput, // placeholder-email fallback and the duplicate-check error
"User", // message live above that call, so they'd otherwise capture the
"Username must be between 3 and 254 characters".to_string(), // raw wire input (e.g. `UpperCase@oxicloud.local`).
)); // See docs/plan/username-lowercase.md § Design decision 5.
} let username = User::validate_username(&dto.username).map_err(|e| {
DomainError::new(ErrorKind::InvalidInput, "User", format!("Username: {e}"))
})?;
// Check for duplicate username // Check for duplicate username
if self if self
.user_storage .user_storage
.get_user_by_username(&dto.username) .get_user_by_username(&username)
.await .await
.is_ok() .is_ok()
{ {
return Err(DomainError::new( return Err(DomainError::new(
ErrorKind::AlreadyExists, ErrorKind::AlreadyExists,
"User", "User",
format!("User '{}' already exists", dto.username), format!("User '{username}' already exists"),
)); ));
} }
// Email: use provided or generate placeholder // Email: use provided or generate placeholder from the
// canonical (lowercase) username.
let email = dto let email = dto
.email .email
.filter(|e| !e.trim().is_empty()) .filter(|e| !e.trim().is_empty())
.unwrap_or_else(|| format!("{}@oxicloud.local", dto.username)); .unwrap_or_else(|| format!("{username}@oxicloud.local"));
// Check email uniqueness // Check email uniqueness
if self.user_storage.get_user_by_email(&email).await.is_ok() { if self.user_storage.get_user_by_email(&email).await.is_ok() {
@@ -3519,7 +3522,7 @@ impl AuthApplicationService {
let user = if is_external { let user = if is_external {
User::new( User::new(
email, email,
Some(dto.username.clone()), Some(username.clone()),
Some(password_hash), Some(password_hash),
None, // federation_kind: admin-created external, no federation link yet None, // federation_kind: admin-created external, no federation link yet
None, // federation_issuer None, // federation_issuer
@@ -3531,7 +3534,7 @@ impl AuthApplicationService {
} else { } else {
User::new( User::new(
email, email,
Some(dto.username.clone()), Some(username.clone()),
Some(password_hash), Some(password_hash),
None, // federation_kind: admin-created local user None, // federation_kind: admin-created local user
None, // federation_issuer None, // federation_issuer
@@ -3750,8 +3753,33 @@ impl AuthApplicationService {
Ok(()) Ok(())
} }
/// Activate or deactivate a user (admin only) /// Activate or deactivate a user (admin only).
///
/// **Reactivation collision handling** — when a deactivated account
/// holds a mixed-case username from before the lowercase-usernames
/// migration (its row was skipped by that migration precisely
/// because it was deactivated), reactivating it can produce a
/// username collision if `LOWER(other.username) == LOWER(this.username)`
/// for an active row. We resolve the collision by:
///
/// 1. Re-normalising via `User::set_username` (returns the
/// canonical lowercase form on success).
/// 2. If the canonical form is already taken by another active
/// row, probe `<lower>-2`, `-3`, … via the shared
/// `find_free_username_suffix` helper.
/// 3. Persist the resolved name BEFORE flipping `active = true`
/// so no time window has two-active-users with the same
/// lowercase form.
///
/// See `docs/plan/username-lowercase.md § Design decision 7`.
/// Without this, un-soft-deleting the only pre-migration
/// mixed-case survivor would refuse-to-boot on the next restart.
pub async fn set_user_active(&self, user_id: Uuid, active: bool) -> Result<(), DomainError> { pub async fn set_user_active(&self, user_id: Uuid, active: bool) -> Result<(), DomainError> {
// Only the activate direction needs the collision-resolution
// dance — deactivation just flips a bit.
if active {
self.resolve_reactivation_collision(user_id).await?;
}
self.user_storage self.user_storage
.set_user_active_status(user_id, active) .set_user_active_status(user_id, active)
.await?; .await?;
@@ -3759,6 +3787,93 @@ impl AuthApplicationService {
Ok(()) Ok(())
} }
/// Pre-flight for `set_user_active(active = true)`: ensures the
/// target user's username is canonical (lowercase) AND unique
/// against currently-active accounts. Renames the target row if
/// either invariant would break.
///
/// NULL usernames (OPAQUE-migrated accounts) are a no-op — nothing
/// to normalise, nothing to collide.
async fn resolve_reactivation_collision(&self, user_id: Uuid) -> Result<(), DomainError> {
let target = self.user_storage.get_user_by_id(user_id).await?;
let Some(current) = target.username().map(str::to_string) else {
return Ok(());
};
let canonical = current.to_ascii_lowercase();
// Look for another ACTIVE user holding the canonical form.
// The migration CLI's `find_free_username_suffix` probes
// directly via a pool; here we don't have the pool
// (`AuthApplicationService` holds a `dyn UserRepository`
// trait object). Use `get_user_by_username` — the repo
// normalises input to lowercase before bind, so this
// resolves against the canonical row.
let collision = self
.user_storage
.get_user_by_username(&canonical)
.await
.ok()
.filter(|other| other.id() != user_id && other.is_active());
let chosen_name = match collision {
None => canonical,
Some(_) => {
// Collision — probe `<canonical>-2`, `-3`, … via
// repository lookups. Same shape as the migration
// CLI's `find_free_username_suffix`, just against
// the repo trait instead of a raw pool. Both paths
// agree by construction on the numbering scheme.
//
// Cap at 10_000 (matches the shared helper's cap —
// see `docs/plan/username-lowercase.md § 3. Suffix-
// collision robustness`). Reaching the cap means
// the account universe has an anomaly worth
// investigating; loud abort beats silent truncation.
const SUFFIX_PROBE_CAP: i32 = 10_000;
let mut chosen: Option<String> = None;
for n in 2..=SUFFIX_PROBE_CAP {
let candidate = format!("{canonical}-{n}");
match self.user_storage.get_user_by_username(&candidate).await {
Ok(_) => continue,
Err(_) => {
chosen = Some(candidate);
break;
}
}
}
let suffixed = chosen.ok_or_else(|| {
DomainError::internal_error(
"User",
format!(
"reactivation-collision suffix probe exhausted \
{SUFFIX_PROBE_CAP} candidates for base '{canonical}'"
),
)
})?;
tracing::info!(
target: "audit",
event = "user.reactivation_renamed",
reason = "collision_with_active",
target_id = %user_id,
from = %current,
to = %suffixed,
"🔄 user reactivation renamed to avoid username collision",
);
suffixed
}
};
if target.username() != Some(chosen_name.as_str()) {
let mut renamed = target;
renamed
.set_username(chosen_name)
.map_err(|e| DomainError::internal_error("User", format!("set_username: {e}")))?;
self.user_storage.update_user(renamed).await?;
}
Ok(())
}
/// Change user role (admin only). /// Change user role (admin only).
/// ///
/// Refuses `role = "admin"` when the target is external (grant-only). /// Refuses `role = "admin"` when the target is external (grant-only).
@@ -4435,10 +4550,14 @@ impl AuthApplicationService {
.clone() .clone()
.or(claims.name.clone()) .or(claims.name.clone())
.unwrap_or_else(|| format!("oidc_{}", &claims.sub[..8.min(claims.sub.len())])); .unwrap_or_else(|| format!("oidc_{}", &claims.sub[..8.min(claims.sub.len())]));
// Placeholder-email fallback when the IdP omits `email` from
// the claim set. Lowercase the local-part so the fake address
// matches the storage convention for other placeholder-email
// paths (see `admin_create_user`'s `<username>@oxicloud.local`).
let oidc_email = claims let oidc_email = claims
.email .email
.clone() .clone()
.unwrap_or_else(|| format!("{}@oidc.local", oidc_username)); .unwrap_or_else(|| format!("{}@oidc.local", oidc_username.to_ascii_lowercase()));
// 5. Look up existing user by OIDC subject. // 5. Look up existing user by OIDC subject.
// //
@@ -4652,16 +4771,30 @@ impl AuthApplicationService {
&oidc_username &oidc_username
}; };
// Filter to valid username characters only, then truncate to 32 chars // Lowercase at JIT derivation. `validate_username` in
// `User::new` would lowercase too, but the collision
// check below (`get_user_by_username`) needs the
// canonical form BEFORE `User::new` is called —
// otherwise `Alice` from an IdP claim would look
// "free" against an `alice` row on the first pass
// and fail the DB unique constraint at INSERT time.
// See `docs/plan/username-lowercase.md § 2. OIDC JIT
// derivation`.
//
// ASCII-only by the char-filter below, so
// `to_ascii_lowercase()` is deterministic and
// locale-safe.
let mut username = base_username let mut username = base_username
.chars() .chars()
.filter(|c| { .filter(|c| {
c.is_ascii_alphanumeric() || *c == '-' || *c == '_' || *c == '.' c.is_ascii_alphanumeric() || *c == '-' || *c == '_' || *c == '.'
}) })
.take(32) .take(32)
.collect::<String>(); .collect::<String>()
.to_ascii_lowercase();
// Filter helper: removes any chars that are not valid in a username // Filter helper: removes any chars that are not valid in a username.
// Lowercases too so the collision-suffix path below writes canonical form.
let filter_username_chars = |s: &str| { let filter_username_chars = |s: &str| {
s.chars() s.chars()
.filter(|c| { .filter(|c| {
@@ -4669,6 +4802,7 @@ impl AuthApplicationService {
}) })
.take(32) .take(32)
.collect::<String>() .collect::<String>()
.to_ascii_lowercase()
}; };
// Ensure minimum length (the padding suffix must also be filtered) // Ensure minimum length (the padding suffix must also be filtered)
@@ -125,10 +125,16 @@ impl StorageUsageService {
} }
/// Same as [`Self::update_user_storage_usage`], keyed by username. /// Same as [`Self::update_user_storage_usage`], keyed by username.
///
/// Lowercases input before bind — same rule as
/// `UserRepository::get_user_by_username`. Usernames are canonical
/// (lowercase) in the DB post-migration; callers may pass any case.
/// See `docs/plan/username-lowercase.md`.
pub async fn update_user_storage_usage_by_username( pub async fn update_user_storage_usage_by_username(
&self, &self,
username: &str, username: &str,
) -> Result<i64, DomainError> { ) -> Result<i64, DomainError> {
let username = username.trim().to_ascii_lowercase();
let total_usage: Option<i64> = sqlx::query_scalar( let total_usage: Option<i64> = sqlx::query_scalar(
r#" r#"
UPDATE auth.users u UPDATE auth.users u
@@ -146,7 +152,7 @@ impl StorageUsageService {
RETURNING u.storage_used_bytes RETURNING u.storage_used_bytes
"#, "#,
) )
.bind(username) .bind(&username)
.fetch_optional(self.pool.as_ref()) .fetch_optional(self.pool.as_ref())
.await .await
.map_err(|e| { .map_err(|e| {
+314
View File
@@ -82,11 +82,35 @@ pub enum Action {
#[arg(long)] #[arg(long)]
dry_run: bool, dry_run: bool,
}, },
/// Lowercase every active user's username in `auth.users`.
///
/// Enforcement-companion for the case-insensitive-usernames
/// migration (see `docs/plan/username-lowercase.md`). The server
/// refuses to boot after upgrade until this has run. Data touched
/// is `auth.users.username` only.
///
/// Collision handling: when `Alice` and `alice` both exist,
/// tiebreak `(last_login_at DESC NULLS LAST, created_at ASC)` —
/// the winner keeps the canonical lowercased name, losers get
/// `<lowercase>-2`, `-3`, … via the shared
/// [`common::username_migration::find_free_username_suffix`]
/// probe. Sessions and grants survive the rename (they key on
/// `user_id`).
///
/// Skipped: soft-deleted / disabled rows and rows where
/// `username IS NULL` (OPAQUE-migrated accounts).
LowercaseUsernames {
/// Print what would change without touching the DB.
#[arg(long)]
dry_run: bool,
},
} }
pub async fn run(action: Action) -> u8 { pub async fn run(action: Action) -> u8 {
match action { match action {
Action::NfcFilenames { dry_run } => run_nfc_filenames(dry_run).await, Action::NfcFilenames { dry_run } => run_nfc_filenames(dry_run).await,
Action::LowercaseUsernames { dry_run } => run_lowercase_usernames(dry_run).await,
} }
} }
@@ -758,3 +782,293 @@ async fn run_folders(pool: &PgPool, dry_run: bool, stats: &mut Stats) -> Result<
} }
Ok(()) Ok(())
} }
// ════════════════════════════════════════════════════════════════════════════
// lowercase-usernames
// ════════════════════════════════════════════════════════════════════════════
#[derive(Default)]
struct UsernameStats {
scanned: u64,
already_lowercase: u64,
normalized_in_place: u64,
/// Multi-member `LOWER(username)` group where the tiebreak
/// winner kept the canonical name.
collision_winners: u64,
/// Multi-member losers renamed to `<lowercase>-N`.
renamed_to_suffix: u64,
/// Rows the scan touched but the loop declined to modify. Today
/// this is inactive rows (soft-deleted / admin-disabled). NULL
/// usernames never enter the scan so they don't contribute here.
skipped: u64,
}
#[derive(Debug)]
struct UsernameRow {
id: Uuid,
username: String,
/// Inactive rows (soft-deleted / admin-disabled) are read but not
/// modified — normalising a name we can't reach anyway risks
/// creating a `<lower>-N` conflict with a future re-activation of
/// the same handle. The loop uses this flag to skip and count.
active: bool,
// Kept for the SQL row-shape roundtrip (the SELECT ordering
// depends on them) even though the Rust-side grouping only
// reads `id` and `username`. Marked `#[allow(dead_code)]`
// so clippy doesn't nag; renaming to `_last_login_at` would
// work too but the SQL column names are load-bearing for the
// sqlx `Row::get` calls below.
#[allow(dead_code)]
last_login_at: Option<DateTime<Utc>>,
#[allow(dead_code)]
created_at: DateTime<Utc>,
}
async fn run_lowercase_usernames(dry_run: bool) -> u8 {
let database_url = match env::var("DATABASE_URL") {
Ok(v) => v,
Err(_) => {
eprintln!("migrate lowercase-usernames: DATABASE_URL not set");
return 2;
}
};
let pool = match PgPool::connect(&database_url).await {
Ok(p) => p,
Err(e) => {
eprintln!("migrate lowercase-usernames: failed to connect: {e}");
return 2;
}
};
if dry_run {
println!("migrate lowercase-usernames: DRY RUN (no writes)");
} else {
println!("migrate lowercase-usernames: applying changes");
}
// Load every user with a non-NULL username, active and inactive
// alike. Inactive rows are surfaced (not filtered at scan-time)
// so the `skipped (inactive)` counter can report them honestly —
// an operator reading the summary sees "10 rows scanned, 2
// inactive were passed on" instead of a phantom 0.
//
// NULL usernames stay out of the scan: there's nothing to
// normalise for OPAQUE-migrated rows, and pulling them would
// inflate `scanned` with rows the migration has no verb for.
//
// Ordering: alphabetic by `LOWER(username)` groups collisions
// together, then the intra-group order is the tiebreak
// (`last_login_at DESC NULLS LAST, created_at ASC` — most
// recently active wins the canonical name).
let rows: Vec<UsernameRow> = match sqlx::query(
r#"
SELECT id, username, active, last_login_at, created_at
FROM auth.users
WHERE username IS NOT NULL
ORDER BY LOWER(username),
(last_login_at IS NULL),
last_login_at DESC NULLS LAST,
created_at ASC
"#,
)
.fetch_all(&pool)
.await
{
Ok(rs) => rs
.into_iter()
.map(|r| UsernameRow {
id: r.get::<Uuid, _>("id"),
username: r.get::<String, _>("username"),
active: r.get::<bool, _>("active"),
last_login_at: r.try_get::<DateTime<Utc>, _>("last_login_at").ok(),
created_at: r.get::<DateTime<Utc>, _>("created_at"),
})
.collect(),
Err(e) => {
eprintln!("migrate lowercase-usernames: initial scan failed: {e}");
return 2;
}
};
let mut stats = UsernameStats::default();
// Group by `LOWER(username)`. Order preserved from the SQL query
// → within a group, the FIRST row is the tiebreak winner.
//
// Inactive rows are filtered OUT of the grouping (not just
// skipped inside the loop) so they can't create a phantom
// collision with an active row sharing their lowercase form.
// Example: inactive `Alice` + active `alice` would otherwise
// look like a two-member group; filtering inactive first leaves
// `alice` as a clean singleton no-op. The count goes to
// `stats.skipped`, surfaced in the summary as `skipped (inactive)`.
let mut groups: Vec<(String, Vec<UsernameRow>)> = Vec::new();
for row in rows {
stats.scanned += 1;
if !row.active {
stats.skipped += 1;
continue;
}
let key = row.username.to_ascii_lowercase();
match groups.last_mut() {
Some((k, v)) if k == &key => v.push(row),
_ => groups.push((key, vec![row])),
}
}
for (lower, members) in groups {
if members.len() == 1 {
let row = &members[0];
if row.username == lower {
stats.already_lowercase += 1;
continue;
}
// Single-member group with a mixed-case name → straight
// rename to the lowercase form. No collision.
if !dry_run && let Err(e) = update_username(&pool, row.id, &lower).await {
eprintln!(
"migrate lowercase-usernames: UPDATE failed for {}: {e}",
row.id
);
return 1;
}
println!(
"NORMALIZE user={} '{}' → '{}'",
row.id, row.username, lower
);
stats.normalized_in_place += 1;
continue;
}
// Multi-member group → collision. Members are already ordered
// by the tiebreak. Winner takes the canonical lowercase name,
// losers get `<lower>-2`, `-3`, … via the shared suffix helper.
//
// ORDER MATTERS: losers must be renamed FIRST. If we renamed
// the winner to `<lower>` while a loser still holds that
// exact name (the common case where the winner is mixed-case
// and the loser is already-lowercase), the UNIQUE constraint
// `users_username_key` fires. Freeing the canonical form by
// suffixing every non-winner member first eliminates the
// race entirely.
let (winner, losers) = members.split_first().expect("non-empty by construction");
// Suffixes assigned inside this group during this run. Used
// to keep dry-run consistent (no DB writes → the shared
// suffix helper would hand the same probe back for every
// loser). During apply, the DB itself deduplicates, but
// tracking here keeps the two modes structurally identical.
let mut reserved_this_group: Vec<String> = Vec::new();
for loser in losers {
let suffixed =
match pick_free_suffix_avoiding(&pool, &lower, &reserved_this_group).await {
Ok(s) => s,
Err(e) => {
eprintln!(
"migrate lowercase-usernames: suffix probe failed for {}: {e}",
loser.id
);
return 1;
}
};
if !dry_run && let Err(e) = update_username(&pool, loser.id, &suffixed).await {
eprintln!(
"migrate lowercase-usernames: loser UPDATE failed for {}: {e}",
loser.id
);
return 1;
}
println!(
"RENAME user={} '{}' → '{}' (collision suffix)",
loser.id, loser.username, suffixed
);
stats.renamed_to_suffix += 1;
reserved_this_group.push(suffixed);
}
if winner.username == lower {
// Winner already holds the canonical name (a lowercase
// row happened to be the most recently active; other
// members are the ones needing renames).
stats.already_lowercase += 1;
} else {
if !dry_run && let Err(e) = update_username(&pool, winner.id, &lower).await {
eprintln!(
"migrate lowercase-usernames: winner UPDATE failed for {}: {e}",
winner.id
);
return 1;
}
println!(
"NORMALIZE user={} '{}' → '{}' (collision winner)",
winner.id, winner.username, lower
);
stats.collision_winners += 1;
}
}
println!();
println!("Summary:");
println!(" scanned: {}", stats.scanned);
println!(" already-lowercase: {}", stats.already_lowercase);
println!(" normalized in place: {}", stats.normalized_in_place);
println!(" collision winners: {}", stats.collision_winners);
println!(" renamed to suffix: {}", stats.renamed_to_suffix);
println!(" skipped (inactive): {}", stats.skipped);
if dry_run
&& (stats.normalized_in_place + stats.collision_winners + stats.renamed_to_suffix) > 0
{
println!();
println!("(dry-run — re-run without --dry-run to apply)");
}
0
}
async fn update_username(pool: &PgPool, id: Uuid, new_name: &str) -> Result<(), sqlx::Error> {
sqlx::query("UPDATE auth.users SET username = $1, updated_at = NOW() WHERE id = $2")
.bind(new_name)
.bind(id)
.execute(pool)
.await?;
Ok(())
}
/// Pick the next free `<base>-<N>` suffix, skipping any suffix already
/// reserved earlier in this migration run.
///
/// Wraps [`crate::common::username_migration::find_free_username_suffix`]
/// with an additional local guard: in dry-run mode, no UPDATEs land so
/// the DB probe would return the same suffix for every loser in a
/// multi-loser group. The `reserved` slice lets the caller feed back
/// the suffixes it has already announced, and the probe steps past
/// them. In apply mode the DB probe alone would be enough (each real
/// UPDATE moves the state forward), but the same code path keeps the
/// two modes structurally identical.
async fn pick_free_suffix_avoiding(
pool: &PgPool,
base: &str,
reserved: &[String],
) -> Result<String, sqlx::Error> {
// Try the shared helper's default candidate first; if it collides
// with a same-run reservation, increment past it and probe again.
let mut n = 2;
loop {
let candidate = format!("{base}-{n}");
let db_taken: (bool,) =
sqlx::query_as("SELECT EXISTS(SELECT 1 FROM auth.users WHERE username = $1)")
.bind(&candidate)
.fetch_one(pool)
.await?;
let locally_taken = reserved.iter().any(|s| s == &candidate);
if !db_taken.0 && !locally_taken {
return Ok(candidate);
}
n += 1;
if n > 10_000 {
// Same cap as the shared helper — a loud panic beats a
// silent truncation for an anomaly this rare.
panic!("pick_free_suffix_avoiding: exhausted 10000 suffix probes for base '{base}'",);
}
}
}
+17 -4
View File
@@ -128,8 +128,15 @@ async fn run_reset(user: Option<String>, all: bool, dry_run: bool) -> u8 {
let rows_result = if all { let rows_result = if all {
sqlx::query(select_sql).fetch_all(&pool).await sqlx::query(select_sql).fetch_all(&pool).await
} else { } else {
let ident = user.as_deref().unwrap(); // Normalise before bind so the CLI accepts any case for
sqlx::query(select_sql).bind(ident).fetch_all(&pool).await // the username branch (email is already case-insensitive
// via a functional index on LOWER(email); lowercasing here
// for both branches is harmless — emails are lowercase
// ASCII in `auth.users.email` too).
// See `docs/plan/username-lowercase.md § 3. Lookup normalization`.
let ident_raw = user.as_deref().unwrap();
let ident = ident_raw.trim().to_ascii_lowercase();
sqlx::query(select_sql).bind(&ident).fetch_all(&pool).await
}; };
let rows = match rows_result { let rows = match rows_result {
Ok(r) => r, Ok(r) => r,
@@ -204,8 +211,14 @@ async fn run_reset(user: Option<String>, all: bool, dry_run: bool) -> u8 {
let write_result = if all { let write_result = if all {
sqlx::query(update_sql_all).execute(&pool).await sqlx::query(update_sql_all).execute(&pool).await
} else { } else {
let ident = user.as_deref().unwrap(); // Same normalisation as the read path above — usernames are
sqlx::query(update_sql_one).bind(ident).execute(&pool).await // canonical lowercase in the DB. See docs/plan/username-lowercase.md.
let ident_raw = user.as_deref().unwrap();
let ident = ident_raw.trim().to_ascii_lowercase();
sqlx::query(update_sql_one)
.bind(&ident)
.execute(&pool)
.await
}; };
let affected = match write_result { let affected = match write_result {
Ok(r) => r.rows_affected(), Ok(r) => r.rows_affected(),
+1
View File
@@ -8,3 +8,4 @@ pub mod mime_detect;
pub mod runtime; pub mod runtime;
pub mod stubs; pub mod stubs;
pub mod text; pub mod text;
pub mod username_migration;
+396
View File
@@ -0,0 +1,396 @@
//! Username-lowercase boot flow: verifier, auto-rename, shared collision helper.
//!
//! The plan (`docs/plan/username-lowercase.md`) makes usernames
//! case-insensitive by canonicalising to lowercase on ingest. Three
//! pieces of infrastructure live here:
//!
//! 1. [`verify_all_usernames_lowercase`] — a **read-only** check that
//! runs after `sqlx::migrate!()` at boot. Classifies every active
//! mixed-case row into one of three outcomes:
//!
//! - [`UsernameCaseCheck::Clean`] — nothing to do.
//! - [`UsernameCaseCheck::AutoRenamable`] — mixed-case rows exist
//! but each `LOWER(username)` form is unique in the active-user
//! set. Safe to lowercase in one atomic transaction; boot proceeds.
//! - [`UsernameCaseCheck::Collisions`] — at least one group has
//! two or more active rows sharing a `LOWER(username)` (e.g.
//! `Alice` + `alice`). Tiebreak requires human judgement; the
//! server refuses to start and prints the CLI command.
//!
//! Follows [[feedback_no_silent_auto_repair]] in spirit: silent
//! action is limited to cases where there is exactly one correct
//! move (rename the sole mixed-case row to its lowercase form).
//! Anywhere ambiguity exists (which of `Alice` and `alice` keeps
//! the canonical name?), boot refuses and defers to `oxicloud
//! migrate lowercase-usernames`.
//!
//! 2. [`apply_auto_renames`] — the one-transaction UPDATE loop that
//! performs the auto-rename path. Emits a structured audit line
//! per row (`user.username_lowercased_on_boot`). All-or-nothing:
//! a mid-tx failure aborts the transaction and boot fails, so the
//! DB is never left in a half-renamed state.
//!
//! 3. [`find_free_username_suffix`] — the shared collision-resolution
//! helper. Called by the migration CLI when it lowercases a name
//! that would clash with an existing row, AND by the un-soft-delete
//! API when it re-normalises a mixed-case account whose lowercase
//! form is now taken by someone else.
//!
//! `NULL` usernames (OPAQUE-migrated accounts) are always skipped — the
//! SQL `WHERE username <> LOWER(username)` predicate is NULL-safe by
//! semantics (`NULL <> anything` yields `NULL`, which `WHERE` excludes).
//! Soft-deleted / disabled accounts (`active = false`) are also skipped:
//! they can't serve traffic anyway.
use sqlx::{PgPool, Row};
/// One mixed-case account row. Used both for the auto-rename list and
/// for reporting collision-group members.
#[derive(Debug, Clone)]
pub struct MixedCaseAccount {
pub id: uuid::Uuid,
pub username: String,
pub last_login_at: Option<chrono::DateTime<chrono::Utc>>,
}
/// A `LOWER(username)` group with two or more active members. At least
/// one member is mixed-case (that's what made the group visible to the
/// verifier); the other member(s) may be already-lowercase (e.g.
/// `Alice` + `alice`) or also mixed-case (`Alice` + `ALICE`).
#[derive(Debug, Clone)]
pub struct CollisionGroup {
/// The lowercase form shared by every member.
pub canonical: String,
/// Members, ordered by the tiebreak that the migration CLI
/// applies: `last_login_at DESC NULLS LAST, created_at ASC`.
pub members: Vec<MixedCaseAccount>,
}
/// The three outcomes of the boot-time verifier.
#[derive(Debug, Clone)]
pub enum UsernameCaseCheck {
/// Every active username is already lowercase (or `NULL`). Boot
/// proceeds unmodified.
Clean,
/// Mixed-case rows exist, but each `LOWER(username)` form is
/// unique among active users. Safe to lowercase atomically at
/// boot; the caller runs [`apply_auto_renames`].
AutoRenamable(Vec<MixedCaseAccount>),
/// At least one `LOWER(username)` group has two or more active
/// members. Tiebreak requires human judgement; the caller formats
/// a refusal message via [`format_refusal_message_collisions`] and
/// aborts boot.
Collisions(Vec<CollisionGroup>),
}
/// Boot-time verifier. Runs AFTER `sqlx::migrate!()` and BEFORE
/// `AppState` is assembled. Read-only: never mutates `auth.users`.
///
/// Returns [`UsernameCaseCheck`] describing what (if anything) the
/// caller should do. Errors are limited to DB failures — semantic
/// outcomes are all `Ok(_)` variants.
pub async fn verify_all_usernames_lowercase(pool: &PgPool) -> Result<UsernameCaseCheck, String> {
// First pass: mixed-case rows that have NO other active row
// sharing their LOWER form. These are safe to auto-rename.
let auto_rows = sqlx::query(
r#"
SELECT u.id, u.username, u.last_login_at
FROM auth.users u
WHERE u.active = true
AND u.username <> LOWER(u.username)
AND NOT EXISTS (
SELECT 1
FROM auth.users u2
WHERE u2.active = true
AND u2.id <> u.id
AND LOWER(u2.username) = LOWER(u.username)
)
ORDER BY LOWER(u.username)
"#,
)
.fetch_all(pool)
.await
.map_err(|e| format!("username lowercase verifier: singleton query failed: {e}"))?;
// Second pass: every active row that belongs to a colliding
// group — a `LOWER(username)` shared by two or more active rows
// where at least one member is mixed-case. Result includes
// already-lowercase members so the refusal report shows the full
// context of each collision.
let collision_rows = sqlx::query(
r#"
WITH colliding_lowers AS (
SELECT LOWER(username) AS canonical
FROM auth.users
WHERE active = true
GROUP BY LOWER(username)
HAVING COUNT(*) > 1
AND SUM(CASE WHEN username <> LOWER(username) THEN 1 ELSE 0 END) >= 1
)
SELECT id, username, last_login_at, LOWER(username) AS canonical
FROM auth.users
WHERE active = true
AND LOWER(username) IN (SELECT canonical FROM colliding_lowers)
ORDER BY LOWER(username),
(last_login_at IS NULL),
last_login_at DESC NULLS LAST,
created_at ASC
"#,
)
.fetch_all(pool)
.await
.map_err(|e| format!("username lowercase verifier: collision query failed: {e}"))?;
if !collision_rows.is_empty() {
// Group by canonical. Rows are already ordered by canonical
// then by tiebreak, so a fold is enough.
let mut groups: Vec<CollisionGroup> = Vec::new();
for r in collision_rows {
let canonical: String = r.get("canonical");
let member = MixedCaseAccount {
id: r.get::<uuid::Uuid, _>("id"),
username: r.get::<String, _>("username"),
last_login_at: r
.try_get::<chrono::DateTime<chrono::Utc>, _>("last_login_at")
.ok(),
};
match groups.last_mut() {
Some(g) if g.canonical == canonical => g.members.push(member),
_ => groups.push(CollisionGroup {
canonical,
members: vec![member],
}),
}
}
return Ok(UsernameCaseCheck::Collisions(groups));
}
if auto_rows.is_empty() {
return Ok(UsernameCaseCheck::Clean);
}
let accounts = auto_rows
.into_iter()
.map(|r| MixedCaseAccount {
id: r.get::<uuid::Uuid, _>("id"),
username: r.get::<String, _>("username"),
last_login_at: r
.try_get::<chrono::DateTime<chrono::Utc>, _>("last_login_at")
.ok(),
})
.collect();
Ok(UsernameCaseCheck::AutoRenamable(accounts))
}
/// Apply the atomic auto-rename transaction. All UPDATEs succeed
/// together or all roll back — the DB is never left in a half-renamed
/// state. Each successful rename emits a structured audit line.
///
/// The `WHERE id = $1 AND username = $3` guard defends against a
/// concurrent rename between the SELECT and this UPDATE. If some
/// other process renamed the row in that window, the UPDATE affects
/// zero rows and we log a warning but do not fail the transaction —
/// the row is already lowercase (that's why the guard didn't match),
/// so the invariant still holds.
pub async fn apply_auto_renames(
pool: &PgPool,
accounts: &[MixedCaseAccount],
) -> Result<(), sqlx::Error> {
let mut tx = pool.begin().await?;
for acc in accounts {
let new_username = acc.username.to_ascii_lowercase();
let res = sqlx::query(
r#"
UPDATE auth.users
SET username = $2
WHERE id = $1
AND username = $3
"#,
)
.bind(acc.id)
.bind(&new_username)
.bind(&acc.username)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
target: "audit",
event = "user.username_lowercase_skipped_on_boot",
reason = "row_changed_between_verify_and_apply",
user_id = %acc.id,
expected_username = %acc.username,
"👮🏻‍♂️ skipped auto-lowercase: row was modified after verifier ran",
);
continue;
}
tracing::info!(
target: "audit",
event = "user.username_lowercased_on_boot",
reason = "unique_lowercase_group",
user_id = %acc.id,
old_username = %acc.username,
new_username = %new_username,
"👮🏻‍♂️ auto-lowercased username at boot",
);
}
tx.commit().await?;
Ok(())
}
/// Format the FATAL error string shown when boot refuses to proceed
/// because at least one `LOWER(username)` group has multiple active
/// members. Self-sufficient — an operator at 3 AM shouldn't need to
/// consult docs to know what to do.
pub fn format_refusal_message_collisions(groups: &[CollisionGroup]) -> String {
use std::fmt::Write;
let total_members: usize = groups.iter().map(|g| g.members.len()).sum();
let mut out = String::new();
let _ = write!(
&mut out,
"\nFATAL: cannot start — {} colliding username group(s) \
({} affected account(s) in total).\n\n\
Non-colliding mixed-case rows are auto-renamed at boot. \
These groups can't be resolved automatically because two or \
more active accounts share the same lowercase form, and only \
a human can decide who keeps the canonical name.\n\n\
Run the migration:\n\n \
oxicloud migrate lowercase-usernames --dry-run # preview the tiebreak\n \
oxicloud migrate lowercase-usernames # apply\n\n\
The tiebreak rule is `last_login_at DESC NULLS LAST, \
created_at ASC` — the most recently active member keeps the \
canonical lowercase name; the losers get `-2`, `-3`, … as a \
suffix. Sessions and grants survive the rename (they key on \
user_id, not username).\n\n\
Collision groups (up to 10 shown):\n",
groups.len(),
total_members
);
for g in groups.iter().take(10) {
let _ = writeln!(&mut out, "\n Canonical form: {}", g.canonical);
for m in &g.members {
let last = m
.last_login_at
.map(|t| t.format("%Y-%m-%d").to_string())
.unwrap_or_else(|| "never".to_string());
let _ = writeln!(
&mut out,
" {} (id: {} last_login: {})",
m.username, m.id, last
);
}
}
if groups.len() > 10 {
let _ = writeln!(
&mut out,
"\n ... and {} more group(s). Run --dry-run for the full list.",
groups.len() - 10
);
}
out
}
/// Cap on the suffix-probe loop. If we ever need `<base>-10000` there's
/// something very wrong with the account universe — collisions in the
/// wild are 2-3 accounts, not 10 K. The loud abort IS the detection.
/// See [`docs/plan/username-lowercase.md § 3. Suffix-collision robustness`].
const SUFFIX_PROBE_CAP: i32 = 10_000;
/// Find the next free `<base>-<N>` suffix for a colliding username.
///
/// Starts at `<base>-2` and increments until an unused suffix is
/// found. Robust against pre-existing rows already occupying some
/// suffixes (the probe steps past them).
///
/// Called by:
/// - The migration CLI when a `LOWER(username)` group has multiple
/// members and the tiebreak winner keeps the canonical name; the
/// losers get `<base>-2`, `-3`, … from this helper.
/// - The un-soft-delete API when re-normalising a mixed-case
/// account whose lowercase form is now taken by an active row.
///
/// Both callers reach for this single function so the two paths
/// agree by construction — no drift risk between the migration and
/// runtime un-soft-delete.
pub async fn find_free_username_suffix(pool: &PgPool, base: &str) -> Result<String, sqlx::Error> {
for n in 2..=SUFFIX_PROBE_CAP {
let candidate = format!("{base}-{n}");
let exists: (bool,) =
sqlx::query_as("SELECT EXISTS(SELECT 1 FROM auth.users WHERE username = $1)")
.bind(&candidate)
.fetch_one(pool)
.await?;
if !exists.0 {
return Ok(candidate);
}
}
// If we get here, something is very wrong. Loud panic beats
// silent truncation to whatever the caller's fallback is.
panic!(
"find_free_username_suffix: exhausted {SUFFIX_PROBE_CAP} suffix probes for base '{base}'; \
the account universe likely has an anomaly worth investigating"
);
}
#[cfg(test)]
mod tests {
use super::*;
fn acc(name: &str) -> MixedCaseAccount {
MixedCaseAccount {
id: uuid::Uuid::nil(),
username: name.into(),
last_login_at: None,
}
}
#[test]
fn refusal_message_lists_groups_and_cli() {
let groups = vec![CollisionGroup {
canonical: "alice".into(),
members: vec![acc("Alice"), acc("alice")],
}];
let msg = format_refusal_message_collisions(&groups);
assert!(msg.contains("1 colliding username group(s)"));
assert!(msg.contains("2 affected account(s)"));
assert!(msg.contains("oxicloud migrate lowercase-usernames"));
assert!(msg.contains("Canonical form: alice"));
assert!(msg.contains("Alice"));
assert!(msg.contains("last_login: never"));
}
#[test]
fn refusal_message_caps_group_display_and_notes_overflow() {
let groups: Vec<_> = (0..15)
.map(|i| CollisionGroup {
canonical: format!("user{i:02}"),
members: vec![acc(&format!("User{i:02}")), acc(&format!("user{i:02}"))],
})
.collect();
let msg = format_refusal_message_collisions(&groups);
// First 10 groups shown by canonical name.
assert!(msg.contains("Canonical form: user00"));
assert!(msg.contains("Canonical form: user09"));
// Overflow tail names how many are hidden.
assert!(msg.contains("and 5 more group(s)"));
}
#[test]
fn refusal_message_reports_total_across_all_groups() {
// Two groups with different sizes — 2 + 3 = 5 members total.
let groups = vec![
CollisionGroup {
canonical: "alice".into(),
members: vec![acc("Alice"), acc("alice")],
},
CollisionGroup {
canonical: "bob".into(),
members: vec![acc("Bob"), acc("BOB"), acc("bob")],
},
];
let msg = format_refusal_message_collisions(&groups);
assert!(msg.contains("2 colliding username group(s)"));
assert!(msg.contains("5 affected account(s)"));
}
}
+93 -14
View File
@@ -322,9 +322,15 @@ impl User {
is_external: bool, is_external: bool,
) -> UserResult<Self> { ) -> UserResult<Self> {
Self::validate_email(&email)?; Self::validate_email(&email)?;
if let Some(ref u) = username { // Shadow `username` with the canonical (trimmed, lowercased)
Self::validate_username(u)?; // form returned by `validate_username`. Every downstream write
} // consumes the shadowed binding, so the row that lands in the
// DB is always the normalised value. See
// `docs/plan/username-lowercase.md`.
let username = match username {
Some(u) => Some(Self::validate_username(&u)?),
None => None,
};
if let Some(ref h) = password_hash if let Some(ref h) = password_hash
&& h.is_empty() && h.is_empty()
{ {
@@ -809,8 +815,10 @@ impl User {
/// renamed: it was display text at creation; the folder is owned /// renamed: it was display text at creation; the folder is owned
/// by `user_id`. /// by `user_id`.
pub fn set_username(&mut self, new_username: String) -> UserResult<()> { pub fn set_username(&mut self, new_username: String) -> UserResult<()> {
Self::validate_username(&new_username)?; // Canonical form (trim + lowercase) — see
self.username = Some(new_username); // `validate_username`. Callers can pass any case; we store
// the normalised value.
self.username = Some(Self::validate_username(&new_username)?);
self.updated_at = Utc::now(); self.updated_at = Utc::now();
Ok(()) Ok(())
} }
@@ -881,20 +889,41 @@ impl User {
/// a handle that shadows another user's email). No leading/trailing /// a handle that shadows another user's email). No leading/trailing
/// dot or hyphen. The character set also prevents XSS payloads from /// dot or hyphen. The character set also prevents XSS payloads from
/// being stored as usernames. /// being stored as usernames.
fn validate_username(username: &str) -> UserResult<()> { /// Validate AND canonicalise a username.
let len = username.chars().count(); ///
/// Two normalisations run first, before every check:
/// - `trim()` — strip whitespace clients may have added.
/// - `to_ascii_lowercase()` — usernames are case-insensitive
/// identifiers. Users type `Alice`, `ALICE`, `alice` on
/// different clients; all three refer to the same account.
/// ASCII-only by construction (charset check below), so
/// `to_ascii_lowercase` is deterministic and locale-safe —
/// no Unicode case-folding surprises (Turkish dotted-I,
/// German ß, Greek final sigma, NFC vs NFD).
///
/// Returns the canonical form on success. Every entity write
/// site consumes the returned string — because the signature
/// changed from `Result<()>` to `Result<String>`, any caller
/// that ignored the result is now a compile error. That's
/// what forces every write path through the normaliser.
///
/// See `docs/plan/username-lowercase.md` for the full design.
pub fn validate_username(username: &str) -> UserResult<String> {
let normalized = username.trim().to_ascii_lowercase();
let len = normalized.chars().count();
if !(2..=64).contains(&len) { if !(2..=64).contains(&len) {
return Err(UserError::InvalidUsername( return Err(UserError::InvalidUsername(
"Username must be between 2 and 64 characters".to_string(), "Username must be between 2 and 64 characters".to_string(),
)); ));
} }
if username.contains('@') { if normalized.contains('@') {
return Err(UserError::InvalidUsername( return Err(UserError::InvalidUsername(
"Username must not contain '@' — use the email field for email addresses" "Username must not contain '@' — use the email field for email addresses"
.to_string(), .to_string(),
)); ));
} }
if !username if !normalized
.chars() .chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == '.') .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == '.')
{ {
@@ -903,16 +932,16 @@ impl User {
.to_string(), .to_string(),
)); ));
} }
if username.starts_with('.') if normalized.starts_with('.')
|| username.starts_with('-') || normalized.starts_with('-')
|| username.ends_with('.') || normalized.ends_with('.')
|| username.ends_with('-') || normalized.ends_with('-')
{ {
return Err(UserError::InvalidUsername( return Err(UserError::InvalidUsername(
"Username must not start or end with a dot or hyphen".to_string(), "Username must not start or end with a dot or hyphen".to_string(),
)); ));
} }
Ok(()) Ok(normalized)
} }
/// Basic but meaningful email validation: /// Basic but meaningful email validation:
@@ -1046,4 +1075,54 @@ mod tests {
assert_eq!(u.display_full(true), "solo@x.com"); assert_eq!(u.display_full(true), "solo@x.com");
assert_eq!(u.display_full(false), "solo@x.com"); assert_eq!(u.display_full(false), "solo@x.com");
} }
// ── validate_username: normalization + rules ─────────────────────────────
//
// Post-lowercase-migration `validate_username` returns the canonical
// (trimmed, lowercased) form on success. Every write-site consumes
// that returned string via the shadow in `User::new` /
// `set_username`, so the invariant "usernames in `auth.users` are
// always canonical" is enforced at the domain boundary.
//
// The rules that DON'T change (charset, length, no leading/trailing
// dot or hyphen, no `@`) get their coverage here too so a future
// rewrite of `validate_username` can't regress them silently.
#[test]
fn validate_username_lowercases_and_trims() {
// Uppercase in the middle → canonical form is lowercase.
assert_eq!(User::validate_username("Alice").unwrap(), "alice");
// All-uppercase.
assert_eq!(User::validate_username("ALICE").unwrap(), "alice");
// Whitespace around a mixed-case name → both stripped.
assert_eq!(User::validate_username(" Alice ").unwrap(), "alice");
// Already-canonical passes through unchanged.
assert_eq!(User::validate_username("alice").unwrap(), "alice");
}
#[test]
fn validate_username_charset_and_boundary_rules_survive_normalization() {
// Trailing hyphen — still rejected after the case-fold.
assert!(User::validate_username("alice-").is_err());
// Leading dot.
assert!(User::validate_username(".alice").is_err());
// Whitespace INSIDE the name (not just around it) — the
// charset check rejects space characters.
assert!(User::validate_username("Al ice").is_err());
// Non-ASCII letter — usernames are ASCII-only.
assert!(User::validate_username("Álice").is_err());
// `@` is forbidden (disjoint namespace with email lookup).
assert!(User::validate_username("alice@example").is_err());
}
#[test]
fn validate_username_length_bounds_apply_after_trim() {
// Two-char minimum satisfied AFTER trim.
assert_eq!(User::validate_username(" ab ").unwrap(), "ab");
// Below the minimum after trim.
assert!(User::validate_username(" a ").is_err());
// Above the maximum after trim.
let too_long = "a".repeat(65);
assert!(User::validate_username(&too_long).is_err());
}
} }
+40
View File
@@ -61,6 +61,46 @@ pub async fn create_database_pools(config: &AppConfig) -> Result<DbPools> {
} }
tracing::info!("Database migrations complete"); tracing::info!("Database migrations complete");
// Username-lowercase verifier — three outcomes:
// * Clean → nothing to do.
// * AutoRenamable → non-colliding mixed-case rows exist; lowercase
// them in one transaction and continue. Silent
// action is bounded to the case where there is
// exactly one correct move ([[feedback_no_silent_auto_repair]]
// in spirit — ambiguity → refusal, unique fix → apply).
// Each rename emits an audit line.
// * Collisions → two or more active rows share a LOWER(username)
// form (e.g. `Alice` + `alice`); tiebreak needs a
// human, refuse to boot and print the CLI command.
// See `common::username_migration::verify_all_usernames_lowercase`.
use crate::common::username_migration::{
UsernameCaseCheck, apply_auto_renames, format_refusal_message_collisions,
verify_all_usernames_lowercase,
};
match verify_all_usernames_lowercase(&primary).await {
Ok(UsernameCaseCheck::Clean) => {}
Ok(UsernameCaseCheck::AutoRenamable(accounts)) => {
let count = accounts.len();
if let Err(e) = apply_auto_renames(&primary, &accounts).await {
return Err(DbError(format!(
"username lowercase auto-rename failed at boot: {e}. \
Run `oxicloud migrate lowercase-usernames --dry-run` to \
inspect the current state, then apply manually."
)));
}
tracing::info!(
target: "audit",
event = "user.usernames_lowercased_on_boot_summary",
renamed = count,
"auto-lowercased {count} non-colliding mixed-case username(s) at boot",
);
}
Ok(UsernameCaseCheck::Collisions(groups)) => {
return Err(DbError(format_refusal_message_collisions(&groups)));
}
Err(msg) => return Err(DbError(msg)),
}
// --- maintenance pool --- // --- maintenance pool ---
let maintenance = create_pool_with_retries( let maintenance = create_pool_with_retries(
&config.database.connection_string, &config.database.connection_string,
@@ -472,8 +472,15 @@ impl UserRepository for UserPgRepository {
Ok((user, flags)) Ok((user, flags))
} }
/// Gets a user by username /// Gets a user by username.
///
/// Lowercases the input before binding: usernames are stored in
/// canonical (lowercase, trimmed) form by `validate_username`
/// (see `docs/plan/username-lowercase.md`), and callers may pass
/// whatever case the user typed at the login form. Normalising
/// here means every caller doesn't have to remember.
async fn get_user_by_username(&self, username: &str) -> UserRepositoryResult<User> { async fn get_user_by_username(&self, username: &str) -> UserRepositoryResult<User> {
let username = username.trim().to_ascii_lowercase();
let row = sqlx::query( let row = sqlx::query(
r#" r#"
SELECT SELECT
@@ -487,7 +494,7 @@ impl UserRepository for UserPgRepository {
WHERE username = $1 WHERE username = $1
"#, "#,
) )
.bind(username) .bind(&username)
.fetch_one(&*self.pool) .fetch_one(&*self.pool)
.await .await
.map_err(Self::map_sqlx_error)?; .map_err(Self::map_sqlx_error)?;
+279 -15
View File
@@ -72,6 +72,86 @@ pub const CDC_AVG_CHUNK: usize = 262_144;
/// Maximum CDC chunk size (1 MB). /// Maximum CDC chunk size (1 MB).
pub const CDC_MAX_CHUNK: usize = 1_048_576; pub const CDC_MAX_CHUNK: usize = 1_048_576;
// ════════════════════════════════════════════════════════════════════════════
// ref_count audit log — attribution trail for ref_count changes
//
// Emitted at every semantic ref-count mutation site so an unexplained drift
// (`manifests_consistency` / `blobs_consistency` finding) can be traced back
// to its calling function within one log query. See
// `docs/plan/refcount-audit.md` for design rationale, retention model, and
// upgrade path to a DB-backed table if log retention proves insufficient.
//
// Enable at runtime with `RUST_LOG=oxicloud::refcount=info`. Off by default;
// info-level so a healthy prod deployment doesn't spam the log stream.
// ════════════════════════════════════════════════════════════════════════════
/// Table names the audit stream uses. Constants (not free strings) so grep
/// across a log stream matches a fixed vocabulary and a typo in a call site
/// fails to compile instead of silently drifting.
mod refcount_audit_table {
pub const CHUNK_MANIFESTS: &str = "chunk_manifests";
pub const BLOBS: &str = "blobs";
}
/// Source labels for the audit stream — one stable string per Rust function
/// that mutates a ref_count. Kept as a module so `grep source=` on the log
/// stream shows a fixed enumeration; a rename here is intentional, a rename
/// at a call site alone doesn't compile.
///
/// See `docs/plan/refcount-audit.md § Callsites to instrument` for the full
/// list. New callers add a new constant here; adding one string at the call
/// site alone is discouraged (breaks the "closed vocabulary" property).
mod refcount_audit_source {
pub const STORE_FROM_STREAM_NEW_MANIFEST: &str = "store_from_stream.new_manifest";
pub const BUMP_MANIFEST_IF_EXISTS: &str = "bump_manifest_if_exists";
pub const ADD_REFERENCE_MANIFEST: &str = "add_reference.manifest";
pub const ADD_REFERENCE_LEGACY: &str = "add_reference.legacy";
pub const REMOVE_MANIFEST_REFERENCE_DECREMENT: &str = "remove_manifest_reference.decrement";
pub const REMOVE_MANIFEST_REFERENCE_DELETE: &str = "remove_manifest_reference.delete";
pub const REMOVE_LEGACY_REFERENCE: &str = "remove_legacy_reference";
pub const STORE_ATTACHED_BLOB_SAME_CONTENT_BALANCE: &str =
"store_attached_blob.same_content_balance";
pub const STORE_ATTACHED_BLOB_REPLACE_RELEASE: &str = "store_attached_blob.replace_release";
}
/// Outcome of [`DedupService::store_attached_blob_if_absent`]. Split
/// so the caller (`thumb_attached_import_service`) can bump its
/// `imported` vs `already` counters without a second query.
#[derive(Debug, Clone)]
pub enum AttachedBlobInsertOutcome {
/// We won the atomic INSERT — the row now points at `hash`.
Inserted { hash: String },
/// A row already existed when the atomic INSERT ran (someone else
/// won, or the migration was re-triggered). `existing_hash` is
/// the row's current `blob_hash` as read moments after the DO
/// NOTHING resolved — useful for the import service's
/// verify-and-unlink readback.
AlreadyPresent { existing_hash: String },
}
/// Emit a single audit line for a ref_count change. Called AFTER the SQL
/// UPDATE / INSERT / DELETE returns Ok, so a rolled-back transaction won't
/// leave a phantom log line (the SQL error path returns before this call).
///
/// `delta` is the signed change (`+1` on increment, `-1` on decrement,
/// `-old_count` when the row is deleted at its last reference — the
/// convention is "resulting ref_count is 0 for reads afterward").
///
/// The tracing span inherits request-scope context (request_id, caller_id
/// from auth middleware, job run id from scheduler) automatically, so
/// no explicit correlation-id plumbing is needed here.
#[inline]
fn audit_ref_count(table: &'static str, hash: &str, delta: i32, source: &'static str) {
tracing::info!(
target: "oxicloud::refcount",
table,
hash = %hash,
delta,
source,
"ref_count {}", if delta >= 0 { "+" } else { "-" }
);
}
// ── CDC helper types ───────────────────────────────────────────────────────── // ── CDC helper types ─────────────────────────────────────────────────────────
/// Everything a streaming chunk ingest learned about its byte stream. /// Everything a streaming chunk ingest learned about its byte stream.
@@ -751,24 +831,146 @@ impl DedupService {
.await .await
.map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?; .map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?;
// A replaced row's old blob loses its only reference from here. Not // Two shapes to balance depending on whether the UPSERT was a
// releasing it would pin those bytes forever — nothing else points at // real content replacement or a same-content re-store:
// a superseded preview. //
if let Some((old_hash,)) = previous // - `previous == Some(old) && old != attached_hash` — different
&& old_hash != attached_hash // content overwritten in the row. Release the old blob's ref
&& let Err(e) = self.remove_reference(&old_hash).await // (its file_attached row is gone; would leak forever otherwise).
{ //
tracing::warn!( // - `previous == Some(old) && old == attached_hash` — SAME-content
target: "oxicloud::dedup", // re-store. `store_from_stream` above incremented the manifest
error = %e, // unconditionally, but the row's blob_hash didn't change so no
"failed to release replaced attached-blob reference for {}", // logical reference was added. Cancel the phantom increment
&old_hash[..old_hash.len().min(12)], // here, or it accumulates one +1 leak per same-content call.
); // Mirrors the pattern `store_derived_blob` uses on its
// `ON CONFLICT DO NOTHING` `inserted == 0` branch.
// See `docs/plan/refcount-audit.md` for how the audit stream
// would surface this class of drift if it reappears.
//
// - `previous == None` — brand new (file_id, kind, variant) row.
// `store_from_stream`'s +1 pairs with the new row's implicit
// reference; nothing to release.
if let Some((old_hash,)) = previous {
let source = if old_hash == attached_hash {
refcount_audit_source::STORE_ATTACHED_BLOB_SAME_CONTENT_BALANCE
} else {
refcount_audit_source::STORE_ATTACHED_BLOB_REPLACE_RELEASE
};
if let Err(e) = self.remove_reference(&old_hash).await {
tracing::warn!(
target: "oxicloud::dedup",
error = %e,
kind = source,
"failed to balance attached-blob reference for {}",
&old_hash[..old_hash.len().min(12)],
);
}
} }
Ok(attached_hash) Ok(attached_hash)
} }
/// Atomic never-overwrite variant of [`Self::store_attached_blob`],
/// for migration/import paths whose semantic is "write if absent,
/// leave alone if present" — mirrors the shape
/// [`Self::store_derived_blob`] already uses.
///
/// The plain `store_attached_blob` reads `previous`, then upserts,
/// then decrements — non-transactional. That's correct for the
/// user-driven PUT thumbnail path (a real replacement should
/// release the superseded blob), but it opens a check-then-act
/// race with concurrent writers when the caller's intent is
/// "only import if this file hasn't already got a preview".
/// `thumb_attached_import_service` is exactly that caller.
///
/// This variant uses a single-statement `INSERT ... ON CONFLICT
/// DO NOTHING` — race-free by construction. If the row already
/// exists (any content), the atomic INSERT is a no-op and we
/// release the reference `store_from_stream` just took. If we
/// won the insert, the reference is legitimately held by our
/// new row.
///
/// Return value discriminates the two cases so the caller can
/// track its own `imported` vs `already` counters:
/// - [`AttachedBlobInsertOutcome::Inserted`] — we wrote the row.
/// - [`AttachedBlobInsertOutcome::AlreadyPresent`] — a row was
/// there when we arrived; we made no change and released our
/// ref. `existing_hash` is the concurrent winner's blob hash,
/// returned via a follow-up SELECT (so it's not strictly
/// atomic with the INSERT, but that's fine — the row's shape
/// is stable now that a concurrent writer can no longer
/// collide with us here; any later change goes through the
/// full `store_attached_blob` UPSERT path, which is out of
/// scope for this method's "if absent" contract).
pub async fn store_attached_blob_if_absent(
&self,
file_id: &str,
kind: &str,
variant: &str,
content_type: &str,
bytes: Bytes,
uploaded_by: uuid::Uuid,
) -> Result<AttachedBlobInsertOutcome, DomainError> {
let stored = self
.store_from_stream(
stream::once(async move { Ok::<Bytes, std::io::Error>(bytes) }),
Some(content_type.to_string()),
)
.await?;
let attached_hash = stored.hash().to_string();
// Single-statement atomic INSERT. `ON CONFLICT DO NOTHING`
// means: if another writer got there first, we silently
// yield. Same primitive `store_derived_blob` uses.
let inserted = sqlx::query(
"INSERT INTO storage.file_attached_blobs
(file_id, kind, variant, blob_hash, content_type, uploaded_by)
VALUES ($1::uuid, $2, $3, $4, $5, $6)
ON CONFLICT (file_id, kind, variant) DO NOTHING",
)
.bind(file_id)
.bind(kind)
.bind(variant)
.bind(&attached_hash)
.bind(content_type)
.bind(uploaded_by)
.execute(self.pool.as_ref())
.await
.map_err(|e| DomainError::internal_error("Dedup", format!("record attached blob: {e}")))?
.rows_affected();
if inserted == 0 {
// Row already existed when we arrived. Release the ref
// `store_from_stream` above took — the row that would
// justify it is not ours. Best-effort: leaving a
// dangling ref is worse than a warn log line.
if let Err(e) = self.remove_reference(&attached_hash).await {
tracing::warn!(
target: "oxicloud::dedup",
error = %e,
"failed to release duplicate attached-blob reference for {}",
&attached_hash[..attached_hash.len().min(12)],
);
}
// Fetch the concurrent winner's hash so the import
// service can readback-verify against its sidecar. The
// window between DO NOTHING and this SELECT is narrow;
// if the row gets updated in it, the sidecar delete
// path fails its verify and keeps the sidecar — the
// conservative fallback.
let existing = self.find_attached_blob(file_id, kind, variant).await;
return Ok(AttachedBlobInsertOutcome::AlreadyPresent {
existing_hash: existing.map(|r| r.blob_hash).unwrap_or_default(),
});
}
Ok(AttachedBlobInsertOutcome::Inserted {
hash: attached_hash,
})
}
/// Look up bytes attached to a file. File-keyed counterpart of /// Look up bytes attached to a file. File-keyed counterpart of
/// [`Self::find_derived_blob`]. /// [`Self::find_derived_blob`].
pub async fn find_attached_blob( pub async fn find_attached_blob(
@@ -1316,6 +1518,17 @@ impl DedupService {
total_size, total_size,
chunk_hashes.len(), chunk_hashes.len(),
); );
// The manifest INSERT above set `ref_count = 1` — that's
// the initial reference held by whatever callsite drove
// this ingest (a file's body, a preview attachment, a
// derivation). Audit-log the +1 so drift investigations
// can find where a manifest first came into existence.
audit_ref_count(
refcount_audit_table::CHUNK_MANIFESTS,
file_hash,
1,
refcount_audit_source::STORE_FROM_STREAM_NEW_MANIFEST,
);
self.fire_blob_creation_hooks(file_hash, content_type.as_deref()); self.fire_blob_creation_hooks(file_hash, content_type.as_deref());
return Ok(DedupResultDto::NewBlob { return Ok(DedupResultDto::NewBlob {
hash: file_hash.to_string(), hash: file_hash.to_string(),
@@ -1747,7 +1960,7 @@ impl DedupService {
/// Bump a manifest's ref_count if it exists; returns its total_size. /// Bump a manifest's ref_count if it exists; returns its total_size.
/// Single statement — no window between the existence check and the bump. /// Single statement — no window between the existence check and the bump.
async fn bump_manifest_if_exists(&self, file_hash: &str) -> Result<Option<i64>, DomainError> { async fn bump_manifest_if_exists(&self, file_hash: &str) -> Result<Option<i64>, DomainError> {
sqlx::query_scalar::<_, i64>( let bumped = sqlx::query_scalar::<_, i64>(
"UPDATE storage.chunk_manifests SET ref_count = ref_count + 1 "UPDATE storage.chunk_manifests SET ref_count = ref_count + 1
WHERE file_hash = $1 WHERE file_hash = $1
RETURNING total_size", RETURNING total_size",
@@ -1757,7 +1970,17 @@ impl DedupService {
.await .await
.map_err(|e| { .map_err(|e| {
DomainError::internal_error("Dedup", format!("Failed to bump manifest ref_count: {e}")) DomainError::internal_error("Dedup", format!("Failed to bump manifest ref_count: {e}"))
}) })?;
if bumped.is_some() {
audit_ref_count(
refcount_audit_table::CHUNK_MANIFESTS,
file_hash,
1,
refcount_audit_source::BUMP_MANIFEST_IF_EXISTS,
);
}
Ok(bumped)
} }
/// Stream → chunk store, WITHOUT creating a manifest. /// Stream → chunk store, WITHOUT creating a manifest.
@@ -2219,6 +2442,12 @@ impl DedupService {
.rows_affected(); .rows_affected();
if manifest_affected > 0 { if manifest_affected > 0 {
audit_ref_count(
refcount_audit_table::CHUNK_MANIFESTS,
hash,
1,
refcount_audit_source::ADD_REFERENCE_MANIFEST,
);
return Ok(()); return Ok(());
} }
@@ -2246,6 +2475,12 @@ impl DedupService {
)); ));
} }
audit_ref_count(
refcount_audit_table::BLOBS,
hash,
1,
refcount_audit_source::ADD_REFERENCE_LEGACY,
);
Ok(()) Ok(())
} }
@@ -2355,6 +2590,17 @@ impl DedupService {
&file_hash[..12], &file_hash[..12],
chunk_hashes.len() chunk_hashes.len()
); );
// Emit AFTER the commit so a rolled-back TX doesn't leave a
// phantom audit line — the "delta = -current_rc" reflects
// "the manifest is gone, effective ref_count is 0". Convention
// for the audit stream: use the delta that would produce a
// read-back of 0.
audit_ref_count(
refcount_audit_table::CHUNK_MANIFESTS,
file_hash,
-current_rc,
refcount_audit_source::REMOVE_MANIFEST_REFERENCE_DELETE,
);
Ok(true) Ok(true)
} else { } else {
// Still has references — just decrement // Still has references — just decrement
@@ -2373,6 +2619,12 @@ impl DedupService {
.map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?; .map_err(|e| DomainError::internal_error("Dedup", format!("Commit: {}", e)))?;
tracing::debug!("Reference removed from manifest {}", &file_hash[..12]); tracing::debug!("Reference removed from manifest {}", &file_hash[..12]);
audit_ref_count(
refcount_audit_table::CHUNK_MANIFESTS,
file_hash,
-1,
refcount_audit_source::REMOVE_MANIFEST_REFERENCE_DECREMENT,
);
Ok(false) Ok(false)
} }
} }
@@ -2430,6 +2682,12 @@ impl DedupService {
self.reap_blob(hash).await; self.reap_blob(hash).await;
tracing::info!("BLOB DELETED: {} (no more references)", &hash[..12]); tracing::info!("BLOB DELETED: {} (no more references)", &hash[..12]);
audit_ref_count(
refcount_audit_table::BLOBS,
hash,
-ref_count,
refcount_audit_source::REMOVE_LEGACY_REFERENCE,
);
Ok(true) Ok(true)
} else { } else {
// Still has references — just decrement // Still has references — just decrement
@@ -2450,6 +2708,12 @@ impl DedupService {
})?; })?;
tracing::debug!("Reference removed from blob {}", &hash[..12]); tracing::debug!("Reference removed from blob {}", &hash[..12]);
audit_ref_count(
refcount_audit_table::BLOBS,
hash,
-1,
refcount_audit_source::REMOVE_LEGACY_REFERENCE,
);
Ok(false) Ok(false)
} }
} }
@@ -274,49 +274,31 @@ impl RecoverableJobHandler for ThumbAttachedImport {
}; };
let file_id_str = file_id.to_string(); let file_id_str = file_id.to_string();
// Already mapped. Checked BEFORE storing, because // Orphan check first — no atomic-insert exists for a
// `store_attached_blob` is ON CONFLICT DO UPDATE and would // file_id whose FK would reject. Same rationale as before;
// release then retake the reference on every run. // the race window between this check and the INSERT is
if let Some(existing) = self // narrow AND covered by the FK constraint if the file is
.dedup // deleted after we look — the atomic INSERT would then
.find_attached_blob(&file_id_str, "preview", &dir_name) // fail loudly instead of silently drift.
.await //
{ // Everything else — "row present" and "row absent" —
already += 1; // used to be split across two branches with a
// Drains on a later run too: importing first and enabling // non-transactional `find_attached_blob` between the
// deletion afterwards is the expected operator sequence, // check and the write. That opened a check-then-act
// so reaching here is the common path rather than an edge // race window: a concurrent thumbnail writer could
// case. // INSERT the row after the check returned None, and the
if delete_imported { // subsequent `store_attached_blob` UPSERT-UPDATE would
let path = self.thumbnails_root.join(&dir_name).join(&name); // fire with same-or-different content. In the
if ThumbDerivedImport::verify_and_unlink( // same-content case that leaked +1 on the manifest ref
&self.dedup, // (pre-fix; guard branch now cancels).
THUMB_ATTACHED_IMPORT_JOB_NAME, //
&file_id_str, // Merged into ONE atomic call
&existing.blob_hash, // `store_attached_blob_if_absent`: single-statement
&path, // `INSERT ... ON CONFLICT DO NOTHING`, race-free by
) // construction. The outcome enum distinguishes the two
.await // paths so `imported` and `already` counters stay
{ // accurate.
deleted += 1; if !self.file_exists(file_id).await {
} else {
unverified += 1;
record_or_log(
store,
THUMB_ATTACHED_IMPORT_JOB_NAME,
"sidecar_delete_unverified",
"anomaly",
None,
serde_json::json!({
"path": position,
"file_id": file_id_str,
"note": "attached blob did not read back; sidecar kept",
}),
)
.await;
}
}
} else if !self.file_exists(file_id).await {
// The file is gone, so this sidecar is unimportable: the // The file is gone, so this sidecar is unimportable: the
// FK on `file_id` would reject the row. Mirrors the // FK on `file_id` would reject the row. Mirrors the
// dead-source case in thumb_derived_import. // dead-source case in thumb_derived_import.
@@ -389,9 +371,10 @@ impl RecoverableJobHandler for ThumbAttachedImport {
let path = self.thumbnails_root.join(&dir_name).join(&name); let path = self.thumbnails_root.join(&dir_name).join(&name);
match fs::read(&path).await { match fs::read(&path).await {
Ok(data) => { Ok(data) => {
use crate::infrastructure::services::dedup_service::AttachedBlobInsertOutcome;
match self match self
.dedup .dedup
.store_attached_blob( .store_attached_blob_if_absent(
&file_id_str, &file_id_str,
"preview", "preview",
&dir_name, &dir_name,
@@ -404,14 +387,39 @@ impl RecoverableJobHandler for ThumbAttachedImport {
) )
.await .await
{ {
Ok(attached_hash) => { Ok(outcome) => {
imported += 1; // Bump the right counter AND pick the
if delete_imported { // hash we'll verify-and-unlink against:
// Inserted — our new blob
// AlreadyPresent — the concurrent
// winner's blob
// Both drain the sidecar identically
// (verify-then-unlink on repair mode).
let verify_hash = match outcome {
AttachedBlobInsertOutcome::Inserted { hash } => {
imported += 1;
hash
}
AttachedBlobInsertOutcome::AlreadyPresent {
existing_hash,
} => {
already += 1;
existing_hash
}
};
// Empty existing_hash only happens if
// the AlreadyPresent path's follow-up
// SELECT was overtaken by another
// writer. verify_and_unlink would
// refuse the sidecar delete in that
// case anyway, but skipping the call
// saves the pointless readback.
if delete_imported && !verify_hash.is_empty() {
if ThumbDerivedImport::verify_and_unlink( if ThumbDerivedImport::verify_and_unlink(
&self.dedup, &self.dedup,
THUMB_ATTACHED_IMPORT_JOB_NAME, THUMB_ATTACHED_IMPORT_JOB_NAME,
&file_id_str, &file_id_str,
&attached_hash, &verify_hash,
&path, &path,
) )
.await .await
@@ -94,6 +94,19 @@ pub async fn basic_auth_middleware(
let (raw_username, password) = let (raw_username, password) =
parse_basic_auth(auth_header).ok_or(NextcloudAuthError::Unauthorized)?; parse_basic_auth(auth_header).ok_or(NextcloudAuthError::Unauthorized)?;
// Canonicalise the whole Basic-Auth username to lowercase.
//
// Usernames are canonical (lowercase) in the DB post-migration
// (`docs/plan/username-lowercase.md`), and NC / DAVX5 clients that
// cached URLs from before the migration keep sending `Alice:pass`
// — the server continues to accept that indefinitely by
// lowercasing here. Safe for the multi-drive `user~drive_uuid`
// composite because UUID hex `[0-9a-f-]` lowercases to itself.
//
// ASCII-only by `validate_username`'s charset check, so
// `to_ascii_lowercase` is deterministic and locale-safe.
let raw_username = raw_username.to_ascii_lowercase();
// ── Multi-drive composite-username parse ──────────────────────── // ── Multi-drive composite-username parse ────────────────────────
// POC wire shape: `{username}~{drive_marker}` may appear in the // POC wire shape: `{username}~{drive_marker}` may appear in the
// Basic Auth header. `~` was chosen because it needs no URL // Basic Auth header. `~` was chosen because it needs no URL
@@ -319,7 +332,13 @@ pub fn parse_basic_auth(header_value: &str) -> Option<(String, String)> {
let decoded = String::from_utf8(decoded).ok()?; let decoded = String::from_utf8(decoded).ok()?;
let (user, pass) = decoded.split_once(':')?; let (user, pass) = decoded.split_once(':')?;
Some((user.to_string(), pass.to_string())) // Canonicalise the username to lowercase here too, so any caller
// that reaches for `parse_basic_auth` directly (bypassing the
// middleware wrapper) also sees the canonical form. Redundant with
// the middleware's explicit `to_ascii_lowercase` on `raw_username`
// — belt-and-braces to keep the invariant local to the parser too.
// See `docs/plan/username-lowercase.md § 4. NextCloud DAV surface`.
Some((user.to_ascii_lowercase(), pass.to_string()))
} }
#[cfg(test)] #[cfg(test)]
+15 -1
View File
@@ -107,7 +107,21 @@ fn extract_url_user(path: &str) -> Option<std::borrow::Cow<'_, str>> {
// common path allocates nothing; only a percent-encoded username owns. The // common path allocates nothing; only a percent-encoded username owns. The
// old `.into_owned()` forced a `String` on EVERY path-scoped NC DAV request // old `.into_owned()` forced a `String` on EVERY path-scoped NC DAV request
// (benches/ROUND19.md §M7). The caller compares by slice. // (benches/ROUND19.md §M7). The caller compares by slice.
urlencoding::decode(user_seg).ok() //
// Lowercase before returning so cached client URLs like
// `/dav/files/Alice/...` compare equal to the canonical
// (lowercase) `session.raw_username`. See
// `docs/plan/username-lowercase.md § 4. NextCloud DAV surface`.
//
// The lowercase transform always allocates (`to_ascii_lowercase`
// on a `str` returns `String`). Trades the "Cow::Borrowed common
// path" of the ROUND19 optimisation for correctness of the case-
// insensitive comparison at line 157 — a `&str` compare with a
// borrowed segment against a lowercase `session.raw_username`
// would silently mismatch for `Alice`. The alloc is one small
// String per NC DAV request; the correctness win is worth it.
let decoded = urlencoding::decode(user_seg).ok()?;
Some(std::borrow::Cow::Owned(decoded.to_ascii_lowercase()))
} }
/// Axum extractor: the shared handle to the request's [`NcSession`]. /// Axum extractor: the shared handle to the request's [`NcSession`].
+160
View File
@@ -0,0 +1,160 @@
# =============================================================
# OxiCloud — usernames are case-insensitive (silent lowercase on ingest)
# =============================================================
# Pin for `docs/plan/username-lowercase.md`. The plan makes usernames
# case-insensitive by canonicalising to lowercase in
# `User::validate_username`. This file covers the wire surface:
#
# 1. Registration with a mixed-case username lands as lowercase.
# 2. Login by the ORIGINAL mixed-case string succeeds (server
# normalises on lookup).
# 3. Login by ALL-CAPS of the same name also succeeds.
# 4. Login by the canonical lowercase form succeeds.
# 5. Profile rename to a mixed-case name lands as lowercase.
# 6. NC Basic Auth accepts every case variant of the same account.
#
# Character-set + boundary rules (no leading dot, no `@`, etc.) are
# NOT re-tested here — that's the Rust unit-test surface. This file
# only pins the end-to-end normalization behaviour.
#
# Runs after `setup.hurl` (admin exists). Uses a self-contained user
# to avoid interfering with other scenarios.
# =============================================================
# ─────────────────────────────────────────────────────────────
# Step 1 — Register with a mixed-case username. Expect the server
# to silently store the lowercase form.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/register
Content-Type: application/json
{
"username": "MixedCaseUser",
"email": "mixedcase@example.com",
"password": "MixedCasePassword1!"
}
HTTP 200
# ─────────────────────────────────────────────────────────────
# Step 2 — Log in with the ORIGINAL mixed-case string. Server
# should normalise on lookup and accept.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "MixedCaseUser", "password": "MixedCasePassword1!" }
HTTP 200
[Asserts]
jsonpath "$.access_token" isString
# The `/auth/me` response inside the login reply exposes the canonical
# stored username. Post-migration, it MUST be lowercase regardless of
# what the caller typed at registration.
jsonpath "$.user.full.user.username" == "mixedcaseuser"
# ─────────────────────────────────────────────────────────────
# Step 3 — Log in with ALL-CAPS. Same account, different casing.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "MIXEDCASEUSER", "password": "MixedCasePassword1!" }
HTTP 200
[Asserts]
jsonpath "$.user.full.user.username" == "mixedcaseuser"
# ─────────────────────────────────────────────────────────────
# Step 4 — Log in with the canonical lowercase form. Same account.
# Capture the token for the profile-rename step below.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "mixedcaseuser", "password": "MixedCasePassword1!" }
HTTP 200
[Captures]
mixed_token: jsonpath "$.access_token"
[Asserts]
jsonpath "$.user.full.user.username" == "mixedcaseuser"
# ─────────────────────────────────────────────────────────────
# Step 5 — Rename via profile PATCH. New name is mixed-case; server
# must store it as lowercase. Same rule as registration, applied on
# the mutation path.
# ─────────────────────────────────────────────────────────────
PATCH {{base_url}}/api/auth/me
Authorization: Bearer {{mixed_token}}
Content-Type: application/json
{ "username": "RenamedTarget" }
HTTP 200
[Asserts]
# The response echoes the stored (canonical) form.
jsonpath "$.full.user.username" == "renamedtarget"
# Old (pre-rename) username no longer resolves — login fails 401.
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "MixedCaseUser", "password": "MixedCasePassword1!" }
HTTP 401
# New (post-rename) mixed-case login succeeds.
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "RENAMEDTARGET", "password": "MixedCasePassword1!" }
HTTP 200
[Asserts]
jsonpath "$.user.full.user.username" == "renamedtarget"
# ─────────────────────────────────────────────────────────────
# Step 6 — NextCloud Basic Auth accepts every case variant.
# The middleware lowercases the decoded username on the auth path,
# and `extract_url_user` lowercases the URL segment. A cached
# client URL like `.../dav/files/RenamedTarget` continues to work
# indefinitely across the migration.
#
# Uses PROPFIND `Depth: 0` on `/remote.php/dav/files/{user}/` — a
# well-formed request that touches Basic Auth + URL parse + chroot
# resolve in one hop. 207 Multi-Status is the expected success shape.
# ─────────────────────────────────────────────────────────────
PROPFIND {{base_url}}/remote.php/dav/files/renamedtarget/
Depth: 0
[BasicAuth]
renamedtarget: MixedCasePassword1!
HTTP 207
PROPFIND {{base_url}}/remote.php/dav/files/RenamedTarget/
Depth: 0
[BasicAuth]
RenamedTarget: MixedCasePassword1!
HTTP 207
PROPFIND {{base_url}}/remote.php/dav/files/RENAMEDTARGET/
Depth: 0
[BasicAuth]
RENAMEDTARGET: MixedCasePassword1!
HTTP 207
# Mixed case in URL, lowercase in Basic Auth — still works because
# both surfaces normalise before comparison.
PROPFIND {{base_url}}/remote.php/dav/files/RenamedTarget/
Depth: 0
[BasicAuth]
renamedtarget: MixedCasePassword1!
HTTP 207
+65
View File
@@ -254,6 +254,71 @@ ATTACHED_2=$(sql "SELECT count(*) FROM storage.file_attached_blobs WHERE file_id
[[ "$ATTACHED_2" == "$ATTACHED_AFTER" ]] || fail "re-run duplicated attached rows" [[ "$ATTACHED_2" == "$ATTACHED_AFTER" ]] || fail "re-run duplicated attached rows"
log "re-run is a no-op: rows and refcounts unchanged." log "re-run is a no-op: rows and refcounts unchanged."
# ── 5c. store_attached_blob same-content guard (regression test) ─────────────
#
# The PUT /api/files/{id}/thumbnail/{size} endpoint calls
# `dedup_service::store_attached_blob` directly — no pre-check like
# `thumb_attached_import_service` does. A same-content re-PUT is the
# ONLY current public surface that exercises the ref-balance branch
# added to `store_attached_blob` after the Sept-2026 manifest-drift
# investigation:
#
# if let Some((old_hash,)) = previous {
# // same-content: cancel `store_from_stream`'s spurious +1
# // different-hash: release the superseded blob's ref
# remove_reference(&old_hash)
# }
#
# Before the fix, same-content re-PUT would leak +1 on the manifest's
# ref_count every time (store_from_stream incremented, guard skipped
# the decrement when old == new). This test PUTs the same thumbnail
# bytes twice and asserts the manifest's ref_count is unchanged.
#
# Uses variant "icon" so we don't collide with the row the import
# job populated above (variant "preview"), keeping the two flows
# independent. Server re-encodes to JPEG deterministically, so both
# PUTs produce byte-identical manifest content.
log "5c. Same-content re-PUT via API does not churn refcount"
# Round 1: first PUT populates the row (INSERT — previous=None, guard
# doesn't fire). This is the fresh-ingest path; ref_count becomes 1.
curl -sf -X PUT \
-H "$AUTH" \
-H "Content-Type: image/jpeg" \
--data-binary "@$UPLOADED_THUMB" \
"$base_url/api/files/$FILE_ID/thumbnail/icon" \
>/dev/null || fail "5c: first PUT of thumbnail (variant=icon) failed"
GUARD_HASH=$(sql "SELECT blob_hash FROM storage.file_attached_blobs \
WHERE file_id='$FILE_ID' AND kind='preview' AND variant='icon' \
LIMIT 1;")
[[ -n "$GUARD_HASH" ]] || fail "5c: first PUT did not land a file_attached_blobs row"
GUARD_REFS_BEFORE=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$GUARD_HASH';")
# Legacy blobs (pre-CDC) don't have a chunk_manifests row — skip
# the test in that case rather than fail on an unrelated path.
if [[ -z "$GUARD_REFS_BEFORE" ]]; then
log "5c: attached blob is on the legacy path (no manifest) — guard test skipped (targets CDC path)"
else
# Round 2: second PUT with the SAME bytes. UPSERT-UPDATE fires,
# previous.blob_hash == new attached_hash, and the guard MUST
# cancel `store_from_stream`'s +1. Without the fix, refcount
# would go from 1 → 2 here.
curl -sf -X PUT \
-H "$AUTH" \
-H "Content-Type: image/jpeg" \
--data-binary "@$UPLOADED_THUMB" \
"$base_url/api/files/$FILE_ID/thumbnail/icon" \
>/dev/null || fail "5c: same-content re-PUT of thumbnail failed"
GUARD_REFS_AFTER=$(sql "SELECT ref_count FROM storage.chunk_manifests WHERE file_hash='$GUARD_HASH';")
[[ "$GUARD_REFS_AFTER" == "$GUARD_REFS_BEFORE" ]] \
|| fail "5c: same-content re-PUT churned refcount: $GUARD_REFS_BEFORE → $GUARD_REFS_AFTER (guard regressed?)"
log "5c: same-content re-PUT stable, refcount=$GUARD_REFS_AFTER"
fi
# ── 5b. Deletion: the destructive half, and the only one that can lose data # ── 5b. Deletion: the destructive half, and the only one that can lose data
# #
# Everything above is additive and recoverable. This unlinks files after a # Everything above is additive and recoverable. This unlinks files after a