cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge branch 'main' into french_translation

Browse Source
This commit is contained in:
Xalares
2026-09-07 15:07:36 +02:00
committed by GitHub
parent 7abb66c19f 3dd4167578
commit c78db6ec6c
246 changed files with 27103 additions and 3456 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cargo/audit.toml
+112
View File
@@ -23,10 +23,61 @@ ignore = [
"RUSTSEC-2026-0098",
"RUSTSEC-2026-0099",
# h2 0.3.27 — RUSTSEC-2026-0258 "unbounded empty DATA frames"
# (GHSA-q83h-524g-xf6h). Transitive via aws-smithy-http-client 1.1.12
# → hyper 0.14.32 → h2 0.3.27. The patched line is 0.4.16+, but hyper
# 0.14's `h2 = "0.3"` requirement pins us to the 0.3.x branch which
# will not receive a backport — real fix requires aws-smithy-http-client
# to migrate to hyper 1.x (which our other h2 copy — 0.4.16, already
# bumped — is on). The 0.4.x copy is fixed via `cargo update`; this
# ignore covers only the 0.3.x chain.
#
# Severity: low (advisory's own classification). Attack is empty-DATA-
# frame flooding by a malicious HTTP/2 peer → memory pressure or panic.
# In this codebase h2 0.3.x runs strictly on the CLIENT side of AWS
# SDK requests to S3 endpoints. Exploitation requires either
# compromising AWS S3 (out-of-scope) or MitM with a valid TLS cert
# for the configured S3 host (bigger problem than the DoS). No
# data-integrity or auth impact; panic path contained by
# request-level unwind.
#
# Un-ignore trigger: aws-smithy-http-client releases a version that
# switches to hyper 1.x (checkable with `cargo tree -i h2@0.3` — the
# command returns no rows once the chain is gone). Track upstream at
# https://github.com/smithy-lang/smithy-rs/issues (search "hyper 1").
"RUSTSEC-2026-0258",
# instant unmaintained — transitive via azure_core 0.21.0 (latest available).
# No direct security impact; no upgrade path exists.
"RUSTSEC-2024-0384",
# azure_core 0.21.0 writes the `authorization` header value to logs —
# `policies/transport.rs` does `debug!("…{request:#?}")` over the whole
# request. For a SharedKey entry that value is the per-request HMAC
# signature; for a SAS entry it is the token. Severity 6.5 (medium).
#
# The advisory says "upgrade to >=0.22.0". That version does not exist:
# `azure_core` jumped 0.21 → 1.x, and `azure_storage_blobs` never left
# 0.21.0 before being archived. So the stated remedy IS the official-SDK
# migration, tracked separately alongside the quick-xml pair above.
#
# Not reachable at our log levels: the line is `debug!` on the
# `azure_core::policies::transport` target, and the default filter is
# `info`. It fires only if an operator explicitly asks for
# `RUST_LOG=…,azure_core=debug`, which is not hypothetical — that is the
# invocation used to diagnose the Azurite migration hang. **Do not run
# `azure_core=debug` against a real Azure account**; it prints request
# signatures to the terminal. Against Azurite it only exposes the
# published dev key's signatures.
#
# A subscriber-level directive pinning that target off was prototyped
# and rejected 2026-09-02 — not worth carrying a filter hack for a
# dependency being replaced.
#
# Un-ignore trigger: the azure_storage_blob 1.x migration lands
# (`cargo tree -i azure_core@0.21` returns no rows).
"RUSTSEC-2026-0275",
# quick-xml 0.31.0 — transitive via azure_core 0.21.0 (unofficial SDK,
# now archived). Our direct dep is already on 0.41.0; the 0.31 copy is
# only reachable through the azure_storage_blobs chain, which parses
@@ -40,6 +91,38 @@ ignore = [
"RUSTSEC-2026-0195",
"RUSTSEC-2026-0194",
# azure_core 0.21.0 — "Legacy azure_core writes the authorization
# header value to logs" (RUSTSEC-2026-0275, 6.5 medium). Same
# unofficial archived SDK, same absent upgrade path as the other
# 0.21.0-chain advisories above: the advisory's "upgrade to
# >=0.22.0" applies to the official azure_core crate line, not to
# the archived 0.21.0 we're pinned on via the unofficial
# azure_storage_blobs SDK. Real fix is the official azure_core 1.0 /
# azure_storage_blob 1.0 SDK migration tracked separately (memory
# project_azure_sdk_migration_pending) — blocked upstream by the
# 1.0 SDK dropping shared-key auth.
#
# Exposure in this codebase is narrow. The advisory covers the
# HTTP client emitting the `Authorization` header value into log
# records; for our Azure backend usage that header value is
# `SharedKey <account>:<HMAC-SHA256>` — the shared key itself
# never appears, only a per-request HMAC signature bound to the
# request's `x-ms-date` and unusable outside the ~15 min clock-skew
# window. Reaching the log path further requires (a) an Azure
# backend actually being configured (S3 and local are the
# alternatives) and (b) the tracing subscriber emitting DEBUG
# records for the `azure_core` target — production defaults are
# INFO. Under both conditions the worst-case leak is replay of
# individual object operations within the skew window by an
# attacker who already has production log read access; the shared
# key cannot be derived.
#
# Un-ignore trigger: the official azure_core 1.0 migration lands —
# at which point this entry and the other azure_core 0.21.0-chain
# entries above (RUSTSEC-2026-0097, -2024-0384, -2026-0195,
# -2026-0194) all go away together.
"RUSTSEC-2026-0275",
# wasmtime 43.0.2 — "Stores can mix up type indices between engines"
# (GHSA-hgjw-h833-99q9). Transitive via extism 1.30.0 (latest published;
# extism `main` still pins wasmtime 43, no upgrade path). The advisory
@@ -58,6 +141,35 @@ ignore = [
# with its own Store (see infrastructure/services/plugins/runtime.rs).
"RUSTSEC-2026-0222",
# wasmtime 43.0.2 — "Filesystem sandbox escape when paths or symlinks
# contain trailing slashes" (RUSTSEC-2026-0269, 8.8 high). Same crate,
# same chain and same absent upgrade path as RUSTSEC-2026-0222 above:
# extism 1.30.0 is the latest published and pins wasmtime 43, while the
# advisory's fixed releases are >=24.0.13 <25, >=36.0.14 <37,
# >=46.0.3 <47, >=47.0.4 — none in the 43.x line, so there is no
# version satisfying extism's requirement that carries the fix.
#
# NOT REACHABLE, and for a stronger reason than the build-feature
# gating: this is a WASI filesystem sandbox escape, and OxiCloud's
# plugin runtime gives plugins no filesystem to escape from.
# `plugins/runtime.rs::compile` builds every plugin with
# `.with_wasi(false)` and declares no `allowed_paths`, so there are no
# preopened directories — the escape needs one to traverse out of.
# `.disallow_all_hosts()` removes outbound network on the same path.
#
# The build-level gating from the entry above still applies on top:
# `plugins` is opt-in and absent from `default`, so the CI release
# binary does not link wasmtime; runtime activation additionally needs
# OXICLOUD_ENABLE_PLUGINS=true; and plugin binaries are admin-supplied,
# not attacker input.
#
# Un-ignore trigger: EITHER extism releases a version on wasmtime
# >=46.0.3 (check with `cargo tree -i wasmtime --features plugins`),
# OR `plugins/runtime.rs` gains `allowed_paths` / `with_wasi(true)` —
# at which point this stops being unreachable and blocks release
# rather than being ignored.
"RUSTSEC-2026-0269",
# astral-tokio-tar 0.5.6 — tar extraction advisories, transitive via
# testcontainers → testcontainers-modules, a DEV-dependency used only by
# the `--cfg integration_tests` harness to spin up throwaway Postgres
.github/workflows/ci.yml
+100
View File
@@ -176,6 +176,21 @@ jobs:
with:
components: clippy
- uses: Swatinem/rust-cache@v2
# `--all-features` includes `bundled-assets`, whose build.rs guard
# requires `static-dist/index.html` at compile time (rust-embed
# scans the folder). Build the SPA first so the lint pass covers
# the embed code paths without needing to enumerate features
# around it. ~90 s once, cached by npm-cache on repeats.
- uses: actions/setup-node@v4
with:
node-version: 26.3.0
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Build SPA (needed for --all-features / bundled-assets)
working-directory: frontend
run: npm ci && npm run build
- run: cargo clippy --all-targets --all-features -- -D warnings
# Mirrors the `wasm-check` justfile recipe. The wasm crate is a
@@ -306,6 +321,18 @@ jobs:
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
# `--all-features` enables `bundled-assets`, whose build.rs guard
# requires `static-dist/index.html` at compile time. Build the SPA
# first so tests can compile the embed code paths. ~90 s, cached.
- uses: actions/setup-node@v4
with:
node-version: 26.3.0
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Build SPA (needed for --all-features / bundled-assets)
working-directory: frontend
run: npm ci && npm run build
- name: Initialize test database
# Applies every migration + seeds the integration-test admin row.
# Same script used by `just test-integration` locally.
@@ -460,6 +487,79 @@ jobs:
path: tests/api/storage/
retention-days: 7
bundled-binary-test:
# `--features bundled-assets` end-to-end integration test.
#
# Builds oxicloud with the SPA baked in via rust-embed, boots it
# against a nonexistent OXICLOUD_STATIC_PATH so the embed path is
# forced, and asserts SPA + locales + immutable-cache headers +
# CSP all serve correctly from the embedded corpus. Guards against
# three failure classes that don't surface in filesystem-served CI:
#
# 1. rust-embed configuration (glob patterns silently producing a
# 0-file embed — hit 2026-08-28).
# 2. Debug-vs-release drift (rust-embed's dynamic-read mode in
# debug builds masks embed bugs; `debug-embed` feature bakes
# files in for both profiles).
# 3. Axum `Path` extractor on fallback routes returning 500 (the
# `serve_root` handler needs `Request` extraction — hit 2026-08-28).
#
# See tests/bundled-binary/run.sh + docs/plan/bundled-binary.md § 2.
#
# Doesn't reuse the `build` job's artifact because that binary is
# compiled with `--features plugins`, not `--features bundled-assets`
# — different feature set = different target. `Swatinem/rust-cache`
# still shares dependency compilation between the two jobs.
name: Bundled-assets binary — embed + SPA-serve integration
needs: changes
if: |
github.event_name == 'pull_request' &&
(needs.changes.outputs.backend == 'true' || needs.changes.outputs.frontend == 'true')
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
# Same disk-hygiene pattern as the `build` job — cargo release
# link + full node_modules install would otherwise squeeze the
# runner disk budget under peak concurrency.
- uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: actions/setup-node@v4
with:
node-version: 26.3.0
cache: npm
cache-dependency-path: frontend/package-lock.json
# Build the SPA up front so the test script can run with
# `SKIP_FRONTEND_BUILD=1` — makes the test-runner log clean
# (no duplicated npm ci noise) and puts the SPA build's cost
# in its own step for CI-side timing visibility.
- name: Build SPA (Vite → static-dist/)
working-directory: frontend
run: npm ci && npm run build
- name: Run bundled-binary integration test
run: bash tests/bundled-binary/run.sh
env:
SKIP_FRONTEND_BUILD: "1"
# Preserve the server log even on failure so a red run doesn't
# require re-running locally to see what happened at boot.
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: bundled-binary-server-log
path: tests/bundled-binary/server.log
retention-days: 7
litmus:
name: WebDAV RFC 4918 — litmus (59/59)
needs: build
.github/workflows/docker-publish.yml
+160 -26
View File
@@ -1,7 +1,30 @@
name: "Docker Hub & GHCR Release"
name: "Docker Publish (release, main, dry-run)"
# Per-run title shown in the Actions tab list — makes it obvious at
# a glance which channel a given run served and (for dispatched
# runs) whether it was a dry-run. Without this, GitHub falls back
# to the commit subject, which is uninformative when multiple
# workflows fire on the same commit.
#
# Falls back to `github.ref_name` for push / release events (which
# don't carry `inputs.*`), and stitches "[DRY-RUN]" onto the
# dispatched cases where `inputs.dry_run` is checked.
run-name: >-
Docker Publish
${{ github.event_name == 'workflow_dispatch' && inputs.dry_run && '[DRY-RUN]' || '' }}
— ${{ github.event.inputs.version || github.ref_name }}
on:
# Every merge to `main` republishes the mutable `:main` tag so users
# tracking the tip of development can pull the freshest image
# without waiting for a release. `:latest` is DELIBERATELY not
# touched on this trigger — it stays pointed at the last released
# version. Provenance for a specific `:main` pull is queryable via
# `docker inspect` (org.opencontainers.image.revision label carries
# the SHA).
push:
branches:
- "main"
tags:
- "v*"
release:
@@ -12,14 +35,44 @@ on:
version:
description: 'Version tag to publish (e.g. v0.5.3)'
required: true
dry_run:
description: 'Dry run — build only, skip push + verify. Prints the tag set that WOULD be pushed. Use to smoke-test workflow edits without touching the registry.'
required: false
type: boolean
default: false
env:
# Docker Hub image name is fixed to the canonical namespace — forks
# that opt in to publishing typically also override this with their
# own DockerHub account name (see the fork guide in
# docs/plan/docker-publish.md, if/when documented).
REGISTRY_IMAGE: diocrafts/oxicloud
GHCR_REGISTRY_IMAGE: ghcr.io/atalayalabs/oxicloud
# GHCR image name follows the repo owner — canonical repo publishes
# to `ghcr.io/atalayalabs/oxicloud`; a fork opting in via
# `vars.ENABLE_DOCKER_PUBLISH=true` publishes to its own owner's
# namespace with zero config edits.
GHCR_REGISTRY_IMAGE: ghcr.io/${{ github.repository_owner }}/oxicloud
# Cancel superseded `:main` builds if commits land in quick succession
# — only the newest one matters, and having two racing builds pushing
# to the same mutable tag is a coin-toss on which one wins. Release-tag
# and manual-dispatch builds never cancel: each release is unique and
# irreversible; every one must publish.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
jobs:
# Run tests before publishing
# Run tests before publishing.
#
# SKIPPED on push-to-`main`: the same test matrix already ran on the
# PR that produced this commit (branch protection ensures PRs pass
# CI before merge). Re-running here would double the CI cost per
# merge for zero signal. Release-tag / manual-dispatch builds still
# test — they're explicit "shipping this" moments where
# belt-and-suspenders matters.
test:
if: github.event_name != 'push' || !startsWith(github.ref, 'refs/heads/')
name: Pre-publish Tests
runs-on: ubuntu-latest
services:
@@ -60,30 +113,64 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 360
needs: test
# Publish gate — TWO conditions must hold:
#
# 1. The `test` job succeeded (or was skipped for push-to-main —
# branch protection ensures PR CI already ran; see the test
# job's `if:`). `always()` unblocks the `needs:` when test is
# skipped; the result check still blocks on real failures.
#
# 2. Publishing is enabled for THIS repo. Canonical
# `AtalayaLabs/OxiCloud` always publishes. Forks stay quiet by
# default (no accidental GHCR packages / wasted CI minutes when
# someone forks just to read code); a fork owner who wants to
# test-publish sets `ENABLE_DOCKER_PUBLISH=true` under
# Settings → Secrets and variables → Actions → Variables.
if: |
always() &&
(needs.test.result == 'success' || needs.test.result == 'skipped') &&
(github.repository == 'AtalayaLabs/OxiCloud' || vars.ENABLE_DOCKER_PUBLISH == 'true')
permissions:
contents: read
packages: write
# Job-level env — `secrets` context is legal here but NOT in
# step-level `if:` conditions. Precomputing the "is DH configured"
# signal as an env var lets downstream steps gate cleanly via
# `env.HAS_DOCKERHUB_TOKEN == 'true'` — see the DockerHub login
# step below.
env:
HAS_DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN != '' }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
# Build the exact tag behind the published release or manual dispatch.
# Build the exact tag behind the published release or manual
# dispatch. On push-to-main, `github.ref` resolves to
# `refs/heads/main` and this checks out the freshly-merged
# commit — exactly what we want to publish as `:main`.
ref: ${{ github.event.inputs.version || github.event.release.tag_name || github.ref }}
- name: Set version tag
id: version
run: |
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
VERSION="${{ github.event.inputs.version }}"
elif [ "${{ github.event_name }}" == "release" ]; then
VERSION="${{ github.event.release.tag_name }}"
else
VERSION="${GITHUB_REF#refs/tags/}"
fi
# Strip leading 'v' if present for Docker tag
VERSION="${VERSION#v}"
echo "VERSION=$VERSION" >> $GITHUB_ENV
echo "version=$VERSION" >> $GITHUB_OUTPUT
- name: Resolve channel + version + tags
id: meta
# Delegates to `scripts/compute-docker-tags.sh` — logic is
# unit-tested via `scripts/test-docker-publish-tags.sh` so
# any change to the tag policy gets caught before hitting a
# runner. The script emits `version`, `channel`, and `tags`
# to $GITHUB_OUTPUT (for later `steps.meta.outputs.*`),
# plus `VERSION` / `CHANNEL` / `SKIP_DOCKERHUB` to
# $GITHUB_ENV (for later steps that read env directly), plus
# a human-readable trailer to stdout for the run log —
# useful in dry-run mode where the tag set is the deliverable.
env:
EVENT_NAME: ${{ github.event_name }}
GITHUB_REF: ${{ github.ref }}
DISPATCH_VERSION: ${{ github.event.inputs.version }}
RELEASE_TAG: ${{ github.event.release.tag_name }}
# Empty DOCKERHUB_TOKEN → skip DH tags cleanly (forks that
# opted in via ENABLE_DOCKER_PUBLISH but didn't set up
# DockerHub still get their GHCR image published).
SKIP_DOCKERHUB: ${{ secrets.DOCKERHUB_TOKEN == '' && 'true' || 'false' }}
run: bash "$GITHUB_WORKSPACE/scripts/compute-docker-tags.sh"
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
@@ -92,6 +179,19 @@ jobs:
uses: docker/setup-buildx-action@v3
- name: Login to DockerHub
# Skipped cleanly when DOCKERHUB_TOKEN isn't configured on
# this repo — the meta step's SKIP_DOCKERHUB env drops DH
# tags from the push set in that case, so we don't need
# DH auth at all. Canonical repo always has the secret and
# always publishes; forks are opt-in via ENABLE_DOCKER_PUBLISH
# AND can further opt in / out of DH separately by
# adding / omitting DOCKERHUB_TOKEN.
#
# `secrets` context is not available in step-level `if:`
# conditions — we read it via the job-level env var
# `HAS_DOCKERHUB_TOKEN` computed above (which CAN reference
# secrets since it lives in `env:`, not `if:`).
if: env.HAS_DOCKERHUB_TOKEN == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
@@ -109,12 +209,17 @@ jobs:
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: |
${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }}
${{ env.REGISTRY_IMAGE }}:latest
${{ env.GHCR_REGISTRY_IMAGE }}:${{ env.VERSION }}
${{ env.GHCR_REGISTRY_IMAGE }}:latest
# `push` flips to `false` for a dry-run — the multi-arch
# build still runs (catches Dockerfile regressions), but
# nothing hits the registry. Only reachable via
# `workflow_dispatch` with `dry_run: true`. Real push
# events (release, branch push) always publish.
push: ${{ github.event.inputs.dry_run != 'true' }}
# Tag set computed in the meta step above — release channel
# publishes `:<version>` + `:latest`; main channel publishes
# just `:main`. Emitted to the build log either way so the
# dry-run mode surfaces "what would ship" in plain sight.
tags: ${{ steps.meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=max
# GitHub Actions env piped through so build.rs stamps
@@ -129,7 +234,36 @@ jobs:
GITHUB_HEAD_REF=${{ github.head_ref }}
- name: Verify published image
# Skipped on dry-run — nothing was pushed to pull back.
# Verifies GHCR unconditionally (always pushed), then adds a
# Docker Hub pull when the DH branch actually ran. Simpler
# than a matrix — the two registries share the same content
# (same multi-arch manifest), so one pull confirms the build
# + push worked; the other is just a "did we auth to both"
# sanity check.
if: github.event.inputs.dry_run != 'true'
run: |
echo "─── Verify GHCR ───"
docker pull ${{ env.GHCR_REGISTRY_IMAGE }}:${{ env.VERSION }}
docker image inspect ${{ env.GHCR_REGISTRY_IMAGE }}:${{ env.VERSION }} > /dev/null
echo "✅ ${{ env.GHCR_REGISTRY_IMAGE }}:${{ env.VERSION }} published"
if [ "${{ env.SKIP_DOCKERHUB }}" != "true" ]; then
echo "─── Verify Docker Hub ───"
docker pull ${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }}
docker image inspect ${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }}
echo "✅ Image ${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }} published successfully"
docker image inspect ${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }} > /dev/null
echo "✅ ${{ env.REGISTRY_IMAGE }}:${{ env.VERSION }} published"
else
echo "ℹ️ Skipped Docker Hub verification (DOCKERHUB_TOKEN not set on this repo)"
fi
- name: Dry-run summary
# Only surfaces in dry-run mode. Mirrors the "Verify" step's
# role — gives the operator running the dry-run a clear
# closing message with the exact tag set the workflow would
# have pushed. The meta step already logged it, this step
# just makes it prominent at the bottom of the run.
if: github.event.inputs.dry_run == 'true'
run: |
echo "🔍 DRY RUN — image built + tagged but NOT pushed."
echo "Would have published:"
echo "${{ steps.meta.outputs.tags }}" | sed 's/^/ /'
.github/workflows/release-binaries.yml
+288
View File
@@ -0,0 +1,288 @@
name: "Release Binaries (musl-linux + macOS)"
# Per-run title shown in the Actions tab — makes it obvious at a
# glance which tag is being packaged and whether a manual run is a
# dry-run (build tarballs into workflow artifacts, DON'T attach to
# any GitHub Release).
run-name: >-
Release Binaries
${{ github.event_name == 'workflow_dispatch' && inputs.dry_run && '[DRY-RUN]' || '' }}
— ${{ github.event.inputs.version || github.ref_name }}
# TRIGGERS — deliberately narrow. This workflow builds 4 platform
# binaries (~15-25 min wall-clock, matrix of native runners) and
# attaches them to a GitHub Release. Running on every push to main
# would be gratuitous CI cost + noise — the point is to package
# releases, not to sanity-check the tip. The bundled-binary
# integration test in ci.yml already covers "does the embed still
# work" on every PR.
on:
push:
tags:
- "v*"
workflow_dispatch:
inputs:
version:
description: 'Existing tag to package (e.g. v0.9.0). Must exist on origin.'
required: true
dry_run:
description: 'Dry run — build + upload tarballs as workflow artifacts, skip attaching to a Release. Use to smoke-test workflow edits without publishing.'
required: false
type: boolean
default: false
# Concurrency key includes the tag ref so different tags don't cancel
# each other; `cancel-in-progress: false` because tag builds are
# unique + immutable — a superseded release build has nothing to cancel.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: write
jobs:
# ── Stage 1: Build the SPA once, share across all platforms ─────────
#
# The SvelteKit build is arch-independent so a single ubuntu runner
# produces static-dist/ for every downstream binary-build matrix
# entry — saves ~2 min × 4 = 8 min vs building it per platform.
frontend-build:
name: Build SPA (Vite → static-dist/)
# Publish gate — same fork-friendly pattern as docker-publish.yml.
# Canonical repo always builds; forks stay quiet unless the fork
# owner opts in via `vars.ENABLE_BINARY_RELEASE=true` under Settings
# → Secrets and variables → Actions → Variables.
if: |
github.repository == 'AtalayaLabs/OxiCloud' ||
vars.ENABLE_BINARY_RELEASE == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# Build the exact tag being packaged. `github.ref` is
# refs/tags/vX.Y.Z on push, refs/heads/... on dispatch (we
# override via `inputs.version` in that case).
ref: ${{ github.event.inputs.version || github.ref }}
- uses: actions/setup-node@v4
with:
node-version: 26.3.0
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Build SPA
working-directory: frontend
run: npm ci && npm run build
- uses: actions/upload-artifact@v4
with:
name: static-dist
# Repo-root output (SvelteKit adapter-static's `pages:
# '../static-dist'`). Downstream jobs restore it to the same
# location so rust-embed's `#[folder = "static-dist/"]`
# resolves without any path juggling.
path: static-dist/
retention-days: 1
# ── Stage 2: Build one binary per target ────────────────────────────
#
# 3-way matrix — 2 musl-linux (native amd64 + arm64) + macOS Apple
# Silicon. Windows and Intel macOS are deferred:
#
# * Intel macOS (`x86_64-apple-darwin` / macos-13 runner) — dropped
# 2026-08-29. Apple is phasing out Intel Macs from GitHub's hosted
# runners; the `macos-13` tier is scheduled for deprecation and
# queues stretched past 1 h during v0.9.0-rc1 build. Intel Mac
# users have three fallbacks: (1) `cargo install oxicloud
# --locked --features bundled-assets` from source, (2) `docker
# pull --platform linux/amd64 ghcr.io/atalayalabs/oxicloud`,
# (3) any of the two Linux musl tarballs via a Linux VM. The
# Intel-Mac install base is small and shrinking (Apple Silicon
# >90% of new sales) so first-class shipping isn't worth the
# CI-availability tax.
# * Windows — separate work when demand appears.
#
# All targets run natively on GitHub-hosted runners with the host's
# glibc + rustup, then cross-compile to their target triple via
# `rustup target add`. The musl-linux targets install `musl-tools`
# (which provides `musl-gcc`) so aws-lc-sys and friends can link
# against musl. macOS runners already have the apple-* triples
# pre-installed.
#
# History: an earlier draft ran Linux builds INSIDE the
# `rust:1.96-alpine3.24` container the Dockerfile uses — matched
# Docker image byte-for-byte. Broke on `ubuntu-22.04-arm`: GitHub
# Actions JS-based actions (`actions/checkout`, artifact steps,
# setup-node) can't run inside Alpine on ARM64 — the Node.js binary
# they ship depends on glibc, and the x64-Alpine workaround doesn't
# extend to arm64. Native ubuntu + musl-tools sidesteps the whole
# thing and produces the same output (both are `cargo build
# --target x86_64-unknown-linux-musl` / `aarch64-...-musl`).
binary-build:
name: Build ${{ matrix.triple }}
needs: frontend-build
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
strategy:
# `fail-fast: false` — one platform's compile failure shouldn't
# cancel the other three. Partial releases are better than none.
fail-fast: false
matrix:
include:
- triple: x86_64-unknown-linux-musl
runner: ubuntu-22.04
rustflags: "-C target-cpu=x86-64-v2"
- triple: aarch64-unknown-linux-musl
runner: ubuntu-22.04-arm
# ARMv8-A baseline — covers Pi 4/5, Graviton, every 64-bit
# ARM Linux server. `generic` is rustc's neutral baseline.
rustflags: "-C target-cpu=generic"
- triple: aarch64-apple-darwin
runner: macos-latest
rustflags: "-C target-cpu=apple-m1"
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.version || github.ref }}
# musl-tools ships `musl-gcc` — the C-compiler wrapper that rustc
# invokes for -musl targets. Without it, `cargo build --target
# aarch64-unknown-linux-musl` fails with "linker `musl-gcc` not
# found" on any C-linked dep (aws-lc-sys, ring, sqlx's native
# backend when enabled).
- name: Install musl-tools (Linux only)
if: contains(matrix.triple, '-linux-musl')
run: sudo apt-get update && sudo apt-get install -y musl-tools
- uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.triple }}
- uses: Swatinem/rust-cache@v2
with:
# Key by triple so the 4 targets don't share caches
# (different feature set + different target triple = different
# compiled artefacts).
key: ${{ matrix.triple }}
- uses: actions/download-artifact@v4
with:
name: static-dist
path: static-dist/
# `--features bundled-assets` bakes static-dist/ into the binary
# via rust-embed. `--bin oxicloud` — the single binary the merge
# (Deliverable 1b) consolidated everything into.
- name: Build binary
env:
# Per-triple CPU baseline — release binaries target the widest
# realistic install base for their arch. See
# docs/plan/bundled-binary.md § 3.
RUSTFLAGS: ${{ matrix.rustflags }}
# Git metadata injection — build.rs reads these env vars to
# stamp GIT_HASH / GIT_BRANCH into the binary. Without them
# `oxicloud --version` reports "unknown".
GITHUB_SHA: ${{ github.sha }}
GITHUB_REF_NAME: ${{ github.ref_name }}
run: |
cargo build --release --features bundled-assets --bin oxicloud --target ${{ matrix.triple }}
# Assemble the tarball layout documented in
# docs/plan/bundled-binary.md § 4: oxicloud + example.env +
# LICENSE + README-install.md, rooted under a per-version-per-
# triple directory so `tar xzf` lands cleanly.
- name: Package tarball
run: |
set -euo pipefail
# Version = tag stripped of leading `v` (workflow_dispatch)
# or ref_name stripped (push tag). Falls back to ref_name
# verbatim if neither strip matches.
RAW_REF="${{ github.event.inputs.version || github.ref_name }}"
VERSION="${RAW_REF#v}"
DIST="oxicloud-${VERSION}-${{ matrix.triple }}"
mkdir -p "dist/${DIST}"
cp "target/${{ matrix.triple }}/release/oxicloud" "dist/${DIST}/oxicloud"
cp example.env "dist/${DIST}/example.env"
cp LICENSE "dist/${DIST}/LICENSE"
# README-install.md may not exist yet in early releases —
# ship a stub that points at the docs site so users have
# something in the tarball. Deliverable 6 replaces it with
# a proper install guide.
if [ -f docs/install/binary.md ]; then
cp docs/install/binary.md "dist/${DIST}/README-install.md"
else
cat > "dist/${DIST}/README-install.md" <<'MD'
# OxiCloud — Installation
Full documentation: https://github.com/AtalayaLabs/OxiCloud/tree/main/docs
Quickstart:
1. Set DATABASE_URL to a PostgreSQL 13+ instance
(with pg_trgm + ltree extensions).
2. Copy example.env → .env, edit as needed.
3. Run ./oxicloud.
Optional: install ffmpeg for server-side video thumbnails
(or set OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false to disable).
MD
fi
# Deterministic tar (owner/group/mtime pinned) so re-running
# the build produces byte-identical archives — helps with
# reproducible-build audits and cheap hash verification.
tar --owner=0 --group=0 -czf "dist/${DIST}.tar.gz" -C dist "${DIST}"
ls -la "dist/${DIST}.tar.gz"
- uses: actions/upload-artifact@v4
with:
name: tarball-${{ matrix.triple }}
path: dist/*.tar.gz
retention-days: 1
# ── Stage 3: Attach all tarballs + SHA256SUMS to the Release ────────
#
# `dry_run: true` (workflow_dispatch only) skips this job — the
# binary tarballs stay as workflow artifacts (accessible from the
# run page for 1 day) but nothing lands on any Release.
release:
name: Attach tarballs to GitHub Release
needs: binary-build
if: |
needs.binary-build.result == 'success' &&
(github.event_name != 'workflow_dispatch' || github.event.inputs.dry_run != 'true')
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
pattern: tarball-*
path: dist/
merge-multiple: true
- name: Compute SHA256SUMS
run: |
set -euo pipefail
cd dist
# Sort output for stable ordering across re-runs — the file
# doubles as a manifest an operator can `diff` between two
# release runs to prove they're identical.
sha256sum *.tar.gz | sort > SHA256SUMS
cat SHA256SUMS
# softprops/action-gh-release@v2 semantics:
# - If the Release for this tag EXISTS (created by release.yml
# which runs in parallel on the same tag push), attaches the
# files to it.
# - If it doesn't yet exist (race — release.yml still running),
# creates a bare Release which release.yml then fills in with
# notes when it finishes.
# Benign either way; see docs/plan/bundled-binary.md § 5
# "Parallel-fire behaviour on tag push".
- name: Attach to Release
uses: softprops/action-gh-release@v2
with:
files: |
dist/*.tar.gz
dist/SHA256SUMS
fail_on_unmatched_files: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Cargo.lock
Generated
+174 -6
View File
@@ -29,6 +29,12 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "adler32"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aae1277d39aeec15cb388266ecc24b11c80469deae6067e17a1a7aa9e5c1f234"
[[package]]
name = "adobe-cmap-parser"
version = "0.4.1"
@@ -601,7 +607,7 @@ dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
"h2 0.3.27",
"h2 0.4.13",
"h2 0.4.16",
"http 0.2.12",
"http 1.4.0",
"http-body 0.4.6",
@@ -1172,6 +1178,16 @@ dependencies = [
"tinyvec",
]
[[package]]
name = "bstr"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab"
dependencies = [
"memchr",
"serde",
]
[[package]]
name = "bumpalo"
version = "3.20.2"
@@ -1962,6 +1978,12 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "dary_heap"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b1e3a325bc115f096c8b77bbf027a7c2592230e70be2d985be950d3d5e60ebe"
[[package]]
name = "dashmap"
version = "6.2.1"
@@ -2884,6 +2906,19 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
[[package]]
name = "globset"
version = "0.4.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e47d37d2ae4464254884b60ab7071be2b876a9c35b696bd018ddcc76847309cd"
dependencies = [
"aho-corasick",
"bstr",
"log",
"regex-automata",
"regex-syntax",
]
[[package]]
name = "group"
version = "0.13.0"
@@ -2916,9 +2951,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.13"
version = "0.4.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54"
checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27"
dependencies = [
"atomic-waker",
"bytes",
@@ -3197,7 +3232,7 @@ dependencies = [
"bytes",
"futures-channel",
"futures-core",
"h2 0.4.13",
"h2 0.4.16",
"http 1.4.0",
"http-body 1.0.1",
"httparse",
@@ -3507,6 +3542,39 @@ dependencies = [
"quick-error",
]
[[package]]
name = "include-flate"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48f173716febb1ad596c16ea5637b5f1790ea32de8e627493ff82bc73b0876ce"
dependencies = [
"include-flate-codegen",
"include-flate-compress",
]
[[package]]
name = "include-flate-codegen"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a7875b62a72ad3f3203cdd8950d4cf9947db036030b974b8b37ceae90c8d8c0"
dependencies = [
"include-flate-compress",
"proc-macro-error3",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "include-flate-compress"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44fbb9c5ccb9a5b67b4afa2974c27e5507ea1bf6d22828cef418e4dfaeca51dd"
dependencies = [
"libflate",
"zstd",
]
[[package]]
name = "indexmap"
version = "1.9.3"
@@ -3802,6 +3870,30 @@ version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "libflate"
version = "2.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4da9b700e758e57152a1fd1c52cbdc5727c1aa6d8743dc1acda917398f1d76c"
dependencies = [
"adler32",
"crc32fast",
"dary_heap",
"libflate_lz77",
"no_std_io2",
]
[[package]]
name = "libflate_lz77"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff7a10e427698aef6eef269482776debfef63384d30f13aad39a1a95e0e098fd"
dependencies = [
"hashbrown 0.16.1",
"no_std_io2",
"rle-decode-fast",
]
[[package]]
name = "libloading"
version = "0.9.0"
@@ -4278,6 +4370,15 @@ dependencies = [
"rawpointer",
]
[[package]]
name = "no_std_io2"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "418abd1b6d34fbf6cae440dc874771b0525a604428704c76e48b29a5e67b8003"
dependencies = [
"memchr",
]
[[package]]
name = "nom"
version = "7.1.3"
@@ -4532,7 +4633,7 @@ dependencies = [
[[package]]
name = "oxicloud"
version = "0.8.7"
version = "0.8.9"
dependencies = [
"accept-language",
"aes-gcm",
@@ -4597,6 +4698,7 @@ dependencies = [
"rand_core 0.6.4",
"rayon",
"reqwest",
"rust-embed",
"serde",
"serde_json",
"sha2 0.11.0",
@@ -5006,6 +5108,28 @@ dependencies = [
"toml_edit",
]
[[package]]
name = "proc-macro-error-attr3"
version = "3.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b0084e6206a967a2dad822180626b2f6b07a3b379325e8f1ec0438e33a469ba7"
dependencies = [
"proc-macro2",
"quote",
]
[[package]]
name = "proc-macro-error3"
version = "3.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cf066225f2373bc711684792b69bdeac0356019b007e721090c24d92d5d5a50"
dependencies = [
"proc-macro-error-attr3",
"proc-macro2",
"quote",
"syn 3.0.2",
]
[[package]]
name = "proc-macro-utils"
version = "0.10.0"
@@ -5558,6 +5682,12 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "rle-decode-fast"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3582f63211428f83597b51b2ddb88e2a91a9d52d12831f9d08f5e624e8977422"
[[package]]
name = "rmp"
version = "0.8.15"
@@ -5597,6 +5727,44 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rust-embed"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9e7760e252aaba7b09f4be00e36476cf585bdb68a53552ac954cdf504ab4bc9"
dependencies = [
"include-flate",
"rust-embed-impl",
"rust-embed-utils",
"walkdir",
]
[[package]]
name = "rust-embed-impl"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3bcfc4d6f53af43755f7a723e4b6b8794fcce052a178dd8c6c1dadc5f5343097"
dependencies = [
"mime_guess",
"proc-macro2",
"quote",
"rust-embed-utils",
"syn 2.0.117",
"walkdir",
]
[[package]]
name = "rust-embed-utils"
version = "8.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42ffa149f6aa81b58a5b3011d01a857c4ed12c7a732d2c51947a4c7c692185f0"
dependencies = [
"globset",
"include-flate",
"sha2 0.11.0",
"walkdir",
]
[[package]]
name = "rust-stemmers"
version = "1.2.0"
@@ -7016,7 +7184,7 @@ dependencies = [
"axum",
"base64 0.22.1",
"bytes",
"h2 0.4.13",
"h2 0.4.16",
"http 1.4.0",
"http-body 1.0.1",
"http-body-util",
Cargo.toml
+90 -22
View File
@@ -1,16 +1,34 @@
[package]
name = "oxicloud"
version = "0.8.7"
version = "0.8.9"
edition = "2024"
default-run = "oxicloud"
# `cargo binstall oxicloud` — fetches the prebuilt release tarball for the
# host triple from GitHub Releases (attached by
# `.github/workflows/release-binaries.yml`) instead of compiling from
# source. Templates match the tarball naming
# `oxicloud-<version>-<triple>.tar.gz` produced by that workflow.
#
# `pkg-fmt = "tgz"` — otherwise binstall guesses from the URL extension;
# being explicit lets `cargo binstall oxicloud` succeed on Windows too
# (where the URL string parsing differs).
#
# Once the first tagged release lands on GitHub, this becomes a one-line
# install for anyone with the Rust toolchain who prefers not to build
# from source and doesn't want Docker either.
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/v{ version }/oxicloud-{ version }-{ target }.tar.gz"
pkg-fmt = "tgz"
bin-dir = "oxicloud-{ version }-{ target }/{ bin }{ binary-ext }"
[dependencies]
mimalloc = { version = "0.1.52", default-features = false }
axum = { version = "0.8.9", features = ["multipart", "http1", "http2", "tokio", "macros"] }
# "process" was previously enabled implicitly through aws-config's feature
# unification; ffmpeg_video_frame_service needs it, so declare it ourselves.
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs", "process"] }
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs", "process", "signal"] }
tokio-util = { version = "0.7.18", features = ["io", "codec", "compat"] }
tokio-stream = { version = "0.1.18", features = ["fs", "sync"] }
bytes = "1.11.1"
@@ -47,6 +65,25 @@ futures = "0.3.32"
async-stream = "0.3.6"
async-trait = "0.1.89"
mime_guess = "2.0.5"
# `rust-embed` — compile-time asset embedding for the `bundled-assets` feature.
# Optional so default builds never pull it in.
#
# Features:
# * `compression` — deflate-compress each embedded file at compile time; the
# handler decompresses lazily on first read (cached per-file in a
# `OnceCell`). Halves the on-disk contribution to the final binary.
# * `include-exclude` — enables the `#[include]` / `#[exclude]` glob
# attributes on the derive.
# * `debug-embed` — CRITICAL. Without this, debug builds read files from
# disk at runtime (dynamic mode) instead of compiling them in. The
# runtime read is fragile (`CARGO_MANIFEST_DIR` resolution + working
# directory dependency) and returned `total=0` for us on 2026-08-28.
# With `debug-embed`, both `cargo build` and `cargo build --release`
# produce a truly self-contained binary — the only sensible default
# for the `bundled-assets` feature.
#
# See src/interfaces/web/embedded.rs for the actual embed struct + handlers.
rust-embed = { version = "8", features = ["compression", "include-exclude", "debug-embed"], optional = true }
uuid = { version = "1.23.3", features = ["v4", "v7", "serde"] }
thiserror = "2.0.18"
arc-swap = "1.9"
@@ -168,6 +205,31 @@ faces-onnx = ["dep:ort", "dep:ndarray"]
# `examples/` can measure them. Off by default — adds nothing to prod builds.
# Run with: `cargo bench --features bench` / `cargo run --release --features bench --example bench_thumbnails_mem`.
bench = []
# Empty marker feature that gates the `generate-openapi` binary out of the
# default release build set. `just openapi` flips it when the SPA needs a
# regenerated openapi.json; end-user release builds never do. Kept separate
# from `test_utils` for the same reason `load_seed_bin` is — enabling
# `dev_tools` on the CLI must not perturb the oxicloud dependency graph.
dev_tools = []
# Bake the SvelteKit build output (`static-dist/` at repo root) into the
# binary at compile time via `rust-embed`. Opt-in and off by default — the
# regular `cargo build --release` still produces a filesystem-served
# binary (matching the current Docker image where assets are separate
# layers, and the `just dev` HMR loop where Vite serves live). Release
# tarballs (`docs/plan/bundled-binary.md` § 2) build with this flag to
# ship a single self-contained executable.
#
# Precedence rule preserved: even when this feature is on, if
# OXICLOUD_STATIC_PATH points at an existing directory, that wins over
# the embedded fallback — ops can override embedded assets for locale
# patches or theming without a rebuild.
#
# Build-time invariant: `cargo build --features bundled-assets` requires
# `static-dist/` at the repo root (SvelteKit adapter-static emits there
# — `frontend/svelte.config.js`'s `pages: '../static-dist'`). `build.rs`
# fails fast with a pointer to `(cd frontend && npm run build)` when the
# directory is missing.
bundled-assets = ["dep:rust-embed"]
[dev-dependencies]
criterion = "0.5"
@@ -186,19 +248,12 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(integration_tests)'] }
[[bin]]
name = "generate-openapi"
path = "src/bin/generate-openapi.rs"
[[bin]]
name = "migrate-nfc-filenames"
path = "src/bin/migrate-nfc-filenames.rs"
[[bin]]
name = "oxicloud-cli"
path = "src/bin/oxicloud-cli.rs"
# Operator toolbox — subcommand-driven CLI for tasks that don't belong
# in the main server. Currently: `oxicloud-cli opaque {setup,reset}`.
# Ships in the release Dockerfile as the single operator-facing helper
# (replaces the earlier per-task `opaque-setup` bin, which was folded
# into `oxicloud-cli opaque setup`).
# Dev-only: regenerates `resources/gen/openapi.json` from the utoipa
# `#[utoipa::path]` annotations in the API handlers. Gated behind the
# `dev_tools` feature so `cargo build --release --bins` (and the prod
# Dockerfile) skip it entirely — end users have no reason to run it.
# Invoked by `just openapi`, which passes `--features dev_tools`.
required-features = ["dev_tools"]
[[bin]]
name = "opaque-hurl-helper"
@@ -207,7 +262,10 @@ path = "src/bin/opaque-hurl-helper.rs"
# handshake against a running server. Invoked from tests/api/run.sh
# after opaque_substrate.hurl to cover the parts Hurl can't (OPRF
# blinding, AKE nonces are per-attempt-random). Not shipped in the
# release Dockerfile (nothing outside tests/ calls it).
# release Dockerfile (nothing outside tests/ calls it). Gated behind
# `test_utils` so `cargo build --release --bins` skips it; `run.sh`
# enables the feature explicitly when building the helper on demand.
required-features = ["test_utils"]
[[bin]]
name = "dpop-hurl-helper"
@@ -216,7 +274,8 @@ path = "src/bin/dpop-hurl-helper.rs"
# the DPoP-Nonce challenge/retry loop, and covers the wire-protocol
# scenarios Hurl can't express (per-request fresh jti/iat, replay
# detection, malformed proofs, wrong htm/htu/alg/typ). Same
# no-ship-in-release status as opaque-hurl-helper.
# no-ship-in-release status as opaque-hurl-helper — gated identically.
required-features = ["test_utils"]
[[bin]]
name = "load-seed"
@@ -938,11 +997,20 @@ opt-level = 1
debug = "line-tables-only"
split-debuginfo = "unpacked"
# Incremental compilation caches per-function IR fingerprints so a
# small edit only recompiles what changed. On a single-crate rebuild
# (oxicloud is one crate) the savings are modest — worth < the ~7 GB
# incremental/ cache costs on disk. Rust-analyzer uses `cargo check`,
# which has its own cache, so LSP responsiveness is unaffected.
incremental = false
# small edit only recompiles what changed. The `target/incremental/`
# cache costs ~7 GB per profile, but at the current codebase size a
# full rebuild is ~10 minutes and an incremental single-file edit is
# seconds — the disk is worth it and then some. Rust-analyzer's own
# `cargo check` cache is separate; LSP responsiveness is unaffected
# either way.
#
# History: this was `= false` early on when the crate was small and
# incremental's savings didn't cover the disk cost. Re-enabled
# 2026-08-29 as the full-rebuild time crossed the "feels annoying"
# threshold on typical dev-loop edits. If disk pressure ever spikes,
# `cargo clean -p oxicloud --profile dev` clears the incremental cache
# without wiping compiled deps.
incremental = true
[profile.bench]
lto = "fat"
Dockerfile
+23 -26
View File
@@ -42,11 +42,8 @@ COPY build.rs ./
# Create a minimal project to download and cache dependencies
RUN mkdir -p src/bin && \
echo 'fn main() { println!("Dummy build for caching dependencies"); }' > src/main.rs && \
echo 'fn main() {}' > src/bin/generate-openapi.rs && \
echo 'fn main() {}' > src/bin/migrate-nfc-filenames.rs && \
echo 'fn main() {}' > src/bin/oxicloud-cli.rs && \
echo 'fn main() {}' > src/bin/opaque-hurl-helper.rs && \
cargo build --release --bin oxicloud --bin generate-openapi --bin migrate-nfc-filenames --bin oxicloud-cli && \
cargo build --release --bin oxicloud && \
rm -rf src static-dist target/release/deps/oxicloud* target/release/build/oxicloud-*
# ─── Stage 3: Build the application ──────────────────────────────────────────
@@ -86,7 +83,7 @@ RUN DATABASE_URL="${DATABASE_URL}" \
GITHUB_SHA="${GITHUB_SHA}" \
GITHUB_REF_NAME="${GITHUB_REF_NAME}" \
GITHUB_HEAD_REF="${GITHUB_HEAD_REF}" \
cargo build --release --bin oxicloud --bin generate-openapi --bin migrate-nfc-filenames --bin oxicloud-cli
cargo build --release --bin oxicloud
# The SPA is built by the Vite frontend stage; bring it in for the runtime copy
# below (build.rs has no asset pipeline — it only injects git metadata).
COPY --from=frontend /static-dist ./static-dist
@@ -126,21 +123,24 @@ RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry,sharin
GITHUB_HEAD_REF="${GITHUB_HEAD_REF}" \
cargo build --release && \
mkdir -p /app/bin && \
cp target/release/oxicloud /app/bin/oxicloud && \
cp target/release/migrate-nfc-filenames /app/bin/migrate-nfc-filenames && \
cp target/release/oxicloud-cli /app/bin/oxicloud-cli
cp target/release/oxicloud /app/bin/oxicloud
# ─── Stage 3c: Select the builder & normalise the binary path ─────────────────
# FROM expands the global ${BUILDER} arg to alias the chosen builder stage
# (`builder` for CI/release, `builder-cache` for the e2e image). It then copies
# the two shipped binaries from the builder-specific ${BIN_DIR} into a single
# stable path (/app/release) so the runtime stage's COPYs are independent of
# which builder ran. `static-dist` already lives at /app/static-dist in both
# the shipped binary from the builder-specific ${BIN_DIR} into a single stable
# path (/app/release) so the runtime stage's COPY is independent of which
# builder ran. `static-dist` already lives at /app/static-dist in both
# builders, so it needs no normalisation.
#
# Single `oxicloud` binary since v0.9.0 — the operator toolbox
# (`opaque setup`, `migrate nfc-filenames`, …) now lives under
# `oxicloud <subcommand>` rather than in standalone `oxicloud-cli` /
# `migrate-nfc-filenames` bins. See docs/plan/bundled-binary.md § 1b.
FROM ${BUILDER} AS app
ARG BIN_DIR
RUN mkdir -p /app/release && \
cp "${BIN_DIR}/oxicloud" "${BIN_DIR}/migrate-nfc-filenames" "${BIN_DIR}/oxicloud-cli" /app/release/
cp "${BIN_DIR}/oxicloud" /app/release/
# ─── Stage 4: Minimal runtime image ──────────────────────────────────────────
FROM alpine:3.24.0
@@ -163,21 +163,18 @@ RUN apk --no-cache upgrade && \
addgroup -g 1001 -S oxicloud && \
adduser -u 1001 -S oxicloud -G oxicloud
# Copy the compiled binary and entrypoint (--chmod avoids extra RUN chmod layers)
# Copy the compiled binary and entrypoint (--chmod avoids extra RUN chmod layers).
#
# Single `oxicloud` binary — since v0.9.0 the operator toolbox lives
# under `oxicloud <subcommand>` rather than as standalone helper bins:
#
# docker run --rm <image> oxicloud opaque setup # print OPAQUE ServerSetup
# docker exec <container> oxicloud migrate nfc-filenames --dry-run
# # NFC-normalize storage.files.name (pre-June-2026 dbs; safe on new installs)
#
# Bare `oxicloud` (Docker CMD default) still starts the server — backwards
# compat preserved. See docs/plan/bundled-binary.md § 1b.
COPY --from=app --chmod=755 /app/release/oxicloud /usr/local/bin/
# Ship the NFC filename migration binary alongside the server so
# operators can run it inside the container without a separate Rust
# toolchain — `docker exec <container> migrate-nfc-filenames --dry-run`
# to preview, drop `--dry-run` to execute. One-shot tool, safe to
# ship; it only mutates `storage.files` rows whose name ≠ NFC(name).
COPY --from=app --chmod=755 /app/release/migrate-nfc-filenames /usr/local/bin/
# Ship the OPAQUE server-setup generator alongside the server so operators
# can generate their `OXICLOUD_AUTH_OPAQUE_SERVER_SETUP` value inside the
# container without a separate Rust toolchain:
# docker run --rm <image> oxicloud-cli opaque setup # prints the base64 value
# One-shot, side-effect-free — safe to include; the runtime doesn't
# invoke it, admins do (see docs/config/authentication.md §OPAQUE).
COPY --from=app --chmod=755 /app/release/oxicloud-cli /usr/local/bin/
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN sed -i 's/\r//' /usr/local/bin/entrypoint.sh && \
chmod 755 /usr/local/bin/entrypoint.sh
README.md
+10
View File
@@ -68,6 +68,16 @@ docker compose up -d
Open `http://localhost:8086`.
### Prebuilt binary
Binary releases (Linux musl amd64/arm64, macOS Intel/Apple Silicon)
are attached to every tagged release on GitHub — the whole SPA + all
operator subcommands + migrations bake into a single self-contained
executable. See [`docs/install/binary.md`](docs/install/binary.md) for
the download / verify / systemd walkthrough.
`cargo binstall oxicloud` works too once a release is out.
### Run from source
Requires Rust 1.93+ and PostgreSQL.
build.rs
+46 -3
View File
@@ -1,15 +1,58 @@
//! build.rs — injects git build metadata into the binary.
//! build.rs — injects git build metadata into the binary and, under the
//! optional `bundled-assets` feature, guards the compile-time embed
//! precondition.
//!
//! Exposes `GIT_HASH` and `GIT_BRANCH` (consumed via `env!()` in `main.rs`).
//! There is no Rust-side asset pipeline: the frontend is built by Vite into
//! `static-dist/` and served directly by the web layer (`interfaces::web`).
//! The frontend is built by Vite into `static-dist/` at the repo root and
//! served directly by the web layer (`interfaces::web`); when
//! `bundled-assets` is on, `src/interfaces/web/embedded.rs` bakes that
//! directory into the binary at compile time via `rust-embed`.
use std::env;
use std::path::Path;
use std::process::Command;
fn main() {
println!("cargo:rerun-if-changed=build.rs");
git_status();
bundled_assets_guard();
}
// ═══════════════════════════════════════════════════════════════════════════════
// Bundled-assets precondition guard
//
// When `--features bundled-assets` is on, `rust-embed`'s `#[folder = "static-dist/"]`
// scans that directory at compile time and errors with a not-very-helpful
// "No such file or directory" if it's missing. Users hit this first when they
// try `cargo build --release --features bundled-assets` before running the
// frontend build — we intercept it here with a clear, actionable message.
//
// Also emits `cargo:rerun-if-changed=static-dist/` so a fresh frontend build
// re-triggers the embed step without needing `cargo clean` — matches what a
// dev on the bundled feature would expect after `just fe-build`.
// ═══════════════════════════════════════════════════════════════════════════════
fn bundled_assets_guard() {
if env::var("CARGO_FEATURE_BUNDLED_ASSETS").is_err() {
return;
}
println!("cargo:rerun-if-changed=static-dist");
let manifest_dir = env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR");
let dist = Path::new(&manifest_dir).join("static-dist");
let index = dist.join("index.html");
if !index.exists() {
// `cargo:warning=` prefixes surface these in the terminal even
// when cargo's default output is quiet; the panic below turns
// them into a compile-time error so the missing prerequisite
// can't slip past a distracted dev.
println!("cargo:warning=`bundled-assets` feature requires static-dist/ at the repo root.");
println!("cargo:warning=Build the SvelteKit SPA first: (cd frontend && npm run build)");
println!("cargo:warning=Or via the workspace shortcut: just fe-build");
panic!(
"build.rs: missing {}/index.html — see the cargo:warning lines above",
dist.display()
);
}
}
// ═══════════════════════════════════════════════════════════════════════════════
clippy.toml
+28
View File
@@ -0,0 +1,28 @@
# Clippy configuration overrides. Kept minimal — each entry documents
# what it's for and when it should be revisited.
# `clippy::result_large_err` — raise the "big Err variant" ceiling to
# 512 bytes.
#
# Rationale: axum handler signatures shaped as
# `Result<impl IntoResponse, impl IntoResponse>` (or `AppError` variants
# that wrap `axum::response::Response`) naturally exceed the default
# 128-byte threshold. `Response` carries a `HeaderMap` (~256 B inline)
# + status + body + extensions; a handful of handlers land in that
# range without doing anything wrong. Fighting the lint per-handler
# with `#[allow]` on every one is churn for zero runtime benefit —
# these Results are constructed on the stack once per request and
# never nested in a hot inner loop.
#
# 512 B keeps the lint's protective value: it still fires on genuinely
# oversized Err variants (embedded `Vec<u8>` blobs, avatar payloads,
# large enum aggregates) that WOULD be worth boxing.
#
# Revisit if:
# * A future refactor slims axum Response OR extracts a small error
# enum with an IntoResponse impl across the handler layer — then
# drop this override back to the default 128.
# * A specific handler exceeds 512 B and clippy re-fires — deal with
# that handler individually (boxed error / small enum) rather than
# raising the ceiling further.
large-error-threshold = 512
docs/.vitepress/config.mts
+1
View File
@@ -170,6 +170,7 @@ export default defineConfig({
{ text: "Storage Quotas", link: "/architecture/storage-quotas" },
{ text: "Backend Storage", link: "/architecture/backend-storage" },
{ text: "File and Blob lifecycle", link: "/architecture/file-and-blob-lifecycle" },
{ text: "Derived and attached blobs", link: "/architecture/derived-and-attached-blobs" },
{ text: "ReBAC & Authorization", link: "/architecture/rebac-authorization" },
{ text: "User lifecycle", link: "/architecture/user-lifecycle" },
{ text: "Authentication model", link: "/architecture/auth-model" },
docs/architecture/backend-storage.md
+28 -17
View File
@@ -26,7 +26,7 @@ probe, and lifecycle behaviour are uniform.
Every entry is declared in `OXICLOUD_STORAGE_ENTRIES` (comma-separated
list of names). The active entry is stored in `admin_settings` and
switched via `oxicloud --select-storage <name>` on the command line
switched via `oxicloud storage select <name>` on the command line
or automatically at the end of a successful `backend_migration`.
Non-active entries stay reachable through the multi-entry API (test,
audit, migrate-into).
@@ -119,7 +119,7 @@ Rendered visually via `xxd -l 15 <blob>`:
Fingerprints are rendered the same colon-hex form (`15:f3:…:50`)
everywhere they appear: boot log, admin panel pair chain, `xxd`
inspection, `oxicloud --fingerprint <base64>` CLI, and the rotate /
inspection, `oxicloud storage fingerprint <base64>` CLI, and the rotate /
migration audit lines. That means an admin can cross-reference by
eye — same string means same key.
@@ -234,27 +234,38 @@ can be safely dropped.
---
## 4. Blob consistency (`blobs_consistency`)
## 4. Blob consistency — two jobs, split by what they read
Read-only recoverable job that walks `storage.blobs` and reports
divergence between the DB registry and the physical backend.
The registry side and the physical side are separate tenants. They
used to be one, with `blobs_consistency` probing the backend once per
row; that probe found strictly less than the merge-join below, at N
round-trips instead of one enumeration, so it was removed.
### Shallow mode (default)
### `blobs_consistency` — database only
Per row:
Walks `storage.blobs` and compares `ref_count` against the reference
count computed from `storage.files.blob_hash` +
`chunk_manifests.chunk_hashes[]`. On mismatch: `refcount_mismatch`
(severity `inconsistent`), repairable under `?repair=true`.
- `blob_exists(hash)` on the active backend → if false, record
`blob_missing_from_backend` (severity `data_loss`)
- Compare `ref_count` against the actual reference count computed
from `SUM` over `storage.files.blob_hash` + `chunk_manifests.chunk_hashes[]`
→ if mismatch, record `refcount_mismatch` (severity `inconsistent`)
It opens no backend and makes no network call. `?storage=<name>` and
`?deep=true` are inert. Cost is one aggregate SQL per row.
Cost: one existence probe + one aggregate SQL per row. Fast on
S3/Azure (single HEAD).
### `backend_consistency` — everything physical
### Deep mode (`?deep=true`)
Merge-joins the backend's enumeration against `storage.blobs`, both
ordered by hash, yielding both deltas in one pass:
Adds a full read of every blob:
- bytes with no registry row → `orphan_blob` (severity `inconsistent`)
- a registry row with no bytes → `blob_missing_from_backend`
(severity `data_loss`)
`?storage=<name>` scopes it to any declared entry rather than the live
backend.
### Deep mode (`?deep=true`, on `backend_consistency`)
For every hash present on both sides, adds a full read:
- Stream the blob through `EncryptedBlobBackend::get_blob_stream`
(strips header, decrypts if needed, applies BLAKE3 rescue for
@@ -336,7 +347,7 @@ readonly (source stays active — writes safe there), and returns
`RunOutcome::Failed`. Operator inspects findings, then either
retries (walk short-circuits on head-format matches → cheap
re-attempt), fixes the source, or explicitly accepts the partial
via `oxicloud --select-storage <target>`.
via `oxicloud storage select <target>`.
---
docs/architecture/caching.md
+63 -4
View File
@@ -1,8 +1,13 @@
# Caching Architecture
OxiCloud uses **moka** (a lock-free, concurrent cache) for write-behind caching that delivers sub-millisecond hot reads.
OxiCloud runs two independent cache layers with different jobs:
## Cache Layers
1. **In-memory metadata caches** — [moka](https://docs.rs/moka) instances that sit in front of PostgreSQL and other hot-path lookups. Sub-millisecond hits, bounded by entry count, TTL-evicted. Cover file metadata, directory listings, blob hashes, audio metadata, small thumbnails, on-the-fly image transcodes.
2. **On-disk blob cache** — an LRU cache of blob **bytes** on local SSD, only meaningful when the storage backend is remote (S3, Azure). Turns remote fetches into local reads for hot content; bounded by a disk-budget in bytes, LRU-evicted. Off by default.
The two layers are orthogonal — the moka caches shave query round-trips regardless of backend; the disk-blob cache shaves network round-trips when the backend is elsewhere.
## Layer 1 — In-memory metadata caches (moka)
| Cache | TTL | Max Entries | Purpose |
|---|---|---|---|
@@ -13,21 +18,63 @@ OxiCloud uses **moka** (a lock-free, concurrent cache) for write-behind caching
| Blob hash | 30 s TTI | 5 000 | BLAKE3 hashes for dedup lookups |
| Audio metadata | — | 2 000 | ID3 tags and duration |
## How It Works
### How it works
1. **Read path:** check cache → if hit, return immediately (sub-ms); if miss, query PostgreSQL, populate cache, return
2. **Write path:** update PostgreSQL → invalidate relevant cache entries
3. **TTL expiry:** entries are evicted after their time-to-live, ensuring eventual consistency
## Why moka?
### Why moka?
- **Lock-free** — no mutex contention under concurrent access
- **Bounded memory** — max entries prevent unbounded growth
- **TTL + TTI** — supports both time-to-live and time-to-idle eviction
- **Async-ready** — works natively with Tokio
## Layer 2 — On-disk blob cache
A local-SSD LRU cache of blob bytes, sitting between OxiCloud and remote storage backends (S3, Azure, or any other `BlobStorageBackend`). Every blob read probes the local cache first; misses fetch from the remote backend and populate the cache. Writes go to the remote backend AND the local cache simultaneously, so a just-uploaded blob is immediately hot for its own re-reads.
Structurally: the bytes live on disk, one `.blob` file per hash, sharded by hash prefix under a configurable directory (default `{root}/.blob-cache/<prefix>/<hash>.blob`). The in-process index is a `moka::sync::Cache` with a byte-weigher — same crate as Layer 1, but weighing by content size not entry count, and only tracking file existence, not payload.
### When it earns its keep
Turn on for any deployment where the backend is not on the same box:
- S3 (AWS, DigitalOcean Spaces, Cloudflare R2, MinIO on another host, …)
- Azure Blob Storage
- Any future network-attached backend
Local backends (`LocalFilesystem`) don't need it — they're already on the same box. Enabling it there just doubles disk usage for zero latency win.
**Thumbnails are the strongest reason to turn this on.** OxiCloud stores thumbnails as blobs alongside primary content (via `content_derived_blobs`, tracked in `storage.blobs` like any other blob) — the sidecar-on-disk layout is gone. On a remote backend this means every thumbnail render is a network fetch: a photos grid with 100 thumbnails is 100 S3 requests, per user, per visit. With Layer 2 on, that cost is paid once per thumbnail hash; every subsequent grid render is local-disk reads.
Concrete impact for the photos / file-listing hot paths:
- **Cold render** (all thumbnails uncached): one remote fetch per thumbnail, latency dominated by the backend's per-request round-trip (S3 typically 30-80 ms per object, more at distance).
- **Warm render** (thumbnails cached): local `open()` + read, sub-millisecond per file.
- **Hit rate in practice**: high — thumbnails are small (typically 5-30 KB per size variant), users re-visit the same folders repeatedly, and the LRU pattern strongly favours recency.
Rule of thumb: if your backend is remote AND you have any user-facing photo grid or file browser, Layer 2 is worth the disk budget. On S3 backends it's the difference between a snappy gallery and a spinner-per-tile browsing experience.
### Sizing guidance
The cache is LRU on a disk-budget basis. A working set larger than the cache size will still work but re-fetch cold blobs from the remote — no correctness cost, just latency. Rough sizing:
- **Home / personal cloud** — 5-10 GB is plenty; the working set for a household of active users is small.
- **Small team / SMB** — 50-100 GB for a hot photo library or shared document store.
- **Large deployment** — size against your top-decile access pattern; the cache doesn't need to cover the whole store.
The default budget is 50 GB (only applied if the cache is enabled). Adjust to what your local SSD can spare.
### Interaction with the moka layer
Independent. A file-metadata hit in Layer 1 tells you the row exists and has a `blob_hash` — but reading the actual bytes still goes through Layer 2 (or straight to the remote backend if disabled). A hit in Layer 2 short-circuits the network fetch; a miss populates it for the next read.
## Configuration
### In-memory metadata caches (Layer 1)
Cache parameters are currently hardcoded in `src/common/config.rs`. Key defaults:
```rust
@@ -35,3 +82,15 @@ file_cache_ttl_ms: 60_000, // 1 minute
directory_cache_ttl_ms: 120_000, // 2 minutes
max_cache_entries: 10_000,
```
### On-disk blob cache (Layer 2)
Environment-tunable — off by default; enable per deployment when the backend is remote:
| Env var | Default | Purpose |
|---|---|---|
| `OXICLOUD_STORAGE_CACHE_ENABLED` | `false` | Master switch. Set `true` to wrap the blob backend with the cache decorator. |
| `OXICLOUD_STORAGE_CACHE_MAX_SIZE` | `53687091200` (50 GB) | Disk-budget in bytes. LRU eviction fires when the cache exceeds this size. |
| `OXICLOUD_STORAGE_CACHE_PATH` | `{root}/.blob-cache` | Where the cache files live. Point at a fast SSD; can be a separate volume from the primary storage root. |
Restart the server after changing any of these — the cache is instantiated once at boot around the configured blob backend.
docs/architecture/derived-and-attached-blobs.md
+284
View File
@@ -0,0 +1,284 @@
# Derived and attached blobs
Two tables hang small artifacts off the blob store: thumbnails,
transcodes, uploaded previews. They look almost identical, and the
difference between them is a security boundary rather than a style
choice.
("Satellite tables" is the shorthand used in the code and in
`satellites_consistency`, the job that walks both. It is a useful
collective noun once you know what it covers; this page is what it
covers.)
- **`storage.content_derived_blobs`** — things the *server derived from
file content*. Keyed by the BLAKE3 of the source.
- **`storage.file_attached_blobs`** — things a *user attached to one
specific file*. Keyed by `file_id`.
Both point into the same blob store underneath (see
[Backend Storage](./backend-storage.md)). The keying is what separates
them.
```mermaid
erDiagram
FILES ||--o{ ATTACHED : "file_id — per FILE"
FILES }o--|| BLOBS : "blob_hash (its content)"
BLOBS ||--o{ DERIVED : "source_hash — per CONTENT"
DERIVED }o--o| ARTIFACT : "blob_hash (NULL = negative)"
ATTACHED }o--|| ARTIFACT : "blob_hash"
FILES {
uuid id PK
text blob_hash
}
DERIVED {
text source_hash PK
text kind PK
text variant PK
text blob_hash "NULL = not worth deriving"
text content_type "NULL iff blob_hash NULL"
}
ATTACHED {
uuid file_id PK
text kind PK
text variant PK
text blob_hash
uuid uploaded_by "no FK; nil = imported"
}
ARTIFACT {
text hash PK
}
```
Read the two arrows into `ARTIFACT`: `DERIVED` reaches it from
**content**, `ATTACHED` from a **file**. Everything below follows from
that.
## Why two tables and not one with a `kind` column
Content keying means identical bytes share one derivation. Upload the
same photo twice and the server renders its thumbnail **once** — both
files resolve to the same `source_hash`, find the same row, and serve
the same blob. That is the entire point for server-derived artifacts:
the derivation is a pure function of the content, so sharing it is
free and correct.
Apply the same keying to *user-supplied* bytes and it becomes an
attack. If uploaded previews were content-keyed, uploading a file whose
content matches someone else's would let you replace the preview they
see — or read yours in place of theirs. The preview is not derived from
the content; it is an assertion *about* a file, made by whoever uploaded
it, and two people can hold different assertions about identical bytes.
A single table with a `kind` discriminator could not express this. The
key would have to be one thing or the other, and either choice is wrong
for half the rows. The split is the enforcement mechanism, not a
filing convenience — which is why `thumb_derived_import` explicitly
refuses `ext-` filenames and `thumb_attached_import` explicitly refuses
hash-named ones, rather than one job handling both trees.
## `storage.content_derived_blobs`
| column | type | notes |
|---|---|---|
| `source_hash` | `VARCHAR(64)` | PK. BLAKE3 of the **source** content. Dependent reference — holds no refcount; the row is reaped with its source. |
| `kind` | `TEXT` | PK. `thumbnail` \| `transcode` (CHECK-constrained). |
| `variant` | `TEXT` | PK. Opaque rendering discriminator — see [Variant](#variant-holds-every-axis-that-can-vary). |
| `blob_hash` | `VARCHAR(64)` | The derived artifact, **or NULL** — see [Negative rows](#negative-rows). Reference **holder** when present. |
| `content_type` | `TEXT` | MIME of the artifact. NULL exactly when `blob_hash` is NULL. |
| `created_at` | `TIMESTAMPTZ` | |
A CHECK keeps `blob_hash` and `content_type` NULL together: a type
without bytes describes nothing, and bytes without a type cannot be
served.
## `storage.file_attached_blobs`
| column | type | notes |
|---|---|---|
| `file_id` | `UUID` | PK. FK to `storage.files` **ON DELETE CASCADE**. |
| `kind` | `TEXT` | PK. `preview` \| `subtitle` \| `cover_art` (CHECK-constrained). |
| `variant` | `TEXT` | PK. |
| `blob_hash` | `VARCHAR(64)` | `NOT NULL` — there is no negative case here. |
| `content_type` | `TEXT` | `NOT NULL`. |
| `uploaded_by` | `UUID` | `NOT NULL`, and deliberately **no FK**. |
| `created_at` | `TIMESTAMPTZ` | |
`uploaded_by` follows the provenance convention: an FK with
`ON DELETE SET NULL` would erase the audit trail exactly when it matters
most, and without an `ON DELETE` clause it would block deleting a user
at all. Deleting the uploader must not rewrite history, so the id is
kept even once it no longer resolves. Rows created by the migration
carry the all-zeros sentinel — "imported, uploader unknown" — rather
than a fabricated owner such as the file's `created_by`, which could
later be misread as evidence that someone replaced a preview.
## Negative rows
`content_derived_blobs.blob_hash` is nullable, and a NULL row means:
**this derivation was attempted and is known not to be worth storing
for this content.**
The case that motivated it: `ImageTranscodeService` can only discover
that WebP comes out *larger* than the original by doing the full decode
and re-encode. Without a record, every request repeats that work to
throw the result away. The same applies to a source that cannot be
decoded, or one over the decode ceiling.
Only failures **deterministic in the content** may be recorded. A
timeout, a closed semaphore, an I/O error reading the source are
properties of the moment, not the bytes; persisting one marks a
perfectly good image as underivable forever, with nothing to retry it.
The asymmetry sets the default — a wrongly-cached transient is silent
and permanent, a missing negative merely costs repeated work — so **when
in doubt, do not write the row.**
A sentinel hash was considered and rejected: it would stop `blob_hash`
naming a real blob, and every consumer joining on it would need to learn
the exception or silently mishandle it. NULL is already SQL's way of
saying "no blob", and joins drop it naturally.
`file_attached_blobs` has no negative case. There is nothing to attempt
— the bytes either arrived from a client or they did not.
### The NULL trap
This has caused two bugs, both found before shipping, and it will cause
more. SQL comparison against NULL yields NULL, so:
```sql
EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.blob_hash)
```
is **false** for every negative row. Whether that is right depends
entirely on what you are asking:
- **Refcounts — correct.** A negative row holds no reference, so it must
not contribute. `content_derived_ref_sql` relies on exactly this.
- **Dangling checks — wrong.** `satellites_consistency` reported every
negative row as `derived_dangling_blob` at `data_loss` severity: a row
correctly pointing at nothing, reported as an artifact that had gone
missing. It needs `d.blob_hash IS NULL OR <exists>`.
- **Enumeration — wrong, and it fails loudly.**
`list_referenced_blobs` decodes `blob_hash` into `String`; the first
NULL takes the whole sweep down. It needs `WHERE blob_hash IS NOT NULL`.
Anything joining on `blob_hash` has to decide which of these it is.
## `variant` holds every axis that can vary
`variant` is an opaque discriminator, and the schema states the rule:
new axes go **inside this string, never into new columns**.
The two tables therefore look asymmetric, and correctly so:
| table | variant | why |
|---|---|---|
| `content_derived_blobs` | `icon.webp`, `preview.jpg` | size **and** format |
| `file_attached_blobs` | `icon` | size only |
A single source legitimately has two thumbnails at one size — WebP for
capable clients, JPEG for the rest — and since the PK is
`(source_hash, kind, variant)`, the format must be inside `variant` or
those rows collide and only one can exist. Uploaded previews have no
format axis: `store_external_thumbnail` re-encodes to JPEG on write, so
`image/jpeg` is a constant and `.jpg` in every variant would carry no
information.
**Rejected alternative: key on `content_type` instead.** Making
`content_type` part of the key would express the same thing, and it was
reasonable until negative rows landed. It is now foreclosed —
`content_type` must be nullable for negative rows, and PostgreSQL does
not allow a nullable column in a primary key. A UNIQUE constraint would
not rescue it either: NULLs compare as *distinct* in a unique index, so
duplicate negative rows for one `(source_hash, kind, variant)` would
become possible, and that row's singularity is what the mechanism
depends on. Two softer objections stand regardless — `content_type` is a
presentation value, and MIME strings are not canonical (`image/jpg` and
`image/jpeg` name one format).
**If the assumption changes**, the migration has a known shape.
`20261022000000_derived_variant_encodes_format.sql` did it once for the
derived table: append the format to existing variants, update the
callsites that build the string. Nothing is lost in the meantime —
`content_type` records the real format — it simply is not part of the
key, so two formats cannot coexist until it is.
## Worked examples
**The same image uploaded twice.** Two `storage.files` rows, one
`blob_hash` between them, **one** `content_derived_blobs` row per
`(kind, variant)`, one thumbnail blob. The second upload renders
nothing; it finds the existing row. Copying either file adds no row at
all — the copy shares the source hash, so it resolves to the same
derivation.
**A PDF with a client-uploaded preview.** One `file_attached_blobs` row
keyed by that `file_id`. Copy the file and the row is **duplicated** for
the new id (`storage.copy_file_satellites`), because the preview belongs
to the file, not to the content. Without that duplication the copy
silently loses its preview — and a PDF has no server-side render path,
so nothing regenerates it.
**A screenshot WebP cannot shrink.** One row, `blob_hash` and
`content_type` both NULL. A reader concludes: the transcode was
attempted, it is known not to help *for this content*, serve the
original and do not retry. Every file sharing those bytes inherits the
verdict.
**One source, thumbnailed and transcoded.** Two rows, same
`source_hash`, `kind` of `thumbnail` and `transcode`. Add a JPEG
fallback thumbnail and it is a third row, differing only in `variant`
(`icon.webp` vs `icon.jpg`).
## Lifecycle
**References.** A positive `blob_hash` is a reference *holder* — it
bumps `chunk_manifests.ref_count` through `DedupService::add_reference`,
so `dedup_gc` cannot reap an artifact a satellite still points at. A
negative row holds none. `source_hash` is a *dependent* reference and
holds nothing.
**Reaping.** Derived rows are removed by `purge_derived_blobs` when
their source blob is reaped. Attached rows vanish by
`ON DELETE CASCADE` when their file is deleted — which happens **inside
the database**, where the Rust lifecycle hooks cannot observe it, so
`trg_file_attached_blobs_decrement_blob_ref` releases the blob reference
on `DELETE`. The trigger fires on DELETE only; replacing a preview
updates `blob_hash` in place and the Rust path handles that reference
swap.
**Writing a derived row requires its source to exist.**
`store_derived_blob` guards the insert with an `EXISTS` on
`chunk_manifests`/`blobs`. Without it, a row written just after its
source was reaped would pin its artifact forever: nothing would ever
reap that `source_hash` again, so `purge_derived_blobs` could never
fire. That is not hypothetical — it shipped once, as a permanent blob
leak.
**Consistency coverage.** `satellites_consistency` is the only job that
walks these tables, and it exists because of a gap the others cannot
close: a satellite row whose *source* is gone breaks no invariant any
other check looks at. The reference is valid, the refcount is correct,
the bytes are present — every Blob-centric job agrees the system is
healthy while the artifact is pinned forever. Blob-side integrity
(missing bytes, orphans, bit-rot) belongs to `backend_consistency`;
refcount arithmetic to `blobs_consistency` and
`manifests_consistency`.
## Adding a third artifact type
Ask one question first: **is it derived from the content, or asserted
about a file?**
Derived from content — a waveform, an extracted page count, an OCR
layer — goes in `content_derived_blobs` under a new `kind`, and shares
across identical content for free.
Supplied by a user — a custom cover image, a hand-authored subtitle
track — goes in `file_attached_blobs`, and must be duplicated on copy
rather than shared.
Getting that backwards is not a performance mistake. Putting
user-supplied bytes in the content-keyed table means one user's upload
is served to everyone whose file happens to match.
docs/architecture/index.md
+1
View File
@@ -75,4 +75,5 @@ src/
- [Resource Listing API →](/architecture/resource-listing)
- [Storage Quotas →](/architecture/storage-quotas)
- [Backend Storage →](/architecture/backend-storage)
- [Derived and Attached Blobs →](/architecture/derived-and-attached-blobs) — thumbnails, transcodes and uploaded previews: why content-keyed and file-keyed artifacts need separate tables
- [Background Jobs →](/architecture/jobs)
docs/config/admin-settings.md
+7 -3
View File
@@ -93,19 +93,23 @@ Runs are recoverable — status, cursor, and per-blob failure findings all live
If an entry is renamed or removed from `.env` while the DB pointer still names the old one, boot aborts with a clear error pointing at:
```
oxicloud --select-storage <name>
oxicloud storage select <name>
```
This one-shot repair command re-runs the same env-parse the server does at boot, verifies `<name>` is declared in `OXICLOUD_STORAGE_ENTRIES`, updates `admin_settings.storage.active_backend_name` in the DB, and exits. Operator then restarts normally. See [Environment Variables — Storage Entries](/config/env#storage-entries-multi-entry-recommended) for the model, and [`oxicloud --help`](https://github.com/oxicloud/oxicloud/blob/main/src/main.rs) for the full flag list.
### Auditing entries other than the active one
`blobs_consistency` and `backend_consistency` (recoverable jobs on the Jobs tab) accept `?storage=<name>` to probe any declared entry — not just the live one. Use this to verify a migration target before cutover, or to audit an old backend after cutover but before decommissioning:
`backend_consistency` (a recoverable job on the Jobs tab) accepts `?storage=<name>` to audit any declared entry — not just the live one. Use this to verify a migration target before cutover, or to audit an old backend after cutover but before decommissioning:
```
POST /api/admin/jobs/blobs_consistency/trigger?storage=<name>
POST /api/admin/jobs/backend_consistency/trigger?storage=<name>
```
Add `?deep=true` to also read every blob back and re-hash it, which catches silent bit-rot. That is a full read of the entry and can take hours.
`blobs_consistency` does *not* accept `?storage=<name>`: it only reads the database, so there is no entry for it to scope.
Unknown names 400 at the HTTP layer.
## Data Storage
docs/config/authentication.md
+3 -3
View File
@@ -135,12 +135,12 @@ Password-using deployments will opt in via three env vars:
2. **`OXICLOUD_AUTH_OPAQUE_SERVER_SETUP`** — generated once and persisted like your JWT secret. Rotating this invalidates every user's registration; treat it as one of the crown jewels. Two ways to generate:
```bash
# Docker (recommended in production — no toolchain needed):
docker run --rm ghcr.io/atalayalabs/oxicloud:latest oxicloud-cli opaque setup
docker run --rm ghcr.io/atalayalabs/oxicloud:latest oxicloud opaque setup
# From a source checkout:
cargo run --bin oxicloud-cli -- opaque setup
cargo run --bin oxicloud -- opaque setup
```
Both print the base64 value on stdout (with guidance on stderr, so shell pipelines like `$(docker run ... oxicloud-cli opaque setup)` capture cleanly).
Both print the base64 value on stdout (with guidance on stderr, so shell pipelines like `$(docker run ... oxicloud opaque setup)` capture cleanly).
3. **`OXICLOUD_AUTH_OPAQUE_KSF_*`** — client-side Argon2id key-stretching cost. Defaults (46 MiB / 1 iter / 1 lane) match OWASP's interactive-auth recommendation. See the next section for the rationale + when to bump.
The `OXICLOUD_HASH_*` variables (server-side legacy Argon2) and `OXICLOUD_AUTH_OPAQUE_KSF_*` (client-side OPAQUE Argon2) are intentionally separate: the server-side path is RAM-bounded by concurrent-login traffic and needs to stay modest; the client-side path is single-user per attempt and can be tuned independently. Tuning them together would force a bad compromise in one direction or the other.
docs/config/env.md
+5 -2
View File
@@ -18,6 +18,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Direct (non-chunked) uploads stream straight into the blob store and need no spool directory. Placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
| `OXICLOUD_METRICS_LISTEN` | (unset) | Prometheus `/metrics` listener address (e.g. `127.0.0.1:9090`, IPv6 allowed as `[::1]:9090`). **Unset = disabled**: no `/metrics` endpoint is bound and no metrics recorder is installed (zero runtime cost). When set, a separate HTTP listener on this address serves the text-format scrape. **Deliberately NOT merged into the main API** — no auth, CSRF, or DPoP layer in front. Bind to loopback or a private interface unless you intend to expose metrics publicly. Starter counters: `oxicloud_dpop_verify_failed_total{reason}`, `oxicloud_dpop_proof_missing_total`, `oxicloud_dpop_header_missing_on_bound_session_total`, `oxicloud_dpop_replay_detected_total`, `oxicloud_dpop_nonce_challenges_issued_total`. |
| `OXICLOUD_STARTUP_JOBS` | `thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true` | Background jobs dispatched once at boot, comma-separated, each `name` or `name?flag=true` using the same syntax as `POST /api/admin/jobs/{name}/trigger`. Flags: `force`, `deep`, `repair`, `storage`. **The default migrates thumbnails out of the legacy `.thumbnails/` directory and deletes the originals**, so the migration completes without anyone triggering it from the admin panel; each sidecar is read back through the normal stack before it is unlinked, and every deletion is audited. An explicit value **replaces** the default; set it empty (`OXICLOUD_STARTUP_JOBS=`) to disable startup jobs, or to `thumb_derived_import,thumb_attached_import` to import without deleting. **Non-blocking** — readiness never waits on a job; entries run sequentially in the background. **Fail-fast** — an unknown job name or flag panics at boot, because a silently-dropped entry means a migration that never runs. A run interrupted by a restart resumes from its cursor on the next boot, so a long migration finishes across restarts. Safe to leave at the default: the jobs are idempotent, and once drained a run does nothing. See [Thumbnail Migration](./thumbnail-migration.md) for the upgrade runbook. |
## Database
@@ -58,7 +59,7 @@ OPAQUE (RFC 9807) is a zero-knowledge password-authenticated key exchange: the p
| Variable | Default | Description |
|---|---|---|
| `OXICLOUD_AUTH_OPAQUE_MODE` | `off` | Runtime mode. `off` = endpoints 404 (default). `migrate` = endpoints live, legacy `POST /api/auth/login` still accepted. `opaque_only` = endpoints live, legacy refused for users with an envelope. **Effective-mode cross-check**: when `password` is not in `OXICLOUD_AUTH_METHODS`, the mode is auto-downgraded to `off` with an audit-channel INFO line (OPAQUE only replaces the password path — nothing to shadow in an OIDC-only or magic-link-only deployment). So OIDC / magic-link-only operators can safely ignore every `OXICLOUD_AUTH_OPAQUE_*` variable. |
| `OXICLOUD_AUTH_OPAQUE_SERVER_SETUP` | — | Base64-encoded `ServerSetup` blob. **Required** when `OXICLOUD_AUTH_OPAQUE_MODE != off` AND password is enabled — the server refuses to start with a helpful error otherwise. Generate once with the `oxicloud-cli opaque setup` subcommand and persist the value like your JWT secret. **Never rotate** — rotating invalidates every user's envelope (they'd all need to reset their passphrase). |
| `OXICLOUD_AUTH_OPAQUE_SERVER_SETUP` | — | Base64-encoded `ServerSetup` blob. **Required** when `OXICLOUD_AUTH_OPAQUE_MODE != off` AND password is enabled — the server refuses to start with a helpful error otherwise. Generate once with the `oxicloud opaque setup` subcommand and persist the value like your JWT secret. **Never rotate** — rotating invalidates every user's envelope (they'd all need to reset their passphrase). |
| `OXICLOUD_AUTH_OPAQUE_KSF_MEMORY_KIB` | `47104` | Client-side Argon2id memory cost in KiB (46 MiB — matches OWASP interactive-auth recommendation). Runs on the user's device during OPAQUE login/registration, TWICE per login. Distinct from `OXICLOUD_HASH_MEMORY_COST` (server-side legacy path). Bumping raises brute-force cost after a hypothetical envelope leak but also raises login latency and risks WASM heap OOM on low-memory devices — see `authentication.md § OPAQUE — KSF parameters` for the full rationale + per-device latency table. |
| `OXICLOUD_AUTH_OPAQUE_KSF_ITERATIONS` | `1` | Client-side Argon2id iteration count (OWASP interactive-auth recommendation). |
| `OXICLOUD_AUTH_OPAQUE_KSF_PARALLELISM` | `1` | Client-side Argon2id parallelism lanes (OWASP recommendation). Higher only helps on multi-core hardware and can hurt single-core / older mobile devices. |
@@ -94,6 +95,8 @@ DPoP cryptographically binds a session cookie to a browser-held ECDSA keypair (P
| `OXICLOUD_ENABLE_TRASH` | `true` | Trash / recycle bin |
| `OXICLOUD_ENABLE_SEARCH` | `true` | Full-text and metadata search |
| `OXICLOUD_ENABLE_MUSIC` | `true` | Music playlists and audio metadata |
| `OXICLOUD_ENABLE_VIDEO_THUMBNAILS` | `true` | Server-side single-frame thumbnail extraction from uploaded videos (one frame → WebP). Requires `ffmpeg` on `PATH` (override with `OXICLOUD_FFMPEG_PATH`). When true and ffmpeg is missing at boot, a WARN log is emitted and videos fall back to a placeholder icon. Set to `false` to skip the ffmpeg lookup entirely — useful on hosts where ffmpeg can't be installed, or when the client uploads video previews itself (some desktop/mobile clients generate thumbnails locally and POST them alongside the video). |
| `OXICLOUD_FFMPEG_PATH` | `ffmpeg` (on PATH) | Absolute path to the ffmpeg binary. Ignored when `OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false`. Useful for pinning a specific static build or when ffmpeg lives outside the default PATH. |
| `OXICLOUD_EXPOSE_SYSTEM_USERS` | `true` | Expose other OxiCloud users as a read-only address book at `GET /api/address-books` |
| `OXICLOUD_GRANT_CLEANUP_ENABLED` | `true` | Background daemon that deletes expired rows from `storage.role_grants`. The authorization engine already filters expired grants out of every permission check at read time (`expires_at IS NULL OR expires_at > NOW()`), so leaving expired rows in place is a hygiene issue — not a security one. This daemon garbage-collects them daily. Set to `false` to keep every expired grant row forever (uncommon; a fresh install rarely wants this). |
| `OXICLOUD_GRANT_CLEANUP_GRACE_DAYS` | `15` | Days past a grant's `expires_at` before the row is eligible for deletion. The grace window preserves the audit / support answer to "what happened to my access?" for a couple of weeks past expiration. Values below 1 are legal but discouraged — the recommendation is **≥ 15 days**. Values above the actual grant TTL used by clients waste index space; a few weeks is the sweet spot. |
@@ -131,7 +134,7 @@ Each declared name `<N>` then reads its own set of per-entry variables:
- A declared name whose required per-entry fields are missing (`_BACKEND` never set, S3 with no `_S3_BUCKET`, Azure with no `_AZURE_CONTAINER`).
- Setting `OXICLOUD_STORAGE_ENTRIES` alongside any of the legacy flat vars below (`OXICLOUD_STORAGE_BACKEND`, `OXICLOUD_S3_*`, `OXICLOUD_AZURE_*`, `OXICLOUD_STORAGE_ENCRYPTION_*`). Pick one mode; the error lists every conflicting var to remove.
- A DB pointer (`admin_settings.storage.active_backend_name`) that names an entry not in the current `_ENTRIES`. The error points at the repair flag `oxicloud --select-storage <name>` — verify + UPDATE DB + exit.
- A DB pointer (`admin_settings.storage.active_backend_name`) that names an entry not in the current `_ENTRIES`. The error points at the repair flag `oxicloud storage select <name>` — verify + UPDATE DB + exit.
**Example** — two entries, local disk plus an S3 target for planned migration:
docs/config/index.md
+1
View File
@@ -7,6 +7,7 @@ OxiCloud is configured entirely via **environment variables** (no config files n
- [Deployment & Docker](/config/deployment) — Docker Compose, Kubernetes Helm chart, image details
- [Environment Variables](/config/env) — complete reference of all `OXICLOUD_*` variables
- [Storage Fine Tuning](/config/storage-fine-tuning) — sizing the upload caps + spool directories; tmpfs vs real disk; NVMe split layouts
- [Thumbnail Migration](/config/thumbnail-migration) — upgrading past `.thumbnails/`: what runs on first boot, taking a snapshot first, verifying afterwards
- [Authentication](/config/authentication) — JWT auth, login, refresh, password changes, and auth status
- [OIDC / SSO](/config/oidc) — single sign-on with Keycloak, Authentik, Authelia, Google, Azure AD
- [WOPI (Office Editing)](/config/wopi) — Collabora Online / OnlyOffice integration
docs/config/thumbnail-migration.md
+153
View File
@@ -0,0 +1,153 @@
# Thumbnail migration runbook
Thumbnails used to live as files under `{STORAGE_PATH}/.thumbnails/`.
They now live in the content-addressed blob store, alongside file
content. This page is for operators upgrading across that change.
**You do not have to do anything.** The migration runs itself, in the
background, on the first boot after the upgrade. The rest of this page
is for operators who want to verify it, take a safety net first, or
understand what it did.
## What runs, and when
Two background jobs, dispatched once at startup and daily thereafter:
| Job | Migrates | Regenerable if lost? |
|---|---|---|
| `thumb_derived_import` | Thumbnails the server rendered from file content | Yes — the next request re-renders |
| `thumb_attached_import` | Previews a client uploaded (`ext-{file_id}.jpg`) | **No** — there is no render path for these |
Both import each sidecar into blob storage, read it back to confirm the
copy is byte-identical, and only then delete the original. When the
directory is empty it is removed, and `.thumbnails/` stops existing.
Startup dispatch is non-blocking — the server is ready immediately and
the migration proceeds behind it. A run interrupted by a restart resumes
from where it stopped, so a large installation finishes over several
restarts rather than starting again each time.
This is controlled by `OXICLOUD_STARTUP_JOBS`, which defaults to:
```
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true
```
To **import without deleting** — migrate now, inspect, delete later:
```
OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import
```
The sidecars then stay on disk. Trigger the deletion when you are ready
from **Admin → Jobs**, using each job's Repair action.
To disable startup jobs entirely, set the variable to an empty value.
## Taking a safety net first
Recommended for any installation where the uploaded previews matter, and
cheap enough to be worth it regardless. Both parts must be captured
together — a database that references blobs a storage snapshot predates
is worse than neither.
**1. Stop the server.** A snapshot taken while writes are in flight can
catch a blob that exists on disk without its database row, or the
reverse.
```bash
systemctl stop oxicloud # or: docker compose stop oxicloud
```
**2. Snapshot the database.**
```bash
pg_dump --format=custom --file=oxicloud-preflight.dump "$DATABASE_URL"
```
Use `--format=custom`; restoring it needs `pg_restore --disable-triggers`,
because the folder table carries a self-referencing foreign key that a
plain SQL restore cannot order correctly.
**3. Snapshot the storage directory.** At minimum `.thumbnails/`, which
is what the migration touches:
```bash
tar -czf oxicloud-thumbnails-preflight.tar.gz -C "$STORAGE_PATH" .thumbnails
```
A whole-directory snapshot is better if you have the space — filesystem
or volume snapshots (ZFS, LVM, EBS) are ideal, since they are atomic and
near-instant:
```bash
zfs snapshot tank/oxicloud@preflight
```
**4. Start the server.** The migration begins in the background.
Keep both snapshots until you have run the verification below and are
satisfied.
## Verifying the migration
Two checks, both from **Admin → Jobs** or the API. Run them after the
migration reports no remaining work.
**1. Every mapping points at a blob that exists.** Run
`satellites_consistency`. It walks both thumbnail tables and reports any
row whose blob or source is gone. A clean run means nothing was lost in
the bookkeeping.
```
POST /api/admin/jobs/satellites_consistency/trigger
```
**2. Every blob still hashes to what it claims.** Run
`backend_consistency` with `?deep=true`. It reads every blob back from
storage and re-hashes it, which covers the migrated thumbnails along
with everything else. This is a full read of your storage and can take
hours on a large installation — schedule it accordingly.
```
POST /api/admin/jobs/backend_consistency/trigger?deep=true
```
A clean pass on both means the thumbnails are readable, correctly
referenced, and byte-intact in their new home. At that point the
snapshots can be discarded.
## Checking it finished
`.thumbnails/` is gone. That is the whole test:
```bash
ls -d "$STORAGE_PATH/.thumbnails" # No such file or directory
```
If you instead find `.thumbnails.migrated/`, the migration completed but
could not remove the directory, because something that is not a
thumbnail was inside it — a `.DS_Store` from macOS Finder is the usual
culprit. The tree was moved aside instead of deleted. Its contents are
no longer used and it is safe to remove by hand once you have looked at
what is in there.
While either directory is absent, the server skips the legacy read path
entirely, at no cost. While `.thumbnails/` is present, reads fall back
to it on a miss, which is what makes the migration invisible to users
while it runs.
## If something looks wrong
Every deletion is written to the audit log, naming the job, the file
removed and the blob that replaced it. To review what a migration
removed:
```bash
journalctl -u oxicloud | grep sidecar_deleted
```
A sidecar is only ever deleted after its replacement has been read back
and compared byte-for-byte, so a file that failed that check is still on
disk. Those show up as findings on the job's run in **Admin → Jobs**,
with the reason recorded per file.
docs/guide/backend-storage.md
+2 -2
View File
@@ -167,13 +167,13 @@ If you rename or remove a backend from `.env` while it was still the active one,
```
active_backend_name = `s3_prod`, but no entry with that name is declared in
OXICLOUD_STORAGE_ENTRIES. Available: [local_main]. […]
oxicloud --select-storage <one-of-the-available-names>
oxicloud storage select <one-of-the-available-names>
```
Run the command it suggests to pick a still-declared backend and the server will boot again on the next start:
```
oxicloud --select-storage local_main
oxicloud storage select local_main
```
This just updates which backend OxiCloud considers active — it doesn't move any data.
docs/install/binary.md
+245
View File
@@ -0,0 +1,245 @@
# Installing OxiCloud from a Binary Release
OxiCloud ships prebuilt binaries for common Linux and macOS platforms
attached to every tagged release on GitHub. This page covers downloading,
verifying, and running one.
If you'd rather run OxiCloud as a container, see the Docker image at
`ghcr.io/atalayalabs/oxicloud`. If you're a Rust developer who just
wants the binary without hand-fetching a tarball, `cargo binstall
oxicloud` picks the right archive for your host automatically.
## Which tarball do I want?
Every release attaches three tarballs plus a `SHA256SUMS` manifest.
Pick by your host's architecture and OS:
| Host | Tarball |
|---|---|
| Linux x86-64 (Intel / AMD servers, most VPS, WSL) | `oxicloud-<version>-x86_64-unknown-linux-musl.tar.gz` |
| Linux ARM64 (Raspberry Pi 4/5, Ampere, Graviton, ARM servers) | `oxicloud-<version>-aarch64-unknown-linux-musl.tar.gz` |
| macOS Apple Silicon (M-series) | `oxicloud-<version>-aarch64-apple-darwin.tar.gz` |
The Linux tarballs link against musl, so they run on ANY glibc version
— Alpine, Debian, Ubuntu, Fedora, Arch, Rocky, and every version in
between. You never need to worry about `GLIBC_x.yy not found`.
**Intel macOS, Windows, and 32-bit ARM are not currently shipped as
prebuilt tarballs.** Intel Mac users have three fallbacks:
1. `cargo install oxicloud --locked --features bundled-assets` from
source (needs the Rust toolchain).
2. Docker: `docker pull --platform linux/amd64 ghcr.io/atalayalabs/oxicloud`.
3. Run one of the two Linux musl tarballs inside a Linux VM
(Multipass, Lima, UTM, etc.).
## Hardware notes
| Model | Notes |
|---|---|
| Pi 5 (4 GB / 8 GB) | Good experience |
| Pi 4 (4 GB / 8 GB) | Solid |
| Pi 4 (2 GB) | Works with face indexing disabled; expect swap under load |
| Pi 3 (any variant) | Marginal — only for a very light single-user personal cloud |
| Pi 2 / Pi Zero / Pi 1 | Not supported (1 GB RAM is below the practical floor) |
| Any ARM64 server | Good — the aarch64 tarball is what you want |
| Any x86-64 server from 2010 or newer | Good — Nehalem / Bulldozer + newer, per the release CPU baseline |
## Verifying the download
Every release ships a `SHA256SUMS` manifest listing every tarball with
its hash. Verify your download before extracting:
```
sha256sum -c SHA256SUMS
```
Only files present in the current directory are checked, so this
succeeds when just the tarball you downloaded matches its entry.
## Extracting
The archive lands as a per-version-per-triple directory next to it:
```
tar xzf oxicloud-<version>-<triple>.tar.gz
cd oxicloud-<version>-<triple>/
ls
# oxicloud example.env LICENSE README-install.md
```
The four files:
- `oxicloud` — the single self-contained binary. The server, all
operator subcommands (`oxicloud opaque setup`, `oxicloud migrate
nfc-filenames`, `oxicloud storage select`), and the SvelteKit web
frontend are all baked in.
- `example.env` — every OxiCloud environment variable documented with
defaults. Copy to `.env` and edit as needed.
- `LICENSE` — the project license.
- `README-install.md` — a shorter version of this page for offline
reference.
## Prerequisites
Only one moving part is required: a PostgreSQL 13+ instance with the
`pg_trgm` and `ltree` extensions available. Anything else you might
need is either baked into the binary or optional.
### Required
- **PostgreSQL 13+** with `pg_trgm` and `ltree` extensions. Any distro
package works (Debian/Ubuntu's `postgresql`, Alpine's `postgresql`,
Homebrew's `postgresql@17`, etc.). Cloud databases like Neon,
Supabase, and RDS also work provided the two extensions are enabled.
### System libraries (usually pre-installed)
- **`ca-certificates`** — for outbound HTTPS (OIDC discovery, S3, magic
links). Pre-installed on essentially every distribution.
- **`tzdata`** — timezone database. Pre-installed on nearly every
distribution; alpine minimal images sometimes need it added.
### Optional
- **`ffmpeg`** — only needed if you want the server to extract a
thumbnail frame from uploaded videos. When ffmpeg is missing the
server logs a warning at boot and videos get a placeholder icon —
everything else keeps working. If your client uploads video
previews itself (some desktop and mobile clients do), or if you
simply don't want thumbnails, set
`OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false` in your `.env` to silence
the warning.
Distro install commands for the optional prerequisite:
| Distro | Command |
|---|---|
| Alpine | `apk add ffmpeg` |
| Debian / Ubuntu | `apt install ffmpeg` |
| Fedora / RHEL | `dnf install ffmpeg` (RPM Fusion for the full codec set) |
| Arch | `pacman -S ffmpeg` |
| macOS | `brew install ffmpeg` |
| Any Linux (portable) | grab a static build from https://github.com/BtbN/FFmpeg-Builds/releases and point `OXICLOUD_FFMPEG_PATH` at it |
## First run
The absolute minimum to boot the server is `DATABASE_URL`:
```
DATABASE_URL="postgres://oxicloud:secret@localhost:5432/oxicloud" \
./oxicloud
```
The binary applies its embedded database migrations on startup, then
listens on `127.0.0.1:8086` by default. Open your browser at
`http://localhost:8086/` and follow the setup flow to create the first
admin account.
For anything more than a smoke test, copy `example.env` to `.env`,
edit it, and run `./oxicloud --config .env` — that pins the config
source and makes stray shell environment variables not silently leak
in.
## Running as a systemd service (Linux)
Move the binary to a system location and create a systemd unit. The
example below runs as a dedicated `oxicloud` user, loads config from
`/etc/oxicloud/oxicloud.env`, and stores data under `/var/lib/oxicloud`.
```
sudo useradd --system --home /var/lib/oxicloud --create-home --shell /usr/sbin/nologin oxicloud
sudo install -m 0755 oxicloud /usr/local/bin/oxicloud
sudo mkdir -p /etc/oxicloud
sudo cp example.env /etc/oxicloud/oxicloud.env
sudo chown -R oxicloud:oxicloud /etc/oxicloud
sudo chmod 0640 /etc/oxicloud/oxicloud.env
```
Create `/etc/systemd/system/oxicloud.service`:
```
[Unit]
Description=OxiCloud self-hosted cloud storage
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=simple
User=oxicloud
Group=oxicloud
WorkingDirectory=/var/lib/oxicloud
ExecStart=/usr/local/bin/oxicloud --config /etc/oxicloud/oxicloud.env
Restart=on-failure
RestartSec=5
# Sandbox — plenty of room to tighten further per your policy
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/oxicloud
PrivateTmp=true
[Install]
WantedBy=multi-user.target
```
Enable and start:
```
sudo systemctl daemon-reload
sudo systemctl enable --now oxicloud
sudo systemctl status oxicloud
journalctl -u oxicloud -f
```
Terminate the reverse-proxy (nginx, Caddy, HAProxy, Traefik) in front
of it for TLS and public exposure — OxiCloud itself binds plaintext
HTTP on `127.0.0.1` by default.
## Upgrading
Replace the binary and restart the service:
```
# Download and verify the new tarball
sha256sum -c SHA256SUMS
tar xzf oxicloud-<new-version>-<triple>.tar.gz
cd oxicloud-<new-version>-<triple>/
sudo systemctl stop oxicloud
sudo install -m 0755 oxicloud /usr/local/bin/oxicloud
sudo systemctl start oxicloud
```
Database migrations apply automatically on startup. Rollbacks are not
supported by sqlx's migration model; if you need to roll back, stop
the server, roll back your Postgres data directory to a snapshot, and
install the previous binary.
## Installing via `cargo binstall`
If you already have the Rust toolchain and just want the binary
without hand-picking a tarball:
```
cargo binstall oxicloud
```
`cargo-binstall` reads the URL template baked into the release
metadata, downloads the tarball for your host triple, verifies its
signature (when present), and installs `oxicloud` into
`~/.cargo/bin`. This resolves to the same tarball you'd download by
hand.
## Where to go from here
- Environment reference — see [`docs/config/env.md`](../config/env.md)
for every `OXICLOUD_*` variable and its default.
- Authentication setup (OPAQUE, OIDC, magic links) — see
[`docs/config/authentication.md`](../config/authentication.md).
- Storage backends (local disk, S3, Azure Blob, encryption) — see
[`docs/config/storage.md`](../config/storage.md) if present, or the
entries under `OXICLOUD_STORAGE_*` in the environment reference.
- File a bug or a feature request — GitHub issues at
https://github.com/AtalayaLabs/OxiCloud.
docs/plan/benchmake-and-performance-tracking.md
+4 -2
View File
@@ -41,7 +41,9 @@ Mirrors the `tests/api/` shell pattern (run.sh, test.env, separate server port).
### 2. Rust bulk seeder — `src/bin/load-seed.rs`
New binary registered in `Cargo.toml` alongside `generate-openapi` and `migrate-nfc-filenames`.
New binary registered in `Cargo.toml` alongside `generate-openapi`
(the historical `migrate-nfc-filenames` bin has since been folded into
`oxicloud migrate nfc-filenames` — see `docs/plan/bundled-binary.md` § 1b).
**CLI:**
```
@@ -158,7 +160,7 @@ Matches existing recipe naming (`test-*`, `front-*`, `api-test`).
- `.github/workflows/load-nightly.yml`, `load-smoke.yml`
**Modify:**
- `Cargo.toml` — add `[[bin]] name = "load-seed" path = "src/bin/load-seed.rs"` after the `migrate-nfc-filenames` entry
- `Cargo.toml` — add `[[bin]] name = "load-seed" path = "src/bin/load-seed.rs"` after the `generate-openapi` entry (the `migrate-nfc-filenames` bin referenced in earlier drafts has been folded into `oxicloud migrate nfc-filenames`)
- `justfile` — append four `load*` recipes
- `.gitignore` — add `tests/load/results/*.json` and `tests/load/storage/`
docs/plan/bundled-binary.md
+888
View File
@@ -0,0 +1,888 @@
# Bundled Binary Distribution — Multi-Platform Plan
## Context
Users have asked for a way to run OxiCloud without Docker — a plain
binary. Today `release.yml` only creates a GitHub Release with notes;
no binary is attached. The Docker workflow (`docker-publish.yml`) ships
multi-arch images, but that's a separate audience.
The blocker for a "just download and run" experience is that the
`oxicloud` binary depends on the SvelteKit build output (`static-dist/`
under `<static_path>/static-dist/`, resolved by
`src/interfaces/web/mod.rs::resolve_static_path` at boot). Two files
to distribute per platform is friction; a single self-contained binary
is what users actually want.
The ask has two parts:
1. Ship **single-file binaries with frontend assets embedded**, for
the common Linux targets and macOS.
2. Audit the current binary set — the crate produces 6+ binaries today,
some of which are test-only. Strip anything that shouldn't ship to
end users.
The intended outcome: a `v0.9.0` release attaches **4 musl-static
tarballs** (Linux amd64/arm64 + macOS Intel/Apple Silicon), each
~15-30 MB, containing a **single `oxicloud` binary** with assets +
operator tools + one-off migrations all baked in. User extracts the
tarball, sets `DATABASE_URL`, runs `./oxicloud` — server up.
Subcommands (`oxicloud opaque setup`, `oxicloud migrate
nfc-filenames --dry-run`) provide operator access to the same tools
currently split across `oxicloud-cli` and `migrate-nfc-filenames`.
Design shape (confirmed 2026-08-27):
- **musl-only Linux** — parity with the existing Docker image (Alpine
base), no glibc-version fragmentation
- **Assets embedded via `rust-embed` with compile-time deflate
compression** — smaller binary
- **`bundled-assets` is opt-in** — default `cargo build` unchanged;
`just dev` still uses the filesystem `ServeDir` with Vite HMR
- **Single unified binary** — `oxicloud`, `oxicloud-cli`, and
`migrate-nfc-filenames` collapse into one clap-driven executable
with implicit-server default (backwards compat with existing Docker
CMD / systemd units)
## Current binary inventory
From `Cargo.toml` + `src/bin/`:
| Binary | Path | Purpose | Ship to end users? |
|---|---|---|---|
| `oxicloud` | `src/main.rs` (implicit) | Server | **YES** |
| `oxicloud-cli` | `src/bin/oxicloud-cli.rs` | Operator toolbox (`opaque setup/reset`) | **MERGED** — absorbed into `oxicloud` per Deliverable 1b |
| `migrate-nfc-filenames` | `src/bin/migrate-nfc-filenames.rs` | One-off filename migration (historical, June 2026 fix) | **MERGED** — absorbed into `oxicloud migrate nfc-filenames` per Deliverable 1a→1b |
| `generate-openapi` | `src/bin/generate-openapi.rs` | Regenerate `resources/gen/openapi.json` | NO — dev tool, gate behind `dev_tools` feature |
| `opaque-hurl-helper` | `src/bin/opaque-hurl-helper.rs` | Hurl test companion (OPRF client) | NO — gate behind `test_utils` feature |
| `dpop-hurl-helper` | `src/bin/dpop-hurl-helper.rs` | Hurl test companion (ES256 DPoP proof) | NO — gate behind `test_utils` feature |
| `load-seed` | `src/bin/load-seed.rs` | Test fixture seeder | Already gated behind `load_seed_bin` feature ✅ |
After Deliverables 1 + 1a + 1b, `cargo build --release --bins`
produces exactly ONE binary: `oxicloud`. That single binary ships in
the tarball and in the Docker image.
## Deliverables
### 1. Squash test/dev binaries with `required-features`
Cargo respects `required-features` per `[[bin]]` — a binary is only
built when its listed features are active. This gates test helpers
out of `cargo build --release --bins` cleanly without needing custom
Cargo commands or shell trimming.
Edits to `Cargo.toml`:
```toml
[features]
# ... existing features ...
dev_tools = [] # NEW: gates ops tooling that shouldn't ship
[[bin]]
name = "opaque-hurl-helper"
path = "src/bin/opaque-hurl-helper.rs"
required-features = ["test_utils"] # NEW gate
[[bin]]
name = "dpop-hurl-helper"
path = "src/bin/dpop-hurl-helper.rs"
required-features = ["test_utils"] # NEW gate
[[bin]]
name = "generate-openapi"
path = "src/bin/generate-openapi.rs"
required-features = ["dev_tools"] # NEW gate — `just openapi` flips it
# [[bin]] name = "migrate-nfc-filenames" ← DELETED per Deliverable 1a
# [[bin]] name = "oxicloud-cli" ← DELETED per Deliverable 1b
```
Existing invocations that need adjustment:
- `just openapi` recipe → add `--features dev_tools` to the underlying
`cargo run --bin generate-openapi` call (currently `cargo run --bin
generate-openapi` per justfile)
- `tests/api/run.sh` → add `--features test_utils` when building the
two hurl helpers (shape confirmed: `cargo build [--release] --bin
opaque-hurl-helper` / same for dpop in each helper's build-if-missing
branch)
After these edits + Deliverables 1a + 1b: `cargo build --release --bins`
produces exactly ONE binary — `oxicloud`. Everything else falls out of
the default build set.
### 1a. Merge `migrate-nfc-filenames` into `oxicloud-cli`
The standalone `migrate-nfc-filenames` binary is a June-2026 one-off:
it cleans up NFD/NFC filename collisions in databases populated
before the write-time fix (`normalize_storage_name()` at
`src/domain/services/path_service.rs:36`, called from
`src/infrastructure/repositories/pg/file_blob_read_repository.rs:1062`).
New installs never need it; only pre-June 2026 databases do.
`oxicloud-cli`'s header docstring (`src/bin/oxicloud-cli.rs:20-23`)
already documents the growth pattern for absorbing tools like this:
> *"each new domain gets its own module below (e.g. `mod opaque`)
> with a `#[derive(Subcommand)]` enum for its actions and a
> `run(args) -> ExitCode` entrypoint. Keep each module self-contained
> so a future extraction is a file move."*
Note: this Deliverable is an intermediate step. Deliverable 1b then
absorbs `oxicloud-cli` itself into `oxicloud`, so the final CLI form
becomes `oxicloud migrate nfc-filenames --dry-run` — but 1a lands
first so the migration logic is proven inside the clap subcommand
tree before the main-binary merge.
Edits:
- **New `mod migrate` in `src/bin/oxicloud-cli.rs`** — moves the ~149
non-boilerplate lines from `migrate-nfc-filenames.rs::main()` into
a `run_nfc_filenames(dry_run: bool) -> ExitCode` function.
`env::args()` parsing goes away; clap handles it.
- **Delete `src/bin/migrate-nfc-filenames.rs`**.
- **Delete the `[[bin]]` entry** in `Cargo.toml`.
- **Update `Dockerfile`** — 6 references to `migrate-nfc-filenames`
(build commands at :46, :49, :89, `cp` steps at :130, :143, doc
comment at :170, `COPY --chmod=755 --from=app` at :173).
- **Update `docs/plan/benchmake-and-performance-tracking.md`** — 2
references to `migrate-nfc-filenames` at lines :44 and :161. Reword
to reference `oxicloud-cli migrate nfc-filenames` (or, after 1b,
`oxicloud migrate nfc-filenames`) and update the Cargo.toml
placement example.
- **Any operator runbook** that documents `docker exec <container>
migrate-nfc-filenames --dry-run` becomes `docker exec <container>
oxicloud-cli migrate nfc-filenames --dry-run` (intermediate) then
`docker exec <container> oxicloud migrate nfc-filenames --dry-run`
after 1b.
Effort: ~1.5 hours mechanical. Extracts the "should the tarball ship
migrate-nfc-filenames?" question entirely — everything now ships as
one operator toolbox binary that also happens to include the
historical migration.
Future v1.0 removal path (deferred): delete `mod migrate` block + one
enum variant + docs. Much cleaner than removing a whole `.rs` file +
Cargo entry + Dockerfile refs.
### 1b. Merge `oxicloud-cli` into `oxicloud`
Single binary — server + operator tools + migrations — with an
**implicit-server** subcommand tree. `oxicloud` with no arguments
starts the server (backwards compat with existing Docker CMD /
systemd units / user configs). Subcommands add operator actions on
top.
After merge, the CLI shape is:
```
$ oxicloud --help
Usage: oxicloud [OPTIONS] [COMMAND]
Commands:
opaque OPAQUE aPAKE substrate management
migrate One-time data migrations
If no command is given, oxicloud starts the server (see docs/config).
```
Concrete forms:
- `oxicloud` — start server (unchanged)
- `oxicloud opaque setup` — was `oxicloud-cli opaque setup`
- `oxicloud opaque reset --user alice --dry-run` — was `oxicloud-cli
opaque reset ...`
- `oxicloud migrate nfc-filenames --dry-run` — was
`migrate-nfc-filenames --dry-run` (via Deliverable 1a)
**Backwards-compat guarantee**: `oxicloud` with no args continues to
start the server. Every existing `CMD ["oxicloud"]`, `ExecStart=/usr/local/bin/oxicloud`,
docker-compose entry, and k8s Deployment keeps working unchanged.
Users updating to v0.9.0 see no surprise.
**Migration impact**: the user-visible break is that `oxicloud-cli
opaque setup` (etc.) no longer exists as a separate binary. Given the
current audience for `oxicloud-cli` is very small (essentially only
the maintainer), the migration cost is trivial. Any user who had
scripted it can adapt with a one-line find/replace.
Edits:
- **`src/main.rs`** — top of `main()`, before the current server
init, parse args via clap. If a subcommand is provided, dispatch
to it and exit; otherwise fall through to the existing server-init
path. Zero-arg startup cost stays ≤ microseconds (clap parse of
empty args).
- **`src/cli/mod.rs`** — NEW module. Contains the `Domain` enum + the
`opaque` and `migrate` submodules moved from
`src/bin/oxicloud-cli.rs`. Each subcommand module keeps its
self-contained shape per the growth pattern documented in the
old `oxicloud-cli.rs` header.
- **Delete `src/bin/oxicloud-cli.rs`** entirely.
- **Delete the `[[bin]] name = "oxicloud-cli"` block** in `Cargo.toml`.
- **`Dockerfile`** — drop all 4 references to `oxicloud-cli` (build
target lines + COPY steps). Simplified build command becomes
`cargo build --release --bin oxicloud` — single-binary.
- **Docs** — all `docker exec <container> oxicloud-cli <domain>
<action>` become `docker exec <container> oxicloud <domain>
<action>`. Same shape, one fewer word.
Effort: ~2 hours mechanical. Comparable to Deliverable 1a but with
slightly more care at the `main.rs` entry point for the args-vs-server
branch.
**Tarball layout simplification** — the tarball now ships exactly
ONE binary:
```
oxicloud-0.9.0-<triple>/
├── oxicloud (single file, server + tools + embedded assets)
├── example.env
├── LICENSE
└── README-install.md
```
That's the "just download and run" ethos in physical form: one file,
one command, done.
### 2. Add `bundled-assets` cargo feature
Purpose: at compile time, choose between filesystem-served static
assets (current behaviour — filesystem `ServeDir`) and
embedded-into-binary assets (via `rust-embed`). Feature is
**opt-in** — the default `cargo build --release` still produces a
filesystem-based binary, matching the current Docker image behaviour
(where assets are separate volume layers). Release tarballs are built
with `--features bundled-assets`.
**Dev mode is untouched.** `just dev` runs `PROFILE=dev cargo run` +
`npm run dev`, neither of which activates `bundled-assets`. The dev
workflow continues to:
- Serve from `frontend/` via Vite's dev server with HMR
- Backend reads static assets from `<static_path>/static-dist/` via the
usual `ServeDir` (or falls back to `frontend/static/` when the
build hasn't been run)
- No rebuild required to change locales, styles, or vendor JS
The `bundled-assets` code paths only compile when the feature is
explicitly enabled — under a `#[cfg(feature = "bundled-assets")]` gate.
The non-feature build's binary shape, ergonomics, and dev loop stay
identical to today.
Measured footprint (2026-08-27):
| Slice | Size | Notes |
|---|---|---|
| Total `static-dist/` uncompressed | **9.8 MB** | 499 files |
| `_app/` (SvelteKit bundle) | 3.3 MB | JS + CSS chunks |
| `vendors/` | 3.6 MB | maplibre-gl 1.0 MB, pdf.worker 1.0 MB, others |
| `locales/` | 2.2 MB | 16 locales, ru.json + hi.json largest at ~116-140 KB |
| `logo/`, `geo/`, `basemaps/`, `workers/`, misc | ~600 KB | |
| **`.tar.gz` compressed** | **4.65 MB** | realistic embed cost after brotli/gzip inside binary |
| **`.tar.xz` compressed** | **4.22 MB** | not what rust-embed uses; reference only |
Expected release-binary size with embed: `oxicloud` today ships in
the 30-60 MB range (stripped, LTO). Add ~5-10 MB for embedded
static-dist. Tarball compression on top → ~20-30 MB shipped per
platform. Four platforms × ~25 MB = ~100 MB per release. Well within
GitHub Releases limits.
Cargo.toml additions:
```toml
[features]
bundled-assets = ["dep:rust-embed", "dep:mime_guess"]
[dependencies]
rust-embed = { version = "8", features = ["compression"], optional = true }
mime_guess = { version = "2", optional = true }
```
Runtime shape — a new module `src/interfaces/web/embedded.rs`:
```rust
#[cfg(feature = "bundled-assets")]
#[derive(rust_embed::RustEmbed)]
#[folder = "static-dist/"] // ← repo-root, matches SvelteKit adapter-static output
#[include = "*"]
#[exclude = "*.br"] // Vite's precompressed sibling — response compression handles on wire
#[exclude = "*.gz"] // ditto
pub struct EmbeddedAssets;
```
The `#[folder]` path is relative to Cargo.toml (repo root), where the
SvelteKit adapter-static config in `frontend/svelte.config.js` emits:
```js
adapter: adapter({
pages: '../static-dist',
assets: '../static-dist',
...
})
```
The current filesystem shape (at `src/interfaces/web/mod.rs:47-106`)
is more than one `ServeDir` — the embed swap replaces FOUR sites, all
downstream of `resolve_static_path()`:
1. **`spa` ServeDir** (`mod.rs:60-63`) — root fallback with
`precompressed_br().precompressed_gzip()` and SPA-shell fallback
pointing at `<static>/index.html`. Under embed: an axum handler
that resolves the request path against `EmbeddedAssets::get()`,
200 with correct MIME (via `mime_guess`) if hit, otherwise return
the embedded `index.html` bytes with `text/html` for SPA client-routing.
2. **`app_immutable` ServeDir** (`mod.rs:66-77`) — nested at
`/_app/immutable` with `Cache-Control: public, max-age=31536000,
immutable`. Under embed: same handler shape as (1), scoped to
the `_app/immutable/` prefix, plus a `.layer()` that stamps the
immutable cache header.
3. **`ServeFile::new(index.html)`** SPA fallback (`mod.rs:63`) —
folds into (1)'s not-found path.
4. **CSP inline-script scan** (`mod.rs:163-233`) — currently reads
every `.html` file in the resolved static dir via
`std::fs::read_dir` + `std::fs::read_to_string` at boot to compute
SHA-256 CSP source expressions for every inline `<script>`. Under
embed: iterate `EmbeddedAssets::iter()` filtered to `.html`
extensions, pull bytes via `::get()`, hash the same way. Same
arithmetic, different source. Boot-time only.
All four flow through `resolve_static_path()` at `src/interfaces/web/mod.rs:25-35`
— that helper is the natural pivot. Add a returned enum:
```rust
#[cfg(feature = "bundled-assets")]
pub enum StaticSource {
Filesystem(PathBuf), // OXICLOUD_STATIC_PATH points at a real dir
Embedded, // fall through to compiled-in bytes
}
```
Then the four callsites (`create_web_routes` + CSP scan) match on
`StaticSource` and pick their implementation. Under the default
feature set (no `bundled-assets`), the enum degrades to a bare
`PathBuf` — zero runtime cost, no cfg pollution across the wider
codebase.
**Locale loading — also needs embed treatment.** Two callsites read
locales at runtime:
- `src/main.rs:599-615` — resolves `<static_path>/locales/` at boot
and passes it to `LocaleRegistry::discover()` at
`src/common/locale.rs:150-221`, which does `fs::read_dir` +
`fs::read_to_string` + `serde_json::from_str` on each of 16 files.
Currently fail-fast panics if the directory is missing.
- `src/infrastructure/services/file_system_i18n_service.rs` — the
runtime translator, `translations_dir: PathBuf` field, does
`tokio::fs::read_to_string` on `<dir>/<code>.json` per lazy-load
miss (cached in `RwLock<HashMap<Locale, Value>>`).
Under `bundled-assets`, both get an alternative implementation that
reads from `EmbeddedAssets` (locale files are at
`static-dist/locales/*.json`, picked up by the same folder embed).
Recommended shape: constructor pair —
`LocaleRegistry::discover_filesystem(path)` and
`#[cfg(feature = "bundled-assets")] LocaleRegistry::discover_embedded()`.
`main.rs` picks based on the resolved `StaticSource`. Simpler than a
trait-based indirection for two static sources with the same interface.
Frontend at runtime ALSO fetches `/locales/*.json` for client-side
i18n — this path is served by the same static router in (1) above,
so no separate work; the embed already covers it.
Precedence rule: even in a bundled build, honour `OXICLOUD_STATIC_PATH`
when it points at an existing directory. Lets ops override embedded
assets for locale patches / theming without a full rebuild. The
`resolve_static_path` return value is checked at boot; a real directory
wins over embedded fallback. If the resolved directory does NOT exist,
fall through to the embedded handler cleanly (log at info level:
"OXICLOUD_STATIC_PATH points at <path> which doesn't exist; serving
embedded assets").
Build-time invariant: `cargo build --features bundled-assets` requires
`static-dist/` to exist AND be non-empty. Add a `build.rs` check that
emits a clear error if missing, pointing at `just fe-build` /
`(cd frontend && npm run build)`.
**Precompression + embed strategy**: minimize binary size by storing
assets compressed inside the binary, and use axum's response
compression on the wire.
`rust-embed`'s `compression` feature deflate-compresses each embedded
file at compile time. Files are decompressed lazily on first access
and cached in a per-file `OnceCell` for the remainder of the process.
Warms up quickly under real traffic — the first user's page load
touches ~30 files, all cached from then on.
On the wire, response compression is handled by axum's
`CompressionLayer` (tower-http) applied to the static router
subtree. Browsers get `Content-Encoding: br` when they Accept-Encoding
brotli; gzip fallback; identity for clients that ask for neither.
Projected embed size after excludes + rust-embed deflate compression:
**~4-5 MB**. Matches the `.tar.xz` reference size and roughly halves
what raw-embed-plus-siblings would cost. Runtime CPU: negligible under
any real load; the compressed variants would benefit from a
reverse-proxy cache in front for CPU-tight hosts (Pi 4/5).
Consequence for the `nginx`/reverse-proxy story users will run in
front: the binary responds correctly to `Accept-Encoding: br, gzip`
without configuration. Users terminating TLS at their proxy get
compressed responses either way (proxy passes through or re-compresses
its cache).
### 3. Target matrix — musl-only Linux
Three triples cover the practical need:
| Triple | Runner + toolchain | Notes |
|---|---|---|
| `x86_64-unknown-linux-musl` | `ubuntu-22.04` + `musl-tools` + `rustup target add` | Static, no glibc dep, runs on ANY Linux distro from Alpine to CentOS 7 to Debian 10 to Ubuntu 25.04. Cross-compiled natively with glibc host + musl target; produces same output as the alpine-container path we originally planned. |
| `aarch64-unknown-linux-musl` | `ubuntu-22.04-arm` + `musl-tools` + `rustup target add` | Same shape as the amd64 twin. Native ARM64 runner (no QEMU). Pi 4/5, ARM servers, Graviton. |
| `aarch64-apple-darwin` | `macos-latest` | Apple Silicon, native |
**Historical note — Intel macOS dropped 2026-08-29** (Apple phasing
out `macos-13`; runner-availability tax exceeded value). Intel Mac
users fall back to `cargo install`, Docker `--platform linux/amd64`,
or one of the Linux musl tarballs inside a Linux VM.
**Historical note — Alpine-container approach abandoned 2026-08-29**
in favour of native cross-compile. Original plan built inside the
Dockerfile's `rust:1.96-alpine3.24` for byte-for-byte parity with
Docker; broke on `ubuntu-22.04-arm` because JS-based GitHub Actions
(checkout, artifact steps, setup-node) can't run inside Alpine on
ARM64 (Node.js binary requires glibc; the x64-Alpine workaround
doesn't extend to arm64). Native `ubuntu-22.04` + `musl-tools` +
`rustup target add` produces the same `--target *-musl` output
without the container gymnastics.
**Rationale for musl-only Linux**:
1. **Parity with Docker.** The Docker image is already Alpine/musl —
users get identical runtime behaviour whether they pull the
container or the tarball. One build shape, one test surface.
2. **Face-indexing regression is a NON-issue.** `faces-onnx` requires
glibc-only `libonnxruntime.so`; it's already unavailable on the
Docker image. Users who want face indexing build from source with
`--features faces-onnx` on a glibc host — same as today, no
change from musl-only tarballs.
3. **Zero glibc-version fragmentation.** No `GLIBC_2.35 not found`
errors on older distros. One binary works everywhere.
4. **Simpler install docs.** "Download this file, run it" without
a "which glibc do you have?" branch.
5. **Marginal perf hit is invisible under I/O-bound OxiCloud workloads.**
Musl's `malloc` and DNS resolver quirks matter for allocation-heavy
/ DNS-heavy servers; OxiCloud is neither.
**External runtime dependencies** — complete list. Codebase audit
2026-08-27 confirmed `ffmpeg` is the ONLY `Command::new` invocation
in `src/`; no other subprocess deps exist.
| Category | Dep | Required? | Notes |
|---|---|---|---|
| Subprocess | `ffmpeg` | Optional | Video thumbnails. Kill switch: `OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false`. Path override: `OXICLOUD_FFMPEG_PATH` |
| System lib | `ca-certificates` | Required | Outbound HTTPS (OIDC, S3, webhooks). Pre-installed on nearly every distro |
| System lib | `tzdata` | Required | Timezone DB for chrono. Pre-installed on nearly every distro |
| External service | PostgreSQL 13+ | Required | With `pg_trgm` + `ltree` extensions. TCP/loopback only — no libpq client lib needed |
| Runtime dylib | `libonnxruntime.so` + ONNX models | N/A for tarball | Face indexing (glibc-only, requires build from source with `--features faces-onnx`). Not shipped in musl tarballs — Docker/tarball users don't have this feature |
**Explicit non-deps** (worth documenting to preempt questions):
- **No libpq** — sqlx uses pure-Rust tokio-postgres
- **No git** — only build-time metadata via `build.rs`, never runtime
- **No ImageMagick / libvips** — image thumbnails via pure-Rust `image` crate
- **No pandoc / rst2html / etc.** — no document conversion
- **No systemd/launchd** — daemon lifecycle user-managed
- **No sendmail / SMTP CLI** — email via pure-Rust SMTP client
**Per-distro install command** (for `README-install.md`):
| Distro | Command |
|---|---|
| Alpine | `apk add ca-certificates tzdata ffmpeg` |
| Debian / Ubuntu | `apt install ca-certificates tzdata ffmpeg` |
| Fedora / RHEL | `dnf install ca-certificates tzdata ffmpeg` (RPMFusion for full codec set) |
| Arch | `pacman -S ca-certificates tzdata ffmpeg` |
| macOS | `brew install ffmpeg` (ca-certificates + tzdata built in) |
| Portable Linux | Static ffmpeg from https://github.com/BtbN/FFmpeg-Builds/releases + `OXICLOUD_FFMPEG_PATH=<path>` |
Postgres install is documented separately (project docs) since it's a
per-distro-per-version story with per-extension setup.
**Windows deliberately deferred** — sqlx feature set, some C deps,
testing story on Windows are all extra work.
**Pi 2 / 32-bit ARM (`armv7-unknown-linux-gnueabihf`) excluded** —
1 GB RAM is below OxiCloud's practical floor even with face indexing
disabled.
**Building strategy for Linux musl targets** — run the compilation
inside the `rust:1.96-alpine3.24` container image the Dockerfile
already uses. Guarantees byte-for-byte parity with what ends up in
the published Docker image; zero new toolchain to maintain. Runner
just needs Docker (all GitHub-hosted Linux runners have it). No
`rustup target add`, no `apt install musl-tools`.
**CPU baseline** — the repo sets `-C target-cpu=native` for x86_64 and
aarch64 hosts (`.cargo/config.toml:11-12`). That flag makes the binary
use every CPU feature the BUILDER exposes — great for local dev,
catastrophic for distributed binaries: a runner with AVX-512 produces
a binary that segfaults on any older CPU. Precedent for the fix at
`.github/workflows/load-smoke.yml:28`, which already overrides with
`RUSTFLAGS="-C target-cpu=x86-64-v3"` for load tests.
Per-target baseline for `release-binaries.yml`:
| Triple | `RUSTFLAGS` |
|---|---|
| `x86_64-unknown-linux-musl` | `-C target-cpu=x86-64-v2` |
| `aarch64-unknown-linux-musl` | `-C target-cpu=generic` (safe ARMv8-A baseline) |
| `aarch64-apple-darwin` | `-C target-cpu=apple-m1` |
`x86-64-v2` covers ~2010+ processors (Nehalem, Bulldozer). Widest
realistic install base for a "runs everywhere" tarball. Notably
different from Docker's `x86-64-v3` (per `load-smoke.yml:28`) — Docker
targets performance-tuned deployments, tarballs target maximum
compatibility.
Trade-off left on the table: BLAKE3 SIMD + image codecs run somewhat
slower on v2 than v3. For a self-hosted personal cloud workload this
is invisible; for anyone who wants max perf, the Docker image is
still their better option.
### 4. Tarball layout
One archive per platform. **Four files inside**, all rooted under a
per-version-per-triple directory so extraction lands cleanly:
```
oxicloud-0.9.0-<triple>/
├── oxicloud ← the single binary (server + tools + embedded assets)
├── example.env ← copied verbatim from repo root (50 KB, all env vars documented)
├── LICENSE ← copied verbatim from repo root
└── README-install.md ← NEW, ~100 lines, tarball-audience-specific
```
Deliberate exclusions:
- **`README.md`** (repo root, 10 KB) — the GitHub landing page: features,
screenshots, tech stack, contribution guide. Wrong orientation for a
downloaded tarball. Users get `README-install.md` instead — shorter,
focused on "how do I run this thing on this box?"
- **`oxicloud.service` systemd unit** — inlined as a copy-paste block in
`README-install.md`. Users have to customize `User=` /
`WorkingDirectory=` anyway; a documented example beats a shipped file
that pretends to be canonical.
- **`CHANGELOG.md`** — the GitHub Release page carries the notes.
Duplicating invites drift.
- **`docs/`** — full documentation stays on GitHub, linked from
`README-install.md`.
`README-install.md` content shape (~100 lines):
- **Quickstart** — required env vars, one-command run
- **PostgreSQL setup** — link to project docs; note `pg_trgm` + `ltree`
extensions
- **Optional: video thumbnails** — mention ffmpeg + the
`OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false` kill switch (first
user-facing surface for this env var, closing the discoverability
gap flagged in memory `bug_env_docs_video_thumbnails_missing`)
- **Systemd unit example** — inline copy-paste block, references
`/etc/oxicloud/oxicloud.env` for env vars
- **First-run** — direct to `/setup` for admin account creation
- **Verification** — `sha256sum -c ../SHA256SUMS` for tarball integrity
- **Upgrading** — replace binary in place, restart service; migrations
run automatically on boot per `sqlx::migrate!()`
- **Support links** — GitHub Issues, docs site
- **Docker note** — for users who want the container path instead
Tarball name: `oxicloud-<version>-<triple>.tar.gz`.
macOS tarballs stay `.tar.gz` too (not `.zip`) — Homebrew formulas
handle either, and it keeps the CI packaging step uniform. Same
extraction UX cross-platform (`tar xzf`).
`SHA256SUMS` file lists all archives with hashes at the release-level
(next to the tarballs, not inside them) — standard OSS practice.
Users verify via `sha256sum -c SHA256SUMS` before extraction.
### 5. New workflow: `.github/workflows/release-binaries.yml`
Three-stage pipeline, shared frontend build:
```
1. frontend-build (ubuntu-latest, single job)
- checkout
- Node 26 setup
- npm ci && npm run build (writes static-dist/ at repo root)
- upload static-dist/ as artifact "static-dist"
2. binary-build (matrix over 4 targets, needs: frontend-build)
- checkout
- download static-dist artifact into repo-root static-dist/
- Linux targets: docker run rust:1.96-alpine3.24, cargo build inside
- macOS targets: rustup target add + native cargo build
- cargo build --release --features bundled-assets --bin oxicloud
- tar czf oxicloud-<version>-<triple>.tar.gz oxicloud-<version>-<triple>/
- upload tarball as per-platform artifact
3. release (ubuntu-latest, needs: binary-build)
- download all tarball artifacts
- compute SHA256SUMS
- softprops/action-gh-release@v2 with files: dist/*
```
Triggers: `push: tags: v*` (real releases) + `workflow_dispatch` with
`dry_run: true` toggle (build tarballs, upload as workflow artifacts,
skip attaching to a release).
Interaction with existing `release.yml`: **new file**, because the
current `release.yml` is tiny (create release + notes) and mixing
concerns would clutter it. `release.yml` stays as "make the GitHub
Release exist"; `release-binaries.yml` stacks binaries into it. Both
trigger on `push: tags: v*`.
**Parallel-fire behaviour on tag push** — on `git push origin v0.9.0`,
three workflows fire simultaneously:
```
tag push v0.9.0
│
├─── release.yml (~1 min) Release + notes
├─── docker-publish.yml (~30-45 min) multi-arch Docker → GHCR + DockerHub
└─── release-binaries.yml (~25-30 min) 4 tarballs → attach to Release
```
Total wall-clock: ~30-45 min (dominated by whichever build is slower).
No sequencing between the three — each has a single responsibility
and runs independently.
Race with `release.yml` is **benign** because `release-binaries.yml`
uses `softprops/action-gh-release@v2`, which:
- **Adds files** to an existing Release if one exists for the tag.
- **Creates** the Release (with default settings, no notes) if
`release.yml` hasn't finished yet.
Worst case: `release-binaries.yml` finishes first on a tiny tag, creates
a bare Release, `release.yml` catches up and fills in the notes. Users
see the Release progressively; nothing breaks. If this becomes annoying
in practice (unlikely — `release.yml` is ~1 min), flip
`release-binaries.yml` to `on: workflow_run: { workflows: ["Release"],
types: [completed] }` to serialize.
Concurrency: same `${{ github.workflow }}-${{ github.ref }}` group as
`docker-publish.yml`, but `cancel-in-progress: false` — every tag is
unique and immutable, so a superseded release build has nothing to
cancel.
Publish gate: same fork-friendly pattern as `docker-publish.yml` —
`if: github.repository == 'AtalayaLabs/OxiCloud' ||
vars.ENABLE_BINARY_RELEASE == 'true'`. Prevents forks from
auto-attaching binaries to their own tag pushes.
### 6. Docs
- **`docs/install/binary.md`** — quickstart per platform, verify
SHA256SUMS, minimum env vars (`DATABASE_URL`), systemd unit
example, Pi-specific advice (link to the "verified on" hardware
table). Prose only — no code snippets that could go stale.
Include a "server-side video thumbnails" callout naming the
`OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false` kill switch — this is
the first user-facing surface where the env var is discoverable
(per memory `bug_env_docs_video_thumbnails_missing`, it's not
in `example.env` nor `docs/env.md` today). Consider fixing the
underlying gap in `example.env` + `docs/env.md` as a companion
edit to this PR — small win, high visibility.
- **`README.md`** — add a one-line pointer under Installation:
"Binary releases attached to each GitHub Release — see
[docs/install/binary.md]". Do NOT list per-triple download links
by hand; they'd rot.
- **This file** — the design record. Kept alongside other
`docs/plan/*.md` docs so the next maintainer sees the rationale
before touching `release-binaries.yml` or the embed layer.
### 7. `Cargo.toml` `[package.metadata.binstall]` block
Free win: `cargo binstall oxicloud` starts working once the tarballs
land on GitHub Releases with predictable names. Two-line metadata
block declares the URL template:
```toml
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/v{ version }/oxicloud-{ version }-{ target }.tar.gz"
bin-dir = "oxicloud-{ version }-{ target }/{ bin }{ binary-ext }"
```
No CI change; the tarballs already follow this shape from Deliverable 4.
## Critical files
- `Cargo.toml` — add `bundled-assets` + `dev_tools` features,
`required-features` on gated bins, `rust-embed` optional dep,
`[package.metadata.binstall]` block. Delete the
`[[bin]] name = "oxicloud-cli"` and `[[bin]] name = "migrate-nfc-filenames"`
blocks (Deliverables 1a + 1b).
- `src/main.rs` — add clap parsing at the top of `main()`. If a
subcommand is present → dispatch via new `src/cli/` module; otherwise
fall through to the existing server-init path (backwards-compat
implicit-server mode).
- `src/cli/mod.rs` — NEW. Root of the operator-tools tree; contains
`Domain` enum + submodules moved from `src/bin/oxicloud-cli.rs`.
- `src/cli/opaque.rs` — NEW. `opaque setup` + `opaque reset` moved
from the old `oxicloud-cli.rs`.
- `src/cli/migrate.rs` — NEW. `migrate nfc-filenames` — the ~149
non-boilerplate lines from the old `migrate-nfc-filenames.rs`,
wrapped as a clap subcommand.
- `src/bin/oxicloud-cli.rs` — DELETE (contents absorbed into `src/cli/`).
- `src/bin/migrate-nfc-filenames.rs` — DELETE (contents absorbed
into `src/cli/migrate.rs`).
- `src/interfaces/web/mod.rs` — 400-line file, owns the static-serving
surface. Four sites gain a `#[cfg(feature = "bundled-assets")]`
alternative:
- `resolve_static_path()` (`:25-35`) — returns a `StaticSource`
enum under bundled mode; a bare `PathBuf` otherwise
- `create_web_routes()` (`:47-106`) — swap the two `ServeDir`
constructions for embedded-asset handlers
- `content_security_policy()` + `inline_script_csp_hashes()`
(`:163-233`) — iterate `EmbeddedAssets::iter()` instead of
`fs::read_dir`
- Import block + type imports for the new source enum
- `src/interfaces/web/embedded.rs` — NEW: `#[derive(RustEmbed)]` struct
+ two axum handlers (root/SPA-fallback + `_app/immutable`-prefixed
with cache header) + shared MIME helper. ~100 lines.
- `src/main.rs:599-615` — locale-source resolution. Under bundled
mode, call `LocaleRegistry::discover_embedded()` instead of the
filesystem variant when `resolve_static_path()` returns
`StaticSource::Embedded`.
- `src/common/locale.rs:150-221` — add `LocaleRegistry::discover_embedded()`
under `#[cfg(feature = "bundled-assets")]`. Same parse + registry
build, source is `EmbeddedAssets::iter()` filtered to `locales/*.json`.
- `src/infrastructure/services/file_system_i18n_service.rs` — either
extend to accept an `EmbeddedLocales` source alongside the
filesystem one, OR ship a second `EmbeddedI18nService` impl of the
same trait. Latter avoids polluting the fast filesystem path with
cfg gates.
- `build.rs` — EXISTS today (injects `GIT_HASH`/`GIT_BRANCH` from git).
Extend with a second block: when the `bundled-assets` feature is
enabled (`env::var("CARGO_FEATURE_BUNDLED_ASSETS").is_ok()`),
check that repo-root `static-dist/` exists and contains at least
`index.html`. Emit a clear compile error pointing at
`just fe-build` / `(cd frontend && npm run build)` if missing.
Also emit `cargo:rerun-if-changed=static-dist/` so a rebuild of
the frontend re-triggers rust-embed's compile-time embed step.
- `.cargo/config.toml` — NO CHANGES. The dev-preserving default of
`-C target-cpu=native` stays. Release CI overrides via per-job
`RUSTFLAGS` env var, per the load-smoke.yml precedent.
- `.github/workflows/release-binaries.yml` — NEW: three-stage pipeline.
- `justfile` — thread `--features dev_tools` into the `openapi` recipe.
- `tests/api/run.sh` — thread `--features test_utils` into the two
hurl-helper build lines.
- `docs/install/binary.md` — NEW: user-facing installation guide.
## Verification
1. **Local squash check**: after Cargo.toml + `src/cli/` edits, run
`cargo build --release --bins` and confirm exactly ONE binary
appears in `target/release/` (`oxicloud`). Run `cargo build --release
--bins --features test_utils` and confirm the two hurl helpers
appear. `cargo build --release --bins --features dev_tools`
should surface `generate-openapi`. Confirm subcommand shape via:
- `target/release/oxicloud --help` — shows `opaque` + `migrate`
domains
- `target/release/oxicloud opaque setup` — prints a fresh
ServerSetup base64 line (unchanged behaviour vs the old
`oxicloud-cli opaque setup`)
- `target/release/oxicloud migrate nfc-filenames --dry-run`
(against a sandbox DB) — same behaviour as the old
`migrate-nfc-filenames --dry-run`
- `target/release/oxicloud` (no args) — starts the server exactly
as today, no clap-related output surprises before the server
init banner.
2. **Local bundled-assets smoke**:
```
(cd frontend && npm ci && npm run build) # writes ../static-dist/
cargo build --release --features bundled-assets --bin oxicloud
# Wipe static-dist/ or point OXICLOUD_STATIC_PATH somewhere
# nonexistent to force the embedded path to be exercised.
mv static-dist/ static-dist.hidden/
OXICLOUD_STATIC_PATH=/tmp/nonexistent DATABASE_URL=... target/release/oxicloud
# Hit http://localhost:8086 — SPA shell + locales must load.
# Restore afterward: mv static-dist.hidden/ static-dist/
```
3. **Filesystem fallback still works in bundled build**: with the
same binary, point `OXICLOUD_STATIC_PATH` at a real static-dist,
confirm files served from disk (change a file, no rebuild → change
visible in browser). Verifies the precedence rule from Deliverable 2.
4. **Non-bundled build still works**: `cargo build --release`
(without `--features bundled-assets`) → binary boots + serves from
`./static/static-dist/` as today. Zero regression on the Docker
image path.
5. **CI dry-run**: dispatch `release-binaries.yml` with `dry_run: true`
from a fork. Confirms all four matrix entries build successfully,
tarballs land in the run's artifact list, no release is created.
6. **Manual extraction test**: download one tarball, extract, run
`./oxicloud` with just `DATABASE_URL` set (against a local
Postgres). Log in, upload a file, check that locale switching
works, confirm `/api/status` returns healthy. Then repeat on a Pi 5
for the `aarch64-unknown-linux-musl` variant if convenient.
## Not in scope
- **Windows target** — separate work when demand appears.
- **glibc Linux tarballs** — musl covers the Linux audience per the
design shape above; users wanting glibc-specific features
(`faces-onnx`) build from source.
- **32-bit ARM (`armv7`)** — hardware below the workload floor.
- **Debian/RPM packages** — post-tarball layer, adds repo-hosting burden.
- **Homebrew tap** — trivial once tarballs exist; separate decision.
- **Signing (Sigstore/GPG)** — worth adding but scope-creeping;
SHA256SUMS is the minimum table stakes for this PR.
## Delivery order
1. **Feature-flag squash** (Deliverable 1). Cargo.toml edits +
`just openapi` / `tests/api/run.sh` invocation fixes. Verify
`cargo build --release --bins` no longer builds hurl helpers.
2. **Merge migrate-nfc-filenames into oxicloud-cli** (Deliverable 1a).
Move logic to `mod migrate` submodule. Delete standalone bin.
Verify `oxicloud-cli migrate nfc-filenames --dry-run` works.
3. **Merge oxicloud-cli into oxicloud** (Deliverable 1b). Move
`src/bin/oxicloud-cli.rs` contents into new `src/cli/` module,
wire clap into `main.rs` with implicit-server default. Delete
`src/bin/oxicloud-cli.rs`. Verify `oxicloud` (no args) still
starts the server; `oxicloud opaque setup` + `oxicloud migrate
nfc-filenames --dry-run` work.
4. **Add `bundled-assets` feature** (Deliverable 2). `rust-embed` +
`build.rs` guard + `src/interfaces/web/embedded.rs` + locale
loader alt + CSP scan alt. Verify locally with the smoke sequence
in Verification §2.
5. **Add `.github/workflows/release-binaries.yml`** (Deliverable 5).
Dry-run on a fork. Iterate until all 4 targets green.
6. **Write docs** (Deliverable 6) — `docs/install/binary.md`. Prose
only, no snippets that will rot. Include the
`OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false` callout for tarball users
without ffmpeg.
7. **Add `[package.metadata.binstall]` block** (Deliverable 7).
One-line change enabling `cargo binstall oxicloud`.
8. **Fix the `OXICLOUD_ENABLE_VIDEO_THUMBNAILS` doc gap** — add to
`example.env` + `docs/env.md` per memory
`bug_env_docs_video_thumbnails_missing`. Small companion edit
surfaced by the binary-install docs work.
9. **Cut a test tag** (`v0.9.0-rc1`?) on a fork with
`vars.ENABLE_BINARY_RELEASE=true`. Confirm tarballs attach to the
Release, SHA256SUMS present, `cargo binstall oxicloud` works.
10. **When happy, cut on canonical.**
Total scope: ~2.5 days of careful work.
- Deliverables 1 + 1a + 1b: ~5 hours mechanical (Cargo config, CLI
merge, subcommand tree)
- Deliverable 2: ~1 day — the only piece with real design surface
(embed swap, four cfg sites, locale + CSP loaders)
- Deliverable 5: ~4 hours workflow authoring + iteration
- Deliverables 6-8: ~4 hours docs + small edits
- Verification + iteration: ~4 hours
docs/plan/derived-blobs.md
+1640 -72
View File
File diff suppressed because it is too large Load Diff
docs/plan/hidden-system.md
+246
View File
@@ -0,0 +1,246 @@
# Plan — Hidden system drive for user-owned objects
**Status:** design captured 2026-08-21. Not implemented. Sibling to
`docs/plan/derived-blobs.md`, which answers "where does *derived*
content live"; this one answers "where do *user-owned binaries* live".
The two share one rule — **point at a file, never at a blob** — and
that rule is the reason neither needs new blob-referencing tables.
## Problem — binaries in the users row
`auth.users.image TEXT` (migration `20260526000000_add_user_image.sql`)
holds the avatar inline, up to 512 KiB. It is the wrong home, and the
cost is already measured rather than theoretical:
- **It TOASTs, and every wide read pays.** The repository comment on
`get_users_by_ids` records a group fan-out that "detoasted + shipped
+ parsed M avatars purely to discard them", fixed by adding a narrow
projection (`benches/ROUND12.md §Q1`, `ROUND13.md §Q1`). That
workaround exists *because* the column is in the wrong place; the
rule it leaves behind — "add a wide sibling rather than widening
this one back" — is a permanent tax on every future query.
- **Base64 inflation.** A ~384 KB image becomes ~512 KB of TEXT.
- **No dedup.** N users sharing a default or IdP-supplied avatar cost
N copies.
- **None of the storage stack applies** — no `EncryptedBlobBackend`,
no backend migration, no key rotation, no local cache, no
consistency coverage.
- **Backups and replication carry it.** Binary weight lands in the
logical dump and on every replica, forever.
The same pressure is coming for the UI background and a signature
image, so this needs a general answer, not another column.
## The rule — point at a file, never at a blob
```sql
ALTER TABLE auth.users
ADD COLUMN avatar_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
ADD COLUMN background_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
ADD COLUMN signature_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL;
```
`storage.files` is already a `BlobReferenceSource`, already covered by
every consistency edge, already GC-integrated, already copy- and
version-aware. A pointer to a *file* therefore adds **zero new
reference sources and zero new consistency edges** — `auth.users`
holds no blob reference at all, only a pointer to a row that does.
Deleting the file decrements the blob refcount through the existing
file-deletion path.
Point at `id`, never at a name or path: a rename must not break a
profile.
For a small fixed set, **columns beat a table.** A table becomes right
only when the object set is open-ended, and it would cost exactly what
the pointer avoids.
### Rejected alternatives
| Option | Why not |
|---|---|
| **New table → blobs/chunks** | Needs a new `BlobReferenceSource`, a new fragment in the manifest sweep, and a new dangling check. Precisely the complexity the pointer removes. |
| **Direct backend paths** (`profile/{uuid}/avatar.png`) | `backend_migration` enumerates *blobs*, so a Local→S3 cutover **silently drops every avatar**. `EncryptedBlobBackend` is hash-keyed, so writes either bypass encryption or need a parallel path. `backend_consistency` raises `unknown_backend_file` (severity `anomaly`, "non-canonical file in blob namespace") per object on every sweep. And a fixed key overwritten in place breaks the immutability everything else rests on, killing `Cache-Control: immutable`. |
| **Reserved folder in the user's own drive** (`.profile/`, `.oxiprofile/`) | The user can write to it, so there are **two write paths and only one is validated** — the upload endpoint's format/size checks are bypassable over WebDAV and sync. It is in the sync tree, so "hidden" is not hidden in the protocols that matter. Existence-by-path drags back the extension problem. And any reserved name squats a namespace users own — `.profile` is a POSIX shell file, so it collides by default for anyone syncing a Linux home. |
| **One drive per user** | Per-user creation at signup, backfill for existing users, per-user quota exemption, and a cascade on account deletion. All avoidable — see below. |
Underneath all of it: these objects are owned by the **application on
the user's behalf**, not by the user as documents. Putting them in a
document tree conflates the two, and every problem above follows.
## The hidden system drive
**One shared drive, not one per user.** `kind = 'system'`, alongside
today's `CHECK (kind IN ('personal', 'shared'))`. Files inside are
owned by their respective users via the normal `created_by` /
ownership columns; the drive is only a container.
Sharing one drive drops per-user creation at signup, backfill for
existing users, and per-user quota exemption. Deleting a user becomes
a query over `storage.files` rather than a drive cascade.
Properties it needs:
- **Hidden at drive enumeration.** This is the single filter point,
and it is why the drive beats a folder: a folder must be filtered in
directory listings, search results, recent items, trash, photo
indexing and sync deltas, whereas a drive is filtered once where
drives are listed. Every surface must honour it — REST, WebDAV,
NextCloud, search, quota reporting. **A missed filter is the
characteristic bug of this design**, so it deserves a test per
surface rather than per call site.
- **Trash disabled.** Otherwise every replaced avatar lands in a trash
nobody can see, holding a blob reference that GC cannot reclaim
while retention keeps it alive — invisible storage growth with no
signal. Deletion here is immediate.
- **Exempt from the user quota envelope.** Nobody should pay quota for
their own avatar.
- **Created at install, fail-fast at boot.** If the drive is missing,
panic rather than silently disabling profile objects — a silently
absent avatar surface is worse than a refusal to start.
- **Visible to admins.** Ops need to see it for storage accounting
even though it is hidden from users.
## Visibility — per kind, in code
Reads go through a service method carrying an explicit policy, audited
like any other authorization decision. Because the column set is fixed
and small, the policy is a `match`, not stored data — there is nothing
to misconfigure, and adding a column forces adding an arm:
| Object | Who may read | Why |
|---|---|---|
| `avatar` | **the same rule as profile visibility** | Not "any authenticated user". `AGENTS.md` has `user_profile.rejected` return **404, never 403**, for an external caller with no relationship, specifically so existence cannot be confirmed. An avatar endpoint answering 200 for any caller is an oracle around that control. |
| `background` | owner only | Nobody else has a reason to fetch it. |
| `signature` | owner only, plus the document render path | A handwritten signature is forgery material. It is "public" only in the sense that it appears on documents you may already read — which argues for rendering it into those documents, not exposing it as a directly-readable object. |
Note the consequence: the drive's own permission model is **not** what
governs these reads. The object lives in a drive and is read through a
different door. That is a deliberate choice, not an oversight — record
it so nobody later "fixes" it by granting cross-user drive access.
## What must NOT live here
> **If losing control of it is a security incident rather than a
> cosmetic bug, it stays in the database.** The system drive is for
> user-facing binaries.
So E2E/Vault key material — public key bundle, passphrase-wrapped
private key, recovery kit — stays in `auth.users` columns. Four
reasons, the last decisive:
1. **Failure-mode asymmetry.** The characteristic bug here is a missed
listing filter. For a wallpaper that is cosmetic; for key material
it is disclosure.
2. **Atomicity.** Rotating a passphrase rewraps the private key
*together with* the credential change. A DB column makes that one
transaction; a file write plus a column update cannot be atomic.
3. **Size.** A few KB — blob storage buys nothing.
4. **`EncryptedBlobBackend` encrypts under a key the server holds.**
For E2E material the whole premise is that the server *cannot*
decrypt. Routing a wrapped private key through the blob layer
encrypts it twice, once under a key the operator controls, adding
no protection while creating the impression of it.
Users who want to store genuinely private *files* already have the
personal drive, with the full AuthZ engine behind it. There is no gap.
## Migrating the avatar off `auth.users.image`
Volume is one row per user, so unlike the thumbnail migration this
needs **no read-through phase** — a single batch job is enough.
**Phase 1.** Add the pointer columns and the system drive. Write path
switches to files; read path prefers `avatar_file_id` and falls back
to `image` when null.
**Phase 2.** `profile_image_import`, a registered `JobRegistry` job
(subject-first naming, per convention). For each user with a non-null
`image`:
1. Decode the data URI; skip and log if it does not parse, rather than
failing the batch.
2. `store_from_stream` the decoded bytes → derived blob + manifest.
3. Insert a `storage.files` row in the system drive, owned by that
user.
4. Set `avatar_file_id`.
Idempotent (`WHERE avatar_file_id IS NULL`), resumable via a user-id
cursor, and reports imported / skipped-unparseable / failed counts.
**Phase 3.** Drop `auth.users.image` and the fallback, gated on the
job reporting zero remaining. Dropping the column is what actually
reclaims the TOAST weight and retires the narrow-projection rule in
`get_users_by_ids`.
**IdP-sourced avatars.** OIDC login already refreshes the avatar
("same IdP avatar, already verified" — `user_pg_repository.rs:1435`).
That path must be converted at Phase 1, not Phase 3, or it keeps
writing to a column the migration is draining.
## Object catalogue
**Now:** avatar, UI background, signature image.
**Strong future candidates** — these are what justify a drive rather
than three columns and a corner:
| Object | Why it fits |
|---|---|
| **Data exports** (GDPR takeout, drive-export zip) | Generated async, large, downloadable, should expire. Today there is nowhere to put them. |
| **Staged imports** (Google Takeout, NextCloud export) | Multi-step ingestion needs durability beyond a temp file. |
| **Share-page branding / logo** | Per-user or per-org, served on public share pages. |
**Same problem, different owner — this drive does not help:**
`carddav.contacts.photo_url TEXT` (contact photos, today a URL or an
inlined data URI) and CalDAV `ATTACH` event attachments. They are keyed
by contact and by event, not by user. But the *pointer* generalises:
`contacts.photo_file_id UUID REFERENCES storage.files(id)` solves them
with no new blob-referencing table either — they simply live in the
address book's or calendar's own drive rather than here. Own plan.
## Operational details
- **Replace must delete.** Write new file → update pointer →
hard-delete the previous file. `ON DELETE SET NULL` protects the
pointer when a file vanishes, but nothing deletes the old file
because the pointer moved.
- **Validation lives at the endpoint** and is now the only write path,
which is the point of not using a user-writable location. Enforce
format, dimensions and size there.
- **Account deletion** deletes the user's files in the system drive
explicitly; the pointer columns are on the row being deleted anyway.
## Non-goals
- **Secrets of any kind.** See the discriminator above.
- **Per-user system drives.** One shared drive; revisit only if
per-user quota or trash semantics ever become necessary.
- **Contact photos and event attachments.** Same pointer pattern,
different owner, different drive — separate plan.
- **A generic "user objects" API.** The column set is fixed and small
on purpose. Reach for a table only when it demonstrably is not.
## Open questions
- Who owns the system drive row itself, and what does
`drives_consistency` expect of a drive with no human owner?
- Does the signature object survive the "owner only" rule, or does
document rendering need a broader read path than expected?
- Should exports live here or in a short-lived namespace with its own
expiry, given they are the only candidate with a natural TTL?
## References
- `docs/plan/derived-blobs.md` — the sibling plan; shares the
point-at-a-file rule and documents the consistency coverage matrix
these objects inherit for free.
- `migrations/20260526000000_add_user_image.sql` — the column being
retired.
- `migrations/20260802100000_drives_schema_additive.sql` — the
`kind IN ('personal','shared')` constraint this extends.
- `src/AGENTS.md` — the backend-abstraction rules, and the
anti-enumeration pattern the avatar visibility rule follows.
- Memory `project_drive_naming_and_vault_reservation` — "Vault"
reserved for the future E2E kind whose key material this plan
explicitly keeps out of the drive.
docs/plan/job-registry.md
+144
View File
@@ -171,6 +171,58 @@ Native services implement this trait on an existing service type (no
new wrapper) and register a single `Arc<dyn JobHandler>` with the
scheduler.
### Self-description — `description` / `mutates` / `repair_description`
Three defaulted methods on both `JobHandler` and `RecoverableJobHandler`
let a job tell the admin UI what it is. `RecoverableAdapter` forwards
them, since the registry only ever holds `dyn JobHandler`.
```rust
fn description(&self) -> &'static str { "" }
fn mutates(&self) -> Mutates { Mutates::Never }
fn repair_description(&self) -> Option<&'static str> { None }
pub enum Mutates { Never, Always, OnRepairOnly }
```
They surface on `JobSummary` (`GET /api/admin/jobs`) and drive the
panel: `Never` earns a read-only badge and triggers straight through,
`Always` confirms first, `OnRepairOnly` is safe to run and confirms only
when the repair variant is picked. `repair_description.is_some()` is
what renders the repair toggle at all, and its text is the confirmation
copy.
**Why three values and not a boolean.** A job can be read-only by
default and destructive under `?repair=true`; a boolean has to answer
wrongly for one of those two modes, and `false` on something that
deletes files is the dangerous direction to be wrong in. It is also
where the recovery framework is heading — discovery-only default,
mutation behind an opt-in — so a tenant that later grows a repair arm
changes this one value and nothing else.
**Why `Option<&str>` and not `supports_repair: bool` + prose.**
Presence gates the toggle, content supplies the wording. Split across
two methods they can disagree; and the frontend cannot invent the
wording itself, because correcting a counter and unlinking files off
disk are not the same warning. The two are independent, not derived
from each other: the thumbnail imports are `Always` *and*
repair-capable.
`OnRepairOnly` with no `repair_description` is rejected at registration
— it claims to mutate only under a flag it does not support, and would
render as safe with no reachable mutating path.
**Why English in the trait, not `locales/*.json`.** A description that
lives away from the behaviour rots the moment a job changes, invisibly,
and a translator cannot know what `manifests_consistency` reconciles.
i18n can layer on later keyed by job name with these as the fallback,
matching the frontend's `t(key, params, fallback)` — a missing
translation then degrades to English from code rather than to a blank
panel. No rework needed to get there.
Defaults exist so the methods could be added without touching every
job at once; every registered job declares all three today.
### `JobOutcome`
```rust
@@ -685,6 +737,98 @@ trigger) resumes any `Paused` row per the normal flow.
Consistency-check.md's existing consistency-scoped sweep collapses
into this general one.
### Startup jobs — `OXICLOUD_STARTUP_JOBS`
A comma-separated list of jobs to dispatch once, in the background,
after the scheduler is ready. Each entry is a registered job name,
optionally with the same query syntax the admin trigger URL uses.
**The default is both migration jobs, in repair mode:**
```
OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true
```
An explicit value replaces that list; an empty value disables startup
jobs entirely.
**Why it exists.** Scheduled ticks deliberately never pass `repair` — a
job that deletes on its default setting is what no-silent-auto-repair
forbids. But that left the migration jobs unable to finish on their
own: a deployment whose operator never opens the admin panel re-imports
sidecars it already imported, forever, and never drains the directory.
**Why the default deletes anyway.** Relying on operators to edit `.env`
has the same failure mode one level up — the ones who never edit it are
exactly the ones whose migration never completes. So this is a
deliberate exception to no-silent-auto-repair, and it rests on three
properties that must keep holding:
- **Nothing is deleted before its replacement has been read back.**
`verify_and_unlink` imports, reads the blob back through the normal
stack, and only then unlinks; a store that reported success but landed
unreadable keeps its sidecar. This matters most for
`thumb_attached_import`, whose bytes are user-uploaded previews with
no render path — a wrong deletion there is permanent, where a wrong
deletion of a server-rendered thumbnail costs a re-render.
- **Sidecars whose source is gone are deleted without a readback**,
because there is nothing to read back and nothing can reference them
again. Unrecoverable and unreachable are different things; these are
both.
- **Every deletion is audited**, so what a boot removed, and from which
source, is reconstructable afterwards.
The consequence to hold in mind: an upgrade deletes on first boot, in
every deployment at once, with no operator action. A regression in the
readback path would be simultaneous and unrecoverable, so that code is
load-bearing. Operators who want to inspect before committing set
`OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import` —
same jobs, import only.
It is not a "run everything in repair mode" switch. Each job is named
individually and carries its own flags.
**Validation is fail-fast.** An unknown job name panics at boot — the
registry is fully populated by then, so a name that doesn't resolve is a
typo or a stale rename, and ignoring it would leave a migration that
silently never runs. Unknown flags panic too: a dropped `?repare=true`
would leave the job in discovery-only mode while the operator believed
the tier was draining, and the symptom ("it never finished") surfaces
months later with nothing pointing back at the config.
**Dispatch is non-blocking.** `tokio::spawn`, so readiness never waits
on a job that may walk a filesystem for hours. Jobs in the list run
sequentially within that task, not concurrently: they contend for the
same directories and pool, and the exclusivity gate would turn overlap
into a *skipped* run rather than a queued one.
**Interrupted runs resume.** The boot recovery sweep above runs first
and flips every abandoned `Running` row to `Paused` with its cursor
intact; `run_or_resume` then picks Resume over a fresh start. So a
migration killed by a restart continues where it stopped, and completes
across however many restarts it takes.
That is a deliberate exception to "do NOT auto-resume" — scoped to the
named jobs only. The rule protects against a restart silently resuming
work nobody asked for; here somebody did ask, in configuration, and not
having to ask again is the entire point. Every other paused run still
waits for an operator.
A resumed run keeps the flags it started with (`repair` / `deep` are
persisted to `params` on the fresh open and read back on resume), so
editing the config mid-migration does not retroactively change a run
already in flight.
**Safe to leave set.** Each job is idempotent and resumable; once the
tier has drained, a run is a `read_dir` over three directories that
returns nothing — and after the directory is removed, not even that.
**Visible in the admin panel.** These are ordinary registered jobs:
they appear in `GET /api/admin/jobs`, are triggerable by hand, and
record the same runs and findings. Rows named here additionally carry a
`startup` object with the configured flags, so an operator can see that
a job deletes files on every boot rather than only when someone clicks.
### Admin surface (recoverable runs)
Same URL taxonomy as Part 1 — resource-first, action second, all
docs/plan/jobs-handling-recoverable-error.md
+265
View File
@@ -0,0 +1,265 @@
# Recoverable errors in jobs — retry, then pause
**Status: not started.** Design settled 2026-08-31, from a live
diagnosis (see [Motivating incident](#motivating-incident)).
A job that hits a failing backend today has two possible endings, and
neither is right for an outage: it fails the run (throwing away a
partially-complete migration, since `Failed` is terminal and only
`Paused` resumes), or it hangs forever inside an SDK retry loop with no
log line and no way to act on it.
This plan adds the third: **retry a bounded number of times, then pause
with the reason recorded**, so an operator resumes when the provider
recovers and the job continues from its cursor.
---
## Motivating incident
`backend_migration ?storage=azurite` hung indefinitely. Diagnosis, after
several wrong theories:
- The job issued a ranged GET carrying `x-ms-range-get-content-crc64`.
- Azurite answered **500** (real Azure supports CRC64 range validation;
the emulator does not).
- `azure_core`'s retry policy classifies 500 as retryable and loops.
- The response was deterministic, so every retry failed identically.
- The job never advanced, never failed, and emitted no per-blob line —
while holding `migration_readonly`, refusing writes **across the whole
application**.
The exact chain was pinned down later (2026-09-02) and is worth having,
because it is not where you would look — the copy itself is innocent:
```
backend_migration_service.rs target.head_check(hash) ← pre-write probe
→ EncryptedBlobBackend::head_check
→ get_blob_range_stream(hash, 0, HEADER_SIZE) ← ~40 bytes
→ azure_core Range::as_headers ← adds the CRC64
(src/request_options/range.rs: any range < 4 MiB) header, no opt-out
```
`copy_blob` reads from the SOURCE, which is local in a local→Azure
migration, so it never touches an Azure range. What hangs is the format
probe against the TARGET, on the first blob, before a byte is copied.
A workaround exists — unranged `get()` for small requests, truncate
client-side — and was **rejected 2026-09-02**: it pays for an emulator
with production read amplification and puts new offset arithmetic on the
read path. See the note on `AzureBlobBackend::get_blob_range_stream`.
The fix is the official SDK, where `range_get_content_crc64` is an
explicit field. **This plan is unaffected either way** — a bounded retry
would have turned the hang into a Paused run with a reason, which is the
point.
Two properties made it invisible: the 500 was only visible at
`azure_core=debug`, and nothing bounded the retry. The same shape would
occur against real Azure or S3 on any persistent 5xx; it is not an
emulator quirk. `AzureBlobBackend` configures no retry policy and no
timeout at all — `grep "retry\|timeout\|ClientOptions"` on
`azure_blob_backend.rs` returns nothing.
Full evidence chain, including the theories ruled out and what each
cost, is in the memory note `bug-azure-put-blob-hangs-no-timeout`.
---
## Why this belongs at the top level
Every recoverable job goes through `run_or_resume`, which already owns
the run lifecycle: it opens the row, persists and restores
[`JobRunArgs`](./job-registry.md) (`63820c3c`), dispatches the handler,
and writes the terminal state. Retry-and-pause is the same kind of
concern — policy about *how a run behaves*, not about what any one job
does.
Implemented there, `backend_rotate`, `transcode_import`, the thumbnail
imports and anything added later inherit it. Implemented per-job, it
gets written once per job and drifts.
It also survives the pending official-Azure-SDK migration untouched,
where per-SDK retry tuning would have to be redone.
**The blocker is that the engine cannot currently act on what it is
told.** A handler returns `RunOutcome::{Completed, Paused, Failed}`, so
a transient backend error is already flattened into `Failed` before the
engine sees it — "the provider is down" and "this data is wrong" are
indistinguishable. Closing that is what makes a top-level
implementation possible, and it is step 1.
---
## Step 1 — errors say whether they are retryable
Today both backends wrap SDK errors into
`DomainError::internal_error("Azure", format!("…{e}"))`, so the status
code survives only inside a formatted string. Recovering it means
string-matching, which is exactly the kind of fragility that turns into
a silent behaviour change when an SDK reformats its `Display`.
Carry the distinction on the type instead — a `retryable` flag, or a
kind the storage ports set deliberately.
**Retryable** (environment, may clear on its own):
- HTTP 5xx, 429 / `SlowDown` / throttling
- connect timeouts, connection resets, DNS failure
**Permanent** (will fail identically forever):
- 4xx other than 429 — 401/403 (credentials), 404 (missing container)
- decode failures, checksum mismatch
- `operation_not_supported`
> **The Azurite 500 is a permanent error wearing a retryable status
> code.** No classification by status alone gets this right, which is
> the case for a bounded cap rather than "retry until it works". The cap
> is the safety net for exactly the errors the taxonomy misjudges.
**Do not double-retry.** The AWS SDK already retries internally with its
own backoff, so a second layer above it multiplies. Check what the S3
backend inherits before adding anything, and consider making the
engine's cap the *outer* bound with SDK retries reduced or disabled.
## Step 2 — an outcome the engine can act on
`RunOutcome` grows a variant meaning "the environment failed, this is
worth trying again later":
```rust
RunOutcome::PausedRetryable { cursor: Vec<u8>, reason: String }
```
Distinct from all three existing outcomes, and the distinction is the
point:
| outcome | meaning | resumes? |
|---|---|---|
| `Failed` | the data or the request is wrong | no — terminal |
| `Paused` | an operator asked it to stop | yes |
| `PausedRetryable` | the environment failed | yes, and says why |
The row lands as `Paused` either way, so resume works unchanged. What
differs is `error_message`, which must let the panel — and an operator —
tell "I paused this" from "the provider went down". Without that
distinction a paused run is an unexplained one.
## Step 3 — the engine implements the policy
In `run_or_resume`:
- bounded exponential backoff, ~5 attempts
- **log each failed attempt at `warn` on our side.** The incident took
several runs to diagnose because the 500 was visible only at
`azure_core=debug`. One line per exhausted operation, naming status,
target and attempt count.
- on exhaustion, write the row as `Paused` with the reason in
`error_message`
Handlers then return the retryable outcome and get the policy for free.
---
## Step 4 — `migration_readonly`, the sharp edge
`backend_migration` holds a gate that refuses writes **application-wide**
until cutover. What happens to it on pause is a correctness question,
not a cosmetic one.
### Start conservative: keep the gate held while paused
Correct, because no writes during the pause means the cursor stays valid
and resume-from-cursor is sound.
It is also **strictly better than today**, which is the thing to
remember when the read-only window looks unattractive: the application
is *already* read-only while the job hangs — there is simply no way to
see why or act. Same lock, now with a reason and two operator actions.
**Cancel must clear the gate.** It ends the run with no swap, so the
source stays active and writes must return. The `failed > 0` path
already clears readonly on the reasoning that "users shouldn't be locked
out because of a partial run" — cancel is the escape hatch operators
will reach for during an outage, and it must work.
**Make the state loud.** Writes refused app-wide should be obvious in
the admin panel, not discovered by reading a run row. The progress
snapshot already feeds the header middleware; a paused migration wants
the same visibility, saying why and offering resume/cancel.
### Why NOT to release the gate on pause (yet)
Tempting — ops should not be locked out during a provider outage — and
unsafe as the job stands:
1. Release readonly; users write again. New blobs land on the source,
which is still active, and are absent from the target.
2. Resume continues **from the cursor**, a position in a hash-ordered
walk.
3. A blob written during the pause whose hash sorts *before* that cursor
is never visited.
4. The run completes, flips the pointer, and reads for that hash 404
against a target that never received it.
This is why the existing `failed > 0` path can safely clear readonly: it
**ends** the run, and an operator retrying starts a new run with a fresh
cursor, so everything is rescanned. Pause-and-resume is what
reintroduces the gap.
Releasing on pause becomes safe only alongside one of:
- **resume rescans from the beginning** rather than trusting the cursor
— cheap, because the walk short-circuits on blobs already present in
the target, so a second pass costs a lookup per blob, not a copy; or
- **a final catch-up pass under readonly before the swap**, with the
pointer flipping only when a complete pass finds nothing new.
### Where this should end up
The second option is the standard online-migration shape, and it is
where this job wants to go regardless: copy the bulk **without** holding
the gate, engage it only for a short final catch-up plus the swap.
That removes what made the incident damaging — writes refused app-wide
for the entire duration of a long copy — and turns releasing-on-pause
into a free consequence rather than a correctness fix. Worth doing as a
follow-up, from the safer base this plan establishes.
---
## Scope: Azure and S3 both
The only reason S3 looks healthy is that the endpoint behaves. A
persistent 5xx from S3 hangs identically — the gap is the absence of a
bound, not anything Azure-specific.
Putting the policy above the `BlobStorageBackend` trait covers Azure,
S3, local and anything added later with one implementation. Per-backend
SDK retry tuning stays a separate, optional refinement — and for Azure
specifically it should wait for the official SDK, since `azure_core`
0.21 is archived and queued for replacement.
## Testing
The `backend_consistency_azure.hurl` scenario and its Azurite service
are already wired (`tests/common/docker-compose.test.yml`,
`spawn-db.sh`), which gives a backend that reliably produces the
failure: Azurite 500s on the CRC64 ranged GET every time. That makes it
a genuine fixture for this work rather than a flake —
**deterministically unretryable-but-retryable-looking**, which is the
hard case.
Note the scenario no longer triggers a migration — it audits Azurite
without cutting over, for the reason in its header. Reaching the fixture
means triggering `backend_migration ?storage=azurite` explicitly, which
is exactly the hang this plan is meant to convert into a Paused run.
Doing that inside the shared suite is what ordering it last was for; it
can go back once the outcome is bounded.
Assert the run reaches `Paused`, that `error_message` names the cause,
and that it does so in bounded time rather than hanging.
`POST /api/admin/settings/storage/test` with `entry_name` is the
pre-flight worth keeping in mind: synchronous, does a real write/read
round-trip, and isolates "the backend is misconfigured" from "the job
is broken". It passing while the migration hung is what ruled out
credentials, container and the write path during the incident.
docs/plan/oidc-account-linking.md
+1 -1
View File
@@ -330,7 +330,7 @@ authenticated session):
**Deferred:**
- Step-up auth before link start
- Admin-mediated link/unlink via `oxicloud-cli federation` (proper for
- Admin-mediated link/unlink via `oxicloud federation` (proper for
"user changed IdP email" recovery scenario)
- OCM link (same shape, different kind)
- Multi-federation (multiple linked identities per user — see
docs/plan/opaque-only.md
+5 -5
View File
@@ -62,7 +62,7 @@ and ordering are the load-bearing decisions here.
## Preconditions before we start the wipe
Every one of these MUST hold. Adding a pre-flight check in
`oxicloud-cli opaque wipe-legacy` (proposed below) that refuses to run
`oxicloud opaque wipe-legacy` (proposed below) that refuses to run
otherwise.
1. **`OXICLOUD_AUTH_OPAQUE_MODE=opaque_only`** on the deployment for at
@@ -174,7 +174,7 @@ it can't, since login-link users just clicked email — no proof-of-current).
### The wipe migration
Delivered as `oxicloud-cli opaque wipe-legacy` — a dedicated subcommand,
Delivered as `oxicloud opaque wipe-legacy` — a dedicated subcommand,
NOT a schema migration. Reasons:
- Idempotent (won't re-wipe already-nulled rows)
- Pre-flight refuses when preconditions aren't met (unlike a migration
@@ -207,7 +207,7 @@ UPDATE auth.users
Output: `N password_hash columns nulled. M users still have password_hash
because they don't meet the OPAQUE-migrated preconditions — inspect via
`oxicloud-cli opaque wipe-legacy --dry-run` and address separately.`
`oxicloud opaque wipe-legacy --dry-run` and address separately.`
The `WHERE` clause is intentionally strict: OIDC users, externals, and
under-migrated users are ALL left alone. The strict version is safer than
@@ -232,7 +232,7 @@ can drop the legacy password code:
6. `has_password` field on `UserDto` / `AdminUserSummaryDto`: delete (always
false, meaningless signal)
7. `admin`-badge `password` chip: delete (same reason)
8. `oxicloud-cli opaque reset --user X` for legacy-recovery: still useful
8. `oxicloud opaque reset --user X` for legacy-recovery: still useful
as an emergency lever (envelope somehow corrupted, need to force
re-registration via recovery-magic-link), but its "silent-migration
handles the recovery" semantics become "recovery-magic-link handles the
@@ -269,7 +269,7 @@ running smoothly for the indicated period."
| G1 | Land task #31: change_password OPAQUE-lockout fix + hybrid-user password gate | Days |
| G2 | Land recovery-magic-link admin reset flow | Weeks |
| G3 | Land OPAQUE-verify-current + change_password redesign that COMPOSES the two (Argon2-verify AND OPAQUE-verify both work; use whichever the user has) | Weeks |
| G4 | Ship `oxicloud-cli opaque wipe-legacy` (dry-run only initially, no destructive flag) | Days |
| G4 | Ship `oxicloud opaque wipe-legacy` (dry-run only initially, no destructive flag) | Days |
| G5 | Add admin-dashboard metric: "N users still on legacy (`password_hash IS NOT NULL AND !opaque_migrated`)" | Days |
| G6 | Operator switches deployment to `opaque_only` mode | ✅ already possible |
| G7 | Wait 90+ days at `opaque_only`, watch the metric drop to 0 | Months |
docs/plan/sessions.md
+401
View File
@@ -0,0 +1,401 @@
# Session Liveness Tracking — `last_seen_at` + Prometheus
Track per-session and per-user "currently active" signals cheaply, and
expose them as Prometheus gauges so an operator (demo instance,
production) can plot concurrency over time. Also unlocks future
features that need "when was this session last used" (idle-timeout
enforcement, per-user session-limit quotas, admin dashboard freshness).
Companion doc for the design decisions covered here — narrative on the
overall session model lives in
[docs/architecture/auth-model.md](../architecture/auth-model.md).
## Purpose — what we want to see
Two distinct signals, deliberately separate:
1. **Online sessions** (`oxicloud_sessions_online`) — count of
non-revoked `auth.sessions` rows that had a request within the last
N minutes. One user with three devices (browser + phone + Nextcloud
desktop) contributes **three** to this count. Useful for
provisioning ("how many concurrent connections do I need to
support?") and load-shape planning.
2. **Online users** (`oxicloud_sessions_online_users`) — count of
DISTINCT `user_id` values behind those online sessions. Same
three-device user contributes **one** to this count. Useful for
billing shape ("how many humans are actually using the system?")
and for the demo landing page's "N users online right now" widget.
The gap between the two IS the multi-device factor. A healthy system
where users routinely have web + desktop client should show `sessions
≈ 2 × users`. A sudden `sessions >> users × 3` is a signal — an app
that opens fresh sessions instead of reusing them, or a
credential-stuffing pattern.
### Terminology — "online" vs "active"
The admin sessions panel already has a lifecycle filter
`Active | Expired | Revoked` — where **active** means
`!revoked && !expired` (row is still usable). That's orthogonal to
"had a request lately", so both concepts fighting for the same word
was going to confuse admins reading the panel.
**Decision (Ed, 2026-08-18)** — "online" is the *presence* signal
throughout the stack:
- **UI**: green-dot badge next to each row when
`SessionSummaryDto::is_online == true`; grey dot + "last seen X ago"
otherwise. Lifecycle filter stays `Active | Expired | Revoked`
unchanged.
- **DTO**: `is_online: bool` on `SessionSummaryDto`, computed
server-side (avoids the SPA doing clock math and drifting from the
server view). Guaranteed `false` on revoked / expired rows so an
admin never sees "Online" on a row they just revoked.
- **Metrics**: `oxicloud_sessions_online` / `_online_users`.
- **Threshold**: single `pub const ONLINE_WINDOW` in
`application/dtos/session_dto.rs` — DTO derivation AND gauge query
read from the same constant so the per-row badge count and the
gauge aggregate stay consistent by construction.
## Why the existing signals don't answer this — DECIDED
`auth.sessions.created_at` is the closest existing proxy. But it
moves on **session rotation**, not per-request:
- Sessions rotate on every silent refresh (`apiFetch`'s 401 → refresh
path). Rotation cadence = `access_token_expiry_secs` (default
3600, i.e. 1 h).
- So `WHERE created_at > NOW() - INTERVAL '1 hour'` catches everyone
who refreshed in the last cycle — but a user who's actively clicking
around for 45 min hasn't rotated yet, so their `created_at` is 45
min old. Threshold `< 30 min` false-negatives them.
- Resolution is capped at the access-token TTL. At the recommended
prod value of 15 min, `created_at` gives 15-min granularity. At the
test value of 60 s it's near-real-time — but no operator wants
to force 60 s token TTL just for observability.
So `created_at` is an OK first-pass proxy but bad enough that we
should add a dedicated column that moves per-request.
## Schema — `last_seen_at`
Migration `<TS>_sessions_last_seen_at.sql`:
```sql
ALTER TABLE auth.sessions
ADD COLUMN last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
-- Partial index — the "active in the last N min" query is the only
-- reason to scan this column, and it always filters on revoked = FALSE.
-- Full index would double the write cost for zero read benefit.
CREATE INDEX idx_sessions_last_seen_at ON auth.sessions(last_seen_at)
WHERE revoked = FALSE;
```
Default `NOW()` — existing rows on migration land at "just seen"
which is slightly optimistic but the alternative (NULL / epoch) makes
every historic session count as long-idle for the first N min after
deploy. `NOW()` matches "assume everyone's active" and the metric
converges to reality within one N-min bucket.
**Not in scope for this schema**: adding `last_seen_at` to
`auth.app_passwords`. Nextcloud-desktop clients authenticate through
that table; separate concern, separate PR if we want desktop-client
liveness.
## Coalescing writes — DECIDED: in-process DashMap + periodic flush
Naive shape (one `UPDATE` per authenticated request) would multiply
the DB write rate by every non-mutating request the SPA fires
(listing pages, thumbnails, delta-upload chunk PUTs). Untenable.
The pattern that scales:
- Middleware stamps `(session_id, Instant::now())` into a shared
`DashMap<Uuid, DateTime<Utc>>`. **O(1)** per request, no I/O.
- Background task drains the map every N seconds (default 30) and
emits ONE batched `UPDATE ... FROM UNNEST(...)` covering every
distinct session seen in the window.
- The map data structure IS the dedup: same `session_id` → same key
→ last write wins. A session seen 100× in the window contributes
ONE row to the batched update, with the latest timestamp.
The `UPDATE` uses `greatest()` to be idempotent under any
retry/race/clock-skew:
```sql
UPDATE auth.sessions AS s
SET last_seen_at = greatest(s.last_seen_at, t.seen_at)
FROM UNNEST($1::uuid[], $2::timestamptz[]) AS t(id, seen_at)
WHERE s.id = t.id;
```
**Restart durability**: up to one flush interval of activity lost on
hard crash. On graceful shutdown (SIGTERM) run one final flush
synchronously before exit — zero loss on planned rolling restarts.
**Failure durability**: if the batched UPDATE fails (PG blip),
DON'T clear the DashMap; next tick retries with the accumulated set
overlaid on any new activity.
### Why not `Arc<Mutex<HashMap>>`
Every authenticated request writes. Under any concurrency (delta
uploads, thumbnail bursts, folder listing paginations firing in
parallel) a single mutex becomes the bottleneck. `DashMap`'s
per-shard locking (16-32 shards by default) parallelises writes
across distinct keys — different session_ids don't contend.
### Why not `Arc<RwLock<HashMap>>`
The workload is write-heavy (every auth'd request writes, reads only
fire in the flusher). RwLock would still serialize the writes for no
benefit.
### Why not PG `NOTIFY` / `LISTEN`
Considered. Trade-offs:
- **Pro**: cross-instance coalescing — multiple OxiCloud processes
behind a load balancer push to one channel, single flusher owns
writes.
- **Con**: every request pays a PG round-trip (`SELECT
pg_notify(...)`) — ~1 ms per request vs ~50 ns for DashMap insert.
On hot endpoints (delta chunk PUTs, thumbnails) this is
measurable.
- **Con**: adds a persistent LISTEN connection to the pool.
- **Con**: OxiCloud is single-instance today. The multi-instance win
doesn't apply.
**Deferred**: if OxiCloud ever grows a multi-instance deployment
story (Kubernetes, active-active behind a load balancer), migrate
the flusher to `NOTIFY`-based ingest — schema stays identical, only
the tracker implementation swaps. Document the migration path in
[Future — multi-instance](#future--multi-instance) below.
## Metric surface — Prometheus
Exposed via the existing `/metrics` endpoint (see
`src/interfaces/metrics.rs`; gated on `OXICLOUD_METRICS_LISTEN`).
### Gauges (polled every 30 s from a background task)
```
# HELP oxicloud_sessions_online Non-revoked sessions seen in the last N min.
# TYPE oxicloud_sessions_online gauge
oxicloud_sessions_online <value>
# HELP oxicloud_sessions_online_users Distinct users behind online sessions.
# TYPE oxicloud_sessions_online_users gauge
oxicloud_sessions_online_users <value>
# HELP oxicloud_sessions_total_non_revoked Total non-revoked sessions
# regardless of activity — the long tail (mobile clients still holding
# refresh tokens they haven't used in weeks).
# TYPE oxicloud_sessions_total_non_revoked gauge
oxicloud_sessions_total_non_revoked <value>
```
Queries powering each:
```sql
-- oxicloud_sessions_online
SELECT COUNT(*) FROM auth.sessions
WHERE revoked = FALSE
AND last_seen_at > NOW() - $1::interval; -- $1 = ONLINE_WINDOW
-- oxicloud_sessions_online_users
SELECT COUNT(DISTINCT user_id) FROM auth.sessions
WHERE revoked = FALSE
AND last_seen_at > NOW() - $1::interval;
-- oxicloud_sessions_total_non_revoked
SELECT COUNT(*) FROM auth.sessions WHERE revoked = FALSE;
```
All three run on the maintenance pool (background polling shouldn't
compete with request-serving connections). Three lightweight
`COUNT(*)` reads every 30 s; measured cost negligible even on
tens-of-thousands-of-rows tables thanks to the partial index.
### Counters (already in-place shape)
`oxicloud_sessions_created_total` and
`oxicloud_sessions_revoked_total{reason}` — extend the existing
counter surface in the auth service (`session.created` audit line
sites) to also `metrics::counter!(...)`. Not strictly needed for the
"how many active" question but useful sanity signal on the
dashboard: rate of creation vs rate of revocation should be
approximately balanced at steady state.
## Config surface
**No new env var.** Ed's call (2026-08-18): tuning the online window
is a deployment-shape question we haven't had to answer in practice,
and adding an env knob invites premature customization. The three
knobs stay hardcoded:
- **Online window** — 5 min. Feels responsive for a demo
landing page without over-fluctuating with tab-open-then-close
blips. Lives at `pub const ONLINE_WINDOW` in
`src/application/dtos/session_dto.rs`; the gauges module in
`src/infrastructure/services/session_liveness_gauges.rs` reads
from that constant so the DTO badge and the gauge aggregate
can't drift.
- **Flush interval** — 30 s. Balances DB write load against gauge
freshness (typical Prometheus scrape at 15 s sees the value
refreshed after at most two scrapes). Lives at `FLUSH_INTERVAL`
in `src/infrastructure/services/last_seen_tracker.rs`.
- **DashMap shard count** — crate default (16). Only worth
surfacing when profiling shows shard contention.
## Middleware wiring
Auth extractor (`CurrentUserId`) already loads the session by
refresh-token cookie / bearer-token subject. Extend the post-load
path:
```rust
// After successful session lookup + auth checks:
state.last_seen_tracker.stamp(session.id);
```
`LastSeenTracker` shape:
```rust
pub struct LastSeenTracker {
seen: Arc<DashMap<Uuid, DateTime<Utc>>>,
pool: Arc<PgPool>,
}
impl LastSeenTracker {
pub fn new(pool: Arc<PgPool>) -> Arc<Self> {
let seen = Arc::new(DashMap::new());
let this = Arc::new(Self { seen: seen.clone(), pool: pool.clone() });
tokio::spawn(this.clone().flush_loop());
this
}
/// Called from the auth middleware on every authenticated request.
/// O(1); no I/O; no round-trip.
pub fn stamp(&self, session_id: Uuid) {
self.seen.insert(session_id, Utc::now());
}
/// Called from the graceful-shutdown handler.
pub async fn flush_now(&self) -> Result<(), sqlx::Error> {
/* drain + one batched UPDATE, same as the loop body */
}
async fn flush_loop(self: Arc<Self>) {
let mut ticker = tokio::time::interval(Duration::from_secs(30));
loop {
ticker.tick().await;
let _ = self.flush_now().await; // errors logged, not propagated
}
}
}
```
Wired in `common/di.rs`; injected into `AppState` and referenced by
the auth extractor.
## Graceful shutdown
Hook into the existing SIGTERM handler in `main.rs` to call
`tracker.flush_now().await` before the runtime exits. Ensures rolling
restarts / container replacements don't lose the last 30 s of
liveness data.
## Admin dashboard integration (deferred, sibling PR)
Once `last_seen_at` exists, the admin sessions panel can render a
"last seen X min ago" column instead of only "created X ago". Small
follow-up — not part of this plan's scope, but the schema addition
unlocks it.
## Testing
Two hermetic units (no DB):
1. `DashMap` dedup test — insert same key 10× with different
timestamps, drain, assert one entry with the latest timestamp.
2. `flush_now()` UPDATE-shape test — mock a `PgExecutor`, assert the
batched UNNEST binds match the drained set. Uses `sqlx-mock` or
equivalent.
One integration (real PG, gated on `integration_tests` cfg):
3. End-to-end — insert a session row, `stamp()` it, call `flush_now`,
assert `last_seen_at > created_at`. Covers the whole write path
including the `greatest()` guard.
## Phasing
1. **Migration** — add column + partial index. Ships alone; zero
application-layer impact. Reversible via `DROP COLUMN`. ✅ **2026-08-18**
(`migrations/20261014000000_sessions_last_seen_at.sql`).
2. **`LastSeenTracker` service** — DashMap + flusher task. Wire into
`AppState`. Middleware calls `stamp()`. Now `last_seen_at` moves
in real time. ✅ **2026-08-18** (`src/infrastructure/services/last_seen_tracker.rs`).
3. **JWT `sid` claim** — token minters carry the fresh session's
id; auth middleware reads it and stamps with no DB round trip.
New tokens carry it, old tokens still validate (Option → no-op).
✅ **2026-08-18** (extends `TokenClaims::sid` on
`application/ports/auth_ports.rs`).
4. **Prometheus gauges** — background poller updates the three
gauges every 30 s using the queries above. Gated on
`OXICLOUD_METRICS_LISTEN` (no recorder → no periodic PG hits).
✅ **2026-08-18** (`src/infrastructure/services/session_liveness_gauges.rs`).
5. **Graceful-shutdown flush** — hook into SIGTERM handler.
✅ **2026-08-18** (added `shutdown_signal` +
`with_graceful_shutdown` in `main.rs`).
6. **Session DTO exposes `last_seen_at` + `is_online`** —
`GET /api/admin/sessions` returns both so the admin table + external
monitors can read them. `is_online` is the server-side derivation
against `ONLINE_WINDOW` (see terminology decision above). ✅
**2026-08-18** (`application/dtos/session_dto.rs`).
7. **Admin dashboard "Online" column** (deferred to a sibling PR) —
render `is_online` as a green/grey dot next to each row plus
"last seen X ago" from `last_seen_at`.
## Future — multi-instance
If OxiCloud grows a multi-instance deployment story (K8s replicaset,
active-active behind a load balancer), the in-process DashMap becomes
insufficient — each process holds its own map, N flushers race
UPDATEs, coalescing across instances doesn't happen.
Migration path when that becomes real:
1. Keep the schema (`last_seen_at` column + partial index).
2. Replace the `LastSeenTracker::stamp()` in-process insert with a
`SELECT pg_notify('oxicloud_session_seen', $session_id)` call.
3. Move the flusher into a **single elected worker** (leader election
via advisory lock in PG). That worker `LISTEN`s the channel,
accumulates into an in-process HashMap, flushes on the same 30 s
ticker.
4. Every OxiCloud instance publishes; one instance consumes.
Trade-off: NOTIFY costs a PG round-trip per request (~1 ms) vs the
current ~50 ns DashMap insert. Only pay that when multi-instance
coalescing actually matters. Schema and gauge queries stay identical;
only the tracker implementation swaps.
## Open questions
1. ~~Definition of "active"~~ — DECIDED 2026-08-18: hardcoded to
5 min, no env var. See [Config surface](#config-surface).
2. **Should the gauge query drop long-inactive sessions?** — a
Nextcloud desktop client checking in every 6 h qualifies as
"active" if the window is 6 h. Probably want two separate metrics
(web-active < 5 min AND dav-active < 6 h) but scope creep for the
initial ship.
3. **Multi-tab dedup on the UI side** — three browser tabs of the
same user share ONE session (same cookies, same refresh token,
same row). Naturally deduped at the DB layer — no action needed.
4. **App-password rows** — Nextcloud desktop / mobile clients use
`auth.app_passwords` on top of the session model. Whether they
should get their own `last_seen_at` (and a companion metric) is
deferred; separate concern, separate PR.
docs/plan/storage-key-rotation.md
+11 -1
View File
@@ -179,7 +179,17 @@ Guardrail:
head pair's fingerprint differs from the second entry's — signals "you
added new keys but haven't rotated legacy blobs yet".
* A **legacy-blob counter** is surfaced in the admin panel per storage entry.
The counter is maintained by `blobs_consistency`: during its normal walk it
> **Retarget (post-split).** This plan names `blobs_consistency` as the
> host for the magic-byte branch throughout. That is no longer the right
> tenant: `blobs_consistency` became database-only and opens no backend,
> while `backend_consistency` owns every physical check and already holds
> the enumeration. Read `backend_consistency` wherever the sections below
> say `blobs_consistency`. Nothing else about the design changes — the
> magic-byte check still rides along on an existing walk, and still lands
> in the run's `stats` bag.
The counter is maintained by that scan: during its normal walk it
branches on the magic-byte check and records the legacy count as a run
statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The
admin panel reads the most recent count and displays it. Refresh cadence is
docs/plan/storage-multi-entry.md
+9 -7
View File
@@ -324,7 +324,7 @@ at), boot fails fast with a clear error. Operator has two ways out:
restart.
2. **CLI repair flag on the `oxicloud` binary itself**:
```
oxicloud --select-storage <name>
oxicloud storage select <name>
```
Behaviour: parse `.env`, verify `<name>` exists in `_ENTRIES` (fail-fast
with the available names listed if not), connect to DB, UPDATE
@@ -334,7 +334,7 @@ at), boot fails fast with a clear error. Operator has two ways out:
The bare-flag on the shipped binary is chosen over a separate `just`
recipe or auxiliary bin because:
- **Docker-friendly**: `docker exec oxicloud oxicloud --select-storage foo`
- **Docker-friendly**: `docker exec oxicloud oxicloud storage select foo`
— no need to install extra tooling in the container.
- **Systemd-friendly**: can be run as a `ExecStartPre=` one-shot before the
main service unit.
@@ -377,7 +377,7 @@ foundational; the rest layer on top independently within reason.
| 5 | Cutover state machine: on migration `Completed`, write `active_backend_name = target_name`, keep read-only on. Boot on new backend after operator restart. | 4 | ~half day |
| 6 | Admin storage tab rewrite: list entries, show active, migrate dropdown, read-only banner. Delete Save form + S3 field editors + .env cutover hint. | 1, 3, 4 | 1 day |
| 7 | `?storage=<name>` on `blobs_consistency` + `backend_consistency`. `JobRunArgs.storage` plumbing, `TriggerJobQuery.storage`, entry-resolver at run start, params records probed name. Retire `verify_migration` + its DTO + its route + its handler. | 1, 3 | 1 day |
| 8 | `oxicloud --select-storage <name>` bare-flag repair command on the main binary. Parses `.env`, verifies entry exists, UPDATEs DB, exits. Boot-time missing-entry error message points at it. See §Fallback. | 2 | ~quarter day |
| 8 | `oxicloud storage select <name>` bare-flag repair command on the main binary. Parses `.env`, verifies entry exists, UPDATEs DB, exits. Boot-time missing-entry error message points at it. See §Fallback. | 2 | ~quarter day |
**Total: ~5-6 days end to end.** Slices 6 and 7 can proceed in parallel with
each other once 1-5 land. Slice 8 is an ops nicety, could ship whenever.
@@ -407,15 +407,17 @@ Per slice, plus these end-to-end scenarios in Hurl:
6. **In-place encryption rotation refused**: two entries, same S3 bucket,
different encryption keys. Trigger migration → refuses with the specific
error message pointing at the encryption case and the two-step workaround.
7. **`?storage=<name>` on blobs_consistency**: run against `s3_prod` before
7. **`?storage=<name>` on backend_consistency**: run against `s3_prod` before
cutover. Full walk, `probed_storage` in run row. Then cutover, then rerun
against `local_main` — verifies old backend still has everything.
8. **Unknown storage name**: `POST /admin/jobs/blobs_consistency/trigger?storage=nope`
(This scenario named `blobs_consistency` until that tenant became
database-only; entry scoping belongs to whichever job opens a backend.)
8. **Unknown storage name**: `POST /admin/jobs/backend_consistency/trigger?storage=nope`
→ 400 with known-names list. No run row created.
9. **Missing entry at boot**: `active_backend_name = "gone"` but `_ENTRIES`
doesn't include it → boot aborts with the specific message pointing at
`oxicloud --select-storage <name>` (with the available names filled in).
Re-run the binary with `--select-storage local_main` → verifies + updates
`oxicloud storage select <name>` (with the available names filled in).
Re-run the binary with `storage select local_main` → verifies + updates
DB + exits 0. Restart the server → boots cleanly on `local_main`.
10. **Encryption key invalid**: `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY=badbase64`
→ boot aborts with entry name + reason (not valid base64 / wrong length).
docs/plan/userdto-refactor.md
+427
View File
@@ -0,0 +1,427 @@
# UserDto Refactor — Three-Layer Split (Public / Full / Self)
> **Status — SHIPPED 2026-08-21.** All eight phases landed and all gates
> pass: `cargo clippy --all-targets --all-features -D warnings` clean,
> `cargo fmt --check` clean, `cargo test three_layer_quarantine` (2/2
> structural-quarantine tests pass), `npm run check` (593 files, 0
> errors, 0 warnings), `npm run test:unit` (414 pass / 1 skipped / 0
> failed), OpenAPI regenerated at `resources/gen/openapi.json`. See the
> [Phasing](#phasing) section below for the per-step outcome. The doc
> is retained as the reference for anyone extending the three-layer
> shape (new field → decide by audience per the rule in the opening
> section).
Establish three DTO shapes for representing a user on the wire, each
with a single unambiguous audience, composed hierarchically so the
overlap between audiences is defined ONCE:
- **`PublicUserDto`** — public identity. What any authenticated caller
may see about *another* user. Returned by `/api/users/{id}`, share
responses, group members, magic-link invitees, recipient enrichment.
- **`FullUserDto`** = `PublicUserDto` + all fields that BOTH an admin
(viewing another user) AND the subject themselves (viewing
themselves) may see. Returned as `Vec<FullUserDto>` by
`/api/admin/users`. Closest DTO to the underlying `auth.users` row.
- **`SelfUserDto`** = `FullUserDto` + self-only preferences,
session-scoped flags, and caller-scoped permissions. Returned by
`/api/auth/me` and by the login / refresh / OIDC / magic-link auth
response.
Composition (`FullUserDto.user: PublicUserDto`,
`SelfUserDto.full: FullUserDto`) means the public-identity contract
has ONE definition; the overlap between admin's view and self's view
is another single definition. Adding a new field naturally finds its
level:
- Useful to any authenticated caller? → `PublicUserDto`.
- Useful only to admin (about another user) and the subject
themselves? → `FullUserDto`.
- Meaningful only to the caller viewing themselves? → `SelfUserDto`.
Companion of `docs/plan/sessions.md` (which introduced `is_online`
and motivated widening the DTO for presence). Same principle: pick
the audience first, structure the DTO around it, don't let the same
field mean different things on different endpoints.
## Why now — the problems this fixes
Today's single `UserDto` conflates three audiences. Symptoms:
1. **The "quiet lie"**. `UserDto::has_password` is populated only by
`/api/auth/me`; every other emitter (`From<User>`) leaves it
`false`. A share-recipient DTO on the wire says
`has_password: false` unconditionally, which an attacker
scraping share responses could misread as "this user is
passwordless" when the truth is "we didn't fill this field in
for you". Same pattern for `force_password_change` and
`is_dpop_bound`. See `src/application/dtos/user_dto.rs:134-138`
for the explicit disclaimer — the convention exists precisely
because the field placement is wrong.
2. **Private signals leak by default**. `last_login_at`,
`notify_on_share`, `ui_preferences`, `preferred_locale`,
`federation_kind`, `email_verified_at`, and
`storage_used_bytes` all ride on `UserDto` and are returned to
any authenticated caller who can see a given user. Group
members can see when their peers last logged in, which IdP they
federate with, and how full their disks are. None of this is
information a share picker or a member listing needs.
3. **`AdminUserSummaryDto` duplicates a chunk of `UserDto` verbatim**
(id / username / email / role / quotas / last_login_at / active /
federation_* / is_external), then adds three admin-only fields
(has_password / opaque_registered / opaque_migrated). The two
shapes drift naturally as new fields are added — no compile-time
guarantee they stay in sync.
4. **N+1 in the admin panel**. Because `AdminUserSummaryDto` doesn't
include `image`, the admin users table fires `/api/users/{id}`
per row so `UserVignette` can render the avatar. Composition
(`FullUserDto.user.image`) lets the admin listing seed the SPA's
per-user cache from the list rows directly.
## Target shapes
### `PublicUserDto` — public identity (9 fields)
Applied rule: "would a share picker / group member listing /
recipient enrichment need this? if no, it doesn't belong here."
```rust
pub struct PublicUserDto {
pub id: String,
pub username: Option<String>,
pub email: String,
pub role: String, // sharee UI renders admin badge
pub image: Option<String>, // avatar
pub is_external: bool, // external badge
pub given_name: Option<String>, // social identity
pub family_name: Option<String>, // social identity
pub is_online: bool, // presence — social signal
}
```
Every existing UserDto emitter site (`From<User>`, share responses,
group members, magic-link invitees, sharee-vignette lookup)
returns this slim shape. All private signals below vanish from
those wire paths.
### `FullUserDto` — admin's view of anyone + self's view of self (13 extras)
The fields the SUBJECT themselves may know about themselves that
an ADMIN may also know about the subject. Composed on top of
`PublicUserDto`. This is the DTO closest to the underlying
`auth.users` row.
```rust
pub struct FullUserDto {
/// Public identity — same set any authenticated caller can see.
pub user: PublicUserDto,
/// IdP linkage. Which SSO provider a peer uses is a soft
/// org-affiliation leak; not needed by share pickers.
pub federation_kind: Option<String>,
pub federation_issuer: Option<String>,
/// Subject's own locale preference. Only THEY or an admin
/// managing them needs this — other callers use their own.
pub preferred_locale: Option<String>,
/// Email-verification stamp. Trust signal — meaningful to admin
/// (auditing verification status) and to self (own record), but
/// not to a share picker rendering a vignette.
pub email_verified_at: Option<DateTime<Utc>>,
/// Row bookkeeping — not rendered on any non-admin surface today.
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
/// Activity signal — private.
pub last_login_at: Option<DateTime<Utc>>,
/// Account-active flag — private (a deactivated user couldn't
/// reach `/me` anyway, but admin needs to see it).
pub active: bool,
/// Storage quotas — personal financials. Admin manages others';
/// self sees own.
pub storage_quota_bytes: i64,
pub storage_used_bytes: i64,
/// Auth capability set — has_password / OPAQUE flags. Kept off
/// public identity because per-user auth adoption leaks through
/// directory endpoints.
pub has_password: bool,
pub opaque_registered: bool,
pub opaque_migrated: bool,
}
```
### `SelfUserDto` — /api/auth/me (5 extras)
Everything the caller may see about themselves that no other
caller (not even an admin) needs to see: pure self-scoped state.
```rust
pub struct SelfUserDto {
/// Full profile. Every field an admin would see about you is
/// here — same shape as one row of /api/admin/users.
pub full: FullUserDto,
/// Opaque UI preferences bag — my own UI state. Cross-device
/// via PATCH /api/auth/me/profile.
pub ui_preferences: serde_json::Value,
/// Whether I want share-notification emails.
pub notify_on_share: bool,
/// Session-scoped: my current session carries a DPoP thumbprint.
/// SPA skips a redundant /api/auth/dpop/bind on load when true.
pub is_dpop_bound: bool,
/// Admin-set temp-password gate — SPA nav guard blocks everything
/// but /change-password until this flips back.
pub force_password_change: bool,
/// Caller-scoped permission: can I edit my own avatar? False for
/// OIDC users whose avatar comes from the IdP. Only meaningful
/// when caller == subject; nonsense on any other DTO.
pub can_edit_image: bool,
}
```
## Endpoint mapping
| Endpoint | Old shape | New shape |
|---|---|---|
| `/api/auth/me` | `UserDto` (fat) | `SelfUserDto` |
| `/api/auth/login` / `/refresh` / OIDC callback / magic-link redemption | `AuthResponseDto { user: UserDto }` | `AuthResponseDto { user: SelfUserDto }` |
| `/api/admin/users` | `Vec<AdminUserSummaryDto>` | `Vec<FullUserDto>` |
| `/api/users/{id}` | `UserDto` (fat) | `PublicUserDto` (9 fields) |
| Share responses / group members / magic-link invitees / recipient enrichment | `UserDto` (fat) | `PublicUserDto` |
**Login response ships `SelfUserDto`, not `PublicUserDto`.** The SPA
needs `has_password`, `ui_preferences`, `is_dpop_bound`, and
`force_password_change` immediately post-login to avoid a UI race
with the first `/me` fetch. Same rationale for refresh and
OIDC/magic-link callback: the SPA's post-auth state must be
complete in one round trip.
## Backend callsite inventory
**DTO layer** (`src/application/dtos/user_dto.rs`):
- Replace `UserDto` with `PublicUserDto` (renamed AND slimmed —
same rename forces every consumer to consciously pick the new
shape rather than silently losing fields).
- Add `FullUserDto` and `SelfUserDto`.
- Delete `AdminUserSummaryDto` (superseded by `FullUserDto`).
- `impl From<User> for PublicUserDto` — the entry point. Maps 1:1
to `User`'s public identity accessors.
- `FullUserDto::build(user: User, flags: UserDerivedFlags)` —
wraps a `PublicUserDto` plus the DB-computed booleans not on
`User` (`has_password`, `opaque_registered`, `opaque_migrated`,
`is_online`). Every other FullUserDto field comes from `User`
directly. Not a `From` impl because the second argument is
needed and Rust's `From` is single-arg.
- `SelfUserDto::build(full: FullUserDto, session_ctx: SessionContext)`
— helper taking a FullUserDto plus caller context (session's
DPoP-bound flag, admin-set force-password-change flag). Same
reason as FullUserDto's builder — not a `From` impl.
**Repository layer**
(`src/infrastructure/repositories/pg/user_pg_repository.rs`):
- **Delete `UserListEntry`** — the narrow projection was a perf
optimization; Path B decision supersedes it.
- `list_users` returns `Vec<(User, UserDerivedFlags)>` where
`UserDerivedFlags` is a small named struct in
`domain/repositories/user_repository.rs`:
```rust
/// DB-computed booleans about a user that aren't fields on the
/// `User` entity itself — either derived from column presence
/// (`password_hash IS NOT NULL`) or from a cross-table lookup
/// (`auth.sessions.last_seen_at` for `is_online`). Companion
/// to `User` on the list projection: the repo computes both,
/// the application layer packs them into `FullUserDto`.
pub struct UserDerivedFlags {
pub has_password: bool,
pub opaque_registered: bool,
pub opaque_migrated: bool,
pub is_online: bool,
}
```
Not "admin-only" — every field ends up on `FullUserDto`, which
both admin AND self read. The name reflects "derived from the
DB row, not intrinsic to the User entity".
- SELECT widens to include `image` (previously narrowed away
per ROUND12 §Q1) + the `EXISTS(...)` scalar for `is_online`,
bound with `ONLINE_WINDOW.as_secs_f64()` via
`make_interval(secs => $N)` — same pattern as
`session_liveness_gauges.rs:104-114`, single source of truth,
no SQL literal.
**Handler / service layer**:
- `/api/auth/me` handler — builds `SelfUserDto` from
`(User, UserDerivedFlags, session_context)`. The
`UserDerivedFlags` for /me comes from a reuse of the
list-repo path scoped to `WHERE id = $me` or a new small
dedicated query (implementer's call — either works).
- `AuthResponseDto` shape follows — `user: SelfUserDto` field.
- Every login/refresh/OIDC/magic-link path that mints an
`AuthResponseDto` computes the same SelfUserDto.
- Admin service `list_users_admin` — returns `Vec<FullUserDto>`.
- `/api/users/{id}` handler — returns `PublicUserDto`. Every
other public consumer stays on `PublicUserDto`.
## Frontend callsite inventory
**Type changes** (`frontend/src/lib/api/types.ts`):
- Rename `User` interface → `PublicUser` and slim to match new
DTO (9 fields).
- Add `FullUser` interface — `{ user: PublicUser, federation_kind, ... }`.
- Add `SelfUser` interface — `{ full: FullUser, ui_preferences, ... }`.
- Delete `AdminUser` (replaced by `FullUser`).
**Store changes**:
- `lib/stores/session.svelte.ts` — reads /me, must handle SelfUser
shape. Recommendation: keep a derived `session.me: SelfUser` for
the full record and shorthand accessors:
`session.user: PublicUser` = `session.me.full.user`,
`session.full: FullUser` = `session.me.full`.
Existing `session.user.username` calls keep working via the
shorthand; new self-only reads go through `session.me.foo` or
`session.full.foo`.
**Component changes**:
- Profile / change-password / DPoP-bind pages — reads
`session.me.has_password`, `session.me.is_dpop_bound`,
`session.me.can_edit_image`, `session.full.preferred_locale`,
etc.
- `routes/admin/[[tab]]/+page.svelte` users table — every
`u.username` → `u.user.username`, every `u.last_login_at` /
`u.active` / `u.has_password` stays top-level (FullUserDto
fields). Also **seed `resolveUser` cache with `u.user`** in the
load path — kills the N+1 that motivated widening the query.
- Admin sessions table's `UserVignette` — no change, `user_id`
passed through unchanged; the users-table cache seed above
satisfies the vignette lookup on cross-table navigation.
- `lib/composables/useOwnerCache.ts` /
`lib/api/endpoints/users.ts` — `resolveUser` returns
`PublicUser`. No signature change; the return shape only gets
smaller. Add a `seedUser(u: PublicUser)` export so the admin
table can prime the cache.
## Phasing
Each step compiles standalone; each is a reasonable review chunk.
1. **Introduce the new DTOs** — add `PublicUserDto`, `FullUserDto`,
and `SelfUserDto` alongside the existing `UserDto`. Don't
change `UserDto` yet. Compiles; no behaviour change.
2. **Widen repo projection** — add `is_online` (via EXISTS
subquery) and `image` back to `list_users` SELECT. Introduce
`AdminExtras` struct. `UserListEntry` still exists but is now
redundant (fields also available on `User`).
3. **Migrate the emitter sites** — `/api/auth/me`,
login/refresh/OIDC/magic-link, admin service. Each now builds
the new nested shape. Old `UserDto` still ships every field.
4. **Rename `UserDto` → `PublicUserDto` and slim** — remove the
moved fields. The Rust compiler flags every remaining consumer
that reads a removed field; those either move to `.full.foo` /
`.user.foo` (embedded) or promote themselves to a Self/Full
DTO.
5. **Delete `UserListEntry` + `AdminUserSummaryDto`** — dead after
the cutover.
6. **Frontend** — rename types (`User` → `PublicUser`), add
`FullUser` / `SelfUser`, update session store, all consumers.
Seed `resolveUser` cache from admin table.
7. **Regenerate OpenAPI** — `cargo run --bin generate-openapi`
picks up the new schemas; the shrunken `PublicUserDto` schema
documents the new contract.
8. **Delete obsolete doc comments** — `has_password` /
`is_dpop_bound` / `force_password_change` comments on the old
UserDto explaining "populated only by `/me`" become obsolete
(the field structurally can't exist on non-self emitters).
## Wire-shape breaking changes
All in-repo consumers (backend + SPA) migrate in the same commit.
External consumers: none today — `/api/admin/users` is
admin-panel-only, `/me` is SPA-only, share/group endpoints are
SPA-only. Ship as one clean break; skip a `?shape=v2` deprecation
window.
Every removed field from a public UserDto path (share responses,
group members, magic-link invitees, `/api/users/{id}`) is a
deliberate leak reduction, not a regression. Any FE consumer that
was reading e.g. `sharee.has_password` was reading a "quiet lie"
anyway (always `false`).
## Testing
**Backend**:
- Round-trip tests for each new DTO type (already have for
UserDto; extend to PublicUserDto + FullUserDto + SelfUserDto).
- **Structural quarantine tests** —
`self_user_dto_does_not_leak_ui_preferences_via_public_paths`:
serialize a `SelfUserDto`, assert `ui_preferences` appears
ONLY at top level, not inside `.full.user` or `.full`. Same for
`FullUserDto` — `has_password` at top level of `FullUserDto`,
not inside `.user`.
- Update every service test that constructs `UserDto` fixtures.
**Frontend**:
- The TS type system catches every consumer that reads a removed
field. `npm run check` surfaces the whole blast radius on the
first pass — no new test infrastructure needed.
- Add one Vitest integration on the admin users table asserting
the presence dot renders AND `/api/users/{id}` is NOT called
per row (checks `apiFetch` mock call count).
**Wire-shape guard**:
- Hurl test hitting `/api/users/{id}` as a non-admin caller,
asserting the response does NOT contain moved fields
(`has_password`, `last_login_at`, `notify_on_share`,
`ui_preferences`, `storage_used_bytes`, `federation_kind`,
`preferred_locale`, `email_verified_at`, `created_at`, etc.).
Anti-regression guard for the whole point of this refactor.
## Non-goals
- **Reworking the `User` domain entity** — this refactor is
DTO-shape only. The entity keeps all its fields.
- **Visibility-rule changes on `/api/users/{id}`** — who can see
whom stays as-is; only the field set narrows.
- **Splitting other DTOs** — `SessionSummaryDto`, `FileDto`, etc.
Same principle would apply, but each is a separate design call.
- **Moving avatars out of the row** — planned separately. This
refactor keeps `image` on `PublicUserDto` so the admin-panel
N+1 fix survives.
- **Flattening the nested shape via `#[serde(flatten)]`** — the
three-level wire shape (`me.full.user.username`) is slightly
deeper than a flat DTO would be, but the structural quarantine
is worth the cost. Reconsider if consumer readability suffers.
## Open questions
1. **Wire nesting depth on `/me`**: `me.full.user.username` is 3
levels. Acceptable? Alternative: `#[serde(flatten)]` on
FullUserDto and SelfUserDto so the wire is flat
(`me.username`, `me.has_password`, `me.ui_preferences` all
at top level), while keeping structural quarantine at compile
time only. Simpler for FE consumers, loses runtime
introspectability (a receiver can't tell which fields are
public vs full vs self from the shape). Recommendation: ship
nested; revisit if FE readability suffers.
2. **`created_at` / `updated_at` on FullUserDto** — not rendered
anywhere currently. Keep for compat unless there's a
compelling reason to drop.
## Memory notes to update on landing
- Extend `project_sessions_last_seen_at_shipped` with a "led to"
pointer at this refactor.
- New note `project_userdto_three_layer_split` — captures the
PublicUserDto / FullUserDto / SelfUserDto pattern + the
decision rule ("would any authenticated caller need this?
PublicUserDto. would self+admin? FullUserDto. self only?
SelfUserDto.").
- Delete the `AdminUserSummaryDto` and `UserListEntry`
references in earlier memory notes.
example.env
+54
View File
@@ -55,6 +55,36 @@ OXICLOUD_SERVER_HOST=0.0.0.0
# Recommended: 127.0.0.1:9090 with node_exporter-style scrapers.
#OXICLOUD_METRICS_LISTEN=127.0.0.1:9090
# ── Startup jobs ──────────────────────────────────────────────────────
# Background jobs dispatched once, after the scheduler is ready.
# Comma-separated; each entry is a registered job name, optionally with
# the same flags the admin trigger URL takes (force, deep, repair,
# storage).
#
# DEFAULT (applied when this variable is unset):
# thumb_derived_import?repair=true,thumb_attached_import?repair=true,transcode_import?repair=true
#
# Those two migrate thumbnails out of the legacy .thumbnails/ directory
# into blob storage and then delete the originals, so the migration
# completes without anyone having to trigger it from the admin panel.
# Each sidecar is read back through the normal stack before it is
# unlinked, and every deletion is written to the audit log.
#
# Dispatch is non-blocking — startup never waits on a job. A run
# interrupted by a restart resumes from its cursor on the next boot, so
# a long migration finishes across restarts. Safe to leave at the
# default: the jobs are idempotent, and once the directory is drained a
# run does nothing at all.
#
# An unknown job name or flag is a FATAL error at boot, not a warning —
# a silently ignored entry means a migration that never runs.
#
# To disable every startup job, set this to the empty value:
#OXICLOUD_STARTUP_JOBS=
#
# To import without deleting (inspect first, delete later by hand):
#OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import
# ── Upload size caps ──────────────────────────────────────────────────
# See docs/config/storage-fine-tuning.md for sizing guidance.
@@ -382,6 +412,30 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud
# Enable music playlists and audio metadata (default: true)
#OXICLOUD_ENABLE_MUSIC=true
# ── Video thumbnails ─────────────────────────────────────────────────────
# Server-side extraction of a single frame from uploaded videos, encoded
# as WebP for the thumbnail grid.
#
# Runtime dependency: `ffmpeg` on PATH (override with OXICLOUD_FFMPEG_PATH).
# When ffmpeg is missing at boot AND this flag is true, the server emits a
# WARN log line ("ffmpeg not found") and falls back to no-thumbnail — the
# API still works, videos just get a placeholder icon.
#
# Turn this off when:
# - You can't install ffmpeg on this host (locked-down image, minimal
# distro, etc.) and want to silence the boot warning.
# - Your client uploads video previews itself (some mobile / desktop
# clients generate thumbnails locally and POST them alongside the
# upload — the server accepts pre-generated thumbnails via the
# upload API and stores them as regular content).
#
# Default: true (extracts server-side when ffmpeg is available).
#OXICLOUD_ENABLE_VIDEO_THUMBNAILS=true
# Explicit path to ffmpeg. Only useful when ffmpeg isn't on PATH or you
# want to pin a specific build (e.g. a static portable ffmpeg). Ignored
# when OXICLOUD_ENABLE_VIDEO_THUMBNAILS=false.
#OXICLOUD_FFMPEG_PATH=/usr/bin/ffmpeg
# Expose other OxiCloud users as a read-only "system" address book
# at GET /api/address-books (default: true)
# Set to false to prevent users from browsing the user directory.
examples/bench_round10_micro.rs
+1
View File
@@ -159,6 +159,7 @@ fn section_identity(iters: u64) {
email: Arc::from("alice.longname@example.com"),
role: "user".to_string(),
dpop_jkt: None,
sid: None,
});
let (bn, ba) = measure("BEFORE String clones + role to_string", iters, || {
examples/bench_round14_queries.rs
+6 -2
View File
@@ -43,9 +43,13 @@ fn stats(mut s: Vec<f64>) -> (f64, f64, f64) {
/// Mirror of `face_pg_repository::bytes_to_embedding` — the per-face
/// `Vec<f32>` decode the BEFORE path pays for a column it never reads.
/// Kept byte-for-byte identical to the shipped path so the benchmark
/// measures apples-to-apples; `as_chunks` swap matches the source.
fn bytes_to_embedding(b: &[u8]) -> Vec<f32> {
b.chunks_exact(4)
.map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
b.as_chunks::<4>()
.0
.iter()
.map(|c| f32::from_le_bytes(*c))
.collect()
}
frontend/src/lib/api/endpoints/admin.test.ts
+1 -4
View File
@@ -64,10 +64,7 @@ describe('admin mutate-based endpoints', () => {
describe('admin read endpoints', () => {
it('call apiJson for the listing/settings reads', async () => {
await admin.listUsers(25, 0);
expect(jsonMock).toHaveBeenCalledWith(
'/api/admin/users?limit=25&offset=0&summary=true',
expect.anything()
);
expect(jsonMock).toHaveBeenCalledWith('/api/admin/users?limit=25&offset=0', expect.anything());
await admin.getDashboard();
await admin.getSmtpInfo();
await admin.getOidcSettings();
frontend/src/lib/api/endpoints/admin.ts
+37 -11
View File
@@ -12,7 +12,7 @@ import type {
DriveMember,
DriveMemberSubject,
DriveRole,
User
FullUser
} from '$lib/api/types';
const JSON_HEADERS = { 'Content-Type': 'application/json' };
@@ -277,19 +277,24 @@ export function revokeAdminSession(sessionId: string): Promise<void> {
// ── Users ───────────────────────────────────────────────────────────────
/** List the compact rows rendered by the management table; full account
* details remain available through {@link getUserAdmin}. */
/** List admin users — always returns `FullUser` rows. The former
* `?summary` toggle is retired; a single canonical shape carries
* the vignette + admin-visible extras the table needs. Single-user
* details still available via {@link getUserAdmin}. */
export function listUsers(limit: number, offset: number): Promise<AdminUsersPage> {
return apiJson<AdminUsersPage>(`/api/admin/users?limit=${limit}&offset=${offset}&summary=true`, {
return apiJson<AdminUsersPage>(`/api/admin/users?limit=${limit}&offset=${offset}`, {
credentials: 'same-origin'
});
}
/**
* Admin-scoped single-user lookup — `GET /api/admin/users/{id}`.
* Returns the full `User` DTO including `storage_quota_bytes` +
* `storage_used_bytes` which the non-admin `/api/users/{id}`
* response omits for privacy.
* Returns the full `FullUser` DTO (public identity in `.user` +
* admin-visible extras like `email_verified_at` / `has_password` /
* `opaque_registered` / `last_login_at` / quotas at top level) —
* same shape as one row of `/api/admin/users` list. The peer-view
* `/api/users/{id}` returns the slim `PublicUser` which omits those
* admin-only signals.
*
* Result promises are cached per id at module scope so multiple
* callers for the same user (e.g. the admin drives table with N
@@ -301,14 +306,14 @@ export function listUsers(limit: number, offset: number): Promise<AdminUsersPage
* still sees the cached value. Callers that need to refresh (e.g.
* after `setUserQuota`) should call `invalidateAdminUserCache`.
*/
const adminUserCache = new Map<string, Promise<User | null>>();
const adminUserCache = new Map<string, Promise<FullUser | null>>();
export function getUserAdmin(id: string): Promise<User | null> {
export function getUserAdmin(id: string): Promise<FullUser | null> {
const hit = adminUserCache.get(id);
if (hit) return hit;
const pending = (async (): Promise<User | null> => {
const pending = (async (): Promise<FullUser | null> => {
try {
return await apiJson<User>(`/api/admin/users/${encodeURIComponent(id)}`, {
return await apiJson<FullUser>(`/api/admin/users/${encodeURIComponent(id)}`, {
credentials: 'same-origin'
});
} catch {
@@ -387,9 +392,30 @@ export interface DriveKindUsage {
}
export interface AdminDashboard {
// ── User accounts (static breakdown of auth.users) ──
// All four are counts of the same table under different
// predicates. Rendered as one grouped section on the dashboard.
total_users: number;
active_users: number;
admin_users: number;
/** Grant-only accounts (magic-link / OIDC-only / OCM recipients).
* Filtered out of `total_users` / `active_users` — those count
* operational seats. Surfaced here as its own metric because
* external-heavy deployments (public-share collab, invited-only
* shops) need the invited population at a glance. */
external_users: number;
// ── Live activity (projection over auth.sessions) ──
// Both change minute-to-minute — a whole different cadence from
// the account counts above. Rendered as a separate section on
// the dashboard with the presence-dot visual cue.
/** Distinct users behind non-revoked sessions active in the last
* 5 min. Same 5-min window as the `oxicloud_sessions_online_users`
* Prometheus gauge; single source of truth on the backend. */
online_users: number;
/** Non-revoked sessions active in the last 5 min. Ratio
* `online_sessions / online_users` is the multi-device factor
* (browser + desktop + phone). */
online_sessions: number;
server_version: string;
drive_usage: DriveKindUsage[];
auth_enabled: boolean;
frontend/src/lib/api/endpoints/adminJobs.ts
+17 -6
View File
@@ -60,11 +60,21 @@ export function listJobs(): Promise<JobSummary[]> {
}
/**
* `POST /api/admin/jobs/{name}/trigger?force=X&deep=X` — dispatch a job
* on-demand. `force` bypasses per-tenant idempotency checks (e.g.
* `trash_cleanup` skipping when nothing is due). `deep` opts into slow
* variants (currently only `storage_consistency`, propagated by
* `consistency_batch` to every child).
* `POST /api/admin/jobs/{name}/trigger?force=X&deep=X&repair=X` —
* dispatch a job on-demand.
*
* - `force` bypasses per-tenant idempotency checks (e.g. `trash_cleanup`
* skipping when nothing is due).
* - `deep` opts into slow variants (currently only `storage_consistency`,
* propagated by `consistency_batch` to every child).
* - `repair` opts into corrective action on the refcount consistency
* tenants (`blobs_consistency`, `manifests_consistency`, and
* `consistency_batch` which fans out to both). Content-safe: only the
* stored counter changes to match the auditor's computed value. Race-
* safe: the corrective UPDATE recomputes the auditor formula in the
* same statement, so a concurrent write can't leave a stale value.
* Default `false` preserves discovery-only behaviour — surface a
* confirm-first flow when calling with `repair: true`.
*
* Throws on 4xx / 5xx with the backend's error message when present.
* A 404 means the job name isn't registered — surface that specifically
@@ -72,11 +82,12 @@ export function listJobs(): Promise<JobSummary[]> {
*/
export async function triggerJob(
name: string,
opts: { force?: boolean; deep?: boolean; storage?: string } = {}
opts: { force?: boolean; deep?: boolean; storage?: string; repair?: boolean } = {}
): Promise<TriggerResponse> {
const params = new URLSearchParams();
if (opts.force) params.set('force', 'true');
if (opts.deep) params.set('deep', 'true');
if (opts.repair) params.set('repair', 'true');
// `storage` scopes tenants that respect JobRunArgs.storage —
// currently blobs_consistency / backend_consistency (probes the
// named entry instead of the live backend). See
frontend/src/lib/api/endpoints/auth.ts
+7 -7
View File
@@ -5,7 +5,7 @@
*/
import { ApiError, apiFetch } from '$lib/api/client';
import { getCsrfHeaders } from '$lib/api/csrf';
import type { AuthResponse, User } from '$lib/api/types';
import type { AuthResponse, SelfUser } from '$lib/api/types';
/**
* Best-effort parse of the backend `ErrorResponse` shape
@@ -45,7 +45,7 @@ const JSON_HEADERS = { 'Content-Type': 'application/json' };
* Failure to build a proof (no keypair, missing WebCrypto) falls back to a
* headerless request — the server still accepts it for unbound sessions.
*/
export async function fetchMe(): Promise<User | null> {
export async function fetchMe(): Promise<SelfUser | null> {
// Build + sign a DPoP proof, send with the header, harvest any
// `DPoP-Nonce` off the response into the shared client cache
// (so the NEXT apiFetch call reuses it — no wasted round trip).
@@ -80,7 +80,7 @@ export async function fetchMe(): Promise<User | null> {
if (dpopMod && dpopMod.isDpopNonceChallenge(res)) res = await send();
if (res.status === 401) return null;
if (!res.ok) throw new Error(`/api/auth/me failed: ${res.status}`);
return (await res.json()) as User;
return (await res.json()) as SelfUser;
}
/**
@@ -408,7 +408,7 @@ export async function setupAdmin(email: string, password: string): Promise<void>
* failure, not an expired access token. Returns the user on success, null on
* any failure so the caller can fall through to the normal login UI.
*/
export async function exchangeOidcCode(code: string): Promise<User | null> {
export async function exchangeOidcCode(code: string): Promise<SelfUser | null> {
try {
const res = await fetch('/api/auth/oidc/exchange', {
method: 'POST',
@@ -417,7 +417,7 @@ export async function exchangeOidcCode(code: string): Promise<User | null> {
body: JSON.stringify({ code })
});
if (!res.ok) return null;
const data = (await res.json()) as { user?: User };
const data = (await res.json()) as { user?: SelfUser };
return data.user ?? null;
} catch {
return null;
@@ -463,7 +463,7 @@ export async function register(email: string, password?: string, username?: stri
* authenticated; a 401 here IS a genuine "session expired" and the
* refresh interceptor is the right response.
*/
export async function upgradeToInternal(password?: string): Promise<User> {
export async function upgradeToInternal(password?: string): Promise<SelfUser> {
const body: Record<string, unknown> = {};
if (password) body.password = password;
const res = await apiFetch('/api/auth/upgrade-to-internal', {
@@ -482,7 +482,7 @@ export async function upgradeToInternal(password?: string): Promise<User> {
message
);
}
return (await res.json()) as User;
return (await res.json()) as SelfUser;
}
export type MagicLinkResult = 'sent' | 'unavailable';
frontend/src/lib/api/endpoints/profile.ts
+3 -3
View File
@@ -1,7 +1,7 @@
/** Profile / account endpoints — ported from views/profile/profile.js. */
import { apiFetch } from '$lib/api/client';
import { getCsrfHeaders } from '$lib/api/csrf';
import type { User } from '$lib/api/types';
import type { SelfUser } from '$lib/api/types';
import { t } from '$lib/i18n/index.svelte';
const JSON_HEADERS = { 'Content-Type': 'application/json' };
@@ -24,7 +24,7 @@ export interface ProfilePatch {
ui_preferences?: Record<string, unknown>;
}
export async function updateProfile(patch: ProfilePatch): Promise<User> {
export async function updateProfile(patch: ProfilePatch): Promise<SelfUser> {
const res = await apiFetch('/api/auth/me/profile', {
method: 'PATCH',
credentials: 'same-origin',
@@ -57,7 +57,7 @@ export async function updateProfile(patch: ProfilePatch): Promise<User> {
}
throw new Error(err.message || err.error || `profile update failed: ${res.status}`);
}
return (await res.json()) as User;
return (await res.json()) as SelfUser;
}
export async function changePassword(currentPw: string, newPw: string): Promise<void> {
frontend/src/lib/api/endpoints/users.ts
+47 -4
View File
@@ -16,15 +16,23 @@ export interface ResolvedUser {
email: string;
image: string | null;
isExternal: boolean;
/** Presence — TRUE when the server observed a request on any of this
* user's non-revoked sessions within the last 5 min (backend
* `PublicUserDto.is_online`). Drives the presence dot overlay on
* `<UserAvatar>` / `<UserVignette>`. `false` when the caller's
* source didn't compute presence (a bare `resolveUser(id)` from
* pre-3-layer callers, an older backend build) — dot stays dark. */
isOnline: boolean;
}
/** Subset of the backend `UserDto` we consume here. */
interface UserDtoShape {
/** Subset of the backend `PublicUserDto` we consume here. */
interface PublicUserShape {
id: string;
username?: string | null;
email?: string | null;
image?: string | null;
is_external: boolean;
is_online?: boolean;
}
// id → in-flight/resolved lookup (the Promise is cached so concurrent callers
@@ -41,13 +49,14 @@ export function resolveUser(id: string): Promise<ResolvedUser | null> {
credentials: 'same-origin'
});
if (!res.ok) return null;
const u = (await res.json()) as UserDtoShape;
const u = (await res.json()) as PublicUserShape;
return {
id: u.id,
name: u.username?.trim() || u.email || u.id,
email: u.email ?? '',
image: u.image ?? null,
isExternal: u.is_external
isExternal: u.is_external,
isOnline: u.is_online ?? false
};
} catch {
return null;
@@ -57,3 +66,37 @@ export function resolveUser(id: string): Promise<ResolvedUser | null> {
cache.set(id, pending);
return pending;
}
/**
* Prime the resolver cache from data the caller already has in hand.
* When a list endpoint (e.g. `/api/admin/users`) ships full
* `PublicUser` rows, the admin page seeds this cache in its load path
* so every subsequent `resolveUser(id)` call (from `UserVignette`
* mounted per-row) hits the cache synchronously — no per-row
* `/api/users/{id}` follow-up fetch. Kills the N+1 that motivated
* widening `/api/admin/users` to include the avatar (see
* `docs/plan/userdto-refactor.md` § N+1).
*
* No-op when the id is already cached (in-flight or resolved). This
* makes seeding safe to call unconditionally — never clobbers an
* authoritative in-flight lookup with a stale seed.
*/
export function seedUser(u: {
id: string;
username?: string | null;
email: string;
image?: string | null;
is_external: boolean;
is_online?: boolean;
}): void {
if (cache.has(u.id)) return;
const resolved: ResolvedUser = {
id: u.id,
name: u.username?.trim() || u.email || u.id,
email: u.email,
image: u.image ?? null,
isExternal: u.is_external,
isOnline: u.is_online ?? false
};
cache.set(u.id, Promise.resolve(resolved));
}
frontend/src/lib/api/types.ts
+146 -126
View File
@@ -179,148 +179,116 @@ export interface TrashResourcesResponse {
export type Role = 'user' | 'admin';
/** Wire shape of `UserDto` (backend: src/application/dtos/user_dto.rs). */
export interface User {
// ─────────────────────────────────────────────────────────────────────────
// Three-layer user family — mirrors src/application/dtos/user_dto.rs.
// See docs/plan/userdto-refactor.md.
//
// `PublicUser` — public identity. Every authenticated caller may see it.
// Returned by /api/users/{id}, share responses, group
// members, magic-link invitees, recipient enrichment.
// `FullUser` — `{ user: PublicUser, ...admin+self extras }`. Returned
// as Vec by /api/admin/users; embedded in `SelfUser`.
// `SelfUser` — `{ full: FullUser, ...self-only extras }`. Returned by
// /api/auth/me and by every auth response.
//
// Adding a field? Decide by audience:
// * Any authenticated caller may see it about another user → PublicUser.
// * Only admin (about another user) AND self (about self) → FullUser.
// * Only self about themselves → SelfUser.
// ─────────────────────────────────────────────────────────────────────────
/** Public identity — 9 fields visible to any authenticated caller. */
export interface PublicUser {
id: string;
username?: string;
email: string;
role: string;
storage_quota_bytes: number;
storage_used_bytes: number;
image?: string | null;
is_external: boolean;
given_name?: string;
family_name?: string;
/** Presence — TRUE when the server observed a request on any of this
* user's non-revoked sessions within the last 5 min. Populated on
* list endpoints; single-user public paths default to `false`.
* Backwards-compat: missing on older backend builds → `false`. */
is_online?: boolean;
}
/** Full user record — public identity + all fields BOTH an admin (viewing
* another user) AND the subject themselves may see. Returned as `Vec` by
* `/api/admin/users`; embedded in `SelfUser` for `/api/auth/me`. */
export interface FullUser {
user: PublicUser;
/** IdP linkage. Load-bearing "is federated?" predicate:
* `full.federation_kind === 'oidc'`. */
federation_kind?: 'oidc' | 'ocm' | 'magic_link';
/** Authority that minted the OIDC/OCM identity — issuer URL for OIDC,
* peer domain for OCM. FE that wants a friendly label maps this
* against `OidcProviders.issuer → provider_name`. */
federation_issuer?: string;
preferred_locale?: string;
email_verified_at?: string;
created_at: string;
updated_at: string;
last_login_at?: string | null;
active: boolean;
/**
* Which trust chain minted this user's federation identity. `null`
* (omitted from wire) for local users (password / OPAQUE only).
* `"oidc" | "ocm" | "magic_link"` for federated users. Predicate:
* `!user.federation_kind` = local; `user.federation_kind === 'oidc'`
* = OIDC user. Mirrors `auth.users.federation_kind` verbatim.
*/
federation_kind?: 'oidc' | 'ocm' | 'magic_link';
/**
* Authority that minted this user's OIDC/OCM identity — issuer URL
* for OIDC (id_token `iss`), peer domain for OCM. `null` (omitted)
* for local users. FE that wants a friendly display label maps this
* against `OidcProviders.issuer → provider_name` when they match;
* shows the raw value otherwise. Renamed from the historical
* `auth_provider` (which held a display label pre-Phase-B and a
* `"local"` sentinel for non-federated users — both are gone).
*/
federation_issuer?: string;
image?: string | null;
can_edit_image: boolean;
is_external: boolean;
given_name?: string;
family_name?: string;
email_verified_at?: string;
preferred_locale?: string;
notify_on_share: boolean;
/**
* Opaque UI preferences bag. Server-side JSONB column that persists
* pure UI toggles (hide-dotfiles, view mode, sidebar collapse, …)
* across devices. The server never inspects the contents — the SPA
* defines the keys (see `lib/stores/preferences.svelte.ts` for the
* typed view). Always an object on the wire (empty bag is `{}`,
* never `null` or missing).
*
* When PATCHing back to the server via
* `PATCH /api/auth/me/profile { ui_preferences: {...} }`, the
* server SHALLOW-merges — only the keys present in the patch are
* touched, so partial writes from one device don't clobber
* preferences set on another. Set a key to `null` in the patch to
* delete it from the bag.
*/
ui_preferences: Record<string, unknown>;
/**
* Mirrors `auth.users.force_password_change_at_next_login`. Only
* populated by `GET /api/auth/me` (see the backend UserDto doc for
* why other UserDto call-sites default to false). When true, the
* SPA MUST lock navigation to the password-change surface — the
* root layout's guard + the backend's `require_no_password_change_pending`
* middleware together enforce this. Optional on the wire because
* older backend builds omit it and `#[serde(default)]` maps
* missing → `false`.
*/
force_password_change?: boolean;
/**
* TRUE when the account has a local Argon2id `password_hash` on
* file. Distinct from `federation_kind`: an OIDC-linked account
* (`federation_kind === 'oidc'`) can ALSO carry a local password
* (hybrid posture — SSO for daily login, local password as
* fallback). The profile page's change-password card gates on this
* flag rather than on the federation shape so hybrid users can
* rotate their local credential. Optional on the wire for older-
* backend compatibility; missing → `false` (safe default: hide the
* card).
*/
has_password?: boolean;
/**
* TRUE when the caller's current session is DPoP-bound (row's
* `dpop_jkt IS NOT NULL`). Populated only by `/api/auth/me`; other
* User-emitting endpoints leave it unset.
*
* The session store reads this to skip a redundant
* `POST /api/auth/dpop/bind` call — the endpoint returns 409
* `already_bound` on repeated attempts (anti-downgrade invariant)
* and each rejection logs at audit INFO, so a naive "bind on
* every load" pattern was cluttering the audit stream. We only
* fire bind now when there's actual work to do (fresh OIDC /
* magic-link session that landed unbound).
*/
is_dpop_bound?: boolean;
storage_quota_bytes: number;
storage_used_bytes: number;
/** TRUE when the account has a local Argon2id `password_hash` on file.
* Distinct from `federation_kind`: an OIDC-linked account can ALSO
* carry a local password (hybrid). */
has_password: boolean;
/** TRUE when the user has an OPAQUE envelope on file. Admin-visible
* rollout signal — kept off `PublicUser` so directory endpoints don't
* leak OPAQUE adoption. */
opaque_registered: boolean;
/** TRUE when the user has completed ≥1 OPAQUE login. Distinct from
* `opaque_registered` — envelope-on-file vs successful-login. */
opaque_migrated: boolean;
}
/** Fields rendered by the paginated admin table. Full account details remain
* available from the detail endpoint; this shape keeps avatars and preference
* documents off every listing page.
*
* The two OPAQUE flags below are ADMIN-ONLY signals: they surface per-user
* OPAQUE rollout progress in the admin table. The backend deliberately keeps
* them off `UserDto` (`/api/auth/me`, share-recipient DTOs, group members)
* so a non-admin can't enumerate the adoption set through third-party
* endpoints. Both optional on the wire — older backend builds omit them and
* `#[serde(default)]` maps missing → `false`. */
export type AdminUserSummary = Pick<
User,
| 'id'
| 'username'
| 'email'
| 'role'
| 'storage_quota_bytes'
| 'storage_used_bytes'
| 'last_login_at'
| 'active'
| 'federation_kind'
| 'federation_issuer'
| 'is_external'
> & {
/** TRUE = user has a server-verifiable password on file (legacy or
* admin-set). Combined with `opaque_registered` and `federation_kind`,
* the admin table derives the full auth capability set — a user with
* `has_password=false`, `opaque_registered=false` AND
* `federation_kind === undefined` (no federation) is passwordless
* (magic-link only, which is the default for externals). */
has_password?: boolean;
/** TRUE = user has an OPAQUE envelope on file (Phase 2 silent migration
* succeeded, or the user completed a manual re-registration). */
opaque_registered?: boolean;
/** TRUE = user has completed at least one successful OPAQUE login.
* Distinct from `opaque_registered` — the envelope may have been
* cleared by an admin reset while a stale migrated=true remains as
* historical signal (backend clears both atomically today, but the
* two-flag shape keeps the option open for a future policy split). */
opaque_migrated?: boolean;
};
/** Self view — everything the caller may see about themselves.
* Returned by `/api/auth/me` and every `AuthResponse` (login / refresh /
* OIDC callback / magic-link redemption ships this so the SPA's post-auth
* state matches its post-`/me` state with no UI race). */
export interface SelfUser {
full: FullUser;
/** Opaque UI-preferences bag. Cross-device store for pure UI toggles
* (view mode, sidebar collapse, hide-dotfiles, …). Server never
* inspects contents; the SPA defines the keys (see
* `lib/stores/preferences.svelte.ts`). Always an object on the wire
* — empty bag is `{}`, never `null`. PATCH via `/api/auth/me/profile`
* shallow-merges; setting a key to `null` removes it. */
ui_preferences: Record<string, unknown>;
/** Whether the user wants share-notification emails. */
notify_on_share: boolean;
/** Session-scoped: my current session is DPoP-bound. SPA reads this
* on `session.load()` to skip a redundant `/api/auth/dpop/bind` call
* (409 `already_bound` otherwise, noisy in the audit stream). */
is_dpop_bound: boolean;
/** Admin-set temp-password gate — SPA nav guard blocks everything
* but /change-password until this flips back. Cleared by a successful
* `POST /api/auth/change-password`. */
force_password_change: boolean;
/** Caller-scoped: can I edit my own avatar? `false` for OIDC users
* whose avatar comes from the IdP. Only meaningful when caller ==
* subject; nonsense on any other DTO. */
can_edit_image: boolean;
}
/** Backwards-compat alias while migrating call-sites. Prefer `PublicUser`
* for public-identity contexts (sharee, group member, invitee) or
* `SelfUser` when reading `/api/auth/me`. Delete once no consumers reference
* the bare `User` name. */
export type User = PublicUser;
export interface AdminUsersPage {
total: number;
users: AdminUserSummary[];
users: FullUser[];
}
export interface AuthResponse {
user: User;
user: SelfUser;
access_token: string;
refresh_token: string;
token_type: string;
@@ -650,8 +618,32 @@ export interface PausedRunBrief {
total?: number;
}
/**
* When a job changes state — `RecoverableJobHandler::mutates()` on the
* backend. Three values rather than a boolean because the interesting
* case is conditional: a job can be read-only by default and destructive
* under `?repair=true`.
*
* - `never` — read-only under every flag. Render a read-only badge; no
* confirmation needed to trigger.
* - `always` — changes state on a plain run. Confirm before triggering.
* - `on_repair_only` — safe to trigger; confirm only when the repair
* toggle is on.
*/
export type Mutates = 'never' | 'always' | 'on_repair_only';
export interface JobSummary {
name: string;
/** One or two sentences on what the job does, in English, authored
* next to the handler. Absent for jobs that haven't declared one —
* omit the line rather than rendering an empty block. */
description?: string;
mutates: Mutates;
/** Present iff `?repair=true` does something beyond a default run;
* describes what it ADDS. Presence is what gates the repair toggle;
* the text is the confirmation copy. Independent of `mutates` — the
* thumbnail import jobs are `always` AND repair-capable. */
repair_description?: string;
interval_ms?: number;
next_run_at?: string;
last_run_at?: string;
@@ -671,6 +663,19 @@ export interface JobSummary {
* for this job. Distinct from `running` — a paused run is
* resumable via the same trigger endpoint. */
paused_run?: PausedRunBrief;
/** Present iff `OXICLOUD_STARTUP_JOBS` names this job — the flags it
* is dispatched with at every boot. Worth showing: a job configured
* with `repair: true` deletes on every restart, and the row would
* otherwise suggest that only happens when someone clicks Run. */
startup?: StartupTrigger;
}
/** Flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with. */
export interface StartupTrigger {
force: boolean;
deep: boolean;
repair: boolean;
storage?: string;
}
/**
@@ -761,12 +766,27 @@ export interface SessionSummary {
user_id: string;
created_at: string;
expires_at: string;
/** Wall-clock (RFC 3339) of the last authenticated request the
* server observed on this session. Trails the true value by at
* most the tracker's flush interval (30 s) on a running server;
* converges after graceful shutdown. Populates the "last seen X
* ago" tooltip on the presence dot. */
last_seen_at: string;
ip_address: string | null;
user_agent: string | null;
is_bound: boolean;
dpop_jkt_prefix: string | null;
is_revoked: boolean;
is_active: boolean;
/** Presence signal — `true` when the server observed a request on
* this session within the last 5 minutes AND the row is `is_active`
* (never `true` on revoked / expired rows). Renders as a filled
* green dot in the Status column; `false` on an otherwise-active
* row renders as an outlined idle dot with a "last seen X ago"
* tooltip. Distinct from `is_active`: that's a lifecycle signal,
* this is a presence signal. Derived server-side against the
* same window that drives `oxicloud_sessions_online[_users]`. */
is_online: boolean;
/** How this session was minted. `unknown` covers pre-migration
* rows and any origin the SPA doesn't yet render. Server enum
* is populated at INSERT (see `Session::new`) and copied on
frontend/src/lib/components/AdminJobsPanel.svelte
+431 -33
View File
@@ -21,6 +21,7 @@
import { SvelteSet } from 'svelte/reactivity';
import Icon from '$lib/icons/Icon.svelte';
import Modal from '$lib/components/Modal.svelte';
import { confirmDialog } from '$lib/stores/dialogs.svelte';
import { t } from '$lib/i18n/index.svelte';
import { errorMessage } from '$lib/utils/errors';
import { ui } from '$lib/stores/ui.svelte';
@@ -66,6 +67,43 @@
else busyKeys.delete(key);
}
// Per-job "Run" split-button menu state. Keyed by job name so
// two rows can open their menus independently (though the
// outside-click handler below closes all on any click outside
// any menu — matching the /files upload dropdown pattern). Only
// rows with `supportsDeep` OR `supportsRepair` render a chevron;
// the plain-Run rows (drives/folders/files/backend/… consistency,
// trash_cleanup, dedup_gc, …) show a bare "Run" button with no
// menu, keeping the common case one-click.
let runMenuOpen = $state<Record<string, boolean>>({});
function toggleRunMenu(name: string) {
runMenuOpen = { ...runMenuOpen, [name]: !runMenuOpen[name] };
}
function closeAllRunMenus() {
runMenuOpen = {};
}
// Global outside-click + Escape dismiss. Only registered while at
// least one menu is open — a background admin tab doesn't hold
// listeners.
$effect(() => {
const anyOpen = Object.values(runMenuOpen).some((v) => v);
if (!anyOpen) return;
const onDown = (e: MouseEvent) => {
if (!(e.target as HTMLElement).closest('.jobs-panel__split')) {
closeAllRunMenus();
}
};
const onKey = (e: KeyboardEvent) => {
if (e.key === 'Escape') closeAllRunMenus();
};
window.addEventListener('pointerdown', onDown);
window.addEventListener('keydown', onKey);
return () => {
window.removeEventListener('pointerdown', onDown);
window.removeEventListener('keydown', onKey);
};
});
// Purge-modal state. Null = closed; otherwise carries the
// draft retention days the operator's picking. Kept separate
// from the top-bar action state so mouse-away doesn't lose
@@ -129,7 +167,7 @@
.slice()
// `consistency_batch` is served by the top-bar
// action buttons; hiding it here removes the
// duplicate table row. `hasBatch` still checks the
// duplicate table row. `batchJob` still reads from the
// full fetched list so the top buttons only render
// when the coordinator is actually registered.
.filter((j) => j.name !== 'consistency_batch')
@@ -142,7 +180,7 @@
// Track whether the coordinator is registered so the
// top-bar buttons can gate on it without checking `jobs`
// (which now filters it out).
hasBatch = fetched.some((j) => j.name === 'consistency_batch');
batchJob = fetched.find((j) => j.name === 'consistency_batch') ?? null;
loadError = null;
} catch (e) {
loadError = errorMessage(e);
@@ -212,13 +250,15 @@
// ─── Expansion toggles ─────────────────────────────────────────────
function toggleJob(name: string) {
if (expandedJob === name) {
function toggleJob(job: JobSummary) {
if (expandedJob === job.name) {
expandedJob = null;
} else {
expandedJob = name;
// Lazy-load on first open, refresh on subsequent opens.
void loadRuns(name);
expandedJob = job.name;
// Lazy-load on first open, refresh on subsequent opens. Only
// recoverable jobs have runs to load — the others expand purely
// to show their description.
if (isRecoverable(job)) void loadRuns(job.name);
}
}
@@ -234,8 +274,10 @@
// ─── Actions ───────────────────────────────────────────────────────
async function onTrigger(name: string, opts: { deep?: boolean } = {}) {
const key = `trigger:${name}${opts.deep ? ':deep' : ''}`;
async function onTrigger(name: string, opts: { deep?: boolean; repair?: boolean } = {}) {
// Key suffix has to keep every dispatched variant distinct so the
// button-disabled state of one doesn't lock out another mid-flight.
const key = `trigger:${name}${opts.deep ? ':deep' : ''}${opts.repair ? ':repair' : ''}`;
markBusy(key, true);
try {
// Fire the trigger + a follow-up loadJobs after a short delay
@@ -265,10 +307,49 @@
if (!res.outcome) {
// dispatched (detached) — no outcome to render
} else if (res.outcome.outcome === 'ok') {
// Repair runs surface a rollup so the operator sees
// whether corrective UPDATEs actually fired. `extra`
// carries `repaired_count` on the two refcount tenants
// directly, and nested under `per_check[*].extra` when
// dispatched via `consistency_batch`. Sum across the
// per_check dict if present, else read the top-level.
let repairedTotal = 0;
let sawRepair = false;
const extra = (res.outcome.extra ?? {}) as {
repair_requested?: boolean;
repaired_count?: number;
per_check?: Record<
string,
{ extra?: { repair_requested?: boolean; repaired_count?: number } }
>;
};
if (extra.repair_requested) {
sawRepair = true;
repairedTotal += extra.repaired_count ?? 0;
}
if (extra.per_check) {
for (const child of Object.values(extra.per_check)) {
if (child?.extra?.repair_requested) {
sawRepair = true;
repairedTotal += child.extra.repaired_count ?? 0;
}
}
}
if (sawRepair) {
ui.notify(
t(
'admin.jobs.triggered_ok_repair',
{ name, n: repairedTotal },
'{{name}}: {{n}} counter(s) repaired'
),
'success'
);
} else {
ui.notify(
t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
'success'
);
}
} else {
ui.notify(
t(
@@ -379,8 +460,9 @@
* Per-severity finding counts from `last_outcome.extra.severity_counts`
* (a JSON object populated by `run_or_resume`). Missing / older
* runs return an empty record — callers should tolerate absent keys.
* The three severity values are the ones consistency tenants emit
* today: `data_loss`, `inconsistent`, `anomaly`.
* Severity values emitted today: `data_loss`, `inconsistent`,
* `anomaly`. The set is open (the column is TEXT), so unknown keys
* must degrade rather than throw.
*/
function lastSeverityCounts(job: JobSummary): Record<string, number> {
if (!job.last_outcome || job.last_outcome.outcome !== 'ok') return {};
@@ -401,6 +483,13 @@
return (s.data_loss ?? 0) + (s.inconsistent ?? 0);
}
/**
* Informational findings. `anomaly` is the wire value; "notice" is
* what the panel calls it — there is no separate `notice` severity.
* A job that acted on what it found (a repair run deleting an
* orphaned sidecar) records the same severity and says so in the
* finding's `detail`.
*/
function anomalyFindingCount(job: JobSummary): number {
return lastSeverityCounts(job).anomaly ?? 0;
}
@@ -548,13 +637,75 @@
// Jobs that respect `?deep=true`:
// * `consistency_batch` — propagates deep to every child that
// understands it
// * `blobs_consistency` — deep mode re-reads + re-hashes every
// blob for silent bit-rot detection (severity `data_loss`).
// Full read of storage; can take hours on big installs — the
// "Run" (normal) button on the same row does the cheap
// existence probes only.
// * `backend_consistency` — deep mode re-reads + re-hashes every
// matched blob for silent bit-rot detection (severity
// `data_loss`). Full read of storage; can take hours on big
// installs — the "Run" button on the same row does the
// enumeration merge-join only. This was `blobs_consistency`
// until that tenant became database-only.
function supportsDeep(name: string): boolean {
return name === 'consistency_batch' || name === 'blobs_consistency';
return name === 'consistency_batch' || name === 'backend_consistency';
}
// Whether `?repair=true` does anything for this job — declared by the
// handler itself via `repair_description()`, not by a name allowlist
// here. The allowlist this replaces named only the two ref_count
// tenants and silently omitted every repair-capable job added since,
// so the thumbnail imports could not be run in repair mode from the
// panel at all despite supporting it.
function supportsRepair(job: JobSummary): boolean {
return !!job.repair_description;
}
// What the repair adds, in the handler's own words. The backend owns
// this string precisely because the wording differs per job: correcting
// a counter and unlinking files off disk are not the same warning, and
// the frontend has no way to tell them apart.
async function onTriggerWithRepairConfirm(job: JobSummary) {
const ok = await confirmDialog({
title: t(
'admin.jobs.run_repair_confirm_title_scoped',
{ name: job.name },
'Run {{name}} in repair mode?'
),
message: job.repair_description ?? '',
confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
danger: true
});
if (ok) await onTrigger(job.name, { repair: true });
}
// Confirmation before a plain run of a job that writes. `never` jobs
// trigger straight through — that is the point of the flag — and
// `on_repair_only` jobs are read-only until the repair variant is
// picked, which carries its own confirm.
async function onTriggerGuarded(job: JobSummary) {
if (job.mutates === 'always') {
const ok = await confirmDialog({
title: t('admin.jobs.run_mutating_confirm_title', { name: job.name }, 'Run {{name}}?'),
message:
job.description ||
t('admin.jobs.run_mutating_confirm_body', 'This job changes stored state when it runs.'),
confirmText: t('admin.jobs.run', 'Run'),
danger: true
});
if (!ok) return;
}
await onTrigger(job.name);
}
// Row badge. `never` is the one worth stating outright — it is the
// answer to "is it safe to click this on production?", and it is the
// question an operator asks before every trigger.
function mutatesLabel(job: JobSummary): string | null {
switch (job.mutates) {
case 'never':
return t('admin.jobs.mutates_never', 'read-only');
case 'on_repair_only':
return t('admin.jobs.mutates_on_repair_only', 'read-only unless repaired');
default:
return null;
}
}
function isRunning(job: JobSummary): boolean {
@@ -575,11 +726,13 @@
// coordinator is registered (should always be true post-Slice 5,
// but check defensively so the button doesn't appear on an old
// deployment before this component is upgraded).
// Coordinator registration flag — set imperatively in
// `loadJobs` because `jobs` no longer contains the
// `consistency_batch` row (filtered out to avoid duplicating the
// top-bar action buttons).
let hasBatch = $state(false);
// Held as the whole summary rather than a boolean because the
// top-bar buttons need its `repair_description` — the coordinator
// describes its own repair semantics, same as every table row.
// Set imperatively in `loadJobs` because `jobs` no longer contains
// the `consistency_batch` row (filtered out to avoid duplicating
// the top-bar action buttons).
let batchJob = $state<JobSummary | null>(null);
</script>
<section class="jobs-panel">
@@ -593,10 +746,12 @@
</p>
</div>
<div class="jobs-panel__header-actions">
{#if hasBatch}
{#if batchJob}
{@const batch = batchJob}
<button
class="jobs-panel__btn jobs-panel__btn--primary"
disabled={busyKeys.has('trigger:consistency_batch')}
title={batch.description || undefined}
onclick={() => onTrigger('consistency_batch')}
>
<Icon name="play" />
@@ -614,10 +769,28 @@
<Icon name="play" />
{t('admin.jobs.run_deep', 'Run deep')}
</button>
<!-- Repair goes behind a confirm because it fans `?repair=true`
out to every sub-check that acts on it. The confirmation
text comes from the coordinator's own
`repair_description` rather than being written here —
what repair means changes as tenants gain repair arms,
and this button would otherwise keep describing only the
two refcount ones. -->
{#if batch.repair_description}
<button
class="jobs-panel__btn jobs-panel__btn--warn"
disabled={busyKeys.has('trigger:consistency_batch:repair')}
title={batch.repair_description}
onclick={() => onTriggerWithRepairConfirm(batch)}
>
<Icon name="cog" />
{t('admin.jobs.run_repair', 'Repair ref_counts')}
</button>
{/if}
{/if}
<!-- Purge is orthogonal to consistency — it works even
when the batch coordinator isn't registered, so it
lives outside the {#if hasBatch}. Opens a modal so
lives outside the batch block. Opens a modal so
the operator picks a retention window with intent
(no accidental delete-all). -->
<button
@@ -661,7 +834,12 @@
{@const runsErr = runsErrorByJob[job.name]}
{@const runsLoading = runsLoadingByJob[job.name]}
{@const expandedRun = expandedRunByJob[job.name] ?? null}
{@const canExpand = isRecoverable(job)}
<!-- Expandable if there is anything to show: a run history,
a description, or both. Gating on `recoverable` alone
would leave the plain periodic jobs (dedup_gc,
trash_cleanup, …) with no way to reach their
description at all. -->
{@const canExpand = isRecoverable(job) || !!job.description}
<tr class="jobs-panel__row" class:jobs-panel__row--expanded={expandedJob === job.name}>
<td>
{#if canExpand}
@@ -669,7 +847,7 @@
type="button"
class="jobs-panel__expand"
aria-expanded={expandedJob === job.name}
onclick={() => toggleJob(job.name)}
onclick={() => toggleJob(job)}
>
<Icon name={expandedJob === job.name ? 'chevron-down' : 'chevron-right'} />
<span class="jobs-panel__name">{job.name}</span>
@@ -677,8 +855,43 @@
{:else}
<span class="jobs-panel__name jobs-panel__name--flat">{job.name}</span>
{/if}
{#if mutatesLabel(job)}
<span class="jobs-panel__pill jobs-panel__pill--readonly">
{mutatesLabel(job)}
</span>
{/if}
</td>
<!-- "At boot" belongs in the cadence column: it answers
WHEN this job runs, which is the same question
`interval_ms` answers. Beside the name it read as a
property of the job rather than of its schedule, and
these two facts have to be read together — a job with
no interval that fires at boot is not on-demand, and
the row said "on-demand" next to a badge saying
otherwise. -->
<td class="jobs-panel__muted">
{cadenceLabel(job)}
{#if job.startup}
<span
class="jobs-panel__pill"
class:jobs-panel__pill--paused={job.startup.repair}
class:jobs-panel__pill--neutral={!job.startup.repair}
title={job.startup.repair
? t(
'admin.jobs.startup_repair_tooltip',
'Configured in OXICLOUD_STARTUP_JOBS to run in repair mode at every boot.'
)
: t(
'admin.jobs.startup_tooltip',
'Configured in OXICLOUD_STARTUP_JOBS to run at every boot.'
)}
>
{job.startup.repair
? t('admin.jobs.startup_repair', 'at boot · repair')
: t('admin.jobs.startup', 'at boot')}
</span>
{/if}
</td>
<td class="jobs-panel__muted">{cadenceLabel(job)}</td>
<td class="jobs-panel__muted">{timeAgo(job.last_run_at)}</td>
<td>
<div class="jobs-panel__outcome-cell">
@@ -754,24 +967,84 @@
{t('admin.jobs.cancel', 'Cancel')}
</button>
{:else}
<!-- Split-button: primary "Run" fires the default
trigger; the chevron opens a menu with the
tenant-specific variants (Run deep / Repair).
Rows without any variant render a bare Run
button — no chevron, no menu, no extra
width. Preserves one-click discovery for
the common case. -->
{@const hasRunVariants = supportsDeep(job.name) || supportsRepair(job)}
<span class="jobs-panel__split">
<button
class="jobs-panel__btn jobs-panel__btn--small"
class:jobs-panel__split-main={hasRunVariants}
disabled={busyKeys.has(`trigger:${job.name}`)}
onclick={() => onTrigger(job.name)}
title={job.description || undefined}
onclick={() => {
closeAllRunMenus();
void onTriggerGuarded(job);
}}
>
{t('admin.jobs.run', 'Run')}
</button>
{#if hasRunVariants}
<button
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__split-toggle"
aria-haspopup="menu"
aria-expanded={runMenuOpen[job.name] ?? false}
aria-label={t('admin.jobs.run_variants_menu', 'Run variants menu')}
onclick={() => toggleRunMenu(job.name)}
>
<Icon name="caret-down" />
</button>
{#if runMenuOpen[job.name]}
<div class="jobs-panel__run-menu" role="menu">
{#if supportsDeep(job.name)}
<button
class="jobs-panel__btn jobs-panel__btn--small"
type="button"
class="jobs-panel__run-menu-item"
role="menuitem"
disabled={busyKeys.has(`trigger:${job.name}:deep`)}
onclick={() => onTrigger(job.name, { deep: true })}
title={t(
'admin.jobs.run_deep_hint',
'Also runs slow variants (blob re-hash, bitrot detection).'
)}
onclick={() => {
closeAllRunMenus();
void onTrigger(job.name, { deep: true });
}}
>
{t('admin.jobs.run_deep', 'Run deep')}
<Icon name="search" />
<span>{t('admin.jobs.run_deep', 'Run deep')}</span>
</button>
{/if}
{#if supportsRepair(job)}
<button
type="button"
class="jobs-panel__run-menu-item jobs-panel__run-menu-item--warn"
role="menuitem"
disabled={busyKeys.has(`trigger:${job.name}:repair`)}
title={job.repair_description}
onclick={() => {
closeAllRunMenus();
void onTriggerWithRepairConfirm(job);
}}
>
<Icon name="cog" />
<span>{t('admin.jobs.run_repair', 'Repair')}</span>
</button>
{/if}
{#if isRunning(job) && canExpand}
</div>
{/if}
{/if}
</span>
{/if}
<!-- `isRecoverable`, not `canExpand`: the latter now also
covers rows that expand only to show a description,
and those must not gain a Cancel button they never
had. -->
{#if isRunning(job) && isRecoverable(job)}
{#if isRecoverable(job)}
<!-- Recoverable running: [Pause] preserves cursor
for later resume; [Cancel] abandons terminally
@@ -811,7 +1084,20 @@
</td>
</tr>
{#if expandedJob === job.name}
<!-- First row of the expanded block, and only visible there:
the description answers "what is this job" before the
run history answers "what did it do", and keeping it
folded keeps the collapsed table scannable — 17 rows of
two-line prose is not a table any more. -->
{#if expandedJob === job.name && job.description}
<tr class="jobs-panel__desc-row">
<td colspan="6">
<p class="jobs-panel__description">{job.description}</p>
</td>
</tr>
{/if}
{#if expandedJob === job.name && isRecoverable(job)}
<tr class="jobs-panel__runs">
<td colspan="6">
<div class="jobs-panel__runs-inner">
@@ -1304,6 +1590,90 @@
color: var(--color-danger-text-alt);
}
/* Warn variant — used for actions that mutate data but are content-
safe / reversible-in-outcome (e.g. Repair ref_counts). Signals
"read the tooltip and the confirm before clicking" without the
danger red reserved for destructive delete-style buttons. */
.jobs-panel__btn--warn {
border-color: var(--color-warning-border);
color: var(--color-warning-text);
}
/* Split-button — inline flex holding a primary "Run" (fires default
action) and a chevron (opens the variants menu). `position:
relative` anchors the menu below the toggle. Only rendered on
rows whose job supports at least one variant; plain-Run rows
sidestep this whole structure. */
.jobs-panel__split {
display: inline-flex;
position: relative;
}
/* Attached-button trick: main loses its right border-radius, toggle
loses its left. Toggle also loses its left border so the two
don't render a double-thick divider. */
.jobs-panel__split-main {
border-top-right-radius: 0;
border-bottom-right-radius: 0;
}
.jobs-panel__split-toggle {
border-top-left-radius: 0;
border-bottom-left-radius: 0;
border-left: none;
padding-left: 0.35rem;
padding-right: 0.35rem;
}
/* The variants menu — dropdown below the toggle, right-aligned so
it doesn't overflow the Actions column edge into the next row's
badge cell. Shadow + surface bg mirror the /files upload
dropdown (`upload-dropdown-menu`); using local CSS here rather
than the ported class so the jobs-panel keeps its scoped styling. */
.jobs-panel__run-menu {
position: absolute;
top: calc(100% + 2px);
right: 0;
z-index: 30;
min-width: 10rem;
background: var(--color-bg-surface);
border: 1px solid var(--color-border);
border-radius: var(--radius-md, 6px);
box-shadow: var(--shadow-md);
padding: 0.25rem 0;
}
.jobs-panel__run-menu-item {
display: flex;
align-items: center;
gap: 0.5rem;
width: 100%;
padding: 0.4rem 0.75rem;
background: transparent;
border: none;
text-align: left;
font: inherit;
color: var(--color-text);
cursor: pointer;
white-space: nowrap;
}
.jobs-panel__run-menu-item:hover:not(:disabled) {
background: var(--color-bg-hover);
}
.jobs-panel__run-menu-item:disabled {
opacity: 0.5;
cursor: not-allowed;
}
/* Warn colour on the menu item mirrors the button variant so the
Repair option carries the same "attention-worthy but not
destructive" visual weight as its top-bar counterpart. */
.jobs-panel__run-menu-item--warn {
color: var(--color-warning-text);
}
.jobs-panel__pill {
display: inline-block;
padding: 0.1rem 0.5rem;
@@ -1342,6 +1712,34 @@
color: var(--color-text-muted);
}
/* "read-only" sits beside the job name and answers the question an
operator asks before every trigger. Deliberately quiet — it marks
the safe case, so it should not compete with outcome pills. */
.jobs-panel__pill--readonly {
margin-left: 0.4rem;
background: var(--color-bg-subtle);
color: var(--color-text-muted);
font-size: 0.72rem;
font-weight: 400;
vertical-align: middle;
}
/* Opens the expanded block, so it carries the drawer's background and
drops its own separator — the runs table below it is part of the
same block, not a new entry. */
.jobs-panel__desc-row td {
padding-left: 2rem; /* clears the chevron, lines up with the name */
border-bottom-color: transparent;
background: var(--color-bg-subtle);
}
.jobs-panel__description {
margin: 0;
color: var(--color-text-muted);
font-size: 0.8rem;
line-height: 1.4;
}
.jobs-panel__runs {
background: var(--color-bg-subtle);
}
frontend/src/lib/components/AppShell.svelte
+13 -13
View File
@@ -437,11 +437,11 @@
{ mode: 'dark', icon: 'moon', label: t('user_menu.theme.dark', 'Dark') }
];
const storagePct = $derived(
session.user && session.user.storage_quota_bytes > 0
? Math.min(100, (session.user.storage_used_bytes / session.user.storage_quota_bytes) * 100)
: 0
);
const storagePct = $derived.by(() => {
const full = session.me?.full;
if (!full || full.storage_quota_bytes <= 0) return 0;
return Math.min(100, (full.storage_used_bytes / full.storage_quota_bytes) * 100);
});
const initials = $derived(userInitials(session.user?.username || session.user?.email));
@@ -655,12 +655,12 @@
<div class="storage-fill" style:width="{storagePct}%"></div>
</div>
<div class="storage-info">
{#if session.user.storage_quota_bytes > 0}
{Math.round(storagePct)}% · {formatBytes(session.user.storage_used_bytes)} / {formatBytes(
session.user.storage_quota_bytes
{#if (session.me?.full.storage_quota_bytes ?? 0) > 0}
{Math.round(storagePct)}% · {formatBytes(session.me?.full.storage_used_bytes ?? 0)} / {formatBytes(
session.me?.full.storage_quota_bytes ?? 0
)}
{:else}
{formatBytes(session.user.storage_used_bytes)}
{formatBytes(session.me?.full.storage_used_bytes ?? 0)}
{/if}
</div>
</div>
@@ -903,18 +903,18 @@
<div class="user-menu-storage-fill" style:width="{storagePct}%"></div>
</div>
<div class="user-menu-storage-text">
{#if session.user.storage_quota_bytes > 0}
{#if (session.me?.full.storage_quota_bytes ?? 0) > 0}
{t(
'storage.used',
{
percentage: Math.round(storagePct),
used: formatBytes(session.user.storage_used_bytes),
total: formatBytes(session.user.storage_quota_bytes)
used: formatBytes(session.me?.full.storage_used_bytes ?? 0),
total: formatBytes(session.me?.full.storage_quota_bytes ?? 0)
},
'{{percentage}}% used ({{used}} / {{total}})'
)}
{:else}
{formatBytes(session.user.storage_used_bytes)}
{formatBytes(session.me?.full.storage_used_bytes ?? 0)}
{/if}
</div>
</div>
frontend/src/lib/components/AppShell.test.ts
+14 -3
View File
@@ -26,16 +26,27 @@ const children = createRawSnippet(() => ({
beforeEach(() => {
vi.clearAllMocks();
pageState.url = new URL('http://localhost/files');
session.user = {
// Post the three-layer UserDto refactor, `session.user` is a
// derived accessor over `session.me.full.user`; only `session.me`
// is settable. Fixture composes the nested shape — public identity
// (username/email/name) on `.full.user`, admin+self extras
// (storage_*, has_password) on `.full`, self-only bag (ui_prefs,
// dpop_bound, force_password_change, can_edit_image) at the top.
// See docs/plan/userdto-refactor.md.
session.me = {
full: {
user: {
id: '1',
username: 'admin',
email: 'a@x.test',
given_name: 'A',
family_name: 'B',
role: 'admin',
storage_used_bytes: 10,
storage_quota_bytes: 100,
is_external: false
},
storage_used_bytes: 10,
storage_quota_bytes: 100
}
} as never;
});
frontend/src/lib/components/EmptyState.svelte
+13 -1
View File
@@ -18,7 +18,19 @@
let { icon, title, hint, error = false, children }: Props = $props();
</script>
<div class="empty-state" class:empty-state--error={error} role={error ? 'alert' : undefined}>
<!-- `data-testid="empty-state"` is a stable Playwright hook: consumers
(ResourceList, ShareList, TrashList, …) only render this component
once the underlying load resolved with no items — so waiting on
this testid = "the listing definitively finished loading and is
empty". Used by `tests/e2e/spa/files.spec.ts` to gate a cold-
navigation upload behind the folder-loaded state (see the guard
in `routes/files/[...path]/+page.svelte::guardUploadFolderReady`). -->
<div
class="empty-state"
class:empty-state--error={error}
role={error ? 'alert' : undefined}
data-testid="empty-state"
>
{#if icon}<Icon name={icon} class="empty-state__icon" />{/if}
{#if title}<p class="empty-state__title">{title}</p>{/if}
{#if hint}<p class="empty-state__hint">{hint}</p>{/if}
frontend/src/lib/components/UserVignette.svelte
+38
View File
@@ -33,6 +33,7 @@
const label = $derived(resolved?.name ?? fallbackLabel ?? userId);
const email = $derived(resolved?.email || fallbackSublabel || '');
const isExternal = $derived(resolved?.isExternal ?? false);
const isOnline = $derived(resolved?.isOnline ?? false);
const image = $derived(resolved?.image ?? null);
const colorIndex = $derived(avatarColorIndex(userId));
const initials = $derived(userInitials(label));
@@ -50,6 +51,22 @@
<Icon name="building-circle-xmark" />
</span>
{/if}
{#if isOnline}
<!-- Presence dot — top-right corner so it doesn't collide with the
external badge at bottom-right. Only rendered when true (absent
= offline reads cleanly on an avatar; no grey placeholder). Uses
`--color-success-alt` (same green as `.badge--active` in the
admin sessions panel + `.presence-dot--online` in the sessions
status column) so the presence signal reads consistently across
every surface. The 2px surface-coloured border visually detaches
the dot from the avatar's own background — matches the pattern
Slack/Teams/Discord use. -->
<span
class="uv__presence"
title={t('share.userOnline', 'Online')}
aria-label={t('share.userOnline', 'Online')}
></span>
{/if}
</span>
<span class="uv__text">
<span class="uv__name">{label}</span>
@@ -128,6 +145,27 @@
font-size: 9px;
}
/* Presence dot — top-right, symmetric with `.uv__badge` at
bottom-right so the two corners don't collide. Slightly smaller
(10x10 vs the badge's 16x16) because it's a pure signal — no
icon, no text. The 2px `--color-bg-surface` border creates a
visual gap between dot and avatar so the green pops out cleanly
regardless of avatar palette (photo, dark initials, light
initials). `box-sizing: border-box` keeps the inner circle's
green footprint at 6x6 — same visual weight the sessions-table
dot has. See `docs/plan/sessions.md` § UI. */
.uv__presence {
position: absolute;
right: -2px;
top: -2px;
width: 10px;
height: 10px;
border-radius: 50%;
background: var(--color-success-alt);
border: 2px solid var(--color-bg-surface);
box-sizing: border-box;
}
.uv__text {
display: flex;
flex-direction: column;
frontend/src/lib/stores/preferences.svelte.ts
+20 -10
View File
@@ -69,12 +69,15 @@ const PATCH_DEBOUNCE_MS = 500;
class PreferencesStore {
/**
* The typed view of the bag. Derived from `session.user?.ui_preferences`
* so signing in / out / refresh flips it in lockstep with the session.
* The typed view of the bag. Derived from `session.me?.ui_preferences`
* (moved from public `User.ui_preferences` to `SelfUser.ui_preferences`
* as part of the three-layer UserDto refactor — the bag is self-only
* state, not something other authenticated callers should see).
* Signing in / out / refresh flips it in lockstep with the session.
* Reads pass through DEFAULTS for any missing key.
*/
private bag = $derived<Record<string, unknown>>(
(session.user?.ui_preferences as Record<string, unknown> | undefined) ?? {}
(session.me?.ui_preferences as Record<string, unknown> | undefined) ?? {}
);
// ── Typed accessors ──────────────────────────────────────────
@@ -100,11 +103,14 @@ class PreferencesStore {
* `jsonb_strip_nulls` after the merge).
*/
set(patch: Partial<Record<keyof UiPreferences, unknown>>): void {
if (!session.user) return;
if (!session.me) return;
// Optimistic local write — mutate the reactive user shallowly.
// Optimistic local write — mutate the reactive me shallowly.
// `ui_preferences` lives on `SelfUser` (self-only), not on the
// public `User` slice, so the mutation stays at the SelfUser
// level. The nested `full` / `full.user` blocks are untouched.
const nextBag = {
...((session.user.ui_preferences as Record<string, unknown> | undefined) ?? {}),
...((session.me.ui_preferences as Record<string, unknown> | undefined) ?? {}),
...patch
};
// Strip any explicit-null locally so the derived getters see the
@@ -114,7 +120,7 @@ class PreferencesStore {
for (const [k, v] of Object.entries(patch)) {
if (v === null) delete (nextBag as Record<string, unknown>)[k];
}
session.user = { ...session.user, ui_preferences: nextBag };
session.me = { ...session.me, ui_preferences: nextBag };
// Accumulate keys so successive `set` calls before the debounce
// fires collapse into a single PATCH body — matters for
@@ -131,16 +137,20 @@ class PreferencesStore {
this.pendingPatch = {};
if (Object.keys(patch).length === 0) return;
const previousUser = session.user;
// `session.user` is a derived read-through on `session.me.full.user`
// — the source of truth is `session.me: SelfUser`. Snapshot + assign
// there so the optimistic update / rollback matches the store shape
// (see `docs/plan/userdto-refactor.md` for the layering).
const previousMe = session.me;
try {
const updated = await updateProfile({ ui_preferences: patch });
session.user = updated;
session.me = updated;
} catch {
// Roll back to whatever the server last confirmed. The
// optimistic local mutation is discarded and the derived
// `hideDotfiles` / other getters snap back on the next
// reactivity tick.
session.user = previousUser;
session.me = previousMe;
ui.notify(
t('preferences.save_failed', "Couldn't save your preference. Please try again."),
'error'
frontend/src/lib/stores/session.svelte.test.ts
+12 -5
View File
@@ -1,5 +1,5 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { User } from '$lib/api/types';
import type { SelfUser } from '$lib/api/types';
// `vi.mock` is hoisted above imports, so the spy it references must be created
// with `vi.hoisted` (a plain top-level const isn't initialised yet when the
@@ -14,7 +14,14 @@ vi.mock('$lib/api/endpoints/auth', () => ({
import { session } from './session.svelte';
const userWithUsage = (used: number) => ({ storage_used_bytes: used }) as unknown as User;
// `storage_used_bytes` moved to `FullUser` (embedded inside `SelfUser`)
// as part of the three-layer UserDto refactor
// (`docs/plan/userdto-refactor.md`). Build a minimal SelfUser shape that
// satisfies the type checker without hand-populating every field the
// production shape carries — the test only cares about the usage read
// path (`session.me.full.storage_used_bytes`).
const userWithUsage = (used: number) =>
({ full: { storage_used_bytes: used } }) as unknown as SelfUser;
describe('session.refresh', () => {
beforeEach(() => {
@@ -25,7 +32,7 @@ describe('session.refresh', () => {
it('pulls the fresh storage usage into the reactive user (upload/delete sync)', async () => {
fetchMeMock.mockResolvedValue(userWithUsage(2048));
await session.refresh();
expect(session.user?.storage_used_bytes).toBe(2048);
expect(session.me?.full.storage_used_bytes).toBe(2048);
});
it('leaves the current user intact when the probe returns null', async () => {
@@ -33,7 +40,7 @@ describe('session.refresh', () => {
await session.refresh();
fetchMeMock.mockResolvedValue(null);
await session.refresh();
expect(session.user?.storage_used_bytes).toBe(2048);
expect(session.me?.full.storage_used_bytes).toBe(2048);
});
it('leaves the current user intact when the probe throws', async () => {
@@ -41,6 +48,6 @@ describe('session.refresh', () => {
await session.refresh();
fetchMeMock.mockRejectedValue(new Error('network'));
await session.refresh();
expect(session.user?.storage_used_bytes).toBe(2048);
expect(session.me?.full.storage_used_bytes).toBe(2048);
});
});
frontend/src/lib/stores/session.svelte.ts
+39 -17
View File
@@ -11,17 +11,39 @@ import { setLogoutInProgress } from '$lib/api/client';
import { hasSessionHint } from '$lib/api/csrf';
import { seedNonceFromCookie } from '$lib/auth/dpop-proof';
import { drives } from '$lib/stores/drives.svelte';
import type { User } from '$lib/api/types';
import type { PublicUser, SelfUser } from '$lib/api/types';
import { ensureActiveUser } from '$lib/utils/localStoragePrefs';
/**
* Session store — the authenticated user and derived flags.
*
* Post the three-layer UserDto refactor (`docs/plan/userdto-refactor.md`),
* `/api/auth/me` returns `SelfUser` (composed:
* `SelfUser.full.user: PublicUser`). Two shorthand accessors keep every
* existing consumer readable:
*
* - `session.user` → `PublicUser` (via `me.full.user`). Every callsite
* that read `session.user.username / email / id / role / image /
* is_external / given_name / family_name / is_online` keeps working.
* - `session.me` → full `SelfUser`. New code that needs self-only or
* admin-visible fields (`has_password`, `is_dpop_bound`, `active`,
* `ui_preferences`, `federation_kind`, `last_login_at`, quotas, …)
* reads through `session.me.full.foo` or `session.me.foo`.
*/
class SessionStore {
user = $state<User | null>(null);
/** Full `/api/auth/me` payload. Null when unauthenticated. */
me = $state<SelfUser | null>(null);
loaded = $state(false);
homeFolderId = $state<string | null>(null);
homeFolderName = $state<string | null>(null);
isExternalUser = $derived(this.user?.is_external ?? false);
isAuthenticated = $derived(this.user !== null);
/** Public-identity shorthand — same fields any authenticated caller
* can see. Every legacy `session.user.foo` read (username, email, id,
* role, image, is_external, given_name, family_name, is_online) still
* works via this derived accessor. */
user = $derived<PublicUser | null>(this.me?.full.user ?? null);
isExternalUser = $derived(this.me?.full.user.is_external ?? false);
isAuthenticated = $derived(this.me !== null);
/**
* TRUE when the backend has set `force_password_change_at_next_login`
* on this account — an admin picked a temporary password and the
@@ -33,7 +55,7 @@ class SessionStore {
* flag (or a malformed `/me` response) doesn't accidentally
* quarantine every user.
*/
mustChangePassword = $derived(this.user?.force_password_change === true);
mustChangePassword = $derived(this.me?.force_password_change === true);
/**
* Resolve the session once. Probes /api/auth/me; on 401 it makes a single
@@ -41,15 +63,15 @@ class SessionStore {
* what to do with an unauthenticated result. Idempotent: subsequent calls
* return the cached result (so client-side navigation doesn't re-probe).
*/
async load(): Promise<User | null> {
if (this.loaded) return this.user;
async load(): Promise<SelfUser | null> {
if (this.loaded) return this.me;
// No JS-visible session hint ⇒ nothing to probe. The server sets
// `oxicloud_csrf` alongside the HttpOnly session cookies and clears
// it on logout, so a missing hint means no session. Skips the
// doomed 2× /me + /refresh burst that would otherwise fire on
// every first landing / post-logout re-mount with no cookies.
if (!hasSessionHint()) {
this.user = null;
this.me = null;
this.loaded = true;
return null;
}
@@ -71,25 +93,25 @@ class SessionStore {
// otherwise clutter the audit stream. Fire-and-forget
// so a slow IndexedDB open doesn't stall app boot.
if (me.is_dpop_bound === false) void bindDpopIfPossible();
} else this.user = null;
} else this.me = null;
} catch {
this.user = null;
this.me = null;
}
this.loaded = true;
return this.user;
return this.me;
}
/**
* Set the authenticated user AND run per-user localStorage cleanup
* (see `$lib/utils/localStoragePrefs::ensureActiveUser`). Direct
* `session.user = …` assignments skip the cleanup — always call
* `session.me = …` assignments skip the cleanup — always call
* `setUser` on login-flow entry points (form login, OIDC exchange,
* existing-session probe) so a switch-account flow inside the same
* tab observes the wipe.
*/
setUser(user: User): void {
this.user = user;
ensureActiveUser(user.id);
setUser(me: SelfUser): void {
this.me = me;
ensureActiveUser(me.full.user.id);
// Any successful login clears the session-teardown gate. Without
// this, a logout → login within the same SPA session leaves the
// gate stuck at `true` — the login POST is exempted via
@@ -115,7 +137,7 @@ class SessionStore {
async refresh(): Promise<void> {
try {
const me = await fetchMe();
if (me) this.user = me;
if (me) this.me = me;
} catch {
/* keep the existing user on a transient /api/auth/me failure */
}
@@ -142,7 +164,7 @@ class SessionStore {
}
reset(): void {
this.user = null;
this.me = null;
this.homeFolderId = null;
this.homeFolderName = null;
// Mark the store as `loaded` so any subsequent `session.load()` —
frontend/src/routes/admin/[[tab]]/+page.svelte
+288 -57
View File
@@ -63,6 +63,7 @@
type StorageTestResult
} from '$lib/api/endpoints/admin';
import { createDrive, updateDrivePolicies } from '$lib/api/endpoints/drives';
import { seedUser } from '$lib/api/endpoints/users';
import {
ensureResolvers,
resolveRecipient,
@@ -70,7 +71,7 @@
type Recipient
} from '$lib/api/endpoints/recipients';
import type {
AdminUserSummary,
FullUser,
Drive,
DriveMember,
DrivePolicies,
@@ -156,11 +157,11 @@
deleteUserModal !== null &&
deleteUserEmailInput.trim().toLowerCase() === deleteUserModal.email.toLowerCase()
);
function openDeleteUser(u: AdminUserSummary) {
function openDeleteUser(u: FullUser) {
deleteUserModal = {
userId: u.id,
username: u.username || u.email,
email: u.email
userId: u.user.id,
username: u.user.username || u.user.email,
email: u.user.email
};
deleteUserEmailInput = '';
}
@@ -817,7 +818,7 @@
}
// Users
let users = $state<AdminUserSummary[]>([]);
let users = $state<FullUser[]>([]);
let total = $state(0);
let pageIndex = $state(0);
let usersError = $state<string | null>(null);
@@ -923,6 +924,12 @@
const page = await listUsers(PAGE_SIZE, pageIndex * PAGE_SIZE);
users = page.users;
total = page.total;
// Seed the per-user resolver cache with the row's `PublicUser`
// slice so every `UserVignette` mounted per row hits the cache
// synchronously — no per-row `/api/users/{id}` follow-up.
// Kills the N+1 that motivated widening `/api/admin/users` to
// carry the avatar (docs/plan/userdto-refactor.md § N+1).
for (const row of page.users) seedUser(row.user);
} catch (e) {
usersError = errorMessage(e);
}
@@ -1003,49 +1010,49 @@
}
/** True for the signed-in admin's own row — guards self-destructive actions. */
function isSelf(u: AdminUserSummary): boolean {
return u.id === currentAdminId;
function isSelf(u: FullUser): boolean {
return u.user.id === currentAdminId;
}
/** OIDC/SSO-provisioned account (no local password to reset). */
function isOidcUser(u: AdminUserSummary): boolean {
function isOidcUser(u: FullUser): boolean {
return u.federation_kind === 'oidc';
}
/** Used-quota percentage (0 when unlimited) for the per-user progress bar. */
function quotaPct(u: AdminUserSummary): number {
function quotaPct(u: FullUser): number {
return u.storage_quota_bytes > 0 ? (u.storage_used_bytes / u.storage_quota_bytes) * 100 : 0;
}
async function toggleRole(u: AdminUserSummary) {
async function toggleRole(u: FullUser) {
if (isSelf(u)) return;
const role = u.role === 'admin' ? 'user' : 'admin';
const role = u.user.role === 'admin' ? 'user' : 'admin';
if (!(await showConfirm(t('admin.confirm_role', { role }, 'Change role to {{role}}?')))) return;
try {
await setUserRole(u.id, role);
await setUserRole(u.user.id, role);
await loadUsers();
} catch (e) {
reportError(e);
}
}
async function toggleActive(u: AdminUserSummary) {
async function toggleActive(u: FullUser) {
if (isSelf(u) && u.active) return;
const msg = u.active
? t('admin.confirm_deactivate', 'Deactivate this user?')
: t('admin.confirm_activate', 'Activate this user?');
if (!(await showConfirm(msg))) return;
try {
await setUserActive(u.id, !u.active);
await setUserActive(u.user.id, !u.active);
await loadUsers();
} catch (e) {
reportError(e);
}
}
function openQuota(u: AdminUserSummary) {
function openQuota(u: FullUser) {
quotaModalError = null;
quotaModal = {
userId: u.id,
username: u.username || u.email,
userId: u.user.id,
username: u.user.username || u.user.email,
initialBytes: u.storage_quota_bytes
};
}
@@ -1069,8 +1076,8 @@
}
}
function openReset(u: AdminUserSummary) {
resetModal = { userId: u.id, username: u.username || u.email };
function openReset(u: FullUser) {
resetModal = { userId: u.user.id, username: u.user.username || u.user.email };
resetPassword = '';
resetError = null;
}
@@ -1094,7 +1101,7 @@
}
}
function removeUser(u: AdminUserSummary) {
function removeUser(u: FullUser) {
if (isSelf(u)) return;
openDeleteUser(u);
}
@@ -1103,20 +1110,20 @@
// provisions a home drive + flips the is_external flag; irreversible
// via the admin UI (there's no demote endpoint on purpose). Backend
// refuses when magic-link login is disabled — surfaced as a toast.
async function promoteExternal(u: AdminUserSummary) {
if (!u.is_external) return;
async function promoteExternal(u: FullUser) {
if (!u.user.is_external) return;
if (
!(await showConfirm(
t(
'admin.confirm_promote_user',
{ name: u.username || u.email },
{ name: u.user.username || u.user.email },
'Promote {{name}} to an internal user? This provisions a home drive and gives the account a normal storage envelope. The account keeps its identity; magic-link login stays the way in unless a password is set later.'
)
))
)
return;
try {
await promoteUserToInternal(u.id);
await promoteUserToInternal(u.user.id);
await loadUsers();
} catch (e) {
reportError(e);
@@ -1240,8 +1247,13 @@
.map(async (d) => {
const ownerMember = nextMembers[d.id]?.find((m) => m.subject.type === 'user');
if (!ownerMember) return;
const user = await getUserAdmin(ownerMember.subject.id);
if (user) nextOwners[d.id] = user;
// `getUserAdmin` returns `FullUser` (admin-visible extras
// + nested `.user: PublicUser`). The drive row only reads
// public-identity fields (username, email, image) so keep
// the map typed as `PublicUser` and unwrap the embedded
// public block on insert. See docs/plan/userdto-refactor.md.
const full = await getUserAdmin(ownerMember.subject.id);
if (full) nextOwners[d.id] = full.user;
})
);
personalDriveOwners = nextOwners;
@@ -1723,6 +1735,16 @@
{:else if !dashboard}
<p class="status">{t('common.loading', 'Loading…')}</p>
{:else}
<!-- Three grouped sections — one per data-nature axis. Static
row counts on top (change on register/deactivate/role toggle),
live-presence signals in the middle (change minute-to-minute,
visually distinguished with the presence dot), system flags at
the bottom (deployment posture, changes rarely). Splitting
what used to be a single 4-card row prevents admins from
misreading "as-of-now count" as "who's here right now". -->
<!-- Section 1: User accounts — static breakdown of auth.users -->
<h2 class="ds-section-title">{t('admin.section_accounts', 'User accounts')}</h2>
<div class="ds-grid">
<div class="ds-card">
<span class="ds-num">{dashboard.total_users}</span>{t('admin.total_users', 'Total users')}
@@ -1733,11 +1755,66 @@
<div class="ds-card">
<span class="ds-num">{dashboard.admin_users}</span>{t('admin.admin_users', 'Admins')}
</div>
<div class="ds-card">
<span class="ds-num">v{dashboard.server_version}</span>{t('admin.version', 'Version')}
<div
class="ds-card"
title={t(
'admin.external_users_tooltip',
'Grant-only accounts — magic-link, OIDC-only, OCM recipients'
)}
>
<span class="ds-num">{dashboard.external_users}</span>{t(
'admin.external_users',
'External'
)}
</div>
</div>
<!-- Section 2: Live activity — projection over auth.sessions.
Same 5-min window as the Prometheus `oxicloud_sessions_online`
gauges. The presence dot before each number signals "this
value changes minute-to-minute" — same green as the
admin > sessions row indicator so admins read one consistent
visual for presence across the panel. -->
<h2 class="ds-section-title">
{t('admin.section_activity', 'Live activity')}
<span
class="ds-section-live"
title={t('admin.live_tooltip', 'Reflects sessions active in the last 5 minutes')}
>
{t('admin.live', 'live')}
</span>
</h2>
<div class="ds-grid">
<div
class="ds-card"
title={t(
'admin.online_users_tooltip',
'Distinct users with a session active in the last 5 minutes'
)}
>
<span class="ds-num ds-num--live">
<span class="presence-dot presence-dot--online" aria-hidden="true"></span>
{dashboard.online_users}
</span>
{t('admin.online_users', 'Online users')}
</div>
<div
class="ds-card"
title={t(
'admin.online_sessions_tooltip',
'Non-revoked sessions active in the last 5 minutes — multi-device users contribute more than one'
)}
>
<span class="ds-num ds-num--live">
<span class="presence-dot presence-dot--online" aria-hidden="true"></span>
{dashboard.online_sessions}
</span>
{t('admin.online_sessions', 'Online sessions')}
</div>
</div>
<!-- Section 3: System — deployment flags + version. -->
<h2 class="ds-section-title">{t('admin.section_system', 'System')}</h2>
<div class="ds-grid">
<div class="ds-card">
<span class="ds-flag" class:ds-flag--on={dashboard.auth_enabled}>
@@ -1761,6 +1838,9 @@
</span>
{t('admin.quotas', 'Quotas')}
</div>
<div class="ds-card">
<span class="ds-num">v{dashboard.server_version}</span>{t('admin.version', 'Version')}
</div>
</div>
{#if dashboard.users_over_quota > 0}
@@ -1802,13 +1882,17 @@
row.kind === 'personal'
? t('admin.quota_personal', 'Personal drives')
: t('admin.quota_shared', 'Shared drives')}
{@const total = row.unlimited_count + row.capped_count}
{@const pct =
row.capped_quota_bytes && row.capped_quota_bytes > 0
? (row.used_bytes / row.capped_quota_bytes) * 100
: null}
{#if row.capped_count > 0 || row.unlimited_count > 0}
<tr>
<th scope="row">{label}</th>
<th scope="row">
<span class="quota-table__count">{total}</span>
{label}
</th>
<td class="quota-table__num">
{#if row.capped_quota_bytes !== null && pct !== null}
{formatBytes(row.used_bytes)} / {formatBytes(row.capped_quota_bytes)}
@@ -2680,15 +2764,15 @@
</tr>
</thead>
<tbody>
{#each users as u (u.id)}
{#each users as u (u.user.id)}
{@const pct = quotaPct(u)}
<tr>
<td>
<div class="user-vignette-cell">
<UserVignette
userId={u.id}
fallbackLabel={u.username || u.email}
fallbackSublabel={u.email}
userId={u.user.id}
fallbackLabel={u.user.username || u.user.email}
fallbackSublabel={u.user.email}
/>
{#if isSelf(u)}
<span class="badge badge--self">{t('admin.you_badge', 'you')}</span>
@@ -2703,11 +2787,11 @@
badge is `white-space: nowrap` so the badge label
itself never wraps mid-word either. -->
<div class="role-badges">
<span class="badge badge--{u.role === 'admin' ? 'admin' : 'user'}">
{#if u.role === 'admin'}<Icon name="shield-alt" />{/if}
{u.role}
<span class="badge badge--{u.user.role === 'admin' ? 'admin' : 'user'}">
{#if u.user.role === 'admin'}<Icon name="shield-alt" />{/if}
{u.user.role}
</span>
{#if u.is_external}
{#if u.user.is_external}
<!-- Origin flag, orthogonal to `role`. Grant-only
accounts (magic-link / OCM) can never be admin
(DB CHECK `users_external_not_admin`) so the two
@@ -2738,7 +2822,7 @@
<td class="auth-cell">
<!--
Auth-capability chip set — ADMIN-ONLY (fields
scoped to `AdminUserSummaryDto`; never on
scoped to `FullUserDto`; never on
`UserDto`). Any user carries ZERO OR MORE of:
* SSO/OIDC — `federation_kind === 'oidc'`,
identity delegated to the IdP; label is
@@ -2758,8 +2842,8 @@
-->
{#if isOidcUser(u)}
<span class="badge badge--oidc" title={u.federation_issuer}>
<Icon name="key" />
<span class="badge__label">{u.federation_issuer}</span>
<Icon name="shield-alt" />
<span class="badge__label">oidc</span>
</span>
{/if}
{#if u.has_password}
@@ -2825,7 +2909,7 @@
</span>
</td>
<td>
{#if u.is_external}
{#if u.user.is_external}
<!-- External accounts have no storage envelope by
design (DB CHECK `users_external_no_storage`
enforces storage_quota_bytes = 0). Rendering the
@@ -2867,10 +2951,10 @@
actions render as invisible placeholders. -->
<div class="actions actions--user">
<!-- Slot 1: quota (internal) OR promote (external). -->
{#if u.is_external}
{#if u.user.is_external}
<button
class="icon-btn icon-btn--success"
data-testid={`admin-user-promote-${u.id}`}
data-testid={`admin-user-promote-${u.user.id}`}
title={t('admin.promote_to_internal_title', 'Promote to internal user')}
aria-label={t('admin.promote_to_internal_title', 'Promote to internal user')}
onclick={() => promoteExternal(u)}
@@ -2880,7 +2964,7 @@
{:else}
<button
class="icon-btn"
data-testid={`admin-user-quota-${u.id}`}
data-testid={`admin-user-quota-${u.user.id}`}
title={t('admin.edit_quota_title', 'Edit quota')}
aria-label={t('admin.edit_quota_title', 'Edit quota')}
onclick={() => openQuota(u)}
@@ -2891,10 +2975,10 @@
<!-- Slot 2: reset password (local internal only —
OIDC and external accounts have no password
to reset). Placeholder otherwise. -->
{#if !isOidcUser(u) && !u.is_external}
{#if !isOidcUser(u) && !u.user.is_external}
<button
class="icon-btn"
data-testid={`admin-user-reset-password-${u.id}`}
data-testid={`admin-user-reset-password-${u.user.id}`}
title={t('admin.reset_password_title', 'Reset password')}
aria-label={t('admin.reset_password_title', 'Reset password')}
onclick={() => openReset(u)}
@@ -2909,16 +2993,16 @@
`change_user_role` + DB CHECK
`users_external_not_admin`). Promotion to
internal is offered separately in slot 1. -->
{#if !u.is_external}
{#if !u.user.is_external}
<button
class="icon-btn"
data-testid={`admin-user-toggle-role-${u.id}`}
data-testid={`admin-user-toggle-role-${u.user.id}`}
title={t('admin.toggle_role_title', 'Toggle admin role')}
aria-label={t('admin.toggle_role_title', 'Toggle admin role')}
disabled={isSelf(u)}
onclick={() => toggleRole(u)}
>
<Icon name={u.role === 'admin' ? 'user' : 'crown'} />
<Icon name={u.user.role === 'admin' ? 'user' : 'crown'} />
</button>
{:else}
<span class="icon-btn icon-btn--placeholder" aria-hidden="true"></span>
@@ -2926,7 +3010,7 @@
<!-- Slot 4: activate/deactivate. -->
<button
class="icon-btn {u.active ? 'icon-btn--danger' : 'icon-btn--success'}"
data-testid={`admin-user-toggle-active-${u.id}`}
data-testid={`admin-user-toggle-active-${u.user.id}`}
title={u.active
? t('admin.deactivate_title', 'Deactivate')
: t('admin.activate_title', 'Activate')}
@@ -2941,7 +3025,7 @@
<!-- Slot 5: delete. -->
<button
class="icon-btn icon-btn--danger"
data-testid={`admin-user-delete-${u.id}`}
data-testid={`admin-user-delete-${u.user.id}`}
title={t('admin.delete_title', 'Delete user')}
aria-label={t('admin.delete_title', 'Delete user')}
disabled={isSelf(u)}
@@ -3082,6 +3166,36 @@
{t('admin.sessions.expired', 'expired')}
</span>
{:else}
<!--
Presence dot — filled green when the server saw a
request in the last 5 min (backend `is_online`),
outlined grey otherwise. Only rendered on active
rows: a revoked-but-recently-seen row would otherwise
flash green post-revocation. Tooltip carries the
human "last seen X ago" so admins don't have to
hover-hunt for the exact timestamp — the
`last_seen_at` DateTime is available as
`title` for the details-on-demand case.
-->
<span
class="presence-dot"
class:presence-dot--online={s.is_online}
class:presence-dot--idle={!s.is_online}
title={s.is_online
? t(
'admin.sessions.presence_online_tooltip',
{ ago: timeAgo(s.last_seen_at) },
'Online — last seen {{ago}}'
)
: t(
'admin.sessions.presence_idle_tooltip',
{ ago: timeAgo(s.last_seen_at) },
'Idle — last seen {{ago}}'
)}
aria-label={s.is_online
? t('admin.sessions.online', 'online')
: t('admin.sessions.idle', 'idle')}
></span>
<span class="badge badge--active">
{t('admin.sessions.active', 'active')}
</span>
@@ -3245,11 +3359,21 @@
fall back to the owner's cap; 0 also means "no limit"
(backend convention — see `User.storage_quota_bytes` doc).
-->
<!-- Personal-drive fallback used to read
`owner.storage_quota_bytes` off the resolved DTO.
Post the UserDto refactor
(docs/plan/userdto-refactor.md) `owner` here is a
`PublicUser` (public identity, no quota); the
envelope quota only lives on `FullUser` /
`SelfUser`. Rather than widen the resolver's shape
just for this fallback, hold the effective quota at
`null` when the drive itself doesn't declare one —
the row renders "—" and the admin can consult the
user's row for their envelope cap. Explicit
shared-drive quota still surfaces as before. -->
{@const effectiveQuota =
d.kind === 'personal'
? owner && owner.storage_quota_bytes > 0
? owner.storage_quota_bytes
: null
? null
: d.quota_bytes && d.quota_bytes > 0
? d.quota_bytes
: null}
@@ -4293,6 +4417,40 @@
margin-bottom: var(--space-4);
}
/* Section title bar above each dashboard grid — labels the
nature of the cards below (accounts vs live activity vs
system). Small, muted, so it structures the page without
competing with the numbers. `text-transform: uppercase` +
`letter-spacing` matches the small-caps section-header pattern
used elsewhere in the admin surface. */
.ds-section-title {
display: flex;
align-items: baseline;
gap: var(--space-2);
margin: var(--space-4) 0 var(--space-2) 0;
font-size: var(--text-xs);
font-weight: var(--weight-semibold);
color: var(--color-text-muted);
text-transform: uppercase;
letter-spacing: 0.06em;
}
/* "live" pill next to the "Live activity" section header —
subtle visual hint that the values in this grid change on
their own cadence. Matches the presence-dot's success token
so the whole live-activity block reads as one visual family. */
.ds-section-live {
display: inline-block;
padding: 0 var(--space-2);
border-radius: var(--radius-full);
background: var(--color-success-bg);
color: var(--color-success-text);
font-size: 0.65rem;
font-weight: var(--weight-bold);
letter-spacing: 0.08em;
vertical-align: middle;
}
.ds-card {
display: flex;
flex-direction: column;
@@ -4311,6 +4469,18 @@
color: var(--color-text-heading);
}
/* Live-count variant — same font size as `.ds-num`, plus a
flex container so the leading presence dot aligns with the
number baseline instead of the top of the digit. Reuses the
`.presence-dot--online` class from the sessions-panel work
so the visual signal for presence is identical across the
admin surface. */
.ds-num.ds-num--live {
display: inline-flex;
align-items: center;
gap: var(--space-2);
}
.ds-bar {
height: 8px;
background: var(--color-bg-muted);
@@ -4403,6 +4573,19 @@
white-space: nowrap;
}
/* Prepended drive count: tabular-nums so single/double/triple digits align
vertically across rows; right-aligned inside a fixed-width box so the
ones-digits line up across "personal" and "shared" rows regardless of
how many digits each count has. */
.quota-table__count {
display: inline-block;
min-width: 1.5em;
margin-right: 0.25em;
text-align: right;
font-variant-numeric: tabular-nums;
color: var(--color-text-heading);
}
.quota-table__num {
font-variant-numeric: tabular-nums;
white-space: nowrap;
@@ -4572,6 +4755,46 @@
text-transform: uppercase;
}
/* Presence dot in the sessions-table Status column — filled green
when the row is `is_online` (a request landed in the last 5 min),
outlined grey when the row is active-but-idle. Only rendered on
active rows: a revoked-but-recently-seen row must never flash
green post-revocation (see the markup guard `{#if s.is_active}`).
The dot sits BEFORE the `active` badge with a small gap, so the
Status cell reads left-to-right as `● active` when online and
`○ active` when idle.
Tokens: `--color-success-alt` / `--color-success-border` for the
filled fill is the same green used by `.badge--active`, keeping
the presence signal visually consistent with the lifecycle one
without stealing the badge's own colour treatment. Grey border
for the idle state uses the neutral `--color-border` token so
both themes (light + dark, driven by `light-dark(...)`) get a
readable contrast. Fixed 8px / 8px sizing — the dot is a signal,
not a click target, so relative units would over-scale on
larger UI densities. */
.presence-dot {
display: inline-block;
width: 8px;
height: 8px;
border-radius: 50%;
margin-right: var(--space-1);
vertical-align: middle;
/* No border on the filled state, so both variants render at
the same 8×8 footprint (the outlined variant's 1px border
is inset via box-sizing: border-box below). */
box-sizing: border-box;
}
.presence-dot--online {
background: var(--color-success-alt);
}
.presence-dot--idle {
background: transparent;
border: 1px solid var(--color-border);
}
/* External / grant-only account marker. Sibling of `.badge--user`
in the same cell so the two stack horizontally; the accent
colour reuses `--color-warning-*` because "external" is the
@@ -5143,9 +5366,17 @@
}
.admin {
max-width: 64rem;
/* Raised from 64rem to 80rem so the data-dense tables (sessions
row with 9+ cells, users table with vignette + role + auth
chips + quota bar) have more horizontal room. At viewports
above 80rem `margin: 0 auto` still centers with the leftover
whitespace — DevTools shows that whitespace as horizontal
margin (not padding) and is what "content looks squeezed"
really means on wide displays. Vertical rhythm and the small
horizontal padding are unchanged. */
max-width: 80rem;
margin: 0 auto;
padding: 1.5rem 1rem;
padding: 1.5rem var(--space-2);
display: flex;
flex-direction: column;
gap: 1rem;
frontend/src/routes/admin/[[tab]]/page.test.ts
+11 -1
View File
@@ -96,16 +96,26 @@ const dashboard = {
users_over_quota: 0
};
// FullUser fixture — post the three-layer UserDto refactor
// (docs/plan/userdto-refactor.md), /api/admin/users returns
// `Vec<FullUserDto>` where public identity nests under `.user`
// and admin-visible extras (quotas, active, has_password, OPAQUE
// flags) live at the top level.
const user = {
user: {
id: 'u1',
username: 'bob',
email: 'bob@x.test',
role: 'user',
is_external: false
},
active: true,
is_active: true,
storage_used_bytes: 10,
storage_quota_bytes: 100,
is_external: false
has_password: true,
opaque_registered: false,
opaque_migrated: false
};
const mount = {
frontend/src/routes/files/[...path]/+page.svelte
+36
View File
@@ -713,8 +713,43 @@
* Upload a batch of files into the current folder, reporting aggregate
* progress through a single bell notification with a progress bar.
*/
/**
* Cold-navigation upload guard.
*
* `currentId` starts `null` and is only populated inside `load()` AFTER
* `session.loadHomeFolder()` resolves (see the `$effect` at the bottom of
* this file that drives `load()`, and the assignment at `currentId =
* folderId` inside `load()`). The hidden `<input data-testid=
* "files-upload-file-input">` is unconditional in the template, so it's
* in the DOM the moment the page shell mounts — before `load()` has
* awaited its first HTTP round-trip.
*
* On a slow network / cold page / Playwright cold `page.goto` immediately
* followed by `setInputFiles`, `onchange` can fire while `currentId` is
* still `null`. Without this guard, `uploadBatch` / `uploadTree` post
* with `folderId: null` and the file silently lands in the caller's
* home root instead of the intended folder — a real user hitting Ctrl+U
* or dropping a file within ~100 ms of navigation hits the same window.
*
* The e2e reproduction: `tests/e2e/spa/files.spec.ts::"upload a file via
* the hidden file input"` flakes on CI where the mount→load round-trip
* outruns Playwright's file-input dispatch.
*
* Returns `true` when it's safe to proceed; `false` + a user-visible
* toast when the folder isn't ready.
*/
function guardUploadFolderReady(): boolean {
if (currentId !== null) return true;
ui.notify(
t('files.upload_folder_not_ready', 'Folder is still loading — please try again in a moment.'),
'warning'
);
return false;
}
async function uploadBatch(files: File[]) {
if (files.length === 0) return;
if (!guardUploadFolderReady()) return;
uploading = true;
// Arm the reload-guard + persist a "batch in flight" marker so a
// page refresh mid-upload (a) prompts the browser's "Leave site?"
@@ -1557,6 +1592,7 @@
*/
async function uploadTree(entries: { file: File; relativePath: string }[]) {
if (entries.length === 0) return;
if (!guardUploadFolderReady()) return;
uploading = true;
// Same reload-guard + interrupted-uploads breadcrumb as uploadBatch —
// the browser prompts on refresh, and if the user reloads anyway
frontend/src/routes/files/page.test.ts
+6
View File
@@ -158,6 +158,12 @@ it('keeps aggregate upload progress exact when one file restarts', async () => {
);
render(FilesPage);
const input = await screen.findByTestId('files-upload-file-input');
// The cold-navigation upload guard (`guardUploadFolderReady` in
// `+page.svelte`) refuses uploads while `currentId` is null — which is
// the initial state before `load()` runs. `load()` sets `currentId =
// folderId` BEFORE it calls `fetchFolderPage`, so waiting on the fetch
// mock is a stable "load() has progressed past the assignment" signal.
await waitFor(() => expect(fetchFolderPage).toHaveBeenCalled());
const uploads = [new File(['a'], 'a.txt'), new File(['b'], 'b.txt')];
Object.defineProperty(input, 'files', { configurable: true, value: uploads });
frontend/src/routes/profile/+page.svelte
+47 -37
View File
@@ -72,11 +72,11 @@
let creatingPw = $state(false);
let autoExpanded = $state(false);
const isOidc = $derived(session.user?.federation_kind === 'oidc');
const isLocal = $derived(!session.user?.federation_kind);
const isOidc = $derived(session.me?.full.federation_kind === 'oidc');
const isLocal = $derived(!session.me?.full.federation_kind);
const usernameClaimed = $derived(!!session.user?.username);
const isAdmin = $derived(session.user?.role === 'admin');
const canEditImage = $derived(session.user?.can_edit_image === true && isLocal);
const canEditImage = $derived(session.me?.can_edit_image === true && isLocal);
// Show the change-password card when the user CAN change their
// local password: they have `password_hash` on file AND the
// deployment offers password login (backend `change_password`
@@ -87,14 +87,16 @@
// password) are a legitimate posture and MUST be able to rotate
// their local credential; the new gate lets them, and the backend
// refusal covers the pure-SSO case where has_password is false.
const showPasswordCard = $derived((session.user?.has_password ?? false) && passwordLoginEnabled);
const showPasswordCard = $derived(
(session.me?.full.has_password ?? false) && passwordLoginEnabled
);
// SSO card gates — see docs/plan/oidc-account-linking.md.
// Connect: only when OIDC is enabled AND the user isn't already linked.
// Disconnect: only when currently OIDC-linked AND the user has an
// alternative auth method (password or OPAQUE-registered) — else
// unlinking would lock them out.
const canConnectSso = $derived(oidcEnabled && !session.user?.federation_kind);
const canConnectSso = $derived(oidcEnabled && !session.me?.full.federation_kind);
// Show the disconnect button whenever the user is OIDC-linked.
// The backend guard (`AuthApplicationService::unlink_oidc`) is the
// source of truth for the "no alternative auth" refusal — it also
@@ -103,7 +105,7 @@
// adoption status through user-directory endpoints). The UI shows
// the button unconditionally and surfaces the backend's 403 as a
// user-facing "set a password first" prompt.
const canDisconnectSso = $derived(session.user?.federation_kind === 'oidc');
const canDisconnectSso = $derived(session.me?.full.federation_kind === 'oidc');
/**
* Mandatory change-password mode. TRUE when the backend has
@@ -136,14 +138,11 @@
}
});
const storagePct = $derived(
session.user && session.user.storage_quota_bytes > 0
? Math.min(
100,
Math.round((session.user.storage_used_bytes / session.user.storage_quota_bytes) * 100)
)
: 0
);
const storagePct = $derived.by(() => {
const full = session.me?.full;
if (!full || full.storage_quota_bytes <= 0) return 0;
return Math.min(100, Math.round((full.storage_used_bytes / full.storage_quota_bytes) * 100));
});
const storageBarClass = $derived(
storagePct > 90 ? 'bar__fill--red' : storagePct > 70 ? 'bar__fill--orange' : 'bar__fill--green'
);
@@ -159,15 +158,20 @@
relativeTimeAgo(value, { empty: t('profile.never', 'Never'), invalidAsString: true });
function hydrate() {
const u = session.user;
if (!u) return;
givenName = u.given_name ?? '';
familyName = u.family_name ?? '';
username = u.username ?? '';
preferredLocale = u.preferred_locale ?? '';
notifyOnShare = u.notify_on_share;
const me = session.me;
if (!me) return;
// Public identity (name / handle) reads via `me.full.user`;
// admin-visible extras (preferred_locale) via `me.full`;
// self-only bag flags (notify_on_share) via `me` directly.
// The three-level indirection makes the audience of each
// field visible at the callsite (docs/plan/userdto-refactor.md).
givenName = me.full.user.given_name ?? '';
familyName = me.full.user.family_name ?? '';
username = me.full.user.username ?? '';
preferredLocale = me.full.preferred_locale ?? '';
notifyOnShare = me.notify_on_share;
// Source of truth is the preferences store, which itself
// derives from `session.user.ui_preferences`. Reading through
// derives from `session.me.ui_preferences`. Reading through
// the store here (rather than the raw bag) means a new
// preference field just needs a getter in the store and its
// own line here — no wire-format knowledge on the page.
@@ -176,21 +180,22 @@
async function saveProfile(e: SubmitEvent) {
e.preventDefault();
const u = session.user;
if (!u) return;
const me = session.me;
if (!me) return;
// Build a sparse patch of only the fields the user actually changed.
// Sending empty strings the user never touched would 400 on the server.
const patch: ProfilePatch = {};
if (!usernameClaimed && username.trim() && username.trim() !== (u.username ?? '')) {
if (!usernameClaimed && username.trim() && username.trim() !== (me.full.user.username ?? '')) {
patch.username = username.trim();
}
if (givenName.trim() !== (u.given_name ?? '')) patch.given_name = givenName.trim();
if (familyName.trim() !== (u.family_name ?? '')) patch.family_name = familyName.trim();
if ((preferredLocale || '') !== (u.preferred_locale ?? '')) {
if (givenName.trim() !== (me.full.user.given_name ?? '')) patch.given_name = givenName.trim();
if (familyName.trim() !== (me.full.user.family_name ?? ''))
patch.family_name = familyName.trim();
if ((preferredLocale || '') !== (me.full.preferred_locale ?? '')) {
patch.preferred_locale = preferredLocale || undefined;
}
if (notifyOnShare !== u.notify_on_share) patch.notify_on_share = notifyOnShare;
if (notifyOnShare !== me.notify_on_share) patch.notify_on_share = notifyOnShare;
// Ship the diff as a partial `ui_preferences` patch — the
// server does a shallow merge, so only the changed key is
// touched; siblings set on other devices survive.
@@ -205,8 +210,13 @@
savingProfile = true;
try {
// PATCH /me/profile echoes SelfUser (same shape as GET /me)
// so the SPA absorbs the just-written state in one round
// trip — no follow-up refresh needed. `session.user` is a
// derived accessor over `session.me.full.user`, so it
// updates in lockstep with the me assignment.
const updated = await updateProfile(patch);
session.user = updated;
session.me = updated;
if (patch.preferred_locale) await setLocale(patch.preferred_locale as Locale);
ui.notify(t('profile.saved', 'Profile saved'), 'success');
} catch (err) {
@@ -445,7 +455,7 @@
// (federation_kind should now be 'oidc').
try {
const me = await fetchMe();
if (me) session.user = me;
if (me) session.me = me;
} catch {
/* stale session is recoverable — next request refreshes */
}
@@ -536,7 +546,7 @@
try {
await unlinkOidc();
const me = await fetchMe();
if (me) session.user = me;
if (me) session.me = me;
ui.notify(t('profile.sso_unlinked_success', 'Single sign-on disconnected.'), 'info');
} catch (err) {
if (err instanceof ApiError && err.errorType === 'NoAlternativeAuth') {
@@ -742,7 +752,7 @@
<Icon name="clock" />
{t('profile.last_login', 'Last Login')}
</div>
<div class="info-value">{timeAgo(session.user.last_login_at)}</div>
<div class="info-value">{timeAgo(session.me?.full.last_login_at)}</div>
</div>
</div>
</div>
@@ -752,20 +762,20 @@
<h2><Icon name="hdd" /> {t('profile.storage', 'Storage')}</h2>
<div class="storage-stats">
<div class="storage-stat">
<div class="stat-value">{formatBytes(session.user.storage_used_bytes)}</div>
<div class="stat-value">{formatBytes(session.me?.full.storage_used_bytes ?? 0)}</div>
<div class="muted">{t('profile.used', 'Used')}</div>
</div>
<div class="storage-stat">
<div class="stat-value">
{session.user.storage_quota_bytes > 0
? formatBytes(session.user.storage_quota_bytes)
{(session.me?.full.storage_quota_bytes ?? 0) > 0
? formatBytes(session.me?.full.storage_quota_bytes ?? 0)
: '∞'}
</div>
<div class="muted">{t('profile.quota', 'Quota')}</div>
</div>
<div class="storage-stat">
<div class="stat-value">
{session.user.storage_quota_bytes > 0 ? `${storagePct}%` : '—'}
{(session.me?.full.storage_quota_bytes ?? 0) > 0 ? `${storagePct}%` : '—'}
</div>
<div class="muted">{t('profile.usage', 'Usage')}</div>
</div>
frontend/src/routes/profile/page.test.ts
+44 -19
View File
@@ -1,10 +1,15 @@
import { it, expect, vi, beforeEach } from 'vitest';
import { render, screen, fireEvent, waitFor } from '@testing-library/svelte';
const { session, ui } = vi.hoisted(() => ({
session: {
loaded: true,
load: vi.fn(),
// Test-double session store. Post the three-layer UserDto refactor
// (docs/plan/userdto-refactor.md), production `session.user` is a
// derived accessor over `session.me.full.user`. The stub here mirrors
// that shape: `me` carries the whole SelfUser tree, and `user` mirrors
// `me.full.user` so any legacy `session.user.foo` read on the tested
// page keeps working through the mock without reproducing the derived
// mechanism.
const buildSelfMe = () => ({
full: {
user: {
id: '1',
username: 'admin',
@@ -12,14 +17,41 @@ const { session, ui } = vi.hoisted(() => ({
given_name: 'A',
family_name: 'B',
role: 'admin',
is_external: false
},
storage_used_bytes: 100,
storage_quota_bytes: 1000,
is_external: false,
has_password: true
}
});
const { session, ui } = vi.hoisted(() => {
const me = {
full: {
user: {
id: '1',
username: 'admin',
email: 'a@x.test',
given_name: 'A',
family_name: 'B',
role: 'admin',
is_external: false
},
storage_used_bytes: 100,
storage_quota_bytes: 1000,
has_password: true
}
};
return {
session: {
loaded: true,
load: vi.fn(),
me,
user: me.full.user
},
ui: { notify: vi.fn() }
}));
};
});
vi.mock('$lib/stores/session.svelte', () => ({ session }));
vi.mock('$lib/stores/ui.svelte', () => ({ ui }));
vi.mock('$lib/stores/dialogs.svelte', () => ({ confirmDialog: vi.fn() }));
@@ -44,20 +76,13 @@ const m = (fn: unknown) => fn as ReturnType<typeof vi.fn>;
beforeEach(() => {
vi.clearAllMocks();
// Reset the shared session each test (handlers may mutate session.user).
// Reset the shared session each test (handlers may mutate session.me
// on save / refresh). `me` is the SelfUser tree; `user` mirrors
// `me.full.user` for legacy `session.user.foo` reads.
session.loaded = true;
session.user = {
id: '1',
username: 'admin',
email: 'a@x.test',
given_name: 'A',
family_name: 'B',
role: 'admin',
storage_used_bytes: 100,
storage_quota_bytes: 1000,
is_external: false,
has_password: true
};
const me = buildSelfMe();
session.me = me;
session.user = me.full.user;
m(profile.listAppPasswords).mockResolvedValue([]);
m(profile.updateProfile).mockResolvedValue(undefined);
m(getOidcProviders).mockResolvedValue({ password_login_enabled: true });
frontend/static/locales/ar.json
+3 -3
View File
@@ -1201,11 +1201,11 @@
"progress_scanned_only_tooltip": "لا يوجد إجمالي متاح لهذا التشغيل (نشر شريط التقدم المسبق أو عدم قيام المستأجر بالإبلاغ عن موضوع قابل للعد).",
"findings_present_tooltip": "قم بتوسيع هذا التشغيل لرؤية تفاصيل كل نتيجة.",
"col_error": "خطأ",
"col_kind": "عطوف",
"col_kind": "نوع",
"col_severity": "خطورة",
"col_resource": "الموارد",
"col_detail": "التفاصيل",
"run": "يجري",
"run": "تشغيل",
"cancel": "يلغي",
"refresh": "ينعش",
"runs_title": "أشواط الأخيرة",
@@ -1218,7 +1218,7 @@
"every_min": "كل دقيقة",
"every_sec": "كل ق",
"outcome_ok": "نعم",
"outcome_err": "يخطئ",
"outcome_err": "خطأ",
"outcome_issues": "مشاكل",
"outcome_notices": "إشعارات",
"n_findings": "‹النتائج",
frontend/static/locales/de.json
+4 -4
View File
@@ -1182,8 +1182,8 @@
"gen_key": "Schlüssel generieren",
"gen_key_warning": "Bewahren Sie diesen Schlüssel sicher auf. Bei Verlust sind die verschlüsselten Daten unwiederbringlich verloren.",
"jobs": {
"run_all_consistency": "Führen Sie alle Konsistenzprüfungen durch",
"run_deep": "Lauf tief",
"run_all_consistency": "Alle Konsistenzprüfungen ausführen",
"run_deep": "Tiefenprüfung",
"run_deep_hint": "Läuft auch langsame Varianten (Blob-Re-Hash, Bitrot-Erkennung).",
"col_name": "Name",
"col_cadence": "Kadenz",
@@ -1205,7 +1205,7 @@
"col_severity": "Schwere",
"col_resource": "Ressource",
"col_detail": "Detail",
"run": "Laufen",
"run": "Ausführen",
"cancel": "Stornieren",
"refresh": "Aktualisieren",
"runs_title": "Aktuelle Läufe",
@@ -1218,7 +1218,7 @@
"every_min": "alle {{n}} Min",
"every_sec": "alle {{n}} s",
"outcome_ok": "OK",
"outcome_err": "ähm",
"outcome_err": "err",
"outcome_issues": "Probleme",
"outcome_notices": "Hinweise",
"n_findings": "{{n}} Erkenntnisse",
frontend/static/locales/en.json
+30
View File
@@ -546,6 +546,7 @@
"empty_hidden_hint": "Files whose name starts with '.' are hidden. Toggle the setting to see them.",
"show_hidden": "Show hidden files",
"upload_dotfile_hidden": "{{n}} file(s) uploaded but hidden by your dotfile preference.",
"upload_folder_not_ready": "Folder is still loading — please try again in a moment.",
"rename_dotfile_hidden": "Renamed to '{{name}}' — now hidden by your preference.",
"new_folder_dotfile_hidden": "Created folder '{{name}}' — hidden by your dotfile preference.",
"dotfiles_hidden_toast": "Dotfiles hidden",
@@ -836,6 +837,17 @@
"total_users": "Total Users",
"active_users": "Active Users",
"admins": "Admins",
"external_users": "External",
"external_users_tooltip": "Grant-only accounts — magic-link, OIDC-only, OCM recipients",
"online_users": "Online users",
"online_users_tooltip": "Distinct users with a session active in the last 5 minutes",
"online_sessions": "Online sessions",
"online_sessions_tooltip": "Non-revoked sessions active in the last 5 minutes — multi-device users contribute more than one",
"section_accounts": "User accounts",
"section_activity": "Live activity",
"section_system": "System",
"live": "live",
"live_tooltip": "Reflects sessions active in the last 5 minutes",
"version": "Version",
"storage_overview": "Storage Overview",
"used": "Used",
@@ -1136,6 +1148,10 @@
"revoked": "revoked",
"expired": "expired",
"active": "active",
"online": "online",
"idle": "idle",
"presence_online_tooltip": "Online — last seen {{ago}}",
"presence_idle_tooltip": "Idle — last seen {{ago}}",
"revoke": "Revoke",
"empty": "No sessions match the current filter.",
"revoke_self_confirm": "⚠️ This is YOUR current session. Revoking it will log YOU out immediately and you'll have to sign back in. Continue?",
@@ -1259,6 +1275,20 @@
"run_all_consistency": "Run all consistency checks",
"run_deep": "Run deep",
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
"run_repair": "Repair ref_counts",
"run_repair_hint": "Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.",
"run_repair_confirm_title_scoped": "Run {{name}} in repair mode?",
"run_mutating_confirm_title": "Run {{name}}?",
"run_mutating_confirm_body": "This job changes stored state when it runs.",
"mutates_never": "read-only",
"mutates_on_repair_only": "read-only unless repaired",
"startup": "at boot",
"startup_repair": "at boot · repair",
"startup_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run at every boot.",
"startup_repair_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run in repair mode at every boot.",
"run_variants_menu": "Run variants menu",
"run_repair_confirm": "Repair",
"triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired",
"col_name": "Name",
"col_cadence": "Cadence",
"col_last_run": "Last run",
frontend/static/locales/es.json
+7 -7
View File
@@ -1183,15 +1183,15 @@
"time_just_now": "En este momento",
"unchanged": "Déjelo en blanco para mantenerse actualizado",
"jobs": {
"run_all_consistency": "Ejecute todas las comprobaciones de coherencia",
"run_deep": "Corre profundo",
"run_all_consistency": "Ejecutar todas las comprobaciones de coherencia",
"run_deep": "Análisis en profundidad",
"run_deep_hint": "También ejecuta variantes lentas (repetición de blobs, detección de bitrot).",
"col_name": "Nombre",
"col_cadence": "Cadencia",
"col_last_run": "última ejecución",
"col_outcome": "Resultado",
"col_state": "Estado",
"col_actions": "Comportamiento",
"col_actions": "Acciones",
"col_started_at": "Comenzó",
"col_status": "Estado",
"col_duration": "Duración",
@@ -1202,24 +1202,24 @@
"progress_scanned_only_tooltip": "No hay un total disponible para esta ejecución (implementación previa a la barra de progreso o el inquilino no informa un asunto contable).",
"findings_present_tooltip": "Amplíe esta ejecución para ver detalles por hallazgo.",
"col_error": "Error",
"col_kind": "Amable",
"col_kind": "Tipo",
"col_severity": "Gravedad",
"col_resource": "Recurso",
"col_detail": "Detalle",
"run": "Correr",
"run": "Ejecutar",
"cancel": "Cancelar",
"refresh": "Refrescar",
"runs_title": "Ejecuciones recientes",
"run_json": "Resumen de ejecución (JSON)",
"findings_title": "Recomendaciones",
"no_runs": "Aún no hay carreras.",
"no_runs": "Aún no hay ejecuciones.",
"no_findings": "No hay resultados: ejecución limpia.",
"on_demand": "Bajo demanda",
"every_h": "cada {{n}} horas",
"every_min": "cada {{n}} minutos",
"every_sec": "cada {{n}} s",
"outcome_ok": "OK",
"outcome_err": "errar",
"outcome_err": "err",
"outcome_issues": "asuntos",
"outcome_notices": "avisos",
"n_findings": "{{n}} hallazgos",
frontend/static/locales/fa.json
+3 -3
View File
@@ -1165,7 +1165,7 @@
"gen_key_warning": "این کلید را به صورت ایمن ذخیره کنید. اگر از بین برود، داده های رمزگذاری شده به طور غیرقابل جبرانی از بین می روند.",
"jobs": {
"run_all_consistency": "تمام بررسی های سازگاری را اجرا کنید",
"run_deep": "عمیق بدو",
"run_deep": "بررسی عمیق",
"run_deep_hint": "همچنین انواع آهسته را اجرا می کند (هش مجدد حباب، تشخیص بیتوت).",
"col_name": "نام",
"col_cadence": "آهنگ",
@@ -1182,7 +1182,7 @@
"progress_scanned_only_tooltip": "مجموع برای این اجرا موجود نیست (پیش از پیشرفت نوار مستقر شده یا مستاجر موضوع قابل شمارش را گزارش نمی کند).",
"findings_present_tooltip": "این اجرا را گسترش دهید تا جزئیات هر یافته را ببینید.",
"col_error": "خطا",
"col_kind": "مهربان",
"col_kind": "نوع",
"col_severity": "شدت",
"col_resource": "منبع",
"col_detail": "جزئیات",
@@ -1199,7 +1199,7 @@
"every_min": "هر {{n}} دقیقه",
"every_sec": "هر {{n}} ثانیه",
"outcome_ok": "باشه",
"outcome_err": "اشتباه کن",
"outcome_err": "خطا",
"outcome_issues": "مسائل",
"outcome_notices": "اطلاعیه ها",
"n_findings": "{{n}} یافته ها",
frontend/static/locales/fr.json
+33 -9
View File
@@ -459,6 +459,7 @@
"empty_hidden_hint": "Les fichiers dont le nom commence par '.' sont masqués. Modifiez le réglage pour les afficher.",
"show_hidden": "Afficher les fichiers masqués",
"upload_dotfile_hidden": "{{n}} fichier(s) téléversé(s) mais masqué(s) par votre préférence.",
"upload_folder_not_ready": "Le dossier est encore en cours de chargement — merci de réessayer dans un instant.",
"rename_dotfile_hidden": "Renommé en \"{{name}}\" — désormais masqué par votre préférence.",
"new_folder_dotfile_hidden": "Dossier \"{{name}}\" créé — masqué par votre préférence.",
"dotfiles_hidden_toast": "Fichiers masqués",
@@ -801,6 +802,17 @@
"total_users": "Utilisateurs totaux",
"active_users": "Utilisateurs actifs",
"admins": "Admins",
"external_users": "Externes",
"external_users_tooltip": "Comptes invités — magic-link, OIDC seulement, destinataires OCM",
"online_users": "Utilisateurs en ligne",
"online_users_tooltip": "Utilisateurs distincts ayant une session active dans les 5 dernières minutes",
"online_sessions": "Sessions en ligne",
"online_sessions_tooltip": "Sessions non révoquées actives dans les 5 dernières minutes — les utilisateurs multi-appareils en contribuent plusieurs",
"section_accounts": "Comptes utilisateurs",
"section_activity": "Activité en direct",
"section_system": "Système",
"live": "en direct",
"live_tooltip": "Reflète les sessions actives dans les 5 dernières minutes",
"version": "Version",
"storage_overview": "Aperçu du stockage",
"used": "Utilisé",
@@ -1191,8 +1203,16 @@
"gen_key_warning": "Conservez cette clé en toute sécurité. En cas de perte, les données cryptées sont irrémédiablement perdues.",
"jobs": {
"run_all_consistency": "Exécuter tous les contrôles de cohérence",
"run_deep": "Exécuter en profondeur",
"run_deep": "Analyse approfondie",
"run_deep_hint": "Exécute également des variantes lentes (re-hachage de blob, détection bitrot).",
"run_repair": "Réparer les compteurs",
"run_repair_hint": "Corrige les compteurs de références (blobs + manifestes) désynchronisés détectés par l'audit. Sûr pour les données — seuls les compteurs changent, pas le contenu.",
"run_repair_confirm_title": "Réparer les compteurs de références ?",
"run_repair_confirm_body": "Lance l'audit sur chaque blob et manifeste, puis applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu des blobs et les enregistrements de fichiers ne sont pas touchés. Vous pouvez exécuter cela à tout moment ; un passage en lecture seule s'exécute d'abord pour visualiser l'écart avant que la réparation ne l'écrase.",
"run_repair_confirm_body_scoped": "Lance {{name}} et applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu et les enregistrements de fichiers ne sont pas touchés.",
"run_variants_menu": "Menu des variantes d'exécution",
"run_repair_confirm": "Réparer",
"triggered_ok_repair": "{{name}} : {{n}} compteur(s) réparé(s)",
"col_name": "Nom",
"col_cadence": "Fréquence",
"col_last_run": "Dernière exécution",
@@ -1202,20 +1222,20 @@
"col_started_at": "Commencé",
"col_status": "Statut",
"col_duration": "Durée",
"col_scanned": "Numérisé",
"col_scanned": "Analysé",
"col_progress": "Progrès",
"col_findings": "Résultats",
"progress_scanned_only": "{{n}} scanné",
"progress_scanned_only_tooltip": "Aucun total disponible pour cette exécution (déploiement préalable de la barre de progression ou le locataire ne signale pas de sujet dénombrable).",
"findings_present_tooltip": "Développez cette analyse pour voir les détails par résultat.",
"col_error": "Erreur",
"col_kind": "Genre",
"col_kind": "Type",
"col_severity": "Gravité",
"col_resource": "Ressource",
"col_detail": "Détail",
"run": "Exécuter",
"cancel": "Annuler",
"refresh": "Rafraîchir",
"refresh": "Actualiser",
"runs_title": "Exécutions récentes",
"run_json": "Résumé de l'exécution (JSON)",
"findings_title": "Résultats",
@@ -1225,22 +1245,22 @@
"every_h": "toutes les {{n}} h",
"every_min": "toutes les {{n}} minutes",
"every_sec": "toutes les {{n}} s",
"outcome_ok": "d'accord",
"outcome_err": "se tromper",
"outcome_ok": "ok",
"outcome_err": "err",
"outcome_issues": "problèmes",
"outcome_notices": "avis",
"n_findings": "{{n}} conclusions",
"n_findings": "{{n}} résultats",
"n_notices": "{{n}} remarques",
"notices_present_tooltip": "Résultats informatifs – aucune action requise. Développez pour plus de détails.",
"purge": "Purger les anciennes exécutions",
"purge_hint": "Supprimez l’historique des exécutions terminées et échouées plus ancienne que la fenêtre de conservation choisie. Les résultats tombent avec leurs exécutions parentes. Les parcours non terminaux sont toujours préservés.",
"purge_hint": "Supprimer l’historique des exécutions terminées et échouées plus ancien que la fenêtre de conservation choisie. Les résultats sont supprimés avec leurs exécutions parentes. Les exécutions non terminales sont toujours préservées.",
"purge_body": "Supprimez l’historique des exécutions terminées et échouées plus ancienne que le nombre de jours choisi. Les résultats tombent avec leurs exécutions parentes. Les exécutions non terminales (en cours d’exécution, en pause, demandées en annulation) sont toujours conservées.",
"purge_days_label": "Rétention (jours)",
"purge_confirm": "Purger",
"purge_done": "{{n}} anciennes exécutions purgées (rétention {{days}} jours)",
"state_running": "en cours d'exécution",
"never": "jamais",
"just_now": "tout à l' heure",
"just_now": "à l'instant",
"n_min_ago": "il y a {{n}} min",
"n_h_ago": "il y a {{n}} h",
"n_d_ago": "il y a {{n}} j",
@@ -1277,6 +1297,10 @@
"revoked": "révoquée",
"expired": "expirée",
"active": "actif",
"online": "en ligne",
"idle": "inactif",
"presence_online_tooltip": "En ligne — vue {{ago}}",
"presence_idle_tooltip": "Inactif — vue {{ago}}",
"revoke": "Révoquer",
"empty": "Aucune session ne correspond au filtre actuel.",
"revoke_self_confirm": "⚠️ Il s'agit de VOTRE session actuelle. La révoquer vous déconnectera immédiatement et vous devrez vous reconnecter. Continuer ?",
frontend/static/locales/hi.json
+5 -5
View File
@@ -1183,11 +1183,11 @@
"gen_key_warning": "इस कुंजी को सुरक्षित रूप से संग्रहित करें. यदि यह खो जाता है, तो एन्क्रिप्टेड डेटा अपरिवर्तनीय रूप से खो जाता है।",
"jobs": {
"run_all_consistency": "सभी संगतता जांचें चलाएँ",
"run_deep": "गहरा रिश्ता",
"run_deep": "गहन जाँच",
"run_deep_hint": "धीमे वेरिएंट (ब्लॉब री-हैश, बिट्रोट डिटेक्शन) भी चलाता है।",
"col_name": "नाम",
"col_cadence": "ताल",
"col_last_run": "आखरी बार",
"col_last_run": "अंतिम रन",
"col_outcome": "नतीजा",
"col_state": "राज्य",
"col_actions": "कार्रवाई",
@@ -1201,11 +1201,11 @@
"progress_scanned_only_tooltip": "इस रन के लिए कोई कुल उपलब्ध नहीं है (पूर्व-प्रगति-बार परिनियोजन या किरायेदार एक गणनीय विषय की रिपोर्ट नहीं करता है)।",
"findings_present_tooltip": "प्रति-खोज विवरण देखने के लिए इस रन का विस्तार करें।",
"col_error": "गलती",
"col_kind": "दयालु",
"col_kind": "प्रकार",
"col_severity": "गंभीरता",
"col_resource": "संसाधन",
"col_detail": "विवरण",
"run": "दौड़ना",
"run": "चलाएँ",
"cancel": "रद्द करना",
"refresh": "ताज़ा करना",
"runs_title": "हालिया रन",
@@ -1218,7 +1218,7 @@
"every_min": "हर {{n}} मिनट",
"every_sec": "हर {{n}} एस",
"outcome_ok": "ठीक है",
"outcome_err": "ग़लती होना",
"outcome_err": "त्रुटि",
"outcome_issues": "समस्याएँ",
"outcome_notices": "नोटिस",
"n_findings": "{{n}} निष्कर्ष",
frontend/static/locales/it.json
+5 -5
View File
@@ -1183,11 +1183,11 @@
"gen_key_warning": "Conserva questa chiave in modo sicuro. In caso di smarrimento, i dati crittografati andranno persi irrimediabilmente.",
"jobs": {
"run_all_consistency": "Esegui tutti i controlli di coerenza",
"run_deep": "Corri in profondità",
"run_deep": "Analisi approfondita",
"run_deep_hint": "Esegue anche varianti lente (re-hash blob, rilevamento bitrot).",
"col_name": "Nome",
"col_cadence": "Cadenza",
"col_last_run": "Ultima corsa",
"col_last_run": "Ultima esecuzione",
"col_outcome": "Risultato",
"col_state": "Stato",
"col_actions": "Azioni",
@@ -1205,20 +1205,20 @@
"col_severity": "Gravità",
"col_resource": "Risorsa",
"col_detail": "Dettaglio",
"run": "Correre",
"run": "Esegui",
"cancel": "Cancellare",
"refresh": "Aggiorna",
"runs_title": "Esecuzioni recenti",
"run_json": "Riepilogo esecuzione (JSON)",
"findings_title": "Risultati",
"no_runs": "Nessuna corsa ancora.",
"no_runs": "Nessuna esecuzione ancora.",
"no_findings": "Nessun risultato: analisi pulita.",
"on_demand": "su richiesta",
"every_h": "ogni {{n}} h",
"every_min": "ogni {{n}} min",
"every_sec": "ogni {{n}} s",
"outcome_ok": "OK",
"outcome_err": "errare",
"outcome_err": "err",
"outcome_issues": "problemi",
"outcome_notices": "avvisi",
"n_findings": "{{n}} risultati",
frontend/static/locales/ja.json
+5 -5
View File
@@ -1183,7 +1183,7 @@
"gen_key_warning": "このキーは安全に保管してください。紛失すると、暗号化されたデータは回復不能に失われます。",
"jobs": {
"run_all_consistency": "すべての整合性チェックを実行する",
"run_deep": "深く走る",
"run_deep": "詳細スキャン",
"run_deep_hint": "低速な亜種 (BLOB 再ハッシュ、ビットロット検出) も実行します。",
"col_name": "名前",
"col_cadence": "ケイデンス",
@@ -1201,14 +1201,14 @@
"progress_scanned_only_tooltip": "この実行で利用できる合計はありません (進行状況バーのデプロイ前、またはテナントがカウント可能な件名を報告しない)。",
"findings_present_tooltip": "この実行を展開すると、結果ごとの詳細が表示されます。",
"col_error": "エラー",
"col_kind": "親切",
"col_kind": "種類",
"col_severity": "重大度",
"col_resource": "リソース",
"col_detail": "詳細",
"run": "走る",
"run": "実行",
"cancel": "キャンセル",
"refresh": "リフレッシュ",
"runs_title": "最近のランニング",
"runs_title": "最近の実行",
"run_json": "実行概要(JSON)",
"findings_title": "調査結果",
"no_runs": "まだ実行はありません。",
@@ -1217,7 +1217,7 @@
"every_h": "{{n}} 時間ごと",
"every_min": "{{n}} 分ごと",
"every_sec": "{{n}} 秒ごと",
"outcome_ok": "わかりました",
"outcome_ok": "OK",
"outcome_err": "エラー",
"outcome_issues": "問題",
"outcome_notices": "通知",
frontend/static/locales/ko.json
+6 -6
View File
@@ -1218,14 +1218,14 @@
"storage_backend_audit": "백엔드 일관성",
"jobs": {
"run_all_consistency": "모든 일관성 검사 실행",
"run_deep": "깊이 달리다",
"run_deep": "심층 스캔",
"run_deep_hint": "또한 느린 변형(블롭 재해시, 비트롯 감지)을 실행합니다.",
"col_name": "이름",
"col_cadence": "운율",
"col_last_run": "마지막 실행",
"col_outcome": "결과",
"col_state": "상태",
"col_actions": "행위",
"col_actions": "작업",
"col_started_at": "시작됨",
"col_status": "상태",
"col_duration": "지속",
@@ -1235,11 +1235,11 @@
"progress_scanned_only_tooltip": "이 실행에 사용할 수 있는 총계가 없습니다(사전 진행률 표시줄 배포 또는 테넌트가 셀 수 있는 주제를 보고하지 않음).",
"findings_present_tooltip": "이 실행을 확장하면 발견 항목별 세부 정보를 볼 수 있습니다.",
"col_error": "오류",
"col_kind": "친절한",
"col_kind": "종류",
"col_severity": "심각성",
"col_resource": "의지",
"col_detail": "세부 사항",
"run": "달리다",
"run": "실행",
"cancel": "취소",
"refresh": "새로 고치다",
"runs_title": "최근 실행",
@@ -1251,8 +1251,8 @@
"every_h": "매 {{n}}시간마다",
"every_min": "{{n}}분마다",
"every_sec": "{{n}}초마다",
"outcome_ok": "좋아요",
"outcome_err": "실수",
"outcome_ok": "OK",
"outcome_err": "오류",
"outcome_issues": "문제",
"outcome_notices": "공지사항",
"n_findings": "{{n}} 조사 결과",
frontend/static/locales/nl.json
+3 -3
View File
@@ -1183,7 +1183,7 @@
"gen_key_warning": "Bewaar deze sleutel veilig. Als het verloren gaat, zijn de gecodeerde gegevens onherstelbaar verloren.",
"jobs": {
"run_all_consistency": "Voer alle consistentiecontroles uit",
"run_deep": "Ren diep",
"run_deep": "Diepe scan",
"run_deep_hint": "Voert ook langzame varianten uit (blob re-hash, bitrot-detectie).",
"col_name": "Naam",
"col_cadence": "Cadans",
@@ -1201,11 +1201,11 @@
"progress_scanned_only_tooltip": "Er is geen totaal beschikbaar voor deze run (implementatie vóór de voortgangsbalk of de tenant rapporteert geen telbaar onderwerp).",
"findings_present_tooltip": "Vouw deze run uit om de details per vondst te bekijken.",
"col_error": "Fout",
"col_kind": "Vriendelijk",
"col_kind": "Soort",
"col_severity": "Ernst",
"col_resource": "Bron",
"col_detail": "Detail",
"run": "Loop",
"run": "Uitvoeren",
"cancel": "Annuleren",
"refresh": "Vernieuwen",
"runs_title": "Recente runs",
frontend/static/locales/pl.json
+6 -6
View File
@@ -1183,11 +1183,11 @@
"gen_key_warning": "Przechowuj ten klucz w bezpiecznym miejscu. W przypadku jego utraty zaszyfrowane dane zostaną utracone bezpowrotnie.",
"jobs": {
"run_all_consistency": "Uruchom wszystkie kontrole spójności",
"run_deep": "Biegnij głęboko",
"run_deep": "Głęboka analiza",
"run_deep_hint": "Uruchamia również powolne warianty (ponowne mieszanie obiektów blob, wykrywanie bitrot).",
"col_name": "Nazwa",
"col_cadence": "Rytm",
"col_last_run": "Ostatni bieg",
"col_last_run": "Ostatnie uruchomienie",
"col_outcome": "Wynik",
"col_state": "Państwo",
"col_actions": "Działania",
@@ -1201,24 +1201,24 @@
"progress_scanned_only_tooltip": "Brak sumy dostępnej dla tego przebiegu (wdrożenie przed paskiem postępu lub dzierżawca nie zgłasza przedmiotu, który można policzyć).",
"findings_present_tooltip": "Rozwiń ten przebieg, aby zobaczyć szczegóły dotyczące każdego znaleziska.",
"col_error": "Błąd",
"col_kind": "Uprzejmy",
"col_kind": "Rodzaj",
"col_severity": "Powaga",
"col_resource": "Ratunek",
"col_detail": "Szczegół",
"run": "Uruchomić",
"cancel": "Anulować",
"refresh": "Odświeżać",
"runs_title": "Ostatnie biegi",
"runs_title": "Ostatnie uruchomienia",
"run_json": "Podsumowanie uruchomienia (JSON)",
"findings_title": "Ustalenia",
"no_runs": "Nie ma jeszcze żadnych biegów.",
"no_runs": "Nie ma jeszcze żadnych uruchomień.",
"no_findings": "Brak wyników – czysty przebieg.",
"on_demand": "na żądanie",
"every_h": "co {{n}} godz",
"every_min": "co {{n}} min",
"every_sec": "co {{n}} s",
"outcome_ok": "OK",
"outcome_err": "błądzić",
"outcome_err": "błąd",
"outcome_issues": "kwestie",
"outcome_notices": "uwagi",
"n_findings": "{{n}} ustalenia",
frontend/static/locales/pt.json
+5 -5
View File
@@ -1182,8 +1182,8 @@
"gen_key": "Gerar chave",
"gen_key_warning": "Armazene esta chave com segurança. Se for perdido, os dados criptografados serão perdidos irrecuperavelmente.",
"jobs": {
"run_all_consistency": "Execute todas as verificações de consistência",
"run_deep": "Corra fundo",
"run_all_consistency": "Executar todas as verificações de consistência",
"run_deep": "Análise profunda",
"run_deep_hint": "Também executa variantes lentas (re-hash de blob, detecção de bitrot).",
"col_name": "Nome",
"col_cadence": "Cadência",
@@ -1205,20 +1205,20 @@
"col_severity": "Gravidade",
"col_resource": "Recurso",
"col_detail": "Detalhe",
"run": "Correr",
"run": "Executar",
"cancel": "Cancelar",
"refresh": "Atualizar",
"runs_title": "Execuções recentes",
"run_json": "Resumo da execução (JSON)",
"findings_title": "Descobertas",
"no_runs": "Ainda não há corridas.",
"no_runs": "Ainda não há execuções.",
"no_findings": "Nenhuma descoberta – execução limpa.",
"on_demand": "Sob demanda",
"every_h": "a cada {{n}} h",
"every_min": "a cada {{n}}min",
"every_sec": "cada {{n}} s",
"outcome_ok": "OK",
"outcome_err": "errar",
"outcome_err": "err",
"outcome_issues": "problemas",
"outcome_notices": "avisos",
"n_findings": "{{n}} descobertas",
frontend/static/locales/ru.json
+8 -8
View File
@@ -1182,8 +1182,8 @@
"gen_key": "Сгенерировать ключ",
"gen_key_warning": "Храните этот ключ в надежном месте. Если он утерян, зашифрованные данные теряются безвозвратно.",
"jobs": {
"run_all_consistency": "Запустите все проверки согласованности",
"run_deep": "Беги глубоко",
"run_all_consistency": "Запустить все проверки согласованности",
"run_deep": "Глубокая проверка",
"run_deep_hint": "Также выполняются медленные варианты (повторное хэширование больших двоичных объектов, обнаружение битротов).",
"col_name": "Имя",
"col_cadence": "Каденс",
@@ -1201,27 +1201,27 @@
"progress_scanned_only_tooltip": "Для этого запуска общая сумма недоступна (развертывание до индикатора выполнения или клиент не сообщает об подсчитываемой теме).",
"findings_present_tooltip": "Разверните этот прогон, чтобы просмотреть детали каждого результата.",
"col_error": "Ошибка",
"col_kind": "Добрый",
"col_kind": "Тип",
"col_severity": "Серьезность",
"col_resource": "Ресурс",
"col_detail": "Деталь",
"run": "Бегать",
"run": "Запустить",
"cancel": "Отмена",
"refresh": "Обновить",
"runs_title": "Недавние запуски",
"run_json": "Сводка выполнения (JSON)",
"findings_title": "Выводы",
"no_runs": "Пробегов пока нет.",
"no_runs": "Запусков пока нет.",
"no_findings": "Никаких результатов — чистый пробег.",
"on_demand": "по требованию",
"every_h": "каждые {{n}} ч",
"every_min": "каждые {{n}} мин.",
"every_sec": "каждые {{n}} с",
"outcome_ok": "хорошо",
"outcome_err": "ошибаться",
"outcome_ok": "ок",
"outcome_err": "ош",
"outcome_issues": "проблемы",
"outcome_notices": "уведомления",
"n_findings": "{{n}} выводы",
"n_findings": "{{n}} результатов",
"n_notices": "{{n}} уведомления",
"notices_present_tooltip": "Информационные выводы — никаких действий не требуется. Разверните для подробностей.",
"purge": "Очистка старых пробегов",
frontend/static/locales/zh-TW.json
+4 -4
View File
@@ -1165,7 +1165,7 @@
"gen_key_warning": "安全地保存此密鑰。如果遺失，加密資料將無法恢復。",
"jobs": {
"run_all_consistency": "執行所有一致性檢查",
"run_deep": "深入運行",
"run_deep": "深度掃描",
"run_deep_hint": "也運行緩慢的變體（blob 重新哈希、bitrot 檢測）。",
"col_name": "姓名",
"col_cadence": "節奏",
@@ -1187,10 +1187,10 @@
"col_severity": "嚴重性",
"col_resource": "資源",
"col_detail": "細節",
"run": "跑步",
"run": "運行",
"cancel": "取消",
"refresh": "重新整理",
"runs_title": "最近的跑步",
"runs_title": "最近的運行",
"run_json": "運行摘要 (JSON)",
"findings_title": "發現",
"no_runs": "還沒有運行。",
@@ -1200,7 +1200,7 @@
"every_min": "每 {{n}} 分鐘",
"every_sec": "每{{n}}秒",
"outcome_ok": "好的",
"outcome_err": "犯錯",
"outcome_err": "錯誤",
"outcome_issues": "問題",
"outcome_notices": "通知",
"n_findings": "{{n}} 研究結果",
frontend/static/locales/zh.json
+4 -4
View File
@@ -1165,7 +1165,7 @@
"gen_key_warning": "安全地保存此密钥。如果丢失，加密数据将无法恢复。",
"jobs": {
"run_all_consistency": "运行所有一致性检查",
"run_deep": "深入运行",
"run_deep": "深度扫描",
"run_deep_hint": "还运行缓慢的变体（blob 重新哈希、bitrot 检测）。",
"col_name": "姓名",
"col_cadence": "节奏",
@@ -1187,10 +1187,10 @@
"col_severity": "严重性",
"col_resource": "资源",
"col_detail": "细节",
"run": "跑步",
"run": "运行",
"cancel": "取消",
"refresh": "刷新",
"runs_title": "最近的跑步",
"runs_title": "最近的运行",
"run_json": "运行摘要 (JSON)",
"findings_title": "发现",
"no_runs": "还没有运行。",
@@ -1200,7 +1200,7 @@
"every_min": "每 {{n}} 分钟",
"every_sec": "每{{n}}秒",
"outcome_ok": "好的",
"outcome_err": "犯错",
"outcome_err": "错误",
"outcome_issues": "问题",
"outcome_notices": "通知",
"n_findings": "{{n}} 研究结果",
justfile
+17 -7
View File
@@ -192,7 +192,7 @@ audit:
cargo audit
openapi:
cargo run --bin generate-openapi
cargo run --features dev_tools --bin generate-openapi
db:
docker compose up -d postgres
@@ -252,10 +252,10 @@ front-design:
# real browser, which the curl-driven
# suite above can't observe.
#
# Same chain runs in CI under the `api-test` job in
# Same chain runs in CI under the `test-api` job in
# .github/workflows/ci.yml; keep the order in sync so a local pass means
# CI passes.
api-test:
test-api:
#!/usr/bin/env bash
set -x
set -euo pipefail
@@ -270,6 +270,12 @@ api-test:
echo "XXX litmus webdav not found, ignore test"
fi
# backward compat
api-test: test-api
test-bundle:
./tests/bundled-binary/run.sh
# CalDAV client-driven conformance suite.
#
# Drives OxiCloud through the maintained `python-caldav` client library
@@ -279,9 +285,9 @@ api-test:
# (RFC 5545 §3.8.4.4), and all-day masters (the shape #528 was filed
# against).
#
# Not chained into `api-test` because it needs python3; run explicitly.
# Not chained into `test-api` because it needs python3; run explicitly.
# The orchestrator spawns its own postgres + server on port 8091 so it
# can run in parallel with api-test/webdav.
# can run in parallel with test-api/webdav.
#
# Runs `cargo build` first so the orchestrator always sees a fresh
# binary. run-pycaldav.sh itself doesn't rebuild — it uses whatever
@@ -301,7 +307,7 @@ test-caldav:
# Manual, human-run: launches OxiCloud with OIDC as the ONLY login method
# (fake IdP on :1081, server on :8090) and waits for you to eyeball the
# /login auto-redirect in a real browser. Not part of `just api-test` —
# /login auto-redirect in a real browser. Not part of `just test-api` —
# there's no automated assertion here, it's a visual check. Ctrl-C to stop.
#oidc-manual-sso-only:
# bash tests/oidc/run-manual-sso-only.sh
@@ -377,6 +383,10 @@ load-baseline:
load-seed:
cargo run --bin load-seed -- --depth 5 --fanout 4 --files-per-leaf 3
# Unit-test the Docker publish tag-policy script (sub-second).
test-docker-tags:
@bash scripts/test-docker-publish-tags.sh
# Check and test everything
# recommanded before pull request
pre-pull-request: check fe-check audit check-migrations test test-integration fe-test build api-test fe-build-e2e front-test
pre-pull-request: test-docker-tags check fe-check audit check-migrations test test-integration fe-test build test-bundle test-api fe-build-e2e front-test
migrations/20261014000000_sessions_last_seen_at.sql
+28
View File
@@ -0,0 +1,28 @@
-- Session liveness tracking — per-request `last_seen_at` stamp on
-- `auth.sessions`, moved by the in-process `LastSeenTracker`
-- (see `src/application/services/last_seen_tracker.rs`) via a
-- batched UPDATE every 30 s.
--
-- Distinct from `created_at`: that column moves on session ROTATION
-- (every silent refresh), so its resolution is capped at the
-- access-token TTL (default 3600 s). `last_seen_at` moves on every
-- authenticated request, so the "active in the last N min" query
-- underlying `oxicloud_sessions_active` / `_active_users` gauges is
-- accurate to the flusher's 30 s cadence regardless of token TTL.
--
-- See `docs/plan/sessions.md` for the full design (why DashMap +
-- periodic flush, why partial index, why `NOW()` default).
ALTER TABLE auth.sessions
ADD COLUMN last_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
-- Partial index — the only reads on this column are the gauge
-- queries in `session_liveness_gauges.rs`, and they always filter
-- `revoked = FALSE`. Indexing only unrevoked rows keeps the write
-- cost of the 30 s batched UPDATE flat: rotated / revoked rows
-- fall out of the index automatically when `revoked` flips to TRUE
-- (partial-index maintenance drops them, no re-scan). A full
-- b-tree on the column would double index size for zero read
-- benefit — every gauge query would skip the revoked half anyway.
CREATE INDEX idx_sessions_last_seen_at ON auth.sessions(last_seen_at)
WHERE revoked = FALSE;
migrations/20261016000000_copy_folder_tree_manifest_refcount.sql
+211
View File
@@ -0,0 +1,211 @@
-- Fix: `storage.copy_folder_tree` never incremented `chunk_manifests.ref_count`.
--
-- The function bumped only `storage.blobs`:
--
-- UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
-- FROM (...) hc WHERE b.hash = hc.blob_hash;
--
-- but a CDC file's `blob_hash` names a MANIFEST, not a chunk. For any
-- multi-chunk file that predicate matches nothing, so a folder copy took
-- NO reference. Delete the original afterwards and `remove_reference`
-- walks the manifest to 0, `dedup_gc` reaps the manifest and every chunk
-- behind it — and the copy is unreadable. Silent data loss on an ordinary
-- UI operation.
--
-- Single-chunk files escaped by accident: their whole-file hash equals
-- their lone chunk's hash, so the UPDATE did match — bumping the wrong
-- counter, which shows up as a manifest under-count plus a blob
-- over-count rather than as loss.
--
-- Reproduced on a 5 MiB / 18-chunk file copied through the UI:
-- `chunk_manifests.ref_count` stayed at 1 while two `storage.files` rows
-- referenced it; `manifests_consistency` reported
-- `manifest_refcount_mismatch` with `delta: 1, reap_risk: true`.
--
-- This migration only rewrites the reference-counting block; everything
-- else is `20260902000001_copy_folder_tree_drop_user_id.sql` verbatim.
--
-- NOTE: existing drift is NOT repaired here. Run `manifests_consistency`
-- to find it — a data fix belongs with the recovery framework, not in a
-- schema migration that cannot know which counter is authoritative.
CREATE OR REPLACE FUNCTION storage.copy_folder_tree(
p_source_id UUID,
p_target_parent_id UUID, -- NULL = copy to root (keeps source drive)
p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name
) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$
DECLARE
v_root_lpath ltree;
v_root_depth INT;
v_max_depth INT;
v_level INT;
v_folders BIGINT := 0;
v_files BIGINT := 0;
v_inserted BIGINT;
v_new_root UUID;
v_dest_drive_id UUID;
BEGIN
-- Validate source exists.
SELECT fo.lpath, nlevel(fo.lpath)
INTO v_root_lpath, v_root_depth
FROM storage.folders fo
WHERE fo.id = p_source_id AND NOT fo.is_trashed;
IF v_root_lpath IS NULL THEN
RAISE EXCEPTION 'Source folder not found: %', p_source_id
USING ERRCODE = 'P0002'; -- no_data_found
END IF;
-- Resolve destination drive_id once up front (cross-drive copy path).
IF p_target_parent_id IS NULL THEN
SELECT fo.drive_id INTO v_dest_drive_id
FROM storage.folders fo
WHERE fo.id = p_source_id;
ELSE
SELECT fo.drive_id INTO v_dest_drive_id
FROM storage.folders fo
WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed;
IF v_dest_drive_id IS NULL THEN
RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id
USING ERRCODE = 'P0002';
END IF;
END IF;
-- Temp mapping: every folder in the subtree → new UUID.
CREATE TEMP TABLE IF NOT EXISTS _copy_map(
old_id UUID PRIMARY KEY,
new_id UUID NOT NULL DEFAULT gen_random_uuid()
) ON COMMIT DROP;
TRUNCATE _copy_map;
INSERT INTO _copy_map(old_id)
SELECT fo.id
FROM storage.folders fo
WHERE NOT fo.is_trashed
AND fo.lpath <@ v_root_lpath;
SELECT cm.new_id INTO v_new_root
FROM _copy_map cm WHERE cm.old_id = p_source_id;
SELECT MAX(nlevel(fo.lpath))
INTO v_max_depth
FROM storage.folders fo
JOIN _copy_map cm ON fo.id = cm.old_id;
-- ── Insert folders level by level ──
-- Post-D7: `user_id` intentionally omitted from the column list so
-- copied rows leave the (now-nullable) column NULL. Provenance is
-- carried by `created_by` / `updated_by` (§14 columns) — preserved
-- from source so authorship survives the copy.
FOR v_level IN v_root_depth .. v_max_depth LOOP
INSERT INTO storage.folders(
id, name, parent_id,
drive_id, created_by, updated_by
)
SELECT cm.new_id,
CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL
THEN p_dest_name ELSE fo.name END,
CASE WHEN fo.id = p_source_id THEN p_target_parent_id
ELSE pm.new_id END,
v_dest_drive_id,
fo.created_by,
fo.updated_by
FROM storage.folders fo
JOIN _copy_map cm ON fo.id = cm.old_id
LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id
WHERE NOT fo.is_trashed
AND nlevel(fo.lpath) = v_level;
GET DIAGNOSTICS v_inserted = ROW_COUNT;
v_folders := v_folders + v_inserted;
END LOOP;
-- Temp mapping for files src→dst (dst ids pre-allocated so we can
-- reference them in the dead-property duplication below).
CREATE TEMP TABLE IF NOT EXISTS _copy_file_map(
old_id UUID PRIMARY KEY,
new_id UUID NOT NULL DEFAULT gen_random_uuid()
) ON COMMIT DROP;
TRUNCATE _copy_file_map;
INSERT INTO _copy_file_map(old_id)
SELECT f.id
FROM storage.files f
JOIN _copy_map cm ON f.folder_id = cm.old_id
WHERE NOT f.is_trashed;
-- ── Batch copy all files (zero-copy: same blob_hash) ──
-- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`.
INSERT INTO storage.files(
id, name, folder_id, blob_hash, size, mime_type,
media_sort_date, drive_id, created_by, updated_by
)
SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size,
f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by,
f.updated_by
FROM storage.files f
JOIN _copy_map cm ON f.folder_id = cm.old_id
JOIN _copy_file_map fm ON fm.old_id = f.id
WHERE NOT f.is_trashed;
GET DIAGNOSTICS v_files = ROW_COUNT;
-- Batch increment reference counts — MANIFEST FIRST, blobs only as
-- fallback. This mirrors `DedupService::add_reference`, and the order
-- is the whole point:
--
-- A CDC file's `blob_hash` names a MANIFEST (`chunk_manifests.file_hash`),
-- not a chunk. The previous version of this block updated only
-- `storage.blobs`, so for a multi-chunk file the predicate
-- `b.hash = hc.blob_hash` matched ZERO rows and the copy took no
-- reference at all. Deleting the original then walked the manifest's
-- ref_count to 0, dedup_gc reaped the manifest and every chunk behind
-- it, and the copy became unreadable. Reproduced via the UI folder
-- copy on a 5 MiB (18-chunk) file: ref_count stayed 1 with two files
-- referencing it.
--
-- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing.
-- For a SINGLE-chunk file the whole-file hash equals its lone chunk's
-- hash, so without it the copy would be counted at both levels and
-- turn an under-count into an over-count.
IF v_files > 0 THEN
WITH hc AS (
SELECT f.blob_hash, COUNT(*)::int AS cnt
FROM storage.files f
JOIN _copy_map cm ON f.folder_id = cm.new_id
WHERE NOT f.is_trashed
GROUP BY f.blob_hash
),
bumped AS (
UPDATE storage.chunk_manifests m
SET ref_count = m.ref_count + hc.cnt
FROM hc
WHERE m.file_hash = hc.blob_hash
RETURNING m.file_hash
)
UPDATE storage.blobs b
SET ref_count = b.ref_count + hc.cnt,
-- Matches add_reference: a blob resurrected inside its GC
-- grace window must lose its orphan stamp.
orphaned_at = NULL
FROM hc
WHERE b.hash = hc.blob_hash
AND NOT EXISTS (SELECT 1 FROM bumped WHERE file_hash = hc.blob_hash);
END IF;
-- Duplicate dead properties per RFC 4918 §8.8 — id-keyed store.
INSERT INTO storage.webdav_dead_properties
(folder_id, namespace, local_name, value)
SELECT cm.new_id, dp.namespace, dp.local_name, dp.value
FROM storage.webdav_dead_properties dp
JOIN _copy_map cm ON dp.folder_id = cm.old_id;
INSERT INTO storage.webdav_dead_properties
(file_id, namespace, local_name, value)
SELECT fm.new_id, dp.namespace, dp.local_name, dp.value
FROM storage.webdav_dead_properties dp
JOIN _copy_file_map fm ON dp.file_id = fm.old_id;
RETURN QUERY SELECT v_new_root::text, v_folders, v_files;
END;
$$ LANGUAGE plpgsql;
migrations/20261017000000_file_delete_trigger_manifest_aware.sql
+108
View File
@@ -0,0 +1,108 @@
-- Fix: `trg_files_decrement_blob_ref` decremented the wrong counter for
-- CDC files.
--
-- The original trigger (2026-03-07 initial schema) unconditionally ran:
--
-- UPDATE storage.blobs
-- SET ref_count = GREATEST(ref_count - 1, 0)
-- WHERE hash = OLD.blob_hash;
--
-- That's correct for a legacy whole-file blob, where `OLD.blob_hash`
-- names a `storage.blobs` row directly. For a CDC file, `OLD.blob_hash`
-- names a `storage.chunk_manifests.file_hash` — the blob table row (if
-- one exists at all) holds a DIFFERENT counter, incremented by the
-- MANIFEST's presence in its own `chunk_hashes[]`, not by the file.
--
-- Consequences before this fix:
-- 1. `storage.chunk_manifests.ref_count` never decremented on file
-- DELETE → over-count grows unboundedly across delete/purge
-- cycles.
-- 2. `storage.blobs.ref_count` decremented for hashes it shouldn't
-- (CDC whole-file hashes) → the counter drops toward 0 while the
-- manifest still legitimately references the chunk. GC then reaps
-- a live blob → downloadable-then-404 data loss.
--
-- Both bugs surfaced by `tests/api/refcount_cascade.hurl` on the
-- 135-byte fixture (single-chunk CDC file, worst case for confusion
-- because the whole-file hash equals its lone chunk's hash). The
-- 2026-08-22 sandbox drift (`storage.blobs.ref_count = 0`,
-- `actual_auditor = 1`) is the same bug at rest.
--
-- Sibling fix: `20261016000000_copy_folder_tree_manifest_refcount.sql`
-- fixed the mirror-image INCREMENT bug in `storage.copy_folder_tree`.
-- This migration closes the decrement half.
--
-- Cross-references:
-- - `DedupService::add_reference` (dedup_service.rs:1703) — app-layer
-- twin for the increment direction: manifest first, blob fallback.
-- - `manifests_consistency` tenant (2026-08-23) — surfaces any
-- residual drift after this fix lands.
--
-- ── DESIGN NOTE — decrement only, no manifest reap here ──
--
-- The trigger DELIBERATELY does not delete manifests or walk chunks on
-- a last-ref decrement. Both actions used to live inside
-- `DedupService::cleanup_if_orphaned` and its callee
-- `remove_manifest_reference`, and both fire `fire_blob_hooks` —
-- the Rust callback that reaps disk artefacts keyed by the whole-file
-- content hash (thumbnails, face embeddings, audio tags, media
-- metadata). SQL triggers can't invoke Rust callbacks, so if this
-- trigger reaped the manifest itself, dedup_gc Phase 1
-- (`dedup_service.rs:2660-2772`) — the ONLY code path that knows to
-- fire `fire_blob_hooks` for a reaped manifest's `file_hash` — would
-- find nothing to do on its next sweep, and every derived artefact
-- would leak on disk. `storage_cleanup_check.sh`'s "N thumbnail
-- file(s) remain on disk" gate catches this class immediately.
--
-- Contract: trigger decrements the correct counter atomically inside
-- the DELETE txn. GC (`dedup_gc`) is responsible for:
-- • finding manifests whose ref_count hit 0 (or that no reference
-- source references, covering bulk-delete paths),
-- • deleting them,
-- • decrementing each chunk in `chunk_hashes[]`,
-- • firing `fire_blob_hooks(file_hash)` so Rust callbacks reap
-- derived disk artefacts,
-- • the corresponding legacy-blob path for ref_count = 0 blobs.
--
-- NOTE: pre-existing drift is NOT repaired here. Run `manifests_
-- consistency` + `blobs_consistency` after deploy; feed the findings
-- into the recovery framework.
CREATE OR REPLACE FUNCTION storage.decrement_blob_ref()
RETURNS trigger AS $$
BEGIN
-- Manifest-first, mirroring the increment side. We touch ONE
-- counter and return — the manifest reap + chunk walk + hook
-- firing lives in `dedup_gc` where Rust callbacks can run.
IF EXISTS (
SELECT 1 FROM storage.chunk_manifests
WHERE file_hash = OLD.blob_hash
) THEN
UPDATE storage.chunk_manifests
SET ref_count = GREATEST(ref_count - 1, 0)
WHERE file_hash = OLD.blob_hash;
ELSE
-- Legacy whole-file blob path: no manifest, blob is referenced
-- directly by this file row. Preserves the original behaviour
-- verbatim for the pre-CDC path.
UPDATE storage.blobs
SET ref_count = GREATEST(ref_count - 1, 0),
orphaned_at = CASE
WHEN GREATEST(ref_count - 1, 0) = 0
THEN now()
ELSE orphaned_at
END
WHERE hash = OLD.blob_hash;
END IF;
RETURN OLD;
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION storage.decrement_blob_ref() IS
'Decrement the correct ref_count when a file is deleted. '
'Manifest-aware (2026-10-17): dispatches to chunk_manifests.ref_count '
'when the file''s blob_hash names a manifest, else to '
'storage.blobs.ref_count for legacy whole-file blobs. Decrement-only: '
'physical cleanup + Rust lifecycle hooks fire from dedup_gc, which '
'can invoke callbacks a SQL trigger cannot.';
migrations/20261017000002_repair_existing_refcount_drift.sql
+162
View File
@@ -0,0 +1,162 @@
-- One-time repair of ref_count drift accumulated under the pre-fix
-- copy/delete code paths.
--
-- Why this is atomic with the upgrade rather than a manual admin action
-- ─────────────────────────────────────────────────────────────────────
-- The two prior migrations on this branch:
-- * `20261016000000_copy_folder_tree_manifest_refcount.sql`
-- (fix the INCREMENT path — copy was bumping the wrong counter for
-- CDC files)
-- * `20261017000000_file_delete_trigger_manifest_aware.sql`
-- (fix the DECREMENT path — trigger was decrementing the wrong
-- counter for CDC files; folder-cascade + trash-empty paths
-- inherited that drift silently)
-- both close the bugs going forward, but production DBs upgrading
-- through this branch may carry accumulated drift from every prior
-- copy → delete cycle a CDC file went through. Under-count is the
-- dangerous direction: the next `dedup_gc` pass would reap a live
-- blob → user-facing 404 → silent data loss.
--
-- Waiting for an operator to open the admin panel and click "Repair
-- ref_counts" is the wrong default for a data-loss-preventing fix.
-- Ed's rule (`[[feedback_no_silent_auto_repair]]`): consistency
-- tenants must default to discovery-only so future bugs surface — but
-- fixing KNOWN pre-existing drift on the upgrade itself is the
-- bounded exception, because at that specific moment the source of
-- drift is known + closed, and there is no upstream mystery to
-- preserve.
--
-- Content-safety guarantees:
-- * Only counter columns change (`storage.chunk_manifests.ref_count`,
-- `storage.blobs.ref_count`). No file rows, no blob rows, no
-- manifest rows, no chunk arrays, no backend files.
-- * The corrective UPDATE sets `stored = actual` where `actual` is
-- computed from the SAME auditor formulas that
-- `manifests_consistency` / `blobs_consistency` use, so this
-- migration and those tenants agree by construction.
-- * Race-safe against concurrent writes (migrations run
-- single-connection at startup before the server serves any
-- traffic; nobody else is writing).
-- * Idempotent — fresh installs and already-clean DBs no-op (both
-- `stored` and `actual` are equal, the `WHERE <>` filters
-- everything out).
--
-- The panel button + `?repair=true` on the trigger endpoints stay for
-- FUTURE drift (regression detector; not for repeat use on this
-- accumulated set).
--
-- ═══════════════════════════════════════════════════════════════════
-- Performance envelope (rewrite 2026-09-02)
-- ═══════════════════════════════════════════════════════════════════
-- Original implementation used correlated subqueries in both SET and
-- WHERE clauses — PG evaluates each subquery twice per row, and the
-- `b.hash = ANY(m.chunk_hashes)` scan is O(blobs × manifests) without
-- a GIN index. On a production customer with a large storage.blobs +
-- storage.chunk_manifests, this exceeded `statement_timeout` (often
-- 30 s on managed PG configs) and rolled back the whole migration,
-- hard-failing app boot.
--
-- Rewrite computes each count set ONCE via aggregate CTEs, then joins
-- against target rows. Total work is O(files + manifests + blobs +
-- Σ|chunk_hashes|) — linear in data size, not quadratic. Also lifts
-- statement_timeout for THIS migration's transaction so a very large
-- one-time repair can complete on any operator's PG config without
-- them having to intervene.
--
-- Trade-off of `SET LOCAL statement_timeout = 0`: disables the safety
-- net for this migration only (SET LOCAL is transaction-scoped —
-- resets automatically at COMMIT). Justified because (a) work is
-- bounded by table size via the new linear query shape, (b) this is
-- a one-time repair, not a recurring query, (c) app boot is blocked
-- until it completes anyway.
--
-- Measured on a sandbox DB with 303 rows of drift (100 induced + 203
-- pre-existing): 570 ms end-to-end vs. timeout in the original form.
-- Lift the timeout for this migration only. Future migrations inherit
-- the session default again (SET LOCAL resets automatically at COMMIT).
SET LOCAL statement_timeout = 0;
DO $$
DECLARE
v_m_fixed int;
v_b_fixed int;
BEGIN
-- Manifest counter: `actual` = # files whose blob_hash names this
-- manifest's file_hash. Same formula as
-- `manifests_consistency_service::manifest_page_sql` (via the
-- BlobReferenceRegistry at RefLevel::Manifest) — inline here
-- because migrations can't call Rust.
--
-- Structure: one GROUP BY over storage.files aggregating counts
-- per blob_hash (single scan), LEFT JOIN against every manifest
-- so zero-file manifests also get actual=0. UPDATE ... FROM
-- walks manifests once, writes only where stored <> actual.
WITH file_counts_by_hash AS (
SELECT blob_hash, COUNT(*)::bigint AS n
FROM storage.files
GROUP BY blob_hash
),
actual_per_manifest AS (
SELECT m.file_hash,
COALESCE(fc.n, 0) AS actual
FROM storage.chunk_manifests m
LEFT JOIN file_counts_by_hash fc ON fc.blob_hash = m.file_hash
)
UPDATE storage.chunk_manifests m
SET ref_count = a.actual
FROM actual_per_manifest a
WHERE a.file_hash = m.file_hash
AND m.ref_count <> a.actual;
GET DIAGNOSTICS v_m_fixed = ROW_COUNT;
-- Blob counter: two-term formula mirroring
-- `blobs_consistency_service.rs:395-408`:
-- (files pointing at this blob AND having NO manifest for their
-- blob_hash — legacy whole-file path)
-- + (manifests including this hash as a chunk in chunk_hashes[])
--
-- Structure: two aggregate CTEs (one per term), then LEFT JOINed
-- against every blob. `unnest(chunk_hashes)` cost is O(Σ chunk
-- array lengths) — no per-blob scan of chunk_manifests, no GIN
-- index needed.
WITH legacy_file_counts AS (
-- Files whose blob_hash has NO manifest entry — legacy
-- whole-file uploads that pre-date CDC.
SELECT f.blob_hash, COUNT(*)::bigint AS legacy_count
FROM storage.files f
WHERE NOT EXISTS (
SELECT 1 FROM storage.chunk_manifests m
WHERE m.file_hash = f.blob_hash
)
GROUP BY f.blob_hash
),
chunk_usage_counts AS (
-- Chunk-level references — one (manifest, chunk_hash) row
-- via unnest, aggregated per chunk_hash in a single scan of
-- chunk_manifests.
SELECT ch AS hash, COUNT(*)::bigint AS chunk_count
FROM storage.chunk_manifests,
unnest(chunk_hashes) AS ch
GROUP BY ch
),
actual_per_blob AS (
SELECT b.hash,
COALESCE(l.legacy_count, 0) + COALESCE(u.chunk_count, 0) AS actual
FROM storage.blobs b
LEFT JOIN legacy_file_counts l ON l.blob_hash = b.hash
LEFT JOIN chunk_usage_counts u ON u.hash = b.hash
)
UPDATE storage.blobs b
SET ref_count = a.actual
FROM actual_per_blob a
WHERE a.hash = b.hash
AND b.ref_count <> a.actual;
GET DIAGNOSTICS v_b_fixed = ROW_COUNT;
-- Landed in the deploy log so an operator upgrading a huge instance
-- can see the migration did work — silent no-op on fresh installs.
RAISE NOTICE '[refcount_repair] fixed % manifest(s), % blob(s)',
v_m_fixed, v_b_fixed;
END;
$$;
migrations/20261018000000_content_derived_blobs.sql
+69
View File
@@ -0,0 +1,69 @@
-- Derived content as blobs — tier-2 refactor, step 5.
-- See `docs/plan/derived-blobs.md`.
--
-- Maps a source Blob to the artifacts derived FROM it: thumbnails today,
-- transcodes next. Both the mapping key and the value are BLAKE3 hashes,
-- but they mean different things:
--
-- * `source_hash` — the Blob the artifact was derived from. A
-- *dependent* reference: it keeps nothing alive (the file does), and
-- when that Blob dies these rows are deleted with it.
-- * `blob_hash` — the derived Blob itself. A reference *holder*: it
-- bumps `chunk_manifests.ref_count`, which is why
-- `ContentDerivedReferenceSource` must be registered before the first
-- row is written, or `dedup_gc` reaps the content on its next sweep.
--
-- KEYING — the rule this table exists to enforce:
--
-- Bytes that are a pure deterministic function of the source content
-- belong here, content-keyed, and dedupe across every file holding
-- that content. Bytes that are user-supplied or user-chosen do NOT:
-- they must be file-keyed, because content-keying them lets one user's
-- upload be served for another user's identical file. Client-uploaded
-- previews (PDF page 1, video poster frames) are the live example and
-- belong in a separate file-keyed table.
--
-- `variant` is opaque text. New axes go INSIDE it, never into new
-- columns: 'preview-avif' beside 'preview', '720p-av1' beside '720p'.
-- That is what keeps this table from growing a column per rendering
-- parameter.
--
-- No FK on either hash column, for the reason
-- `20260701000000_content_search_index.sql` already documents: a hash
-- resolves to either `storage.blobs` (legacy whole blob) or
-- `storage.chunk_manifests` (CDC file hash), so the reference cannot be
-- expressed as a single FK. Orphans are reclaimed by GC and reported by
-- the consistency jobs instead.
--
-- No `size` column: the bytes are content-addressed, so their length is
-- an immutable fact the blob layer already owns via `blob_hash`.
-- `content_type` IS stored — the thumbnail handler byte-sniffs every
-- response today, and this retires that.
CREATE TABLE IF NOT EXISTS storage.content_derived_blobs (
source_hash VARCHAR(64) NOT NULL,
kind TEXT NOT NULL CHECK (kind IN ('thumbnail', 'transcode')),
variant TEXT NOT NULL,
blob_hash VARCHAR(64) NOT NULL,
content_type TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (source_hash, kind, variant)
);
-- Reverse lookup: "what still references this derived Blob?" — used by
-- the manifest-level refcount recompute in `manifests_consistency` and by
-- `dedup_gc`'s reap predicate.
CREATE INDEX IF NOT EXISTS idx_content_derived_blobs_blob_hash
ON storage.content_derived_blobs (blob_hash);
COMMENT ON TABLE storage.content_derived_blobs IS
'Server-derived artifacts (thumbnails, transcodes) keyed by the BLAKE3 of their SOURCE content. Content-keyed on purpose: identical content shares one derivation. User-supplied bytes must NOT be stored here — see docs/plan/derived-blobs.md.';
COMMENT ON COLUMN storage.content_derived_blobs.source_hash IS
'The Blob this was derived from. Dependent reference — holds no ref_count; rows are deleted when the source Blob is reaped.';
COMMENT ON COLUMN storage.content_derived_blobs.blob_hash IS
'The derived Blob. Reference HOLDER — bumps chunk_manifests.ref_count via DedupService::add_reference.';
COMMENT ON COLUMN storage.content_derived_blobs.variant IS
'Opaque rendering discriminator (icon | preview | large | 720p...). New axes go inside this string, never into new columns.';
migrations/20261019000000_copy_file_satellites.sql
+332
View File
@@ -0,0 +1,332 @@
-- Step 8 of `docs/plan/derived-blobs.md` — single-source the copy fan-out.
--
-- "What follows a file when the file is copied" was written twice: once in
-- the `copy_file` CTE (Rust, `file_blob_write_repository.rs`) and once in
-- `storage.copy_folder_tree`. They had already drifted — the tree path
-- bumped `storage.blobs` only, missing manifests entirely, which was silent
-- data loss on a multi-chunk file (fixed in `20261016000000`, and the fix
-- had to be written a second time rather than in one place).
--
-- The plan adds file-keyed satellite tables (`file_attached_blobs`, step 9).
-- Adding them against two copy sites means writing the same cascade a third
-- and fourth time, into sites that have already proven they drift. So the
-- fan-out gets exactly one home first.
--
-- Two functions land here:
--
-- * `storage.add_blob_references(TEXT[])` — the manifest-first reference
-- contract, expressed once for SQL callers. `DedupService::add_reference`
-- is the Rust twin; they must change together, which is why the shared
-- contract is spelled out in both doc comments.
--
-- * `storage.copy_file_satellites(UUID[], UUID[])` — everything that
-- follows a file on copy. The body IS the copy-semantics declaration:
-- what is absent is a documented decision (see the trailing comments),
-- not an omission someone has to notice.
--
-- Set-based rather than per-row on purpose. A per-row helper would have made
-- a 10k-file folder copy 10k function calls; taking arrays keeps the tree
-- path's single-statement cost while still having one implementation. The
-- single-file path passes one-element arrays.
-- ── The reference contract, for SQL callers ──────────────────────────────
--
-- Increment the reference count for each hash in `p_hashes`, counting
-- repeats (pass the hash once per referencing row). Returns the hashes that
-- matched NEITHER table, so callers can decide how loud to be — a copy
-- inherits a pre-existing breakage and should warn, whereas an ingest
-- referencing a nonexistent blob is a hard error.
--
-- MANIFEST FIRST, `storage.blobs` only as fallback. The order is the whole
-- point: a CDC file's `blob_hash` names a manifest
-- (`chunk_manifests.file_hash`), not a chunk, so bumping `storage.blobs`
-- first would match nothing for a multi-chunk file and take no reference at
-- all.
--
-- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing. For a
-- SINGLE-chunk file the whole-file hash EQUALS its lone chunk's hash (both
-- are BLAKE3 over the same bytes), so without the guard one reference would
-- be counted at both levels — turning an under-count into an over-count.
--
-- Mirrors `DedupService::add_reference`, including the asymmetry on
-- `orphaned_at`: only `storage.blobs` carries that column, so only the blobs
-- branch clears it. A chunk resurrected inside its GC grace window must lose
-- its orphan stamp or `dedup_gc` reaps live content.
CREATE OR REPLACE FUNCTION storage.add_blob_references(p_hashes TEXT[])
RETURNS TEXT[] AS $$
DECLARE
v_unmatched TEXT[];
BEGIN
IF p_hashes IS NULL OR cardinality(p_hashes) = 0 THEN
RETURN ARRAY[]::TEXT[];
END IF;
WITH hc AS (
SELECT h AS blob_hash, COUNT(*)::int AS cnt
FROM unnest(p_hashes) AS h
WHERE h IS NOT NULL
GROUP BY h
),
bumped_manifests AS (
UPDATE storage.chunk_manifests m
SET ref_count = m.ref_count + hc.cnt
FROM hc
WHERE m.file_hash = hc.blob_hash
RETURNING m.file_hash
),
bumped_blobs AS (
UPDATE storage.blobs b
SET ref_count = b.ref_count + hc.cnt,
orphaned_at = NULL
FROM hc
WHERE b.hash = hc.blob_hash
AND NOT EXISTS (
SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash
)
RETURNING b.hash
)
SELECT COALESCE(array_agg(hc.blob_hash), ARRAY[]::TEXT[])
INTO v_unmatched
FROM hc
WHERE NOT EXISTS (SELECT 1 FROM bumped_manifests WHERE file_hash = hc.blob_hash)
AND NOT EXISTS (SELECT 1 FROM bumped_blobs WHERE hash = hc.blob_hash);
RETURN v_unmatched;
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION storage.add_blob_references(TEXT[]) IS
'Manifest-first blob reference increment for SQL callers. Returns hashes '
'that matched no registry row. Rust twin: DedupService::add_reference — '
'change both together.';
-- ── What follows a file on copy ──────────────────────────────────────────
--
-- `p_old_ids[i]` is copied to `p_new_ids[i]`; the new `storage.files` rows
-- must already be inserted and visible (both callers insert in an earlier
-- statement of the same transaction).
--
-- Every satellite of a copied file belongs in this body. What is NOT here is
-- listed at the bottom, with the reason — the taxonomy is executable rather
-- than living in a document that drifts from the code.
CREATE OR REPLACE FUNCTION storage.copy_file_satellites(
p_old_ids UUID[],
p_new_ids UUID[]
) RETURNS void AS $$
DECLARE
v_unmatched TEXT[];
BEGIN
IF p_old_ids IS NULL OR cardinality(p_old_ids) = 0 THEN
RETURN;
END IF;
IF p_new_ids IS NULL OR cardinality(p_old_ids) <> cardinality(p_new_ids) THEN
-- Positional correspondence is the whole interface; a length
-- mismatch would silently attach satellites to the wrong file.
RAISE EXCEPTION
'copy_file_satellites: id arrays must correspond positionally (% old vs % new)',
cardinality(p_old_ids), COALESCE(cardinality(p_new_ids), 0);
END IF;
-- 1. WebDAV dead properties. RFC 4918 §8.8 requires COPY to duplicate
-- them: properties describe the resource, and the copy is a resource.
INSERT INTO storage.webdav_dead_properties
(file_id, namespace, local_name, value)
SELECT m.new_id, dp.namespace, dp.local_name, dp.value
FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id)
JOIN storage.webdav_dead_properties dp ON dp.file_id = m.old_id;
-- 2. A reference on the copied content, so deleting the original cannot
-- reap bytes the copy still needs. Read from the NEW rows rather than
-- the old ones: that is what makes an unreferenceable copy impossible
-- to create, since a row that failed to insert contributes nothing.
SELECT storage.add_blob_references(array_agg(f.blob_hash))
INTO v_unmatched
FROM unnest(p_new_ids) AS n(id)
JOIN storage.files f ON f.id = n.id
WHERE NOT f.is_trashed;
IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN
-- Warn, do not abort. A missing registry row means the SOURCE file
-- was already broken; the copy merely inherits it. Failing here
-- would abort an entire folder copy over one pre-existing fault,
-- which is worse than completing it and reporting. The blob-level
-- audit jobs are what surface the underlying breakage.
RAISE WARNING
'copy_file_satellites: % copied file(s) reference a blob with no registry row (first: %); source was already broken',
cardinality(v_unmatched), v_unmatched[1];
END IF;
-- ── Deliberately absent ──────────────────────────────────────────────
--
-- storage.comments (future): NOT copied. A copy is a new artifact; the
-- discussion belongs to the original.
--
-- storage.file_attached_blobs (step 9): WILL be copied here, with a
-- reference taken per attached blob_hash via add_blob_references.
--
-- content_derived_blobs, blob_extracted_text, faces.faces: content-keyed.
-- The copy shares the source's hash, so it already sees them — copying
-- would duplicate rows that are keyed on the very thing being shared.
--
-- storage.favorites, recent_items, shares: properties of the ORIGINAL's
-- relationship to users, not of its content.
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION storage.copy_file_satellites(UUID[], UUID[]) IS
'Single source of truth for what follows a file on copy. Both copy paths '
'(single-file and copy_folder_tree) call it. Adding a file-keyed satellite '
'table means editing this function, and only this function.';
-- ── Route copy_folder_tree through it ────────────────────────────────────
--
-- Only two blocks change versus `20261016000000`: the inline reference bump
-- and the per-file dead-property INSERT are both replaced by one
-- `copy_file_satellites` call. The folder dead-property INSERT stays inline
-- — folders are not files and have no satellite fan-out to share.
CREATE OR REPLACE FUNCTION storage.copy_folder_tree(
p_source_id UUID,
p_target_parent_id UUID, -- NULL = copy to root (keeps source drive)
p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name
) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$
DECLARE
v_root_lpath ltree;
v_root_depth INT;
v_max_depth INT;
v_level INT;
v_folders BIGINT := 0;
v_files BIGINT := 0;
v_inserted BIGINT;
v_new_root UUID;
v_dest_drive_id UUID;
BEGIN
-- Validate source exists.
SELECT fo.lpath, nlevel(fo.lpath)
INTO v_root_lpath, v_root_depth
FROM storage.folders fo
WHERE fo.id = p_source_id AND NOT fo.is_trashed;
IF v_root_lpath IS NULL THEN
RAISE EXCEPTION 'Source folder not found: %', p_source_id
USING ERRCODE = 'P0002'; -- no_data_found
END IF;
-- Resolve destination drive_id once up front (cross-drive copy path).
IF p_target_parent_id IS NULL THEN
SELECT fo.drive_id INTO v_dest_drive_id
FROM storage.folders fo
WHERE fo.id = p_source_id;
ELSE
SELECT fo.drive_id INTO v_dest_drive_id
FROM storage.folders fo
WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed;
IF v_dest_drive_id IS NULL THEN
RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id
USING ERRCODE = 'P0002';
END IF;
END IF;
-- Temp mapping: every folder in the subtree → new UUID.
CREATE TEMP TABLE IF NOT EXISTS _copy_map(
old_id UUID PRIMARY KEY,
new_id UUID NOT NULL DEFAULT gen_random_uuid()
) ON COMMIT DROP;
TRUNCATE _copy_map;
INSERT INTO _copy_map(old_id)
SELECT fo.id
FROM storage.folders fo
WHERE NOT fo.is_trashed
AND fo.lpath <@ v_root_lpath;
SELECT cm.new_id INTO v_new_root
FROM _copy_map cm WHERE cm.old_id = p_source_id;
SELECT MAX(nlevel(fo.lpath))
INTO v_max_depth
FROM storage.folders fo
JOIN _copy_map cm ON fo.id = cm.old_id;
-- ── Insert folders level by level ──
-- Post-D7: `user_id` intentionally omitted from the column list so
-- copied rows leave the (now-nullable) column NULL. Provenance is
-- carried by `created_by` / `updated_by` (§14 columns) — preserved
-- from source so authorship survives the copy.
FOR v_level IN v_root_depth .. v_max_depth LOOP
INSERT INTO storage.folders(
id, name, parent_id,
drive_id, created_by, updated_by
)
SELECT cm.new_id,
CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL
THEN p_dest_name ELSE fo.name END,
CASE WHEN fo.id = p_source_id THEN p_target_parent_id
ELSE pm.new_id END,
v_dest_drive_id,
fo.created_by,
fo.updated_by
FROM storage.folders fo
JOIN _copy_map cm ON fo.id = cm.old_id
LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id
WHERE NOT fo.is_trashed
AND nlevel(fo.lpath) = v_level;
GET DIAGNOSTICS v_inserted = ROW_COUNT;
v_folders := v_folders + v_inserted;
END LOOP;
-- Temp mapping for files src→dst (dst ids pre-allocated so we can hand
-- both sides to copy_file_satellites below).
CREATE TEMP TABLE IF NOT EXISTS _copy_file_map(
old_id UUID PRIMARY KEY,
new_id UUID NOT NULL DEFAULT gen_random_uuid()
) ON COMMIT DROP;
TRUNCATE _copy_file_map;
INSERT INTO _copy_file_map(old_id)
SELECT f.id
FROM storage.files f
JOIN _copy_map cm ON f.folder_id = cm.old_id
WHERE NOT f.is_trashed;
-- ── Batch copy all files (zero-copy: same blob_hash) ──
-- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`.
INSERT INTO storage.files(
id, name, folder_id, blob_hash, size, mime_type,
media_sort_date, drive_id, created_by, updated_by
)
SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size,
f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by,
f.updated_by
FROM storage.files f
JOIN _copy_map cm ON f.folder_id = cm.old_id
JOIN _copy_file_map fm ON fm.old_id = f.id
WHERE NOT f.is_trashed;
GET DIAGNOSTICS v_files = ROW_COUNT;
-- Everything that follows a file on copy — blob references and dead
-- properties — in one call, shared with the single-file copy path.
--
-- Both aggregates order by `old_id`, which is what makes the two arrays
-- correspond positionally; `array_agg` without a matching ORDER BY would
-- be free to pair a file with another file's satellites.
IF v_files > 0 THEN
PERFORM storage.copy_file_satellites(
(SELECT array_agg(old_id ORDER BY old_id) FROM _copy_file_map),
(SELECT array_agg(new_id ORDER BY old_id) FROM _copy_file_map)
);
END IF;
-- Folder dead properties. Files are handled inside copy_file_satellites;
-- folders have no other satellites, so this stays here.
INSERT INTO storage.webdav_dead_properties
(folder_id, namespace, local_name, value)
SELECT cm.new_id, dp.namespace, dp.local_name, dp.value
FROM storage.webdav_dead_properties dp
JOIN _copy_map cm ON dp.folder_id = cm.old_id;
RETURN QUERY SELECT v_new_root::text, v_folders, v_files;
END;
$$ LANGUAGE plpgsql;
migrations/20261020000000_file_attached_blobs.sql
+157
View File
@@ -0,0 +1,157 @@
-- Step 9 of `docs/plan/derived-blobs.md` — the file-keyed half of the pair.
--
-- `content_derived_blobs` holds bytes that are a pure deterministic function
-- of a file's content, so they are keyed by that content and shared across
-- every file holding it. This table holds the opposite: bytes a USER supplied
-- or chose. Those must never be shared across files, and the key is what
-- enforces it.
--
-- The distinction is a security boundary, not a modelling preference. If a
-- client-uploaded preview were content-keyed, user A could upload a file plus
-- a preview that misrepresents it; when user B later uploads the same bytes,
-- dedup would match and B would be served A's preview. Content-keying is only
-- safe when the bytes are derivable from the content by the server — nothing
-- to poison, because anyone with the same input gets the same output.
--
-- Required now rather than deferred: the SPA already generates and PUTs
-- previews for PDFs, and there is no server-side regeneration path for them,
-- so the sidecar migration has nowhere else to put those bytes.
CREATE TABLE IF NOT EXISTS storage.file_attached_blobs (
file_id UUID NOT NULL REFERENCES storage.files(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('preview', 'subtitle', 'cover_art')),
variant TEXT NOT NULL,
blob_hash VARCHAR(64) NOT NULL,
content_type TEXT NOT NULL,
-- Provenance convention: NOT NULL and NO foreign key. A FK with
-- ON DELETE SET NULL loses the audit trail exactly when it matters most,
-- and without an ON DELETE clause it would block deleting a user
-- outright. Deleting the uploader must not rewrite history, so the id is
-- retained even once it no longer resolves. Rows imported with no known
-- uploader carry the all-zeros sentinel.
uploaded_by UUID NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (file_id, kind, variant)
);
-- Reverse lookup for the reference recompute and for dedup_gc's reap
-- predicate: both ask "does any row reference this blob?".
CREATE INDEX IF NOT EXISTS idx_file_attached_blobs_blob_hash
ON storage.file_attached_blobs (blob_hash);
-- ── The routing rule, recorded on both tables ────────────────────────────
-- Choosing the wrong table is a silent poisoning bug rather than a compile
-- error, so the rule lives where an implementor will actually meet it.
COMMENT ON TABLE storage.file_attached_blobs IS
'User-supplied or user-chosen artifacts (client previews, subtitles, cover art) keyed by FILE. File-keyed on purpose: these bytes are not derivable from the file''s content, so sharing them across files with identical content would let one user''s upload be served for another user''s file. Server-derived bytes must NOT be stored here — see docs/plan/derived-blobs.md.';
COMMENT ON COLUMN storage.file_attached_blobs.file_id IS
'The file these bytes are attached to. ON DELETE CASCADE: the attachment has no meaning without it. Deleting the row does NOT release the blob reference — the owning service does that in its on_file_deleted hook.';
COMMENT ON COLUMN storage.file_attached_blobs.blob_hash IS
'The attached Blob. Reference HOLDER — bumps chunk_manifests.ref_count via DedupService::add_reference. Dedup still applies to the bytes themselves; what is forbidden is sharing the MAPPING across files.';
COMMENT ON COLUMN storage.file_attached_blobs.variant IS
'Opaque discriminator within a kind (preview | en | fr | cover...). New axes go inside this string, never into new columns.';
COMMENT ON COLUMN storage.file_attached_blobs.uploaded_by IS
'Who supplied these bytes. Retained after the user is deleted — deleting a user must not rewrite provenance. The only trace that an Editor on a shared file replaced the owner''s preview.';
COMMENT ON TABLE storage.content_derived_blobs IS
'Server-derived artifacts (thumbnails, transcodes) keyed by the BLAKE3 of their SOURCE content. Content-keyed on purpose: identical content shares one derivation. ROUTING RULE — bytes that are a pure deterministic function of the file''s content belong here; bytes that are user-supplied or user-chosen belong in storage.file_attached_blobs, which is file-keyed and never shared. See docs/plan/derived-blobs.md.';
-- ── Teach the copy fan-out about it ──────────────────────────────────────
--
-- Only the attached-blobs arm is new versus `20261019000000`; everything
-- else is that definition verbatim. Adding a file-keyed table is now one
-- edit in one function, which is the whole point of having consolidated the
-- two copy paths first.
CREATE OR REPLACE FUNCTION storage.copy_file_satellites(
p_old_ids UUID[],
p_new_ids UUID[]
) RETURNS void AS $$
DECLARE
v_unmatched TEXT[];
BEGIN
IF p_old_ids IS NULL OR cardinality(p_old_ids) = 0 THEN
RETURN;
END IF;
IF p_new_ids IS NULL OR cardinality(p_old_ids) <> cardinality(p_new_ids) THEN
-- Positional correspondence is the whole interface; a length
-- mismatch would silently attach satellites to the wrong file.
RAISE EXCEPTION
'copy_file_satellites: id arrays must correspond positionally (% old vs % new)',
cardinality(p_old_ids), COALESCE(cardinality(p_new_ids), 0);
END IF;
-- 1. WebDAV dead properties. RFC 4918 §8.8 requires COPY to duplicate
-- them: properties describe the resource, and the copy is a resource.
INSERT INTO storage.webdav_dead_properties
(file_id, namespace, local_name, value)
SELECT m.new_id, dp.namespace, dp.local_name, dp.value
FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id)
JOIN storage.webdav_dead_properties dp ON dp.file_id = m.old_id;
-- 2. A reference on the copied content, so deleting the original cannot
-- reap bytes the copy still needs. Read from the NEW rows rather than
-- the old ones: that is what makes an unreferenceable copy impossible
-- to create, since a row that failed to insert contributes nothing.
SELECT storage.add_blob_references(array_agg(f.blob_hash))
INTO v_unmatched
FROM unnest(p_new_ids) AS n(id)
JOIN storage.files f ON f.id = n.id
WHERE NOT f.is_trashed;
IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN
-- Warn, do not abort. A missing registry row means the SOURCE file
-- was already broken; the copy merely inherits it. Failing here
-- would abort an entire folder copy over one pre-existing fault,
-- which is worse than completing it and reporting. The blob-level
-- audit jobs are what surface the underlying breakage.
RAISE WARNING
'copy_file_satellites: % copied file(s) reference a blob with no registry row (first: %); source was already broken',
cardinality(v_unmatched), v_unmatched[1];
END IF;
-- 3. File-keyed attachments — client previews, subtitles, cover art.
-- DUPLICATED rather than shared, because the key is `file_id` and the
-- copy is a different file. `uploaded_by` carries over: the person
-- who supplied the bytes did not change because someone copied the
-- file, and rewriting it to the copier would forge provenance.
--
-- Each duplicated row is a new reference on the same blob, so the
-- bytes are still deduplicated — it is the MAPPING that must not be
-- shared, not the content.
WITH copied AS (
INSERT INTO storage.file_attached_blobs
(file_id, kind, variant, blob_hash, content_type, uploaded_by)
SELECT m.new_id, a.kind, a.variant, a.blob_hash, a.content_type, a.uploaded_by
FROM unnest(p_old_ids, p_new_ids) AS m(old_id, new_id)
JOIN storage.file_attached_blobs a ON a.file_id = m.old_id
RETURNING blob_hash
)
SELECT storage.add_blob_references(array_agg(blob_hash))
INTO v_unmatched
FROM copied;
IF v_unmatched IS NOT NULL AND cardinality(v_unmatched) > 0 THEN
RAISE WARNING
'copy_file_satellites: % attached blob(s) reference no registry row (first: %); source was already broken',
cardinality(v_unmatched), v_unmatched[1];
END IF;
-- ── Deliberately absent ──────────────────────────────────────────────
--
-- storage.comments (future): NOT copied. A copy is a new artifact; the
-- discussion belongs to the original.
--
-- content_derived_blobs, blob_extracted_text, faces.faces: content-keyed.
-- The copy shares the source's hash, so it already sees them — copying
-- would duplicate rows that are keyed on the very thing being shared.
--
-- storage.favorites, recent_items, shares: properties of the ORIGINAL's
-- relationship to users, not of its content.
END;
$$ LANGUAGE plpgsql;
migrations/20261021000000_file_attached_blobs_decrement_trigger.sql
+28
View File
@@ -0,0 +1,28 @@
-- Release the blob reference when an attachment row goes away.
--
-- `storage.file_attached_blobs.file_id` is `ON DELETE CASCADE`, so deleting a
-- file removes its attachment rows inside the database — invisible to Rust.
-- The lifecycle hook cannot cover this: `on_file_deleted` fires AFTER
-- `delete_file`, by which point the cascade has already run and there is
-- nothing left to read. The references would survive with no row behind them,
-- and `dedup_gc` would see a positive count forever — bytes pinned for good.
--
-- `storage.decrement_blob_ref()` already exists for exactly this, on
-- `storage.files`. It keys off `OLD.blob_hash` and is otherwise
-- table-agnostic, so it applies verbatim — and reusing it keeps the
-- manifest-first decrement contract defined in one place rather than
-- transcribed into a second trigger that can drift.
--
-- Only DELETE. Replacing a preview updates `blob_hash` in place
-- (`store_attached_blob` is ON CONFLICT DO UPDATE), and the reference to the
-- superseded blob is released there, in Rust. Adding UPDATE here would
-- double-decrement it.
CREATE OR REPLACE TRIGGER trg_file_attached_blobs_decrement_blob_ref
AFTER DELETE ON storage.file_attached_blobs
FOR EACH ROW
EXECUTE FUNCTION storage.decrement_blob_ref();
COMMENT ON TRIGGER trg_file_attached_blobs_decrement_blob_ref
ON storage.file_attached_blobs IS
'Releases the blob reference held by an attachment row. Needed because file_id is ON DELETE CASCADE, so rows vanish inside the DB where the Rust lifecycle hooks cannot see them.';
migrations/20261022000000_derived_variant_encodes_format.sql
+56
View File
@@ -0,0 +1,56 @@
-- Put the output format inside `variant`, where the plan says new axes go.
--
-- `content_derived_blobs.variant` held the size alone (`icon` | `preview` |
-- `large`), so a size could hold exactly ONE stored artifact regardless of
-- codec. That surfaced when the read order flipped (step 10c): a JPEG request
-- matched the WebP row and would have been served the wrong codec, which the
-- old ordering hid because the `.jpg` sidecar won first. The flip had to be
-- gated to WebP, which in turn means JPEG clients can never leave the sidecar
-- — so the sidecar can never be deleted.
--
-- It blocks transcodes harder still: those are multi-format by nature, so
-- without a format term two output codecs of one source collide on the
-- primary key.
--
-- Per the column's own comment — "new axes go inside this string, never into
-- new columns" — the axis goes in the string rather than into a fourth PK
-- column. The PK stays `(source_hash, kind, variant)`.
--
-- Shape: `{size}.{ext}` — `preview.webp`, `icon.jpg`, and later `720p.webp`
-- for transcodes.
--
-- The backfill is deterministic rather than a guess: `store_derived_blob` has
-- only ever been called with `"image/webp"` for thumbnails, so every existing
-- thumbnail row is WebP. `content_type` is checked anyway rather than assumed
-- — if that assumption is ever wrong, the row is left alone for a human to
-- look at instead of being silently mislabelled.
UPDATE storage.content_derived_blobs
SET variant = variant || '.webp'
WHERE kind = 'thumbnail'
AND content_type = 'image/webp'
-- Idempotent: skip anything already carrying a format suffix, so a
-- re-applied migration cannot produce `preview.webp.webp`.
AND variant NOT LIKE '%.%';
-- Anything left without a format suffix did not match the WebP assumption.
-- Surfaced as a warning rather than coerced: the read path will simply miss
-- those rows and fall back to the sidecar, which is safe, whereas guessing a
-- codec would serve the wrong bytes.
DO $$
DECLARE
v_unsuffixed INT;
BEGIN
SELECT COUNT(*) INTO v_unsuffixed
FROM storage.content_derived_blobs
WHERE kind = 'thumbnail' AND variant NOT LIKE '%.%';
IF v_unsuffixed > 0 THEN
RAISE WARNING
'derived_variant_encodes_format: % thumbnail row(s) have no format suffix (content_type was not image/webp). They will be ignored by the read path and re-derived on demand; inspect before deleting the sidecars.',
v_unsuffixed;
END IF;
END $$;
COMMENT ON COLUMN storage.content_derived_blobs.variant IS
'Opaque discriminator carrying every axis but the source and the kind: size AND output format, as {size}.{ext} (preview.webp | icon.jpg | 720p.webp). New axes go inside this string, never into new columns. A format term is required — without one, two codecs of the same source collide on the primary key, and the read path cannot tell which codec a row holds.';
migrations/20261023000000_content_derived_blobs_negative_rows.sql
+63
View File
@@ -0,0 +1,63 @@
-- Negative rows in `storage.content_derived_blobs`.
--
-- Some derivations can only be known to be useless by doing the whole
-- expensive job. `ImageTranscodeService` learns that WebP is not smaller
-- than the original by decoding and re-encoding the whole image; a
-- thumbnail renderer learns a source is undecodable, or over the
-- 50-megapixel ceiling, only by attempting it. Recomputing that verdict
-- on every request is the same cost as computing it the first time.
--
-- Today those verdicts live in RAM (moka's zero-weight empty-Bytes
-- convention) and, for transcodes, as zero-byte `.skip` files on local
-- disk. Both vanish: moka evicts, and the local disk is exactly what
-- this plan is deleting. So the verdict is stored here, next to the
-- positive derivations, as a row whose derived Blob is NULL.
--
-- ## Why NULL rather than a sentinel hash
--
-- A reserved hash was considered and rejected. It would stop
-- `blob_hash` naming a real Blob, and every consumer — the refcount
-- recompute in `manifests_consistency`, `dedup_gc`'s reap predicate,
-- `satellites_consistency`'s dangling check — would need to learn the
-- exception or silently mis-handle it. NULL is already the SQL way to
-- say "no Blob", and those consumers all join on `blob_hash`, so a NULL
-- drops out of the join instead of matching something fictional.
--
-- ## The CHECK matters
--
-- A row with a `blob_hash` but no `content_type` is unserveable; a row
-- with a `content_type` but no `blob_hash` claims a type for bytes that
-- do not exist. Both are bugs that would surface far from their cause,
-- so the pair moves together or not at all.
--
-- ## What must NOT become a negative row
--
-- Only failures that are DETERMINISTIC IN THE CONTENT. A transcode that
-- was not smaller, or a source that cannot be decoded, will fail the
-- same way forever — those are worth remembering. A generation timeout,
-- a closed semaphore, an I/O error reading the source Blob are
-- properties of the moment, not the content; persisting one marks a
-- perfectly good image as underivable permanently, and nothing ever
-- retries it. The asymmetry sets the default: a wrongly-cached
-- transient is silent and forever, a missing negative merely costs
-- repeated work. When in doubt, do not write the row.
ALTER TABLE storage.content_derived_blobs
ALTER COLUMN blob_hash DROP NOT NULL,
ALTER COLUMN content_type DROP NOT NULL;
ALTER TABLE storage.content_derived_blobs
DROP CONSTRAINT IF EXISTS content_derived_blobs_positive_or_negative;
ALTER TABLE storage.content_derived_blobs
ADD CONSTRAINT content_derived_blobs_positive_or_negative
CHECK (
(blob_hash IS NOT NULL AND content_type IS NOT NULL)
OR (blob_hash IS NULL AND content_type IS NULL)
);
COMMENT ON COLUMN storage.content_derived_blobs.blob_hash IS
'The derived Blob, or NULL for a NEGATIVE row: the derivation was attempted and is known not to be worth storing (transcode came out larger, source undecodable, source over the decode ceiling). Reference HOLDER when present — bumps chunk_manifests.ref_count via DedupService::add_reference. Only content-deterministic failures may be recorded as negatives; transient ones (timeout, semaphore, I/O) must not, or a momentary failure becomes permanent.';
COMMENT ON COLUMN storage.content_derived_blobs.content_type IS
'MIME type of the derived Blob. NULL exactly when blob_hash is NULL — the CHECK keeps the pair together, since a type without bytes describes nothing and bytes without a type cannot be served.';
migrations/20261024000000_nfc_name_column_comments.sql
+52
View File
@@ -0,0 +1,52 @@
-- COMMENT ON COLUMN for every user-visible-name column whose invariant
-- ("stored bytes are NFC") is enforced by the write repository, not by
-- the DB itself.
--
-- Why this migration exists
-- ─────────────────────────
-- Before 2026-09-04 the NFC invariant lived at `File::new` /
-- `Folder::new_folder` entity constructors — plausible-looking but
-- DEAD CODE for the create path, because every real caller went
-- straight from a DTO string to `sqlx::bind()` inside the repos
-- without ever constructing the entity first. Result: 22 audited
-- entry points, every single one shipped raw client input to the DB.
-- macOS Finder / DAVX5 / NC-desktop uploads landed NFD; NFC-
-- normalizing clients then failed to find their own content by URL
-- (AtalayaLabs/OxiCloud#706).
--
-- The fix moved normalization to the repository methods that own the
-- INSERT / UPDATE. The next contributor writing a new write surface
-- may reasonably wonder where to enforce the invariant — this comment
-- puts the answer next to the column so grep-hunting the codebase is
-- not required. Purely documentation; no runtime effect. A stronger
-- form (CHECK CONSTRAINT `name = normalize(name, NFC)`) was
-- considered and rejected for now — that would rely on every
-- historical row already being NFC (which we deliberately do NOT
-- migrate on read, so pre-fix rows stay in place until an operator
-- runs `oxicloud migrate nfc-filenames`), and would fail-boot any
-- upgrade path where the migrate has not yet been applied.
--
-- Idempotent. COMMENT ON COLUMN replaces any prior comment on the
-- same target, so re-running has no effect.
COMMENT ON COLUMN storage.files.name IS
'User-visible file name. MUST be NFC (Unicode Normalization Form C). '
'Invariant enforced at write time by '
'src/infrastructure/repositories/pg/file_blob_write_repository.rs — the '
'`save_file_with_blob_impl`, `copy_file`, `rename_file`, '
'`register_file_deferred`, and `copy_folder_tree` methods each call '
'`normalize_storage_name(_owned)` before binding. No DB-level CHECK '
'constraint (historical NFD rows may still exist on pre-2026-09-04 '
'databases until `oxicloud migrate nfc-filenames` is run). New write '
'surfaces MUST land in one of those repo methods; direct INSERT '
'bypasses the invariant.';
COMMENT ON COLUMN storage.folders.name IS
'User-visible folder name. MUST be NFC (Unicode Normalization Form C). '
'Invariant enforced at write time by '
'src/infrastructure/repositories/pg/folder_db_repository.rs — the '
'`create_folder` and `rename_folder` methods each call '
'`normalize_storage_name_owned` before binding. See also '
'storage.files.name — identical contract, different table. No DB-level '
'CHECK (see that column comment). New write surfaces MUST land in one of '
'those repo methods; direct INSERT bypasses the invariant.';
scripts/backup.sh
Executable
+5
View File
@@ -0,0 +1,5 @@
#!/bin/bash
NOW="$(date '+%Y-%m-%d %H:%M:%S')"
pg_dump postgres://postgres:postgres@localhost:5432/oxicloud -F c --disable-triggers > "backup.${NOW}.dump"
scripts/compute-docker-tags.sh
Executable
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env bash
# =============================================================
# Compute Docker channel + version + tag set for the
# `docker-publish` workflow. Extracted from the workflow so the
# logic can be unit-tested via `scripts/test-docker-publish-tags.sh`
# without needing to dispatch the workflow itself.
#
# The workflow's meta step invokes this via `bash scripts/
# compute-docker-tags.sh` with the GITHUB_* env vars set; the
# same call form works from a local shell for smoke checks
# ("what would we publish if I tag v0.8.8 tomorrow?").
#
# Inputs (env vars — missing required inputs exit non-zero):
# EVENT_NAME workflow_dispatch | release | push
# GITHUB_REF refs/heads/main | refs/tags/vX.Y.Z | ...
# (required for the `push` event)
# DISPATCH_VERSION workflow_dispatch only, e.g. v0.5.3 or 0.5.3
# RELEASE_TAG release event only, e.g. v0.8.7
# REGISTRY_IMAGE Docker Hub image (e.g. diocrafts/oxicloud)
# GHCR_REGISTRY_IMAGE GHCR image (e.g. ghcr.io/atalayalabs/oxicloud)
# SKIP_DOCKERHUB optional. When "true", omits Docker Hub tags
# from the output — used by forks whose
# DOCKERHUB_TOKEN secret isn't configured. The
# workflow only pushes to GHCR (which needs no
# external secret; auth via GITHUB_TOKEN).
#
# Outputs:
# Always writes `version=<v>`, `channel=<c>`, and a `tags:` block
# to stdout — visible in workflow logs and captured by the test
# harness for diff-based assertions.
#
# When `GITHUB_OUTPUT` is set (inside a GHA `run:` step), also
# emits the same values via GHA's `>> $GITHUB_OUTPUT` convention
# so subsequent steps can reference `${{ steps.meta.outputs.tags }}`.
#
# Channel semantics (mirrors the workflow's tag policy):
# release — tag push / release event / manual dispatch:
# publish `:<version>` AND move `:latest`.
# main — push to `main` branch: publish `:main` (mutable
# tip) only. Never touches `:latest`, never emits a
# per-commit `:main-<sha>` (would balloon the
# registry across every merge).
# =============================================================
set -euo pipefail
: "${EVENT_NAME:?EVENT_NAME required}"
: "${REGISTRY_IMAGE:?REGISTRY_IMAGE required}"
: "${GHCR_REGISTRY_IMAGE:?GHCR_REGISTRY_IMAGE required}"
# Both GHCR and Docker Hub reject mixed-case namespace / image names
# ("repository name must be lowercase"). `${{ github.repository_owner
# }}` in the workflow inserts the GitHub username verbatim, and GitHub
# expression syntax has no `lower()` function. So we normalise here
# — the workflow keeps its declarative `env:` block, the script owns
# the case-safety contract, and tests cover it (see
# `test-docker-publish-tags.sh` for mixed-case cases).
REGISTRY_IMAGE=$(echo "$REGISTRY_IMAGE" | tr '[:upper:]' '[:lower:]')
GHCR_REGISTRY_IMAGE=$(echo "$GHCR_REGISTRY_IMAGE" | tr '[:upper:]' '[:lower:]')
# Docker Hub is optional — a fork without DOCKERHUB_TOKEN configured
# still publishes to GHCR (its own namespace, auth via GITHUB_TOKEN)
# but skips Docker Hub cleanly. The workflow sets this to "true" when
# `secrets.DOCKERHUB_TOKEN` is empty; the tag set below omits DH
# entries in that case.
SKIP_DOCKERHUB="${SKIP_DOCKERHUB:-false}"
case "$EVENT_NAME" in
workflow_dispatch)
: "${DISPATCH_VERSION:?DISPATCH_VERSION required for workflow_dispatch}"
VERSION="${DISPATCH_VERSION#v}"
CHANNEL="release"
;;
release)
: "${RELEASE_TAG:?RELEASE_TAG required for release event}"
VERSION="${RELEASE_TAG#v}"
CHANNEL="release"
;;
push)
: "${GITHUB_REF:?GITHUB_REF required for push event}"
if [ "$GITHUB_REF" = "refs/heads/main" ]; then
VERSION="main"
CHANNEL="main"
else
# Tag push — strip refs/tags/ prefix and leading `v`.
VERSION="${GITHUB_REF#refs/tags/}"
VERSION="${VERSION#v}"
CHANNEL="release"
fi
;;
*)
echo "compute-docker-tags: unsupported EVENT_NAME: $EVENT_NAME" >&2
exit 1
;;
esac
# Assemble the multi-line tag set. Format matches what the
# `docker/build-push-action` `tags:` input consumes — one tag
# per line, whitespace ignored between lines. Docker Hub entries
# omitted entirely when SKIP_DOCKERHUB=true — the build-push-action
# just doesn't see the tags, so no auth is attempted for them.
if [ "$CHANNEL" = "main" ]; then
if [ "$SKIP_DOCKERHUB" = "true" ]; then
TAGS="${GHCR_REGISTRY_IMAGE}:main"
else
TAGS="${REGISTRY_IMAGE}:main
${GHCR_REGISTRY_IMAGE}:main"
fi
else
if [ "$SKIP_DOCKERHUB" = "true" ]; then
TAGS="${GHCR_REGISTRY_IMAGE}:${VERSION}
${GHCR_REGISTRY_IMAGE}:latest"
else
TAGS="${REGISTRY_IMAGE}:${VERSION}
${REGISTRY_IMAGE}:latest
${GHCR_REGISTRY_IMAGE}:${VERSION}
${GHCR_REGISTRY_IMAGE}:latest"
fi
fi
# Emit to $GITHUB_OUTPUT when running under GHA — subsequent
# workflow steps read via `${{ steps.meta.outputs.tags }}`.
if [ -n "${GITHUB_OUTPUT:-}" ]; then
{
echo "version=$VERSION"
echo "channel=$CHANNEL"
echo "tags<<EOF"
echo "$TAGS"
echo "EOF"
} >> "$GITHUB_OUTPUT"
fi
# Also emit VERSION + CHANNEL + SKIP_DOCKERHUB to $GITHUB_ENV —
# later steps (`Verify published image`, DockerHub-gated conditionals)
# read these directly. Keeps the step-scoped env aligned with the
# steps.meta.outputs.* set for consumers that prefer one or the other.
#
# Also OVERRIDE the workflow-level REGISTRY_IMAGE / GHCR_REGISTRY_IMAGE
# env vars with the lowercased forms. Without this, the verify step
# would read the workflow-declared mixed-case value from
# `${{ github.repository_owner }}` (e.g. `ghcr.io/EdouardVanbelle/
# oxicloud`) and `docker pull` would reject it — despite the tags
# themselves being lowercased in the actual push. Step-level env
# additions take precedence over workflow-level for subsequent steps.
if [ -n "${GITHUB_ENV:-}" ]; then
{
echo "VERSION=$VERSION"
echo "CHANNEL=$CHANNEL"
echo "SKIP_DOCKERHUB=$SKIP_DOCKERHUB"
echo "REGISTRY_IMAGE=$REGISTRY_IMAGE"
echo "GHCR_REGISTRY_IMAGE=$GHCR_REGISTRY_IMAGE"
} >> "$GITHUB_ENV"
fi
# Always echo to stdout — visible in workflow logs (useful for
# dry-run verification, when the push step is skipped) and
# consumed by the test harness for equality checks.
echo "version=$VERSION"
echo "channel=$CHANNEL"
echo "tags:"
echo "$TAGS" | sed 's/^/ /'
scripts/restore.sh
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/bin/bash
# scripts/restore.sh — restore a pg_dump custom-format snapshot into a
# fresh oxicloud DB.
#
# Drops + recreates the whole DB before restoring so migrations added
# AFTER the dump was taken don't block --clean's DROPs. Symptom of
# that class (hit 2026-08-30):
#
# pg_restore: erreur : ...
# cannot drop constraint files_pkey on table storage.files because
# other objects depend on it
# DÉTAIL : constraint file_attached_blobs_file_id_fkey on table
# storage.file_attached_blobs depends on index storage.files_pkey
#
# The dump only knows about objects that existed at dump time;
# `pg_restore --clean` DROPs exactly those. Anything added since
# (`file_attached_blobs` in the example above) survives and blocks
# DROPs of things it depends on. Drop-and-recreate the whole DB
# sidesteps the problem entirely.
#
# Usage:
# ./scripts/restore.sh <dump-file>
#
# Companion of the pg_dump command in memory
# bug_pg_dump_folders_circular_fk.md:
# pg_dump postgres://... -F c --disable-triggers > backup.$NOW.dump
#
# NB: this restores the DB ONLY. Blob storage on disk
# (${OXICLOUD_STORAGE_PATH}) is NOT touched — snapshot + restore that
# separately with rsync if you need lockstep DB/disk state.
set -euo pipefail
DUMP="${1:?usage: $0 <dump-file>}"
[[ -f "$DUMP" ]] || { echo "[restore] ERROR: dump file not found: $DUMP" >&2; exit 1; }
# Admin connection — connect to the `postgres` maintenance DB so we can
# drop `oxicloud` itself (can't drop the DB you're connected to). Both
# connection strings share credentials from the sandbox setup.
ADMIN="postgres://postgres:postgres@localhost:5432/postgres"
TARGET="postgres://postgres:postgres@localhost:5432/oxicloud"
# 1. Terminate every connection to `oxicloud` so DROP DATABASE can
# proceed. OxiCloud running against 5432? rust-analyzer with
# sqlx-cli open? Any lingering psql session? All of them block
# `DROP DATABASE` with "database is being accessed by other users".
# pg_terminate_backend kicks them cleanly (they'll reconnect if
# they retry).
echo "[restore] Terminating connections to oxicloud..."
psql "$ADMIN" -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'oxicloud'
AND pid <> pg_backend_pid();
" >/dev/null
# 2. Drop + recreate. IF EXISTS on DROP so a fresh workstation without
# an existing `oxicloud` database doesn't error on the first-ever
# invocation.
echo "[restore] Dropping + recreating oxicloud database..."
psql "$ADMIN" <<'SQL'
DROP DATABASE IF EXISTS oxicloud;
CREATE DATABASE oxicloud OWNER postgres;
SQL
# 3. Restore. `--clean --if-exists` no longer needed (fresh DB from
# step 2). The remaining flags:
#
# * --disable-triggers — turn triggers off during data load so the
# `storage.folders.parent_id` self-FK doesn't reject rows whose
# parent hasn't been inserted yet in the same COPY batch. See
# memory bug_pg_dump_folders_circular_fk for background.
# * --single-transaction — atomic restore (all-or-nothing) AND
# lets --disable-triggers work without superuser (superuser is
# required otherwise).
# * --no-owner --no-privileges — portable, ignores ownership /
# GRANTs from the source system so a dump taken from one machine
# restores cleanly on another with different user names.
echo "[restore] Restoring from $DUMP..."
pg_restore \
--disable-triggers \
--single-transaction \
--no-owner \
--no-privileges \
-d "$TARGET" \
"$DUMP"
echo "[restore] Done. Database restored to snapshot state in $DUMP."
scripts/test-docker-publish-tags.sh
Executable
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env bash
# =============================================================
# Unit tests for `scripts/compute-docker-tags.sh`.
#
# Exercises every trigger channel the docker-publish workflow
# supports, plus the invalid-input path. Runs standalone in
# under a second — cheap regression check to lock the tag-set
# contract before pushing changes to `.github/workflows/
# docker-publish.yml`.
#
# Run:
# bash scripts/test-docker-publish-tags.sh
# =============================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT="$SCRIPT_DIR/compute-docker-tags.sh"
if [ ! -f "$SCRIPT" ]; then
echo "compute-docker-tags.sh not found at $SCRIPT" >&2
exit 2
fi
REGISTRY_IMAGE=diocrafts/oxicloud
GHCR_REGISTRY_IMAGE=ghcr.io/atalayalabs/oxicloud
pass=0
fail=0
# Runs the script with the given env, compares stdout against expected.
# `env "$@" bash ...` passes the env vars only for this invocation so
# leftover state from a prior case can't leak across.
expect() {
local name="$1" expected="$2"
shift 2
local actual rc
actual=$(env -i \
REGISTRY_IMAGE="$REGISTRY_IMAGE" \
GHCR_REGISTRY_IMAGE="$GHCR_REGISTRY_IMAGE" \
PATH="/usr/bin:/bin" \
"$@" \
bash "$SCRIPT" 2>&1)
rc=$?
if [ "$rc" -ne 0 ]; then
echo "FAIL: $name — script exited $rc:"
echo "$actual" | sed 's/^/ /'
fail=$((fail + 1))
return
fi
if [ "$actual" = "$expected" ]; then
echo "PASS: $name"
pass=$((pass + 1))
else
echo "FAIL: $name"
echo " expected:"
echo "$expected" | sed 's/^/ /'
echo " actual:"
echo "$actual" | sed 's/^/ /'
fail=$((fail + 1))
fi
}
expect_fail() {
local name="$1"
shift
if env -i \
REGISTRY_IMAGE="$REGISTRY_IMAGE" \
GHCR_REGISTRY_IMAGE="$GHCR_REGISTRY_IMAGE" \
PATH="/usr/bin:/bin" \
"$@" \
bash "$SCRIPT" >/dev/null 2>&1
then
echo "FAIL: $name (script should have exited non-zero)"
fail=$((fail + 1))
else
echo "PASS: $name"
pass=$((pass + 1))
fi
}
# ── Happy paths ─────────────────────────────────────────────────
expect "push to main → :main only, no :latest" \
"version=main
channel=main
tags:
diocrafts/oxicloud:main
ghcr.io/atalayalabs/oxicloud:main" \
EVENT_NAME=push GITHUB_REF=refs/heads/main
expect "push of version tag → :<v> + :latest" \
"version=0.8.7
channel=release
tags:
diocrafts/oxicloud:0.8.7
diocrafts/oxicloud:latest
ghcr.io/atalayalabs/oxicloud:0.8.7
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=push GITHUB_REF=refs/tags/v0.8.7
expect "release event → :<v> + :latest" \
"version=0.9.0
channel=release
tags:
diocrafts/oxicloud:0.9.0
diocrafts/oxicloud:latest
ghcr.io/atalayalabs/oxicloud:0.9.0
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=release RELEASE_TAG=v0.9.0
expect "workflow_dispatch with 'v' prefix" \
"version=1.0.0
channel=release
tags:
diocrafts/oxicloud:1.0.0
diocrafts/oxicloud:latest
ghcr.io/atalayalabs/oxicloud:1.0.0
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=workflow_dispatch DISPATCH_VERSION=v1.0.0
expect "workflow_dispatch without 'v' prefix (permissive)" \
"version=1.0.0
channel=release
tags:
diocrafts/oxicloud:1.0.0
diocrafts/oxicloud:latest
ghcr.io/atalayalabs/oxicloud:1.0.0
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=workflow_dispatch DISPATCH_VERSION=1.0.0
expect "release with pre-release version" \
"version=0.9.0-rc1
channel=release
tags:
diocrafts/oxicloud:0.9.0-rc1
diocrafts/oxicloud:latest
ghcr.io/atalayalabs/oxicloud:0.9.0-rc1
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=release RELEASE_TAG=v0.9.0-rc1
# ── SKIP_DOCKERHUB path (forks without DOCKERHUB_TOKEN) ─────────
expect "push to main + SKIP_DOCKERHUB → GHCR only" \
"version=main
channel=main
tags:
ghcr.io/atalayalabs/oxicloud:main" \
EVENT_NAME=push GITHUB_REF=refs/heads/main SKIP_DOCKERHUB=true
expect "release + SKIP_DOCKERHUB → GHCR :<v> + :latest only" \
"version=0.9.0
channel=release
tags:
ghcr.io/atalayalabs/oxicloud:0.9.0
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=release RELEASE_TAG=v0.9.0 SKIP_DOCKERHUB=true
expect "dispatch + SKIP_DOCKERHUB → GHCR :<v> + :latest only" \
"version=1.0.0
channel=release
tags:
ghcr.io/atalayalabs/oxicloud:1.0.0
ghcr.io/atalayalabs/oxicloud:latest" \
EVENT_NAME=workflow_dispatch DISPATCH_VERSION=v1.0.0 SKIP_DOCKERHUB=true
expect "explicit SKIP_DOCKERHUB=false behaves like default (both registries)" \
"version=main
channel=main
tags:
diocrafts/oxicloud:main
ghcr.io/atalayalabs/oxicloud:main" \
EVENT_NAME=push GITHUB_REF=refs/heads/main SKIP_DOCKERHUB=false
# ── Case-safety — GHCR / DH reject mixed-case names ─────────────
#
# Regression pin: `${{ github.repository_owner }}` inserts a
# GitHub username verbatim, which is often mixed-case
# (e.g. EdouardVanbelle). The registries reject that with
# "repository name must be lowercase". The script normalises
# both inputs; these cases assert it.
# Local override so we can pass a mixed-case owner without touching
# the harness's defaults on other cases.
_orig_ghcr="$GHCR_REGISTRY_IMAGE"
_orig_dh="$REGISTRY_IMAGE"
GHCR_REGISTRY_IMAGE=ghcr.io/EdouardVanbelle/OxiCloud \
REGISTRY_IMAGE=ghcr.io/EdouardVanbelle/OxiCloud \
expect "mixed-case owner and image lowercased in tags" \
"version=main
channel=main
tags:
ghcr.io/edouardvanbelle/oxicloud:main" \
EVENT_NAME=push GITHUB_REF=refs/heads/main \
REGISTRY_IMAGE=DioCrafts/OxiCloud \
GHCR_REGISTRY_IMAGE=ghcr.io/EdouardVanbelle/OxiCloud \
SKIP_DOCKERHUB=true
expect "release with mixed-case DH namespace lowercased" \
"version=0.8.7
channel=release
tags:
diocrafts/oxicloud:0.8.7
diocrafts/oxicloud:latest
ghcr.io/edouardvanbelle/oxicloud:0.8.7
ghcr.io/edouardvanbelle/oxicloud:latest" \
EVENT_NAME=release RELEASE_TAG=v0.8.7 \
REGISTRY_IMAGE=DioCrafts/OxiCloud \
GHCR_REGISTRY_IMAGE=ghcr.io/EdouardVanbelle/OxiCloud
REGISTRY_IMAGE="$_orig_dh"
GHCR_REGISTRY_IMAGE="$_orig_ghcr"
unset _orig_dh _orig_ghcr
# ── Error paths ─────────────────────────────────────────────────
expect_fail "unknown event rejected" \
EVENT_NAME=cron GITHUB_REF=refs/heads/main
expect_fail "push without GITHUB_REF rejected" \
EVENT_NAME=push
expect_fail "release without RELEASE_TAG rejected" \
EVENT_NAME=release
expect_fail "dispatch without DISPATCH_VERSION rejected" \
EVENT_NAME=workflow_dispatch
# ── Report ──────────────────────────────────────────────────────
echo ""
echo "─────────────────────────"
echo "Passed: $pass Failed: $fail"
[ "$fail" -eq 0 ]
src/application/dtos/session_dto.rs
+179 -1
View File
@@ -14,6 +14,8 @@
//! separate batch fetch (extra round-trip). Frontend cross-references
//! `user_id` against its cached user list.
use std::time::Duration;
use chrono::{DateTime, Utc};
use serde::Serialize;
use utoipa::ToSchema;
@@ -21,6 +23,20 @@ use uuid::Uuid;
use crate::domain::entities::session::{Session, SessionOrigin};
/// The "recently seen" threshold that turns a session's
/// `last_seen_at` into a green-dot "Online" badge on the admin
/// sessions panel — AND the same window that drives the
/// `oxicloud_sessions_online[_users]` Prometheus gauges (see
/// `src/infrastructure/services/session_liveness_gauges.rs`).
/// The two MUST agree so the dashboard's per-row badge count
/// matches the gauge's aggregate — one source of truth here.
///
/// 5 min feels responsive without over-fluctuating with
/// tab-open-then-close blips. Deliberately hardcoded, not an
/// env var — see `docs/plan/sessions.md` §"Config surface" for
/// the reasoning.
pub const ONLINE_WINDOW: Duration = Duration::from_secs(5 * 60);
/// Authenticated-caller context — the caller's identity + session-
/// bound signals a service method might key off. Constructed at the
/// handler boundary from `AuthUser` and passed through unchanged;
@@ -53,6 +69,17 @@ pub struct SessionSummaryDto {
pub user_id: Uuid,
pub created_at: DateTime<Utc>,
pub expires_at: DateTime<Utc>,
/// Wall-clock time this session was last observed serving an
/// authenticated request. Moved forward per request by the
/// in-process [`LastSeenTracker`](crate::infrastructure::services::last_seen_tracker)
/// via a batched UPDATE every 30 s — so this value trails the
/// true "last seen" by at most one flush interval on a running
/// server. On DB read it always converges after a graceful
/// shutdown flush. Distinct from `created_at`: that only moves
/// on session rotation (silent refresh), so its resolution is
/// capped at the access-token TTL. The admin table renders a
/// "last seen X ago" column off this field.
pub last_seen_at: DateTime<Utc>,
pub ip_address: Option<String>,
pub user_agent: Option<String>,
/// `true` iff the session is DPoP-bound. Rendered as a lock icon
@@ -68,7 +95,24 @@ pub struct SessionSummaryDto {
pub is_revoked: bool,
/// Whether this row is currently usable — `!revoked && expires_at > now()`.
/// Kept server-side so the SPA doesn't drift if the browser clock is off.
/// **Distinct from [`is_online`](Self::is_online)** — this is a
/// *lifecycle* signal (row still has authority), that one is a
/// *presence* signal (a request landed on it lately).
pub is_active: bool,
/// Whether the session was actually observed serving a request in the
/// last [`ONLINE_WINDOW`] (5 min). Presence signal, orthogonal to
/// [`is_active`](Self::is_active): a session may be active-and-online
/// (green dot in the admin table), active-and-idle (no dot, "last
/// seen 12 min ago"), or non-active-and-offline (expired / revoked
/// rows are never online). Derived server-side against
/// [`ONLINE_WINDOW`] so the row-level badge stays consistent with
/// the `oxicloud_sessions_online[_users]` Prometheus aggregates.
///
/// Guaranteed `false` for revoked / expired rows — those short-
/// circuit before the recency check so a revoked row that happened
/// to receive a request in its final second before revocation
/// doesn't confusingly render "Online" post-revocation.
pub is_online: bool,
// NOTE: no `oidc_sid` / `oidc_sid_prefix` field. The IdP-emitted
// sid identifies the row's upstream session and stays server-side
// (used by Back-Channel Logout matching). Exposing even a prefix
@@ -104,23 +148,40 @@ impl SessionSummaryDto {
pub fn from_session(s: Session, caller_jkt: Option<&str>) -> Self {
let is_revoked = s.is_revoked();
let is_expired = s.is_expired();
let is_active = !is_revoked && !is_expired;
let jkt = s.dpop_jkt().map(|s| s.to_owned());
let dpop_jkt_prefix = jkt.as_ref().map(|t| t.chars().take(8).collect::<String>());
let is_current = match (jkt.as_deref(), caller_jkt) {
(Some(row), Some(caller)) => row == caller,
_ => false,
};
// Presence check gated on lifecycle — a revoked or expired
// row's `last_seen_at` may still be fresh (the last request
// that arrived just before revocation), but calling it
// "Online" post-revocation would confuse an admin reading
// the panel. Short-circuit on !is_active.
let online_cutoff = match chrono::Duration::from_std(ONLINE_WINDOW) {
Ok(d) => Utc::now() - d,
// Cast can only fail on a Duration too large for i64
// milliseconds; not reachable with our 5 min constant.
// Fall back to "never online" rather than panic — a
// wrong badge is fixable, a request-path panic is not.
Err(_) => DateTime::<Utc>::MAX_UTC,
};
let is_online = is_active && s.last_seen_at() > online_cutoff;
Self {
id: s.id(),
user_id: s.user_id(),
created_at: s.created_at(),
expires_at: s.expires_at(),
last_seen_at: s.last_seen_at(),
ip_address: s.ip_address().map(str::to_owned),
user_agent: s.user_agent().map(str::to_owned),
is_bound: jkt.is_some(),
dpop_jkt_prefix,
is_revoked,
is_active: !is_revoked && !is_expired,
is_active,
is_online,
origin: s.origin(),
is_current,
}
@@ -166,6 +227,7 @@ mod tests {
Some(sid.to_string()),
None,
crate::domain::entities::session::SessionOrigin::Oidc,
Utc::now(),
)
}
@@ -241,6 +303,121 @@ mod tests {
assert!(dto.is_active);
}
#[test]
fn dto_exposes_last_seen_at() {
// Regression: the admin table renders "last seen X ago"
// straight off this field, and clients that build
// dashboards off the session API rely on it too. Guards
// against a struct field being removed / renamed silently.
let s = base(false, None);
let expected = s.last_seen_at();
let dto = SessionSummaryDto::from(s);
assert_eq!(dto.last_seen_at, expected);
let json = serde_json::to_string(&dto).unwrap();
assert!(
json.contains("\"last_seen_at\""),
"wire shape must include `last_seen_at`: {json}"
);
}
/// A freshly-minted, unbound, unrevoked session ships with
/// `last_seen_at = Utc::now()` from `Session::new`, so it
/// MUST render as online. This is the green-dot happy path
/// the admin panel keys off — regression here means the
/// dashboard misses every currently-active session.
#[test]
fn dto_is_online_when_last_seen_is_fresh() {
let dto = SessionSummaryDto::from(base(false, None));
assert!(dto.is_online, "fresh session must be online: {dto:?}");
assert!(dto.is_active);
}
/// A session whose `last_seen_at` is older than the
/// [`ONLINE_WINDOW`] MUST render as offline even when the
/// row is otherwise Active — that's the whole point of the
/// presence vs lifecycle split. Constructed via `from_raw`
/// so we can stamp a stale timestamp deterministically.
#[test]
fn dto_is_not_online_when_last_seen_is_stale() {
let stale = Utc::now() - chrono::Duration::hours(1);
let s = Session::from_raw(
Uuid::new_v4(),
Uuid::new_v4(),
"rt".to_string(),
Utc::now() + Duration::days(30),
None,
None,
stale,
false,
Uuid::new_v4(),
None,
None,
None,
SessionOrigin::Password,
stale,
);
let dto = SessionSummaryDto::from(s);
assert!(!dto.is_online, "1h-idle session must not be online");
assert!(dto.is_active, "stale-but-alive session stays active");
}
/// Anti-confusion guard: a revoked row whose `last_seen_at`
/// happens to be fresh (the last request that landed just
/// before revocation) must NOT surface as "Online" — an admin
/// reading the panel post-revocation expects the green dot
/// gone. `is_online` short-circuits on `!is_active`.
#[test]
fn dto_is_not_online_when_revoked_even_if_fresh() {
let dto = SessionSummaryDto::from(base(true, None));
assert!(dto.is_revoked);
assert!(!dto.is_active);
assert!(
!dto.is_online,
"revoked-but-fresh row must never render as online",
);
}
/// Same anti-confusion guard for expiry: a session that's
/// past `expires_at` but whose last request landed in the
/// last 5 min must not surface as online.
#[test]
fn dto_is_not_online_when_expired_even_if_fresh() {
let past = Utc::now() - Duration::days(1);
let s = Session::from_raw(
Uuid::new_v4(),
Uuid::new_v4(),
"rt".to_string(),
past, // expires_at in the past
None,
None,
past,
false,
Uuid::new_v4(),
None,
None,
None,
SessionOrigin::Password,
Utc::now(), // last_seen_at fresh
);
let dto = SessionSummaryDto::from(s);
assert!(!dto.is_active, "expired session is not active");
assert!(
!dto.is_online,
"expired-but-fresh row must never render as online",
);
}
#[test]
fn fresh_session_has_last_seen_equal_to_created_at() {
// The DB default is `NOW()` and `Session::new` mirrors
// that with `Utc::now()` for BOTH columns — so a
// freshly-minted session immediately counts as "recently
// active" for the liveness gauges rather than showing up
// as long-idle for the first flush interval.
let s = base(false, None);
assert_eq!(s.created_at(), s.last_seen_at());
}
#[test]
fn from_raw_expired_session_is_not_active() {
let past = Utc::now() - Duration::days(1);
@@ -258,6 +435,7 @@ mod tests {
None,
None,
crate::domain::entities::session::SessionOrigin::Unknown,
past,
);
let dto = SessionSummaryDto::from(s);
assert!(!dto.is_active);
src/application/dtos/settings_dto.rs
+35 -5
View File
@@ -108,14 +108,15 @@ pub struct AdminResetPasswordDto {
pub new_password: String,
}
/// Query parameters for listing users
/// Query parameters for listing users. `/api/admin/users` used to
/// bifurcate on `?summary=` (flat `PublicUserDto` vs nested
/// `FullUserDto`); that split was retired — the endpoint now always
/// returns `FullUserDto`. Unknown query params are ignored, so
/// existing callers still passing `?summary=true` keep working.
#[derive(Debug, Serialize, Deserialize)]
pub struct ListUsersQueryDto {
pub limit: Option<i64>,
pub offset: Option<i64>,
/// Return only the fields rendered by the paginated management table.
/// Defaults to `false` so existing API clients keep the full user shape.
pub summary: Option<bool>,
}
/// Query parameters for the admin sessions listing.
@@ -163,10 +164,39 @@ pub struct DashboardStatsDto {
pub auth_enabled: bool,
pub oidc_configured: bool,
pub quotas_enabled: bool,
// User stats
// ── User accounts (static breakdown of auth.users) ──
// All four are counts of the SAME table under different
// predicates. `active`, `admin`, `external` are all subsets of
// `total`. `external` is disjoint from `admin` by DB constraint
// (`users_external_not_admin`). The dashboard renders these as
// one grouped section separate from the live-activity section
// below, so admins don't confuse "as-of-now row count" with
// "who's here right now".
pub total_users: i64,
pub active_users: i64,
pub admin_users: i64,
/// Grant-only accounts (magic-link / OIDC-only / OCM recipients).
/// Filtered out of `total_users` / `active_users` since those
/// columns count operational seats (see the SELECT comment). Here
/// as its own metric because operators of external-heavy
/// deployments (public shares, invited-collab shops) need to see
/// the invited population at a glance.
pub external_users: i64,
// ── Live activity (projection over auth.sessions) ──
// Both fields change minute-to-minute, unlike the user counts
// above which only move on register/deactivate/role-toggle.
// Same 5-min window as the Prometheus gauges
// (`oxicloud_sessions_online[_users]` in
// `session_liveness_gauges.rs`), computed via the shared
// `ONLINE_WINDOW` constant so per-user badges + aggregate
// counts + this dashboard number stay consistent by construction.
/// Distinct users behind non-revoked sessions active in the last
/// 5 min. Answers "how many humans are here right now?".
pub online_users: i64,
/// Non-revoked sessions active in the last 5 min. Answers "how
/// many concurrent connections must I serve?". Ratio
/// `online_sessions / online_users` is the multi-device factor.
pub online_sessions: i64,
// ── Per-drive-kind quota accounting ──
// One row per drive kind (personal, shared). Pre-dedup, logical
// file sizes summed from `drives.used_bytes` (personal rolls up
src/application/dtos/user_dto.rs
+378 -239
View File
@@ -1,5 +1,4 @@
use crate::domain::entities::user::User;
use crate::domain::repositories::user_repository::UserListEntry;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use smol_str::SmolStr;
@@ -7,287 +6,283 @@ use std::sync::Arc;
use utoipa::ToSchema;
use uuid::Uuid;
// ────────────────────────────────────────────────────────────────────────
// Three-layer user DTO family — see docs/plan/userdto-refactor.md.
//
// `PublicUserDto` — public identity. Every authenticated caller may see it.
// Returned by /api/users/{id}, share responses, group
// members, magic-link invitees, recipient enrichment.
// `FullUserDto` — `{ user: PublicUserDto, ...admin+self extras }`.
// Returned as `Vec<FullUserDto>` by /api/admin/users;
// embedded in `SelfUserDto`. Closest DTO to the
// `auth.users` row.
// `SelfUserDto` — `{ full: FullUserDto, ...self-only extras }`. Returned
// by /api/auth/me and by every AuthResponseDto path.
//
// Adding a field? Decide by audience:
// * Any authenticated caller may see it about another user → `PublicUserDto`.
// * Only admin (about another user) AND self (about self) → `FullUserDto`.
// * Only self about themselves → `SelfUserDto`.
// ────────────────────────────────────────────────────────────────────────
/// Public identity — what any authenticated caller may see about ANOTHER
/// user. Returned by `/api/users/{id}` and everywhere a user is
/// referenced by another surface (share responses, group members,
/// magic-link invitees, recipient enrichment).
///
/// This is the audience-narrowest DTO: adding a field here means every
/// authenticated caller can see it about every visible user. Fields that
/// are meaningful only to the subject themselves (preferences, session
/// state) or only to an admin (auth adoption signals) belong on
/// [`SelfUserDto`] or [`FullUserDto`] respectively.
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct UserDto {
pub struct PublicUserDto {
pub id: String,
/// Optional handle. `None` for users who have not claimed one
/// (externals, fresh email-only signups). Frontend display callers
/// should walk `username → given/family → email` as their fallback
/// chain. Omitted from JSON when None (consistent with the existing
/// given_name / family_name fields).
#[serde(skip_serializing_if = "Option::is_none")]
pub username: Option<String>,
pub email: String,
/// Role string ("admin" | "user"). Kept public because the sharee /
/// group-member vignette renders an admin badge.
pub role: String,
pub storage_quota_bytes: i64,
pub storage_used_bytes: i64,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub last_login_at: Option<DateTime<Utc>>,
pub active: bool,
/// Which trust chain minted this user's federation identity —
/// `"oidc" | "ocm" | "magic_link"` — or `None` for pure local
/// users. Load-bearing for "is this user OIDC?"-shape predicates:
/// use `federation_kind == "oidc"` rather than string-scraping
/// `federation_issuer`. Serialized only when populated.
///
/// Mirrors `auth.users.federation_kind` verbatim — same name at
/// DB, entity, and wire layers so there's no translation to reason
/// about. See docs/plan/ocm.md § Identity & auth model.
#[serde(skip_serializing_if = "Option::is_none")]
pub federation_kind: Option<String>,
/// The authority that mints this user's `federation_subject` —
/// issuer URL for OIDC (id_token `iss` claim), peer domain for
/// OCM, `null` for local users (password / OPAQUE only).
///
/// Renamed from `auth_provider` (which was a `String` with the
/// sentinel `"local"` for non-federated users, and a human-readable
/// label like `"MockSSO"` before Phase B). This shape mirrors the
/// `auth.users.federation_issuer` column directly: nullable when
/// there's no federation involved. FE predicates for "is this user
/// federated?" should read `federation_kind`, not
/// string-compare this value.
///
/// When populated, FE code that wants a friendly display label
/// looks this value up against `OidcProviderInfoDto.issuer →
/// provider_name` to render the deployment's configured display
/// name; falls back to the raw issuer for foreign IdPs / legacy
/// rows still holding a pre-Phase-B label.
#[serde(skip_serializing_if = "Option::is_none")]
pub federation_issuer: Option<String>,
/// Avatar payload (base64 data-URI up to 512 KiB). Public so a share
/// picker can render the recipient's face directly. Will move to a
/// dedicated avatar endpoint in a future refactor — this shape is
/// transitional.
pub image: Option<String>,
pub can_edit_image: bool,
/// `true` for grant-only external recipients (magic-link, OIDC-only,
/// future OCM federated). External users have no home folder and
/// can't own storage; their quota is always 0. Internal users
/// default to `false`.
/// future OCM federated). Renders the "external" badge on the vignette.
pub is_external: bool,
/// Optional first/given name. Populated from the OIDC `given_name`
/// claim at JIT provisioning, or via a profile-edit endpoint.
/// `None` until explicitly set — `skip_serializing_if = "Option::is_none"`
/// keeps the wire format compact for the common case.
/// Optional first/given name. Social identity.
#[serde(skip_serializing_if = "Option::is_none")]
pub given_name: Option<String>,
/// Optional last/family name. Same provenance + serde rules as
/// `given_name`.
/// Optional last/family name. Social identity.
#[serde(skip_serializing_if = "Option::is_none")]
pub family_name: Option<String>,
/// When the user first demonstrated control of their email (PR 23).
/// `None` = unverified (omitted from JSON). Stamped on the first
/// successful magic-link redemption or OIDC JIT with verified
/// claim. Idempotent — the original timestamp is preserved on
/// subsequent verifications.
#[serde(skip_serializing_if = "Option::is_none")]
pub email_verified_at: Option<DateTime<Utc>>,
/// User-chosen locale for server-rendered surfaces (emails,
/// future authenticated HTML). `None` = no preference (the server
/// resolves to `OXICLOUD_DEFAULT_LOCALE` when rendering). Round-trips
/// through `/api/auth/me` and `PATCH /api/auth/me/profile`.
#[serde(skip_serializing_if = "Option::is_none")]
pub preferred_locale: Option<String>,
/// Whether the user wants an email when someone shares a resource
/// with them. `true` (default) = receive share-notification mails;
/// `false` = grants are still created but no email is sent. Honored
/// only on the plain-notification path — magic-link first-invitations
/// to brand-new external users always send, otherwise the recipient
/// could never claim the share. Round-trips through `/api/auth/me`
/// and `PATCH /api/auth/me/profile`.
pub notify_on_share: bool,
/// Opaque UI preferences bag. Cross-device store for pure UI
/// toggles (hide dotfiles, view mode, sidebar collapse, …). The
/// server never inspects the contents — this DTO field just echoes
/// what was PATCHed via `PATCH /api/auth/me/profile`. Shape is a
/// JSON object; the frontend defines the keys it cares about (see
/// `frontend/src/lib/stores/preferences.svelte.ts`). Always present
/// on the wire; empty bag is `{}`, never `null`.
pub ui_preferences: serde_json::Value,
/// Mirrors `auth.users.force_password_change_at_next_login`. Set
/// TRUE by the admin password-reset flow (see
/// `AuthApplicationService::admin_reset_password`) and cleared by
/// a successful self-service `POST /api/auth/change-password`.
///
/// Populated only by the `/api/auth/me` handler and the login
/// response minter (via a distinct code path). `From<User>` — used
/// by admin listings, share-recipient responses, group-member DTOs,
/// etc. — leaves it at `false`. The flag is a per-session-account
/// concern (does *this* user need to change their password before
/// they can proceed?), not a general user attribute worth
/// surfacing on every list row.
///
/// The load-bearing consumer is the SPA's session store: on
/// startup and after every refresh, `/me` returns the current
/// flag value and the SPA's nav-guard blocks navigation to
/// anything but the change-password surface until it flips
/// back to false. Backend enforcement is separate (see the
/// `require_no_password_change_pending` middleware) — this DTO
/// field is what the SPA reads to render the mandatory-mode UI.
/// Presence signal — TRUE when the server observed a request on any
/// of this user's non-revoked sessions within the last
/// [`ONLINE_WINDOW`](crate::application::dtos::session_dto::ONLINE_WINDOW)
/// (5 min). Sourced from an EXISTS subquery when the DTO is built
/// from a list-projection path; single-user endpoints that don't
/// enrich presence ship `false`.
#[serde(default)]
pub force_password_change: bool,
/// TRUE when the account has a local Argon2id `password_hash` on
/// file. Distinct from `federation_kind`: an OIDC-linked account
/// (`federation_kind == "oidc"`) can ALSO carry a local password if
/// it was set at signup or later — a hybrid posture. The SPA
/// gates the profile page's change-password card on this flag,
/// so hybrid users can rotate their local password even though
/// they normally sign in via SSO.
///
/// Populated only by the `/api/auth/me` handler. `From<User>` in
/// this file leaves it `false` — other UserDto emitters (admin
/// listings, share-recipient responses, group members) do not
/// need to surface per-user credential state.
#[serde(default)]
pub has_password: bool,
/// TRUE when the caller's current session carries a DPoP JWK
/// thumbprint (`session.dpop_jkt IS NOT NULL`). Sourced from the
/// caller's JWT `cnf.jkt` claim — `is_some()` means the session
/// was bound at token-mint time.
///
/// Populated only by the `/api/auth/me` handler; other UserDto
/// emitters leave it `false`. The SPA reads this on `session.load()`
/// to skip a redundant `POST /api/auth/dpop/bind` call when the
/// session is already bound (which would 409 and log noisily under
/// the audit stream — see the `already_bound` reject). Only the
/// OIDC / magic-link redirect flows land here as `false` on first
/// visit; password login binds at session-mint time so the very
/// first `/me` after login already reports `true`.
#[serde(default)]
pub is_dpop_bound: bool,
pub is_online: bool,
}
/// Compact row returned by the paginated admin user table.
/// Full user record — public identity + all fields BOTH an admin
/// (viewing another user) AND the subject themselves may see. Returned
/// as `Vec<FullUserDto>` by `/api/admin/users`; embedded in
/// [`SelfUserDto`] for `/api/auth/me`.
///
/// Account-detail fields deliberately do not appear here. In particular,
/// omitting `image` and `ui_preferences` prevents a 100-row page from turning
/// into tens of MiB when users have uploaded avatars. `GET /api/admin/users/:id`
/// remains the full-detail endpoint.
/// This is the DTO closest to the underlying `auth.users` row. Adding a
/// field here means an admin looking at any user can see it, and the
/// subject themselves can see it in their `/me` response — but the field
/// stays off the public [`PublicUserDto`] surface.
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct AdminUserSummaryDto {
pub id: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub username: Option<String>,
pub email: String,
pub role: String,
pub storage_quota_bytes: i64,
pub storage_used_bytes: i64,
pub last_login_at: Option<DateTime<Utc>>,
pub active: bool,
/// See `UserDto::federation_kind` — same semantics, same wire spelling.
pub struct FullUserDto {
/// Public identity — same set every authenticated caller sees.
pub user: PublicUserDto,
/// Which trust chain minted this user's federation identity —
/// `"oidc" | "ocm" | "magic_link"` — or `None` for pure local users.
/// Kept off `PublicUserDto` because a peer's federation kind is a
/// soft org-affiliation leak; only self + admin need it.
#[serde(skip_serializing_if = "Option::is_none")]
pub federation_kind: Option<String>,
/// See `UserDto::federation_issuer` — same semantics, same wire spelling.
/// The authority that minted this user's `federation_subject` —
/// issuer URL for OIDC (id_token `iss` claim), peer domain for OCM,
/// `None` for local users. Same rationale as `federation_kind`.
#[serde(skip_serializing_if = "Option::is_none")]
pub federation_issuer: Option<String>,
pub is_external: bool,
/// TRUE when the user has a server-verifiable password on file
/// (`password_hash IS NOT NULL`). The admin table uses this
/// alongside `federation_issuer` and `opaque_registered` to render
/// the user's full capability set: a `password` chip lights up
/// here, an OIDC provider name renders the SSO badge, an
/// envelope-on-file flips the OPAQUE chip. A user with none of
/// the three is passwordless (magic-link only — the SPA renders
/// a distinct `passwordless` chip in that case). Admin-only
/// exposure — see the DTO doc for why this isn't on `UserDto`.
#[serde(default)]
/// Subject's own locale preference. Only THEY or an admin managing
/// them needs this — other callers use their own locale.
#[serde(skip_serializing_if = "Option::is_none")]
pub preferred_locale: Option<String>,
/// When the user first demonstrated control of their email. Trust
/// signal — meaningful to admin (auditing verification status) and
/// to self (own record), but not to a share picker rendering a
/// vignette.
#[serde(skip_serializing_if = "Option::is_none")]
pub email_verified_at: Option<DateTime<Utc>>,
/// Row bookkeeping.
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
/// Activity signal — private to the subject; admin sees it too.
pub last_login_at: Option<DateTime<Utc>>,
/// Account-active flag — a deactivated user couldn't reach `/me`
/// anyway, but admin needs to see it.
pub active: bool,
/// Storage quotas — personal financials. Admin manages others';
/// self sees own.
pub storage_quota_bytes: i64,
pub storage_used_bytes: i64,
/// TRUE when the account has a server-verifiable password
/// (`password_hash IS NOT NULL`). Kept off `PublicUserDto` because
/// per-user auth adoption leaks through directory endpoints.
pub has_password: bool,
/// Mirrors `UserListEntry::opaque_registered` — TRUE when the user
/// has an OPAQUE envelope on file. Surfaced on the admin table so
/// operators can see per-user rollout progress during the
/// migration window. **Admin-only exposure**: this field is NOT
/// on `UserDto` — putting it there would leak adoption status
/// through every user-directory-adjacent endpoint (share targets,
/// group members, invite listings). `#[serde(default)]` keeps
/// older SPA builds tolerant of the added field.
#[serde(default)]
/// TRUE when the user has an OPAQUE envelope on file.
pub opaque_registered: bool,
/// Mirrors `UserListEntry::opaque_migrated` — TRUE when the user
/// has completed at least one successful OPAQUE login. Distinct
/// from `opaque_registered`: an admin can invalidate the envelope
/// (`clear_registration`) leaving the user registered=false but
/// with a historical migrated=true; the SPA's admin table shows
/// both so this operational nuance is visible.
#[serde(default)]
/// TRUE when the user has completed ≥1 successful OPAQUE login.
/// Distinct from `opaque_registered`: an admin can invalidate the
/// envelope leaving the user registered=false but with historical
/// migrated=true.
pub opaque_migrated: bool,
}
impl From<UserListEntry> for AdminUserSummaryDto {
fn from(entry: UserListEntry) -> Self {
Self {
id: entry.id.to_string(),
username: entry.username,
email: entry.email,
role: entry.role.to_string(),
storage_quota_bytes: entry.storage_quota_bytes,
storage_used_bytes: entry.storage_used_bytes,
last_login_at: entry.last_login_at,
active: entry.active,
federation_kind: entry.federation_kind,
federation_issuer: entry.federation_issuer,
is_external: entry.is_external,
has_password: entry.has_password,
opaque_registered: entry.opaque_registered,
opaque_migrated: entry.opaque_migrated,
}
}
/// Self view — everything the caller may see about themselves.
/// Returned by `/api/auth/me` and by every `AuthResponseDto` path
/// (login / refresh / OIDC callback / magic-link redemption).
///
/// Composed on top of [`FullUserDto`] so `/me` and `/admin/users` share
/// the SAME "full profile" contract for the fields both need — new
/// self+admin-visible fields go on `FullUserDto` and both endpoints get
/// them together. Fields here are pure self-scoped state: preferences,
/// session-scoped flags, and caller-scoped permissions.
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct SelfUserDto {
/// Full profile — same shape as one row of `/api/admin/users`.
pub full: FullUserDto,
/// Opaque UI preferences bag — my own UI state. Cross-device store
/// for pure UI toggles (view mode, sidebar collapse, hide dotfiles,
/// …). The server never inspects the contents. Always present on
/// the wire; empty bag is `{}`, never `null`.
pub ui_preferences: serde_json::Value,
/// Whether I want share-notification emails.
pub notify_on_share: bool,
/// Session-scoped: my current session carries a DPoP thumbprint.
/// SPA reads this on `session.load()` to skip a redundant
/// `POST /api/auth/dpop/bind` when the session is already bound.
pub is_dpop_bound: bool,
/// Admin-set temp-password gate — SPA nav guard blocks everything
/// but `/change-password` until this flips back. Cleared by a
/// successful `POST /api/auth/change-password`.
pub force_password_change: bool,
/// Caller-scoped permission: can I edit my own avatar? `false` for
/// OIDC users whose avatar comes from the IdP. Only meaningful when
/// caller == subject; nonsense on any other DTO.
pub can_edit_image: bool,
}
impl From<User> for UserDto {
fn from(user: User) -> Self {
// `user` is owned and dropped here, so every owned field is MOVED out
// via `into_parts` rather than cloned through the borrowing accessors —
// the accessor form deep-cloned `image` (a data URI up to 512 KiB) and
// the whole `ui_preferences` JSON tree on every `/api/auth/me` and admin
// user listing (benches/ROUND20.md §A2). The two derived values read the
// entity before the move.
impl PublicUserDto {
/// Construct a `PublicUserDto` from a `User` entity + an explicit
/// `is_online` signal.
///
/// **Why not `From<User>`?** The `User` entity models a row in
/// `auth.users`; `is_online` is a cross-table lookup on
/// `auth.sessions` (see the EXISTS subquery in
/// `list_users_with_derived_flags` and `get_user_with_derived_flags`
/// on the user repo). A `From<User>` impl couldn't compute it
/// honestly — it would have to ship a `false` default that lies to
/// the FE presence dot on every emitter that didn't remember to
/// override. Making presence a required constructor argument
/// removes that footgun: every callsite has to declare its intent.
///
/// Two shapes at the callsite:
///
/// - Presence matters (single-user `/api/users/{id}`, list
/// projections, self-view): pair with
/// `user_storage.get_user_with_derived_flags(id)` and pass
/// `flags.is_online`.
/// - Presence is out of scope (register / update-profile response,
/// post-mutation echo where the FE ignores the field): pass
/// `false` with a short comment explaining why. The receiver's
/// presence read is a no-op — no dot lights up on the stale
/// value.
pub fn new(user: User, is_online: bool) -> Self {
let role = format!("{}", user.role());
let can_edit_image = !user.is_oidc_user();
// has_password is derivable from the entity — read before the
// move. Cheap (bool from Option::is_some), no extra DB round-
// trip, so From<User> can populate it uniformly rather than
// leaving it false and requiring per-call-site backfill.
let has_password = user.has_password();
let p = user.into_parts();
Self {
id: p.id.to_string(),
username: p.username,
email: p.email,
role,
storage_quota_bytes: p.storage_quota_bytes,
storage_used_bytes: p.storage_used_bytes,
image: p.image,
is_external: p.is_external,
given_name: p.given_name,
family_name: p.family_name,
is_online,
}
}
}
impl FullUserDto {
/// Construct a `FullUserDto` from a `User` entity plus the DB-derived
/// flags the entity doesn't carry (`has_password`, OPAQUE flags,
/// `is_online`). Both are typically produced together by the users
/// list repo projection.
///
/// Not a `From` impl because it takes two arguments; not a `From
/// <(User, UserDerivedFlags)>` because that reads awkwardly at
/// callsites — `FullUserDto::build(user, flags)` is clearer.
pub fn build(
user: User,
flags: crate::domain::repositories::user_repository::UserDerivedFlags,
) -> Self {
let role = format!("{}", user.role());
let p = user.into_parts();
Self {
user: PublicUserDto {
id: p.id.to_string(),
username: p.username,
email: p.email,
role,
image: p.image,
is_external: p.is_external,
given_name: p.given_name,
family_name: p.family_name,
is_online: flags.is_online,
},
federation_kind: p.federation_kind.map(|k| k.as_str().to_string()),
federation_issuer: p.federation_issuer,
preferred_locale: p.preferred_locale,
email_verified_at: p.email_verified_at,
created_at: p.created_at,
updated_at: p.updated_at,
last_login_at: p.last_login_at,
active: p.active,
// NULL on both fields for local users (no federation wired).
// FE predicates use `!!federation_kind` for "is federated?" —
// no "local" sentinel string; the null tells the whole story.
federation_kind: p.federation_kind.map(|k| k.as_str().to_string()),
federation_issuer: p.federation_issuer,
image: p.image,
storage_quota_bytes: p.storage_quota_bytes,
storage_used_bytes: p.storage_used_bytes,
has_password: flags.has_password,
opaque_registered: flags.opaque_registered,
opaque_migrated: flags.opaque_migrated,
}
}
}
impl SelfUserDto {
/// Assemble the `/me` response from a `FullUserDto` plus the two
/// session-scoped booleans that can't be derived from `User` alone:
/// the caller's DPoP-binding state (from the JWT `cnf.jkt` claim)
/// and the admin-set force-password-change flag (from the auth
/// service's cache).
///
/// The other self-only fields (`ui_preferences`, `notify_on_share`,
/// `can_edit_image`) come from `User` and are read off the entity
/// before it's moved into the FullUserDto; this method takes those
/// as explicit parameters so the caller can decide when to read
/// them (typically at the same point they read the DPoP-binding
/// state).
pub fn build(
full: FullUserDto,
ui_preferences: serde_json::Value,
notify_on_share: bool,
is_dpop_bound: bool,
force_password_change: bool,
can_edit_image: bool,
) -> Self {
Self {
full,
ui_preferences,
notify_on_share,
is_dpop_bound,
force_password_change,
can_edit_image,
is_external: p.is_external,
given_name: p.given_name,
family_name: p.family_name,
email_verified_at: p.email_verified_at,
preferred_locale: p.preferred_locale,
notify_on_share: p.notify_on_share,
ui_preferences: p.ui_preferences,
// Defaults to false. The `/me` handler + the login-response
// minter populate this via a distinct code path (a
// repo read that goes through the auth service's cache);
// admin listings and other UserDto consumers deliberately
// leave it false — the flag is per-session-account state,
// not a general user attribute.
force_password_change: false,
has_password,
// Populated only by `/api/auth/me` — the handler overlays
// the caller's session's actual DPoP binding state after
// this `From<User>` runs. Other UserDto emitters leave
// this at `false` (they lack session context).
is_dpop_bound: false,
}
}
}
// ────────────────────────────────────────────────────────────────────────
// End of three-layer user DTO family.
// ────────────────────────────────────────────────────────────────────────
#[derive(Debug, Serialize, Deserialize, Clone, ToSchema)]
pub struct LoginDto {
/// Identifier the user typed. Accepts BOTH a username (no `@`) and
@@ -425,7 +420,12 @@ impl UpdateProfileDto {
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
pub struct AuthResponseDto {
pub user: UserDto,
/// Full self view — identical shape to `/api/auth/me`. Every
/// login / refresh / OIDC-callback / magic-link redemption ships
/// this so the SPA's post-auth state matches its post-`/me` state
/// (no UI race between `AuthResponseDto` and the first `/me`
/// fetch). See `docs/plan/userdto-refactor.md` § Endpoint mapping.
pub user: SelfUserDto,
pub access_token: String,
pub refresh_token: String,
pub token_type: String,
@@ -562,7 +562,7 @@ pub struct OidcProviderInfoDto {
/// users JIT-provisioned via this IdP.
///
/// Populated so the frontend can resolve display: when
/// `UserDto.federation_issuer` equals this `issuer`, render
/// `PublicUserDto.federation_issuer` equals this `issuer`, render
/// `provider_name` as the human-friendly label (avoids showing raw
/// issuer URLs like `https://sso.example.com/realms/main` in the
/// admin badge / profile view). Falls back to the raw issuer when
@@ -606,3 +606,142 @@ pub struct OidcUserInfoDto {
pub name: Option<String>,
pub groups: Vec<String>,
}
#[cfg(test)]
mod three_layer_quarantine {
use super::*;
use serde_json::Value;
/// Structural-quarantine guard for `SelfUserDto`. The self-only
/// bag (`ui_preferences`, `notify_on_share`, `is_dpop_bound`,
/// `force_password_change`, `can_edit_image`) MUST live at the
/// top level, NOT nested inside `.full` or `.full.user`. If a
/// future refactor accidentally moves one of them down, the
/// wire shape leaks it through every `PublicUserDto` /
/// `FullUserDto` emitter (share responses, group members,
/// `/api/admin/users`, magic-link invitees) — exactly what the
/// three-layer split exists to prevent. Fails loudly here.
#[test]
fn self_only_fields_stay_at_top_level_of_self_user_dto() {
let self_dto = SelfUserDto {
full: FullUserDto {
user: PublicUserDto {
id: "00000000-0000-0000-0000-000000000001".into(),
username: None,
email: "self@example.invalid".into(),
role: "user".into(),
image: None,
is_external: false,
given_name: None,
family_name: None,
is_online: false,
},
federation_kind: None,
federation_issuer: None,
preferred_locale: None,
email_verified_at: None,
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
last_login_at: None,
active: true,
storage_quota_bytes: 0,
storage_used_bytes: 0,
has_password: true,
opaque_registered: false,
opaque_migrated: false,
},
ui_preferences: serde_json::json!({}),
notify_on_share: true,
is_dpop_bound: false,
force_password_change: false,
can_edit_image: true,
};
let json: Value = serde_json::to_value(&self_dto).expect("SelfUserDto serialises");
assert!(
json.get("ui_preferences").is_some(),
"top-level ui_preferences"
);
assert!(
json.get("full")
.expect("full block")
.get("ui_preferences")
.is_none(),
"ui_preferences must NOT appear inside `.full`"
);
assert!(
json.pointer("/full/user/ui_preferences").is_none(),
"ui_preferences must NOT appear inside `.full.user`"
);
// Same guard for the other self-only fields.
for k in [
"notify_on_share",
"is_dpop_bound",
"force_password_change",
"can_edit_image",
] {
assert!(json.get(k).is_some(), "{k} at top level");
assert!(
json.pointer(&format!("/full/{k}")).is_none(),
"{k} must NOT nest in .full"
);
assert!(
json.pointer(&format!("/full/user/{k}")).is_none(),
"{k} must NOT nest in .full.user"
);
}
}
/// Structural-quarantine guard for `FullUserDto`. Admin-visible
/// extras (`has_password`, OPAQUE flags, `federation_*`,
/// `last_login_at`, `active`, quotas, `preferred_locale`,
/// `email_verified_at`) MUST live at the top level of
/// `FullUserDto`, NOT inside `.user`. If a future refactor
/// accidentally lifts one of them onto `PublicUserDto` (the
/// embedded `user` field), it leaks through `/api/users/{id}`
/// and every other public directory endpoint.
#[test]
fn admin_only_fields_stay_at_top_level_of_full_user_dto() {
let full = FullUserDto {
user: PublicUserDto {
id: "00000000-0000-0000-0000-000000000002".into(),
username: Some("bob".into()),
email: "bob@example.invalid".into(),
role: "user".into(),
image: None,
is_external: false,
given_name: None,
family_name: None,
is_online: false,
},
federation_kind: None,
federation_issuer: None,
preferred_locale: None,
email_verified_at: None,
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
last_login_at: None,
active: true,
storage_quota_bytes: 10_737_418_240,
storage_used_bytes: 0,
has_password: true,
opaque_registered: false,
opaque_migrated: false,
};
let json: Value = serde_json::to_value(&full).expect("FullUserDto serialises");
for k in [
"has_password",
"opaque_registered",
"opaque_migrated",
"last_login_at",
"active",
"storage_quota_bytes",
"storage_used_bytes",
] {
assert!(json.get(k).is_some(), "{k} at top level of FullUserDto");
assert!(
json.pointer(&format!("/user/{k}")).is_none(),
"{k} must NOT nest in .user"
);
}
}
}
src/application/ports/auth_ports.rs
+39 -10
View File
@@ -3,7 +3,6 @@ use crate::domain::entities::app_password::AppPassword;
use crate::domain::entities::device_code::DeviceCode;
use crate::domain::entities::session::Session;
use crate::domain::entities::user::User;
use crate::domain::repositories::user_repository::UserListEntry;
use std::sync::Arc;
use uuid::Uuid;
@@ -63,6 +62,14 @@ pub struct TokenClaims {
/// The DPoP middleware reads it from the already-validated token
/// (no DB round trip) to enforce "bound session → proof required".
pub dpop_jkt: Option<String>,
/// Session identifier — the `auth.sessions.id` this access token
/// was minted for. Read by the auth middleware to stamp
/// per-session liveness via [`LastSeenTracker`](crate::infrastructure::services::last_seen_tracker)
/// with no DB round trip. `None` for tokens minted by builds
/// that predate the `sid` claim (backward compat during rollout;
/// harmless — the missing sid just means no stamp fires, and
/// the token still authenticates normally).
pub sid: Option<Uuid>,
}
/// Port for JWT token operations.
@@ -81,6 +88,7 @@ pub trait TokenServicePort: Send + Sync + 'static {
fn generate_access_token(
&self,
user: &User,
session_id: Option<Uuid>,
dpop_jkt: Option<&str>,
) -> Result<String, DomainError>;
@@ -114,6 +122,36 @@ pub trait UserStoragePort: Send + Sync + 'static {
/// Gets a user by ID
async fn get_user_by_id(&self, id: Uuid) -> Result<User, DomainError>;
/// Fetch the full `User` + [`UserDerivedFlags`] in one query. See
/// [`UserRepository::get_user_with_derived_flags`](crate::domain::repositories::user_repository::UserRepository::get_user_with_derived_flags)
/// for the contract and the rationale for the single-query shape.
async fn get_user_with_derived_flags(
&self,
id: Uuid,
) -> Result<
(
User,
crate::domain::repositories::user_repository::UserDerivedFlags,
),
DomainError,
>;
/// Paginated admin user listing with derived flags. See
/// [`UserRepository::list_users_with_derived_flags`](crate::domain::repositories::user_repository::UserRepository::list_users_with_derived_flags)
/// for the contract and rationale.
async fn list_users_with_derived_flags(
&self,
limit: i64,
offset: i64,
include_external: bool,
) -> Result<
Vec<(
User,
crate::domain::repositories::user_repository::UserDerivedFlags,
)>,
DomainError,
>;
/// Batch-loads users by id. Order is unspecified; missing ids are
/// silently dropped. Used by group-recipient expansion in
/// `RecipientNotificationService` to avoid N+1 lookups when notifying
@@ -155,15 +193,6 @@ pub trait UserStoragePort: Send + Sync + 'static {
include_external: bool,
) -> Result<Vec<User>, DomainError>;
/// Narrow user-list projection for management tables. Keeps heavyweight
/// account-detail fields off the database and JSON hot path.
async fn list_user_summaries(
&self,
limit: i64,
offset: i64,
include_external: bool,
) -> Result<Vec<UserListEntry>, DomainError>;
/// Searches users by username or email (SQL ILIKE) with a limit.
/// See [`list_users`] for the meaning of `include_external`.
async fn search_users(
src/application/ports/blob_reference_ports.rs
+313
View File
@@ -0,0 +1,313 @@
//! `BlobReferenceSource` — the extension point that teaches ref-counting
//! and the consistency jobs about a table holding blob references.
//!
//! Before this port, "who references this hash" was hardcoded SQL in two
//! places (`dedup_gc`'s reap predicate and `blobs_consistency`'s refcount
//! recompute), both naming `storage.files` and `storage.chunk_manifests`
//! directly. Any new blob-owning table therefore risked silent orphaning:
//! `dedup_gc` sees `ref_count = 0`, or a manifest with no `storage.files`
//! row behind it, and reaps live content.
//!
//! See `docs/plan/derived-blobs.md` for the design and the coverage matrix.
//!
//! # Two levels, and why a source may span both
//!
//! [`DedupService::add_reference`] bumps `chunk_manifests.ref_count` first
//! and only falls back to `storage.blobs.ref_count`. So a reference lands
//! on whichever counter its hash names, and the two must be recomputed
//! separately — mixing them double-counts, systematically:
//!
//! * A **Blob** (`chunk_manifests.file_hash`) is "the content of a file".
//! * A **Chunk** (`storage.blobs.hash`) is a physical byte payload.
//! * For a single-chunk Blob the two hashes are **equal**, because both are
//! BLAKE3 over the same bytes. That aliasing is why today's chunk-level
//! recompute carries a `NOT EXISTS` clause, and why every fragment here
//! must be level-correct rather than merely plausible.
//!
//! A source is not confined to one level: [`RefLevel::Chunk`] and
//! [`RefLevel::Manifest`] fragments are requested independently, and
//! `storage.files` legitimately contributes to both — a manifest-less
//! legacy row references a chunk, a CDC row references a Blob.
//!
//! # Why SQL fragments rather than a per-hash count
//!
//! `blobs_consistency` recomputes refcounts with **one query per page**,
//! the expected count inlined as correlated subqueries. Asking each source
//! for a count per hash would turn that into `sources × rows` round-trips —
//! a catastrophic regression on a table with millions of rows. So sources
//! contribute a *fragment* that the registry sums into the existing page
//! query, and [`BlobReferenceSource::count_references`] exists only for the
//! on-demand path (`dedup_gc` checking a single reap candidate, where the
//! candidate set is already filtered to `ref_count = 0`).
use std::sync::Arc;
use async_trait::async_trait;
use crate::domain::errors::DomainError;
/// Which counter a source's references land on.
///
/// Not a property of the source — see the module docs; the same source may
/// contribute at both levels.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum RefLevel {
/// References a physical chunk. Feeds `storage.blobs.ref_count`.
Chunk,
/// References a Blob via its manifest. Feeds
/// `chunk_manifests.ref_count`.
Manifest,
}
impl RefLevel {
/// Both levels, for callers that sweep each in turn.
pub const ALL: [RefLevel; 2] = [RefLevel::Chunk, RefLevel::Manifest];
/// Stable name for logs and consistency-finding fields.
pub fn as_str(self) -> &'static str {
match self {
RefLevel::Chunk => "chunk",
RefLevel::Manifest => "manifest",
}
}
}
/// One table that holds references to blob hashes.
///
/// Implementors are registered on [`BlobReferenceRegistry`] during DI.
/// Adding a blob-owning table **without** registering it is the failure
/// this port exists to prevent.
#[async_trait]
pub trait BlobReferenceSource: Send + Sync {
/// Short stable identifier for logs and consistency-finding `source`
/// fields — `"files"`, `"chunks"`, `"content_derived"`, …
///
/// Stable across releases: log aggregators key off it.
fn source_name(&self) -> &'static str;
/// A correlated-subquery fragment counting this source's references
/// **at `level`** to `outer_hash_expr`, or `None` when this source
/// holds no references at that level.
///
/// `outer_hash_expr` is the SQL expression naming the hash of the row
/// being recomputed — `"b.hash"` when sweeping `storage.blobs`,
/// `"m.file_hash"` when sweeping `storage.chunk_manifests`. The
/// fragment must be a parenthesised scalar subquery so the registry can
/// join fragments with `+`.
///
/// **Identifiers only.** `outer_hash_expr` is supplied by the sweep, never
/// by a request; no fragment may interpolate caller input.
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String>;
/// Existence form of [`Self::ref_count_sql`] — a boolean fragment, true
/// when this source holds at least one reference at `level`.
///
/// Defaults to `(<count>) > 0`. Override when the source can express a
/// short-circuiting `EXISTS`, which the planner can stop at the first
/// matching row: `dedup_gc`'s reap predicate runs this per candidate
/// manifest, and a heavily-deduplicated blob has many referrers, so
/// counting all of them where existence would do is a real regression.
fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
self.ref_count_sql(level, outer_hash_expr)
.map(|fragment| format!("{fragment} > 0"))
}
/// Count of references this source holds on `blob_hash`, across both
/// levels.
///
/// **On-demand path only** — `dedup_gc` checking a single reap
/// candidate. The consistency sweeps must use [`Self::ref_count_sql`];
/// calling this per row would turn one query per page into
/// `sources × rows` round-trips.
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;
/// Iterate the hashes this source references, paged by the
/// implementation's natural cursor (typically a primary key).
///
/// Used by `backend_consistency` to walk the backend against the union
/// of all sources. Returns the page plus the cursor to resume from,
/// `None` when exhausted.
async fn list_referenced_blobs(
&self,
cursor: Option<Vec<u8>>,
limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;
/// Notification that `dedup_gc` reaped this blob.
///
/// Sources maintaining a denormalised refcount can clean up here. Most
/// leave the default noop — the mapping row is normally deleted by the
/// owning service's `on_blob_deleted` hook instead.
fn on_blob_reaped(&self, _blob_hash: &str) {}
}
/// The set of registered [`BlobReferenceSource`]s.
///
/// Assembled once during DI and shared (`Arc`) by `dedup_gc` and the
/// consistency jobs, so all three agree on what "referenced" means.
#[derive(Default)]
pub struct BlobReferenceRegistry {
sources: Vec<Arc<dyn BlobReferenceSource>>,
}
impl BlobReferenceRegistry {
pub fn new() -> Self {
Self::default()
}
/// Register a source. Order is irrelevant — fragments are summed and
/// counts added.
pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>) {
self.sources.push(source);
}
pub fn sources(&self) -> &[Arc<dyn BlobReferenceSource>] {
&self.sources
}
/// The summed SQL expression counting every source's references at
/// `level` to `outer_hash_expr`.
///
/// Returns `"0"` when no source contributes at this level, which keeps
/// the caller's query valid without a special case.
pub fn ref_count_expr(&self, level: RefLevel, outer_hash_expr: &str) -> String {
let fragments: Vec<String> = self
.sources
.iter()
.filter_map(|s| s.ref_count_sql(level, outer_hash_expr))
.collect();
if fragments.is_empty() {
"0".to_string()
} else {
fragments.join("\n + ")
}
}
/// Predicate selecting rows that **no** registered source references at
/// `level` — i.e. reap candidates.
///
/// Returns `None` when no source contributes at this level, and callers
/// **must** treat that as "refuse to act" rather than substituting a
/// default. The natural default would be the sum-equals-zero form, which
/// on an empty registry reduces to `0 = 0` — vacuously true for every
/// row, i.e. "delete everything". Returning `None` makes that
/// unrepresentable at the call site instead of merely discouraged.
pub fn no_reference_predicate(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
let fragments: Vec<String> = self
.sources
.iter()
.filter_map(|s| s.ref_exists_sql(level, outer_hash_expr))
.collect();
if fragments.is_empty() {
return None;
}
Some(format!("NOT ({})", fragments.join("\n OR ")))
}
/// Total references held on `hash` across every source.
///
/// On-demand path only — see [`BlobReferenceSource::count_references`].
pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError> {
let mut total = 0u64;
for source in &self.sources {
total = total.saturating_add(source.count_references(hash).await?);
}
Ok(total)
}
/// Fan out a reap notification to every source.
pub fn notify_reaped(&self, blob_hash: &str) {
for source in &self.sources {
source.on_blob_reaped(blob_hash);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
struct Stub {
name: &'static str,
chunk: Option<&'static str>,
manifest: Option<&'static str>,
count: u64,
}
#[async_trait]
impl BlobReferenceSource for Stub {
fn source_name(&self) -> &'static str {
self.name
}
fn ref_count_sql(&self, level: RefLevel, outer: &str) -> Option<String> {
let tmpl = match level {
RefLevel::Chunk => self.chunk?,
RefLevel::Manifest => self.manifest?,
};
Some(tmpl.replace("{outer}", outer))
}
async fn count_references(&self, _blob_hash: &str) -> Result<u64, DomainError> {
Ok(self.count)
}
async fn list_referenced_blobs(
&self,
_cursor: Option<Vec<u8>>,
_limit: usize,
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
Ok((Vec::new(), None))
}
}
fn registry() -> BlobReferenceRegistry {
let mut r = BlobReferenceRegistry::new();
r.register(Arc::new(Stub {
name: "a",
chunk: Some("(SELECT 1 WHERE {outer} = 'x')"),
manifest: None,
count: 2,
}));
r.register(Arc::new(Stub {
name: "b",
chunk: Some("(SELECT 2 WHERE {outer} = 'y')"),
manifest: Some("(SELECT 3 WHERE {outer} = 'z')"),
count: 5,
}));
r
}
#[test]
fn chunk_level_sums_every_contributing_source() {
let expr = registry().ref_count_expr(RefLevel::Chunk, "b.hash");
assert!(expr.contains("b.hash = 'x'"), "{expr}");
assert!(expr.contains("b.hash = 'y'"), "{expr}");
assert!(expr.contains('+'), "fragments must be summed: {expr}");
}
/// A source returning `None` for a level must contribute nothing there —
/// this is what keeps manifest-only tables out of the chunk recompute,
/// where they would double-count against the single-chunk hash alias.
#[test]
fn manifest_level_skips_non_contributing_sources() {
let expr = registry().ref_count_expr(RefLevel::Manifest, "m.file_hash");
assert!(expr.contains("m.file_hash = 'z'"), "{expr}");
assert!(!expr.contains('+'), "only one source contributes: {expr}");
}
/// An empty level must still yield a valid scalar expression, so callers
/// need no special case before a registry is fully populated.
#[test]
fn empty_level_yields_zero_literal() {
let r = BlobReferenceRegistry::new();
assert_eq!(r.ref_count_expr(RefLevel::Chunk, "b.hash"), "0");
}
#[tokio::test]
async fn total_references_adds_across_sources() {
assert_eq!(registry().total_references("deadbeef").await.unwrap(), 7);
}
}
src/application/ports/blob_storage_ports.rs
+19 -2
View File
@@ -244,11 +244,28 @@ pub trait BlobStorageBackend: Send + Sync + 'static {
///
/// * `cursor` — opaque continuation token from a prior call, or
/// `None` to start from the beginning. Format is per-backend
/// (local = last path visited; S3 = continuation token; Azure
/// = list marker); callers treat it as opaque.
/// — **the last blob hash returned by the previous page**.
/// Enumeration resumes strictly AFTER that hash.
///
/// This is deliberately NOT an opaque backend token. Callers may
/// synthesise a cursor from any hash they hold, which is what lets a
/// consistency sweep merge-join this stream against a
/// `storage.blobs` walk and resume both sides from one checkpoint.
/// An opaque token would force the backend side to re-enumerate from
/// the beginning on every resume.
/// * `limit` — soft cap on batch size; backends may return
/// fewer (e.g. end of a shard directory).
///
/// **Entries MUST be returned in ascending hash order**, and pages must
/// be contiguous in that order. Every shipped backend already satisfies
/// this — local sorts within each shard and walks shards `00`..`ff`
/// (the shard IS the hash prefix, so that is globally sorted); S3 and
/// Azure list lexicographically by key, and `blobs/<xx>/<hash>` sorts
/// identically to `<hash>`. It is stated here because the merge-join in
/// `backend_consistency` depends on it: an unordered backend would
/// silently emit bogus `blob_missing_from_backend` findings at
/// `data_loss` severity.
///
/// Returns `(entries, next_cursor)`. `next_cursor = None` means
/// enumeration is complete. Each `BackendBlobEntry` carries the
/// hash + optional mtime for grace-window filtering.
src/application/ports/dedup_ports.rs
+44
View File
@@ -24,6 +24,39 @@ pub struct BlobMetadataDto {
pub content_type: Option<String>,
}
/// A stored server-derived artifact: which blob holds it, and what it is.
///
/// `content_type` is carried so the read path can set the response header
/// without byte-sniffing the payload, which is what it does today.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DerivedBlobRef {
pub blob_hash: String,
pub content_type: String,
}
/// What the derived tier knows about one `(source_hash, kind, variant)`.
///
/// Three answers, not two. `Option<DerivedBlobRef>` could only say
/// "have it" or "don't", which collapses the two cases that matter most
/// to a caller deciding whether to spend a decode:
///
/// * [`Missing`](Self::Missing) — never attempted. Derive it.
/// * [`NotDerivable`](Self::NotDerivable) — attempted, and the attempt
/// is known to be a waste for this content: the transcode came out
/// larger than the original, the source cannot be decoded, the source
/// is over the decode ceiling. Serve the original and do not retry.
/// * [`Found`](Self::Found) — here are the bytes.
///
/// Only failures that are deterministic in the CONTENT may be recorded
/// as `NotDerivable`. A timeout or an I/O error is a property of the
/// moment; persisting one would mark a good image underivable forever.
#[derive(Debug, Clone)]
pub enum DerivedLookup {
Missing,
NotDerivable,
Found(DerivedBlobRef),
}
/// Result of a deduplication store operation.
#[derive(Debug, Clone)]
pub enum DedupResultDto {
@@ -83,6 +116,17 @@ pub trait DedupPort: Send + Sync + 'static {
/// Check if a blob with the given hash exists.
async fn blob_exists(&self, hash: &str) -> bool;
/// Look up a server-derived artifact by the content it was derived from.
///
/// The read counterpart of `store_derived_blob`. Returns `None` when no
/// such variant has been derived yet — the caller then renders it.
async fn find_derived_blob(
&self,
source_hash: &str,
kind: &str,
variant: &str,
) -> Option<DerivedBlobRef>;
/// Get metadata for a blob.
async fn get_blob_metadata(&self, hash: &str) -> Option<BlobMetadataDto>;
src/application/ports/mod.rs
+1
View File
@@ -1,6 +1,7 @@
pub mod auth_ports;
pub mod authorization_ports;
pub mod blob_lifecycle;
pub mod blob_reference_ports;
pub mod blob_storage_ports;
pub mod cache_ports;
pub mod calendar_ports;
src/application/ports/thumbnail_ports.rs
+18
View File
@@ -77,6 +77,12 @@ pub enum ThumbnailFormat {
}
impl ThumbnailFormat {
/// Every format, for callers that must handle all of them — notably
/// `thumb_derived_import`, which claims one sidecar extension per format
/// and would silently strand a codec if this list and the write path
/// drifted apart.
pub const ALL: [ThumbnailFormat; 2] = [ThumbnailFormat::Webp, ThumbnailFormat::Jpeg];
/// Stable name, byte-identical to the derived `Debug` output (see
/// [`ThumbnailSize::as_str`] — same ETag-stability contract).
pub fn as_str(self) -> &'static str {
@@ -94,6 +100,18 @@ impl ThumbnailFormat {
}
}
/// Media type, for `content_derived_blobs.content_type` and for any
/// response serving these bytes.
///
/// Beside `ext` deliberately: the two must agree, and an extension
/// without a matching media type is how a WebP ends up labelled JPEG.
pub fn mime(self) -> &'static str {
match self {
ThumbnailFormat::Webp => "image/webp",
ThumbnailFormat::Jpeg => "image/jpeg",
}
}
/// Pick the output format from a request `Accept` header: WebP when the
/// client advertises `image/webp`, JPEG otherwise. A plain substring check
/// is sufficient — no client sends `image/webp;q=0`, and every WebP-capable
src/application/ports/transcode_ports.rs
+6
View File
@@ -85,9 +85,15 @@ pub trait ImageTranscodePort: Send + Sync + 'static {
/// Returns `(content, mime_type, was_transcoded)`.
/// If transcoding is not beneficial (output larger than input), returns the
/// original content with `was_transcoded = false`.
///
/// `source_hash` is the BLAKE3 of the original content, which is how the
/// durable derived tier is keyed. `None` restricts the implementation to
/// its local cache — correct for callers with no hash (external mounts),
/// and the behaviour of every caller before that tier existed.
async fn get_transcoded(
&self,
file_id: &str,
source_hash: Option<&str>,
original_content: Bytes,
original_mime: &str,
target_format: OutputFormat,

Some files were not shown because too many files have changed in this diff Show More