cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #656 from EdouardVanbelle/feat/storage-key-rotation

Browse Source
This commit is contained in:
Dionisio Pozo
2026-08-02 18:10:45 +02:00
committed by GitHub
parent 0660ac1e6c 6ab87d83e8
commit 74866bea81
60 changed files with 6494 additions and 707 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/.vitepress/config.mts
+1
View File
@@ -168,6 +168,7 @@ export default defineConfig({
{ text: "ReBAC Authorization", link: "/architecture/rebac-authorization" },
{ text: "Share Integration", link: "/architecture/share-integration" },
{ text: "Storage Quotas", link: "/architecture/storage-quotas" },
{ text: "Backend Storage", link: "/architecture/backend-storage" },
{ text: "File and Blob lifecycle", link: "/architecture/file-and-blob-lifecycle" },
{ text: "ReBAC & Authorization", link: "/architecture/rebac-authorization" },
{ text: "User lifecycle", link: "/architecture/user-lifecycle" },
docs/architecture/backend-storage.md
+421
View File
@@ -0,0 +1,421 @@
# Backend Storage
Reference for implementors of storage backends, encryption layers,
consistency checks, and migration/rotation jobs.
The user-facing "how to configure S3" guide lives in
[`docs/config/env.md`](../config/env.md); the operational plan
history is in [`docs/plan/storage-multi-entry.md`](../plan/storage-multi-entry.md)
and [`docs/plan/storage-key-rotation.md`](../plan/storage-key-rotation.md).
This page is the "what the code actually does and why" reference.
---
## Supported backends
Three concrete implementations of `BlobStorageBackend` ship in the
tree today. All go through the same `EncryptedBlobBackend` wrapper
(see §6) so encryption, header format, BLAKE3 rescue, smart-skip
probe, and lifecycle behaviour are uniform.
| Backend | `StorageBackendType` | Env key prefix | Config surface | Impl |
|---|---|---|---|---|
| **Local filesystem** | `Local` | `OXICLOUD_STORAGE_<name>_ROOT_DIR` | Root directory on the host FS. Shard tree `<root>/.blobs/<xx>/`. Atomic replace via tempfile + `rename(2)`. | [`local_blob_backend.rs`](../../src/infrastructure/services/local_blob_backend.rs) |
| **S3-compatible** | `S3` | `OXICLOUD_STORAGE_<name>_BUCKET` + `_REGION` + `_ACCESS_KEY` + `_SECRET_KEY` + optional `_ENDPOINT_URL` + `_FORCE_PATH_STYLE` | AWS S3, Cloudflare R2, Backblaze B2, MinIO, DigitalOcean Spaces, Wasabi — anything speaking the S3 API. | [`s3_blob_backend.rs`](../../src/infrastructure/services/s3_blob_backend.rs) |
| **Azure Blob Storage** | `Azure` | `OXICLOUD_STORAGE_<name>_ACCOUNT_NAME` + `_ACCOUNT_KEY` (or `_SAS_TOKEN`) + `_CONTAINER` + optional `_ENDPOINT_URL` | Azure Blob Storage; `_ENDPOINT_URL` targets Azurite (local emulator) or private endpoints. | [`azure_blob_backend.rs`](../../src/infrastructure/services/azure_blob_backend.rs) |
Every entry is declared in `OXICLOUD_STORAGE_ENTRIES` (comma-separated
list of names). The active entry is stored in `admin_settings` and
switched via `oxicloud --select-storage <name>` on the command line
or automatically at the end of a successful `backend_migration`.
Non-active entries stay reachable through the multi-entry API (test,
audit, migrate-into).
Adding a new backend = new struct implementing `BlobStorageBackend`
+ a new arm on `StorageBackendType` + a new branch in
`entry_backend::build_base_backend`. The wrapper stack, key rotation,
consistency check, and migration all pick it up for free (see §6).
---
## 1. The `file → blob → chunk` model
Three separate concerns, three separate storage layers:
```text
PostgreSQL Backend
(Local/S3/Azure)
┌─────────────────────┐ ┌──────────────────────┐ ┌──────────────┐
│ storage.files │ │ storage.blobs │ │ │
│ - id │────▶│ - hash (BLAKE3) │────────▶│ .blobs/xx/ │
│ - name │ │ - size, ref_count │ │ <hash>.blob│
│ - folder_id │ │ - content_type │ │ │
│ - blob_hash │ │ │ │ │
└─────────────────────┘ └──────────────────────┘ └──────────────┘
▲
│ (for chunked files only)
┌───────┴──────────────┐
│ storage.chunk_manifests
│ - file_hash │
│ - chunk_hashes[] │
│ - total_size │
└──────────────────────┘
```
**File** (`storage.files`) — DB row. Has a name, a folder, a drive, an
owner, a size, a MIME type. Points at exactly one **content descriptor**:
either a whole-file blob or a chunk manifest, both keyed by BLAKE3 hash.
Files are what users see; nothing about them lives on the backend.
**Blob** (`storage.blobs`) — DB row + **physical bytes on a backend**.
Content-addressable: the row's primary key is `hash = BLAKE3(plaintext_bytes)`.
`ref_count` is the number of live references (files or manifests) pointing
at this blob; when it reaches zero, `dedup_gc` removes both the row and
the physical bytes (after a grace window). Every backend lays blobs out
under a two-char shard directory: `.blobs/<first-two-hex>/<full-hash>.blob`
— Local's filesystem tree, S3's object keys, Azure's blob names. See
`LocalBlobBackend::object_key` and its S3/Azure counterparts.
**Chunk** (`storage.chunk_manifests`) — content-defined-chunking (CDC)
subdivision of a file. When an upload exceeds the whole-file threshold,
the ingest pipeline splits it into ≤1 MiB chunks and stores each as
its own blob. The manifest records the chunk sequence + total size;
downloads stream through the manifest, fetching each chunk blob in
order. **Chunk blobs are indistinguishable from whole-file blobs** at
the backend layer — they're just blobs. `storage.chunk_manifests` is
pure PG state with no backend bytes.
**Why this matters for backend implementors:** you only ever deal with
blobs. You never see file paths, folder trees, chunks, manifests, or
users. Your API is `(hash) → put/get/exists/delete bytes`. Everything
else is orchestrated above.
---
## 2. The v1 blob header (`OXCPT`)
Every blob written since the key-rotation implementation landed
starts with a 15-byte header:
```text
byte 0..4 "OXCPT" magic marker (5 bytes)
byte 5..6 0x00 0x01 format version (2 bytes, big-endian u16)
byte 7..14 <key_fp> key fingerprint (8 bytes)
```
Then either:
- **plaintext-v1**: `key_fp` is all zeros; the header is followed by
raw plaintext bytes.
- **encrypted-v1**: `key_fp` is `SHA-256(key_material)[..8]`; the
header is followed by a 12-byte AES-GCM nonce, then the ciphertext,
then the 16-byte GCM tag. Total overhead: **43 bytes**.
Rendered visually via `xxd -l 15 <blob>`:
```text
4f58 4350 54 00 01 00 00 00 00 00 00 00 00 OXCPT.......... ← plaintext-v1
4f58 4350 54 00 01 15 f3 8f 80 2c ae 2c 50 OXCPT......,.,P ← encrypted-v1 with key_fp = 15:f3:8f:80:2c:ae:2c:50
```
Fingerprints are rendered the same colon-hex form (`15:f3:…:50`)
everywhere they appear: boot log, admin panel pair chain, `xxd`
inspection, `oxicloud --fingerprint <base64>` CLI, and the rotate /
migration audit lines. That means an admin can cross-reference by
eye — same string means same key.
### Why the header exists
Before the key-rotation implementation, blobs had no header.
Reading a legacy blob meant "try to decrypt with the
currently-configured key; if it works, it's encrypted; if not,
it's plaintext." This has three problems:
1. **Ambiguity on key change.** If the operator changed the key,
every existing blob became unreadable — nothing on disk said
which key was used.
2. **No way to smart-skip.** A rotation or migration couldn't tell
whether a target blob was already in the desired state without
reading and re-hashing every byte.
3. **No forward compatibility.** Any future format change (E2E,
compression, alternate cipher) would need magic-byte detection
layered on top.
The header solves all three. Magic bytes disambiguate legacy from
v1. Version bytes let us evolve the format. `key_fp` lets us
identify *which* key was used without trying every candidate.
### Future: end-to-end encryption
The current `EncryptedV1` variant is **server-side encryption at
rest** — the server holds the key. E2E encryption (client holds the
key, server sees only ciphertext) is designed to slot in as a new
version:
```text
byte 5..6 0x00 0x02 format version = 2 (E2E)
byte 7..14 <key_fp> hint identifying the client key
byte 15.. <opaque body> client-encrypted payload, opaque to server
```
The read/write pipeline stays the same at the backend layer — the
server just passes bytes through. `read_dispatch` grows a match arm
for version 2 that skips server decryption entirely. This is why the
version bytes exist as a distinct field: the file format is
extensible without a magic-byte rewrite.
### BLAKE3 rescue for legacy plaintext
Deployments that predate the key-rotation implementation have blobs
on disk with no `OXCPT` header — the pipeline calls these `Legacy`
format. Reads try to
decrypt with each pair-list key; if all fail, a last-resort branch
computes `BLAKE3(raw_bytes)` and returns the bytes as plaintext iff
the digest matches the expected hash. Zero-false-positive by
construction (content-addressable proof). Emits
`encryption.legacy_plaintext_rescued` audit lines so operators can
spot which blobs still need re-writing. See
`encrypted_blob_backend.rs::read_dispatch` last branch.
The rescue is transparent — downloads, thumbnails, consistency
checks, and rotation all benefit. The first time `backend_rotate`
sweeps a legacy-plaintext blob, it classifies it as `Legacy`,
rewrites it through the wrapper, and the resulting blob has a proper
v1 header. After one rotate pass, rescue never fires again.
---
## 3. Key rotation
### Pair-list config
Encryption is configured per storage entry via a comma-separated
**pair list**:
```bash
OXICLOUD_STORAGE_<name>_ENCRYPTION_KEY='aes_gcm:<b64_key1>,aes_gcm:<b64_key2>,none:'
```
The **head** is the leftmost entry — writes use this key. Every entry
in the list is available for reads (fallback loop). `none:` in the
list declares "raw plaintext is a legitimate on-disk shape for this
backend" — the leftmost `none:` becomes the head if placed first,
otherwise it enables the plaintext-fallback branch of `read_dispatch`.
### Rotate job (`backend_rotate`)
Recoverable job that iterates every blob on the current backend and
rewrites any whose header doesn't match the current head format.
Decision table via `BlobFormat::classify` compared against
`EncryptedBlobBackend::head_format`:
| Current on-disk | Head | Action |
|---|---|---|
| `EncryptedV1 { key_fp: A }` | `EncryptedV1 { key_fp: A }` | skip |
| `EncryptedV1 { key_fp: A }` | `EncryptedV1 { key_fp: B }` | rewrite (key change) |
| `PlaintextV1` | `EncryptedV1 { key_fp: X }` | rewrite (encrypt in place) |
| `EncryptedV1` | `PlaintextV1` | rewrite (decrypt in place) |
| `Legacy` | anything v1 | rewrite (upgrade header) |
Reports per-blob outcomes (`rewritten`, `skipped`, `failed`) and the
final head format/fp in the run's `extra_stats`. Each rewrite goes
through `put_blob_from_bytes_replace` — see §6.
### Head-key vs fallback keys
- **Head** — used for **writes only**. Rotating just means "declare
a new head and run `backend_rotate` to catch up existing bytes."
- **Fallback keys** — read-only. Kept in the pair list until every
blob on disk has been rewritten under the head, then safe to
remove from `.env`.
The admin panel shows the whole pair chain per entry with the head
badged; after a successful rotation with `failed=0`, non-head keys
can be safely dropped.
---
## 4. Blob consistency (`blobs_consistency`)
Read-only recoverable job that walks `storage.blobs` and reports
divergence between the DB registry and the physical backend.
### Shallow mode (default)
Per row:
- `blob_exists(hash)` on the active backend → if false, record
`blob_missing_from_backend` (severity `data_loss`)
- Compare `ref_count` against the actual reference count computed
from `SUM` over `storage.files.blob_hash` + `chunk_manifests.chunk_hashes[]`
→ if mismatch, record `refcount_mismatch` (severity `inconsistent`)
Cost: one existence probe + one aggregate SQL per row. Fast on
S3/Azure (single HEAD).
### Deep mode (`?deep=true`)
Adds a full read of every blob:
- Stream the blob through `EncryptedBlobBackend::get_blob_stream`
(strips header, decrypts if needed, applies BLAKE3 rescue for
legacy plaintext)
- Recompute `BLAKE3(plaintext)` and compare against the row's `hash`
- If bytes match: no finding
- If bytes differ: record `blob_corrupted` (silent bit-rot)
- If the read pipeline errors (missing key, unreadable header,
network failure): record `blob_unreadable`
Both `blob_corrupted` and `blob_unreadable` carry the list of files
that reference the offending hash (`affected_files`) so the operator
can decide whether to re-upload or drop.
The `deep` flag is persisted in the run's `params` on Fresh open so
it survives a mid-run restart — resume continues in deep mode
without the operator re-specifying it.
---
## 5. Backend migration (`backend_migration`)
Recoverable job that copies every blob from the current active
backend to a target entry, then hot-swaps the active pointer on
completion.
### Cursor + resume
Iterates blobs in ascending hash order. Checkpoints the last-visited
hash to `jobs.recoverable_runs.cursor` after each batch. On restart,
`WHERE hash > cursor` picks up from where the previous session
stopped. `MigrationProgress` counter is seeded from
`stats.scanned_count` on resume so the maintenance banner shows
continued progress, not a fresh `0`.
### Smart-skip via `head_check`
Before each copy, `target.head_check(hash)` returns one of:
- `Match` → target blob already carries the current head's
format+`key_fp`; skip. Debug-log
`backend_migration.blob_skipped_head_match`.
- `Mismatch(current_format)` → target blob exists with a different
header (legacy shape, old key, plaintext vs encrypted). Overwrite
via `put_blob_from_bytes_replace`. Info-log
`backend_migration.blob_overwritten` — this is the **legacy
skip-check residual repair** log line: exactly the blobs that
historically escaped re-encryption because the pre-rotation
migration path had an `if target exists { skip }` short-circuit
(see §5 below).
- `Absent` → fresh write. Info-log `backend_migration.blob_written`.
The check is one 15-byte range-read via `get_blob_range_stream(hash, 0, Some(15))`
on the inner backend — cheap on Local (`pread`), cheap on S3/Azure
(single GET with `Range: bytes=0-14`).
### Legacy skip-check residual repair (the historic bug)
Before the key-rotation implementation, the migration path hit
`if target.blob_exists(hash) { continue }` before every copy — it
silently skipped blobs already present on the target, even if the
target's current head key was different from the key used at the
historical write time. Result: mixed-key target backends, and blobs
that suddenly failed to read when the old key was later removed
from the pair list.
The rotation work removed the app-layer skip. Backend-agnostic
write-side fixes followed: every backend's
`put_blob_from_bytes_replace` was overridden to bypass the internal
HEAD-probe skip (`S3BlobBackend`, `AzureBlobBackend`) or use
`O_CREAT|O_TRUNC` via tempfile+rename (`LocalBlobBackend`). See §6
for the full contract.
### Failure gate
`finish_completed` refuses to flip the active-backend pointer if
`failed > 0`. Emits `storage_migration.aborted` audit, clears
readonly (source stays active — writes safe there), and returns
`RunOutcome::Failed`. Operator inspects findings, then either
retries (walk short-circuits on head-format matches → cheap
re-attempt), fixes the source, or explicitly accepts the partial
via `oxicloud --select-storage <target>`.
---
## 6. Implementor contract — the `BlobStorageBackend` trait
### Always wrapped in `EncryptedBlobBackend`
Every entry is built via `entry_backend::build_entry_backend_typed`,
which unconditionally wraps the raw backend in `EncryptedBlobBackend`
— regardless of whether the entry has an encryption key. A `none:`
head gets an `EncryptedBlobBackend` with `head_cipher = None` that
writes plaintext-v1 blobs. This means:
- Every read goes through `read_dispatch` → magic-byte inspection →
v1 or legacy branch → decrypt (if needed) → BLAKE3 rescue (if
legacy plaintext).
- Every write goes through the wrapper's write path → prepend the
15-byte header → encrypt with head cipher (or leave plaintext) →
hand to inner backend.
- The **inner backend never sees plaintext application content** —
only the header-wrapped or encrypted body.
Your job as a backend implementor: implement `BlobStorageBackend`
for opaque byte payloads. Never inspect or modify the bytes.
### Required overrides
The trait provides defaults but they lie about correctness for the
rotate/migrate use case. **Every production backend MUST override
`put_blob_from_bytes_replace`.**
| Method | Semantics | Default | Must override? |
|---|---|---|---|
| `put_blob` | Content-addressed upload. May skip if hash exists (dedup, idempotency). | — | yes |
| `put_blob_from_bytes` | Same, in-memory bytes. May skip if exists. | — | yes |
| `put_blob_from_bytes_unsynced` | Unconditional PUT. Durability not required on return; caller batches `sync_blobs`. | delegates to `put_blob_from_bytes` (WRONG for skip-backends) | recommended (dedup fast-path) |
| **`put_blob_from_bytes_replace`** | **Unconditional overwrite. Durable on return.** Used by rotate + migration. | delegates to `put_blob_from_bytes` (WRONG for every current backend) | **yes** |
| `get_blob_stream` | Full-blob stream. | — | yes |
| `get_blob_range_stream` | Range stream — used by the 15-byte `head_check` probe. | — | yes |
| `blob_exists` | Cheap presence check. | — | yes |
| `delete_blob` | Physical deletion. | — | yes |
| `sync_blobs` | Fsync barrier for `_unsynced` writes. | no-op | Local only |
| `initialize` | Called at boot. Verify creds, create shard dirs, reap tempfiles. | no-op | yes |
The `put_blob*` skip-if-exists semantics is correct for **uploads**
— dedup hits should short-circuit. It's wrong for **rewrites** —
rotate needs to overwrite the header, migration needs to overwrite
with new-key ciphertext. `put_blob_from_bytes_replace` is the
escape hatch. On S3/Azure it delegates to `put_blob_from_bytes_unsynced`
(unconditional PUT, durable on return). On Local it does
write-to-tempfile + `rename(2)` + fsync — atomic overwrite on POSIX.
### Reference implementations
- [`LocalBlobBackend`](../../src/infrastructure/services/local_blob_backend.rs)
— filesystem tree under a configurable root. Shard directory
`.blobs/<xx>/`. Tempfiles named `<hash>.replace.<pid>.<counter>.tmp`,
reaped at boot by `initialize`.
- [`S3BlobBackend`](../../src/infrastructure/services/s3_blob_backend.rs)
— AWS SDK v2. Object key `<xx>/<hash>.blob` under a configurable
bucket. `put_blob_from_bytes_replace` delegates to
`put_blob_from_bytes_unsynced` (skips the HEAD probe).
- [`AzureBlobBackend`](../../src/infrastructure/services/azure_blob_backend.rs)
— Azure SDK. Same shape as S3.
Read them side-by-side before implementing a new backend — the three
follow the same skeleton so the diff is where your backend's
semantics genuinely differ.
---
## Related docs
- [File and blob lifecycle →](./file-and-blob-lifecycle.md) — the
hook system that observes file/blob CRUD events.
- [Background jobs →](./jobs.md) — how to plug in a new job tenant.
- [Storage quotas →](./storage-quotas.md) — usage accounting layer
(independent of the backend).
- [Storage multi-entry plan →](../plan/storage-multi-entry.md) —
design history: why multiple entries, why hot-swap migration.
- [Storage key rotation plan →](../plan/storage-key-rotation.md) —
design history and slice-by-slice implementation notes.
docs/architecture/index.md
+1
View File
@@ -74,4 +74,5 @@ src/
- [Caching Architecture →](/architecture/caching)
- [Resource Listing API →](/architecture/resource-listing)
- [Storage Quotas →](/architecture/storage-quotas)
- [Backend Storage →](/architecture/backend-storage)
- [Background Jobs →](/architecture/jobs)
docs/architecture/jobs.md
+1 -1
View File
@@ -41,7 +41,7 @@ operator benefit.
| Job name | Cadence | Force semantic | Service |
|---|---|---|---|
| `trash_cleanup` | 24 h (hardcoded in DI, no env var yet) | ignored | [`trash_cleanup_service.rs`](../../src/infrastructure/services/trash_cleanup_service.rs) |
| `storage_reconcile`| `OXICLOUD_STORAGE_USAGE_RECONCILE_SECS` (default 600s, min 30s) | ignored | [`storage_usage_service.rs`](../../src/application/services/storage_usage_service.rs) |
| `usage_reconcile` | `OXICLOUD_STORAGE_USAGE_RECONCILE_SECS` (default 600s, min 30s) | ignored | [`storage_usage_service.rs`](../../src/application/services/storage_usage_service.rs) |
| `dedup_gc` | on-demand only (trash cleanup runs it inline as its tail step) | `force=true` → `garbage_collect_force()` (skip orphan grace) | [`dedup_service.rs`](../../src/infrastructure/services/dedup_service.rs) |
| `grant_cleanup` | `OXICLOUD_GRANT_CLEANUP_INTERVAL_HOURS` (default 24h) — feature-gated by `OXICLOUD_GRANT_CLEANUP_ENABLED` | `force=true` → `purge(Some(0))` (grace_days=0) | [`grant_cleanup_service.rs`](../../src/infrastructure/services/grant_cleanup_service.rs) |
docs/config/admin-settings.md
+1 -1
View File
@@ -79,7 +79,7 @@ The admin storage tab operates on the **named storage entries** declared in `.en
| `POST` | `/api/admin/storage/migration/resume` | Resume a paused run (target read from `params.target_name`, no body needed) |
| `GET` | `/api/admin/storage/migration` | Poll the current run's progress |
Runs are recoverable — status, cursor, and per-blob failure findings all live in `jobs.recoverable_runs` / `jobs.run_findings`. The same run history is browsable via `GET /api/admin/jobs/storage_migration/runs`.
Runs are recoverable — status, cursor, and per-blob failure findings all live in `jobs.recoverable_runs` / `jobs.run_findings`. The same run history is browsable via `GET /api/admin/jobs/backend_migration/runs`.
### Cutover flow (moving the active pointer)
docs/config/env.md
+5 -3
View File
@@ -103,8 +103,7 @@ Each declared name `<N>` then reads its own set of per-entry variables:
| `OXICLOUD_STORAGE_<N>_AZURE_CONTAINER` | — | Azure-only: blob container name (required when backend=azure) |
| `OXICLOUD_STORAGE_<N>_AZURE_SAS_TOKEN` | — | Azure-only: SAS token (alternative to account key) |
| `OXICLOUD_STORAGE_<N>_AZURE_ENDPOINT_URL` | — | Azure-only: custom endpoint (Azurite, private deployments) |
| `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY` | — | Base64-encoded 32-byte AES-256 key. **Presence implies encryption is enabled** on this entry — no separate enable flag. Bad base64 / wrong length aborts boot. |
| `OXICLOUD_STORAGE_<N>_ENCRYPTION_CIPHER` | `aes-256-gcm` when `_ENCRYPTION_KEY` is set | Cipher choice for this entry. Only `aes-256-gcm` is accepted today (future-proofing knob — the enum is ready for a second cipher, the implementation still hardcodes AES-256-GCM). Setting the cipher without a key aborts boot. |
| `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY` | — | Comma-separated list of `<cipher>:<base64 key>` pairs (or bare `<base64 key>`, which defaults to `aes-256-gcm`). **Presence implies encryption is enabled** on this entry — no separate enable flag. The LAST pair wins on writes; every pair is a candidate for reads. Supported ciphers: `aes-256-gcm` and `none` (empty-key sentinel used at pair-list head/tail for encrypt/decrypt-in-place rotations). Bad base64, wrong length, duplicate keys, or multiple `none` pairs abort boot. See [Storage key rotation](../plan/storage-key-rotation.md) for rotation recipes. |
**Fail-fast rules** (boot aborts with actionable message):
@@ -125,7 +124,10 @@ OXICLOUD_STORAGE_s3_prod_S3_BUCKET=my-oxicloud-bucket
OXICLOUD_STORAGE_s3_prod_S3_REGION=us-east-1
OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY=…
OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY=…
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=… # openssl rand -base64 32
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:… # openssl rand -base64 32
# Rotation window (two pairs, last wins on writes):
# OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<OLD>,aes-256-gcm:<NEW>
```
## Storage Backend (DEPRECATED — legacy single-backend)
docs/guide/backend-storage.md
+64 -1
View File
@@ -40,15 +40,78 @@ A few things to know:
Any backend can be encrypted at rest by adding an encryption key to it:
```
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=… # base64 of 32 random bytes
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:… # base64 of 32 random bytes
```
A key can be generated from **Settings → Storage → Generate key** in the admin panel. Set the same key on a new backend during migration and OxiCloud re-encrypts the data as it copies.
The bare shorthand `OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=<key>` (no `aes-256-gcm:` prefix) also works and is treated as AES-256-GCM. Use the explicit form once you have more than one key in the list — see [Rotating an encryption key](#rotating-an-encryption-key) below.
::: warning
If you lose the encryption key, the data encrypted with it is unrecoverable. Store the key somewhere as safe as you'd store a database backup.
:::
### Rotating an encryption key
When it's time to rotate a key — after a suspected leak, on a periodic policy, or after a staff turnover — you don't need to provision a second bucket. Add the new key **alongside** the old one; the server writes with the new key while reads keep working through the old one; then a background job re-encrypts existing data under the new key; then you drop the old key.
Step by step:
1. Generate a new key:
```
openssl rand -base64 32
```
2. Append it to the entry's key list. **Order matters — the NEW key goes LAST.**
```
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<OLD>,aes-256-gcm:<NEW>
```
3. Restart the server. New uploads are now encrypted with the new key; existing files still open normally because the old key is still in the list.
4. On the **Storage** tab, click **Rotate encryption key** on the entry. This dispatches a background job that re-encrypts every existing file under the new key. All operations keep working during rotation — uploads, browsing, downloads, sharing.
5. Wait for the job to complete. Progress shows in a top banner and on the **Jobs** tab.
6. Once the entry card says "*Rotation complete — safe to remove the old key*", remove the OLD key from the list:
```
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<NEW>
```
7. Restart the server. Rotation is done.
::: warning
Do NOT remove the old key before the rotation job reports "safe to remove". Any file still encrypted under the old key would become unreadable.
:::
### Encrypting a backend that started unencrypted
Same shape as key rotation, using the `none:` sentinel to represent "the current writes are plaintext":
1. Generate a new key.
2. Add it AFTER `none:`:
```
OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=none:,aes-256-gcm:<NEW>
```
3. Restart. New uploads are encrypted; existing plaintext files stay readable.
4. Trigger **Rotate encryption key** on the entry.
5. Once the entry card says the rotation is done, remove `none:` from the list; restart.
### Decrypting an encrypted backend
The symmetric flow — add `none:` AFTER the current key, restart, rotate, drop the old key:
```
OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=aes-256-gcm:<KEY>,none:
```
After the rotation job completes and the entry card says done, remove the AES pair (or the whole variable) and restart. All files on that entry are now plaintext.
## Checking a backend
Once a backend is declared, it appears on the admin **Storage** tab as a card:
docs/plan/consistency-check.md
+1 -1
View File
@@ -565,7 +565,7 @@ CREATE SCHEMA IF NOT EXISTS admin;
CREATE TABLE jobs.recoverable_runs (
id UUID PRIMARY KEY,
job_name TEXT NOT NULL, -- 'consistency_blobs', 'storage_migration', 'reextract_audio', ...
job_name TEXT NOT NULL, -- 'consistency_blobs', 'backend_migration', 'reextract_audio', ...
status TEXT NOT NULL, -- Running / Paused / Completed / Failed / CancelRequested
started_at TIMESTAMPTZ NOT NULL,
last_progress_at TIMESTAMPTZ NOT NULL,
docs/plan/job-registry.md
+4 -4
View File
@@ -716,7 +716,7 @@ rationale + the merges/separations that fall out of the rule.
| `files_consistency` | `storage.files` | file UUID | `parent_folder_trashed` (live file under trashed folder), `missing_blob` (severity `data_loss` — `blob_hash` present in neither `storage.blobs` nor `storage.chunk_manifests`), `chunk_missing` (severity `data_loss` — manifest exists but points at chunks absent from `storage.blobs`; typical dedup GC race), `blob_size_mismatch` (denormalised `files.size` diverges from the authoritative size — manifest first, blob fallback) | Shipped Slice 6, CDC-aware Slice 10. Handles both storage paths: `storage.chunk_manifests` (post-Apr-2026 FastCDC ingest, dominant path) and `storage.blobs` (pre-CDC whole-file blob, legacy fallback). Physical backend-existence checks (chunk bytes actually on disk) belong in `storage_consistency`. Room to grow: `drive_id_parent_mismatch`, mime-type reconciliation. |
| `storage_consistency` | Storage backend (fs / S3) | object key / path | Each blob has a `storage.blobs` row (orphan detection) | `?deep=true` adds re-BLAKE3 + mime sniff. Orphan-side of the old bidirectional blob check + former `blob_integrity` + former `thumbnail_consistency`. |
| `grants_consistency` (future) | `storage.role_grants` | grant UUID | subject/resource/granted_by exist | |
| `storage_migration` | `storage.blobs` (source) → target backend | blob hash | Copy bytes; failures → `stats.failed_blobs` (and eventually `jobs.run_findings`) | Retires `Arc<RwLock<MigrationState>>` in `migration_job.rs`. |
| `backend_migration` | `storage.blobs` (source) → target backend | blob hash | Copy bytes; failures → `stats.failed_blobs` (and eventually `jobs.run_findings`) | Retires `Arc<RwLock<MigrationState>>` in `migration_job.rs`. |
| `reextract_audio` | `storage.files` where audio | file UUID | Re-run audio-tag parser, upsert `audio_metadata` | Retires synchronous admin-request execution. |
| `reextract_image` | `storage.files` where image/video | file UUID | Re-run EXIF/container date parser, upsert capture date | Same shape as reextract_audio. |
| `consistency_batch` (wrapper) | Iterates registered `*_consistency` jobs | — (JobHandler, not RecoverableJobHandler) | Sequentially triggers each sub-job; `?deep=true` propagates | Shipped Slice 5. One-click "run all" without per-job clicks; exclusivity via `job_name` prevents concurrent batches from stepping on each other. Batch itself always returns `Ok` — child failures land in `outcome.extra.per_check[<name>].outcome`. |
@@ -729,7 +729,7 @@ SELECT + one UPDATE. Kept as its own admin endpoint; do NOT fold into
### Verification (Part 2)
1. **Compile + schema-migration idempotence.**
2. **Fresh run:** `POST /api/admin/jobs/storage_migration/trigger` → new row with
2. **Fresh run:** `POST /api/admin/jobs/backend_migration/trigger` → new row with
`status='Running'`, `cursor=NULL`.
3. **Concurrent trigger:** second `POST` while the first is running
returns the SAME `run_id` (idempotent, DB unique index enforces).
@@ -787,7 +787,7 @@ reference for any external tool that still expects the old paths:
| Legacy (retired) | Replacement |
|---|---|
| `POST /admin/internal/trigger-sweep` | `POST /admin/jobs/storage_reconcile/trigger` |
| `POST /admin/internal/trigger-sweep` | `POST /admin/jobs/usage_reconcile/trigger` |
| `POST /admin/internal/trigger-gc?force=X` | `POST /admin/jobs/dedup_gc/trigger?force=X` |
| `POST /admin/internal/trigger-grant-cleanup?force=X` | `POST /admin/jobs/grant_cleanup/trigger?force=X` |
@@ -816,7 +816,7 @@ complete. Rough shape:
│ Name Cadence Last run Status Actions │
│ ───────────────────────────────────────────────────────────────────│
│ trash_cleanup every 24 h 3h ago ok [Run] │
│ storage_reconcile every 10 m 4m ago ok [Run] │
│ usage_reconcile every 10 m 4m ago ok [Run] │
│ dedup_gc on-demand 1d ago ok [Run] │
│ grant_cleanup every 24 h never — [Run] │
│ drives_consistency on-demand never — [Run] │
docs/plan/storage-key-rotation.md
+609
View File
@@ -0,0 +1,609 @@
# Plan — Storage encryption-key rotation
## Context
Today an encrypted storage entry declares a single key:
```env
OXICLOUD_STORAGE_s3_prod_BACKEND=s3
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=<base64 32 bytes>
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_CIPHER=aes-256-gcm # optional, defaults to aes-256-gcm
```
Blobs land at `<hash>.blob` as raw AES-GCM output (`nonce | ciphertext | tag`) —
no discriminating prefix. That key can never change. Any real deployment needs
to rotate keys — after a suspected leak, on a periodic policy, or after a staff
turnover. The current answer is "create a second entry and run a migration to
it," which:
* forces the admin to provision a second bucket / directory,
* churns object storage costs,
* and does not scale to routine rotations.
The proper fix is **in-place key rotation** inside the same entry. The storage
config holds a *list* of keys, the server writes with the newest one, reads
with any of them, and a background job re-encrypts every blob under the newest
key. When the job completes, the admin removes the old key from the list.
At the same time we solve two related on-disk problems that today's format
leaves open:
1. **No self-description.** A `<hash>.blob` today is ambiguous: could be raw
plaintext (unencrypted deployment) or raw AES-GCM output (encrypted
deployment). Reads guess based on the entry config; a misconfigured key or
a wrong-cipher default silently returns garbage.
2. **No path from encrypted to plaintext.** Once encryption is on for an
entry, the only way off is via a second entry and full migration — same
ergonomics problem as key rotation.
Both fall out of introducing a self-describing on-blob header (`v1`). The
object-key suffix stays `<hash>.blob` for both eras — the magic bytes at the
top of every v1 blob are the discriminator, so legacy files stay readable
byte-identically and new files coexist alongside them at the same suffix.
Migration is lazy.
Related memory (this plan supersedes it):
> **[Encryption key rotation in place — deferred, unsafe today]** — the previous
> "namespace object keys by encryption generation" direction (`<hash>.k2.blob`)
> is DROPPED in favour of the pair-list + v1-header approach below. That
> approach fought content-addressability by generation-namespacing per key;
> this one keeps a stable content-addressable object key and encodes generation
> in a compact on-blob header.
## Design
### The pair-list config
Replace the singular `_ENCRYPTION_KEY` + `_ENCRYPTION_CIPHER` with a single
comma-separated list of pairs on `_ENCRYPTION_KEY` alone:
```env
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<base64 K1>,aes-256-gcm:<base64 K2>
```
Rules:
* **Format** — `[<cipher>:]<base64 key>` per pair, comma-separated. Whitespace
around commas / colons tolerated.
* **Cipher optional.** Only one real cipher (`aes-256-gcm`) exists today, so
`<base64 key>` on its own is legal and behaves as `aes-256-gcm:<key>`. When a
second cipher lands, the colon-prefixed form is the disambiguator.
* **The `none` cipher.** Legal, has no key material (`none:` — trailing colon
with an empty key). Enables migrating BOTH ways: encrypt a previously-plaintext
deployment (`none:,aes-256-gcm:<K>`), or decrypt an encrypted deployment
(`aes-256-gcm:<K>,none:`). Correctness relies on the v1 header (see
*Encrypted-blob format*) discriminating encrypted vs plaintext, NOT on
fallback ordering. A `none` pair by itself in the list is equivalent to
omitting `_ENCRYPTION_KEY` entirely (kept for symmetry).
* **Order matters. Last pair wins on writes.** For v1-header reads, the
`key_fp` in the header selects the exact pair; the list order is irrelevant
at read time. For legacy reads (see *Coexistence*), the head pair is used
because pre-v1 deployments had only one key by construction.
* **At least one pair.** Empty list → boot aborts.
* **Uniqueness.** Same key appearing twice = boot aborts (config drift smell).
`none` may appear at most once.
* **Fingerprint.** For each real-cipher pair, log a truncated SHA-256 of the
key material at boot (12 hex chars) so a swap-order accident is loud in the
audit stream. `none` logs as `none:—`.
### Retire `_ENCRYPTION_CIPHER`
`_ENCRYPTION_CIPHER` was added recently and has not shipped in a release. Drop
it entirely in the same slice — the pair-list makes it redundant, and the on-blob
v1 header carries the cipher choice explicitly per blob anyway. No back-compat
shim needed.
### Encrypted-blob format (v1 header)
Every v1 blob starts with a fixed self-describing header. Layout:
```
"OXCPT" 5 bytes — magic marker (0x4F 0x58 0x43 0x50 0x54)
<version> 2 bytes — big-endian u16; v1 = 0x0001
<key_fp> 8 bytes — sha256(key material)[..8]; identifies which pair
decrypts. For a `none` (plaintext) v1 blob this is
eight zero bytes and the fields below are absent.
<nonce> 12 bytes — random per blob (AES-GCM standard nonce length)
<ciphertext> N bytes — encrypted payload; same length as the plaintext
<auth_tag> 16 bytes — AEAD authentication tag
────
fixed overhead: 43 bytes for encrypted v1; 15 bytes for plaintext v1
```
* **The magic `OXCPT`** unambiguously distinguishes a v1 blob from legacy raw
bytes. It's the sole discriminator — no filename convention, no DB column,
the first five bytes tell the reader everything.
* **`<version>`** is the only field that can change format without touching
every blob. v1 is what this plan ships; v2 slots are reserved
(see *Forward compatibility*).
* **`<key_fp>`** eliminates guess-work on read. The pair-list lookup becomes
O(1) — no fallback tag-check loop. A key-fp with no matching pair means
`NoKeyForBlob` (500 with a clear message), NEVER silent garbage.
* **`<nonce>`** and **`<auth_tag>`** are the AES-GCM primitive's own outputs.
Their length is fixed by v1 = AES-256-GCM. If a future v2 uses a different
AEAD with the same shape (say ChaCha20-Poly1305, tag = 16 B, nonce = 12 B)
v2 can keep the layout; a truly-different AEAD triggers a new version.
* **Plaintext v1 blobs** (produced when the entry's head pair is `none`) use
the same magic + version + all-zero key_fp, then the raw plaintext bytes
directly (no nonce, no ciphertext framing, no tag). The header alone is
enough to say "this is plaintext v1", cleanly distinguished from encrypted
v1 and from legacy raw-plaintext.
**Why no CRC.** The AEAD tag covers ciphertext + nonce integrity. Content
addressability (`<hash>` = BLAKE3(plaintext)) covers whole-blob integrity for
either format. A CRC over the header adds nothing — a corrupt version byte
fails on decode; a corrupt key_fp fails to find a pair. Both fail loud.
### Coexistence with legacy blobs
Legacy and v1 blobs share the `<hash>.blob` object-key namespace. The magic
bytes at position 0 discriminate them safely:
| First 5 bytes | Interpretation | Read path |
|---|---|---|
| `"OXCPT"` | v1 blob | verify magic, dispatch on version + key_fp |
| anything else | legacy | apply entry config: raw plaintext if no encryption pair, single-key AES-GCM decode with the head pair if encryption is set (by construction pre-v1 used one key, and any pair-list produced by upgrading from a pre-v1 config keeps that key at the head) |
Collision probability of legacy bytes accidentally matching `"OXCPT"` at
position 0 is 2⁻⁴⁰ per blob. If a match does happen, the read continues into
the v1 path, fails at version/key_fp check with `UnsupportedBlobVersion` or
`NoKeyForBlob`, and returns a hard error. **Never silent corruption.** For
context: on a 10 M-blob deployment, expected number of collision hard-errors
across the lifetime of legacy files is ~10⁻⁵ blobs. Effectively never.
Properties this buys us:
* **No forced migration.** Legacy `<hash>.blob` files stay readable
indefinitely. Existing deployments upgrade to v1 code with zero
format-conversion work.
* **Lazy conversion on hot paths.** Any COW overwrite (WebDAV MOVE, PUT-over,
content-hash re-upload) naturally lands as v1 at the same object key.
* **Explicit conversion via `backend_rotate`.** The rotate job walks
`storage.blobs`, reads each blob via the magic-byte dispatch, and if the
blob is not already v1 with the head-pair key, PUTs it back as v1 with
the head pair — in place, same object key.
* **No schema changes.** Nothing new in `storage.blobs`; nothing for
`blobs_consistency`, `storage_migrate`, or any other sibling job to learn.
### Legacy-pair guardrail
Once any legacy blob remains on disk, the pre-v1 single key that produced
those blobs MUST remain in the pair-list at the head position (or in the list
somewhere the legacy read path can find it). Removing it early breaks every
legacy read.
Guardrail:
* Boot logs a `warn` line if any pair-list has more than one entry AND the
head pair's fingerprint differs from the second entry's — signals "you
added new keys but haven't rotated legacy blobs yet".
* A **legacy-blob counter** is surfaced in the admin panel per storage entry.
The counter is maintained by `blobs_consistency`: during its normal walk it
branches on the magic-byte check and records the legacy count as a run
statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The
admin panel reads the most recent count and displays it. Refresh cadence is
whatever consistency scan cadence the deployment has (weekly by default; on
demand from the admin panel).
* The *"Rotation complete — safe to remove the old key"* hint appears in the
entry card only when the last `backend_rotate` run completed with zero
findings AND the most recent consistency scan reported zero legacy blobs.
* Nothing enforces removal at code level — the admin is trusted, given a
clear signal, and warned.
### Read path
Per blob read:
1. Fetch `<hash>.blob` from the backend.
2. Check first 5 bytes.
3. If `"OXCPT"` → v1 read path:
* Read `<version>`. Not `0x0001` → `UnsupportedBlobVersion`.
* Read `<key_fp>`. If zero → plaintext-v1; return raw bytes (post-header
payload). Else find the pair whose `sha256(key)[..8]` matches; not
found → `NoKeyForBlob`; found → AES-GCM decrypt with `<nonce>` /
`<ciphertext>` / `<auth_tag>` and return plaintext.
4. Otherwise → legacy read path:
* If entry has any real-cipher pair, attempt AES-GCM decrypt with the
head pair's key; return plaintext on success, `DecryptFailed` on tag
failure.
* If entry has only `none` (or no pair at all), return raw bytes as
plaintext.
Never falls through silently. Every failure is a distinct typed error the
handler can map to a 500 with an actionable message.
### Write path
1. Always writes to `<hash>.blob`.
2. Head pair is `none` → write v1 header (magic + version + zero key_fp) +
raw plaintext.
3. Head pair is a real cipher → compute AEAD; write v1 header (magic +
version + key_fp) + nonce + ciphertext + auth_tag.
v1 code never produces a legacy-format blob. Any leftover legacy blobs on
disk pre-date the v1-code deployment.
### The rotation job
New `RecoverableJobHandler` tenant, `backend_rotate`. Mirrors
`backend_migration`'s shape:
* **Iterates `storage.blobs`** in hash-lex order. Cursor is the last-processed
hash (64 hex chars). Same cursor encoding as `backend_migration` and
`blobs_consistency`.
* **Per blob:**
1. Fetch `<hash>.blob` and dispatch via the standard read path.
2. Decide if a rewrite is needed based on what actually decoded:
* Legacy blob (no v1 magic) → always rewrite (upgrade to v1 with head
pair).
* v1 encrypted, decrypted under a pair-index other than head → rewrite
(key rotation).
* v1 encrypted, already under head — skip.
* v1 plaintext, head pair is `none` — skip.
* v1 plaintext, head pair is a real cipher — rewrite
(encrypt-in-place upgrade).
* v1 encrypted, head pair is `none` — rewrite
(decrypt-in-place downgrade).
3. Write via the standard v1 write path — same object key, atomic
replace.
4. Checkpoint. On per-blob failure, record a `rotation_failed` finding
with severity `data_loss` (bytes may not have crossed), continue.
* **Concurrency-safe by construction.**
* A concurrent upload during rotation writes v1 with the head key. The
rotate walk will short-circuit on that hash if it reaches it later.
* A COW overwrite is the same story.
* The v1 write is atomic at object-storage level (S3 replace, Local
rename-into-place). A concurrent reader sees either state.
* No readonly mode. This is a critical improvement over
`backend_migration`: rotation is per-blob idempotent, so we don't need
to freeze writes.
* **Restart-survivable** — same boot-time sweep as every other recoverable
handler.
* **`?deep=true` unused** — rotation has no "slow variant" mode. Parameter
accepted for uniformity, ignored.
### Admin UX
In the admin storage-panel entry card, add a **Rotate encryption key** action.
Preconditions:
* Entry has encryption enabled with ≥ 2 pairs OR entry has legacy blobs
outstanding OR the pair-list has otherwise changed since the last rotation.
* Otherwise the button is disabled with the tooltip *"Add a second pair in
`OXICLOUD_STORAGE_<name>_ENCRYPTION_KEY` first, or wait for legacy blobs to
accumulate — nothing to rotate right now."*
Clicking the button dispatches the `backend_rotate` job for that entry. The
job's progress rides on the same `X-Server-Status` header infrastructure the
maintenance banner uses — but this time WITHOUT engaging read-only mode.
Banner variant reads *"Rotating encryption key on `<entry>` — X% (Y / Z
blobs). All operations continue normally."* and disappears on completion.
The entry card shows a **legacy-blob counter** sourced from the most recent
`blobs_consistency` run: *"N legacy-format blobs remain (upgrade included in
next rotation)"*. Refresh-on-demand button next to it triggers a targeted
`blobs_consistency` scan (already available via the admin surface). The
*"Rotation complete — safe to remove the old key"* hint appears only when
N = 0 and the last `backend_rotate` run completed with zero findings.
### Removing the old pair
Not automated. The admin edits `.env`, restarts. Rationale:
* We do not want the running server to silently mutate its own `.env`.
* A retained old pair is a benign cost (nothing hits it on read since
`key_fp` lookup is O(1)).
* Explicit human step matches the "add a new pair → restart" symmetry.
## Deployment flow
### First-time upgrade to v1
Zero admin work required. On upgrade:
* v1 code deploys.
* New blobs are written in v1 format at the existing `<hash>.blob` object
key.
* Existing legacy blobs stay readable via the magic-byte dispatch — the
legacy read path is preserved verbatim.
* Admin can optionally trigger a `backend_rotate` run to consolidate every
legacy blob into v1 format. Not required — legacy blobs migrate
opportunistically via COW overwrites and stay readable indefinitely
otherwise.
### Rotating an encryption key
The user-facing recipe (goes verbatim into `docs/guide/backend-storage.md`):
```
1. Generate a new key:
openssl rand -base64 32
2. Append it to the entry's key list. Order matters — the NEW key goes LAST:
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<OLD>,aes-256-gcm:<NEW>
3. Restart OxiCloud. New uploads are now encrypted with the new key; existing
blobs still decrypt with the old one.
4. In the admin panel, click "Rotate encryption key" on the entry. This
dispatches a background job that re-encrypts every existing blob under
the new key AND upgrades any remaining legacy-format blobs to v1. All
operations keep working during rotation.
5. Wait for the job to complete (progress shows in the top banner and in
the Jobs admin page).
6. Remove the OLD key from the list:
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<NEW>
7. Restart OxiCloud. Rotation is complete.
```
### Encrypting a previously-plaintext deployment
```
1. Generate a new key (as above).
2. Add it AFTER `none`:
OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=none:,aes-256-gcm:<K>
3. Restart. New uploads are encrypted; existing plaintext blobs stay readable.
4. Run `backend_rotate` to encrypt existing blobs in place.
5. Remove `none:` from the list; restart.
```
### Decrypting an encrypted deployment
```
1. Add `none:` AFTER the current encryption key:
OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=aes-256-gcm:<K>,none:
2. Restart. New uploads are plaintext; existing encrypted blobs stay readable.
3. Run `backend_rotate` to decrypt existing blobs in place.
4. Remove the key pair, keep `none:` only (or drop `_ENCRYPTION_KEY` entirely);
restart.
```
## Testing strategy
**Most of this ships as Rust tests, not Hurl.** The rotation surface is
byte-level, cryptographic, and stateful across restart boundaries — none of
which Hurl can meaningfully assert. Push almost everything down into
`#[cfg(test)]` blocks alongside the code, following the codebase's existing
inline-tests convention (see AGENTS.md — "tests are primarily `#[cfg(test)]`
modules within source files").
**What lives in Rust (unit + integration):**
* **Pair-list parser.** Unit tests in `config.rs` on `_ENCRYPTION_KEY` parsing:
1-pair, 2-pair, cipher-optional shape, `none:` alone, `none:,aes:K`,
`aes:K,none:`, whitespace tolerance, duplicate rejection, multiple-`none`
rejection, empty-list rejection, retired-`_ENCRYPTION_CIPHER` rejection,
base64 length validation. Table-driven; fast.
* **v1 header round-trip.** Unit tests in
`encrypted_blob_backend.rs::tests`: encrypt payload → verify header bytes
(magic, version, key_fp) → decrypt → assert plaintext identity. One test per
format flavour (encrypted, plaintext-with-`none`-head). Verify `<hash>` =
BLAKE3(plaintext) with no offset for plaintext-v1 (the "recovery tool"
invariant).
* **Legacy read-path preservation.** Fixture: a hand-crafted raw AES-GCM
ciphertext (`nonce | ct | tag`) produced by the pre-v1 code path. Assert
that magic-byte dispatch falls through to the legacy branch and decrypts
correctly. Fixture stays in-tree as `tests/fixtures/legacy_blob.bin` so
regressions on the legacy path can never sneak through.
* **`OXCPT` collision on random data.** Property test: generate random N-byte
payloads, feed through legacy read path. Assert the ~2⁻⁴⁰ magic-collision
case fails with `UnsupportedBlobVersion` or `NoKeyForBlob` — hard error,
never silent misread. Keeps the "collisions can't silently corrupt" claim
in the plan honest.
* **Rotation decision tree.** Unit tests on the per-blob `decide()` helper of
`backend_rotate_service.rs`. All six cases from *The rotation job* section
as separate tests with clear names (`legacy_upgrades_to_v1`,
`v1_encrypted_under_head_skips`,
`v1_encrypted_under_older_pair_rewrites`,
`v1_plaintext_with_none_head_skips`,
`v1_plaintext_encrypts_when_head_is_cipher`,
`v1_encrypted_decrypts_when_head_is_none`).
* **Recoverable-job round-trip.** Integration test in
`backend_rotate_service::tests` using the existing recoverable-run harness:
seed N blobs (mix of legacy + v1-under-old-key), trigger rotation, assert
every blob ends v1-with-head, `format_generation` (if we add it later) or
the consistency-scan count reports zero legacy remaining, findings=0.
* **Crash recovery.** Same harness: interrupt mid-run at cursor position K,
restart, assert resume from K and eventual completion with correct final
state. Same discipline `backend_migration` already uses.
* **Concurrency safety.** Test that a `put_blob` call during a rotation
targeting the same hash produces exactly one v1 blob at end-state (either
the rotate's or the concurrent write's — both are head-format so the final
state is indistinguishable). Verifies the "per-blob idempotent" claim.
* **Config-restart semantics.** Table-driven test on the pair-list state
machine: start with pair-list [K1], add K2 → [K1,K2], run rotation,
drop K1 → [K2], assert every blob still decrypts. Exercises the "safe to
remove old key" transition end-to-end without going through the admin UI.
**What Hurl covers (thin — the API surface, not the mechanics):**
* Trigger endpoint auth: `POST /api/admin/storage/entries/{name}/rotate` is
admin-only (403 for non-admins, 401 for unauthenticated).
* Trigger endpoint preconditions: 400 or similar when pairs < 2 AND no
legacy blobs exist AND pair-list hasn't changed.
* Trigger endpoint concurrency: second POST while a run is Active returns
409.
* Progress header: `X-Server-Status` includes a `rotation` payload during a
running job and drops it on completion.
Hurl does NOT try to:
* Inspect on-disk bytes.
* Restart the server with a new pair-list.
* Seed a legacy blob directly.
* Verify decryption after old-key removal.
Those all belong in Rust tests where we can hold the pool, the backend, and
the config in the same test's memory.
**Test data.** Legacy-blob fixtures are byte-frozen (`tests/fixtures/*.bin`)
and committed. Don't regenerate them from live code — the whole point is that
they were produced by pre-v1 code and won't ever again be. Regeneration
scripts (kept outside `tests/`) are OK for one-time refreshes if the format
changes.
## Slices
Same discipline as `storage-multi-entry.md`. Each slice is a mergeable
increment; no slice depends on a later one.
### Slice K1 — Pair-list config parsing
**Scope.** Config layer only. No behaviour change on the write / read path
yet — the parser produces `Vec<KeyPair>` and the existing code keeps calling
`pairs.last()`.
* `NamedStorageEntry.encryption` becomes `Option<Vec<KeyPair>>` where
`KeyPair = { cipher: CipherKind, key_material: Option<[u8; 32]> }`.
`CipherKind` = `{ AesGcm256, None }`; `None` carries no key.
* Parser: split on `,`, per-pair split on `:` (1 or 2 parts), decode base64
when a real cipher, reject empty list, reject duplicates, reject multiple
`none`.
* Retire `OXICLOUD_STORAGE_<N>_ENCRYPTION_CIPHER` — parser errors on it with
guidance to move the cipher into the pair.
* Boot logs each pair's fingerprint (`sha256(key)[..12]` for real ciphers,
`—` for `none`) and marks the head. Warn line if head fp differs from any
non-head fp (rotation window signal).
* Docs: `example.env`, `docs/config/env.md`, `docs/guide/backend-storage.md`
updated with the pair syntax and the three recipes (rotate / encrypt /
decrypt).
**Exit criteria.** Boot with a 1-pair config behaves identically to today.
Boot with a 2-pair config succeeds and logs both fingerprints. Boot with
`_ENCRYPTION_CIPHER` alongside `_ENCRYPTION_KEY` fails with a migration hint.
### Slice K2 — v1 read/write paths in `EncryptedBlobBackend`
**Scope.** The encrypted-backend decorator learns the v1 header format. Reads
dispatch on magic bytes; writes always emit v1 headers. Legacy read path
preserved verbatim. `blobs_consistency` gains a magic-byte branch to track
the legacy-blob count.
* `EncryptedBlobBackend` refactored around `BlobFormat::V1` writer + reader.
Legacy path preserved but read-only from new code.
* Read dispatches on magic; write always emits v1 header at `<hash>.blob`.
* `blobs_consistency`: during its normal walk, per-blob magic-byte check;
legacy count recorded on the run's `stats` JSON. No schema hit — reuses
the existing `stats` bag on `jobs.recoverable_runs`.
* `storage_migrate`: no changes needed. It copies raw bytes between backends;
format is preserved on the target automatically because the object bytes
are opaque to it.
**Exit criteria.** A brand-new deployment writes only v1 blobs. An upgraded
deployment reads existing legacy blobs and writes new v1 blobs at the same
object-key. `blobs_consistency` reports a legacy-blob count in its run stats.
### Slice K3 — The `backend_rotate` recoverable job
**Scope.** New handler, admin-triggered, iterates blobs, per-blob decision
tree (legacy → v1 upgrade, v1 with old key → v1 with head key, plaintext ↔
encrypted where applicable), records findings.
* New file `src/infrastructure/services/backend_rotate_service.rs`.
* Registered in `JobRegistry` as `backend_rotate`. Runs on the same
recoverable-runs engine (crash recovery, cursor persistence, pause/resume).
* Trigger endpoint: `POST /api/admin/storage/entries/{name}/rotate`.
Requires admin. Refuses if no work would happen (all blobs already at
head format + head key). Refuses if a `backend_rotate` or
`backend_migration` run is already Active for any entry.
* Per-blob decision tree per *The rotation job* section above. In-place
atomic replace at the same `<hash>.blob` object key.
* No readonly mode engaged. `X-Server-Status` header payload gains a
`rotation` field alongside `migration`.
**Exit criteria.** On a filled sandbox: (1) legacy-only entry rotates to
v1; (2) encrypted entry with 2 pairs rotates so head-pair-only remains
decryptable; (3) plaintext entry with `none,cipher` head rotates to
encrypted-v1; (4) encrypted entry with `cipher,none` head rotates to
plaintext-v1. Each round-trip verified by dropping the retired pair from
config and successfully reading every blob.
### Slice K4 — Admin panel action + banner + docs
**Scope.** UI wiring, banner variant, guide docs, legacy-blob counter.
* Entry card: **Rotate encryption key** button + tooltip states as above.
* Legacy-blob counter chip on each entry card. Sources the count from the
most recent `blobs_consistency` run's stats (already surfaced in the Jobs
admin page). Refresh button triggers a fresh scan.
* `ReadOnlyBanner.svelte` gains a third variant `variant="rotating"` — same
shape, milder tone (info not warning), copy makes it clear that operations
continue.
* Server-status header payload gains `rotation?: {entry, migrated, total,
percent}` alongside the existing `migration?:` field. `readonly` stays
false during rotation.
* `docs/guide/backend-storage.md`: new **Rotating an encryption key**,
**Encrypting a plaintext deployment**, and **Decrypting an encrypted
deployment** sections with the recipes above.
* `docs/config/env.md`: pair syntax section under `_ENCRYPTION_KEY`.
* Admin panel *"Rotation complete — safe to remove the old key"* hint gated
on findings=0 AND legacy count=0.
**Exit criteria.** Round-trip on a real deployment (Ed's OVH S3): add pair
→ restart → rotate → verify blobs (including legacy) → remove old pair →
restart → blobs still readable.
## Forward compatibility (v2 and beyond)
The `<version>` field is the single load-bearing knob for future format
changes. Reserved slots:
* `0x0001` — this plan. AES-256-GCM, server-side keys, individual blob
integrity via AEAD tag + content-hash.
* `0x0002…0x00FF` — reserved for server-side format bumps: new AEAD,
chain-authenticated CDC chunks (either *manifest HMAC* — no header changes,
`chunk_manifests` gains a server-computed HMAC over the ordered chunk-hash
list — or *Merkle root in header* — v2 grows to carry `manifest_root` +
`chunk_index`, letting any streamed chunk be verified in isolation). Both
defend against manifest reorder / truncation / injection attacks that
content-addressability alone doesn't cover.
* `0x0100…0x01FF` — reserved for client-side encryption (E2E) variants.
`<key_fp>` becomes the client-key fingerprint; the server can no longer
decrypt and passes ciphertext through the streaming path unchanged.
v1 and v2 coexist in the same storage indefinitely — the magic-byte read
dispatch handles arbitrary versions at position 5-6. Migration between
generations reuses `backend_rotate`'s pattern: rewrite each blob with the
new-generation writer, in-place at the same object key.
## Non-goals
* **Asymmetric / KMS-backed keys.** Out of scope. Symmetric AES-256-GCM only.
A KMS-backed variant is a separate slice built on top of this one.
* **Chunk-chain integrity / manifest tampering defense.** Recognised as a
real gap (chunk reorder / truncate / inject via DB write). Deferred to the
v2 header slice — see *Forward compatibility*.
* **Client-side encryption (E2E).** Same — v2 slot reserved, separate slice.
* **Automatic old-key removal.** Explicit human step, see above.
* **Rotation of a key on the "in-transit" side** (client → server TLS).
Handled by the reverse proxy; unrelated.
* **Per-blob key derivation** (KDF from a master key + blob hash). A
legitimate hardening path but orthogonal to the rotation story; folds in as
a future slice under this same pair-list config.
* **Rollback safety across format generations.** If v1 code is rolled back
to pre-v1 after new v1 blobs have been written, old code will 500 on those
blobs (AES-GCM tag fails on OXCPT-prefixed bytes). Not addressed by this
plan; the safety net is restore-from-backup. Deploying an
incompatibility-breaking format change is a pre-planned event, not a hot
rollback scenario.
## Open questions
* **Should we throttle the rotate job?** Same question `backend_migration` had.
Answer: not in v1. If throughput bites, add a `_ROTATE_MAX_MB_PER_SEC` on
the entry later.
* **Should rotation be idempotent under repeat trigger?** Yes. Running it a
second time with the same config is a no-op walk (every blob is already at
head format + head key).
* **Should we ship a `format_generation` column on `storage.blobs` as a
query-optimizer?** No in v1. Consistency-scan stats cover the "how many
legacy blobs remain" question at admin cadence, and per-read magic-byte
dispatch is a 5-byte compare. If a real hot-path or admin-panel latency
need emerges, the column is a small back-fill migration to add later.
* **Should the legacy-blob counter block key removal at code level?** No —
keeping the trust model consistent with "admin edits .env, we don't fight
them" is the current default. The counter + hint is enough. If real
incidents happen we can escalate to a hard guard later.
docs/plan/storage-multi-entry.md
+3 -3
View File
@@ -213,7 +213,7 @@ if permission.is_write() && self.migration_readonly.load(Ordering::Relaxed) {
- **Reads are unaffected**. Users can still browse and download during
migration.
- **Boot-time clearing**: if boot detects `migration_readonly=true` AND no
in-flight `storage_migration` row (no `Running`/`Paused`) AND
in-flight `backend_migration` row (no `Running`/`Paused`) AND
`active_backend_name` matches the entry the app booted onto → assume
successful cutover completed on prior boot, clear the flag. Otherwise leave
it set; admin knows they still need to finish something.
@@ -230,7 +230,7 @@ if permission.is_write() && self.migration_readonly.load(Ordering::Relaxed) {
strings — but the identity check still runs as a second-line defence
against the encryption-in-place case)
- Write admin_settings.storage.migration_readonly = true
- Trigger `storage_migration` recoverable job with
- Trigger `backend_migration` recoverable job with
params = { source_name: "local_main", target_name: "s3_prod" }
3. Migration runs — target resolved fresh each batch by NAME lookup, so
@@ -473,5 +473,5 @@ Per slice, plus these end-to-end scenarios in Hurl:
`PgAclEngine::check_inner` is where write-permission short-circuits live;
the new global read-only clause lands next to the per-drive one.
- `docs/plan/job-registry.md` Part 2 — recoverable-run engine that
`storage_migration` runs on; `params` field, resume semantics, boot
`backend_migration` runs on; `params` field, resume semantics, boot
sweep.
example.env
+18 -8
View File
@@ -371,9 +371,16 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud
# `OXICLOUD_STORAGE_<N>_BACKEND` (local | s3 | azure) plus the
# backend-specific fields below. A missing required field aborts
# boot with the exact var name in the error message.
# * Presence of `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY` implies AES-256
# encryption is enabled on that entry (no separate enable flag).
# Bad base64 / wrong length aborts boot with the entry name.
# * `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY` is a comma-separated LIST
# of `<cipher>:<key>` pairs. Presence implies encryption is
# enabled on that entry (no separate enable flag). The LAST pair
# wins on writes; every pair is a candidate for reads. Supported
# ciphers: `aes-256-gcm` (default when no cipher prefix is given)
# and `none` (empty-key sentinel used at pair-list head for a
# decrypt-in-place rotation, or at tail for an encrypt-in-place
# rotation). Bad base64 / wrong length / duplicate keys /
# multiple `none` pairs abort boot with the entry name. See
# `docs/plan/storage-key-rotation.md` for rotation recipes.
# * SETTING `_ENTRIES` alongside the legacy flat vars below (e.g.
# `OXICLOUD_STORAGE_BACKEND` + `OXICLOUD_S3_BUCKET`) is a FAIL-FAST
# boot error — pick one mode. Migrate any leftover flat vars into
@@ -393,11 +400,14 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud
#OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY=
#OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY=
#OXICLOUD_STORAGE_s3_prod_S3_FORCE_PATH_STYLE=false
#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY= # generate: openssl rand -base64 32
# Cipher declaration — future-proofing. Today only `aes-256-gcm` is
# accepted (and it's the default when `_ENCRYPTION_KEY` is set), so
# this line can be omitted. Explicit here as documentation.
#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_CIPHER=aes-256-gcm
#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm: # generate: openssl rand -base64 32
# The bare shorthand `<base64 key>` (no `aes-256-gcm:` prefix)
# also works; the explicit form makes the cipher visible and is
# required once you have more than one pair in the list.
#
# Two-pair rotation example (paste both pairs in .env, restart,
# then trigger the format-upgrade job, then drop the OLD pair):
#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<OLD_KEY>,aes-256-gcm:<NEW_KEY>
#
# Repair flag: if you rename an entry in .env while the DB still points
# at the old name, boot aborts with an actionable error pointing at:
examples/bench_s3_put.rs
+1 -1
View File
@@ -353,7 +353,7 @@ async fn main() {
// Full production composition: Cache(Encrypted(Retry(S3))).
let cache_dir_b = tempfile::tempdir().expect("tempdir");
let full_stack: Arc<dyn BlobStorageBackend> = Arc::new(CachedBlobBackend::new(
Arc::new(EncryptedBlobBackend::new(
Arc::new(EncryptedBlobBackend::new_single_aes(
Arc::new(RetryBlobBackend::new(
backend.clone() as Arc<dyn BlobStorageBackend>,
RetryPolicy::default(),
frontend/src/lib/api/endpoints/admin.ts
+63 -4
View File
@@ -65,9 +65,13 @@ export function reextractPhotoMetadata(): Promise<ReextractResult> {
}
/** A freshly generated AES-256 at-rest blob-encryption key (base64) plus a
* data-loss warning authored by the server. */
* data-loss warning authored by the server. The `fingerprint` is the
* SSH-style colon-hex render the boot log / admin pair-chain / rotate
* reports all use — admins can paste the key into `.env`, restart, and
* check the fingerprint matches to confirm the key made it in intact. */
export interface GeneratedKey {
key: string;
fingerprint: string;
warning: string;
}
@@ -338,20 +342,32 @@ export function promoteUserToInternal(userId: string): Promise<void> {
// ── Dashboard ───────────────────────────────────────────────────────────
export interface DriveKindUsage {
kind: 'personal' | 'shared';
used_bytes: number;
// null when there are no capped drives of this kind — the FE
// hides the ratio and just renders "N unlimited"
capped_quota_bytes: number | null;
unlimited_count: number;
capped_count: number;
}
export interface AdminDashboard {
total_users: number;
active_users: number;
admin_users: number;
server_version: string;
total_used_bytes: number;
total_quota_bytes: number;
storage_usage_percent: number;
drive_usage: DriveKindUsage[];
auth_enabled: boolean;
oidc_configured: boolean;
quotas_enabled: boolean;
registration_enabled?: boolean;
users_over_80_percent: number;
users_over_quota: number;
// Backend physical accounting — omitted when the dedup service
// is unavailable. Renders as "—" in that case.
total_bytes_stored?: number;
dedup_ratio?: number;
}
export function getDashboard(): Promise<AdminDashboard> {
@@ -466,6 +482,25 @@ export function saveOidc(body: Record<string, unknown>): Promise<void> {
// ── Storage settings + migration ───────────────────────────────────────────
/**
* One `<cipher>:<key>` pair rendered for the admin storage panel.
* Never carries key material — only cipher name + SSH-style
* fingerprint safe to show operators.
*/
export interface StorageEncryptionPair {
/** `"aes-256-gcm"` for a real-cipher pair, `"none"` for a `none:` sentinel. */
cipher: string;
/**
* SSH-style colon-hex 8-byte fingerprint of the key. Matches
* `backend_rotate`'s `head_key_fp` and the `oxicloud --fingerprint`
* CLI output — enables one-glance identification of which key is
* which. `undefined` for `none:` pairs (no key material).
*/
fingerprint?: string;
/** True for the LAST pair in the list — the write pair (head). */
is_head: boolean;
}
export interface StorageEntrySummary {
name: string;
backend: string;
@@ -473,6 +508,14 @@ export interface StorageEntrySummary {
encryption_enabled: boolean;
/** Human-readable physical hint (root_dir / bucket / container). */
location_hint?: string | null;
/**
* Ordered pair-list summary — oldest first, head last. Empty when
* the entry has no `_ENCRYPTION_KEY` declared at all. Used by the
* entry card to render the pair chain so admins can identify
* which key is the current head + which are safe to remove after
* a completed rotation.
*/
encryption_pairs: StorageEncryptionPair[];
}
export interface StorageSettings {
@@ -538,6 +581,22 @@ export function migrationAction(
return mutate(`/api/admin/storage/migration/${action}`, 'POST', body);
}
/**
* K4 (storage-key-rotation): trigger `backend_rotate` on a specific
* storage entry. Normalises every blob on `<name>` to that entry's
* head-pair format: legacy → v1, plaintext ↔ encrypted, old-key →
* new-key. Fire-and-forget — poll `GET /api/admin/jobs/backend_rotate`
* for status.
*
* Backend: `POST /api/admin/storage/entries/{name}/rotate`
* (`admin_handler::trigger_backend_rotate`). Refuses (400) on unknown
* entry name or when a `backend_rotate` / `backend_migration` run is
* already in flight.
*/
export function rotateStorageEntry(name: string): Promise<void> {
return mutate(`/api/admin/storage/entries/${encodeURIComponent(name)}/rotate`, 'POST', undefined);
}
// verifyMigration + MigrationVerifyResult retired in slice 7 of
// docs/plan/storage-multi-entry.md — the corresponding backend
// endpoint's sample-based check is superseded by
frontend/src/lib/api/endpoints/adminJobs.ts
+55 -11
View File
@@ -17,18 +17,37 @@ const JSON_HEADERS = { 'Content-Type': 'application/json' };
* `ok: true` means "dispatch reached the handler"; the handler's own
* pass/fail is in `outcome.outcome`. For `consistency_batch`, per-child
* outcomes are inside `outcome.extra.per_check`.
*
* `outcome` is absent for detached jobs (currently only
* `backend_migration`) — the endpoint returns `202 Accepted` with
* `dispatched: true` immediately and the run continues in the
* background. Progress polling shows the state; there's no synchronous
* outcome to surface.
*/
export interface TriggerResponse {
ok: boolean;
outcome: JobOutcome;
outcome?: JobOutcome;
dispatched?: boolean;
detached?: boolean;
}
/** Envelope from `POST /api/admin/jobs/{name}/cancel`. `run_id` is
* the id of the run whose `Running` status was flipped to
* `CancelRequested` (null when nothing was in flight to cancel). */
/** Envelope from `POST /api/admin/jobs/{name}/cancel` — terminal
* cancel. `run_id` populated iff a non-terminal row was flipped
* (Running/CancelRequested get the intent stamp; Paused gets a
* direct DB flip to Cancelled). */
export interface CancelResponse {
ok: boolean;
run_id: string | null;
cancelled: boolean;
run_id?: string;
reason?: string;
note?: string;
}
/** Envelope from `POST /api/admin/jobs/{name}/pause` — soft pause. */
export interface PauseResponse {
paused: boolean;
run_id?: string;
reason?: string;
note?: string;
}
/**
@@ -84,11 +103,11 @@ export async function triggerJob(
}
/**
* `POST /api/admin/jobs/{name}/cancel` — cooperatively request cancel
* of the currently running instance. The handler observes it on its
* next `store.status()` poll and returns `RunOutcome::Paused` at the
* next safe boundary. If nothing is running, this is a no-op that
* returns `run_id: null`.
* `POST /api/admin/jobs/{name}/cancel` — TERMINAL cancel. Abandons
* the run: Running/CancelRequested rows get stamped with the intent
* flag and land as `Cancelled` when the handler yields; Paused rows
* get flipped directly to `Cancelled`. Not resumable. Use `pauseJob`
* for interruption-with-resume semantics.
*/
export async function cancelJob(name: string): Promise<CancelResponse> {
const res = await apiFetch(`/api/admin/jobs/${encodeURIComponent(name)}/cancel`, {
@@ -109,6 +128,31 @@ export async function cancelJob(name: string): Promise<CancelResponse> {
return (await res.json()) as CancelResponse;
}
/**
* `POST /api/admin/jobs/{name}/pause` — cooperative pause. Row lands
* as `Paused` when the handler yields; a subsequent trigger click
* resumes from the cursor via `run_or_resume`. Use `cancelJob` to
* abandon terminally.
*/
export async function pauseJob(name: string): Promise<PauseResponse> {
const res = await apiFetch(`/api/admin/jobs/${encodeURIComponent(name)}/pause`, {
method: 'POST',
credentials: 'same-origin',
headers: { ...JSON_HEADERS, ...getCsrfHeaders() }
});
if (!res.ok) {
let msg = `pause failed: ${res.status}`;
try {
const body = (await res.json()) as { error?: string; message?: string };
msg = body.error ?? body.message ?? msg;
} catch {
/* no JSON body */
}
throw new Error(msg);
}
return (await res.json()) as PauseResponse;
}
/**
* `GET /api/admin/jobs/{name}/runs?limit=N` — history of recoverable
* runs for `name`, newest first. Backend caps `limit` at 100.
frontend/src/lib/api/types.ts
+34 -1
View File
@@ -543,6 +543,19 @@ export type JobOutcome =
* Cadence + last-run bookkeeping. `interval_ms` / `next_run_at` are
* `undefined` on on-demand jobs (serde skips `Option::None`).
*/
/**
* Enough info about a paused recoverable run for the admin panel to
* render "Resume (scanned/total)" on the job row without opening the
* drawer. Absent when no `Paused` row exists for this job. `total`
* is absent when the tenant didn't seed a countable subject —
* fallback UI is just "Resume".
*/
export interface PausedRunBrief {
id: string;
scanned: number;
total?: number;
}
export interface JobSummary {
name: string;
interval_ms?: number;
@@ -550,6 +563,20 @@ export interface JobSummary {
last_run_at?: string;
last_outcome?: JobOutcome;
running: boolean;
/**
* `true` iff the job persists runs + findings to
* `jobs.recoverable_runs`. Consumed by the admin panel to decide
* whether the row is expandable (drawer with run history +
* findings) and to gate the retention/purge action — replaces
* the pre-K3 name-based allowlist that missed newly-added
* recoverable tenants (`backend_rotate` shipped first without a
* row-expand until this flag was added).
*/
recoverable: boolean;
/** Populated iff a `Paused` row exists in `jobs.recoverable_runs`
* for this job. Distinct from `running` — a paused run is
* resumable via the same trigger endpoint. */
paused_run?: PausedRunBrief;
}
/**
@@ -557,7 +584,13 @@ export interface JobSummary {
* non-terminal set (Running / Paused / CancelRequested) is what the
* DB's `one_active_run_per_job` partial unique index scopes.
*/
export type RunStatus = 'Running' | 'Paused' | 'CancelRequested' | 'Completed' | 'Failed';
export type RunStatus =
| 'Running'
| 'Paused'
| 'CancelRequested'
| 'Completed'
| 'Failed'
| 'Cancelled';
/**
* `RunSummary` — one row per recoverable-job run from
frontend/src/lib/components/AdminJobsPanel.svelte
+213 -41
View File
@@ -28,6 +28,7 @@
listJobs,
listRuns,
listFindings,
pauseJob,
triggerJob,
cancelJob,
purgeJobRuns
@@ -237,11 +238,33 @@
const key = `trigger:${name}${opts.deep ? ':deep' : ''}`;
markBusy(key, true);
try {
const res = await triggerJob(name, opts);
// Fire the trigger + a follow-up loadJobs after a short delay
// in parallel. Long jobs (backend_migration) come back 202
// immediately; short jobs (consistency checks) come back on
// completion. Either way, the `running` badge / Pause button
// should appear within a render cycle rather than waiting
// for the next 5s poll tick.
const triggerPromise = triggerJob(name, opts);
// Give the backend a moment to register the run's
// `current_run_start` before we ask "is it running?" — this
// races against the trigger acknowledgment for detached
// jobs. 300 ms is well under the 5 s poll cadence and
// invisible to the operator.
setTimeout(() => {
void loadJobs();
if (expandedJob === name) void loadRuns(expandedJob);
}, 300);
const res = await triggerPromise;
// The trigger envelope carries the child's outcome — surface
// its pass/fail immediately so operators don't have to click
// through to see whether the run completed cleanly.
if (res.outcome.outcome === 'ok') {
// through to see whether the run completed cleanly. Detached
// jobs come back with `dispatched: true` and no outcome —
// silence the notify for those (the "started" state is
// already visible via the badge).
if (!res.outcome) {
// dispatched (detached) — no outcome to render
} else if (res.outcome.outcome === 'ok') {
ui.notify(
t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
'success'
@@ -265,23 +288,71 @@
}
}
async function onCancel(name: string) {
const key = `cancel:${name}`;
async function onPause(name: string) {
const key = `pause:${name}`;
markBusy(key, true);
try {
const res = await cancelJob(name);
if (res.run_id) {
const res = await pauseJob(name);
if (res.paused) {
ui.notify(
t(
'admin.jobs.cancel_requested',
'admin.jobs.pause_requested',
{ name },
'Cancel requested — {{name}} will pause at the next safe boundary'
'Pause requested — {{name}} will pause at the next checkpoint (progress preserved)'
),
'info'
);
} else {
ui.notify(
t('admin.jobs.cancel_noop', { name }, 'Nothing to cancel — {{name}} is not running'),
t('admin.jobs.pause_noop', { name }, 'Nothing to pause — {{name}} is not running'),
'info'
);
}
await loadJobs();
if (expandedJob === name) await loadRuns(name);
} catch (e) {
ui.notify(errorMessage(e), 'error');
} finally {
markBusy(key, false);
}
}
async function onCancel(name: string) {
// Terminal cancel confirmation — this is destructive (marks the
// run as Cancelled, cursor preserved for post-mortem but not
// resumable). Skip the confirm for non-recoverable jobs since
// there's no persistent state to lose there today.
if (
!window.confirm(
t(
'admin.jobs.cancel_confirm',
{ name },
'Cancel run of {{name}}? The run will be marked as Cancelled and cannot be resumed. Progress bytes on disk stay put — this only affects the run row.'
)
)
) {
return;
}
const key = `cancel:${name}`;
markBusy(key, true);
try {
const res = await cancelJob(name);
if (res.cancelled) {
ui.notify(
t(
'admin.jobs.cancel_requested',
{ name },
'Cancel requested — {{name}} will land in Cancelled at the next batch boundary (Paused rows flip immediately)'
),
'info'
);
} else {
ui.notify(
t(
'admin.jobs.cancel_noop',
{ name },
'Nothing to cancel — {{name}} has no non-terminal run'
),
'info'
);
}
@@ -400,11 +471,41 @@
return 'jobs-panel__pill jobs-panel__pill--ok';
case 'Failed':
return 'jobs-panel__pill jobs-panel__pill--err';
case 'Cancelled':
return 'jobs-panel__pill jobs-panel__pill--neutral';
default:
return 'jobs-panel__pill jobs-panel__pill--neutral';
}
}
/**
* Human-facing label for a `RunStatus`. Translates the internal
* DB status enum into text an operator can read at a glance —
* notably renders `CancelRequested` as "Pausing" for the
* recoverable-run case (the mechanism is a cancel flag, but the
* user intent is pause). Non-recoverable cancels aren't a thing
* today because non-recoverable jobs run to completion inline,
* so `CancelRequested` here is always the pause path.
*/
function statusLabel(status: RunStatus): string {
switch (status) {
case 'Running':
return t('admin.jobs.status_running', 'Running');
case 'Paused':
return t('admin.jobs.status_paused', 'Paused');
case 'CancelRequested':
return t('admin.jobs.status_ending', 'Ending');
case 'Completed':
return t('admin.jobs.status_completed', 'Completed');
case 'Failed':
return t('admin.jobs.status_failed', 'Failed');
case 'Cancelled':
return t('admin.jobs.status_cancelled', 'Cancelled');
default:
return status;
}
}
/** Coarse "3 min ago" / "2 h ago" — same shape as the parent
* admin page's timeAgo(). Duplicated locally so the component
* stays self-contained; extract if a third caller emerges. */
@@ -461,16 +562,13 @@
}
function isRecoverable(job: JobSummary): boolean {
// Heuristic: recoverable jobs are the ones that publish runs via
// `jobs.recoverable_runs`. There's no direct flag on JobSummary
// (Part 1 handlers and Part 2 adapters share the same summary
// shape by design). Name-based recognition is fine for now — the
// admin panel is the only consumer; broader use would call for
// a `recoverable: bool` field in JobSummary.
return name_is_recoverable(job.name);
}
function name_is_recoverable(name: string): boolean {
return name.endsWith('_consistency') || name === 'storage_migration';
// Backend authoritative source: the `recoverable` flag on
// `JobSummary` is set at registration time by
// `RecoverableAdapter::is_recoverable() -> true`. Every tenant
// registered via `register_recoverable_job` flips it
// automatically. No name-based allowlists — a new recoverable
// tenant is expandable in the UI as soon as it's registered.
return job.recoverable;
}
// Consistency batch shortcut — top button. Only shown when the
@@ -487,7 +585,6 @@
<section class="jobs-panel">
<header class="jobs-panel__header">
<div class="jobs-panel__header-text">
<h2>{t('admin.jobs.title', 'Jobs')}</h2>
<p class="jobs-panel__hint">
{t(
'admin.jobs.hint',
@@ -620,30 +717,94 @@
{/if}
</td>
<td class="jobs-panel__actions">
<button
class="jobs-panel__btn jobs-panel__btn--small"
disabled={busyKeys.has(`trigger:${job.name}`)}
onclick={() => onTrigger(job.name)}
>
{t('admin.jobs.run', 'Run')}
</button>
{#if supportsDeep(job.name)}
{#if job.paused_run}
<!-- Paused row: [Resume (X/Y)] to continue, [Cancel]
to abandon the checkpoint (marks run Cancelled). -->
{@const p = job.paused_run}
{@const label =
p.total && p.total > 0
? t(
'admin.jobs.resume_progress',
{ scanned: p.scanned, total: p.total },
'Resume ({{scanned}}/{{total}})'
)
: t('admin.jobs.resume', 'Resume')}
<button
class="jobs-panel__btn jobs-panel__btn--small"
disabled={busyKeys.has(`trigger:${job.name}:deep`)}
onclick={() => onTrigger(job.name, { deep: true })}
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__btn--primary"
disabled={busyKeys.has(`trigger:${job.name}`)}
onclick={() => onTrigger(job.name)}
title={t(
'admin.jobs.resume_title',
'Continue the paused run from its last checkpoint.'
)}
>
{t('admin.jobs.run_deep', 'Run deep')}
{label}
</button>
{/if}
{#if isRunning(job) && canExpand}
<button
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__btn--danger"
disabled={busyKeys.has(`cancel:${job.name}`)}
onclick={() => onCancel(job.name)}
title={t(
'admin.jobs.cancel_paused_title',
'Abandon the paused run — marks it as Cancelled. Not resumable.'
)}
>
{t('admin.jobs.cancel', 'Cancel')}
</button>
{:else}
<button
class="jobs-panel__btn jobs-panel__btn--small"
disabled={busyKeys.has(`trigger:${job.name}`)}
onclick={() => onTrigger(job.name)}
>
{t('admin.jobs.run', 'Run')}
</button>
{#if supportsDeep(job.name)}
<button
class="jobs-panel__btn jobs-panel__btn--small"
disabled={busyKeys.has(`trigger:${job.name}:deep`)}
onclick={() => onTrigger(job.name, { deep: true })}
>
{t('admin.jobs.run_deep', 'Run deep')}
</button>
{/if}
{/if}
{#if isRunning(job) && canExpand}
{#if isRecoverable(job)}
<!-- Recoverable running: [Pause] preserves cursor
for later resume; [Cancel] abandons terminally
(engine writes Cancelled when the handler yields). -->
<button
class="jobs-panel__btn jobs-panel__btn--small"
disabled={busyKeys.has(`pause:${job.name}`)}
onclick={() => onPause(job.name)}
title={t(
'admin.jobs.pause_title',
'Signal a graceful pause at the next batch boundary. Run row stays as `Paused` — Resume picks up from the checkpoint.'
)}
>
{t('admin.jobs.pause', 'Pause')}
</button>
<button
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__btn--danger"
disabled={busyKeys.has(`cancel:${job.name}`)}
onclick={() => onCancel(job.name)}
title={t(
'admin.jobs.cancel_running_title',
'Abandon the run — marks it as Cancelled at the next batch boundary. Not resumable.'
)}
>
{t('admin.jobs.cancel', 'Cancel')}
</button>
{:else}
<button
class="jobs-panel__btn jobs-panel__btn--small jobs-panel__btn--danger"
disabled={busyKeys.has(`cancel:${job.name}`)}
onclick={() => onCancel(job.name)}
>
{t('admin.jobs.cancel', 'Cancel')}
</button>
{/if}
{/if}
</td>
</tr>
@@ -710,7 +871,7 @@
</td>
<td>
<span class={statusClass(run.status)}>
{run.status}
{statusLabel(run.status)}
</span>
</td>
<td class="jobs-panel__muted">
@@ -1009,10 +1170,6 @@
flex-wrap: wrap;
}
.jobs-panel__header-text h2 {
margin: 0 0 0.25rem;
}
.jobs-panel__hint {
margin: 0;
color: var(--color-text-muted);
@@ -1254,9 +1411,24 @@
background: var(--color-bg-surface);
padding: 0.5rem;
border-radius: 4px;
overflow-x: auto;
font-size: 0.8rem;
margin: 0.5rem 0 0;
/* Wrap long values (cursor_hex is 128 hex chars) instead of
expanding the table cell — the run-drawer sits inside a
`<td colspan>` that would otherwise grow horizontally past
the viewport and blow out the page layout. `pre-wrap`
preserves the multi-line JSON.stringify(…, 2) indent;
`word-break: break-all` breaks the long hex strings mid-run
without hyphens.
`overflow-x: auto` is kept as a defense-in-depth for any
future field that pre-wrap can't handle (e.g. a single
unbroken word longer than max-width). It only kicks in
when wrapping isn't enough. */
white-space: pre-wrap;
word-break: break-all;
max-width: 100%;
overflow-x: auto;
}
.jobs-panel__findings h4 {
frontend/src/lib/components/AppShell.svelte
+7
View File
@@ -1041,6 +1041,13 @@
different copy. -->
{#if serverStatus().readonly}
<ReadOnlyBanner variant="maintenance" progress={serverStatus().migration} />
{:else if serverStatus().rotation}
<!-- K4 storage-key-rotation: rotation is running but
`readonly` is false — writes continue as normal.
Distinct banner variant so the copy reads
"background maintenance" rather than "server
frozen". -->
<ReadOnlyBanner variant="rotating" progress={serverStatus().rotation} />
{/if}
{@render children()}
</div>
frontend/src/lib/components/ReadOnlyBanner.svelte
+59 -29
View File
@@ -1,37 +1,27 @@
<script lang="ts">
/**
* Read-only banner — one component, two variants.
* Read-only banner — one component, three variants.
*
* ## `variant="drive"` (default) — drive-scoped freeze
*
* Rendered at the top of any page whose content lives in (or is scoped
* to) a drive whose `policies.read_only === true`. Members see the
* banner and understand why upload / rename / delete / share
* affordances elsewhere in the app fail with a generic error toast —
* the backend engine gate refuses every non-`Read` permission on
* resources in the drive.
*
* Only `Read` permissions pass; the banner does not need to gate any
* behavior itself. It's pure signage. Backed by
* `docs/plan/drive.md` §8 (`read_only`).
*
* Consumed by:
* - `routes/config/drive/[uuid]/+page.svelte` — always shown when
* the drive being configured is frozen.
* - `routes/files/[...path]/+page.svelte` — shown when the current
* folder's owning drive is frozen (parent looks up drive via
* `drives.findByRootFolderId`/`findById`).
* to) a drive whose `policies.read_only === true`.
*
* ## `variant="maintenance"` — server-wide freeze
*
* Rendered inside `AppShell` above `{children}` when the
* `x-server-status` header (see `middleware::server_status`) says
* the whole server is in read-only mode — typically during a
* storage-backend migration. Optional `progress` lets the banner
* show target + percentage.
* `x-server-status` header says the whole server is in read-only
* mode — typically during a `backend_migration` cutover.
*
* Shape / accent is identical between both variants — the design
* system reads them as the same family. Only the copy differs.
* ## `variant="rotating"` — background key rotation
*
* K4 storage-key-rotation. `backend_rotate` walks blobs in place;
* writes/reads continue normally throughout. Copy makes it clear
* this is a background maintenance banner, not a freeze — the app
* is fully usable.
*
* Shape / accent is identical across variants — the design system
* reads them as the same family. Only the copy differs.
*/
import { t } from '$lib/i18n/index.svelte';
import Icon from '$lib/icons/Icon.svelte';
@@ -47,12 +37,13 @@
/**
* `"drive"` — a specific drive is frozen (default; back-compat
* with pre-migration call sites). `"maintenance"` — the whole
* server is in read-only mode.
* server is in read-only mode. `"rotating"` — a background key
* rotation is running; writes continue.
*/
variant?: 'drive' | 'maintenance';
variant?: 'drive' | 'maintenance' | 'rotating';
/** Drive-name shown in the body (variant="drive" only). */
driveName?: string;
/** Migration progress (variant="maintenance" only). */
/** Migration/rotation progress (variant="maintenance" | "rotating" only). */
progress?: Progress;
}
@@ -61,19 +52,28 @@
<div
class="read-only-banner"
class:read-only-banner--rotating={variant === 'rotating'}
role="region"
aria-label={variant === 'maintenance'
? t('server_status.readonly_banner_aria', 'Server maintenance in progress')
: t('drive.read_only_banner.aria', 'This drive is read-only')}
data-testid={variant === 'maintenance' ? 'server-status-banner' : 'read-only-banner'}
: variant === 'rotating'
? t('server_status.rotating_banner_aria', 'Storage key rotation in progress')
: t('drive.read_only_banner.aria', 'This drive is read-only')}
data-testid={variant === 'maintenance'
? 'server-status-banner'
: variant === 'rotating'
? 'server-status-rotating-banner'
: 'read-only-banner'}
>
<div class="read-only-banner__icon" aria-hidden="true">
<Icon name="lock" />
<Icon name={variant === 'rotating' ? 'key' : 'lock'} />
</div>
<div class="read-only-banner__body">
<strong>
{#if variant === 'maintenance'}
{t('server_status.readonly_title', 'Server maintenance in progress')}
{:else if variant === 'rotating'}
{t('server_status.rotating_title', 'Storage key rotation in progress')}
{:else if driveName}
{t(
'drive.read_only_banner.title_named',
@@ -103,6 +103,24 @@
'Uploads, renames, deletes, and shares are refused temporarily. Reads and downloads work as normal.'
)}
{/if}
{:else if variant === 'rotating'}
{#if progress}
{t(
'server_status.rotating_progress',
{
target: progress.target,
migrated: progress.migrated,
total: progress.total,
percent: progress.percent
},
'Rotating encryption on `{{target}}` — {{percent}}% ({{migrated}} / {{total}} blobs). All operations continue normally; this is a background maintenance task.'
)}
{:else}
{t(
'server_status.rotating_body',
'A background key rotation is normalising storage. All operations continue normally.'
)}
{/if}
{:else}
{t(
'drive.read_only_banner.body',
@@ -143,6 +161,18 @@
font-size: var(--text-lg);
}
/* K4 rotating variant — same shape, info accent (softer than the
default), signalling "background task, no user-facing freeze".
Uses `--color-info` when the palette defines it, falls back to
`--color-accent` otherwise. */
.read-only-banner--rotating {
border-left-color: var(--color-info, var(--color-accent));
}
.read-only-banner--rotating .read-only-banner__icon {
color: var(--color-info, var(--color-accent));
}
.read-only-banner__body {
display: flex;
flex-direction: column;
frontend/src/lib/stores/serverStatus.svelte.ts
+29 -12
View File
@@ -15,17 +15,34 @@
*/
/**
* JSON shape emitted in the `x-server-status` header. Optional
* `migration` field is present only while a migration is running.
* Progress snapshot shared by both migration and rotation fields.
* Server-side struct is `ProgressHeader` — see
* `middleware::server_status`.
*/
export interface ProgressStatus {
target: string;
migrated: number;
total: number;
percent: number;
}
/**
* JSON shape emitted in the `x-server-status` header.
*
* * `migration` — present only during a `backend_migration` run;
* engages `readonly = true` (all writes are refused).
* * `rotation` — present only during a `backend_rotate` run (K4
* storage-key-rotation); `readonly` stays false, uploads and
* reads continue normally throughout.
*
* Both can be `undefined` on the same response — that's the steady-
* state "nothing running" case and the header may be omitted
* entirely.
*/
export interface ServerStatus {
readonly: boolean;
migration?: {
target: string;
migrated: number;
total: number;
percent: number;
};
migration?: ProgressStatus;
rotation?: ProgressStatus;
}
const DEFAULT: ServerStatus = { readonly: false };
@@ -48,10 +65,10 @@ export function serverStatus(): ServerStatus {
*/
export function updateFromHeader(rawHeader: string | null): void {
if (rawHeader == null) {
// No header on this response = server not in maintenance
// mode = reset the store to the default so any lingering
// banner disappears. Cheap idempotent write.
if (current.readonly || current.migration) current = DEFAULT;
// No header on this response = nothing running server-side
// = reset the store to the default so any lingering banner
// disappears. Cheap idempotent write.
if (current.readonly || current.migration || current.rotation) current = DEFAULT;
return;
}
try {
frontend/src/routes/admin/[[tab]]/+page.svelte
+551 -38
View File
@@ -34,6 +34,7 @@
setUserRole,
testOidc,
testStorage,
rotateStorageEntry,
createExternalMount,
deleteExternalMount,
listExternalMounts,
@@ -75,6 +76,7 @@
User
} from '$lib/api/types';
import { triggerJob } from '$lib/api/endpoints/adminJobs';
import { serverStatus } from '$lib/stores/serverStatus.svelte';
import AdminJobsPanel from '$lib/components/AdminJobsPanel.svelte';
import Icon from '$lib/icons/Icon.svelte';
import Modal from '$lib/components/Modal.svelte';
@@ -408,19 +410,45 @@
}
}
// Entry-card action confirmations use `ui.notify()` (viewport
// toast) rather than `storageMsg` (top-of-tab banner). The
// buttons live on cards that can be scrolled far below the
// storage-msg region — a banner confirmation is invisible
// when the user is looking at the card that triggered it.
async function doAuditEntry(name: string) {
try {
await triggerJob('blobs_consistency', { storage: name });
storageMsg = {
text: t(
ui.notify(
t(
'admin.storage_audit_triggered',
{ name },
'blobs_consistency triggered for `{{name}}` — watch it on the Jobs tab.'
),
ok: true
};
'success'
);
} catch (e) {
storageMsg = { text: errorMessage(e), ok: false };
ui.notify(errorMessage(e), 'error');
}
}
// K4: `backend_consistency` — the mirror of `blobs_consistency`.
// Walks the entry's backend and reports blobs physically present
// on it that have no matching row in `storage.blobs` (orphans on
// disk/S3). Meaningful for ANY entry, not just the active one —
// useful for spotting leftover data on a deprecated backend.
async function doStorageConsistency(name: string) {
try {
await triggerJob('backend_consistency', { storage: name });
ui.notify(
t(
'admin.storage_backend_audit_triggered',
{ name },
'backend_consistency triggered for `{{name}}` — watch it on the Jobs tab.'
),
'success'
);
} catch (e) {
ui.notify(errorMessage(e), 'error');
}
}
@@ -438,6 +466,36 @@
await doMigration('start', name);
}
// K4 storage-key-rotation: normalise every blob on `<name>` to
// that entry's head-pair format. Unlike migration, rotation does
// NOT engage read-only mode — uploads/reads keep working
// throughout. Fire-and-forget; the Jobs tab surfaces progress.
async function doRotateEntry(name: string) {
if (
!confirm(
t(
'admin.backend_rotate_confirm',
{ name },
'Start a background rotation on `{{name}}`? Every existing blob is rewritten under the entry’s head pair (v1 header + head key). All operations continue normally during rotation — no read-only mode. Progress shows in the top banner and on the Jobs tab.'
)
)
)
return;
try {
await rotateStorageEntry(name);
ui.notify(
t(
'admin.backend_rotate_triggered',
{ name },
'Rotation started on `{{name}}` — watch it on the Jobs tab (`backend_rotate`).'
),
'success'
);
} catch (e) {
ui.notify(errorMessage(e), 'error');
}
}
// Migration
let migration = $state<MigrationStatus | null>(null);
let migrationTimer: ReturnType<typeof setInterval> | null = null;
@@ -1585,7 +1643,7 @@
communicates which admin area we're in — the plain "Admin"
h1 was informationless once the tab bar moved out.
-->
<h1>{tabLabel}</h1>
<h1>{t('admin.title', 'Admin')} > {tabLabel}</h1>
{#if tab === 'dashboard'}
{#if dashboardError}
@@ -1656,20 +1714,105 @@
</div>
{/if}
<div class="card">
<h2>{t('admin.storage', 'Storage')}</h2>
<div class="ds-bar">
<div
class="ds-fill"
class:ds-fill--warn={dashboard.storage_usage_percent > 70}
class:ds-fill--danger={dashboard.storage_usage_percent > 90}
style:width="{Math.min(dashboard.storage_usage_percent, 100)}%"
></div>
<div class="storage-cards">
<div class="card storage-cards__quota">
<h2>{t('admin.quota_usage', 'Quota usage')}</h2>
<p class="muted storage-cards__hint">
{t(
'admin.quota_usage_hint',
'Pre-dedup, logical file sizes. Includes trashed files until permanent deletion.'
)}
</p>
<table class="quota-table">
<tbody>
{#each dashboard.drive_usage ?? [] as row (row.kind)}
{@const label =
row.kind === 'personal'
? t('admin.quota_personal', 'Personal drives')
: t('admin.quota_shared', 'Shared drives')}
{@const pct =
row.capped_quota_bytes && row.capped_quota_bytes > 0
? (row.used_bytes / row.capped_quota_bytes) * 100
: null}
{#if row.capped_count > 0 || row.unlimited_count > 0}
<tr>
<th scope="row">{label}</th>
<td class="quota-table__num">
{#if row.capped_quota_bytes !== null && pct !== null}
{formatBytes(row.used_bytes)} / {formatBytes(row.capped_quota_bytes)}
<span class="quota-table__pct">({pct.toFixed(1)}%)</span>
{:else}
{formatBytes(row.used_bytes)}
{/if}
</td>
<td class="quota-table__bar">
{#if pct !== null}
<div class="ds-bar">
<div
class="ds-fill"
class:ds-fill--warn={pct > 70}
class:ds-fill--danger={pct > 90}
style:width="{Math.min(pct, 100)}%"
></div>
</div>
{/if}
</td>
<td class="quota-table__meta">
{#if row.unlimited_count > 0}
<span class="quota-table__unlimited">
{t(
'admin.quota_unlimited',
{ n: row.unlimited_count },
'{{n}} unlimited'
)}
</span>
{/if}
</td>
</tr>
{/if}
{/each}
</tbody>
</table>
</div>
<div class="card storage-cards__backend">
<h2>{t('admin.backend_storage', 'Backend storage')}</h2>
{#if dashboard.total_bytes_stored !== undefined}
<dl class="storage-cards__stats">
<div>
<dt>{t('admin.backend_stored', 'Stored')}</dt>
<dd>{formatBytes(dashboard.total_bytes_stored)}</dd>
</div>
<div
class="storage-cards__stat-hint"
title={t(
'admin.backend_referenced_hint',
'Sum of blob references (size × ref_count). Can exceed the drive total because thumbnails, derived assets, and blobs pending garbage collection still hold references.'
)}
>
<dt>
{t('admin.backend_referenced', 'Referenced')}
<Icon name="info-circle" />
</dt>
<dd>
{formatBytes(
Math.round((dashboard.total_bytes_stored ?? 0) * (dashboard.dedup_ratio ?? 1))
)}
</dd>
</div>
<div>
<dt>{t('admin.backend_dedup_ratio', 'Dedup ratio')}</dt>
<dd>
{dashboard.dedup_ratio !== undefined
? `${dashboard.dedup_ratio.toFixed(2)}×`
: '—'}
</dd>
</div>
</dl>
{:else}
<p class="muted">—</p>
{/if}
</div>
<p class="muted">
{formatBytes(dashboard.total_used_bytes)} / {formatBytes(dashboard.total_quota_bytes)}
({dashboard.storage_usage_percent.toFixed(1)}%)
</p>
</div>
{#if dashboard.registration_enabled !== undefined}
@@ -1938,8 +2081,52 @@
The legacy form + related handlers/state live in git
history; deleted here in one sweep.
══════════════════════════════════════════════════════════ -->
<!-- Section 1 — Content store: global DB blob stats,
independent of any backend entry. Rendered first because
it's the "what's actually in the system" answer;
Storage backend + Encryption below are the "where /
how it's stored" answers. -->
{#if storage}
<section
class="card storage-content-stats"
data-testid="admin-storage-content-stats"
aria-labelledby="admin-storage-content-stats-title"
>
<h2 id="admin-storage-content-stats-title">
{t('admin.storage_content_stats_title', 'Content store')}
</h2>
<p class="muted storage-content-stats__hint">
{t(
'admin.storage_content_stats_hint',
'Aggregate over the DB blob store — independent of which backend entry holds the bytes.'
)}
</p>
<dl class="storage-content-stats__grid">
<div>
<dt>{t('admin.storage_blobs', 'Blobs')}</dt>
<dd>{storage.total_blobs ?? '—'}</dd>
</div>
<div>
<dt>{t('admin.storage_size', 'Stored')}</dt>
<dd>
{storage.total_bytes_stored != null ? formatBytes(storage.total_bytes_stored) : '—'}
</dd>
</div>
<div>
<dt>{t('admin.storage_dedup', 'Dedup ratio')}</dt>
<dd>
{storage.dedup_ratio != null ? `${storage.dedup_ratio.toFixed(2)}x` : '—'}
</dd>
</div>
</dl>
</section>
{/if}
<!-- Section 2 — Storage backend: per-entry cards (backend
type, location, actions). Encryption pair-chain moved
out to its own section below. -->
<div class="card">
<h2>{t('admin.storage_tab', 'Storage entries')}</h2>
<h2>{t('admin.storage_title', 'Storage backend')}</h2>
<p class="muted">
{t(
'admin.storage_move_hint',
@@ -1974,7 +2161,17 @@
<dd>{storage.dedup_ratio != null ? `${storage.dedup_ratio.toFixed(2)}x` : '—'}</dd>
</dl>
{:else}
{#if storage.migration_readonly}
<!-- Banner keys off `serverStatus().readonly` (live-updated
via `x-server-status` on every API response) rather
than `storage.migration_readonly` (a snapshot from
the one-shot `getStorageSettings()` fetch). Prevents
the "stale until force-refresh" bug when navigating
into /admin/storage while a migration is running:
any API call that fires on tab entry — even
`loadStorage` itself — updates the store from the
response header, so the banner shows within the
first render cycle. -->
{#if serverStatus().readonly}
<div
class="cutover-hint cutover-hint--readonly"
data-testid="admin-migration-readonly-banner"
@@ -1995,9 +2192,9 @@
<!-- Card-per-entry layout — most installs have 1 backend
(occasionally 2 during a migration), so a rich card
reads better than a wide table. Active entry gets
the sub-stats + a highlight ring. Migrate & activate
is per-card and only shown on non-active cards when
no other migration is in flight. -->
a highlight ring. Migrate & activate is per-card
and only shown on non-active cards when no other
migration is in flight. -->
{@const migrationInFlight =
migration != null && (migration.status === 'running' || migration.status === 'paused')}
<div class="entries-list" data-testid="admin-storage-entries-list">
@@ -2049,6 +2246,7 @@
data-testid={`admin-storage-test-${entry.name}`}
onclick={() => doTestEntry(entry.name)}
>
<Icon name="vial" />
{test?.busy
? t('admin.storage_testing', 'Testing…')
: t('admin.storage_test', 'Test')}
@@ -2062,6 +2260,20 @@
<Icon name="check-double" />
{t('admin.storage_audit', 'Blob consistency')}
</button>
<!-- Storage-side consistency (K4): the mirror of Blob
consistency. `blobs_consistency` walks the DB and
checks the backend has each blob; `backend_consistency`
walks the backend and checks the DB has each hash.
Together they close the reference graph. -->
<button
type="button"
class="btn btn-sm btn-secondary entry-card__action-btn"
data-testid={`admin-storage-backend-audit-${entry.name}`}
onclick={() => doStorageConsistency(entry.name)}
>
<Icon name="database" />
{t('admin.storage_backend_audit', 'Backend consistency')}
</button>
{#if !entry.is_active && !migrationInFlight}
<button
type="button"
@@ -2069,6 +2281,7 @@
data-testid={`admin-storage-migrate-${entry.name}`}
onclick={() => doMigrateActivate(entry.name)}
>
<Icon name="crown" />
{t('admin.storage_migrate_activate', 'Migrate & activate')}
</button>
{:else}
@@ -2079,9 +2292,51 @@
tabindex={-1}
disabled
>
<Icon name="crown" />
{t('admin.storage_migrate_activate', 'Migrate & activate')}
</button>
{/if}
<!-- Rotate encryption key (K4 storage-key-rotation).
ACTIVE ENTRY ONLY — `storage.blobs` describes the
active backend; rotating a non-active entry would
produce a `rotation_failed` finding per blob that
isn't there (backend refuses this with a 400 too).
Placeholder slot on non-active cards keeps the
three-button row aligned across the grid. Also
disabled while any migration is in flight — backend
refuses concurrent encryption-touching jobs. -->
{#if entry.is_active}
<button
type="button"
class="btn btn-sm btn-secondary entry-card__action-btn"
disabled={migrationInFlight}
data-testid={`admin-storage-rotate-${entry.name}`}
onclick={() => doRotateEntry(entry.name)}
title={migrationInFlight
? t(
'admin.backend_rotate_disabled_migration',
'Cannot rotate while a migration is in flight.'
)
: t(
'admin.backend_rotate_tooltip',
'Normalise every blob on this entry to the head pair’s format (upgrade legacy blobs, re-encrypt under a new key, etc.).'
)}
>
<Icon name="key" />
{t('admin.backend_rotate', 'Rotate key')}
</button>
{:else}
<button
type="button"
class="btn btn-sm btn-secondary entry-card__action-btn entry-card__action-btn--placeholder"
aria-hidden="true"
tabindex={-1}
disabled
>
<Icon name="key" />
{t('admin.backend_rotate', 'Rotate key')}
</button>
{/if}
</div>
</header>
<dl class="entry-card__grid">
@@ -2089,21 +2344,42 @@
<dd>{entry.backend}</dd>
<dt>{t('admin.entry_location', 'Location')}</dt>
<dd class="entry-card__mono">{entry.location_hint ?? '—'}</dd>
{#if entry.is_active}
<dt>{t('admin.storage_blobs', 'Blobs')}</dt>
<dd>{storage.total_blobs ?? '—'}</dd>
<dt>{t('admin.storage_size', 'Stored')}</dt>
<dd>
{storage.total_bytes_stored != null
? formatBytes(storage.total_bytes_stored)
: '—'}
</dd>
<dt>{t('admin.storage_dedup', 'Dedup ratio')}</dt>
<dd>
{storage.dedup_ratio != null ? `${storage.dedup_ratio.toFixed(2)}x` : '—'}
</dd>
{/if}
</dl>
<!-- Pair-list chain — one row per configured pair,
head marked. Empty state (no `_ENCRYPTION_KEY`
declared at all) hides the whole block; a single
`none:` pair renders as one row so admins can see
"yes, encryption declaration exists but head is
plaintext" vs "no encryption declared". -->
{#if entry.encryption_pairs?.length}
<section class="entry-card__pairs" aria-label="Encryption keys">
<h4 class="entry-card__pairs-title">
{t('admin.storage_pair_list', 'Encryption keys')}
</h4>
<ol class="entry-card__pair-chain">
{#each entry.encryption_pairs as pair, i (i)}
<li class="entry-card__pair" class:entry-card__pair--head={pair.is_head}>
<span class="entry-card__pair-idx">key{i + 1}:</span>
<span class="entry-card__pair-cipher">{pair.cipher}</span>
<code class="entry-card__pair-fp">
{pair.fingerprint ?? '—'}
</code>
{#if pair.is_head}
<span class="entry-card__pair-head-badge">
{t('admin.storage_pair_head', 'head')}
</span>
{/if}
</li>
{/each}
</ol>
<p class="entry-card__pairs-help muted">
{t(
'admin.storage_pair_help',
'Head is the write key. After a successful rotation with 0 failures, any non-head key can be safely removed from `.env`.'
)}
</p>
</section>
{/if}
{#if test?.result != null || test?.error != null}
<footer class="entry-card__test-result">
{#if test.error}
@@ -2227,6 +2503,17 @@
{t('common.copy', 'Copy')}
</button>
</p>
<p class="muted gen-key-fp">
{t('admin.gen_key_fingerprint', 'Fingerprint')}:
<code>{generatedKey.fingerprint}</code>
<span class="muted">
—
{t(
'admin.gen_key_fingerprint_hint',
'appears in the boot log and the pair chain above once loaded from `.env`.'
)}
</span>
</p>
<p class="alert alert--warn">
<Icon name="exclamation-triangle" />
{t(
@@ -3728,6 +4015,109 @@
background: var(--color-error-text);
}
.storage-cards {
display: grid;
grid-template-columns: 2fr 1fr;
gap: var(--space-3);
margin-bottom: var(--space-4);
}
@media (width <= 40rem) {
.storage-cards {
grid-template-columns: 1fr;
}
}
.storage-cards__hint {
margin-top: calc(-1 * var(--space-2));
margin-bottom: var(--space-3);
font-size: var(--text-xs);
}
.storage-cards__stats {
display: flex;
flex-direction: column;
gap: var(--space-2);
margin: 0;
}
.storage-cards__stats > div {
display: flex;
justify-content: space-between;
align-items: center;
gap: var(--space-3);
}
.storage-cards__stats dt {
display: inline-flex;
align-items: center;
gap: var(--space-1);
color: var(--color-text-muted);
font-size: var(--text-sm);
}
.storage-cards__stats dd {
margin: 0;
font-weight: var(--weight-semibold);
color: var(--color-text-heading);
font-variant-numeric: tabular-nums;
}
.storage-cards__stat-hint {
cursor: help;
}
.quota-table {
width: 100%;
border-collapse: collapse;
}
.quota-table th,
.quota-table td {
padding: var(--space-2) var(--space-2);
text-align: left;
vertical-align: middle;
font-size: var(--text-sm);
}
.quota-table th {
font-weight: var(--weight-semibold);
color: var(--color-text-heading);
white-space: nowrap;
}
.quota-table__num {
font-variant-numeric: tabular-nums;
white-space: nowrap;
}
.quota-table__pct {
color: var(--color-text-muted);
}
.quota-table__bar {
width: 40%;
min-width: 6rem;
}
.quota-table__bar .ds-bar {
margin-bottom: 0;
}
.quota-table__meta {
text-align: right;
white-space: nowrap;
}
.quota-table__unlimited {
display: inline-block;
padding: 2px var(--space-2);
border-radius: var(--radius-full);
background: var(--color-bg-muted);
color: var(--color-text-muted);
font-size: var(--text-xs);
}
.kv {
display: grid;
grid-template-columns: auto 1fr;
@@ -4000,6 +4390,16 @@
flex-wrap: wrap;
}
.gen-key-fp {
margin-top: var(--space-2);
font-size: var(--text-sm);
}
.gen-key-fp code {
font-family: var(--font-mono);
color: var(--color-text-heading);
}
.maint-row {
display: flex;
align-items: center;
@@ -4110,6 +4510,47 @@
color: var(--color-danger-text, var(--color-text));
}
.storage-content-stats {
margin-bottom: var(--space-4);
}
.storage-content-stats h2 {
margin: 0 0 var(--space-1);
}
.storage-content-stats__hint {
margin: 0 0 var(--space-3);
font-size: var(--text-sm);
}
.storage-content-stats__grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(9rem, 1fr));
gap: var(--space-3);
margin: 0;
}
.storage-content-stats__grid > div {
display: flex;
flex-direction: column;
gap: 2px;
}
.storage-content-stats__grid dt {
font-size: var(--text-xs);
color: var(--color-text-muted);
text-transform: uppercase;
letter-spacing: 0.03em;
}
.storage-content-stats__grid dd {
margin: 0;
font-size: var(--text-md);
font-weight: var(--weight-semibold);
color: var(--color-text-heading);
font-variant-numeric: tabular-nums;
}
.entries-list {
display: flex;
flex-direction: column;
@@ -4221,6 +4662,78 @@
border-top: 1px solid var(--color-border);
}
/* K3.7 pair-chain — one row per configured pair. Head is bolded
and gets an "← head" badge so the write pair pops out. Aligns
the fingerprint column so admins can eyeball-diff between
entries. */
.entry-card__pairs {
margin-top: var(--space-3);
padding-top: var(--space-2);
border-top: 1px solid var(--color-border);
}
.entry-card__pairs-title {
font-size: var(--text-xs, 0.75rem);
font-weight: var(--weight-semibold, 600);
text-transform: uppercase;
letter-spacing: 0.5px;
color: var(--color-text-muted);
margin: 0 0 var(--space-2);
}
.entry-card__pair-chain {
list-style: none;
margin: 0;
padding: 0;
display: flex;
flex-direction: column;
gap: var(--space-1);
}
.entry-card__pair {
display: grid;
grid-template-columns: 3rem 6.5rem 1fr auto;
align-items: center;
gap: var(--space-2);
font-size: var(--text-sm);
}
.entry-card__pair-idx {
color: var(--color-text-muted);
font-variant-numeric: tabular-nums;
}
.entry-card__pair-cipher {
color: var(--color-text);
}
.entry-card__pair-fp {
font-family: var(--font-mono, monospace);
font-size: var(--text-xs);
color: var(--color-text-muted);
word-break: keep-all;
}
.entry-card__pair--head .entry-card__pair-cipher,
.entry-card__pair--head .entry-card__pair-fp {
color: var(--color-text);
font-weight: var(--weight-semibold, 600);
}
.entry-card__pair-head-badge {
font-size: var(--text-xs);
padding: 0 var(--space-1);
background: var(--color-surface);
border: 1px solid var(--color-border);
border-radius: var(--radius-sm);
color: var(--color-accent);
}
.entry-card__pairs-help {
margin: var(--space-2) 0 0;
font-size: var(--text-xs);
}
.mig-status {
margin: var(--space-3) 0;
}
frontend/static/locales/en.json
+4 -2
View File
@@ -1120,7 +1120,9 @@
"storage_preset": "Preset",
"storage_size": "Stored",
"storage_tab": "Storage",
"storage_test": "Test connection",
"storage_test": "Test",
"storage_backend_audit": "Backend consistency",
"storage_backend_audit_triggered": "backend_consistency triggered for `{{name}}` — watch it on the Jobs tab.",
"time_day_ago": "{{n}} d ago",
"time_hour_ago": "{{n}} h ago",
"time_just_now": "just now",
@@ -1203,7 +1205,7 @@
"jobs": {
"tab": "Jobs",
"title": "Jobs",
"hint": "Fires periodic + on-demand jobs. Consistency checks are safe to run at any time — they are read-only.",
"hint": "This section concerns internal jobs: periodic + on-demand jobs. Consistency checks are safe to run at any time — they are read-only.",
"run_all_consistency": "Run all consistency checks",
"run_deep": "Run deep",
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
frontend/static/locales/fr.json
+3
View File
@@ -778,6 +778,9 @@
"storage_key_placeholder": "Saisir une nouvelle clé",
"storage_path_style": "Forcer le style de chemin",
"storage_path_style_hint": "Requis pour MinIO et certains services compatibles S3",
"storage_test": "Test",
"storage_backend_audit": "Cohérence du backend",
"storage_backend_audit_triggered": "backend_consistency lancé pour `{{name}}` — suivez son avancement dans l’onglet Tâches.",
"storage_test_connection": "Tester la connexion",
"storage_test_success": "Connexion réussie",
"storage_test_failure": "Échec de la connexion",
src/application/dtos/settings_dto.rs
+77 -4
View File
@@ -118,6 +118,30 @@ pub struct ListUsersQueryDto {
pub summary: Option<bool>,
}
/// One row of the dashboard's quota panel — usage aggregate for a
/// single drive kind. Unlimited caps are excluded from `capped_quota_bytes`
/// and counted in `unlimited_count` so the panel can render the ratio
/// honestly ("X / Y over N capped drives · M unlimited").
#[derive(Debug, Serialize, Deserialize)]
pub struct DriveKindUsageDto {
/// `"personal"` or `"shared"`.
pub kind: String,
/// Total bytes stored across drives of this kind. Excludes trashed
/// files (see `bug_trash_excluded_from_quota` for the known gap).
pub used_bytes: i64,
/// Sum of caps over capped drives only. `None` when there are no
/// capped drives of this kind (would otherwise report `0 / 0`
/// meaninglessly).
pub capped_quota_bytes: Option<i64>,
/// Count of drives (personal: users) with no cap. Personal-kind
/// unlimited = `auth.users.storage_quota_bytes = 0`; shared-kind
/// unlimited = `storage.drives.quota_bytes IS NULL`.
pub unlimited_count: i64,
/// Count of drives with a numeric cap. Used to hide rows with
/// zero drives and denominate the ratio.
pub capped_count: i64,
}
/// Dashboard statistics
#[derive(Debug, Serialize, Deserialize)]
pub struct DashboardStatsDto {
@@ -130,12 +154,27 @@ pub struct DashboardStatsDto {
pub total_users: i64,
pub active_users: i64,
pub admin_users: i64,
// Storage stats
pub total_quota_bytes: i64,
pub total_used_bytes: i64,
pub storage_usage_percent: f64,
// ── Per-drive-kind quota accounting ──
// One row per drive kind (personal, shared). Pre-dedup, logical
// file sizes summed from `drives.used_bytes` (personal rolls up
// via the user envelope). Cap sums exclude unlimited entries;
// `unlimited_count` tracks them separately so the ratio stays
// honest.
pub drive_usage: Vec<DriveKindUsageDto>,
pub users_over_80_percent: i64,
pub users_over_quota: i64,
// ── Backend physical accounting ──
// Bytes actually stored on the active backend (`storage.blobs`
// aggregate) plus the dedup ratio (referenced / stored).
// `total_bytes_stored` is typically << `total_used_bytes` on a
// healthy deployment — dedup + shared blobs mean many user file
// rows resolve to one physical blob. `None` when the dedup
// stats service is unavailable or errored (dashboard renders as
// "—" in that case).
#[serde(skip_serializing_if = "Option::is_none")]
pub total_bytes_stored: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub dedup_ratio: Option<f64>,
pub registration_enabled: bool,
}
@@ -211,6 +250,40 @@ pub struct StorageEntrySummaryDto {
/// Azure). Cosmetic — helps the admin distinguish two Local
/// entries pointing at different disks.
pub location_hint: Option<String>,
/// Ordered pair-list summary — one entry per configured pair in
/// `OXICLOUD_STORAGE_<NAME>_ENCRYPTION_KEY`, oldest first, head
/// last. Empty vec means the entry has no `_ENCRYPTION_KEY`
/// declared at all (pure plaintext-v1 writes today, no crypto).
///
/// Frontend renders this on the entry card so operators can:
/// - See which pairs are configured + their SSH-style
/// fingerprints without inspecting `.env`.
/// - Cross-reference the head pair against the `head_key_fp`
/// from the last `backend_rotate` completion — if they
/// match AND `failed = 0`, every on-disk blob is under the
/// head, and non-head pairs are safe to remove.
#[serde(default)]
pub encryption_pairs: Vec<StorageEncryptionPairDto>,
}
/// One `<cipher>:<key>` pair rendered for the admin UI. Never
/// carries key material — only cipher name + a truncated fingerprint
/// safe to show operators.
#[derive(Debug, Serialize, Deserialize)]
pub struct StorageEncryptionPairDto {
/// `"aes-256-gcm"` for a real-cipher pair, `"none"` for a
/// `none:` sentinel (writes as plaintext-v1).
pub cipher: String,
/// SSH-style colon-hex 8-byte truncation of `sha256(key)`.
/// Matches the v1 header's `<key_fp>` field and the CLI's
/// `oxicloud --fingerprint <key>` output. `None` for `none:`
/// pairs (no key material to fingerprint).
#[serde(skip_serializing_if = "Option::is_none")]
pub fingerprint: Option<String>,
/// True for the LAST pair in the list — the write pair. UI
/// badges it distinctly ("← head" or an arrow). Exactly one
/// pair has `is_head = true` when the list is non-empty.
pub is_head: bool,
}
/// Request body for saving storage settings from the admin panel
src/application/ports/blob_storage_ports.rs
+45
View File
@@ -128,6 +128,51 @@ pub trait BlobStorageBackend: Send + Sync + 'static {
self.put_blob_from_bytes(hash, data)
}
/// Store a blob from in-memory bytes, **replacing** any existing
/// object at that hash. Distinct from [`Self::put_blob_from_bytes`]:
/// the standard variant is idempotent-skip (correct for uploads —
/// same plaintext always produces bytes that decrypt back to the
/// same plaintext), whereas this variant is required by callers
/// that need the on-disk BYTES to change even when the CONTENT
/// hash doesn't:
///
/// * `backend_rotate` — rewrites every blob under the head pair's
/// format (legacy → v1 header, old key → new key, plaintext ↔
/// encrypted). If the target's `put_blob_from_bytes` silently
/// skipped, rotation would report success while leaving the old
/// format on disk.
/// * `backend_migration` — same story when a target already has a
/// blob at that hash from an earlier state (Ed hit this on
/// 2026-08-01 in the S3 → local migration test).
///
/// Must be **atomic** — a concurrent reader must see either the
/// old bytes or the new bytes, never a truncated partial write.
/// On POSIX that's a `write-to-tempfile + rename(2)` pattern; on
/// object storage (S3, Azure) it's a straight `PUT` (already
/// overwrites atomically).
///
/// Default: delegates to `put_blob_from_bytes`. That default is
/// WRONG for every current backend — Local uses `O_EXCL` and
/// S3/Azure both HEAD-probe before writing, so `put_blob_from_bytes`
/// is silently a no-op when the target already exists. Every
/// production backend MUST override this to guarantee overwrite:
///
/// - `LocalBlobBackend` — tempfile + rename(2) + fsync
/// - `S3BlobBackend` / `AzureBlobBackend` — unconditional PUT
/// (same body as `put_blob_from_bytes_unsynced`, which already
/// skips the HEAD probe; object-store PUTs are durable on
/// return so no separate sync is needed)
///
/// The default remains only for the `#[cfg(test)]` mocks that
/// never exercise rotate/migrate.
fn put_blob_from_bytes_replace(
&self,
hash: &str,
data: Bytes,
) -> BoxFut<'_, Result<u64, DomainError>> {
self.put_blob_from_bytes(hash, data)
}
/// Make previously written blobs durable in one batched operation.
///
/// Durability barrier for blobs written via `put_blob_from_bytes_unsynced`:
src/application/services/storage_settings_service.rs
+42 -13
View File
@@ -265,16 +265,36 @@ impl StorageSettingsService {
let entries: Vec<StorageEntrySummaryDto> = self
.storage_entries
.iter()
.map(|e| StorageEntrySummaryDto {
name: e.name.clone(),
backend: match e.backend {
StorageBackendType::Local => "local".to_string(),
StorageBackendType::S3 => "s3".to_string(),
StorageBackendType::Azure => "azure".to_string(),
},
is_active: e.name == active_entry_name,
encryption_enabled: e.encryption_key_base64.is_some(),
location_hint: entry_location_hint(e),
.map(|e| {
// Render the pair-list summary — one row per pair,
// head marked. Never emits key material; only
// cipher + fingerprint. See `StorageEncryptionPairDto`
// for the display contract.
let pairs = e.encryption_pairs();
let head_idx = pairs.len().saturating_sub(1);
let encryption_pairs = pairs
.iter()
.enumerate()
.map(|(i, kp)| {
crate::application::dtos::settings_dto::StorageEncryptionPairDto {
cipher: kp.cipher.as_str().to_string(),
fingerprint: kp.fingerprint_short(),
is_head: i == head_idx,
}
})
.collect();
StorageEntrySummaryDto {
name: e.name.clone(),
backend: match e.backend {
StorageBackendType::Local => "local".to_string(),
StorageBackendType::S3 => "s3".to_string(),
StorageBackendType::Azure => "azure".to_string(),
},
is_active: e.name == active_entry_name,
encryption_enabled: e.is_encrypted(),
location_hint: entry_location_hint(e),
encryption_pairs,
}
})
.collect();
@@ -400,14 +420,23 @@ impl StorageSettingsService {
entry,
std::path::Path::new(&self.env_storage_config.root_dir),
);
let backend_kind = backend.backend_type().to_string();
// Post-K2 (always-wrap): `backend.backend_type()` returns
// the outer wrapper's kind ("v1-plaintext" / "encrypted"),
// NOT the underlying storage backend. `health_check()`
// formats the wrapper-inner combo as
// `"<wrapper>(<inner>)"` — that's the more informative
// string for the admin panel's Test-connection result.
// We use the wrapper-only string as a fallback for the
// health-check-failed branch, where there's no formatted
// status to draw from.
let fallback_backend_kind = backend.backend_type().to_string();
let status = match backend.health_check().await {
Ok(s) => s,
Err(e) => {
return Ok(StorageTestResultDto {
connected: false,
message: format!("health-check failed: {e}"),
backend_type: backend_kind,
backend_type: fallback_backend_kind,
available_bytes: None,
roundtrip_passed: None,
phase_reached: None,
@@ -421,7 +450,7 @@ impl StorageSettingsService {
let mut out = StorageTestResultDto {
connected: status.connected,
message: status.message,
backend_type: backend_kind,
backend_type: status.backend_type,
available_bytes: status.available_bytes,
roundtrip_passed: None,
phase_reached: None,
src/application/services/storage_usage_service.rs
+9 -4
View File
@@ -65,7 +65,7 @@ impl StorageUsageService {
/// `GET /api/drives` therefore lags by up to the cache TTL (30 s),
/// which matches the sibling caches' accepted UX phantom for
/// drive-name staleness. Tests / operators that need immediate
/// freshness call `POST /api/admin/jobs/storage_reconcile/trigger`,
/// freshness call `POST /api/admin/jobs/usage_reconcile/trigger`,
/// which runs `update_all_drives_storage_usage` → this method.
///
/// Security posture unaffected: `check_drive_quota` reads
@@ -576,7 +576,7 @@ impl StorageUsageService {
}
}
pub const STORAGE_RECONCILE_JOB_NAME: &str = "storage_reconcile";
pub const USAGE_RECONCILE_JOB_NAME: &str = "usage_reconcile";
use crate::infrastructure::scheduler::{JobHandler, JobOutcome, JobRegistry, JobRunArgs};
use async_trait::async_trait;
@@ -600,7 +600,7 @@ impl StorageUsageService {
#[async_trait]
impl JobHandler for StorageUsageService {
fn name(&self) -> &str {
STORAGE_RECONCILE_JOB_NAME
USAGE_RECONCILE_JOB_NAME
}
/// Runs both reconciliation sweeps — drives first, then users —
@@ -752,6 +752,12 @@ impl StorageUsagePort for StorageUsageService {
/// FROM` guard to skip no-op rewrites so idle drives don't churn
/// dead tuples. Runs from the same reconciliation ticker as the
/// user sweep; failure is logged but doesn't stop the next tick.
///
/// Trashed files ARE included in the sum — matching the hot-path
/// delta which never decrements on `move_to_trash`. Trash weight
/// stays billed to the drive/user until permanent deletion (that's
/// when the delta subtracts). Excluding trash here would make
/// `used_bytes` oscillate between sweep runs and delta writes.
async fn update_all_drives_storage_usage(&self) -> Result<u64, DomainError> {
debug!("Starting drive storage-usage reconciliation sweep");
let result = sqlx::query(
@@ -762,7 +768,6 @@ impl StorageUsagePort for StorageUsageService {
LEFT JOIN (
SELECT drive_id, SUM(size)::bigint AS total
FROM storage.files
WHERE NOT is_trashed
GROUP BY drive_id
) t ON t.drive_id = d2.id
WHERE d.id = d2.id
src/common/config.rs
+824 -125
View File
File diff suppressed because it is too large Load Diff
src/common/di.rs
+36 -7
View File
@@ -366,7 +366,7 @@ impl AppServiceFactory {
//
// When `storage_entries` is non-empty, encryption is already
// applied inside `build_entry_backend` from the entry's own
// `encryption_key_base64` (per-entry key). This block is the
// pair-list (head-pair key). This block is the
// pre-multi-entry fallback that reads the flat
// `OXICLOUD_STORAGE_ENCRYPTION_*` vars — reachable only for
// fresh installs with no explicit storage config at all
@@ -387,7 +387,7 @@ impl AppServiceFactory {
let key: [u8; 32] = key_bytes.try_into().expect(
"OXICLOUD_STORAGE_ENCRYPTION_KEY must be exactly 32 bytes (base64 of 32 bytes)",
);
blob_backend = Arc::new(EncryptedBlobBackend::new(blob_backend, &key));
blob_backend = Arc::new(EncryptedBlobBackend::new_single_aes(blob_backend, &key));
tracing::info!("Blob storage encryption decorator enabled (AES-256-GCM) — legacy path");
}
@@ -2029,6 +2029,7 @@ impl AppServiceFactory {
authorization: authorization.clone(),
migration_readonly: migration_readonly.clone(),
migration_progress: Arc::new(std::sync::RwLock::new(None)),
rotation_progress: Arc::new(std::sync::RwLock::new(None)),
drive_repo: drive_repo.clone(),
drive_management_service: Arc::new(
crate::application::services::drive_management_service::DriveManagementService::new(
@@ -2244,7 +2245,7 @@ impl AppServiceFactory {
dyn crate::infrastructure::scheduler::JobStoreProvider,
> = app_state.core.job_store_provider.clone();
let _ = Arc::new(
crate::infrastructure::services::storage_migration_service::StorageMigrationService::new(
crate::infrastructure::services::backend_migration_service::BackendMigrationService::new(
app_state
.maintenance_pool
.clone()
@@ -2261,6 +2262,25 @@ impl AppServiceFactory {
.register_recoverable_job(&app_state.core.job_registry, &job_store_provider_dyn)
.await;
// K3: `backend_rotate` recoverable-job tenant. Same
// pattern as `backend_migration` but without the
// cutover/readonly plumbing — rotation writes in place on
// whichever entry the trigger endpoint names. Target name
// comes from `params.target_name` per run.
let _ = Arc::new(
crate::infrastructure::services::backend_rotate_service::BackendRotateService::new(
app_state
.maintenance_pool
.clone()
.expect("maintenance_pool set above"),
app_state.core.config.storage_entries.clone(),
self.storage_path.clone(),
app_state.rotation_progress.clone(),
),
)
.register_recoverable_job(&app_state.core.job_registry, &job_store_provider_dyn)
.await;
// 9b-2. Log whether system needs first-time admin setup
if !admin_svc.is_system_initialized().await {
tracing::warn!("╔══════════════════════════════════════════════════════════╗");
@@ -2457,7 +2477,7 @@ impl AppServiceFactory {
// Migration-readonly boot-clear rule. See
// `docs/plan/storage-multi-entry.md` §"Read-only mode".
//
// If the flag was set true at boot AND no storage_migration
// If the flag was set true at boot AND no backend_migration
// run is currently non-terminal AND active_backend_name
// matches the entry the app actually booted onto — that means
// the cutover completed on a prior boot (the run reached
@@ -2475,11 +2495,11 @@ impl AppServiceFactory {
.migration_readonly
.load(std::sync::atomic::Ordering::Relaxed)
{
use crate::infrastructure::services::storage_migration_service::STORAGE_MIGRATION_JOB_NAME;
use crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME;
let has_in_flight = match app_state
.core
.job_store_provider
.list_runs(STORAGE_MIGRATION_JOB_NAME, 5)
.list_runs(BACKEND_MIGRATION_JOB_NAME, 5)
.await
{
Ok(runs) => runs.iter().any(|r| {
@@ -2495,7 +2515,7 @@ impl AppServiceFactory {
target: "oxicloud::scheduler",
event = "storage.migration_readonly.clear_check_failed",
error = %e,
"failed to list storage_migration runs during readonly-clear check; \
"failed to list backend_migration runs during readonly-clear check; \
leaving migration_readonly flag as-is"
);
// Play it safe: assume in-flight to avoid clearing prematurely.
@@ -2795,6 +2815,15 @@ pub struct AppState {
/// user's session banner about maintenance progress without
/// polling. See `MigrationProgress` for the field shape.
pub migration_progress: Arc<std::sync::RwLock<Option<crate::common::migration_progress::MigrationProgress>>>,
/// Live progress snapshot for the storage-rotate handler
/// (`backend_rotate` — K3 of the storage-key-rotation plan).
/// `Some(_)` while a rotation is running; `None` otherwise.
/// Held separately from `migration_progress` so the
/// server-status header can broadcast the two states
/// independently: migration engages readonly mode, rotation
/// does not. Same `MigrationProgress` type — both are "walk
/// progress" fundamentally.
pub rotation_progress: Arc<std::sync::RwLock<Option<crate::common::migration_progress::MigrationProgress>>>,
/// Drive entity repository — `GET /api/drives`, the personal-drive
/// lifecycle hook, and (post-D2) shared-drive creation flow all read
/// through this. Backing table is `storage.drives`; membership is
src/infrastructure/scheduler/handler.rs
+16
View File
@@ -76,4 +76,20 @@ pub trait JobHandler: Send + Sync {
///
/// See trait-level docs for guidance on when to return Ok vs Err.
async fn run(&self, args: &JobRunArgs) -> JobOutcome;
/// `true` iff this handler persists per-run rows to
/// `jobs.recoverable_runs` (cursor + findings + resume). Surfaced
/// on [`crate::infrastructure::scheduler::registry::JobSummary`]
/// so the admin UI can decide whether the row is expandable to
/// show a run history + findings drawer, without hardcoding a
/// name-based allowlist.
///
/// Default is `false` — Part 1 periodic handlers (`TrashCleanup`,
/// `StorageReconcile`, `GrantCleanup`, `DedupGc`) don't have runs
/// or findings. `RecoverableAdapter` overrides to `true` so every
/// tenant registered via `register_recoverable_job` flips the flag
/// automatically at registration time.
fn is_recoverable(&self) -> bool {
false
}
}
src/infrastructure/scheduler/mod.rs
+1 -1
View File
@@ -37,5 +37,5 @@ pub use recoverable::{
RecoverableJobHandler, RunOutcome, RunProgress, RunStatus, RunSummary, derive_progress,
record_or_log, run_or_resume,
};
pub use registry::{JobEntry, JobRegistry, JobSummary, RegisterError};
pub use registry::{JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError};
pub use types::{ErrCause, JobOutcome, JobRunArgs};
src/infrastructure/scheduler/pg_job_store.rs
+160
View File
@@ -229,6 +229,51 @@ impl JobStore for PgJobStore {
Ok(row.and_then(|(v,)| v))
}
async fn scanned_count(&self) -> Result<u64, DomainError> {
// `(stats->>'scanned_count')::BIGINT` — text cast rather than
// `->` numeric extraction because the stored value has been
// written via `((...)::text)::jsonb` in `checkpoint`, which
// may present as either a JSON number or a JSON string
// depending on prior versions. `::BIGINT` handles both.
let row: Option<(Option<i64>,)> = sqlx::query_as(
"SELECT (stats ->> 'scanned_count')::BIGINT FROM jobs.recoverable_runs WHERE id = $1",
)
.bind(self.run_id)
.fetch_optional(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("scanned_count", e))?;
Ok(row.and_then(|(v,)| v).unwrap_or(0).max(0) as u64)
}
async fn merge_stats(
&self,
extras: &serde_json::Map<String, serde_json::Value>,
) -> Result<(), DomainError> {
if extras.is_empty() {
return Ok(());
}
// JSONB concat (`||`) is a shallow merge — right side wins on
// key conflict, which matches the "last-write-wins" semantic
// in the trait doc. Existing keys from the engine's own
// `scanned_count` / `finding_count` (written via `checkpoint`
// / `record_finding`) are preserved because handlers never
// emit those keys in their extras map.
sqlx::query(
r#"
UPDATE jobs.recoverable_runs
SET stats = COALESCE(stats, '{}'::jsonb) || $1::jsonb,
last_progress_at = NOW()
WHERE id = $2
"#,
)
.bind(serde_json::Value::Object(extras.clone()))
.bind(self.run_id)
.execute(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("merge_stats", e))?;
Ok(())
}
async fn mark_completed(&self) -> Result<(), DomainError> {
sqlx::query(
r#"
@@ -300,6 +345,45 @@ impl JobStore for PgJobStore {
.map_err(|e| map_sqlx_err("mark_failed", e))?;
Ok(())
}
async fn mark_cancelled(&self, cursor: Option<Vec<u8>>) -> Result<(), DomainError> {
// Mirror of `mark_paused`'s two-branch cursor handling: preserve
// the last-known cursor for post-mortem inspection (an operator
// can see how far the abandoned run got) even though nothing
// will resume it.
if let Some(c) = cursor {
sqlx::query(
r#"
UPDATE jobs.recoverable_runs
SET status = 'Cancelled',
cursor = $2,
completed_at = NOW(),
last_progress_at = NOW()
WHERE id = $1
"#,
)
.bind(self.run_id)
.bind(&c[..])
.execute(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("mark_cancelled", e))?;
} else {
sqlx::query(
r#"
UPDATE jobs.recoverable_runs
SET status = 'Cancelled',
completed_at = NOW(),
last_progress_at = NOW()
WHERE id = $1
"#,
)
.bind(self.run_id)
.execute(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("mark_cancelled", e))?;
}
Ok(())
}
}
// ─── PgJobStoreProvider — registry-level ops ────────────────────────────────
@@ -513,6 +597,82 @@ impl JobStoreProvider for PgJobStoreProvider {
.map_err(|e| map_sqlx_err("request_cancel", e))?;
Ok(flipped.map(|(id,)| id))
}
async fn request_terminal_cancel(&self, job_name: &str) -> Result<Option<Uuid>, DomainError> {
// Latest non-terminal row for this job. Order by started_at DESC
// + LIMIT 1 defends against partial-index churn during retries.
let row: Option<(Uuid, String)> = sqlx::query_as(
r#"
SELECT id, status FROM jobs.recoverable_runs
WHERE job_name = $1
AND status IN ('Running', 'CancelRequested', 'Paused')
ORDER BY started_at DESC
LIMIT 1
"#,
)
.bind(job_name)
.fetch_optional(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("request_terminal_cancel.select", e))?;
let Some((id, status)) = row else {
return Ok(None);
};
match status.as_str() {
"Paused" => {
// Direct DB flip — no handler is running to observe
// the intent flag, so we transition immediately.
sqlx::query(
r#"
UPDATE jobs.recoverable_runs
SET status = 'Cancelled',
completed_at = NOW(),
last_progress_at = NOW()
WHERE id = $1
AND status = 'Paused'
"#,
)
.bind(id)
.execute(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("request_terminal_cancel.paused_flip", e))?;
Ok(Some(id))
}
"Running" | "CancelRequested" => {
// Stamp intent + flip to CancelRequested in one statement.
// The handler's next `store.status()` poll observes
// CancelRequested, returns `RunOutcome::Paused` at the
// next boundary; the engine wrap reads the intent and
// calls `mark_cancelled` instead of `mark_paused`.
//
// If the row was already CancelRequested (admin clicked
// Pause first, then Cancel), the status update is a
// no-op but the intent flag stamps — the engine wrap
// upgrades the pending Paused into Cancelled at yield
// time.
sqlx::query(
r#"
UPDATE jobs.recoverable_runs
SET status = 'CancelRequested',
params = jsonb_set(COALESCE(params, '{}'::jsonb),
'{cancel_intent}',
'"terminate"'::jsonb),
last_progress_at = NOW()
WHERE id = $1
"#,
)
.bind(id)
.execute(self.pool.as_ref())
.await
.map_err(|e| map_sqlx_err("request_terminal_cancel.running_flip", e))?;
Ok(Some(id))
}
other => Err(DomainError::internal_error(
"JobStore",
format!("request_terminal_cancel: unexpected status `{other}`"),
)),
}
}
}
// ─── Shared row → RunSummary decoder ────────────────────────────────────────
src/infrastructure/scheduler/recoverable.rs
+269 -22
View File
@@ -54,13 +54,21 @@ use super::types::{JobOutcome, JobRunArgs};
/// Mirror of the `TEXT` values allowed in `jobs.recoverable_runs.status`.
///
/// Terminal set = `{Completed, Failed}`. Non-terminal set (the one the
/// exclusivity partial unique index scopes) =
/// Terminal set = `{Completed, Failed, Cancelled}`. Non-terminal set
/// (the one the exclusivity partial unique index scopes) =
/// `{Running, Paused, CancelRequested}`.
///
/// `CancelRequested` IS non-terminal — the run is still shutting down.
/// A second trigger arriving during cancel MUST NOT spawn a parallel
/// run; the trigger endpoint returns the surviving row instead.
///
/// `Cancelled` IS terminal — admin explicitly abandoned the run. Distinct
/// from `Failed` because it's user-driven, not a handler error. Distinct
/// from `Paused` because it's not resumable. Runs land in `Cancelled` via
/// two paths: (1) admin cancel on a Running row (sets
/// `params.cancel_intent = "terminate"` alongside the CancelRequested
/// flip; engine post-processes handler's Paused return → Cancelled), or
/// (2) admin cancel on an already-Paused row (direct DB flip).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
pub enum RunStatus {
Running,
@@ -68,6 +76,7 @@ pub enum RunStatus {
CancelRequested,
Completed,
Failed,
Cancelled,
}
impl RunStatus {
@@ -79,6 +88,7 @@ impl RunStatus {
RunStatus::CancelRequested => "CancelRequested",
RunStatus::Completed => "Completed",
RunStatus::Failed => "Failed",
RunStatus::Cancelled => "Cancelled",
}
}
@@ -91,6 +101,7 @@ impl RunStatus {
"CancelRequested" => Some(RunStatus::CancelRequested),
"Completed" => Some(RunStatus::Completed),
"Failed" => Some(RunStatus::Failed),
"Cancelled" => Some(RunStatus::Cancelled),
_ => None,
}
}
@@ -105,6 +116,14 @@ impl RunStatus {
}
}
/// Value written to `params.cancel_intent` to tell the engine's
/// terminal-write wrap how to interpret a subsequent
/// [`RunOutcome::Paused`] return. Absent → treat as ordinary pause
/// (write `Paused`). Present with this value → the admin asked to
/// abandon, not just yield, so write `Cancelled` instead.
pub const CANCEL_INTENT_PARAM: &str = "cancel_intent";
pub const CANCEL_INTENT_TERMINATE: &str = "terminate";
// ─── Run outcome (handler → engine) ─────────────────────────────────────────
/// What a [`RecoverableJobHandler`] returns from `run_resumable`.
@@ -119,9 +138,65 @@ impl RunStatus {
/// writes `status = Failed` with the message.
#[derive(Debug, Clone)]
pub enum RunOutcome {
Completed,
Paused { cursor: Vec<u8> },
Failed { message: String },
/// The run walked the whole subject space.
///
/// `extra_stats` is merged into the run row's `stats` JSONB
/// alongside the engine-owned `scanned_count` + `finding_count`
/// / `severity_counts`. Handlers use it to surface per-run
/// summary counters (e.g. `backend_rotate` reports
/// `{"rewritten": N, "skipped": M, "failed": K}`) — the outcome
/// message in `JobOutcome.extra` and every downstream reader
/// of `RunSummary.stats` see the merged fields.
///
/// Empty map = "no tenant-specific extras" — same shape as the
/// pre-K3 bare `Completed` variant. Handlers that don't
/// summarise their work call [`Self::completed`].
Completed {
extra_stats: serde_json::Map<String, serde_json::Value>,
},
Paused {
cursor: Vec<u8>,
},
Failed {
message: String,
},
}
impl RunOutcome {
/// Convenience for the common case: handler has nothing to add
/// to `stats` beyond what the engine already tracks (finding /
/// scanned counters). Equivalent to
/// `Completed { extra_stats: Map::new() }`.
pub fn completed() -> Self {
RunOutcome::Completed {
extra_stats: serde_json::Map::new(),
}
}
/// Convenience for handlers that want to surface per-run
/// summary counters. Takes any JSON object literal produced by
/// `serde_json::json!({...})`; panics if the top-level value
/// isn't an Object (programmer bug — the contract is
/// object-shaped).
///
/// Example — a rotate handler at run-complete:
///
/// ```ignore
/// return RunOutcome::completed_with(serde_json::json!({
/// "rewritten": rewritten_count,
/// "skipped": skipped_count,
/// "failed": failed_count,
/// }));
/// ```
pub fn completed_with(extras: serde_json::Value) -> Self {
match extras {
serde_json::Value::Object(map) => RunOutcome::Completed { extra_stats: map },
other => panic!(
"RunOutcome::completed_with expected a JSON object, got {}",
other
),
}
}
}
// ─── Traits — implementor + port ────────────────────────────────────────────
@@ -248,7 +323,7 @@ pub trait JobStore: Send + Sync {
/// Set an arbitrary string field on `params` (JSONB). Used by
/// handlers on a Fresh run to persist per-run configuration that
/// must survive a mid-run restart — e.g. `storage_migration`
/// must survive a mid-run restart — e.g. `backend_migration`
/// stamping `params.target_name` at run start so a resume can
/// pick up the same target without the admin re-specifying it.
///
@@ -264,6 +339,18 @@ pub trait JobStore: Send + Sync {
/// stamped.
async fn get_string_param(&self, key: &str) -> Result<Option<String>, DomainError>;
/// Current `stats.scanned_count` for this run. Used by handlers
/// on a Resume path to reconstruct progress state that isn't
/// persisted in `params` — e.g. `backend_migration` seeds its
/// user-facing `MigrationProgress` counter with this so the
/// admin banner shows continued progress across a restart
/// instead of resetting to 0.
///
/// Returns `0` if the key is absent (fresh row) or not a
/// number. Callers on a Fresh run can safely skip this — the
/// answer is trivially 0 and the write path starts fresh.
async fn scanned_count(&self) -> Result<u64, DomainError>;
/// Persist one finding to `jobs.run_findings` and bump
/// `stats.finding_count` on the parent run. Consistency handlers
/// call this in place of the transitional
@@ -295,6 +382,24 @@ pub trait JobStore: Send + Sync {
detail: serde_json::Value,
) -> Result<(), DomainError>;
/// **Engine-only.** Merge `extras` into the run row's `stats`
/// JSONB (SQL `stats = stats || $1`). Called by [`run_or_resume`]
/// on [`RunOutcome::Completed`] to persist the handler's
/// per-run summary counters alongside the engine-owned
/// `scanned_count` / `finding_count`. Handler code MUST NOT
/// call this directly — return an `extra_stats` map on
/// `Completed` and the engine handles the write.
///
/// Idempotent: merging the same map twice yields the same row.
/// A stats key that already exists is OVERWRITTEN by the
/// merge (last-write-wins) — a handler that emits e.g.
/// `"rewritten": 300` at run end always displaces any prior
/// per-batch write of the same key.
async fn merge_stats(
&self,
extras: &serde_json::Map<String, serde_json::Value>,
) -> Result<(), DomainError>;
// ─── Terminal writes — engine-only. Do not call from handler code.
/// Engine-only. Called by [`run_or_resume`] on
@@ -309,6 +414,15 @@ pub trait JobStore: Send + Sync {
/// Engine-only. Called by [`run_or_resume`] on
/// [`RunOutcome::Failed`]. Handler code MUST NOT call this.
async fn mark_failed(&self, message: &str) -> Result<(), DomainError>;
/// Engine-only. Called by [`run_or_resume`] when the handler
/// returns [`RunOutcome::Paused`] AND
/// `params.cancel_intent = "terminate"` — the admin asked to
/// abandon the run, not just yield. Writes `status = 'Cancelled'`
/// + `completed_at = NOW()`. Preserves the cursor for post-mortem
/// (an operator can see how far it got before being killed).
/// Handler code MUST NOT call this.
async fn mark_cancelled(&self, cursor: Option<Vec<u8>>) -> Result<(), DomainError>;
}
/// Registry-level operations on `jobs.recoverable_runs` — NOT bound
@@ -365,6 +479,25 @@ pub trait JobStoreProvider: Send + Sync {
/// completes naturally.
async fn request_cancel(&self, job_name: &str) -> Result<Option<Uuid>, DomainError>;
/// Request TERMINAL cancellation — admin abandons the run rather
/// than yielding it for later resume. Two paths depending on the
/// current row's status:
///
/// - **`Running` / `CancelRequested`** — same DB flip as
/// [`Self::request_cancel`] (Running → CancelRequested) BUT
/// also stamps `params.cancel_intent = "terminate"`. When the
/// handler yields and the engine wraps `RunOutcome::Paused`, it
/// reads the intent and calls
/// [`JobStore::mark_cancelled`] instead of `mark_paused`.
/// - **`Paused`** — no handler is running, so the engine wrap
/// never fires. Direct DB flip `Paused → Cancelled +
/// completed_at = NOW()`.
/// - **Terminal or absent** — no-op (`Ok(None)`).
///
/// Returns the affected run's id when any transition happened,
/// `None` otherwise.
async fn request_terminal_cancel(&self, job_name: &str) -> Result<Option<Uuid>, DomainError>;
/// Findings for a specific run, newest-last, paginated.
/// Powers `GET /api/admin/jobs/{name}/runs/{id}/findings`.
/// `limit` caps rows; the API layer clamps it too. `offset` is
@@ -630,8 +763,22 @@ pub async fn run_or_resume(
let stats = fetch_outcome_stats(&*provider, run_id).await;
match outcome {
RunOutcome::Completed => {
RunOutcome::Completed { extra_stats } => {
// Merge tenant-supplied extras into the run's stats
// JSONB BEFORE the terminal mark, so downstream readers
// see the merged view atomically. `fetch_outcome_stats`
// (a few lines up) already ran and reflects the state
// WITHOUT the merge — re-fetch so the outer JobOutcome
// includes the tenant counters too.
if !extra_stats.is_empty() {
log_terminal_write_err(
"merge_stats",
run_id,
store.merge_stats(&extra_stats).await,
);
}
log_terminal_write_err("mark_completed", run_id, store.mark_completed().await);
let stats = fetch_outcome_stats(&*provider, run_id).await;
JobOutcome::ok_with(
stats.finding_count,
serde_json::json!({
@@ -640,23 +787,59 @@ pub async fn run_or_resume(
"finding_count": stats.finding_count,
"scanned_count": stats.scanned_count,
"severity_counts": stats.by_severity,
"extra_stats": serde_json::Value::Object(extra_stats),
}),
)
}
RunOutcome::Paused { cursor } => {
// Read the intent stamped by `/api/admin/jobs/{name}/cancel`
// (terminal cancel path). Absent → ordinary pause. Present
// with `terminate` → admin asked to abandon; write
// Cancelled instead of Paused. Any read error falls
// through to Paused — errs on preserving-progress side.
let terminate = store
.get_string_param(CANCEL_INTENT_PARAM)
.await
.ok()
.flatten()
.as_deref()
== Some(CANCEL_INTENT_TERMINATE);
let cursor_hex = hex::encode(&cursor);
log_terminal_write_err("mark_paused", run_id, store.mark_paused(Some(cursor)).await);
JobOutcome::ok_with(
stats.finding_count,
serde_json::json!({
"paused": true,
"run_id": run_id.to_string(),
"cursor_hex": cursor_hex,
"finding_count": stats.finding_count,
"scanned_count": stats.scanned_count,
"severity_counts": stats.by_severity,
}),
)
if terminate {
log_terminal_write_err(
"mark_cancelled",
run_id,
store.mark_cancelled(Some(cursor)).await,
);
JobOutcome::ok_with(
stats.finding_count,
serde_json::json!({
"cancelled": true,
"run_id": run_id.to_string(),
"cursor_hex": cursor_hex,
"finding_count": stats.finding_count,
"scanned_count": stats.scanned_count,
"severity_counts": stats.by_severity,
}),
)
} else {
log_terminal_write_err(
"mark_paused",
run_id,
store.mark_paused(Some(cursor)).await,
);
JobOutcome::ok_with(
stats.finding_count,
serde_json::json!({
"paused": true,
"run_id": run_id.to_string(),
"cursor_hex": cursor_hex,
"finding_count": stats.finding_count,
"scanned_count": stats.scanned_count,
"severity_counts": stats.by_severity,
}),
)
}
}
RunOutcome::Failed { message } => {
log_terminal_write_err("mark_failed", run_id, store.mark_failed(&message).await);
@@ -802,6 +985,14 @@ impl JobHandler for RecoverableAdapter {
async fn run(&self, args: &JobRunArgs) -> JobOutcome {
run_or_resume(self.inner.clone(), self.provider.clone(), args).await
}
fn is_recoverable(&self) -> bool {
// Every tenant registered through `register_recoverable_job` is
// wrapped by this adapter, so this flag flips true for exactly
// the set of jobs whose runs + findings the admin UI should
// let operators drill into. No name-based allowlists needed
// downstream.
true
}
}
// ─── Ergonomics: JobRegistry extension for recoverable jobs ─────────────────
@@ -857,6 +1048,10 @@ mod tests {
progress_total: Option<u64>,
progress_kind: Option<ProgressKind>,
string_params: std::collections::HashMap<String, String>,
/// K3+: extras merged into the run's stats JSONB via
/// `merge_stats` at Completed time. Tests observe the merged
/// view by reading this map alongside `scanned_count`.
extra_stats: serde_json::Map<String, serde_json::Value>,
}
#[async_trait]
@@ -916,6 +1111,19 @@ mod tests {
async fn get_string_param(&self, key: &str) -> Result<Option<String>, DomainError> {
Ok(self.state.lock().unwrap().string_params.get(key).cloned())
}
async fn scanned_count(&self) -> Result<u64, DomainError> {
Ok(self.state.lock().unwrap().scanned_count)
}
async fn merge_stats(
&self,
extras: &serde_json::Map<String, serde_json::Value>,
) -> Result<(), DomainError> {
let mut s = self.state.lock().unwrap();
for (k, v) in extras {
s.extra_stats.insert(k.clone(), v.clone());
}
Ok(())
}
async fn mark_completed(&self) -> Result<(), DomainError> {
self.state.lock().unwrap().status = RunStatus::Completed;
Ok(())
@@ -934,6 +1142,14 @@ mod tests {
s.error_message = Some(message.to_string());
Ok(())
}
async fn mark_cancelled(&self, cursor: Option<Vec<u8>>) -> Result<(), DomainError> {
let mut s = self.state.lock().unwrap();
s.status = RunStatus::Cancelled;
if let Some(c) = cursor {
s.cursor = Some(c);
}
Ok(())
}
}
// ─── In-memory JobStoreProvider ────────────────────────────────────────
@@ -968,6 +1184,7 @@ mod tests {
progress_total: None,
progress_kind: None,
string_params: std::collections::HashMap::new(),
extra_stats: serde_json::Map::new(),
}),
});
let id = store.run_id;
@@ -1027,6 +1244,7 @@ mod tests {
progress_total: None,
progress_kind: None,
string_params: std::collections::HashMap::new(),
extra_stats: serde_json::Map::new(),
}),
});
stores.push(store.clone());
@@ -1163,7 +1381,10 @@ mod tests {
let before = stores.len();
stores.retain(|s| {
let state = s.state.lock().unwrap();
!matches!(state.status, RunStatus::Completed | RunStatus::Failed)
!matches!(
state.status,
RunStatus::Completed | RunStatus::Failed | RunStatus::Cancelled
)
});
Ok((before - stores.len()) as u64)
}
@@ -1179,6 +1400,32 @@ mod tests {
}
Ok(None)
}
async fn request_terminal_cancel(
&self,
_job_name: &str,
) -> Result<Option<Uuid>, DomainError> {
let stores = self.stores.lock().unwrap();
if let Some(s) = stores.last() {
let mut state = s.state.lock().unwrap();
match state.status {
RunStatus::Paused => {
state.status = RunStatus::Cancelled;
return Ok(Some(s.run_id));
}
RunStatus::Running | RunStatus::CancelRequested => {
state.status = RunStatus::CancelRequested;
state.string_params.insert(
CANCEL_INTENT_PARAM.to_string(),
CANCEL_INTENT_TERMINATE.to_string(),
);
return Ok(Some(s.run_id));
}
_ => {}
}
}
Ok(None)
}
}
// ─── Handlers ──────────────────────────────────────────────────────────
@@ -1196,7 +1443,7 @@ mod tests {
_resume_cursor: Option<Vec<u8>>,
) -> RunOutcome {
store.checkpoint(vec![1, 2, 3], 5).await.unwrap();
RunOutcome::Completed
RunOutcome::completed()
}
}
@@ -1251,7 +1498,7 @@ mod tests {
resume_cursor: Option<Vec<u8>>,
) -> RunOutcome {
*self.saw_cursor.lock().unwrap() = resume_cursor;
RunOutcome::Completed
RunOutcome::completed()
}
}
src/infrastructure/scheduler/registry.rs
+38
View File
@@ -225,6 +225,12 @@ impl JobRegistry {
last_run_at,
last_outcome,
running: state.current_run_start.is_some(),
recoverable: entry.handler.is_recoverable(),
// Populated in `list_jobs` handler via a single
// DB round-trip — kept out of the registry
// snapshot to avoid pulling a DB dependency into
// the in-memory scheduler state.
paused_run: None,
}
})
.collect()
@@ -288,6 +294,13 @@ pub enum RegisterError {
/// - `running` — true iff the in-flight permit is currently held
/// (either the supervisor tick is in progress or an admin trigger
/// raced in).
/// - `recoverable` — true iff the job persists runs + findings to
/// `jobs.recoverable_runs`. Consumed by the admin UI to decide
/// whether the row is expandable (drawer with run history +
/// findings) and to gate the retention/purge action.
/// - `paused_run` — populated iff a `Paused` row exists in
/// `jobs.recoverable_runs` for this job. The UI uses it to render
/// "Resume (scanned/total)" instead of "Run".
#[derive(Debug, Clone, Serialize)]
pub struct JobSummary {
pub name: String,
@@ -300,6 +313,31 @@ pub struct JobSummary {
#[serde(skip_serializing_if = "Option::is_none")]
pub last_outcome: Option<JobOutcome>,
pub running: bool,
pub recoverable: bool,
/// Populated iff a `Paused` row exists in `jobs.recoverable_runs`
/// for this job. Distinct from `running` — a paused run is
/// resumable via the same trigger endpoint (`run_or_resume`
/// picks Resume when the latest row is Paused).
#[serde(skip_serializing_if = "Option::is_none")]
pub paused_run: Option<PausedRunBrief>,
}
/// Enough info about a paused recoverable run for the admin panel to
/// render "Resume (scanned/total)" on the job row without opening the
/// drawer. Populated by `list_jobs` in the admin handler from a
/// single `SELECT job_name, id, stats->>'scanned_count',
/// params->>'total_rows' FROM jobs.recoverable_runs WHERE status =
/// 'Paused'` — indexed by the `one_active_run_per_job` partial UNIQUE.
///
/// `total` is `None` when the tenant doesn't seed a countable subject
/// (`RecoverableJobHandler::count_total`); the UI then shows just
/// "Resume" without progress.
#[derive(Debug, Clone, Serialize)]
pub struct PausedRunBrief {
pub id: uuid::Uuid,
pub scanned: u64,
#[serde(skip_serializing_if = "Option::is_none")]
pub total: Option<u64>,
}
#[cfg(test)]
src/infrastructure/scheduler/types.rs
+2 -2
View File
@@ -24,7 +24,7 @@ use serde::{Deserialize, Serialize};
/// Semantics of `force`, per job:
/// - `dedup_gc` — skip the orphan grace window (grace = 0).
/// - `grant_cleanup` — grace = 0.
/// - Others (trash_cleanup, storage_reconcile, …) — ignored.
/// - Others (trash_cleanup, usage_reconcile, …) — ignored.
///
/// Semantics of `deep`, per job:
/// - `consistency_batch` — propagate to sub-jobs; only `storage_consistency`
@@ -37,7 +37,7 @@ use serde::{Deserialize, Serialize};
///
/// Semantics of `storage`, per job (added for the multi-entry storage
/// design — see `docs/plan/storage-multi-entry.md`):
/// - `storage_migration` — the NAME of the target storage entry to
/// - `backend_migration` — the NAME of the target storage entry to
/// copy blobs INTO. Required on a Fresh run (handler refuses
/// without it); ignored on a Resumed run (target read from the
/// persisted `params.target_name`).
src/infrastructure/services/azure_blob_backend.rs
+17
View File
@@ -173,6 +173,23 @@ impl BlobStorageBackend for AzureBlobBackend {
})
}
/// Atomic overwrite path used by `backend_rotate` and
/// `backend_migration` when re-writing an already-present blob
/// under a new head key/format. Trait default delegates to
/// `put_blob_from_bytes` which `get_properties`-probes and
/// silently skips — exactly wrong for the rotate/migrate use
/// case (the whole point is to replace the existing bytes).
/// Override delegates to the same unconditional PUT as
/// `put_blob_from_bytes_unsynced` — Azure's PUT is durable on
/// return, no separate sync barrier needed.
fn put_blob_from_bytes_replace(
&self,
hash: &str,
data: Bytes,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
self.put_blob_from_bytes_unsynced(hash, data)
}
fn get_blob_stream(
&self,
hash: &str,
src/infrastructure/services/backend_consistency_service.rs
+3 -3
View File
@@ -321,7 +321,7 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
backend = backend.backend_type(),
"backend refused enumeration (typical during migration or on backends without list support)"
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
return RunOutcome::Failed {
message: format!("backend list failed mid-scan: {e}"),
@@ -373,7 +373,7 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
"backend_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
// Batch DB probe: which of these hashes have a
@@ -451,7 +451,7 @@ impl RecoverableJobHandler for BackendConsistencyCheck {
"backend_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
}
}
src/infrastructure/services/storage_migration_service.rs → src/infrastructure/services/backend_migration_service.rs
+281 -97
View File
@@ -46,11 +46,12 @@
//! cancel + cursor discipline; the batch loop is I/O-bound anyway.
//! Add concurrency later if a real throughput need appears.
use std::path::{Path, PathBuf};
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use async_trait::async_trait;
use bytes::Bytes;
use futures::StreamExt;
use sqlx::PgPool;
@@ -61,11 +62,12 @@ use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
};
use crate::infrastructure::services::encrypted_blob_backend::{EncryptedBlobBackend, HeadCheck};
use crate::infrastructure::services::entry_backend::{
build_entry_backend, persist_active_backend_name, persist_migration_readonly,
build_entry_backend_typed, persist_active_backend_name, persist_migration_readonly,
};
pub const STORAGE_MIGRATION_JOB_NAME: &str = "storage_migration";
pub const BACKEND_MIGRATION_JOB_NAME: &str = "backend_migration";
/// The `params` JSONB key under which the run's target entry name is
/// stashed at Fresh-open time via `JobStore::set_string_param`.
@@ -75,6 +77,15 @@ pub const STORAGE_MIGRATION_JOB_NAME: &str = "storage_migration";
/// projections read the same constant.
pub const TARGET_NAME_PARAM: &str = "target_name";
/// Companion to [`TARGET_NAME_PARAM`] — records the source entry
/// name (the active backend at Fresh-open time) so a run row read
/// months later self-describes the migration direction. Without
/// this, an operator inspecting a Completed row from an old
/// deployment could see "migrated to `s3_prod`" but had to
/// cross-reference `admin_settings` history to know what it came
/// from. Stamped once on Fresh open; Resume reads it back.
pub const SOURCE_NAME_PARAM: &str = "source_name";
/// Rows per batch. Copies are I/O-bound (source read + target write);
/// larger batches amortise fewer SQL round-trips but the checkpoint
/// / cancel-poll cadence lengthens. 100 balances the two — one
@@ -82,7 +93,7 @@ pub const TARGET_NAME_PARAM: &str = "target_name";
/// every 100 rows too. Match `blobs_consistency` for consistency.
const BATCH_SIZE: i64 = 100;
pub struct StorageMigrationService {
pub struct BackendMigrationService {
pool: Arc<PgPool>,
/// Backend the running app is bound to at handler-construction
/// time. Refers to the hot-swap wrapper when multi-entry is
@@ -129,7 +140,7 @@ pub struct StorageMigrationService {
Arc<std::sync::RwLock<Option<crate::common::migration_progress::MigrationProgress>>>,
}
impl StorageMigrationService {
impl BackendMigrationService {
#[allow(clippy::too_many_arguments)]
pub fn new(
pool: Arc<PgPool>,
@@ -172,9 +183,9 @@ impl StorageMigrationService {
}
#[async_trait]
impl RecoverableJobHandler for StorageMigrationService {
impl RecoverableJobHandler for BackendMigrationService {
fn name(&self) -> &str {
STORAGE_MIGRATION_JOB_NAME
BACKEND_MIGRATION_JOB_NAME
}
/// Definitive count — one row per blob. `SELECT COUNT(*) FROM
@@ -189,7 +200,7 @@ impl RecoverableJobHandler for StorageMigrationService {
Err(e) => {
tracing::debug!(
target: "oxicloud::migration",
event = "storage_migration.count_total_failed",
event = "backend_migration.count_total_failed",
error = %e,
"count_total failed — run will not surface a progress bar"
);
@@ -223,7 +234,7 @@ impl RecoverableJobHandler for StorageMigrationService {
let Some(name) = args.storage.clone() else {
return RunOutcome::Failed {
message:
"storage_migration requires `target_name` on a fresh run — trigger via \
"backend_migration requires `target_name` on a fresh run — trigger via \
POST /api/admin/storage/migration/start with `{\"target_name\": \"<entry>\"}`."
.to_string(),
};
@@ -259,11 +270,56 @@ impl RecoverableJobHandler for StorageMigrationService {
// reference reads from this local. A hot-swap that fires
// mid-run (e.g., a second migration starting after this one
// completes) doesn't reshape our decisions from underneath.
let active_backend_name = self
.active_backend_name
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone();
//
// For Fresh runs we ALSO stamp this into `params.source_name`
// so an audit-log reader can self-describe the migration
// direction without cross-referencing `admin_settings`
// history. On Resume we read it back — the ORIGINAL source
// (from when the run was opened) is what's audit-worthy,
// not whatever the active backend happens to be at resume
// time.
let active_backend_name = if is_fresh {
let snap = self
.active_backend_name
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone();
if let Err(e) = store.set_string_param(SOURCE_NAME_PARAM, &snap).await {
return RunOutcome::Failed {
message: format!("failed to persist source_name to params: {e}"),
};
}
snap
} else {
match store.get_string_param(SOURCE_NAME_PARAM).await {
Ok(Some(name)) => name,
Ok(None) => {
// Paused row predates K3.8's source-stamping.
// Fall back to current active name and log a
// note so the audit trail is at least
// approximately correct.
let fallback = self
.active_backend_name
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone();
tracing::warn!(
target: "oxicloud::migration",
event = "backend_migration.legacy_paused_row_source_defaulted",
run_id = %store.run_id(),
fallback_source = %fallback,
"resumed run has no source_name in params (pre-K3.8 row) — defaulting \
to current active backend for the audit line"
);
fallback
}
Err(e) => {
return RunOutcome::Failed {
message: format!("read {SOURCE_NAME_PARAM} from params: {e}"),
};
}
}
};
// First-line guard: target name equals the currently-active
// entry. Silent no-op if we let it through — the app would
@@ -274,11 +330,11 @@ impl RecoverableJobHandler for StorageMigrationService {
if target_name == active_backend_name {
tracing::warn!(
target: "audit",
event = "storage_migration.refused_noop",
event = "backend_migration.refused_noop",
run_id = %store.run_id(),
target_name = %target_name,
active = %active_backend_name,
"storage_migration refused: target equals the currently-active entry"
"backend_migration refused: target equals the currently-active entry"
);
return RunOutcome::Failed {
message: format!(
@@ -328,7 +384,10 @@ impl RecoverableJobHandler for StorageMigrationService {
if let Some(source) = source_entry
&& entry_identity(source) == entry_identity(target_entry)
{
let key_differs = source.encryption_key_base64 != target_entry.encryption_key_base64;
// Compare head-pair materials — the "write key" for each
// entry. A non-head pair difference (mid-rotation) doesn't
// count as a key change for the purposes of this refusal.
let key_differs = source.head_key_material() != target_entry.head_key_material();
let hint = if key_differs {
" (encryption key differs → this looks like an in-place key rotation; \
create a new entry pointing at a DIFFERENT bucket / dir, migrate to it, \
@@ -338,12 +397,12 @@ impl RecoverableJobHandler for StorageMigrationService {
};
tracing::warn!(
target: "audit",
event = "storage_migration.refused_same_physical_storage",
event = "backend_migration.refused_same_physical_storage",
run_id = %store.run_id(),
target_name = %target_name,
source_name = %active_backend_name,
encryption_differs = key_differs,
"storage_migration refused: named target differs from source but physical storage matches"
"backend_migration refused: named target differs from source but physical storage matches"
);
return RunOutcome::Failed {
message: format!(
@@ -356,7 +415,13 @@ impl RecoverableJobHandler for StorageMigrationService {
// Build target backend via the shared factory — same code
// path boot uses, so the encryption decorator wrapping is
// uniform.
let target = build_entry_backend(target_entry, &self.storage_path_fallback);
// Typed variant: we need the wrapper's `is_at_head_format`
// + `put_blob_from_bytes_replace` for the smart-skip probe
// and overwrite path (the trait-object `put_blob` silently
// no-ops on existing target blobs — that's the bug this
// commit fixes end-to-end). The `Arc<dyn>` coercion is free
// for the swap-hot-swap call in `finish_completed`.
let target = build_entry_backend_typed(target_entry, &self.storage_path_fallback);
if let Err(e) = target.initialize().await {
return RunOutcome::Failed {
message: format!("target backend init: {e}"),
@@ -404,29 +469,45 @@ impl RecoverableJobHandler for StorageMigrationService {
.await
.map(|n| n.max(0) as u64)
.unwrap_or(0);
// On Resume, seed the counter with what's already been done
// in prior sessions — else the banner shows "500 / 1536"
// right after resuming a run that had reached 900/1536,
// which misleads admins into thinking the migration
// regressed. Fresh run reports 0. `stats.scanned_count`
// was written by `checkpoint` after each batch, so it's
// durable across restarts.
let already_scanned = if is_fresh {
0
} else {
store.scanned_count().await.unwrap_or(0)
};
{
let mut guard = self
.migration_progress
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = Some(crate::common::migration_progress::MigrationProgress::new(
let mut progress = crate::common::migration_progress::MigrationProgress::new(
target_name.clone(),
total_blobs,
));
);
if already_scanned > 0 {
progress.bump(already_scanned);
}
*guard = Some(progress);
}
let source_kind = self.source.backend_type();
let target_kind = target.backend_type();
tracing::info!(
target: "audit",
event = "storage_migration.run_started",
event = "backend_migration.run_started",
run_id = %store.run_id(),
source_name = %active_backend_name,
target_name = %target_name,
source_kind = source_kind,
target_kind = target_kind,
resuming = !is_fresh,
"storage_migration starting {active_backend_name} ({source_kind}) → {target_name} ({target_kind})"
"backend_migration starting {active_backend_name} ({source_kind}) → {target_name} ({target_kind})"
);
// Cursor = the last-visited blob hash, UTF-8-encoded. On resume
@@ -446,7 +527,13 @@ impl RecoverableJobHandler for StorageMigrationService {
};
let mut copied_count = 0u64;
let mut skipped_count = 0u64;
// Populated by the smart-skip probe below: target blob
// already exists at the current head format+key, so a
// rewrite would be identical bytes. Cheap (15-byte range
// read via `is_at_head_format`), massive latency win on
// resume + on backends where the source was rotated to the
// same key as the target already had.
let mut skipped_count: u64 = 0;
let mut failed_count = 0u64;
let mut source_missing_count = 0u64;
@@ -456,13 +543,13 @@ impl RecoverableJobHandler for StorageMigrationService {
Ok(RunStatus::CancelRequested) => {
tracing::info!(
target: "oxicloud::migration",
event = "storage_migration.cancelled",
event = "backend_migration.cancelled",
run_id = %store.run_id(),
copied = copied_count,
skipped = skipped_count,
failed = failed_count,
source_missing = source_missing_count,
"storage_migration cancelled cooperatively, pausing"
"backend_migration cancelled cooperatively, pausing"
);
return RunOutcome::Paused {
cursor: cursor
@@ -533,7 +620,7 @@ impl RecoverableJobHandler for StorageMigrationService {
source_missing_count += 1;
tracing::warn!(
target: "oxicloud::migration",
event = "storage_migration.source_missing",
event = "backend_migration.source_missing",
run_id = %store.run_id(),
hash = %hash,
source = source_kind,
@@ -541,7 +628,7 @@ impl RecoverableJobHandler for StorageMigrationService {
);
record_or_log(
store,
STORAGE_MIGRATION_JOB_NAME,
BACKEND_MIGRATION_JOB_NAME,
"source_missing",
"data_loss",
None,
@@ -564,7 +651,7 @@ impl RecoverableJobHandler for StorageMigrationService {
// it.
tracing::warn!(
target: "oxicloud::migration",
event = "storage_migration.source_probe_error",
event = "backend_migration.source_probe_error",
run_id = %store.run_id(),
hash = %hash,
error = %e,
@@ -574,36 +661,77 @@ impl RecoverableJobHandler for StorageMigrationService {
}
}
// Skip when the target already has it — supports
// idempotent resume and cheap re-runs against a
// partially-migrated target.
match target.blob_exists(hash).await {
Ok(true) => {
skipped_count += 1;
continue;
}
Ok(false) => {}
Err(e) => {
tracing::warn!(
target: "oxicloud::migration",
event = "storage_migration.blob_exists_error",
run_id = %store.run_id(),
hash = %hash,
error = %e,
"blob_exists probe on target failed; attempting copy anyway"
);
}
// Smart skip via v1 header inspection: read the
// target's first 15 bytes and only rewrite if the
// stored format+key_fp differs from the target's
// current head. This is the "if file exists and
// destination key mismatch, overwrite it" rule.
//
// Failure of the probe → treat as "not at head" →
// fall through to overwrite. Safe because the
// overwrite is atomic (Local: tempfile + rename;
// S3/Azure: unconditional PUT via the newly-fixed
// `put_blob_from_bytes_replace` overrides).
//
// Historical note: earlier we had `blob_exists`
// skip (K1.0), which silently skipped mismatched
// keys and produced mixed-encryption backends. K1.2
// removed that skip and made every blob rewrite
// unconditionally. This slice replaces the
// unconditional rewrite with a smart skip that's
// both correct (checks the header, not just
// existence) AND fast (skips the ~99% of resume
// blobs already at head format).
let head_check = target.head_check(hash).await;
if matches!(head_check, HeadCheck::Match) {
skipped_count += 1;
tracing::debug!(
target: "oxicloud::migration",
event = "backend_migration.blob_skipped_head_match",
run_id = %store.run_id(),
hash = %hash,
head_format = %target.head_format(),
"target blob already at head format — skipping rewrite"
);
continue;
}
match copy_blob(self.source.as_ref(), target.as_ref(), hash).await {
match copy_blob(self.source.as_ref(), target.clone(), hash).await {
Ok(()) => {
copied_count += 1;
// Log the concrete action: overwrite (blob
// existed with wrong format — the K1.2 repair
// case) vs fresh write (blob absent). Info
// level for both so a single log tail shows
// operators exactly what happened per blob.
match head_check {
HeadCheck::Mismatch(prev) => tracing::info!(
target: "oxicloud::migration",
event = "backend_migration.blob_overwritten",
run_id = %store.run_id(),
hash = %hash,
previous_format = %prev,
new_format = %target.head_format(),
"🔄 target blob existed with different format — overwritten"
),
HeadCheck::Absent => tracing::info!(
target: "oxicloud::migration",
event = "backend_migration.blob_written",
run_id = %store.run_id(),
hash = %hash,
new_format = %target.head_format(),
"✍️ fresh blob written to target"
),
// Unreachable in practice — we already
// early-`continue`d on Match above.
HeadCheck::Match => {}
}
}
Err(e) => {
failed_count += 1;
tracing::warn!(
target: "oxicloud::migration",
event = "storage_migration.blob_failed",
event = "backend_migration.blob_failed",
run_id = %store.run_id(),
hash = %hash,
error = %e,
@@ -614,7 +742,7 @@ impl RecoverableJobHandler for StorageMigrationService {
// where the admin UI reads it.
record_or_log(
store,
STORAGE_MIGRATION_JOB_NAME,
BACKEND_MIGRATION_JOB_NAME,
"migration_failed",
"data_loss",
None,
@@ -676,7 +804,7 @@ impl RecoverableJobHandler for StorageMigrationService {
}
}
impl StorageMigrationService {
impl BackendMigrationService {
/// Terminal successful path — reached from both Completed sites
/// in the batch loop (empty-first-batch and short-batch).
///
@@ -716,6 +844,68 @@ impl StorageMigrationService {
failed: u64,
source_missing: u64,
) -> RunOutcome {
// ── Failure gate ───────────────────────────────────────────
// Refuse to flip the active backend if ANY blob failed to
// migrate. Flipping to a partial target strands live traffic
// on incomplete data — reads for the missing hashes would
// 404. Findings are already recorded per-blob in the batch
// loop; operator inspects, then either:
// - retries (walk short-circuits on already-present blobs,
// so the retry costs only the failed ones), OR
// - fixes the source, retries, OR
// - accepts the loss and manually flips via
// `oxicloud --select-storage <target>`.
//
// Readonly is cleared either way — the source is still the
// active backend, and users shouldn't be locked out because
// of a partial run. Progress snapshot cleared too so the
// header middleware stops emitting.
if failed > 0 {
let readonly_persisted = persist_migration_readonly(self.pool.as_ref(), false)
.await
.is_ok();
self.migration_readonly.store(false, Ordering::Relaxed);
{
let mut guard = self
.migration_progress
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = None;
}
if !readonly_persisted {
tracing::warn!(
target: "oxicloud::migration",
event = "backend_migration.readonly_clear_persist_failed",
run_id = %store.run_id(),
"cleared migration_readonly in memory (writes allowed against source) but the \
DB persist failed. Boot-clear rule will fix on next restart."
);
}
tracing::info!(
target: "audit",
event = "backend_migration.aborted",
reason = "blobs_failed",
run_id = %store.run_id(),
active_backend_name = previous_active,
target_name = target_name,
copied = copied,
skipped = skipped,
failed = failed,
source_missing = source_missing,
"🛑 backend_migration aborted — {failed} blob(s) failed, active backend left at \
`{previous_active}`, readonly cleared. Inspect findings and retry, or accept \
the partial migration via `oxicloud --select-storage {target_name}`."
);
return RunOutcome::Failed {
message: format!(
"{failed} blob(s) failed to migrate — active backend NOT switched \
(still `{previous_active}`). Retry the run (short-circuits on already-copied \
blobs) or accept the partial migration manually via \
`oxicloud --select-storage {target_name}`."
),
};
}
// 1. DB pointer.
if let Err(e) = persist_active_backend_name(self.pool.as_ref(), target_name).await {
return RunOutcome::Failed {
@@ -761,7 +951,7 @@ impl StorageMigrationService {
if !readonly_persisted {
tracing::warn!(
target: "oxicloud::migration",
event = "storage_migration.readonly_clear_persist_failed",
event = "backend_migration.readonly_clear_persist_failed",
run_id = %store.run_id(),
"cleared migration_readonly in memory (writes allowed) but the DB persist \
failed. If the server crashes before next boot, boot will re-seed the flag \
@@ -771,7 +961,7 @@ impl StorageMigrationService {
tracing::info!(
target: "audit",
event = "storage_migration.completed",
event = "backend_migration.completed",
run_id = %store.run_id(),
active_backend_name = target_name,
previous_active = previous_active,
@@ -779,10 +969,18 @@ impl StorageMigrationService {
skipped = skipped,
failed = failed,
source_missing = source_missing,
"✅ storage_migration completed — hot-swapped runtime backend to `{target_name}`, \
"✅ backend_migration completed — hot-swapped runtime backend to `{target_name}`, \
writes resumed. No restart required."
);
RunOutcome::Completed
// Per-run summary counters merged into `stats` for the admin
// UI drawer. Same shape as `backend_rotate`'s extras + one
// extra `source_missing` counter unique to migration.
RunOutcome::completed_with(serde_json::json!({
"copied": copied,
"skipped": skipped,
"failed": failed,
"source_missing": source_missing,
}))
}
}
@@ -838,53 +1036,39 @@ fn entry_identity(entry: &NamedStorageEntry) -> String {
/// cleans up on reboot).
async fn copy_blob(
source: &dyn BlobStorageBackend,
target: &dyn BlobStorageBackend,
target: Arc<EncryptedBlobBackend>,
hash: &str,
) -> Result<(), DomainError> {
let tmp_dir = std::env::temp_dir().join("oxicloud-migration");
tokio::fs::create_dir_all(&tmp_dir).await.map_err(|e| {
DomainError::internal_error(
"StorageMigration",
format!("create temp dir {}: {e}", tmp_dir.display()),
)
})?;
let tmp_path = tmp_dir.join(format!("{hash}.tmp"));
if let Err(e) = write_source_to_tmp(source, hash, &tmp_path).await {
let _ = tokio::fs::remove_file(&tmp_path).await;
return Err(e);
}
let put_result = target.put_blob(hash, &tmp_path).await;
let _ = tokio::fs::remove_file(&tmp_path).await;
put_result.map(|_bytes_written| ())
// Read the full plaintext from source (source is wrapped in
// `EncryptedBlobBackend`, so its `get_blob_stream` decrypts
// transparently — even legacy plaintext blobs come out clean
// via the BLAKE3 rescue). Collected in-memory because the
// target's `put_blob_from_bytes_replace` needs a `Bytes`.
//
// For CDC chunks (every blob written since chunking landed)
// this is ≤ 1 MiB. Legacy whole-file blobs pay a full-blob
// buffer here; acceptable given rotate/migration are admin-
// triggered operations. Streaming through a temp file (the
// old shape) doesn't help — target still needs the bytes.
let stream = source.get_blob_stream(hash).await?;
let plaintext = collect_stream_bytes(stream).await?;
target
.put_blob_from_bytes_replace(hash, plaintext)
.await
.map(|_bytes_written| ())
}
async fn write_source_to_tmp(
source: &dyn BlobStorageBackend,
hash: &str,
tmp_path: &Path,
) -> Result<(), DomainError> {
use tokio::io::AsyncWriteExt;
let stream = source.get_blob_stream(hash).await?;
let mut file = tokio::fs::File::create(tmp_path).await.map_err(|e| {
DomainError::internal_error(
"StorageMigration",
format!("create temp file {}: {e}", tmp_path.display()),
)
})?;
async fn collect_stream_bytes(
stream: crate::application::ports::blob_storage_ports::BlobStream,
) -> Result<Bytes, DomainError> {
use bytes::BytesMut;
let mut buf = BytesMut::new();
let mut stream = std::pin::pin!(stream);
while let Some(chunk) = stream.next().await {
let bytes = chunk.map_err(|e| {
DomainError::internal_error("StorageMigration", format!("source stream read: {e}"))
})?;
file.write_all(&bytes).await.map_err(|e| {
DomainError::internal_error("StorageMigration", format!("temp file write: {e}"))
DomainError::internal_error("BackendMigration", format!("source stream read: {e}"))
})?;
buf.extend_from_slice(&bytes);
}
file.flush()
.await
.map_err(|e| DomainError::internal_error("StorageMigration", format!("temp flush: {e}")))?;
Ok(())
Ok(buf.freeze())
}
src/infrastructure/services/backend_rotate_service.rs
+546
View File
@@ -0,0 +1,546 @@
//! Storage-format rotation as a recoverable-run tenant (K3 of
//! `docs/plan/storage-key-rotation.md`).
//!
//! Iterates `storage.blobs` for a target entry, decides per blob
//! whether the on-disk format matches what the entry's head pair
//! would write, and rewrites in place when it doesn't. Covers four
//! transitions with a single equality check:
//!
//! * Legacy blob (no `OXCPT` magic) → rewrite as v1 with the head
//! pair's format.
//! * v1 encrypted, decrypted under a pair-index other than head →
//! rewrite (key rotation).
//! * v1 plaintext with head=`aes:K` → rewrite (encrypt-in-place).
//! * v1 encrypted with head=`none:` → rewrite (decrypt-in-place).
//!
//! ### No readonly, no cutover
//!
//! `backend_rotate` is per-blob idempotent — repeat rewrites are
//! byte-safe (content-addressability holds; the wrapper always
//! produces the head format). Concurrent user writes coexist: they
//! land as head-format themselves, so when the walk reaches that
//! hash the classifier reports "already at head format" and the
//! decision tree collapses to `skip`. No app-wide read-only gate is
//! ever engaged — a critical improvement over `backend_migration`,
//! whose target-different-from-source cutover forces one.
//!
//! ### Restart survival
//!
//! Cursor + per-blob failure findings are persisted after every
//! batch. On restart, boot flips any abandoned `Running` row to
//! `Paused`; an admin trigger resumes from the checkpointed cursor.
//! The last checkpoint window (~100 blobs) re-processes; each of
//! those blobs is now head-format from the previous run's rewrite,
//! so the walk short-circuits without re-writing. Effectively free.
//!
//! ### Design notes
//!
//! * **Cursor** — UTF-8 hex of the last-processed blob hash (64
//! chars). Same encoding as `backend_migration` and
//! `blobs_consistency`.
//! * **Target lookup** — the entry NAME is stashed in `params` at
//! Fresh-open time and re-read on Resume. The wrapper for that
//! entry is rebuilt at the top of every run via
//! `build_entry_backend_typed`; mid-run config changes are
//! ignored until the next run (mirrors `backend_migration`).
//! * **Per-blob failures don't fail the run** — each failure records
//! a `rotation_failed` finding (severity `data_loss` — the bytes
//! didn't get rewritten) and the walk continues. A run that
//! completes with zero findings is proof every blob is at head
//! format.
//! * **`?deep=true` is unused** — rotation has no slow variant.
//! Parameter accepted for uniformity with other tenants; ignored.
use std::path::PathBuf;
use std::sync::Arc;
use async_trait::async_trait;
use bytes::Bytes;
use sqlx::PgPool;
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::NamedStorageEntry;
use crate::common::migration_progress::MigrationProgress;
use crate::infrastructure::scheduler::{
JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
RunStatus, record_or_log,
};
use crate::infrastructure::services::encrypted_blob_backend::BlobFormat;
use crate::infrastructure::services::entry_backend::build_entry_backend_typed;
pub const BACKEND_ROTATE_JOB_NAME: &str = "backend_rotate";
/// The `params` JSONB key under which the run's target entry name is
/// stashed at Fresh-open time via `JobStore::set_string_param`.
/// Kept identical to `backend_migration`'s TARGET_NAME_PARAM so
/// operators grepping run rows see the same convention across both
/// storage-touching tenants.
pub const TARGET_NAME_PARAM: &str = "target_name";
/// Rows per batch. Matches `backend_migration` / `blobs_consistency`
/// so the checkpoint + cancel-poll cadence is uniform across tenants.
const BATCH_SIZE: i64 = 100;
pub struct BackendRotateService {
pool: Arc<PgPool>,
/// Immutable per-deploy snapshot; used to look up the target
/// entry by name at run start. Matches `AppConfig.storage_entries`.
storage_entries: Vec<NamedStorageEntry>,
/// Ambient `AppConfig.storage_path` used as the `root_dir`
/// fallback for a Local target entry that doesn't declare its
/// own `_ROOT_DIR`. Same fallback rule as boot
/// (`build_entry_backend`).
storage_path_fallback: PathBuf,
/// Shared in-memory progress snapshot for the server-status
/// header middleware. `Some(_)` while a rotation is
/// running/paused, `None` otherwise. Distinct from
/// `AppState.migration_progress` so the header can broadcast
/// migration + rotation states independently.
rotation_progress: Arc<std::sync::RwLock<Option<MigrationProgress>>>,
}
impl BackendRotateService {
pub fn new(
pool: Arc<PgPool>,
storage_entries: Vec<NamedStorageEntry>,
storage_path_fallback: PathBuf,
rotation_progress: Arc<std::sync::RwLock<Option<MigrationProgress>>>,
) -> Self {
Self {
pool,
storage_entries,
storage_path_fallback,
rotation_progress,
}
}
/// Chainable self-registration — mirrors the `*_consistency`
/// tenants and `backend_migration`. On-demand only (no periodic
/// tick).
pub async fn register_recoverable_job(
self: Arc<Self>,
registry: &JobRegistry,
provider: &Arc<dyn JobStoreProvider>,
) -> Arc<Self> {
registry
.register_recoverable_job(self.clone(), provider.clone(), None)
.await;
self
}
}
#[async_trait]
impl RecoverableJobHandler for BackendRotateService {
fn name(&self) -> &str {
BACKEND_ROTATE_JOB_NAME
}
/// Definitive count — one row per blob. Same query as
/// `backend_migration::count_total`; the two walk the same rows.
async fn count_total(&self) -> Option<u64> {
let row: Result<(i64,), sqlx::Error> = sqlx::query_as("SELECT COUNT(*) FROM storage.blobs")
.fetch_one(self.pool.as_ref())
.await;
match row {
Ok((n,)) => Some(n.max(0) as u64),
Err(e) => {
tracing::debug!(
target: "oxicloud::rotate",
event = "backend_rotate.count_total_failed",
error = %e,
"count_total failed — run will not surface a progress bar"
);
None
}
}
}
async fn run_resumable(
&self,
store: &dyn JobStore,
args: &JobRunArgs,
resume_cursor: Option<Vec<u8>>,
) -> RunOutcome {
// Resolve target entry name — same shape as `backend_migration`.
let is_fresh = resume_cursor.is_none();
let target_name = if is_fresh {
let Some(name) = args.storage.clone() else {
return RunOutcome::Failed {
message: "backend_rotate requires `target_name` on a fresh run — trigger via \
POST /api/admin/storage/entries/{name}/rotate"
.to_string(),
};
};
if let Err(e) = store.set_string_param(TARGET_NAME_PARAM, &name).await {
return RunOutcome::Failed {
message: format!("failed to persist target_name to params: {e}"),
};
}
name
} else {
match store.get_string_param(TARGET_NAME_PARAM).await {
Ok(Some(name)) => name,
Ok(None) => {
return RunOutcome::Failed {
message: format!(
"resumed run has no {TARGET_NAME_PARAM} in params — cancel + trigger \
fresh."
),
};
}
Err(e) => {
return RunOutcome::Failed {
message: format!("read {TARGET_NAME_PARAM} from params: {e}"),
};
}
}
};
// Look up the target entry.
let target_entry = match self.storage_entries.iter().find(|e| e.name == target_name) {
Some(e) => e,
None => {
let available = if self.storage_entries.is_empty() {
"(none)".to_string()
} else {
self.storage_entries
.iter()
.map(|e| e.name.as_str())
.collect::<Vec<_>>()
.join(", ")
};
return RunOutcome::Failed {
message: format!(
"target entry `{target_name}` not declared in `OXICLOUD_STORAGE_ENTRIES` — \
available: {available}."
),
};
}
};
// Build the wrapper for this entry — typed so we can call
// `read_and_classify` + `head_format` directly.
let wrapper = build_entry_backend_typed(target_entry, &self.storage_path_fallback);
if let Err(e) = wrapper.initialize().await {
return RunOutcome::Failed {
message: format!("target entry `{target_name}` failed to initialize: {e}"),
};
}
let head_format = wrapper.head_format();
tracing::info!(
target: "audit",
event = "backend_rotate.run_started",
run_id = %store.run_id(),
target_name = %target_name,
// `%` (Display) → SSH-style `encrypted-v1 key_fp=83:96:...`
// instead of the raw `[131, 150, 255, ...]` byte-array
// shape Debug produces. Matches how `xxd` renders the
// header bytes on disk.
head_format = %head_format,
resuming = !is_fresh,
"backend_rotate started on `{target_name}` (head_format = {head_format})"
);
// Seed the progress snapshot. Total = count_total's estimate;
// if that failed we still surface the header without a
// denominator so the banner shows "rotation in progress" at
// minimum.
let total = self.count_total().await.unwrap_or(0);
{
let mut guard = self
.rotation_progress
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = Some(MigrationProgress::new(target_name.clone(), total));
}
let mut cursor: Option<String> = match resume_cursor {
None => None,
Some(bytes) if bytes.is_empty() => None,
Some(bytes) => match String::from_utf8(bytes) {
Ok(s) => Some(s),
Err(e) => {
self.clear_progress();
return RunOutcome::Failed {
message: format!("invalid cursor: not valid UTF-8: {e}"),
};
}
},
};
let mut rewritten_count = 0u64;
let mut skipped_count = 0u64;
let mut failed_count = 0u64;
loop {
// Cooperative cancel poll between batches.
match store.status().await {
Ok(RunStatus::CancelRequested) => {
self.clear_progress();
tracing::info!(
target: "oxicloud::rotate",
event = "backend_rotate.cancelled",
run_id = %store.run_id(),
rewritten = rewritten_count,
skipped = skipped_count,
failed = failed_count,
"backend_rotate cancelled cooperatively, pausing"
);
return RunOutcome::Paused {
cursor: cursor
.as_ref()
.map(|s| s.as_bytes().to_vec())
.unwrap_or_default(),
};
}
Ok(_) => {}
Err(e) => {
self.clear_progress();
return RunOutcome::Failed {
message: format!("status poll: {e}"),
};
}
}
// Fetch the next batch. Same keyset pagination shape as
// `backend_migration` — `hash > $1` on the PK, index-only.
let rows: Vec<(String,)> = match sqlx::query_as(
r#"
SELECT hash
FROM storage.blobs
WHERE ($1::text IS NULL OR hash > $1)
ORDER BY hash
LIMIT $2
"#,
)
.bind(cursor.as_deref())
.bind(BATCH_SIZE)
.fetch_all(self.pool.as_ref())
.await
{
Ok(r) => r,
Err(e) => {
self.clear_progress();
return RunOutcome::Failed {
message: format!("batch fetch: {e}"),
};
}
};
if rows.is_empty() {
return self
.finish_completed(
store,
&target_name,
head_format,
rewritten_count,
skipped_count,
failed_count,
)
.await;
}
for (hash,) in &rows {
// Read + classify in one round-trip. Failure here is
// a real read failure (e.g. blob missing on disk),
// recorded as a finding.
let (plaintext, current_format) = match wrapper.read_and_classify(hash).await {
Ok(pair) => pair,
Err(e) => {
failed_count += 1;
tracing::warn!(
target: "oxicloud::rotate",
event = "backend_rotate.read_failed",
run_id = %store.run_id(),
hash = %hash,
error = %e,
"failed to read blob for classification; recording finding"
);
record_or_log(
store,
BACKEND_ROTATE_JOB_NAME,
"rotation_failed",
"data_loss",
None,
serde_json::json!({
"hash": hash,
"phase": "read",
"error": e.to_string(),
}),
)
.await;
continue;
}
};
// The whole decision tree collapses to one equality
// check thanks to `BlobFormat`'s `PartialEq`. Six
// cases in the plan → one branch here.
if current_format == head_format {
skipped_count += 1;
continue;
}
// Rewrite via the atomic-replace write path.
//
// **NOT** `put_blob_from_bytes`: that variant is
// idempotent-skip (`O_CREAT|O_EXCL` on
// `LocalBlobBackend`) — correct for uploads (same
// plaintext ↔ any ciphertext at hash decrypts back)
// but a silent no-op for us. Rotate NEEDS the on-disk
// bytes to change (legacy → v1 header, old key → new
// key, plaintext ↔ encrypted). Ed hit this on
// 2026-08-02: rotation reported success in 9s but
// every blob on disk still had the legacy shape.
// `put_blob_from_bytes_replace` writes to a tempfile
// + atomic `rename(2)`s over the existing object key.
if let Err(e) = wrapper
.put_blob_from_bytes_replace(hash, Bytes::from(plaintext.to_vec()))
.await
{
failed_count += 1;
tracing::warn!(
target: "oxicloud::rotate",
event = "backend_rotate.write_failed",
run_id = %store.run_id(),
hash = %hash,
error = %e,
"failed to rewrite blob; recording finding"
);
record_or_log(
store,
BACKEND_ROTATE_JOB_NAME,
"rotation_failed",
"data_loss",
None,
serde_json::json!({
"hash": hash,
"phase": "write",
"from": format!("{current_format}"),
"to": format!("{head_format}"),
"error": e.to_string(),
}),
)
.await;
continue;
}
rewritten_count += 1;
}
// Advance cursor + checkpoint. `delta_count` = work
// attempted this batch, so the progress bar advances even
// when a batch is dominated by skips (steady-state
// re-run) or failures.
let last_hash = rows.last().map(|(h,)| h.clone()).expect("non-empty rows");
cursor = Some(last_hash.clone());
let batch_len = rows.len() as u64;
if let Err(e) = store.checkpoint(last_hash.into_bytes(), batch_len).await {
self.clear_progress();
return RunOutcome::Failed {
message: format!("checkpoint: {e}"),
};
}
{
let mut guard = self
.rotation_progress
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some(progress) = guard.as_mut() {
progress.bump(batch_len);
}
}
if (rows.len() as i64) < BATCH_SIZE {
return self
.finish_completed(
store,
&target_name,
head_format,
rewritten_count,
skipped_count,
failed_count,
)
.await;
}
}
}
}
impl BackendRotateService {
/// Terminal successful path — clear the header snapshot and log a
/// final audit line. Unlike `backend_migration::finish_completed`
/// there's no cutover / hot-swap step: rotation writes in place
/// on the entry that's already there.
///
/// `head_format` — the target format at run completion. Persisted
/// into the run row's `stats` as `head_format` (Display) +
/// `head_key_fp` (raw hex) so operators have a durable record of
/// "at time T, all blobs on entry E were normalised to fingerprint
/// F". Combined with `failed = 0`, that's the signal to remove
/// obsolete keys from `.env` — any key NOT matching `head_key_fp`
/// no longer decrypts any live blob and can be safely dropped.
async fn finish_completed(
&self,
store: &dyn JobStore,
target_name: &str,
head_format: BlobFormat,
rewritten: u64,
skipped: u64,
failed: u64,
) -> RunOutcome {
self.clear_progress();
// Render two fingerprint shapes:
// * `head_format` — Display impl, e.g.
// `encrypted-v1 key_fp=15:f3:8f:80:2c:ae:2c:50` — human
// friendly for audit logs + admin UI.
// * `head_key_fp` — bare 16-hex string, matches what an
// operator gets from `openssl dgst -sha256 <keyfile> | head -c 16`
// so post-hoc verification against the raw key material
// is trivial.
let head_display = format!("{head_format}");
let head_key_fp_hex = match head_format {
BlobFormat::EncryptedV1 { key_fp } => hex::encode(key_fp),
BlobFormat::PlaintextV1 => String::new(), // all-zero, uninformative
BlobFormat::Legacy => String::new(), // never emitted at head
};
tracing::info!(
target: "audit",
event = "backend_rotate.run_completed",
run_id = %store.run_id(),
target_name = %target_name,
rewritten = rewritten,
skipped = skipped,
failed = failed,
head_format = %head_display,
"backend_rotate completed on `{target_name}` — {rewritten} rewritten, {skipped} skipped, {failed} failed; head = {head_display}"
);
// Surface the per-run summary counters as extras merged into
// the run row's `stats` JSONB. Frontend renders whatever keys
// are present, so no wire-format bumping is needed — the
// admin UI's run drawer just picks these up alongside the
// engine-owned `finding_count` + `scanned_count`.
//
// `head_key_fp` empty string when head is not an encrypted
// pair (plaintext-v1) — frontend can render "all in clear"
// vs "all under fp <X>" based on that discriminator.
RunOutcome::completed_with(serde_json::json!({
"rewritten": rewritten,
"skipped": skipped,
"failed": failed,
"head_format": head_display,
"head_key_fp": head_key_fp_hex,
}))
}
fn clear_progress(&self) {
let mut guard = self
.rotation_progress
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
*guard = None;
}
}
src/infrastructure/services/blobs_consistency_service.rs
+100 -13
View File
@@ -19,6 +19,15 @@
//! runs when the operator passes `?deep=true` because it costs a
//! full read of every blob.
//!
//! * `blob_unreadable` (severity `data_loss`, deep mode only) —
//! `blob_exists` returned true but the read pipeline errored (can't
//! decrypt, network glitch, permission error, etc.). Distinct from
//! `blob_corrupted` (which requires successful read + hash mismatch);
//! here we can't get bytes out at all. Same operator impact — any
//! file referencing this hash is inaccessible — but the remedy
//! differs (key recovery, retry, or blob replacement, depending on
//! the recorded `error` field).
//!
//! * `refcount_mismatch` (severity `inconsistent`) —
//! `storage.blobs.ref_count` disagrees with the actual reference
//! count computed from `storage.files.blob_hash` +
@@ -66,7 +75,7 @@ pub const BLOBS_CONSISTENCY_JOB_NAME: &str = "blobs_consistency";
/// `params` JSONB key under which the entry name being probed is
/// stashed on a Fresh run (matches `TARGET_NAME_PARAM` on
/// `storage_migration`). Resumed runs re-read it so a paused audit
/// `backend_migration`). Resumed runs re-read it so a paused audit
/// survives restart without the admin re-specifying the target.
pub const PROBED_STORAGE_PARAM: &str = "probed_storage";
@@ -180,7 +189,7 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
resume_cursor: Option<Vec<u8>>,
) -> RunOutcome {
// Resolve the backend to probe. Two paths, mirroring the
// Fresh/Resumed split the storage_migration handler uses:
// Fresh/Resumed split the backend_migration handler uses:
//
// * Fresh + args.storage=Some — probe that named entry
// instead of the live backend. Stamp probed_storage in
@@ -251,6 +260,12 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
);
}
// Snapshot "is this a Fresh run?" BEFORE the resume_cursor
// match consumes it — otherwise the `is_none()` check later
// borrows a partially-moved value. Fresh = no cursor bytes
// at all; Resumed = cursor bytes present (possibly empty).
let is_fresh = resume_cursor.is_none();
// Cursor = the last-visited `hash` string, UTF-8-encoded. On
// resume, we walk `WHERE hash > $cursor` in ASC order. First
// batch: NULL cursor → start from the smallest hash.
@@ -272,10 +287,49 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
// `record_finding` on each emission).
let mut finding_count = 0u64;
// Deep mode = re-hash bytes for bit-rot detection. Logged
// once at run start so operators tailing tracing know why the
// scan is taking hours.
if args.deep {
// Deep mode is a per-run flag with two consumers:
// 1. This handler — decides whether to re-hash bytes.
// 2. The admin panel — needs to display whether the run
// was deep so operators know what the scan actually
// verified.
//
// On a Fresh run we take it from `deep` (the trigger
// endpoint stamps `?deep=true` onto the args) and stash it
// in `params.deep` so:
// * Resume picks up the same mode (would previously become
// non-deep on Resume — a Paused deep scan silently lost
// its `deep` intent).
// * The admin panel run-detail view can render
// `params.deep = "true"` alongside `target_name`,
// `progress_kind`, etc.
//
// Persist BEFORE the walk so a mid-fresh-batch crash still
// leaves a Paused row with the right mode marker.
let deep = if is_fresh {
let deep = args.deep;
let v = if deep { "true" } else { "false" };
if let Err(e) = store.set_string_param("deep", v).await {
return RunOutcome::Failed {
message: format!("failed to persist deep flag to params: {e}"),
};
}
deep
} else {
// Resumed run — read the persisted flag. Default to
// false (fast mode) if the row is a pre-K3.5 Paused
// scan without the param stashed.
match store.get_string_param("deep").await {
Ok(Some(v)) => v == "true",
Ok(None) => false,
Err(e) => {
return RunOutcome::Failed {
message: format!("read `deep` from params: {e}"),
};
}
}
};
if deep {
tracing::info!(
target: "oxicloud::consistency",
event = "blobs_consistency.deep_mode_active",
@@ -377,11 +431,11 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
event = "blobs_consistency.completed",
run_id = %store.run_id(),
finding_count = finding_count,
deep = args.deep,
deep = deep,
"blobs_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
let grace_cutoff = Utc::now() - CREATE_GRACE;
@@ -472,7 +526,7 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
// `expected_hash` was NOT reused as a name to
// avoid mistaking it for "the hash we expect to
// see on disk (i.e. what will fix this)".
if args.deep {
if deep {
match recompute_hash(backend.as_ref(), &row.hash).await {
Ok(computed_hash) if computed_hash == row.hash => {}
Ok(computed_hash) => {
@@ -495,13 +549,46 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
.await;
}
Err(e) => {
// Blob can't be read at all — record as
// `blob_unreadable`. Distinct from
// `blob_corrupted` (hash mismatch = we
// can read but content differs): here
// we can't get bytes out to hash. Common
// causes: decrypt failure (missing key),
// network glitch on S3/Azure, missing
// file on Local, permission error.
//
// Recorded as `data_loss` because from
// the file's perspective the outcome is
// the same as corruption: content is
// inaccessible. Admins triage the error
// string to distinguish transient
// (retry-safe) from permanent (needs
// key recovery or blob replacement).
finding_count += 1;
let affected = affected_files(self.pool.as_ref(), &row.hash).await;
record_or_log(
store,
BLOBS_CONSISTENCY_JOB_NAME,
"blob_unreadable",
"data_loss",
None,
serde_json::json!({
"hash": row.hash,
"size": row.size,
"ref_count": row.ref_count,
"affected_files": affected,
"error": e.to_string(),
}),
)
.await;
tracing::warn!(
target: "oxicloud::consistency",
event = "blobs_consistency.recompute_hash_error",
event = "blobs_consistency.blob_unreadable",
run_id = %store.run_id(),
hash = %row.hash,
error = %e,
"recompute_hash failed; not a corruption signal on its own"
"🚨 blob unreadable in deep mode — recorded finding, continuing"
);
}
}
@@ -524,11 +611,11 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
event = "blobs_consistency.completed",
run_id = %store.run_id(),
finding_count = finding_count,
deep = args.deep,
deep = deep,
"blobs_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
}
}
src/infrastructure/services/dedup_service.rs
+1 -1
View File
@@ -3672,7 +3672,7 @@ mod rechunk_integration_tests {
let inner = Arc::new(LocalBlobBackend::new(&dir.path().join("blobs")));
inner.initialize().await.expect("init backend");
let key = EncryptedBlobBackend::generate_key();
let backend = Arc::new(EncryptedBlobBackend::new(inner, &key));
let backend = Arc::new(EncryptedBlobBackend::new_single_aes(inner, &key));
DedupService::new(backend, pool.clone(), pool.clone())
}
src/infrastructure/services/drives_consistency_service.rs
+9 -7
View File
@@ -1,9 +1,12 @@
//! First tenant of Part 2 (recoverable-run engine).
//!
//! Iterates `storage.drives` and reports each drive whose cached
//! `used_bytes` differs from `SUM(files.size) WHERE NOT is_trashed`
//! for that drive. **Read-only** — reports drift as findings but does
//! NOT fix it. The existing `storage_reconcile` job (Part 1) is what
//! `used_bytes` differs from `SUM(files.size)` for that drive.
//! Includes trashed files — matches the hot-path delta (upload
//! writes never decrement on `move_to_trash`) and the sweep at
//! `storage_usage_service.rs::update_all_drives_storage_usage`.
//! **Read-only** — reports drift as findings but does
//! NOT fix it. The existing `usage_reconcile` job (Part 1) is what
//! corrects the counter; this check surfaces WHEN drift happens so
//! operators can trace it back to root cause (missed delta call,
//! delta failed silently, race, etc.).
@@ -142,7 +145,7 @@ impl RecoverableJobHandler for DrivesConsistencyCheck {
// Fetch next batch of drives + their actual SUM in one
// query. LEFT JOIN via correlated subquery gets us both
// sides in one round-trip; the storage_reconcile sweep
// sides in one round-trip; the usage_reconcile sweep
// uses the same shape.
// Grace window: skip drives created within the last hour.
// A drive being created RIGHT NOW may still have its first
@@ -167,7 +170,6 @@ impl RecoverableJobHandler for DrivesConsistencyCheck {
SELECT SUM(size)::bigint
FROM storage.files
WHERE drive_id = d.id
AND NOT is_trashed
), 0) AS actual_bytes
FROM storage.drives d
LEFT JOIN storage.folders rf ON rf.id = d.root_folder_id
@@ -199,7 +201,7 @@ impl RecoverableJobHandler for DrivesConsistencyCheck {
"drives_consistency completed with {} drift finding(s)",
drift_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
// Per-row check: cached vs actual. This is the ONE check
@@ -255,7 +257,7 @@ impl RecoverableJobHandler for DrivesConsistencyCheck {
"drives_consistency completed with {} drift finding(s)",
drift_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
}
}
src/infrastructure/services/encrypted_blob_backend.rs
+1086 -84
View File
File diff suppressed because it is too large Load Diff
src/infrastructure/services/entry_backend.rs
+46 -30
View File
@@ -194,11 +194,47 @@ pub async fn resolve_active_entry<'a>(
///
/// Both are boot-fatal and indicate a code (not config) bug, so
/// panic is the honest response.
pub fn build_entry_backend(
/// Typed variant of [`build_entry_backend`] — returns the concrete
/// [`EncryptedBlobBackend`] wrapper so callers that need K3's
/// introspection API (`read_and_classify`, `head_format`, …) can hit
/// it directly without a downcast.
///
/// Same construction path as `build_entry_backend`; the trait-object
/// version delegates through this. Preferred for job handlers
/// (`backend_rotate`) that need typed access. The trait-object
/// version stays for the DI hot-path where the caller only needs
/// the generic `BlobStorageBackend` contract.
pub fn build_entry_backend_typed(
entry: &NamedStorageEntry,
local_storage_path_fallback: &Path,
) -> Arc<crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend> {
let base = build_base_backend(entry, local_storage_path_fallback);
let pairs = entry.encryption.clone().unwrap_or_default();
let mode = match entry.head_cipher() {
Some(crate::common::config::CipherKind::AesGcm256) => "encrypted-v1",
_ => "plaintext-v1",
};
tracing::info!(
"Storage entry `{}` — {} wrapper (pairs: {})",
entry.name,
mode,
pairs.len()
);
Arc::new(
crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend::new(
base, pairs,
),
)
}
/// Construct just the raw backend for the entry (no wrapper). Split
/// out of [`build_entry_backend`] so the typed variant can share the
/// switch on backend type without duplicating panic messages.
fn build_base_backend(
entry: &NamedStorageEntry,
local_storage_path_fallback: &Path,
) -> Arc<dyn BlobStorageBackend> {
let base: Arc<dyn BlobStorageBackend> = match entry.backend {
match entry.backend {
StorageBackendType::Local => {
let path = entry
.root_dir
@@ -227,32 +263,12 @@ pub fn build_entry_backend(
});
Arc::new(crate::infrastructure::services::azure_blob_backend::AzureBlobBackend::new(az))
}
};
// Encryption decorator — presence-implies-enabled, per plan §Encryption.
let Some(key_b64) = entry.encryption_key_base64.as_ref() else {
return base;
};
use crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend;
let key_bytes = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, key_b64)
.unwrap_or_else(|e| {
panic!(
"entry `{}` encryption key is not valid base64: {e} — parser was supposed to \
catch this at boot",
entry.name
)
});
let key: [u8; 32] = key_bytes.try_into().unwrap_or_else(|v: Vec<u8>| {
panic!(
"entry `{}` encryption key decoded to {} bytes; must be 32 — parser was supposed to \
catch this at boot",
entry.name,
v.len()
)
});
tracing::info!(
"Storage entry `{}` encrypted with AES-256-GCM (key from env)",
entry.name
);
Arc::new(EncryptedBlobBackend::new(base, &key))
}
}
pub fn build_entry_backend(
entry: &NamedStorageEntry,
local_storage_path_fallback: &Path,
) -> Arc<dyn BlobStorageBackend> {
build_entry_backend_typed(entry, local_storage_path_fallback)
}
src/infrastructure/services/files_consistency_service.rs
+2 -2
View File
@@ -312,7 +312,7 @@ impl RecoverableJobHandler for FilesConsistencyCheck {
"files_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
for row in &rows {
@@ -490,7 +490,7 @@ impl RecoverableJobHandler for FilesConsistencyCheck {
"files_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
}
}
src/infrastructure/services/folders_consistency_service.rs
+2 -2
View File
@@ -255,7 +255,7 @@ impl RecoverableJobHandler for FoldersConsistencyCheck {
"folders_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
// Per-row branches. Add new ones here — same pattern as
@@ -352,7 +352,7 @@ impl RecoverableJobHandler for FoldersConsistencyCheck {
"folders_consistency completed with {} finding(s)",
finding_count
);
return RunOutcome::Completed;
return RunOutcome::completed();
}
}
}
src/infrastructure/services/local_blob_backend.rs
+136 -2
View File
@@ -181,6 +181,40 @@ async fn write_blob_bytes(blob_path: &Path, data: &Bytes) -> Result<Option<File>
Ok(Some(file))
}
/// Delete every `*.replace.*.tmp` file in `dir` (best-effort).
///
/// Companion to `put_blob_from_bytes_replace`: those tempfiles are
/// created under `<hash>.replace.<pid>.<counter>.tmp` immediately
/// before the atomic `rename(2)` over the target. A crash between
/// `write_all + sync_all` and `rename` leaves the tempfile behind
/// with no owner (writer process gone). Since no other job cleans
/// them (`dedup_gc` and `backend_consistency` operate on canonical
/// `<hash>.blob` names), reap at boot in `initialize()`.
///
/// Silent on errors: a shard we can't read has bigger problems than
/// leaked tmp files, and the boot flow's own `create_dir_all` will
/// surface the underlying I/O error separately.
async fn reap_replace_tmpfiles_in(dir: &Path) {
let mut entries = match fs::read_dir(dir).await {
Ok(rd) => rd,
Err(_) => return,
};
while let Ok(Some(entry)) = entries.next_entry().await {
let name = entry.file_name();
let name_str = match name.to_str() {
Some(s) => s,
None => continue,
};
// Match `<hash>.replace.<pid>.<counter>.tmp` — precise-enough
// to avoid nuking anything a future feature might drop next
// to blobs. Requires the `.replace.` marker AND the `.tmp`
// suffix; a plain `<hash>.blob` never matches.
if name_str.contains(".replace.") && name_str.ends_with(".tmp") {
let _ = fs::remove_file(entry.path()).await;
}
}
}
/// Bench-only public wrapper (feature = "bench") over the private chunk
/// writer so `examples/bench_storage_micro.rs` can A/B the open strategy.
#[cfg(feature = "bench")]
@@ -290,11 +324,21 @@ impl BlobStorageBackend for LocalBlobBackend {
.await
.map_err(DomainError::from)?;
// Create the 256 hash-prefix directories (00-ff)
// Create the 256 hash-prefix directories (00-ff), and while
// we're iterating them, reap any `*.replace.*.tmp` files
// that a previous run's `put_blob_from_bytes_replace` may
// have leaked (crashed between write + fsync + rename). No
// existing job GCs these — `dedup_gc` operates on blob
// hashes, `backend_consistency` reports orphans as
// findings but doesn't delete. Reaping at boot is cheap
// (one `read_dir` per shard, ~256 fast enumerations) and
// guarantees a clean slate.
for prefix in &HEX_PREFIXES {
fs::create_dir_all(self.blob_root.join(prefix))
let shard = self.blob_root.join(prefix);
fs::create_dir_all(&shard)
.await
.map_err(DomainError::from)?;
reap_replace_tmpfiles_in(&shard).await;
}
Ok(())
})
@@ -422,6 +466,96 @@ impl BlobStorageBackend for LocalBlobBackend {
})
}
/// **Atomic replace**: write to a same-directory tempfile, fsync,
/// then `rename(2)` over the target. `write_blob_bytes`'s
/// `O_CREAT|O_EXCL` idempotent-skip (the right choice for uploads)
/// silently no-ops when the target already exists — wrong for
/// callers like `backend_rotate` that need the bytes to change.
/// See the trait doc for the full picture.
///
/// Tempfile lives beside the target under the same shard directory
/// so `rename` is a cheap same-filesystem operation (never an
/// EXDEV cross-device copy fallback). The tempfile name embeds
/// the process pid + a monotonic counter so parallel replaces on
/// the same hash from different tasks don't clobber each other.
fn put_blob_from_bytes_replace(
&self,
hash: &str,
data: Bytes,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let size = data.len() as u64;
// Tempfile in the SAME directory as the target → rename is
// cheap same-filesystem, never EXDEV. Counter ensures
// uniqueness under parallel replaces (rare — rotate is
// sequential per-blob today, but future concurrency won't
// corrupt).
static REPLACE_COUNTER: std::sync::atomic::AtomicU64 =
std::sync::atomic::AtomicU64::new(0);
let counter = REPLACE_COUNTER.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let tmp_path = blob_path.with_file_name(format!(
"{}.replace.{}.{}.tmp",
hash,
std::process::id(),
counter
));
// Create + write + fsync the tempfile. `create_new(true)`
// stays here to catch the astronomically-unlikely case of
// two tasks colliding on the same counter value (belt-and-
// braces; the pid+counter naming already prevents it).
{
let mut tmp = fs::File::options()
.write(true)
.create_new(true)
.open(&tmp_path)
.await
.map_err(|e| {
DomainError::internal_error(
"Blob",
format!("Failed to create replace-tmp: {}", e),
)
})?;
if let Err(e) = tmp.write_all(&data).await {
let _ = fs::remove_file(&tmp_path).await;
return Err(DomainError::internal_error(
"Blob",
format!("Failed to write replace-tmp: {}", e),
));
}
if let Err(e) = tmp.sync_all().await {
let _ = fs::remove_file(&tmp_path).await;
return Err(DomainError::internal_error(
"Blob",
format!("Failed to fsync replace-tmp: {}", e),
));
}
}
// Atomic replace. On POSIX `rename(2)` is atomic within a
// filesystem — a concurrent reader sees either the old or
// new bytes, never a truncated view. Older bytes drop out
// as soon as no reader holds an open fd.
if let Err(e) = fs::rename(&tmp_path, &blob_path).await {
let _ = fs::remove_file(&tmp_path).await;
return Err(DomainError::internal_error(
"Blob",
format!("Failed to atomically replace blob: {}", e),
));
}
// fsync the parent directory so the dirent change (i.e. the
// rename result) survives a power loss, same discipline as
// the create path in `put_blob_from_bytes`.
fsync_parent_dir(&blob_path).await;
Ok(size)
})
}
fn sync_blobs(
&self,
hashes: &[String],
src/infrastructure/services/mod.rs
+2 -1
View File
@@ -1,6 +1,8 @@
pub mod audio_metadata_service;
pub mod azure_blob_backend;
pub mod backend_consistency_service;
pub mod backend_migration_service;
pub mod backend_rotate_service;
pub mod blobs_consistency_service;
pub mod cached_blob_backend;
pub mod chunked_upload_service;
@@ -45,7 +47,6 @@ pub mod s3_blob_backend;
pub mod search_index;
pub mod share_unlock_cookie;
pub mod smtp_email_sender;
pub mod storage_migration_service;
pub mod swappable_blob_backend;
pub mod thumbnail_service;
#[cfg(test)]
src/infrastructure/services/s3_blob_backend.rs
+22
View File
@@ -206,6 +206,11 @@ impl BlobStorageBackend for S3BlobBackend {
/// dedup layer already filtered out chunks the database knows about,
/// so the probe was a pure extra round-trip on every NEW chunk of
/// every upload (2 RTTs -> 1, benches/S3-PUT.md).
///
/// Shares the body with `put_blob_from_bytes_replace` below —
/// S3 PUT is durable on return, so "unsynced" and "replace"
/// collapse to the same semantics here (unlike Local, where
/// `_replace` needs tempfile-rename + fsync).
fn put_blob_from_bytes_unsynced(
&self,
hash: &str,
@@ -232,6 +237,23 @@ impl BlobStorageBackend for S3BlobBackend {
})
}
/// Atomic overwrite path used by `backend_rotate` and
/// `backend_migration` when re-writing an already-present blob
/// under a new head key/format. Trait default delegates to
/// `put_blob_from_bytes` which HEAD-probes and silently skips —
/// exactly wrong for the rotate/migrate use case (the whole
/// point is to replace the existing bytes). Override delegates
/// to the same unconditional PUT as `put_blob_from_bytes_unsynced`
/// — S3's PUT is durable on return, no separate sync barrier
/// needed.
fn put_blob_from_bytes_replace(
&self,
hash: &str,
data: Bytes,
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
self.put_blob_from_bytes_unsynced(hash, data)
}
fn get_blob_stream(
&self,
hash: &str,
src/interfaces/api/handlers/admin_handler.rs
+382 -42
View File
@@ -16,10 +16,10 @@ use crate::application::dtos::plugin_dto::{
SetEnabledDto,
};
use crate::application::dtos::settings_dto::{
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, ListUsersQueryDto,
MigrationStateDto, SaveOidcSettingsDto, SaveStorageSettingsDto, SendSmtpTestDto, SmtpInfoDto,
SmtpTestResultDto, StartMigrationDto, TestOidcConnectionDto, TestStorageConnectionDto,
UpdateUserActiveDto, UpdateUserQuotaDto, UpdateUserRoleDto,
AdminCreateUserDto, AdminResetPasswordDto, DashboardStatsDto, DriveKindUsageDto,
ListUsersQueryDto, MigrationStateDto, SaveOidcSettingsDto, SaveStorageSettingsDto,
SendSmtpTestDto, SmtpInfoDto, SmtpTestResultDto, StartMigrationDto, TestOidcConnectionDto,
TestStorageConnectionDto, UpdateUserActiveDto, UpdateUserQuotaDto, UpdateUserRoleDto,
};
use crate::application::dtos::user_dto::{AdminUserSummaryDto, UserDto};
use crate::application::ports::authorization_ports::AuthorizationEngine;
@@ -30,7 +30,7 @@ use crate::application::ports::plugin_ports::{LogQuery, PluginManagementPort, Pl
use crate::common::di::AppState;
use crate::domain::repositories::drive_repository::DriveRepository;
use crate::domain::services::authorization::{Resource, Subject};
use crate::infrastructure::scheduler::JobStoreProvider;
use crate::infrastructure::scheduler::{JobStoreProvider, PausedRunBrief};
use crate::interfaces::api::handlers::dedup_handler::{get_stats, recalculate_stats};
use crate::interfaces::api::handlers::search_handler::clear_search_cache;
use crate::interfaces::errors::AppError;
@@ -75,9 +75,9 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
.route("/settings/storage", put(save_storage_settings))
.route("/settings/storage/test", post(test_storage_connection))
// Storage migration — thin shims over the recoverable-run
// engine (job_name = "storage_migration"). Retained under
// engine (job_name = "backend_migration"). Retained under
// /storage/migration/* until the admin UI is rewired to
// /api/admin/jobs/storage_migration/*; both paths route to
// /api/admin/jobs/backend_migration/*; both paths route to
// the same underlying JobRegistry dispatch. The old /complete
// endpoint is retired — a finished run is a Completed row,
// there's nothing to acknowledge.
@@ -85,6 +85,14 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
.route("/storage/migration/start", post(start_migration))
.route("/storage/migration/pause", post(pause_migration))
.route("/storage/migration/resume", post(resume_migration))
// K3 (storage-key-rotation): per-entry rotate trigger.
// Normalises every blob on the named entry to its head-pair
// format (legacy → v1, plaintext ↔ encrypted, old-key →
// new-key). No readonly mode; safe under normal traffic.
.route(
"/storage/entries/{name}/rotate",
post(trigger_backend_rotate),
)
// NOTE: /storage/migration/verify retired in slice 7 (see the
// comment near where `verify_migration` used to live). Use
// `POST /api/admin/jobs/blobs_consistency/trigger?storage=<name>`.
@@ -169,6 +177,7 @@ pub fn admin_routes() -> Router<Arc<AppState>> {
.route("/jobs", get(list_jobs))
.route("/jobs/{name}/trigger", post(trigger_job))
.route("/jobs/{name}/cancel", post(cancel_job))
.route("/jobs/{name}/pause", post(pause_job))
.route("/jobs/{name}/runs", get(list_job_runs))
.route("/jobs/{name}/runs/{id}", get(get_job_run))
.route(
@@ -375,7 +384,7 @@ async fn test_storage_connection(
/// GET /api/admin/storage/migration — current migration progress.
///
/// Shim over the recoverable-run engine: reads the latest
/// `storage_migration` run from `jobs.recoverable_runs` (via the
/// `backend_migration` run from `jobs.recoverable_runs` (via the
/// `JobStoreProvider`) and projects it into the legacy
/// `MigrationStateDto` shape the admin storage tab expects. When no
/// run has ever been triggered the response is an empty "idle" DTO —
@@ -395,11 +404,11 @@ async fn test_storage_connection(
pub async fn get_migration_status(
State(state): State<Arc<AppState>>,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::storage_migration_service::STORAGE_MIGRATION_JOB_NAME;
use crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME;
let provider = state.core.job_store_provider.clone();
let latest = provider
.list_runs(STORAGE_MIGRATION_JOB_NAME, 1)
.list_runs(BACKEND_MIGRATION_JOB_NAME, 1)
.await
.map_err(AppError::from)?
.into_iter()
@@ -432,7 +441,7 @@ pub async fn get_migration_status(
/// POST /api/admin/storage/migration/start — begin background migration.
///
/// Shim that forwards to `JobRegistry::trigger("storage_migration",
/// Shim that forwards to `JobRegistry::trigger("backend_migration",
/// ...)`. `run_or_resume` (the RecoverableAdapter's inner dispatch)
/// resumes a Paused run or starts a fresh one — one endpoint covers
/// both. Exclusivity is enforced at the DB layer (the partial unique
@@ -492,7 +501,7 @@ pub async fn start_migration(
dto.target_name
)));
}
trigger_storage_migration(state, Some(dto.target_name)).await
trigger_backend_migration(state, Some(dto.target_name)).await
}
/// POST /api/admin/storage/migration/pause — pause a running migration.
@@ -516,18 +525,18 @@ pub async fn start_migration(
pub async fn pause_migration(
State(state): State<Arc<AppState>>,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::services::storage_migration_service::STORAGE_MIGRATION_JOB_NAME;
use crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME;
tracing::info!(
target: "audit",
event = "storage_migration.pause_requested",
"👮🏻‍♂️ Admin requested storage_migration pause"
event = "backend_migration.pause_requested",
"👮🏻‍♂️ Admin requested backend_migration pause"
);
let flipped = state
.core
.job_store_provider
.request_cancel(STORAGE_MIGRATION_JOB_NAME)
.request_cancel(BACKEND_MIGRATION_JOB_NAME)
.await
.map_err(AppError::from)?;
@@ -568,7 +577,7 @@ pub async fn resume_migration(
// it from `params.target_name` stamped on the original Fresh
// open. Refuses gracefully via RunOutcome::Failed if there is
// no Paused row to resume.
trigger_storage_migration(state, None).await
trigger_backend_migration(state, None).await
}
// verify_migration endpoint retired (slice 7 of
@@ -588,18 +597,18 @@ pub async fn resume_migration(
/// desync `current_run_start` from the actually-running task). The
/// admin UI polls `GET /storage/migration` for progress; the trigger
/// itself is fire-and-forget.
async fn trigger_storage_migration(
async fn trigger_backend_migration(
state: Arc<AppState>,
target_name: Option<String>,
) -> Result<axum::response::Response, AppError> {
use crate::infrastructure::scheduler::JobRunArgs;
use crate::infrastructure::services::storage_migration_service::STORAGE_MIGRATION_JOB_NAME;
use crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME;
tracing::info!(
target: "audit",
event = "storage_migration.trigger_requested",
event = "backend_migration.trigger_requested",
target_name = target_name.as_deref().unwrap_or("<resume>"),
"👮🏻‍♂️ Admin triggered storage_migration"
"👮🏻‍♂️ Admin triggered backend_migration"
);
let registry = state.core.job_registry.clone();
@@ -608,7 +617,7 @@ async fn trigger_storage_migration(
..JobRunArgs::default()
};
tokio::spawn(async move {
registry.trigger(STORAGE_MIGRATION_JOB_NAME, &args).await;
registry.trigger(BACKEND_MIGRATION_JOB_NAME, &args).await;
});
Ok((
@@ -621,6 +630,139 @@ async fn trigger_storage_migration(
.into_response())
}
/// POST /api/admin/storage/entries/{name}/rotate — trigger the
/// `backend_rotate` recoverable job on a specific entry.
///
/// Normalises every blob on `<name>` to the entry's head-pair
/// format: legacy → v1, plaintext ↔ encrypted, old-key → new-key.
/// See `docs/plan/storage-key-rotation.md` §"The rotation job".
///
/// Unlike migration, rotation does NOT engage read-only mode —
/// rewrites happen in place under normal traffic. Concurrent user
/// writes coexist safely.
///
/// The handler validates the entry name synchronously (400 on
/// unknown entry); the actual walk detaches into a
/// `tokio::spawn` so the HTTP call returns immediately.
#[utoipa::path(
post,
path = "/api/admin/storage/entries/{name}/rotate",
responses(
(status = 202, description = "Rotation dispatched"),
(status = 400, description = "Unknown entry"),
(status = 401, description = "Unauthorized"),
(status = 403, description = "Admin required")
),
params(
("name" = String, Path, description = "Storage entry name to rotate")
),
security(("bearerAuth" = [])),
tag = "admin"
)]
pub async fn trigger_backend_rotate(
State(state): State<Arc<AppState>>,
axum::extract::Path(name): axum::extract::Path<String>,
) -> Result<impl IntoResponse, AppError> {
use crate::infrastructure::scheduler::JobRunArgs;
use crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME;
use crate::infrastructure::services::backend_rotate_service::BACKEND_ROTATE_JOB_NAME;
// Synchronous entry-existence check — a bad name would fail the
// run anyway, but returning 400 here spares the operator an
// audit-log round-trip.
let entries = &state.core.config.storage_entries;
if entries.iter().all(|e| e.name != name) {
let available = if entries.is_empty() {
"(none)".to_string()
} else {
entries
.iter()
.map(|e| e.name.as_str())
.collect::<Vec<_>>()
.join(", ")
};
return Err(AppError::bad_request(format!(
"unknown storage entry `{name}`. Available: [{available}]"
)));
}
// Refuse on non-active entry. `storage.blobs` describes what's on
// the ACTIVE backend; walking it against a stale target produces a
// `rotation_failed` finding per missing blob (pure noise) and can't
// actually normalise anything the app reads. The right recipe for
// "normalise a different backend" is: migrate to it (blobs land in
// the head-pair's format on arrival — no rotation needed).
let active = state
.core
.active_backend_name
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone();
if name != active {
return Err(AppError::bad_request(format!(
"backend_rotate refuses non-active entry `{name}` — the DB blob registry \
describes the active entry (`{active}`), so walking it against a stale \
target produces spurious `rotation_failed` findings. Activate `{name}` \
first via `Migrate & activate`, then rotate."
)));
}
// Concurrency guard per plan: at most one encryption-touching
// recoverable run at a time across the whole app. Rotation
// rewrites blobs in place; migration copies + swaps; running
// both simultaneously could interleave writes on the same
// hash. Cheap check — `list_runs` limit 1 with the status
// filter is an index scan.
let provider = state.core.job_store_provider.clone();
for job_name in [BACKEND_ROTATE_JOB_NAME, BACKEND_MIGRATION_JOB_NAME] {
let in_flight = provider
.list_runs(job_name, 5)
.await
.map_err(AppError::from)?
.into_iter()
.any(|r| {
matches!(
r.status,
crate::infrastructure::scheduler::RunStatus::Running
| crate::infrastructure::scheduler::RunStatus::Paused
| crate::infrastructure::scheduler::RunStatus::CancelRequested
)
});
if in_flight {
return Err(AppError::bad_request(format!(
"cannot start backend_rotate on `{name}` — `{job_name}` is already Running / \
Paused / CancelRequested. Wait for it to finish (or cancel via \
`POST /api/admin/jobs/{job_name}/cancel`)."
)));
}
}
tracing::info!(
target: "audit",
event = "backend_rotate.trigger_requested",
target_name = %name,
"👮🏻‍♂️ Admin triggered backend_rotate on `{name}`"
);
let registry = state.core.job_registry.clone();
let args = JobRunArgs {
storage: Some(name.clone()),
..JobRunArgs::default()
};
tokio::spawn(async move {
registry.trigger(BACKEND_ROTATE_JOB_NAME, &args).await;
});
Ok((
StatusCode::ACCEPTED,
Json(serde_json::json!({
"message": format!("Rotation dispatched on `{name}` — poll GET /api/admin/jobs/{BACKEND_ROTATE_JOB_NAME} for progress"),
"detached": true,
})),
)
.into_response())
}
/// Idle-state DTO — no run has been triggered yet.
fn idle_migration_dto() -> MigrationStateDto {
MigrationStateDto {
@@ -658,6 +800,10 @@ fn run_to_migration_dto(
RunStatus::CancelRequested => "paused",
RunStatus::Completed => "completed",
RunStatus::Failed => "failed",
// Cancelled is user-abandoned but terminal — same visual as
// failed for the migration status endpoint (both mean "not
// going to finish, look at findings/logs to know why").
RunStatus::Cancelled => "cancelled",
}
.to_string();
@@ -701,9 +847,22 @@ pub async fn generate_encryption_key() -> Result<impl IntoResponse, AppError> {
crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend::generate_key(
);
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
// Fingerprint uses the same colon-hex rendering as the boot log,
// the pair chain in admin/storage, and `oxicloud --fingerprint`.
// Ed can cross-reference it against `.env` after pasting the key
// in — if the fingerprints match, the key made it into the
// config intact.
let fingerprint =
crate::common::config::fingerprint_from_base64_key(&key_b64).unwrap_or_else(|_| {
// Should never happen — we JUST generated a 32-byte key
// and base64-encoded it — but if the fingerprint helper
// rejects, degrade gracefully rather than 500.
"—".to_string()
});
Ok(Json(serde_json::json!({
"key": key_b64,
"fingerprint": fingerprint,
"warning": "Store this key securely. If lost, encrypted data is IRRECOVERABLY LOST."
})))
}
@@ -756,8 +915,6 @@ pub async fn get_dashboard_stats(
COUNT(*)::INT8 as total_users,
COUNT(*) FILTER (WHERE active = true)::INT8 as active_users,
COUNT(*) FILTER (WHERE role::text = 'admin')::INT8 as admin_users,
COALESCE(SUM(storage_quota_bytes)::INT8, 0) as total_quota_bytes,
COALESCE(SUM(storage_used_bytes)::INT8, 0) as total_used_bytes,
COUNT(*) FILTER (WHERE storage_quota_bytes > 0 AND storage_used_bytes > storage_quota_bytes * 0.8)::INT8 as users_over_80,
COUNT(*) FILTER (WHERE storage_quota_bytes > 0 AND storage_used_bytes > storage_quota_bytes)::INT8 as users_over_quota
FROM auth.users
@@ -769,13 +926,80 @@ pub async fn get_dashboard_stats(
.map_err(|e| AppError::internal_error(format!("Database query failed: {}", e)))?;
use sqlx::Row;
let total_quota: i64 = stats_row.get("total_quota_bytes");
let total_used: i64 = stats_row.get("total_used_bytes");
let usage_percent = if total_quota > 0 {
(total_used as f64 / total_quota as f64) * 100.0
} else {
0.0
// Per-drive-kind quota panel:
//
// - **Personal** rolls up via the user envelope
// (`auth.users.storage_quota_bytes`; `= 0` means unlimited),
// because personal drives inherit their cap from the user per
// `docs/plan/drive.md`. The "N unlimited" here counts USERS
// with unlimited envelope, not drives.
// - **Shared** uses `storage.drives.quota_bytes` directly
// (`IS NULL` means unlimited).
//
// Both rows sum `used_bytes` — for personal that's
// `auth.users.storage_used_bytes`, which is itself
// `SUM(drives.used_bytes) WHERE kind='personal'` per the sweep
// in `storage_usage_service.rs`. For shared it's the drive's own
// `used_bytes`. Trashed files are excluded from both — see
// `bug_trash_excluded_from_quota` for the known gap.
let personal_row = sqlx::query(
r#"
SELECT
COALESCE(SUM(storage_used_bytes)::INT8, 0) AS used_bytes,
COALESCE(SUM(storage_quota_bytes) FILTER (WHERE storage_quota_bytes > 0)::INT8, 0) AS capped_quota_bytes,
COUNT(*) FILTER (WHERE storage_quota_bytes = 0)::INT8 AS unlimited_count,
COUNT(*) FILTER (WHERE storage_quota_bytes > 0)::INT8 AS capped_count
FROM auth.users
WHERE is_external = false
"#,
)
.fetch_one(db_pool.as_ref())
.await
.map_err(|e| AppError::internal_error(format!("Personal-drive stats failed: {}", e)))?;
let shared_row = sqlx::query(
r#"
SELECT
COALESCE(SUM(used_bytes)::INT8, 0) AS used_bytes,
COALESCE(SUM(quota_bytes) FILTER (WHERE quota_bytes IS NOT NULL)::INT8, 0) AS capped_quota_bytes,
COUNT(*) FILTER (WHERE quota_bytes IS NULL)::INT8 AS unlimited_count,
COUNT(*) FILTER (WHERE quota_bytes IS NOT NULL)::INT8 AS capped_count
FROM storage.drives
WHERE kind::text = 'shared'
"#,
)
.fetch_one(db_pool.as_ref())
.await
.map_err(|e| AppError::internal_error(format!("Shared-drive stats failed: {}", e)))?;
let build_row = |kind: &str, row: sqlx::postgres::PgRow| DriveKindUsageDto {
kind: kind.to_string(),
used_bytes: row.get("used_bytes"),
// Only surface the cap when there's at least one capped drive
// — else the FE would render "0 / 0 (NaN%)" for a kind that's
// entirely unlimited.
capped_quota_bytes: {
let capped_count: i64 = row.get("capped_count");
if capped_count > 0 {
Some(row.get("capped_quota_bytes"))
} else {
None
}
},
unlimited_count: row.get("unlimited_count"),
capped_count: row.get("capped_count"),
};
let drive_usage = vec![
build_row("personal", personal_row),
build_row("shared", shared_row),
];
// Backend physical stats (post-dedup, post-encryption) —
// rendered in the dashboard's "Backend Storage" card next to
// the user-quota panel. Same source `StorageSettingsDto` uses;
// cheap aggregate over `storage.blobs`.
let dedup_stats = state.core.dedup_service.get_stats().await;
let stats = DashboardStatsDto {
server_version: env!("CARGO_PKG_VERSION").to_string(),
@@ -785,11 +1009,11 @@ pub async fn get_dashboard_stats(
total_users: stats_row.get("total_users"),
active_users: stats_row.get("active_users"),
admin_users: stats_row.get("admin_users"),
total_quota_bytes: total_quota,
total_used_bytes: total_used,
storage_usage_percent: (usage_percent * 100.0).round() / 100.0,
drive_usage,
users_over_80_percent: stats_row.get("users_over_80"),
users_over_quota: stats_row.get("users_over_quota"),
total_bytes_stored: Some(dedup_stats.total_bytes_stored as i64),
dedup_ratio: Some(dedup_stats.dedup_ratio),
registration_enabled: {
if let Some(svc) = state.admin_settings_service.as_ref() {
svc.get_registration_enabled().await
@@ -2120,7 +2344,54 @@ pub async fn delete_drive_admin(
tag = "admin"
)]
pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse {
let summary = state.core.job_registry.snapshot().await;
let mut summary = state.core.job_registry.snapshot().await;
// Enrich with paused-run info for recoverable jobs so the admin
// panel can render "Resume (scanned/total)" on the row instead of
// just "Run". One indexed SELECT hits `jobs.recoverable_runs`
// (`one_active_run_per_job` partial UNIQUE keys the lookup);
// failures fall back to the pre-enrichment shape so the endpoint
// stays useful when the jobs DB is temporarily unreachable.
if let Some(pool) = state.db_pool.as_ref() {
let paused_rows: Vec<(String, uuid::Uuid, Option<i64>, Option<i64>)> = sqlx::query_as(
r#"
SELECT
job_name,
id,
(stats ->> 'scanned_count')::BIGINT AS scanned,
(params ->> 'total_rows')::BIGINT AS total
FROM jobs.recoverable_runs
WHERE status = 'Paused'
"#,
)
.fetch_all(pool.as_ref())
.await
.unwrap_or_default();
let by_name: std::collections::HashMap<String, PausedRunBrief> = paused_rows
.into_iter()
.map(|(name, id, scanned, total)| {
(
name,
PausedRunBrief {
id,
scanned: scanned.unwrap_or(0).max(0) as u64,
total: total.filter(|t| *t > 0).map(|t| t as u64),
},
)
})
.collect();
for job in summary.iter_mut() {
if job.recoverable
&& !job.running
&& let Some(paused) = by_name.get(&job.name)
{
job.paused_run = Some(paused.clone());
}
}
}
(StatusCode::OK, Json(summary)).into_response()
}
@@ -2129,7 +2400,7 @@ pub async fn list_jobs(State(state): State<Arc<AppState>>) -> impl IntoResponse
/// `force=true` requests acceleration semantics from handlers that
/// support it (dedup_gc → grace = 0, grant_cleanup → grace = 0).
/// Silently ignored by handlers that don't (trash_cleanup,
/// storage_reconcile).
/// usage_reconcile).
///
/// `deep=true` opts into slow variants — `consistency_batch` fans it
/// out to sub-jobs; `storage_consistency` (when implemented) will
@@ -2142,7 +2413,7 @@ pub struct TriggerJobQuery {
pub deep: bool,
/// Optional named storage entry to scope the run against — used by
/// tenants that respect `JobRunArgs.storage` (currently
/// `storage_migration` for its target; `blobs_consistency` /
/// `backend_migration` for its target; `blobs_consistency` /
/// `backend_consistency` will pick this up in slice 7 to probe a
/// non-active entry). Ignored by tenants that don't declare a
/// semantic for it. Unknown-name validation is per-tenant — the
@@ -2199,7 +2470,7 @@ pub async fn trigger_job(
storage: query.storage.clone(),
};
// Jobs that can run for hours (storage_migration, future
// Jobs that can run for hours (backend_migration, future
// reextract_*) are detached: `tokio::spawn` the trigger so the
// HTTP request returns immediately. Without this, browser HTTP
// timeouts drop the request future mid-await → the SemaphorePermit
@@ -2255,7 +2526,7 @@ pub async fn trigger_job(
/// there's a second long-running tenant that justifies the plumbing.
/// See the comment in `trigger_job` for why detach matters.
fn is_detached_job(name: &str) -> bool {
matches!(name, "storage_migration")
matches!(name, "backend_migration")
}
/// `POST /api/admin/jobs/{name}/cancel` — cooperative cancel of the
@@ -2293,16 +2564,28 @@ pub async fn cancel_job(
target: "audit",
event = "job.cancel_requested",
job = %name,
"👮🏻‍♂️ Admin requested cancel for job {}",
"👮🏻‍♂️ Admin requested TERMINAL cancel for job {}",
name,
);
match state.core.job_store_provider.request_cancel(&name).await {
// Terminal semantics: stamps `params.cancel_intent = "terminate"`
// when a Running / CancelRequested row is present so the engine
// upgrades the handler's yield to `Cancelled` instead of `Paused`.
// When the current row is `Paused` (no handler running), does a
// direct DB flip Paused → Cancelled. See
// `PgJobStoreProvider::request_terminal_cancel`.
match state
.core
.job_store_provider
.request_terminal_cancel(&name)
.await
{
Ok(Some(run_id)) => (
StatusCode::OK,
Json(serde_json::json!({
"cancelled": true,
"run_id": run_id.to_string(),
"status": "CancelRequested",
"note": "Running row → will land in Cancelled at next batch boundary; \
Paused row → flipped to Cancelled immediately.",
})),
)
.into_response(),
@@ -2310,7 +2593,7 @@ pub async fn cancel_job(
StatusCode::OK,
Json(serde_json::json!({
"cancelled": false,
"reason": "no running run for this job",
"reason": "no non-terminal run for this job",
})),
)
.into_response(),
@@ -2318,6 +2601,63 @@ pub async fn cancel_job(
}
}
/// `POST /api/admin/jobs/{name}/pause` — cooperative PAUSE of the
/// currently-running recoverable run for `{name}`.
///
/// Same DB mechanism as the old cancel (Running → CancelRequested,
/// handler yields to Paused), but no `cancel_intent` stamp so the
/// engine writes `Paused`. Use this to interrupt a long-running
/// job and resume it later; use `/cancel` to abandon it terminally.
///
/// Idempotent: if the row is already Paused, returns 200 with
/// `paused: false, reason: "already_paused"`.
#[utoipa::path(
post,
path = "/api/admin/jobs/{name}/pause",
params(("name" = String, Path, description = "Registered job name")),
responses(
(status = 200, description = "Pause signalled (or no-op if nothing was running)"),
(status = 401, description = "Unauthorized"),
(status = 403, description = "Admin required"),
(status = 500, description = "DB error"),
),
security(("bearerAuth" = [])),
tag = "admin"
)]
pub async fn pause_job(
State(state): State<Arc<AppState>>,
axum::extract::Path(name): axum::extract::Path<String>,
) -> impl IntoResponse {
use crate::infrastructure::scheduler::JobStoreProvider as _;
tracing::info!(
target: "audit",
event = "job.pause_requested",
job = %name,
"👮🏻‍♂️ Admin requested pause for job {}",
name,
);
match state.core.job_store_provider.request_cancel(&name).await {
Ok(Some(run_id)) => (
StatusCode::OK,
Json(serde_json::json!({
"paused": true,
"run_id": run_id.to_string(),
"note": "Handler will yield at the next batch boundary; row will land in Paused.",
})),
)
.into_response(),
Ok(None) => (
StatusCode::OK,
Json(serde_json::json!({
"paused": false,
"reason": "no running run for this job",
})),
)
.into_response(),
Err(e) => AppError::internal_error(format!("pause failed: {e}")).into_response(),
}
}
/// Query parameters for `GET /api/admin/jobs/{name}/runs`.
#[derive(serde::Deserialize)]
pub struct ListRunsQuery {
src/interfaces/middleware/server_status.rs
+77 -42
View File
@@ -9,23 +9,22 @@
//!
//! ## Cost model
//!
//! On the *hot path* (no migration running — the ~100% case in normal
//! operation) this middleware does:
//! On the *hot path* (no migration AND no rotation running — the
//! ~100% case in normal operation) this middleware does:
//! 1. one `AtomicBool::load(Relaxed)` — sub-nanosecond;
//! 2. an early return when `false`.
//! 2. one `RwLock::read` on `rotation_progress` — uncontended;
//! 3. an early return when both are inactive.
//!
//! No allocation, no lock, no formatting. Adds no measurable latency
//! at any user count.
//! No allocation, no formatting on the hot path. The rotation-check
//! `RwLock::read` is cheap because writers only fire on batch
//! checkpoints (~every 100 blobs); worst-case contention is
//! sub-microsecond.
//!
//! On the *cold path* (migration in progress) this middleware does:
//! 1. the atomic load above;
//! 2. one `RwLock::read` (uncontended — writers are the migration
//! handler, one per batch every ~100 blobs);
//! 3. one small `serde_json::to_string` call on a 4-field struct
//! (a few dozen bytes);
//! 4. one header insertion.
//! On the *cold path* (migration OR rotation in progress) the
//! payload builder pulls the progress snapshot(s), formats a small
//! JSON struct (~a few dozen bytes) and inserts the header.
//!
//! Total per-request work in this branch: microseconds.
//! Total per-request work on cold path: microseconds.
use axum::extract::Request;
use axum::extract::State;
@@ -53,11 +52,20 @@ pub const SERVER_STATUS_HEADER: &str = "x-server-status";
struct HeaderPayload {
readonly: bool,
#[serde(skip_serializing_if = "Option::is_none")]
migration: Option<MigrationHeader>,
migration: Option<ProgressHeader>,
/// K3: independent of `readonly` — rotation does NOT engage the
/// app-wide read-only flag, so the frontend needs a distinct
/// signal to know "rotation is running, show the rotation
/// banner instead of migration banner".
#[serde(skip_serializing_if = "Option::is_none")]
rotation: Option<ProgressHeader>,
}
/// Shared progress shape used by both `migration` and `rotation`
/// header fields — same struct name, same JSON field names. Frontend
/// treats them identically at the render layer.
#[derive(serde::Serialize)]
struct MigrationHeader {
struct ProgressHeader {
// `target` is owned here — the RwLock guard is released before
// serialisation, so a borrowed slice wouldn't survive. Names
// are small (`[a-z0-9_-]{1,32}`) so the copy is trivial.
@@ -67,49 +75,76 @@ struct MigrationHeader {
percent: u8,
}
impl ProgressHeader {
fn from_snapshot(p: &crate::common::migration_progress::MigrationProgress) -> Self {
Self {
target: p.target_name.clone(),
migrated: p.migrated_blobs,
total: p.total_blobs,
percent: p.percent,
}
}
}
pub async fn server_status_middleware(
State(state): State<Arc<AppState>>,
request: Request,
next: Next,
) -> Response {
// Hot-path fast return. When no migration is running the flag is
// false and there's nothing to emit — a bare atomic load and out.
let readonly = state.migration_readonly.load(Ordering::Relaxed);
// Rotation snapshot check — cheap uncontended `read`; if `None`
// and readonly is also false, hot-path returns without a header.
let rotation_active = state
.rotation_progress
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.is_some();
let mut response = next.run(request).await;
if !readonly {
if !readonly && !rotation_active {
return response;
}
// Cold path — build the payload from the shared progress
// snapshot. If the snapshot is absent (readonly is true but the
// handler hasn't seeded progress yet, or a restart-during-
// migration scenario) we still emit `readonly: true` so the
// banner shows — the frontend renders a "maintenance in progress"
// message even when specific numbers aren't available.
// Cold path — build the payload from whichever snapshots are
// active. `readonly:true` fires the migration banner even if
// the migration handler hasn't seeded its progress yet
// (restart-mid-migration scenario). `rotation` is populated
// independently.
let payload = {
let guard = state
.migration_progress
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let migration = if readonly {
state
.migration_progress
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.as_ref()
.map(ProgressHeader::from_snapshot)
} else {
None
};
let rotation = if rotation_active {
state
.rotation_progress
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.as_ref()
.map(ProgressHeader::from_snapshot)
} else {
None
};
HeaderPayload {
readonly: true,
migration: guard.as_ref().map(|p| MigrationHeader {
target: p.target_name.clone(),
migrated: p.migrated_blobs,
total: p.total_blobs,
percent: p.percent,
}),
readonly,
migration,
rotation,
}
};
// `serde_json::to_string` on this 4-field struct is a few
// dozen-byte allocation — negligible against the response body.
// A serialize failure here would be a programming bug (all
// fields are trivially serializable), so we degrade to a
// minimal `readonly: true` string rather than skipping the
// header entirely.
// `serde_json::to_string` on this struct is a few dozen-byte
// allocation — negligible against the response body. A
// serialize failure here would be a programming bug, so we
// degrade to a minimal string rather than skipping the header.
let value =
serde_json::to_string(&payload).unwrap_or_else(|_| r#"{"readonly":true}"#.to_string());
serde_json::to_string(&payload).unwrap_or_else(|_| r#"{"readonly":false}"#.to_string());
if let Ok(header_value) = HeaderValue::from_str(&value) {
response
.headers_mut()
src/main.rs
+64
View File
@@ -168,6 +168,50 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
};
select_storage = Some(name);
}
"--fingerprint" => {
// One-shot helper: compute the SSH-style colon-hex
// fingerprint of a base64-encoded AES-256 key and
// print to stdout. Same truncation used by the v1
// header's `<key_fp>` field + the `backend_rotate`
// completion summary — so an admin can:
// 1. Look at the `head_key_fp` reported by the
// last rotate run.
// 2. Run `oxicloud --fingerprint <base64key>` for
// each candidate in `.env`.
// 3. Match — the key that produces the reported
// fingerprint is the current head; any other
// key in `_ENCRYPTION_KEY` no longer decrypts
// any live blob and can be dropped.
//
// Also accepts `-` for stdin so keys never touch the
// shell history:
// echo -n '<base64>' | oxicloud --fingerprint -
let Some(key_b64) = args.next() else {
eprintln!("--fingerprint requires a base64 key argument (or `-` for stdin)");
std::process::exit(2);
};
let key_b64 = if key_b64 == "-" {
use std::io::Read;
let mut buf = String::new();
if let Err(e) = std::io::stdin().read_to_string(&mut buf) {
eprintln!("failed to read key from stdin: {e}");
std::process::exit(2);
}
buf.trim().to_string()
} else {
key_b64
};
match oxicloud::common::config::fingerprint_from_base64_key(&key_b64) {
Ok(fp) => {
println!("{fp}");
return Ok(());
}
Err(e) => {
eprintln!("--fingerprint: {e}");
std::process::exit(2);
}
}
}
"--help" | "-h" => {
print_help();
return Ok(());
@@ -250,6 +294,13 @@ fn print_help() {
println!(" oxicloud --select-storage <name> One-shot repair — set the active");
println!(" storage entry in the DB and exit.");
println!();
println!(" oxicloud --fingerprint <base64key|-> One-shot helper — print the SSH-style");
println!(" fingerprint of a base64 AES-256 key.");
println!(" Same shape used by the v1 blob header");
println!(" + `backend_rotate` completion summary.");
println!(" Read stdin with `-` to keep keys out");
println!(" of shell history.");
println!();
println!(" oxicloud --version Print version + commit and exit.");
println!();
println!(" oxicloud --help Print this help and exit.");
@@ -273,6 +324,19 @@ fn print_help() {
println!(" when that happens). See `docs/plan/storage-multi-entry.md`");
println!(" §Fallback for the full recovery flow.");
println!();
println!(" --fingerprint <base64key | ->");
println!(" Compute the SSH-style colon-hex fingerprint (16-hex, 8-byte");
println!(" truncation of sha256) of a base64-encoded AES-256 key. Matches the");
println!(" `head_key_fp` field the `backend_rotate` job reports on completion,");
println!(" and the raw <key_fp> field embedded in every v1 blob header. Used");
println!(" to identify which key in `OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY`");
println!(" corresponds to the current on-disk head — safe to drop any key");
println!(" whose fingerprint does NOT match the last-successful rotate's");
println!(" `head_key_fp`. Pass `-` to read the key from stdin so it never");
println!(" touches shell history:");
println!();
println!(" echo -n '<base64>' | oxicloud --fingerprint -");
println!();
println!(" --version, -V");
println!(" Print the version, git branch, and commit hash. Exits 0.");
println!();
tests/api/admin_jobs.hurl
+10 -8
View File
@@ -7,7 +7,7 @@
#
# Coverage:
# * Listing returns the four registered tenants
# (trash_cleanup, storage_reconcile, dedup_gc, grant_cleanup).
# (trash_cleanup, usage_reconcile, dedup_gc, grant_cleanup).
# * Scheduled jobs report `interval_ms`; on-demand jobs
# (`dedup_gc`) omit it via `skip_serializing_if=None`.
# * Triggering a job updates its `last_outcome` in the next list.
@@ -52,7 +52,7 @@ bob_token: jsonpath "$.access_token"
# ─────────────────────────────────────────────────────────────
# Step 1 — Admin lists jobs. All four Part 1 tenants must appear.
# Scheduled jobs (trash_cleanup, storage_reconcile, grant_cleanup)
# Scheduled jobs (trash_cleanup, usage_reconcile, grant_cleanup)
# report `interval_ms`; on-demand jobs (`dedup_gc`) omit it via
# serde's `skip_serializing_if = "Option::is_none"`.
#
@@ -69,7 +69,7 @@ HTTP 200
[Asserts]
# Names — the four scheduler tenants.
jsonpath "$[*].name" contains "trash_cleanup"
jsonpath "$[*].name" contains "storage_reconcile"
jsonpath "$[*].name" contains "usage_reconcile"
jsonpath "$[*].name" contains "dedup_gc"
jsonpath "$[*].name" contains "grant_cleanup"
@@ -90,21 +90,23 @@ jsonpath "$..interval_ms" count == 3
# Every entry carries a `running` bool — same aggregate primitive.
# Count matches the registered-tenant count: 4 Part 1 periodics
# (trash_cleanup, storage_reconcile, dedup_gc, grant_cleanup) + 5
# (trash_cleanup, usage_reconcile, dedup_gc, grant_cleanup) + 5
# Part 2 recoverables (drives_consistency, folders_consistency,
# files_consistency, blobs_consistency, backend_consistency —
# wrapped by RecoverableAdapter so they appear here alongside the
# periodics) + 1 coordinator (consistency_batch — a plain
# JobHandler that dispatches every registered `*_consistency`) +
# 1 on-demand admin op (storage_migration — recoverable, no
# periodic tick, triggered by the admin panel's backend cutover).
# 2 on-demand admin ops (backend_migration — the readonly-mode +
# cutover backend swap; backend_rotate — K3, in-place per-blob
# format normalisation, no readonly).
# Bump when a new tenant registers.
jsonpath "$..running" count == 11
jsonpath "$..running" count == 12
jsonpath "$[*].name" contains "drives_consistency"
jsonpath "$[*].name" contains "folders_consistency"
jsonpath "$[*].name" contains "files_consistency"
jsonpath "$[*].name" contains "consistency_batch"
jsonpath "$[*].name" contains "storage_migration"
jsonpath "$[*].name" contains "backend_migration"
jsonpath "$[*].name" contains "backend_rotate"
# ─────────────────────────────────────────────────────────────
tests/api/cross_drive_copy.hurl
+5 -5
View File
@@ -30,7 +30,7 @@
# same `v_dest_drive_id` variable, so (a) passing implies
# file rows used the same value and (b) cross-checks it.
#
# Sweep convergence: `/api/admin/jobs/storage_reconcile/trigger` is the
# Sweep convergence: `/api/admin/jobs/usage_reconcile/trigger` is the
# deterministic synchronisation point — without it the
# fire-and-forget delta hook may not yet have updated the cached
# `used_bytes` when we read it.
@@ -136,7 +136,7 @@ file_id: jsonpath "$.id"
# numbers, the late hook adds its delta on top, and used_bytes ends
# up high by exactly one file's size. Symptom: expected 32, got 64.
# Real fix is await'ing the hook inline server-side.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -171,7 +171,7 @@ HTTP 200
[Captures]
shared_file_id: jsonpath "$.successful[0].id"
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -259,7 +259,7 @@ HTTP 201
# the file's size into the cached counter. 200 ms is well above
# the tokio task latency on any reasonable box; the deterministic
# fix would be intra-transaction hooks, deferred until D7.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -348,7 +348,7 @@ jsonpath "$.name" == "dc-subtree-inner"
# the Step 6 file copy (32) = 96. Anything other than (96, 96)
# would mean the file INSERT in copy_folder_tree used the wrong
# drive_id.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
tests/api/cross_drive_move.hurl
+5 -5
View File
@@ -17,7 +17,7 @@
# a file inside and watching the destination drive's
# `used_bytes` jump by the descendant's size (not 0).
#
# Sweep convergence: `/api/admin/jobs/storage_reconcile/trigger` is the
# Sweep convergence: `/api/admin/jobs/usage_reconcile/trigger` is the
# deterministic synchronisation point — it recomputes every
# drive's cached `used_bytes` from `SUM(file.size) WHERE
# drive_id = d.id`. If the file/folder move didn't update
@@ -125,7 +125,7 @@ file_id: jsonpath "$.id"
# Baseline used_bytes after the upload settles. Trigger-sweep is
# the deterministic sync point — but only after the spawn'd hook
# has had a chance to land (bug_trigger_sweep_vs_spawn_hook_race.md).
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -161,7 +161,7 @@ Content-Type: application/json
HTTP 200
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -245,7 +245,7 @@ nested_file_id: jsonpath "$.id"
# size. Symptom: expected 64, got 96 (one extra hook landed late).
# Real fix is await'ing the hook inline server-side; until then this
# delay deflakes the test.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -278,7 +278,7 @@ HTTP 200
# shared: nested hello-copy.txt now charged here (32)
# Anything other than (32, 32) means the descendant file's
# drive_id wasn't cascaded by the trigger.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
tests/api/drive_quota.hurl
+10 -10
View File
@@ -120,7 +120,7 @@ small_file_id: jsonpath "$.id"
# Ed's 2026-07-17 design call: the sweep is the escape hatch
# for tests / operators that need immediate cache freshness;
# per-write invalidation would nuke the cache on every upload.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -157,7 +157,7 @@ HTTP 201
# `used_bytes` climbs to 64 (32 + 32). Same trigger-sweep pattern
# as the first assertion — the delta is fire-and-forget and the
# listing cache lags until the sweep invalidates it.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -194,7 +194,7 @@ HTTP 507
# consumed by the intervening GET which re-populated the cache
# with the pre-refused-write value. Sweep + re-check for
# determinism.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
@@ -238,7 +238,7 @@ HTTP 201
# Unlimited drive's `used_bytes` climbs to the file's exact size
# (5 MiB = 5_242_880 bytes). Trigger-sweep pattern (see above).
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -266,7 +266,7 @@ jsonpath "$[?(@.id=='{{unlimited_drive_id}}')].used_bytes" == 5242880
# tight drive; the 5 MiB is in the unlimited one).
# b) Permanently delete via empty-trash.
# c) Trigger the reconciliation sweep on demand —
# `POST /api/admin/jobs/storage_reconcile/trigger`.
# `POST /api/admin/jobs/usage_reconcile/trigger`.
# d) `GET /api/drives` now shows the corrected counter.
# ─────────────────────────────────────────────────────────────
DELETE {{base_url}}/api/files/{{small_file_id}}
@@ -284,7 +284,7 @@ HTTP 200
# Sweep is fire-and-forget on a ticker (default 600 s). Run it now
# so the assertion below is deterministic instead of polling.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
@@ -310,7 +310,7 @@ jsonpath "$[?(@.id=='{{unlimited_drive_id}}')].used_bytes" == 5242880
# the route in production configs; here we just confirm a
# non-admin caller is refused even when the feature is on.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{owner_token}}
HTTP 403
@@ -415,7 +415,7 @@ HTTP 200
# operations above never wrote anything. Trigger-sweep so the
# check reads live SQL (see the class doc on the earlier
# sweep + GET pair for the design rationale).
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
@@ -570,7 +570,7 @@ HTTP 201
soft_shrink_file_id: jsonpath "$.id"
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -654,7 +654,7 @@ Authorization: Bearer {{owner_token}}
HTTP 200
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
tests/api/recoverable_jobs.hurl
+6 -5
View File
@@ -137,10 +137,11 @@ jsonpath "$.stats" exists
# ─────────────────────────────────────────────────────────────
# Step 5 — Cancel-on-idle is a no-op. No Running row means no
# Running→CancelRequested flip. Response is 200 with
# `cancelled: false` (NOT a 404 — the job name is
# registered, cancel just found nothing to cancel).
# Step 5 — Cancel-on-idle is a no-op. `cancel` is TERMINAL — it
# acts on any non-terminal row (Running, CancelRequested,
# or Paused). No such row → 200 with `cancelled: false`.
# NOT a 404 (job name is registered, cancel just found
# nothing to abandon).
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/jobs/drives_consistency/cancel
Authorization: Bearer {{admin_token}}
@@ -148,7 +149,7 @@ Authorization: Bearer {{admin_token}}
HTTP 200
[Asserts]
jsonpath "$.cancelled" == false
jsonpath "$.reason" == "no running run for this job"
jsonpath "$.reason" == "no non-terminal run for this job"
# ─────────────────────────────────────────────────────────────
tests/api/storage_cleanup_check.sh
+3 -3
View File
@@ -268,7 +268,7 @@ log "API confirms trash is empty."
# disk state to be quiescent NOW. The two JobRegistry admin triggers
# below (production surface, always on) make this deterministic:
#
# 1. storage_reconcile — reconciles users.storage_used_bytes and
# 1. usage_reconcile — reconciles users.storage_used_bytes and
# drives.used_bytes from SUM(size) — keeps
# the cached counters honest for any quota
# assertions that follow.
@@ -280,8 +280,8 @@ log "API confirms trash is empty."
# row-delete → unlink window the grace
# normally protects.
curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/storage_reconcile/trigger" >/dev/null \
|| fail "storage_reconcile trigger failed"
curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/usage_reconcile/trigger" >/dev/null \
|| fail "usage_reconcile trigger failed"
log "Reconciliation sweep triggered."
GC_RESULT=$(curl -sf -X POST -H "$AUTH" "$base_url/api/admin/jobs/dedup_gc/trigger?force=true")
tests/api/storage_multi_entry.hurl
+7 -1
View File
@@ -87,7 +87,13 @@ Content-Type: application/json
HTTP 200
[Asserts]
jsonpath "$.connected" == true
jsonpath "$.backend_type" == "local"
# Post-K2 (storage-key-rotation): every entry is wrapped in the v1
# blob-format decorator, so `backend_type` reports the WRAPPER's kind
# in `"<wrapper>(<inner>)"` form. `local_main` is unencrypted → wrapper
# is `v1-plaintext`. Match on the inner name via `contains` so the
# assertion survives future wrapper renames.
jsonpath "$.backend_type" contains "local"
jsonpath "$.backend_type" contains "v1-plaintext"
jsonpath "$.roundtrip_passed" == true
jsonpath "$.phase_reached" == "cleanup_ok"
tests/api/user_envelope_quota.hurl
+5 -5
View File
@@ -19,7 +19,7 @@
# 4. Sweep self-heals — after trashing the personal file and
# `trigger-sweep`, `/me.storage_used_bytes` returns to 0.
#
# `POST /api/admin/jobs/storage_reconcile/trigger` is the
# `POST /api/admin/jobs/usage_reconcile/trigger` is the
# deterministic synchronisation point: it runs the drive-side sweep
# then the user-side sweep (both under the periodic scheduler), so
# both cached counters are authoritative ground-truth by the time
@@ -139,7 +139,7 @@ HTTP 201
# acts as the synchronisation point for the user-envelope
# assertion below — the sweep is the authoritative
# ground-truth for both drive- and user-side counters.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
[Options]
delay: 200ms
@@ -159,7 +159,7 @@ jsonpath "$[?(@.id=='{{shared_drive_id}}')].used_bytes" == 32
# If the delta path incorrectly fired the user counter, the sweep
# would still correct it back to 0 (the new SQL excludes shared
# drives) — this also validates the sweep formula.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
@@ -207,7 +207,7 @@ jsonpath "$.storage_used_bytes" == 32
# Confirm the sweep agrees with the delta — both code paths must
# give the same number.
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200
@@ -237,7 +237,7 @@ Authorization: Bearer {{owner_token}}
HTTP 200
POST {{base_url}}/api/admin/jobs/storage_reconcile/trigger
POST {{base_url}}/api/admin/jobs/usage_reconcile/trigger
Authorization: Bearer {{admin_token}}
HTTP 200