Oxicloud/docs/plan/storage-multi-entry.md

Plan — Multi-entry storage config + name-selected active backend

Context

Today OxiCloud has a single storage backend, configured via a flat set of env vars (OXICLOUD_STORAGE_BACKEND, OXICLOUD_S3_*, OXICLOUD_STORAGE_ENCRYPTION_KEY, …). The admin panel has a second, parallel storage-config surface backed by admin_settings.storage.* rows, used by the migration tool to pick a target. Priority is env > DB > defaults, so the DB config effectively acts as a staging area for "what the migration should copy INTO" but never wins at boot.

Two chronic problems fall out:

1. Split-brain config. Admin edits DB via the panel; app boot ignores DB. Migration completes; live backend hasn't moved. Admin has to remember to copy env vars into .env and restart. Two sources of truth for the same setting. Cutover is a manual multi-step flow; users routinely get it wrong.

2. Migration data-loss window on concurrent writes. The copy walks storage.blobs in hash order. A blob whose hash is lex-lower than the current cursor, written to source AFTER migration passed it, is never copied to target. passed=true, findings=0 completion does NOT guarantee target has every blob. Silent.

3. Migration target selection is fragile. DTO passes the whole S3 config at trigger time; secrets sit plaintext in admin_settings. Any future pluggable-storage story compounds this (Azure, GCS, WebDAV-as-source, …).

This plan replaces the split-brain model with a single-source-of-truth architecture:

• .env declares N named storage entries (immutable per-deploy).
• admin_settings.storage.active_backend_name holds ONE row — which named entry the app currently runs on. That's the whole runtime config.
• Migration is the atomic transition from one active entry to another. Server is put in read-only mode for the copy window; on completion, the active pointer flips; a restart cuts over.

Named entries also solve two adjacent problems Ed flagged during design:

• Per-entry encryption keys enable local (raw) → s3 (encrypted K1) moves AND s3 (K1) → s3-new-bucket (K2) key-rotation moves.
• ?storage=<name> on blobs_consistency/backend_consistency lets operators audit any registered entry (target verification, pre-decommission check, etc.), replacing the sample-based verify_migration endpoint with a full-walk audit.

Design decisions

Named entries in .env — explicit allowlist

OXICLOUD_STORAGE_ENTRIES=local_main,s3_prod

OXICLOUD_STORAGE_local_main_BACKEND=local
OXICLOUD_STORAGE_local_main_ROOT_DIR=/data

OXICLOUD_STORAGE_s3_prod_BACKEND=s3
OXICLOUD_STORAGE_s3_prod_S3_BUCKET=my-bucket
OXICLOUD_STORAGE_s3_prod_S3_ENDPOINT_URL=https://s3.example.com
OXICLOUD_STORAGE_s3_prod_S3_REGION=us-east-1
OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY=...
OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY=...
OXICLOUD_STORAGE_s3_prod_S3_FORCE_PATH_STYLE=true
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=<base64-32-bytes>

• Explicit _ENTRIES allowlist — order-independent, admin-authored names. Env-pattern-matching was considered and rejected: too fragile (someone typos OXICLOUD_STORAGE_s3_prud_BUCKET, gets a silently-registered ghost entry). Explicit list forces a declaration.
• Names are admin-chosen strings matching [a-z0-9_-]{1,32}, unique within the list. Parsed at boot; unparseable → fail-fast with the offending name in the error.
• Order does not matter — DB says which is active. Reordering entries in .env never changes runtime behaviour.

Legacy flat-var interaction — three states, one is fail-fast

Multi-entry lives alongside the pre-existing single-backend flat vars (OXICLOUD_STORAGE_BACKEND, OXICLOUD_S3_*, OXICLOUD_AZURE_*, OXICLOUD_STORAGE_ENCRYPTION_KEY, _ENABLED). The parser resolves the interaction as follows:

_ENTRIES	Legacy storage-backend vars present	Behaviour
Unset / empty	Absent	storage_entries = []. Boot uses framework defaults (Local at storage/).
Unset / empty	Present	Synthesize a single entry named default from the legacy vars. Preserves upgrade path — existing deployments keep working without touching .env.
Set (e.g. foo,bar)	Absent	Parse each named entry from _STORAGE_<NAME>_* vars. Fail-fast if any declared entry is missing its required per-name fields.
Set (e.g. foo,bar)	Present	FAIL FAST. Boot aborts with an error listing every legacy var found. Admin must remove them or migrate them into per-entry _STORAGE_<NAME>_* form.

Why fail-fast on the "both set" case: without it, admin state is ambiguous — someone edits OXICLOUD_S3_BUCKET expecting it to matter, but it's silently ignored because _ENTRIES won. The subtle-bug cost is much higher than the one-time cleanup cost. Refusing to boot forces the conversion once, with a clear message naming the exact vars to remove.

Set of "legacy storage-backend vars" counted for the conflict check: OXICLOUD_STORAGE_BACKEND, all seven OXICLOUD_S3_*, all five OXICLOUD_AZURE_*, OXICLOUD_STORAGE_ENCRYPTION_ENABLED, and OXICLOUD_STORAGE_ENCRYPTION_KEY. OXICLOUD_STORAGE_PATH is NOT in this set — it drives multiple non-backend things (chunk dir default, etc.) and remains a valid ambient path; per-entry _ROOT_DIR falls back to it for Local entries when unset.

One DB row: active_backend_name

Single setting in admin_settings:

storage.active_backend_name = "local_main"

• Boot logic:
  1. Parse AppConfig.storage_entries from env.
  2. Read active_backend_name from admin_settings.
  3. If unset (fresh install): use the FIRST name in _ENTRIES. Log which one, don't fail.
  4. Look up the entry by name. Build blob_backend from it.
  5. If the name doesn't exist in current env (deploy drift — someone removed an entry): fail-fast at boot with a clear error naming the missing entry AND listing the available ones. Admin fixes env or overrides the setting via a fallback CLI (see §Fallback).
• Everything the admin panel currently writes about storage (s3.bucket, s3.access_key, …) is removed from DB. save_storage_settings is deleted along with those rows.

Encryption is per-entry

OXICLOUD_STORAGE_<NAME>_ENCRYPTION_KEY (base64 of exactly 32 bytes) on an entry → that entry's backend is wrapped in EncryptedBlobBackend at build time. Absent → raw backend.

• Presence-implies-enabled. No separate _ENCRYPTION_ENABLED toggle — one env var per entry is enough.
• Fail-fast on invalid key: bad base64, wrong decoded length → boot aborts with the entry name in the error message. A bad key is a real deployment error, must be caught at boot, never silently disabled.
• Legacy OXICLOUD_STORAGE_ENCRYPTION_KEY remains honoured for the synthesized default entry when _ENTRIES is empty (upgrade path).

Cross-entry encryption combinations work out of the box because EncryptedBlobBackend is a decorator and copy_blob in the migration handler always spools plaintext to a tmp file:

Source	Target	Migration behaviour
Raw local	Encrypted S3 (K1)	Read plaintext → write encrypts with K1
Encrypted S3 (K1)	Raw local	Read decrypts with K1 → write plaintext
Encrypted S3 (K1)	Encrypted S3 (K2) on different bucket	Read decrypts K1 → write encrypts K2 (rotation via new bucket)
Encrypted S3 (K1)	Encrypted S3 (K2) on SAME bucket	REFUSED — see below

In-place key rotation is refused. Same physical bucket + different encryption key would have the migration overwrite <hash>.blob with K2 ciphertext while the LIVE backend is still K1-configured → readers get K1-decrypt-of-K2-bytes → 500. Silent data-loss window. The is_source_target_identical guard (see §Migration flow) catches this because storage_identity deliberately excludes the encryption key. The refusal message spells the case out and recommends the two-step workaround (rotate via a temp bucket).

Proper in-place rotation (deferred future slice)

The safe implementation is to namespace object keys by encryption generation — <hash>.k2.blob or k2/<hash>.blob (backend-specific convention). Then:

• Both key generations coexist in the same bucket during rotation.
• Reads continue against K1 (LIVE backend) — old keys untouched.
• Target writes go to K2 keys.
• Cutover restarts with K2 as live; K1 objects can be reaped async by a follow-up sweep.
• The is_source_target_identical guard's storage_identity string changes to INCLUDE the encryption generation — same-bucket different-generation then correctly registers as a legitimate migration, no longer refused.

Schema changes required:

• EncryptedBlobBackend gains a generation: u32 field.
• Object keys become <hash>.k<gen>.blob (or backend-specific equivalent — S3 key naming, Azure blob naming).
• Read path tries current-gen first, falls back to prior-gen for the rotation window (bounded by cutover-completion + async-reap duration).

Not built now — real key-rotation demand is rare enough that the two-step workaround via temp bucket is acceptable. File this as a follow-up slice AFTER multi-entry lands; the named-entry infrastructure is a prerequisite (generation-versioned keys only make sense when there's an entry model to hold "which generation is active" as configuration).

Read-only mode reuses the existing AuthZ short-circuit

DrivePolicies.read_only already gates writes at PgAclEngine::check_inner — short-circuits Create|Update|Delete|Share Permission checks on any resource in a read-only drive. We extend that clause with a global check:

// Inside PgAclEngine::check_inner, before per-drive read_only check:
if permission.is_write() && self.migration_readonly.load(Ordering::Relaxed) {
    return AclDecision::Denied("server in migration read-only mode");
}

• The flag is a AtomicBool on AppState, backed by admin_settings.storage.migration_readonly so it survives restart (server crashes mid-migration → boots read-only → admin retriggers → still safe).
• Admin operations bypass as they already do — admin can still exit read-only, cancel migration, restart the server.
• Reads are unaffected. Users can still browse and download during migration.
• Boot-time clearing: if boot detects migration_readonly=true AND no in-flight backend_migration row (no Running/Paused) AND active_backend_name matches the entry the app booted onto → assume successful cutover completed on prior boot, clear the flag. Otherwise leave it set; admin knows they still need to finish something.

Migration flow — atomic + restart-proof

1. Admin picks target entry from a dropdown → clicks "Migrate to s3_prod"

2. Backend:
   - Verify target_name exists in AppConfig.storage_entries
   - Verify target_name != active_backend_name (no-op guard; existing
     `is_source_target_identical` refactored to compare NAMES not identity
     strings — but the identity check still runs as a second-line defence
     against the encryption-in-place case)
   - Write admin_settings.storage.migration_readonly = true
   - Trigger `backend_migration` recoverable job with
     params = { source_name: "local_main", target_name: "s3_prod" }

3. Migration runs — target resolved fresh each batch by NAME lookup, so
   the run's params carries only the name. No secrets in params. If the
   process restarts mid-run, the resume path re-resolves the entry from
   env by the persisted name. Env is the source of truth for credentials.

4. On Completed:
   - Write admin_settings.storage.active_backend_name = "s3_prod"
   - Log "Cutover complete — restart the server to switch to the new backend"
   - LEAVE read-only mode on. The app is still running with local_main as
     the live backend; if we lifted read-only now, writes would go to
     local_main even though the DB pointer says s3_prod. Forces the operator
     restart, which resolves the ambiguity.

5. Admin restarts the server:
   - Boot reads active_backend_name = "s3_prod" → live backend is now S3
   - Boot's read-only-clear rule fires (no in-flight migration + active
     matches booted entry) → migration_readonly cleared
   - Server writable, on the new backend. Cutover complete.

Handling in-progress restart (server dies while migration is running):

• Boot sweep flips Running → Paused on the migration row (existing Part 2 machinery).
• active_backend_name unchanged. Boots on old backend.
• migration_readonly stays true (in-flight migration → boot-time clear rule DOESN'T fire).
• Admin retriggers migration → run_or_resume reads params.target_name → resumes from cursor with the same target.

No secret ever reaches the DB. Params holds only the entry names. Credentials stay in env; migration resolves them at each batch by name.

Concurrent-write safety — read-only for the copy window

The full-quiesce trade-off: users can browse/download during migration but cannot upload, rename, delete, or share. For a multi-hour migration this is noticeable; the alternative (dual-write decorator) is genuinely a week of work (runtime backend swapping, failure-mode reconciliation, MigrationBlobBackend rebuild) and only justified if migration is a routine op. Read-only is the honest v1 answer — pick a low-traffic window, run the migration, restart.

Read-only is engaged at trigger time and cleared at boot after cutover. Nothing more elaborate. Dual-write is filed as a future upgrade if operator demand appears.

?storage=<name> for consistency audits

Once entries are named, ?storage=<name> becomes the generic "probe any registered entry" knob on the two tenants that touch a backend:

• blobs_consistency?storage=<name> — DB → backend probe.
• backend_consistency?storage=<name> — backend → DB probe.

Unspecified → falls through to the active backend (today's behaviour, preserved).

Use cases this unlocks:

• Pre-cutover verification: blobs_consistency?storage=s3_prod after migration completes — full walk (not a sample) proving target has every blob before the .env flip + restart. Retires verify_migration — sample-check becomes redundant when full audit is one click away.
• Pre-decommission verification: ?storage=local_main after cutover — check the old backend still has every blob DB expects before rm -rf the local .blobs/.
• Ad-hoc audit of any registered entry, backup-restore verification, etc.

Plumbing:

• JobRunArgs gains storage: Option<String>.
• TriggerJobQuery on the admin trigger endpoint parses ?storage=<name>.
• BlobsConsistencyCheck and BackendConsistencyCheck constructors take Arc<StorageSettingsService> (or a smaller EntryResolver port); at run start they use args.storage to pick the backend, falling back to the injected active-backend Arc.
• Unknown entry: fail-fast at HTTP layer (400 before the run ever starts) with known: [local_main, s3_prod] in the response. Cheaper than burning a run row.
• Combined with ?deep=true: legit — "full byte-level integrity audit of a named entry, not the live one". Audit log records both.
• Params on the run row records probed_storage: <name> (or "active") for post-hoc diagnosis.

Fallback for the "boot fails on missing entry" case

If admin renames an entry in .env (or removes one that DB still points at), boot fails fast with a clear error. Operator has two ways out:

1. Fix .env — add the missing entry back OR update _ENTRIES to include an alternative that DOES exist, plus flip active_backend_name before restart.
2. CLI repair flag on the oxicloud binary itself:

   oxicloud storage select <name>
   

   Behaviour: parse .env, verify <name> exists in _ENTRIES (fail-fast with the available names listed if not), connect to DB, UPDATE admin_settings.storage.active_backend_name, print confirmation, exit 0. Does NOT continue to boot the server — one-shot repair; admin starts the server normally afterwards.

The bare-flag on the shipped binary is chosen over a separate just recipe or auxiliary bin because:

• Docker-friendly: docker exec oxicloud oxicloud storage select foo — no need to install extra tooling in the container.
• Systemd-friendly: can be run as a ExecStartPre= one-shot before the main service unit.
• No dep on just being installed (dev-machine tool, not typical prod).
• Same binary, same env-parse code path: the repair uses THE SAME .env parser the server does, so "verified present" means the server will succeed on next boot too. No parser drift possible.

The boot-time error message points at this flag explicitly, with the exact command line filled in.

Interaction with existing surfaces

• /admin/storage tab rewritten:
  • Read-only listing of entries (name, backend type, encryption on/off, is active).
  • "Migrate to X" dropdown (choose target entry).
  • Migration progress + verify/cancel (existing).
  • Read-only banner when migration_readonly is on.
  • The Save form, S3-field editors, .env cutover hint — all deleted (no settings edit here anymore).
• test_storage_connection loses the S3-fields DTO; becomes a round-trip probe against a named entry: POST .../test?storage=<name>.
• verify_migration deleted; the "Verify integrity" button on the storage tab is rewired to trigger blobs_consistency?storage=<name> against the migration target (or dropped in favour of the standard admin/jobs Run button — TBD in slice 6).

Slice breakdown

Single PR is too big; splitting into a coherent sequence. Slices 1-2 are foundational; the rest layer on top independently within reason.

#	Slice	Depends on	Rough size
1	Config parser: OXICLOUD_STORAGE_ENTRIES + per-entry vars + _ENCRYPTION_KEY → Vec<NamedStorageEntry> on AppConfig. Legacy synthesis for empty _ENTRIES (default entry from legacy flat vars).	—	1 day
2	Boot: read active_backend_name from admin_settings; look up entry; build blob_backend via a shared build_entry_backend(&NamedStorageEntry) factory (wraps encryption decorator when key present). Fail-fast on missing entry with actionable message.	1	~half day
3	Migration handler rewrite: params carry target_name only. Handler resolves target by name from storage_settings.build_entry_backend(name). is_source_target_identical guard refactored to compare NAMES first; keeps physical-identity check as second-line refusal (with encryption-differs-specific message). Retire the migration DTO S3-field body.	1, 2	~half day
4	Global migration_readonly flag on AppState, backed by admin_settings.storage.migration_readonly. One clause added to PgAclEngine::check_inner. Boot-time clear rule (no in-flight + active matches booted → clear).	2	~half day
5	Cutover state machine: on migration Completed, write active_backend_name = target_name, keep read-only on. Boot on new backend after operator restart.	4	~half day
6	Admin storage tab rewrite: list entries, show active, migrate dropdown, read-only banner. Delete Save form + S3 field editors + .env cutover hint.	1, 3, 4	1 day
7	?storage=<name> on blobs_consistency + backend_consistency. JobRunArgs.storage plumbing, TriggerJobQuery.storage, entry-resolver at run start, params records probed name. Retire verify_migration + its DTO + its route + its handler.	1, 3	1 day
8	oxicloud storage select <name> bare-flag repair command on the main binary. Parses .env, verifies entry exists, UPDATEs DB, exits. Boot-time missing-entry error message points at it. See §Fallback.	2	~quarter day

Total: ~5-6 days end to end. Slices 6 and 7 can proceed in parallel with each other once 1-5 land. Slice 8 is an ops nicety, could ship whenever.

Verification

Per slice, plus these end-to-end scenarios in Hurl:

1. Fresh install, no _ENTRIES: boot uses synthesized default entry from legacy vars, active_backend_name unset. GET /admin/settings/storage returns one entry, active. No cutover UI.
2. Two entries, no active set: boot picks first in _ENTRIES, logs it, proceeds. Admin panel shows both entries with the picked one marked active.
3. Change active without migration (rarely useful but must be safe): POST /admin/settings/storage/active-name (or however the pointer is exposed) updates the row; next restart boots on the new entry. Existing blobs on the new entry NOT verified — operator's problem, but blobs_consistency catches the drift on next run.
4. Migration happy path: two entries, entry A active, migrate to B. Read-only comes on. Copy runs. active_backend_name flips to B on completion. Read-only stays on until restart. After restart, live is B, read-only cleared.
5. Restart mid-migration: kill server after checkpoint N. Boot: row Paused, active still A, migration_readonly still true. Admin retriggers → resumes from checkpoint N against target B (resolved by name from params). Complete → pointer flips → restart → live is B.
6. In-place encryption rotation refused: two entries, same S3 bucket, different encryption keys. Trigger migration → refuses with the specific error message pointing at the encryption case and the two-step workaround.
7. ?storage=<name> on backend_consistency: run against s3_prod before cutover. Full walk, probed_storage in run row. Then cutover, then rerun against local_main — verifies old backend still has everything. (This scenario named blobs_consistency until that tenant became database-only; entry scoping belongs to whichever job opens a backend.)
8. Unknown storage name: POST /admin/jobs/backend_consistency/trigger?storage=nope → 400 with known-names list. No run row created.
9. Missing entry at boot: active_backend_name = "gone" but _ENTRIES doesn't include it → boot aborts with the specific message pointing at oxicloud storage select <name> (with the available names filled in). Re-run the binary with storage select local_main → verifies + updates DB + exits 0. Restart the server → boots cleanly on local_main.
10. Encryption key invalid: OXICLOUD_STORAGE_<N>_ENCRYPTION_KEY=badbase64 → boot aborts with entry name + reason (not valid base64 / wrong length).
11. Legacy vars alongside _ENTRIES: set _ENTRIES=foo AND leave a stale OXICLOUD_S3_BUCKET=... in .env. Boot aborts with the full list of legacy vars detected, tells the admin to remove them (or move them into OXICLOUD_STORAGE_<NAME>_S3_BUCKET form). Removing the legacy var → next boot succeeds.
12. Legacy synthesis path: _ENTRIES unset, OXICLOUD_STORAGE_BACKEND=s3
    • OXICLOUD_S3_BUCKET=... set. Boot synthesizes one entry named default from those vars. storage_entries.len() == 1, name is default, backend is S3, config carries the flat-var values.

Out of scope

• Dual-write decorator / zero-downtime migration: read-only is v1; dual-write is filed under "if operator demand appears". Would rebuild MigrationBlobBackend (deleted in the recoverable-migration PR — retained in git for the same reason).
• In-place encryption key rotation (same-bucket, different key): refused by the identity guard; workaround is two-step via temp bucket. Proper fix (per-generation object naming) is documented inline in the Encryption section above; deferred until real demand appears.
• Runtime backend hot-swap without restart: not attempted. Requires Arc<RwLock<Arc<dyn BlobStorageBackend>>> indirection at every call site plus per-request coordination; huge blast radius. Read-only + restart is the honest answer.
• Per-user or per-drive storage backends: everything in this plan is server-scoped. If per-drive storage becomes a real need, the named-entry registry is the right substrate but blob_backend on AppState becomes a EntryResolver and every call site changes. Not now.
• DB storage config UI: retired. Admin panel is a controller (list + test + migrate + audit), never a persister. If persisted per-entry knobs become a need (retention days per entry, quota per entry, …), those live in DB rows keyed by entry name — a small extension, not a return to the old model.
• Secret encryption in DB: admin_settings.storage.* secret rows are deleted along with save_storage_settings. If the migration params approach ever grows to store secrets (e.g., a future "supply the target creds inline for one-off migrations"), those get encrypted at rest with a KMS-provided key. Not needed for this plan — params holds only names.

Related memory notes

• feedback_no_abbreviated_env_vars — full-word env var names (OXICLOUD_STORAGE_local_main_S3_ENDPOINT_URL, not OXICLOUD_STORAGE_local_main_S3_EP).
• feedback_config_file_overrides_shell — dotenvy behaviour on explicit config path. Matters when _ENTRIES values are loaded from --config vs shell.
• project_admin_middleware_layer — admin ops bypass read-only via existing middleware; the AuthZ short-circuit added in slice 4 doesn't need any new admin carve-out.
• bug_drive_rename_editor_can_do_it — reminder that PgAclEngine::check_inner is where write-permission short-circuits live; the new global read-only clause lands next to the per-drive one.
• docs/plan/job-registry.md Part 2 — recoverable-run engine that backend_migration runs on; params field, resume semantics, boot sweep.