Oxicloud/docs/plan/storage-key-rotation.md

Plan — Storage encryption-key rotation

Context

Today an encrypted storage entry declares a single key:

OXICLOUD_STORAGE_s3_prod_BACKEND=s3
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=<base64 32 bytes>
OXICLOUD_STORAGE_s3_prod_ENCRYPTION_CIPHER=aes-256-gcm   # optional, defaults to aes-256-gcm

Blobs land at <hash>.blob as raw AES-GCM output (nonce | ciphertext | tag) — no discriminating prefix. That key can never change. Any real deployment needs to rotate keys — after a suspected leak, on a periodic policy, or after a staff turnover. The current answer is "create a second entry and run a migration to it," which:

• forces the admin to provision a second bucket / directory,
• churns object storage costs,
• and does not scale to routine rotations.

The proper fix is in-place key rotation inside the same entry. The storage config holds a list of keys, the server writes with the newest one, reads with any of them, and a background job re-encrypts every blob under the newest key. When the job completes, the admin removes the old key from the list.

At the same time we solve two related on-disk problems that today's format leaves open:

1. No self-description. A <hash>.blob today is ambiguous: could be raw plaintext (unencrypted deployment) or raw AES-GCM output (encrypted deployment). Reads guess based on the entry config; a misconfigured key or a wrong-cipher default silently returns garbage.
2. No path from encrypted to plaintext. Once encryption is on for an entry, the only way off is via a second entry and full migration — same ergonomics problem as key rotation.

Both fall out of introducing a self-describing on-blob header (v1). The object-key suffix stays <hash>.blob for both eras — the magic bytes at the top of every v1 blob are the discriminator, so legacy files stay readable byte-identically and new files coexist alongside them at the same suffix. Migration is lazy.

Related memory (this plan supersedes it):

[Encryption key rotation in place — deferred, unsafe today] — the previous "namespace object keys by encryption generation" direction (<hash>.k2.blob) is DROPPED in favour of the pair-list + v1-header approach below. That approach fought content-addressability by generation-namespacing per key; this one keeps a stable content-addressable object key and encodes generation in a compact on-blob header.

Design

The pair-list config

Replace the singular _ENCRYPTION_KEY + _ENCRYPTION_CIPHER with a single comma-separated list of pairs on _ENCRYPTION_KEY alone:

OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<base64 K1>,aes-256-gcm:<base64 K2>

Rules:

• Format — [<cipher>:]<base64 key> per pair, comma-separated. Whitespace around commas / colons tolerated.
• Cipher optional. Only one real cipher (aes-256-gcm) exists today, so <base64 key> on its own is legal and behaves as aes-256-gcm:<key>. When a second cipher lands, the colon-prefixed form is the disambiguator.
• The none cipher. Legal, has no key material (none: — trailing colon with an empty key). Enables migrating BOTH ways: encrypt a previously-plaintext deployment (none:,aes-256-gcm:<K>), or decrypt an encrypted deployment (aes-256-gcm:<K>,none:). Correctness relies on the v1 header (see Encrypted-blob format) discriminating encrypted vs plaintext, NOT on fallback ordering. A none pair by itself in the list is equivalent to omitting _ENCRYPTION_KEY entirely (kept for symmetry).
• Order matters. Last pair wins on writes. For v1-header reads, the key_fp in the header selects the exact pair; the list order is irrelevant at read time. For legacy reads (see Coexistence), the head pair is used because pre-v1 deployments had only one key by construction.
• At least one pair. Empty list → boot aborts.
• Uniqueness. Same key appearing twice = boot aborts (config drift smell). none may appear at most once.
• Fingerprint. For each real-cipher pair, log a truncated SHA-256 of the key material at boot (12 hex chars) so a swap-order accident is loud in the audit stream. none logs as none:—.

Retire _ENCRYPTION_CIPHER

_ENCRYPTION_CIPHER was added recently and has not shipped in a release. Drop it entirely in the same slice — the pair-list makes it redundant, and the on-blob v1 header carries the cipher choice explicitly per blob anyway. No back-compat shim needed.

Encrypted-blob format (v1 header)

Every v1 blob starts with a fixed self-describing header. Layout:

"OXCPT"        5 bytes  — magic marker (0x4F 0x58 0x43 0x50 0x54)
<version>      2 bytes  — big-endian u16; v1 = 0x0001
<key_fp>       8 bytes  — sha256(key material)[..8]; identifies which pair
                          decrypts. For a `none` (plaintext) v1 blob this is
                          eight zero bytes and the fields below are absent.
<nonce>       12 bytes  — random per blob (AES-GCM standard nonce length)
<ciphertext>   N bytes  — encrypted payload; same length as the plaintext
<auth_tag>    16 bytes  — AEAD authentication tag
                          ────
fixed overhead: 43 bytes for encrypted v1; 15 bytes for plaintext v1

• The magic OXCPT unambiguously distinguishes a v1 blob from legacy raw bytes. It's the sole discriminator — no filename convention, no DB column, the first five bytes tell the reader everything.
• <version> is the only field that can change format without touching every blob. v1 is what this plan ships; v2 slots are reserved (see Forward compatibility).
• <key_fp> eliminates guess-work on read. The pair-list lookup becomes O(1) — no fallback tag-check loop. A key-fp with no matching pair means NoKeyForBlob (500 with a clear message), NEVER silent garbage.
• <nonce> and <auth_tag> are the AES-GCM primitive's own outputs. Their length is fixed by v1 = AES-256-GCM. If a future v2 uses a different AEAD with the same shape (say ChaCha20-Poly1305, tag = 16 B, nonce = 12 B) v2 can keep the layout; a truly-different AEAD triggers a new version.
• Plaintext v1 blobs (produced when the entry's head pair is none) use the same magic + version + all-zero key_fp, then the raw plaintext bytes directly (no nonce, no ciphertext framing, no tag). The header alone is enough to say "this is plaintext v1", cleanly distinguished from encrypted v1 and from legacy raw-plaintext.

Why no CRC. The AEAD tag covers ciphertext + nonce integrity. Content addressability (<hash> = BLAKE3(plaintext)) covers whole-blob integrity for either format. A CRC over the header adds nothing — a corrupt version byte fails on decode; a corrupt key_fp fails to find a pair. Both fail loud.

Coexistence with legacy blobs

Legacy and v1 blobs share the <hash>.blob object-key namespace. The magic bytes at position 0 discriminate them safely:

First 5 bytes	Interpretation	Read path
"OXCPT"	v1 blob	verify magic, dispatch on version + key_fp
anything else	legacy	apply entry config: raw plaintext if no encryption pair, single-key AES-GCM decode with the head pair if encryption is set (by construction pre-v1 used one key, and any pair-list produced by upgrading from a pre-v1 config keeps that key at the head)

Collision probability of legacy bytes accidentally matching "OXCPT" at position 0 is 2⁻⁴⁰ per blob. If a match does happen, the read continues into the v1 path, fails at version/key_fp check with UnsupportedBlobVersion or NoKeyForBlob, and returns a hard error. Never silent corruption. For context: on a 10 M-blob deployment, expected number of collision hard-errors across the lifetime of legacy files is ~10⁻⁵ blobs. Effectively never.

Properties this buys us:

• No forced migration. Legacy <hash>.blob files stay readable indefinitely. Existing deployments upgrade to v1 code with zero format-conversion work.
• Lazy conversion on hot paths. Any COW overwrite (WebDAV MOVE, PUT-over, content-hash re-upload) naturally lands as v1 at the same object key.
• Explicit conversion via backend_rotate. The rotate job walks storage.blobs, reads each blob via the magic-byte dispatch, and if the blob is not already v1 with the head-pair key, PUTs it back as v1 with the head pair — in place, same object key.
• No schema changes. Nothing new in storage.blobs; nothing for blobs_consistency, storage_migrate, or any other sibling job to learn.

Legacy-pair guardrail

Once any legacy blob remains on disk, the pre-v1 single key that produced those blobs MUST remain in the pair-list at the head position (or in the list somewhere the legacy read path can find it). Removing it early breaks every legacy read.

Guardrail:

• Boot logs a warn line if any pair-list has more than one entry AND the head pair's fingerprint differs from the second entry's — signals "you added new keys but haven't rotated legacy blobs yet".

• A legacy-blob counter is surfaced in the admin panel per storage entry.

  Retarget (post-split). This plan names blobs_consistency as the host for the magic-byte branch throughout. That is no longer the right tenant: blobs_consistency became database-only and opens no backend, while backend_consistency owns every physical check and already holds the enumeration. Read backend_consistency wherever the sections below say blobs_consistency. Nothing else about the design changes — the magic-byte check still rides along on an existing walk, and still lands in the run's stats bag.

  The counter is maintained by that scan: during its normal walk it branches on the magic-byte check and records the legacy count as a run statistic on jobs.recoverable_runs (existing surface, no schema hit). The admin panel reads the most recent count and displays it. Refresh cadence is whatever consistency scan cadence the deployment has (weekly by default; on demand from the admin panel).

• The "Rotation complete — safe to remove the old key" hint appears in the entry card only when the last backend_rotate run completed with zero findings AND the most recent consistency scan reported zero legacy blobs.

• Nothing enforces removal at code level — the admin is trusted, given a clear signal, and warned.

Read path

Per blob read:

1. Fetch <hash>.blob from the backend.
2. Check first 5 bytes.
3. If "OXCPT" → v1 read path:
   • Read <version>. Not 0x0001 → UnsupportedBlobVersion.
   • Read <key_fp>. If zero → plaintext-v1; return raw bytes (post-header payload). Else find the pair whose sha256(key)[..8] matches; not found → NoKeyForBlob; found → AES-GCM decrypt with <nonce> / <ciphertext> / <auth_tag> and return plaintext.
4. Otherwise → legacy read path:
   • If entry has any real-cipher pair, attempt AES-GCM decrypt with the head pair's key; return plaintext on success, DecryptFailed on tag failure.
   • If entry has only none (or no pair at all), return raw bytes as plaintext.

Never falls through silently. Every failure is a distinct typed error the handler can map to a 500 with an actionable message.

Write path

1. Always writes to <hash>.blob.
2. Head pair is none → write v1 header (magic + version + zero key_fp) + raw plaintext.
3. Head pair is a real cipher → compute AEAD; write v1 header (magic + version + key_fp) + nonce + ciphertext + auth_tag.

v1 code never produces a legacy-format blob. Any leftover legacy blobs on disk pre-date the v1-code deployment.

The rotation job

New RecoverableJobHandler tenant, backend_rotate. Mirrors backend_migration's shape:

• Iterates storage.blobs in hash-lex order. Cursor is the last-processed hash (64 hex chars). Same cursor encoding as backend_migration and blobs_consistency.
• Per blob:
  1. Fetch <hash>.blob and dispatch via the standard read path.
  2. Decide if a rewrite is needed based on what actually decoded:
     • Legacy blob (no v1 magic) → always rewrite (upgrade to v1 with head pair).
     • v1 encrypted, decrypted under a pair-index other than head → rewrite (key rotation).
     • v1 encrypted, already under head — skip.
     • v1 plaintext, head pair is none — skip.
     • v1 plaintext, head pair is a real cipher — rewrite (encrypt-in-place upgrade).
     • v1 encrypted, head pair is none — rewrite (decrypt-in-place downgrade).
  3. Write via the standard v1 write path — same object key, atomic replace.
  4. Checkpoint. On per-blob failure, record a rotation_failed finding with severity data_loss (bytes may not have crossed), continue.
• Concurrency-safe by construction.
  • A concurrent upload during rotation writes v1 with the head key. The rotate walk will short-circuit on that hash if it reaches it later.
  • A COW overwrite is the same story.
  • The v1 write is atomic at object-storage level (S3 replace, Local rename-into-place). A concurrent reader sees either state.
  • No readonly mode. This is a critical improvement over backend_migration: rotation is per-blob idempotent, so we don't need to freeze writes.
• Restart-survivable — same boot-time sweep as every other recoverable handler.
• ?deep=true unused — rotation has no "slow variant" mode. Parameter accepted for uniformity, ignored.

Admin UX

In the admin storage-panel entry card, add a Rotate encryption key action. Preconditions:

• Entry has encryption enabled with ≥ 2 pairs OR entry has legacy blobs outstanding OR the pair-list has otherwise changed since the last rotation.
• Otherwise the button is disabled with the tooltip "Add a second pair in OXICLOUD_STORAGE_<name>_ENCRYPTION_KEY first, or wait for legacy blobs to accumulate — nothing to rotate right now."

Clicking the button dispatches the backend_rotate job for that entry. The job's progress rides on the same X-Server-Status header infrastructure the maintenance banner uses — but this time WITHOUT engaging read-only mode. Banner variant reads "Rotating encryption key on <entry> — X% (Y / Z blobs). All operations continue normally." and disappears on completion.

The entry card shows a legacy-blob counter sourced from the most recent blobs_consistency run: "N legacy-format blobs remain (upgrade included in next rotation)". Refresh-on-demand button next to it triggers a targeted blobs_consistency scan (already available via the admin surface). The "Rotation complete — safe to remove the old key" hint appears only when N = 0 and the last backend_rotate run completed with zero findings.

Removing the old pair

Not automated. The admin edits .env, restarts. Rationale:

• We do not want the running server to silently mutate its own .env.
• A retained old pair is a benign cost (nothing hits it on read since key_fp lookup is O(1)).
• Explicit human step matches the "add a new pair → restart" symmetry.

Deployment flow

First-time upgrade to v1

Zero admin work required. On upgrade:

• v1 code deploys.
• New blobs are written in v1 format at the existing <hash>.blob object key.
• Existing legacy blobs stay readable via the magic-byte dispatch — the legacy read path is preserved verbatim.
• Admin can optionally trigger a backend_rotate run to consolidate every legacy blob into v1 format. Not required — legacy blobs migrate opportunistically via COW overwrites and stay readable indefinitely otherwise.

Rotating an encryption key

The user-facing recipe (goes verbatim into docs/guide/backend-storage.md):

1. Generate a new key:
   openssl rand -base64 32

2. Append it to the entry's key list. Order matters — the NEW key goes LAST:
   OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<OLD>,aes-256-gcm:<NEW>

3. Restart OxiCloud. New uploads are now encrypted with the new key; existing
   blobs still decrypt with the old one.

4. In the admin panel, click "Rotate encryption key" on the entry. This
   dispatches a background job that re-encrypts every existing blob under
   the new key AND upgrades any remaining legacy-format blobs to v1. All
   operations keep working during rotation.

5. Wait for the job to complete (progress shows in the top banner and in
   the Jobs admin page).

6. Remove the OLD key from the list:
   OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:<NEW>

7. Restart OxiCloud. Rotation is complete.

Encrypting a previously-plaintext deployment

1. Generate a new key (as above).
2. Add it AFTER `none`:
   OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=none:,aes-256-gcm:<K>
3. Restart. New uploads are encrypted; existing plaintext blobs stay readable.
4. Run `backend_rotate` to encrypt existing blobs in place.
5. Remove `none:` from the list; restart.

Decrypting an encrypted deployment

1. Add `none:` AFTER the current encryption key:
   OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=aes-256-gcm:<K>,none:
2. Restart. New uploads are plaintext; existing encrypted blobs stay readable.
3. Run `backend_rotate` to decrypt existing blobs in place.
4. Remove the key pair, keep `none:` only (or drop `_ENCRYPTION_KEY` entirely);
   restart.

Testing strategy

Most of this ships as Rust tests, not Hurl. The rotation surface is byte-level, cryptographic, and stateful across restart boundaries — none of which Hurl can meaningfully assert. Push almost everything down into #[cfg(test)] blocks alongside the code, following the codebase's existing inline-tests convention (see AGENTS.md — "tests are primarily #[cfg(test)] modules within source files").

What lives in Rust (unit + integration):

• Pair-list parser. Unit tests in config.rs on _ENCRYPTION_KEY parsing: 1-pair, 2-pair, cipher-optional shape, none: alone, none:,aes:K, aes:K,none:, whitespace tolerance, duplicate rejection, multiple-none rejection, empty-list rejection, retired-_ENCRYPTION_CIPHER rejection, base64 length validation. Table-driven; fast.
• v1 header round-trip. Unit tests in encrypted_blob_backend.rs::tests: encrypt payload → verify header bytes (magic, version, key_fp) → decrypt → assert plaintext identity. One test per format flavour (encrypted, plaintext-with-none-head). Verify <hash> = BLAKE3(plaintext) with no offset for plaintext-v1 (the "recovery tool" invariant).
• Legacy read-path preservation. Fixture: a hand-crafted raw AES-GCM ciphertext (nonce | ct | tag) produced by the pre-v1 code path. Assert that magic-byte dispatch falls through to the legacy branch and decrypts correctly. Fixture stays in-tree as tests/fixtures/legacy_blob.bin so regressions on the legacy path can never sneak through.
• OXCPT collision on random data. Property test: generate random N-byte payloads, feed through legacy read path. Assert the ~2⁻⁴⁰ magic-collision case fails with UnsupportedBlobVersion or NoKeyForBlob — hard error, never silent misread. Keeps the "collisions can't silently corrupt" claim in the plan honest.
• Rotation decision tree. Unit tests on the per-blob decide() helper of backend_rotate_service.rs. All six cases from The rotation job section as separate tests with clear names (legacy_upgrades_to_v1, v1_encrypted_under_head_skips, v1_encrypted_under_older_pair_rewrites, v1_plaintext_with_none_head_skips, v1_plaintext_encrypts_when_head_is_cipher, v1_encrypted_decrypts_when_head_is_none).
• Recoverable-job round-trip. Integration test in backend_rotate_service::tests using the existing recoverable-run harness: seed N blobs (mix of legacy + v1-under-old-key), trigger rotation, assert every blob ends v1-with-head, format_generation (if we add it later) or the consistency-scan count reports zero legacy remaining, findings=0.
• Crash recovery. Same harness: interrupt mid-run at cursor position K, restart, assert resume from K and eventual completion with correct final state. Same discipline backend_migration already uses.
• Concurrency safety. Test that a put_blob call during a rotation targeting the same hash produces exactly one v1 blob at end-state (either the rotate's or the concurrent write's — both are head-format so the final state is indistinguishable). Verifies the "per-blob idempotent" claim.
• Config-restart semantics. Table-driven test on the pair-list state machine: start with pair-list [K1], add K2 → [K1,K2], run rotation, drop K1 → [K2], assert every blob still decrypts. Exercises the "safe to remove old key" transition end-to-end without going through the admin UI.

What Hurl covers (thin — the API surface, not the mechanics):

• Trigger endpoint auth: POST /api/admin/storage/entries/{name}/rotate is admin-only (403 for non-admins, 401 for unauthenticated).
• Trigger endpoint preconditions: 400 or similar when pairs < 2 AND no legacy blobs exist AND pair-list hasn't changed.
• Trigger endpoint concurrency: second POST while a run is Active returns 409.
• Progress header: X-Server-Status includes a rotation payload during a running job and drops it on completion.

Hurl does NOT try to:

• Inspect on-disk bytes.
• Restart the server with a new pair-list.
• Seed a legacy blob directly.
• Verify decryption after old-key removal.

Those all belong in Rust tests where we can hold the pool, the backend, and the config in the same test's memory.

Test data. Legacy-blob fixtures are byte-frozen (tests/fixtures/*.bin) and committed. Don't regenerate them from live code — the whole point is that they were produced by pre-v1 code and won't ever again be. Regeneration scripts (kept outside tests/) are OK for one-time refreshes if the format changes.

Slices

Same discipline as storage-multi-entry.md. Each slice is a mergeable increment; no slice depends on a later one.

Slice K1 — Pair-list config parsing

Scope. Config layer only. No behaviour change on the write / read path yet — the parser produces Vec<KeyPair> and the existing code keeps calling pairs.last().

• NamedStorageEntry.encryption becomes Option<Vec<KeyPair>> where KeyPair = { cipher: CipherKind, key_material: Option<[u8; 32]> }. CipherKind = { AesGcm256, None }; None carries no key.
• Parser: split on ,, per-pair split on : (1 or 2 parts), decode base64 when a real cipher, reject empty list, reject duplicates, reject multiple none.
• Retire OXICLOUD_STORAGE_<N>_ENCRYPTION_CIPHER — parser errors on it with guidance to move the cipher into the pair.
• Boot logs each pair's fingerprint (sha256(key)[..12] for real ciphers, — for none) and marks the head. Warn line if head fp differs from any non-head fp (rotation window signal).
• Docs: example.env, docs/config/env.md, docs/guide/backend-storage.md updated with the pair syntax and the three recipes (rotate / encrypt / decrypt).

Exit criteria. Boot with a 1-pair config behaves identically to today. Boot with a 2-pair config succeeds and logs both fingerprints. Boot with _ENCRYPTION_CIPHER alongside _ENCRYPTION_KEY fails with a migration hint.

Slice K2 — v1 read/write paths in EncryptedBlobBackend

Scope. The encrypted-backend decorator learns the v1 header format. Reads dispatch on magic bytes; writes always emit v1 headers. Legacy read path preserved verbatim. blobs_consistency gains a magic-byte branch to track the legacy-blob count.

• EncryptedBlobBackend refactored around BlobFormat::V1 writer + reader. Legacy path preserved but read-only from new code.
• Read dispatches on magic; write always emits v1 header at <hash>.blob.
• blobs_consistency: during its normal walk, per-blob magic-byte check; legacy count recorded on the run's stats JSON. No schema hit — reuses the existing stats bag on jobs.recoverable_runs.
• storage_migrate: no changes needed. It copies raw bytes between backends; format is preserved on the target automatically because the object bytes are opaque to it.

Exit criteria. A brand-new deployment writes only v1 blobs. An upgraded deployment reads existing legacy blobs and writes new v1 blobs at the same object-key. blobs_consistency reports a legacy-blob count in its run stats.

Slice K3 — The backend_rotate recoverable job

Scope. New handler, admin-triggered, iterates blobs, per-blob decision tree (legacy → v1 upgrade, v1 with old key → v1 with head key, plaintext ↔ encrypted where applicable), records findings.

• New file src/infrastructure/services/backend_rotate_service.rs.
• Registered in JobRegistry as backend_rotate. Runs on the same recoverable-runs engine (crash recovery, cursor persistence, pause/resume).
• Trigger endpoint: POST /api/admin/storage/entries/{name}/rotate. Requires admin. Refuses if no work would happen (all blobs already at head format + head key). Refuses if a backend_rotate or backend_migration run is already Active for any entry.
• Per-blob decision tree per The rotation job section above. In-place atomic replace at the same <hash>.blob object key.
• No readonly mode engaged. X-Server-Status header payload gains a rotation field alongside migration.

Exit criteria. On a filled sandbox: (1) legacy-only entry rotates to v1; (2) encrypted entry with 2 pairs rotates so head-pair-only remains decryptable; (3) plaintext entry with none,cipher head rotates to encrypted-v1; (4) encrypted entry with cipher,none head rotates to plaintext-v1. Each round-trip verified by dropping the retired pair from config and successfully reading every blob.

Slice K4 — Admin panel action + banner + docs

Scope. UI wiring, banner variant, guide docs, legacy-blob counter.

• Entry card: Rotate encryption key button + tooltip states as above.
• Legacy-blob counter chip on each entry card. Sources the count from the most recent blobs_consistency run's stats (already surfaced in the Jobs admin page). Refresh button triggers a fresh scan.
• ReadOnlyBanner.svelte gains a third variant variant="rotating" — same shape, milder tone (info not warning), copy makes it clear that operations continue.
• Server-status header payload gains rotation?: {entry, migrated, total, percent} alongside the existing migration?: field. readonly stays false during rotation.
• docs/guide/backend-storage.md: new Rotating an encryption key, Encrypting a plaintext deployment, and Decrypting an encrypted deployment sections with the recipes above.
• docs/config/env.md: pair syntax section under _ENCRYPTION_KEY.
• Admin panel "Rotation complete — safe to remove the old key" hint gated on findings=0 AND legacy count=0.

Exit criteria. Round-trip on a real deployment (Ed's OVH S3): add pair → restart → rotate → verify blobs (including legacy) → remove old pair → restart → blobs still readable.

Forward compatibility (v2 and beyond)

The <version> field is the single load-bearing knob for future format changes. Reserved slots:

• 0x0001 — this plan. AES-256-GCM, server-side keys, individual blob integrity via AEAD tag + content-hash.
• 0x0002…0x00FF — reserved for server-side format bumps: new AEAD, chain-authenticated CDC chunks (either manifest HMAC — no header changes, chunk_manifests gains a server-computed HMAC over the ordered chunk-hash list — or Merkle root in header — v2 grows to carry manifest_root + chunk_index, letting any streamed chunk be verified in isolation). Both defend against manifest reorder / truncation / injection attacks that content-addressability alone doesn't cover.
• 0x0100…0x01FF — reserved for client-side encryption (E2E) variants. <key_fp> becomes the client-key fingerprint; the server can no longer decrypt and passes ciphertext through the streaming path unchanged.

v1 and v2 coexist in the same storage indefinitely — the magic-byte read dispatch handles arbitrary versions at position 5-6. Migration between generations reuses backend_rotate's pattern: rewrite each blob with the new-generation writer, in-place at the same object key.

Non-goals

• Asymmetric / KMS-backed keys. Out of scope. Symmetric AES-256-GCM only. A KMS-backed variant is a separate slice built on top of this one.
• Chunk-chain integrity / manifest tampering defense. Recognised as a real gap (chunk reorder / truncate / inject via DB write). Deferred to the v2 header slice — see Forward compatibility.
• Client-side encryption (E2E). Same — v2 slot reserved, separate slice.
• Automatic old-key removal. Explicit human step, see above.
• Rotation of a key on the "in-transit" side (client → server TLS). Handled by the reverse proxy; unrelated.
• Per-blob key derivation (KDF from a master key + blob hash). A legitimate hardening path but orthogonal to the rotation story; folds in as a future slice under this same pair-list config.
• Rollback safety across format generations. If v1 code is rolled back to pre-v1 after new v1 blobs have been written, old code will 500 on those blobs (AES-GCM tag fails on OXCPT-prefixed bytes). Not addressed by this plan; the safety net is restore-from-backup. Deploying an incompatibility-breaking format change is a pre-planned event, not a hot rollback scenario.

Open questions

• Should we throttle the rotate job? Same question backend_migration had. Answer: not in v1. If throughput bites, add a _ROTATE_MAX_MB_PER_SEC on the entry later.
• Should rotation be idempotent under repeat trigger? Yes. Running it a second time with the same config is a no-op walk (every blob is already at head format + head key).
• Should we ship a format_generation column on storage.blobs as a query-optimizer? No in v1. Consistency-scan stats cover the "how many legacy blobs remain" question at admin cadence, and per-read magic-byte dispatch is a 5-byte compare. If a real hot-path or admin-panel latency need emerges, the column is a small back-fill migration to add later.
• Should the legacy-blob counter block key removal at code level? No — keeping the trust model consistent with "admin edits .env, we don't fight them" is the current default. The counter + hint is enough. If real incidents happen we can escalate to a hard guard later.