From e16468977170c0168c79364663bf1d813e892a35 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sat, 1 Aug 2026 21:59:48 +0200 Subject: [PATCH] feat(storage key rot): remove dead born OXICLOUD_STORAGE__ENCRYPTION_CIPHER + alway ovewrite on storage migration (got issue when migrating with blob already existing and a key change) --- docs/config/env.md | 8 +- docs/guide/backend-storage.md | 65 +++- example.env | 26 +- .../services/storage_settings_service.rs | 2 +- src/common/config.rs | 355 ++++++++++-------- src/common/di.rs | 2 +- src/infrastructure/services/entry_backend.rs | 34 +- .../services/storage_migration_service.rs | 61 +-- 8 files changed, 332 insertions(+), 221 deletions(-) diff --git a/docs/config/env.md b/docs/config/env.md index c39651e7..05febfdb 100644 --- a/docs/config/env.md +++ b/docs/config/env.md @@ -103,8 +103,7 @@ Each declared name `` then reads its own set of per-entry variables: | `OXICLOUD_STORAGE__AZURE_CONTAINER` | — | Azure-only: blob container name (required when backend=azure) | | `OXICLOUD_STORAGE__AZURE_SAS_TOKEN` | — | Azure-only: SAS token (alternative to account key) | | `OXICLOUD_STORAGE__AZURE_ENDPOINT_URL` | — | Azure-only: custom endpoint (Azurite, private deployments) | -| `OXICLOUD_STORAGE__ENCRYPTION_KEY` | — | Base64-encoded 32-byte AES-256 key. **Presence implies encryption is enabled** on this entry — no separate enable flag. Bad base64 / wrong length aborts boot. | -| `OXICLOUD_STORAGE__ENCRYPTION_CIPHER` | `aes-256-gcm` when `_ENCRYPTION_KEY` is set | Cipher choice for this entry. Only `aes-256-gcm` is accepted today (future-proofing knob — the enum is ready for a second cipher, the implementation still hardcodes AES-256-GCM). Setting the cipher without a key aborts boot. | +| `OXICLOUD_STORAGE__ENCRYPTION_KEY` | — | Comma-separated list of `:` pairs (or bare ``, which defaults to `aes-256-gcm`). **Presence implies encryption is enabled** on this entry — no separate enable flag. The LAST pair wins on writes; every pair is a candidate for reads. Supported ciphers: `aes-256-gcm` and `none` (empty-key sentinel used at pair-list head/tail for encrypt/decrypt-in-place rotations). Bad base64, wrong length, duplicate keys, or multiple `none` pairs abort boot. See [Storage key rotation](../plan/storage-key-rotation.md) for rotation recipes. | **Fail-fast rules** (boot aborts with actionable message): @@ -125,7 +124,10 @@ OXICLOUD_STORAGE_s3_prod_S3_BUCKET=my-oxicloud-bucket OXICLOUD_STORAGE_s3_prod_S3_REGION=us-east-1 OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY=… OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY=… -OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=… # openssl rand -base64 32 +OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:… # openssl rand -base64 32 + +# Rotation window (two pairs, last wins on writes): +# OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:,aes-256-gcm: ``` ## Storage Backend (DEPRECATED — legacy single-backend) diff --git a/docs/guide/backend-storage.md b/docs/guide/backend-storage.md index 6735711a..19d04bdd 100644 --- a/docs/guide/backend-storage.md +++ b/docs/guide/backend-storage.md @@ -40,15 +40,78 @@ A few things to know: Any backend can be encrypted at rest by adding an encryption key to it: ``` -OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=… # base64 of 32 random bytes +OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:… # base64 of 32 random bytes ``` A key can be generated from **Settings → Storage → Generate key** in the admin panel. Set the same key on a new backend during migration and OxiCloud re-encrypts the data as it copies. +The bare shorthand `OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=` (no `aes-256-gcm:` prefix) also works and is treated as AES-256-GCM. Use the explicit form once you have more than one key in the list — see [Rotating an encryption key](#rotating-an-encryption-key) below. + ::: warning If you lose the encryption key, the data encrypted with it is unrecoverable. Store the key somewhere as safe as you'd store a database backup. ::: +### Rotating an encryption key + +When it's time to rotate a key — after a suspected leak, on a periodic policy, or after a staff turnover — you don't need to provision a second bucket. Add the new key **alongside** the old one; the server writes with the new key while reads keep working through the old one; then a background job re-encrypts existing data under the new key; then you drop the old key. + +Step by step: + +1. Generate a new key: + + ``` + openssl rand -base64 32 + ``` + +2. Append it to the entry's key list. **Order matters — the NEW key goes LAST.** + + ``` + OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:,aes-256-gcm: + ``` + +3. Restart the server. New uploads are now encrypted with the new key; existing files still open normally because the old key is still in the list. + +4. On the **Storage** tab, click **Rotate encryption key** on the entry. This dispatches a background job that re-encrypts every existing file under the new key. All operations keep working during rotation — uploads, browsing, downloads, sharing. + +5. Wait for the job to complete. Progress shows in a top banner and on the **Jobs** tab. + +6. Once the entry card says "*Rotation complete — safe to remove the old key*", remove the OLD key from the list: + + ``` + OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm: + ``` + +7. Restart the server. Rotation is done. + +::: warning +Do NOT remove the old key before the rotation job reports "safe to remove". Any file still encrypted under the old key would become unreadable. +::: + +### Encrypting a backend that started unencrypted + +Same shape as key rotation, using the `none:` sentinel to represent "the current writes are plaintext": + +1. Generate a new key. +2. Add it AFTER `none:`: + + ``` + OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=none:,aes-256-gcm: + ``` + +3. Restart. New uploads are encrypted; existing plaintext files stay readable. +4. Trigger **Rotate encryption key** on the entry. +5. Once the entry card says the rotation is done, remove `none:` from the list; restart. + +### Decrypting an encrypted backend + +The symmetric flow — add `none:` AFTER the current key, restart, rotate, drop the old key: + +``` +OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=aes-256-gcm:,none: +``` + +After the rotation job completes and the entry card says done, remove the AES pair (or the whole variable) and restart. All files on that entry are now plaintext. + ## Checking a backend Once a backend is declared, it appears on the admin **Storage** tab as a card: diff --git a/example.env b/example.env index f84b531a..45b9b06e 100644 --- a/example.env +++ b/example.env @@ -371,9 +371,16 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud # `OXICLOUD_STORAGE__BACKEND` (local | s3 | azure) plus the # backend-specific fields below. A missing required field aborts # boot with the exact var name in the error message. -# * Presence of `OXICLOUD_STORAGE__ENCRYPTION_KEY` implies AES-256 -# encryption is enabled on that entry (no separate enable flag). -# Bad base64 / wrong length aborts boot with the entry name. +# * `OXICLOUD_STORAGE__ENCRYPTION_KEY` is a comma-separated LIST +# of `:` pairs. Presence implies encryption is +# enabled on that entry (no separate enable flag). The LAST pair +# wins on writes; every pair is a candidate for reads. Supported +# ciphers: `aes-256-gcm` (default when no cipher prefix is given) +# and `none` (empty-key sentinel used at pair-list head for a +# decrypt-in-place rotation, or at tail for an encrypt-in-place +# rotation). Bad base64 / wrong length / duplicate keys / +# multiple `none` pairs abort boot with the entry name. See +# `docs/plan/storage-key-rotation.md` for rotation recipes. # * SETTING `_ENTRIES` alongside the legacy flat vars below (e.g. # `OXICLOUD_STORAGE_BACKEND` + `OXICLOUD_S3_BUCKET`) is a FAIL-FAST # boot error — pick one mode. Migrate any leftover flat vars into @@ -393,11 +400,14 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud #OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY= #OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY= #OXICLOUD_STORAGE_s3_prod_S3_FORCE_PATH_STYLE=false -#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY= # generate: openssl rand -base64 32 -# Cipher declaration — future-proofing. Today only `aes-256-gcm` is -# accepted (and it's the default when `_ENCRYPTION_KEY` is set), so -# this line can be omitted. Explicit here as documentation. -#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_CIPHER=aes-256-gcm +#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm: # generate: openssl rand -base64 32 +# The bare shorthand `` (no `aes-256-gcm:` prefix) +# also works; the explicit form makes the cipher visible and is +# required once you have more than one pair in the list. +# +# Two-pair rotation example (paste both pairs in .env, restart, +# then trigger the format-upgrade job, then drop the OLD pair): +#OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:,aes-256-gcm: # # Repair flag: if you rename an entry in .env while the DB still points # at the old name, boot aborts with an actionable error pointing at: diff --git a/src/application/services/storage_settings_service.rs b/src/application/services/storage_settings_service.rs index a96af68e..672df02b 100644 --- a/src/application/services/storage_settings_service.rs +++ b/src/application/services/storage_settings_service.rs @@ -273,7 +273,7 @@ impl StorageSettingsService { StorageBackendType::Azure => "azure".to_string(), }, is_active: e.name == active_entry_name, - encryption_enabled: e.encryption_key_base64.is_some(), + encryption_enabled: e.is_encrypted(), location_hint: entry_location_hint(e), }) .collect(); diff --git a/src/common/config.rs b/src/common/config.rs index ee53b08b..e68ae87f 100644 --- a/src/common/config.rs +++ b/src/common/config.rs @@ -383,80 +383,38 @@ impl Default for RetryConfig { } } -/// AES-GCM / AEAD cipher choice for a `NamedStorageEntry`. -/// -/// Today the only shipping variant is `Aes256Gcm` — the same cipher -/// `EncryptedBlobBackend` has always hardcoded. The enum is -/// future-proofing so `OXICLOUD_STORAGE__ENCRYPTION_CIPHER` is -/// already an accepted knob when a second cipher lands -/// (chacha20-poly1305, aes-256-siv, …). Absent + `_ENCRYPTION_KEY` -/// present → defaults to `Aes256Gcm` for back-compat with entries -/// declared before this field existed. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum EncryptionCipher { - /// AES-256 in Galois/Counter Mode. 96-bit nonce + 128-bit tag, - /// random nonce per blob. Layout on disk / S3: - /// `[12-byte nonce] [ciphertext] [16-byte GCM tag]`. - Aes256Gcm, -} - -impl EncryptionCipher { - /// Stable env-var value → variant. Case-insensitive. Extend with - /// new variants as new ciphers land; the surface is one match - /// arm here + one branch in `EncryptedBlobBackend::new_for_cipher` - /// (when that gets added). - pub fn parse(raw: &str) -> Option { - match raw.to_ascii_lowercase().as_str() { - "aes-256-gcm" | "aes256gcm" => Some(EncryptionCipher::Aes256Gcm), - _ => None, - } - } - - /// Stable env-var-friendly name — the exact string - /// `OXICLOUD_STORAGE__ENCRYPTION_CIPHER` accepts, and what - /// admin surfaces render back to operators. - pub fn as_str(self) -> &'static str { - match self { - EncryptionCipher::Aes256Gcm => "aes-256-gcm", - } - } -} - // ───────────────────────────────────────────────────────────────────── -// K1: pair-list encryption config (post-storage-multi-entry, pre-v1-header). -// See `docs/plan/storage-key-rotation.md`. +// K1: pair-list encryption config (post-storage-multi-entry, +// pre-v1-header). See `docs/plan/storage-key-rotation.md`. // -// The types and parser below REPLACE the singular `EncryptionCipher` + -// `encryption_key_base64` + `encryption_cipher` model with a -// list-of-pairs model where the last pair wins on writes and every -// pair is a candidate for reads. The `none` cipher is a first-class -// citizen so plaintext ↔ encrypted transitions can be expressed as a -// single pair-list evolution. +// [`CipherKind`], [`KeyPair`] and [`parse_encryption_pair_list`] +// replace the singular pre-K1 `EncryptionCipher` + `encryption_key_base64` +// + `encryption_cipher` model with a list-of-pairs model where the +// last pair wins on writes and every pair is a candidate for reads. +// The `none` cipher is a first-class citizen so plaintext ↔ encrypted +// transitions can be expressed as a single pair-list evolution. // -// K1 introduces these types but does NOT yet wire them into -// `NamedStorageEntry` — that flip lands in K2 alongside the v1 header -// read/write paths. The parser is exercised via unit tests only in -// this slice. +// K1.2 wires the pair-list into `NamedStorageEntry`; K2 replaces the +// on-disk format (`.blob` → v1 header at same suffix) and the read +// path (magic-byte dispatch + `` lookup). See the plan. // ───────────────────────────────────────────────────────────────────── /// The AEAD (or absence of one) used by a single [`KeyPair`]. /// -/// Distinct from the older [`EncryptionCipher`] enum in two ways: -/// -/// * Adds a `None` variant. A pair with `CipherKind::None` says +/// * `AesGcm256` — the only real AEAD OxiCloud ships today. On the +/// wire (v1 header): `[12-byte nonce] [ciphertext] [16-byte tag]`. +/// * `None` — no cipher. A pair with `CipherKind::None` says /// "writes routed to this pair produce raw plaintext". This is /// what makes the encrypt-a-plaintext-deployment and /// decrypt-an-encrypted-deployment recipes expressible without a /// second storage entry — see `docs/plan/storage-key-rotation.md` /// §"Encrypting a previously-plaintext deployment" and the /// symmetric decrypt recipe. -/// * Is the type carried by the v1 on-disk header's cipher field -/// (indirectly via `` in v1, but explicitly if a future -/// v2 adds a cipher byte to the header — the enum grows without -/// touching call sites). /// -/// Kept alongside [`EncryptionCipher`] during K1; K2 retires the -/// older enum in the same slice that flips `NamedStorageEntry`. +/// The type is carried by the v1 on-disk header's cipher field +/// (indirectly via `` in v1, but explicitly if a future +/// v2 adds a cipher byte to the header — the enum grows without +/// touching call sites). #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum CipherKind { /// AES-256 in Galois/Counter Mode. 96-bit nonce + 128-bit tag. @@ -473,9 +431,9 @@ pub enum CipherKind { } impl CipherKind { - /// Parse an env-var token: `"aes-256-gcm"` (with the - /// `aes256gcm` alias kept for continuity with - /// [`EncryptionCipher::parse`]) or `"none"`. Case-insensitive. + /// Parse an env-var token: `"aes-256-gcm"` (with the `aes256gcm` + /// alias kept for continuity with the pre-K1 spelling) or + /// `"none"`. Case-insensitive. pub fn parse(raw: &str) -> Option { match raw.to_ascii_lowercase().as_str() { "aes-256-gcm" | "aes256gcm" => Some(CipherKind::AesGcm256), @@ -695,6 +653,84 @@ pub fn parse_encryption_pair_list(entry_name: &str, raw: &str) -> Result { + tracing::info!( + target: "oxicloud::storage", + entry = %entry.name, + "storage entry `{}` — unencrypted", entry.name + ); + } + Some(pairs) => { + let head_idx = pairs.len() - 1; + let rendered: Vec = pairs + .iter() + .enumerate() + .map(|(i, kp)| { + let fp = kp.fingerprint_short().unwrap_or_else(|| "—".to_string()); + let head_mark = if i == head_idx { " ← head" } else { "" }; + format!("{}:{}{}", kp.cipher.as_str(), fp, head_mark) + }) + .collect(); + + tracing::info!( + target: "oxicloud::storage", + entry = %entry.name, + pairs = pairs.len(), + "storage entry `{}` — {} pair(s): {}", + entry.name, + pairs.len(), + rendered.join(", ") + ); + + // Warn on head-vs-other fingerprint divergence — the + // signal for "rotation in progress, don't drop the + // older key yet". Uses fingerprints (not raw + // material) so the comparison stays cheap and the + // log line reveals no key bytes. + let head_fp = pairs[head_idx].fingerprint_short(); + let non_head_fps: Vec> = pairs[..head_idx] + .iter() + .map(|kp| kp.fingerprint_short()) + .collect(); + if !non_head_fps.is_empty() && non_head_fps.iter().any(|fp| fp != &head_fp) { + tracing::warn!( + target: "oxicloud::storage", + entry = %entry.name, + head_fp = ?head_fp, + "storage entry `{}` has a rotation window open — head pair differs \ + from at least one older pair. Existing blobs written under an older \ + key stay readable, but keep the older pair in `.env` until the \ + format-upgrade job has reconciled them.", + entry.name + ); + } + } + } + } +} + /// One named storage entry declared in `.env`. /// /// See `docs/plan/storage-multi-entry.md`. Each entry is a fully-realised @@ -724,22 +760,67 @@ pub struct NamedStorageEntry { pub s3: Option, /// Azure configuration for `Azure`. `None` for other backends. pub azure: Option, - /// Per-entry AES-256-GCM key (base64, exactly 32 bytes decoded). - /// Presence implies encryption is enabled on this entry — no - /// separate `_ENCRYPTION_ENABLED` toggle. Absence = raw backend. - /// Validated at parse time; boot aborts on invalid key. - pub encryption_key_base64: Option, - /// Cipher choice for this entry. `Some(Aes256Gcm)` today when - /// `_ENCRYPTION_KEY` is set (whether or not the operator - /// declared `_ENCRYPTION_CIPHER=aes-256-gcm` explicitly, since - /// that's currently the only supported variant). - /// `None` when no encryption key is set. When new ciphers land, - /// the parser reads `_ENCRYPTION_CIPHER` to distinguish; today - /// the field is effectively a boolean because there's exactly - /// one variant. Kept as an enum on the type so downstream code - /// pattern-matches the concept explicitly and we don't lose the - /// intent when a second cipher is added. - pub encryption_cipher: Option, + /// Ordered list of `:` pairs from + /// `OXICLOUD_STORAGE__ENCRYPTION_KEY`. `None` means the + /// operator did not set the variable — the entry is + /// unencrypted, writes and reads pass through the raw backend. + /// + /// `Some(pairs)` is guaranteed non-empty (parser rejects the + /// empty-list case). The LAST pair is the write pair; every + /// pair is a candidate for reads. See + /// `docs/plan/storage-key-rotation.md` §"The pair-list config". + /// + /// Access this field through the [`Self::head_key_material`], + /// [`Self::head_cipher`], [`Self::is_encrypted`], and + /// [`Self::encryption_pairs`] helpers rather than pattern-matching + /// directly — they encapsulate the "unencrypted vs `none:`-headed + /// pair-list" distinction and keep call sites stable across + /// future K2 changes. + pub encryption: Option>, +} + +impl NamedStorageEntry { + /// Head pair — the one used for writes (K1) and, once K2 wires + /// the header, for read-dispatch of blobs whose header advertises + /// the head pair's `key_fp`. + /// + /// `None` when the entry has no `_ENCRYPTION_KEY` at all OR when + /// the head pair is `none:` (writes produce plaintext, so no key + /// material to hand to the AEAD). + pub fn head_key_material(&self) -> Option<&[u8; 32]> { + self.encryption + .as_ref() + .and_then(|pairs| pairs.last()) + .and_then(|kp| kp.key_material.as_ref()) + } + + /// Head pair's cipher choice. `None` when the entry has no + /// `_ENCRYPTION_KEY` at all. `Some(CipherKind::None)` when the + /// head pair is explicitly `none:` — semantically distinct from + /// "unconfigured", useful during decrypt-in-place migrations. + pub fn head_cipher(&self) -> Option { + self.encryption + .as_ref() + .and_then(|pairs| pairs.last()) + .map(|kp| kp.cipher) + } + + /// `true` iff writes to this entry produce ciphertext right now. + /// Distinct from "the operator has ever configured encryption" — + /// during a decrypt-in-place migration this returns `false` (head + /// is `none:`) even though older pairs in the list are real + /// ciphers used to READ existing encrypted blobs. + pub fn is_encrypted(&self) -> bool { + self.head_cipher().is_some_and(|c| c != CipherKind::None) + } + + /// The whole pair list, or an empty slice when the entry is + /// unencrypted. Used by K2's read path to walk pairs and by + /// `storage_rotate` to enumerate legacy pairs. Callers that only + /// need the write pair should prefer [`Self::head_key_material`]. + pub fn encryption_pairs(&self) -> &[KeyPair] { + self.encryption.as_deref().unwrap_or(&[]) + } } /// Validation for a `NamedStorageEntry.name`. Restricts to a safe subset @@ -989,56 +1070,22 @@ fn parse_named_entry(name: &str) -> Result { } } - // Encryption key — presence implies enabled. Validate now (base64 + - // decoded length) so we fail at boot, not at first blob write. - let encryption_key_base64 = match env::var(format!("OXICLOUD_STORAGE_{name}_ENCRYPTION_KEY")) { - Ok(k) if !k.is_empty() => { - validate_encryption_key(name, &k)?; - Some(k) - } + // Encryption — pair-list format. Parser accepts both the K1 + // shape (`aes-256-gcm:` or bare ``) and future shapes + // (2-pair rotation, `none:` head, …). See + // `docs/plan/storage-key-rotation.md`. + let encryption = match env::var(format!("OXICLOUD_STORAGE_{name}_ENCRYPTION_KEY")) { + Ok(raw) if !raw.trim().is_empty() => Some(parse_encryption_pair_list(name, &raw)?), _ => None, }; - // Cipher — future-proofing knob. Only `aes-256-gcm` today. - // Explicit value → parse (unknown = fail-fast so a typo isn't - // silently defaulted). Absent + key set → default to - // `Aes256Gcm`. Absent + no key → `None` (no encryption at all). - let encryption_cipher = match env::var(format!("OXICLOUD_STORAGE_{name}_ENCRYPTION_CIPHER")) { - Ok(raw) if !raw.is_empty() => match EncryptionCipher::parse(&raw) { - Some(c) => Some(c), - None => { - return Err(format!( - "OXICLOUD_STORAGE_{name}_ENCRYPTION_CIPHER=`{raw}` is not a known cipher — \ - supported: `aes-256-gcm`." - )); - } - }, - _ => { - if encryption_key_base64.is_some() { - Some(EncryptionCipher::Aes256Gcm) - } else { - None - } - } - }; - - // Nonsense combo — cipher declared without a key. Refuse rather - // than silently ignore the operator's declared intent. - if encryption_cipher.is_some() && encryption_key_base64.is_none() { - return Err(format!( - "OXICLOUD_STORAGE_{name}_ENCRYPTION_CIPHER is set but \ - OXICLOUD_STORAGE_{name}_ENCRYPTION_KEY is not — set the key too, or remove the cipher." - )); - } - Ok(NamedStorageEntry { name: name.to_string(), backend, root_dir, s3, azure, - encryption_key_base64, - encryption_cipher, + encryption, }) } @@ -1104,20 +1151,18 @@ fn synthesize_default_from_legacy_vars() -> Result { } } - let encryption_key_base64 = match env::var("OXICLOUD_STORAGE_ENCRYPTION_KEY") { - Ok(k) if !k.is_empty() => { - validate_encryption_key("default", &k)?; - Some(k) - } + // Legacy synthesis path: the pre-multi-entry world had a single + // `OXICLOUD_STORAGE_ENCRYPTION_KEY` (raw base64, no cipher/none + // syntax). The pair-list parser accepts that shape as a + // 1-pair `aes-256-gcm:` after defaulting the cipher, so we + // route through it and get the same validation for free. + // + // The legacy flat surface has no cipher variable, so no + // agreement check is needed here. + let encryption = match env::var("OXICLOUD_STORAGE_ENCRYPTION_KEY") { + Ok(k) if !k.trim().is_empty() => Some(parse_encryption_pair_list("default", &k)?), _ => None, }; - // Legacy synthesis: no `OXICLOUD_STORAGE_ENCRYPTION_CIPHER` env - // var exists in the legacy flat surface — always default to - // AES-256-GCM when a legacy key is set (matches the hardcoded - // pre-multi-entry behaviour). - let encryption_cipher = encryption_key_base64 - .as_ref() - .map(|_| EncryptionCipher::Aes256Gcm); Ok(NamedStorageEntry { name: "default".to_string(), @@ -1125,33 +1170,10 @@ fn synthesize_default_from_legacy_vars() -> Result { root_dir, s3, azure, - encryption_key_base64, - encryption_cipher, + encryption, }) } -/// Validate a base64-encoded 32-byte encryption key. Called at parse -/// time (not first-use) so a bad key aborts boot with a clear -/// per-entry message instead of failing much later inside -/// `EncryptedBlobBackend::new`. -fn validate_encryption_key(entry_name: &str, key_b64: &str) -> Result<(), String> { - use base64::Engine; - let decoded = base64::engine::general_purpose::STANDARD - .decode(key_b64) - .map_err(|e| { - format!("OXICLOUD_STORAGE_{entry_name}_ENCRYPTION_KEY is not valid base64: {e}") - })?; - if decoded.len() != 32 { - return Err(format!( - "OXICLOUD_STORAGE_{entry_name}_ENCRYPTION_KEY decodes to {} bytes; must be exactly \ - 32 bytes (AES-256). Generate a fresh key via \ - `POST /api/admin/settings/storage/generate-key`.", - decoded.len() - )); - } - Ok(()) -} - impl Default for StorageConfig { fn default() -> Self { // Architecture-appropriate max upload size to avoid overflow on 32-bit systems @@ -2907,6 +2929,7 @@ impl AppConfig { config.storage_entries = parse_storage_entries().unwrap_or_else(|e| { panic!("Invalid storage configuration in environment: {e}"); }); + log_storage_encryption_summary(&config.storage_entries); // Storage backend selection if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") { @@ -3405,7 +3428,7 @@ mod tests { assert_eq!(entries[0].backend, StorageBackendType::Local); assert_eq!(entries[0].root_dir.as_deref(), Some("/data")); assert!(entries[0].s3.is_none()); - assert!(entries[0].encryption_key_base64.is_none()); + assert!(entries[0].encryption.is_none()); } #[test] @@ -3433,10 +3456,12 @@ mod tests { set("OXICLOUD_STORAGE_ENCRYPTION_KEY", VALID_KEY_B64); let entries = parse_storage_entries().unwrap(); assert_eq!(entries.len(), 1); - assert_eq!( - entries[0].encryption_key_base64.as_deref(), - Some(VALID_KEY_B64) - ); + // Legacy flat-var path routes through `parse_encryption_pair_list` + // and produces a 1-pair `aes-256-gcm:` list. + let pairs = entries[0].encryption.as_ref().expect("expected pair list"); + assert_eq!(pairs.len(), 1); + assert_eq!(pairs[0].cipher, CipherKind::AesGcm256); + assert_eq!(pairs[0].key_material, Some([0u8; 32])); } #[test] @@ -3492,10 +3517,10 @@ mod tests { set("OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY", VALID_KEY_B64); let entries = parse_storage_entries().unwrap(); assert_eq!(entries.len(), 1); - assert_eq!( - entries[0].encryption_key_base64.as_deref(), - Some(VALID_KEY_B64) - ); + let pairs = entries[0].encryption.as_ref().expect("expected pair list"); + assert_eq!(pairs.len(), 1); + assert_eq!(pairs[0].cipher, CipherKind::AesGcm256); + assert_eq!(pairs[0].key_material, Some([0u8; 32])); } // ── Cell (Yes, Yes) — fail fast on conflict diff --git a/src/common/di.rs b/src/common/di.rs index ca555373..5565483c 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -366,7 +366,7 @@ impl AppServiceFactory { // // When `storage_entries` is non-empty, encryption is already // applied inside `build_entry_backend` from the entry's own - // `encryption_key_base64` (per-entry key). This block is the + // pair-list (head-pair key). This block is the // pre-multi-entry fallback that reads the flat // `OXICLOUD_STORAGE_ENCRYPTION_*` vars — reachable only for // fresh installs with no explicit storage config at all diff --git a/src/infrastructure/services/entry_backend.rs b/src/infrastructure/services/entry_backend.rs index fa427f21..0c915256 100644 --- a/src/infrastructure/services/entry_backend.rs +++ b/src/infrastructure/services/entry_backend.rs @@ -229,30 +229,24 @@ pub fn build_entry_backend( } }; - // Encryption decorator — presence-implies-enabled, per plan §Encryption. - let Some(key_b64) = entry.encryption_key_base64.as_ref() else { + // Encryption decorator — presence-implies-enabled, per plan + // §Encryption. K1 preserves single-head-pair behaviour: writes and + // reads use the head pair's material. K2 replaces the decorator + // with a header-aware read/write path that consults the full pair + // list; this call site becomes a wrapper construction rather than + // a single-key handoff at that point. + let Some(key) = entry.head_key_material() else { + // No pair list at all, OR head pair is `CipherKind::None` + // (mid-decrypt-migration state). Both cases mean "writes go + // straight to the raw backend today"; older encrypted pairs + // in the list stay unreachable until K2 wires the read + // fallback. return base; }; use crate::infrastructure::services::encrypted_blob_backend::EncryptedBlobBackend; - let key_bytes = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, key_b64) - .unwrap_or_else(|e| { - panic!( - "entry `{}` encryption key is not valid base64: {e} — parser was supposed to \ - catch this at boot", - entry.name - ) - }); - let key: [u8; 32] = key_bytes.try_into().unwrap_or_else(|v: Vec| { - panic!( - "entry `{}` encryption key decoded to {} bytes; must be 32 — parser was supposed to \ - catch this at boot", - entry.name, - v.len() - ) - }); tracing::info!( - "Storage entry `{}` encrypted with AES-256-GCM (key from env)", + "Storage entry `{}` encrypted with AES-256-GCM (head pair from env)", entry.name ); - Arc::new(EncryptedBlobBackend::new(base, &key)) + Arc::new(EncryptedBlobBackend::new(base, key)) } diff --git a/src/infrastructure/services/storage_migration_service.rs b/src/infrastructure/services/storage_migration_service.rs index 6436ed47..49390df8 100644 --- a/src/infrastructure/services/storage_migration_service.rs +++ b/src/infrastructure/services/storage_migration_service.rs @@ -328,7 +328,10 @@ impl RecoverableJobHandler for StorageMigrationService { if let Some(source) = source_entry && entry_identity(source) == entry_identity(target_entry) { - let key_differs = source.encryption_key_base64 != target_entry.encryption_key_base64; + // Compare head-pair materials — the "write key" for each + // entry. A non-head pair difference (mid-rotation) doesn't + // count as a key change for the purposes of this refusal. + let key_differs = source.head_key_material() != target_entry.head_key_material(); let hint = if key_differs { " (encryption key differs → this looks like an in-place key rotation; \ create a new entry pointing at a DIFFERENT bucket / dir, migrate to it, \ @@ -446,7 +449,14 @@ impl RecoverableJobHandler for StorageMigrationService { }; let mut copied_count = 0u64; - let mut skipped_count = 0u64; + // K1.2: with the target-skip short-circuit gone (see the + // detailed comment further down), no blob is ever "skipped" + // during a migration walk today. The counter stays wired + // through the log lines + `finish_completed` so K3's + // format-aware smart-skip can re-populate it without + // touching the observability surface. Not mutated in this + // slice — hence no `mut`. + let skipped_count: u64 = 0; let mut failed_count = 0u64; let mut source_missing_count = 0u64; @@ -574,26 +584,33 @@ impl RecoverableJobHandler for StorageMigrationService { } } - // Skip when the target already has it — supports - // idempotent resume and cheap re-runs against a - // partially-migrated target. - match target.blob_exists(hash).await { - Ok(true) => { - skipped_count += 1; - continue; - } - Ok(false) => {} - Err(e) => { - tracing::warn!( - target: "oxicloud::migration", - event = "storage_migration.blob_exists_error", - run_id = %store.run_id(), - hash = %hash, - error = %e, - "blob_exists probe on target failed; attempting copy anyway" - ); - } - } + // Always copy — do NOT short-circuit on + // `target.blob_exists(hash)`. Ed hit this on 2026-08-01 + // during S3 → local migration testing with encryption + // enabled on the target: the target had pre-existing + // plaintext blobs from an earlier local-active session, + // so `blob_exists` returned true and the migration + // silently skipped them. Result: the "encrypted" + // target ended up with mixed plaintext + ciphertext + // blobs — undetectable until a subsequent read failed. + // + // The old skip was justified by two use cases: + // (a) resume idempotency — the last cursor-checkpoint + // window (~100 blobs) gets re-processed on resume; + // (b) target-side dedup — same content already present. + // + // Both are now handled by unconditional overwrite: the + // re-copy is bounded by the checkpoint window (small), + // and dedup-hit content is rare in practice + // (content-addressability means duplicate blobs ARE + // the same blob unless two backends were seeded + // separately from the same source). + // + // K3's `storage_rotate` job will restore a smart skip + // via the v1 header's `` field — "already at + // head format+key" then becomes cheaply detectable + // without reading target bytes. Until then, correct > + // fast. match copy_blob(self.source.as_ref(), target.as_ref(), hash).await { Ok(()) => {