Re-chunk pre-CDC legacy blobs into CDC manifests at startup

Files uploaded before chunk_manifests landed (20260414000000) are stored
as ONE whole-file blob with no manifest. Every legacy fallback in
DedupService exists to serve them, and the cost concentrates on Range
reads: with encryption enabled, seeking inside a legacy video decrypts
the ENTIRE blob (AES-GCM is all-or-nothing), where a CDC file decrypts
only the overlapping <=1 MiB chunks.

This adds a one-time, idempotent background migration (spawned from the
composition root after dedup init, maintenance pool) that converts each
legacy blob into a regular CDC file, indistinguishable from a native
upload:

  1. Spool the blob through the normal read path (decrypts when
     encryption is on) to a per-attempt-unique temp file, verifying
     BLAKE3 == hash; sizes come from the verified spool, never from the
     legacy storage.blobs.size column (the manifest's total_size drives
     Range arithmetic).
  2. CDC-chunk + store chunks via the existing store_chunks (one
     manifest reference per distinct chunk).
  3. One short accounting TX with the blob row locked: manifest INSERT
     with ref_count = N current file references, blob ref_count -= N,
     row deleted only at exactly 0 - so single-chunk files (chunk hash
     == file hash) keep the physical blob, which IS the chunk; only
     bookkeeping moves, no bytes are rewritten.
  4. Physical whole-file blob deleted only when its row dropped.

Races lean on the row lock: a concurrent identical upload landing a
legacy reference after commit keeps the blob row alive and that file
readable via the fallback (bounded space leak, never data loss); a
crash between chunk store and the TX over-counts one file's chunk refs
(also a bounded leak). Corrupt blobs (content != hash) are logged,
counted, excluded from the sweep and left untouched, with a hard cap
before aborting.

Per-hash failures never block the sweep; manifests are the resumability
marker, so a restart continues where it left off. The legacy read/write
fallbacks stay in place as the safety net while a deployment converges;
they can be deleted once fleets report "legacy re-chunk: nothing to do".

Opt-out via OXICLOUD_LEGACY_RECHUNK=false (documented in example.env)
for metered remote backends where the one-time re-read should be
scheduled deliberately.

Covered by five integration tests against real PostgreSQL (multi-chunk
accounting + Range across a chunk boundary, single-chunk physical-blob
preservation, corrupt-blob isolation, empty blob, and the full
encrypted-backend roundtrip); they run concurrently, which also
exercises the cross-sweep race handling.

https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1

src/common/config.rs

@@ -254,6 +254,14 @@ pub struct StorageConfig {
     /// bound on how stale an ancestor folder's ETag can be after a change.
     /// Default: 500. Env: `OXICLOUD_TREE_ETAG_FLUSH_MS`.
     pub tree_etag_flush_ms: u64,
+    /// Startup background migration that re-chunks legacy whole-file blobs
+    /// (written before CDC chunking landed) into chunk manifests, so Range
+    /// reads stop paying a full-blob read — and, with encryption enabled, a
+    /// full-blob decrypt. Idempotent and incremental; a no-op (one COUNT
+    /// query) once no legacy blobs remain. Disable on metered remote
+    /// backends where the one-time re-read of every legacy blob should be
+    /// scheduled deliberately. Default: true. Env: `OXICLOUD_LEGACY_RECHUNK`.
+    pub legacy_rechunk_enabled: bool,
     /// Which blob storage backend to use (`local`, `s3`, or `azure`).
     pub backend: StorageBackendType,
     /// S3-compatible backend configuration (used when `backend == S3`).
@@ -397,6 +405,7 @@ impl Default for StorageConfig {
             chunk_dir: None,
             usage_reconcile_secs: 600, // 10 minutes
             tree_etag_flush_ms: 500,
+            legacy_rechunk_enabled: true,
             backend: StorageBackendType::Local,
             s3: None,
             azure: None,
@@ -1305,6 +1314,12 @@ impl AppConfig {
             config.storage.tree_etag_flush_ms = val;
         }
 
+        // Legacy whole-file blob re-chunk migration (startup background task)
+        if let Ok(enabled) = env::var("OXICLOUD_LEGACY_RECHUNK") {
+            config.storage.legacy_rechunk_enabled =
+                enabled.eq_ignore_ascii_case("true") || enabled == "1";
+        }
+
         // Storage backend selection
         if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
             match backend.to_lowercase().as_str() {

src/common/di.rs

@@ -319,6 +319,18 @@ impl AppServiceFactory {
         );
         dedup_service.initialize().await?;
 
+        // One-time background migration: re-chunk pre-CDC whole-file blobs
+        // into chunk manifests so Range reads (and, with encryption, partial
+        // decrypts) stop paying for the entire blob. No-op once converged.
+        if self.config.storage.legacy_rechunk_enabled {
+            dedup_service.spawn_legacy_rechunk();
+        } else {
+            tracing::info!(
+                "Legacy re-chunk migration disabled (OXICLOUD_LEGACY_RECHUNK=false) — \
+                 pre-CDC whole-file blobs, if any, will keep using the legacy read path"
+            );
+        }
+
         tracing::info!(
             "Core services initialized: path service, file content cache, thumbnails, chunked upload, image transcode, dedup (PRIMARY blob storage)"
         );