diff --git a/docs/plan/drive.md b/docs/plan/drive.md index 8897cc96..e2f0e989 100644 --- a/docs/plan/drive.md +++ b/docs/plan/drive.md @@ -314,7 +314,9 @@ For reference, the equivalent (broken) one-CTE form looks like: WITH new_drive AS ( INSERT INTO storage.drives (kind, default_for_user, quota_bytes, policies) - VALUES ('personal', $user_id, $quota, '{}'::jsonb) + VALUES ('personal', $user_id, NULL, '{}'::jsonb) -- personal drives carry + -- NULL quota; the cap is + -- the user envelope, §7 RETURNING id ), new_root AS ( @@ -366,10 +368,12 @@ The fix is the four-step transaction described above. Rust: ```rust let mut tx = pool.begin().await?; +// Personal drives carry NULL quota_bytes — the cap is the user envelope +// (`auth.users.storage_quota_bytes`, §7), not the per-drive column. let drive_id: Uuid = sqlx::query_scalar( r#"INSERT INTO storage.drives (kind, default_for_user, quota_bytes) - VALUES ('personal', $1, $2) RETURNING id"#, -).bind(owner).bind(quota).fetch_one(&mut *tx).await?; + VALUES ('personal', $1, NULL) RETURNING id"#, +).bind(owner).fetch_one(&mut *tx).await?; let folder_id: Uuid = sqlx::query_scalar( r#"INSERT INTO storage.folders @@ -450,7 +454,7 @@ that try to bypass it now hit a DB-level wall. | Per-resource grant outward | yes (subject to drive policies) | yes | yes | | Cross-drive move | yes (subject to `forbid_cross_drive_move`) | yes | yes | | Kind conversion | no — always default-personal | yes → may be promoted to `kind='shared'` later (drops the single-user restriction, picks up members) | no | -| Change `quota_bytes` | **OxiCloud admin only** (not the drive owner — §7) | **OxiCloud admin only** | **OxiCloud admin only** | +| Change `quota_bytes` | N/A — `drives.quota_bytes` is NULL for personal drives. The envelope is `auth.users.storage_quota_bytes` (admin-only — §7) | N/A — same | **OxiCloud admin only** (not the drive owner — §7) | ### 4. Roles → permission bundles @@ -493,7 +497,7 @@ against the same table. | Event | Behaviour | |---|---| -| New internal user registers | Auto-create a default personal drive (`kind='personal'`, `default_for_user=`, `quota_bytes=`) + its root folder (`name='Personal'`, `parent_id=NULL`, drive_id pinned) + the Owner role_grant (`role_grants(subject_type='user', subject_id=, resource_type='drive', resource_id=, role='owner')`) — **all four writes in one CTE statement** (§3), atomic against server crash. | +| New internal user registers | Auto-create a default personal drive (`kind='personal'`, `default_for_user=`, `quota_bytes=NULL` — the envelope lives on `auth.users.storage_quota_bytes`, see §7) + its root folder (`name='Personal'`, `parent_id=NULL`, drive_id pinned) + the Owner role_grant (`role_grants(subject_type='user', subject_id=, resource_type='drive', resource_id=, role='owner')`) — **all four writes in one transaction** (§3), atomic against server crash. | | External user invited (magic-link only) | **No personal drive created.** External users are grant-only recipients with no storage. | | External user converts to internal (future flow) | Default personal drive created at conversion time. | | User deleted | **Default** personal drive cascade-deletes via `ON DELETE CASCADE` on `default_for_user`. **Secondary** personal drives (`kind='personal' AND default_for_user IS NULL` and whose sole owner `role_grants` row points at the user) are deleted by an application-layer pass in the same transaction. `role_grants` rows referencing the deleted user are removed from all shared drives. If a removal would leave a shared drive with zero owners, deletion is refused — admin must transfer first. | @@ -506,67 +510,157 @@ against the same table. ### 7. Quota model -The per-user `auth.users.storage_quota_bytes` field is **migrated to -the user's personal drive's `quota_bytes`** in one step, then the -column is deprecated (kept for one release cycle as a no-op, dropped -in a later migration). +Two ceilings, two different jobs: -After the cutover: -- Every drive owns its quota. Files inside a drive count against that - drive's `used_bytes` only. -- A user who collaborates in a 1 TB shared drive sees their personal - drive's quota as "their" quota; the shared drive's quota is owned - by the team. -- New drives default to a tenant-configured `OXICLOUD_DEFAULT_DRIVE_QUOTA_BYTES` - setting (separate env var, replacing today's per-user equivalent). +- **Per-user envelope** — `auth.users.storage_quota_bytes` stays + as the canonical "how much can this user store on this server." + It caps the sum of `used_bytes` across **every personal drive + the user owns** (default + any secondaries — see §2). Shared + drives never count against any user envelope. +- **Per-drive ceiling** — `drives.quota_bytes` is a per-drive cap + that applies **only to shared drives**. For personal drives + this column is `NULL` (unlimited at the drive layer); the + effective cap comes from the user envelope. -`used_bytes` is maintained incrementally on every file insert/delete -(plus a periodic reconciliation job to fix drift, similar to the -existing per-user accounting). +Why the asymmetry: a user's personal storage is a single budget +that the operator has agreed to provide; splitting it into +sub-quotas per personal drive is a sub-quota UX trap (users now +have to plan how to allocate "their" bytes between drives they +own). A shared drive's quota IS the team's resource budget, owned +by the operator, set independently. + +#### Upload gate + +Pre-upload, both checks run, in order: + +1. **Drive cap** (`drives.quota_bytes`) — skipped when NULL. + Always skipped for personal drives by virtue of the NULL + convention; applies for shared drives. +2. **User envelope** (`auth.users.storage_quota_bytes`) — runs + only when the target drive is personal. The check sums + `used_bytes` across the caller's personal drives (or, fast + path while no secondaries exist on the UI, reads the cached + `auth.users.storage_used_bytes`). + +For shared-drive uploads, only the per-drive check applies and the +user envelope is untouched — collaborating in a 1 TB shared drive +costs no personal bytes. + +#### `used_bytes` accounting + +Maintained incrementally on every file insert/delete in +`storage.drives.used_bytes`. The user-side cached counter +(`auth.users.storage_used_bytes`) is updated **only when the +target drive is personal** — the per-upload delta hook reads +`drives.kind` from the same query that already fetches +`drives.used_bytes` for the drive-cap check, so the hot path adds +zero round-trips. + +A periodic reconciliation job rebuilds both counters from ground +truth: + +```sql +-- Per-drive: unchanged from today. +UPDATE storage.drives SET used_bytes = ( + SELECT COALESCE(SUM(size), 0) FROM storage.files + WHERE drive_id = d.id AND NOT is_trashed +) d; + +-- Per-user: sum of personal-drive used_bytes owned by the user. +UPDATE auth.users u SET storage_used_bytes = COALESCE(( + SELECT SUM(d.used_bytes) + FROM storage.drives d + JOIN storage.role_grants g + ON g.resource_type = 'drive' AND g.resource_id = d.id + AND g.role = 'owner' + AND g.subject_type = 'user' AND g.subject_id = u.id + WHERE d.kind = 'personal' +), 0); +``` + +Fast-path variant while only default personals are exposed: + +```sql +UPDATE auth.users u SET storage_used_bytes = COALESCE(( + SELECT used_bytes FROM storage.drives WHERE default_for_user = u.id +), 0); +``` + +Reconciliation runs on the maintenance pool — never blocks +uploads. Drift between deltas and the sweep is bounded by the +sweep interval (default 10 min). #### Quota mutation is OxiCloud-admin only -Changing `drives.quota_bytes` is **not** in the drive `owner` role -bundle (§4). It requires the tenant-level OxiCloud admin role -(`auth.users.role = 'admin'`), checked at -`PATCH /api/admin/drives/{id}/quota` — the only callsite that -mutates the column. Drive owners can rename, edit policies, and -manage members; they cannot self-grant capacity. +Changing `drives.quota_bytes` (shared drives only) is **not** in +the drive `owner` role bundle (§4). It requires the tenant-level +OxiCloud admin role (`auth.users.role = 'admin'`), checked at +`PATCH /api/admin/drives/{id}/quota`. Drive owners can rename, +edit policies, and manage members; they cannot self-grant +capacity. + +Changing `auth.users.storage_quota_bytes` (the personal envelope) +is likewise admin-only — same surface and audit pattern as today. Why this seam matters: -- **Resource allocation is a tenant concern, not a drive - concern.** Storage bytes are a finite system resource the - operator pays for. The drive owner is empowered over the - drive's *use*; the admin is empowered over its *budget*. Same - separation that exists today between a user and the operator - who set `OXICLOUD_DEFAULT_QUOTA_BYTES`. -- **Privilege-escalation seam closed.** Without this carve-out, - any user with a personal drive (= every internal user) could - raise their own quota by virtue of being its sole owner — - trivially defeating the quota system. +- **Resource allocation is a tenant concern.** Storage bytes are + a finite system resource the operator pays for. The drive owner + is empowered over the drive's *use*; the admin is empowered + over its *budget*. +- **Privilege-escalation seam closed.** Without the per-drive + carve-out, an Owner of a shared drive could raise its quota. + Without the per-user carve-out, any internal user could raise + their own envelope by virtue of owning their personal drive. - **Shared-drive coherence.** A shared drive's quota is set by the operator at provisioning; subsequent capacity requests go - through the admin, not the drive's group owners. Keeps the - capacity decision auditable and out of intra-team politics. + through the admin, not the drive's group owners. -The admin endpoint is the same surface the operator uses today to -change `auth.users.storage_quota_bytes`; D4 simply re-targets the -write at `storage.drives.quota_bytes`. Audit log emits -`drive.quota_changed` with `granted_by=` and the -old/new values, mirroring the existing user-quota change event. +Audit log emits `drive.quota_changed` (shared drives) and +`user.quota_changed` (envelope) with `granted_by=` +and the old/new values. -**Chunk dedup vs per-drive quota.** With the CDC chunk store landed -in v0.7.0 (see `delta_upload_service`, `upload_ingest`, instant -upload by hash), a single chunk can be referenced by files in -multiple drives. The accounting decision: **each drive counts the -file's logical size in full against its own `used_bytes`** — dedup -savings are server-side only and never visible in the per-drive -quota number. This matches the existing per-user blob-dedup model -and avoids the alternative "pro-rated quota" trap (which makes -quota math depend on cross-drive content and breaks the user's -mental model of "I have 1 TB free"). Reconciliation job sums file -sizes per drive, not chunk allocations. +#### Multiple personal drives — schema-ready, no public surface + +The schema and service layer treat personal drives as "any +personal drive owned by a user counts against the envelope," so +secondary personal drives (`kind='personal' AND +default_for_user IS NULL`) just work the day they ship. Today +there is **no public API surface to create them** — the only +`POST /api/drives` flow creates shared drives, and personal-drive +provisioning happens at user registration via the lifecycle hook +(§6). The capability matrix (§3) keeps the secondary column for +the migration backfill path and for the future, but it is not +user-reachable. + +When secondary personals are eventually exposed (e.g. a "Vault" +end-to-end-encrypted drive kind, or a "Work" silo with a stricter +policy bag), the quota model needs no change — the sum-of-personal +formula already accounts for them. + +#### Chunk dedup vs per-drive quota + +With the CDC chunk store landed in v0.7.0 (see +`delta_upload_service`, `upload_ingest`, instant upload by hash), +a single chunk can be referenced by files in multiple drives. The +accounting decision: **each drive counts the file's logical size +in full against its own `used_bytes`** — dedup savings are +server-side only and never visible in the per-drive quota number. +This matches the existing per-user blob-dedup model and avoids +the "pro-rated quota" trap (which makes quota math depend on +cross-drive content and breaks the user's mental model of "I have +1 TB free"). Reconciliation sums file sizes per drive, not chunk +allocations. + +#### Migration + +One-shot at deploy: NULL out `drives.quota_bytes` for every +`kind='personal'` row (D4 backfilled them from +`auth.users.storage_quota_bytes` for the original "every drive +owns its quota" plan). Then run the new reconciliation sweep once +to resync `auth.users.storage_used_bytes` to "sum of personal +drives" (excludes any shared-drive bytes the old delta path may +have charged to it). Both steps idempotent. ### 8. Policies (JSONB, extensible) diff --git a/migrations/20260806000000_personal_drive_quota_to_user_envelope.sql b/migrations/20260806000000_personal_drive_quota_to_user_envelope.sql new file mode 100644 index 00000000..bb1b0caf --- /dev/null +++ b/migrations/20260806000000_personal_drive_quota_to_user_envelope.sql @@ -0,0 +1,44 @@ +-- Switch personal-drive quota semantics from "every drive owns its quota" +-- to "user envelope on the SUM of personal-drive `used_bytes`". +-- See docs/plan/drive.md §7. +-- +-- Two idempotent steps: +-- 1. NULL `drives.quota_bytes` for every `kind='personal'` row. After this +-- migration the column is meaningful only for shared drives; personal +-- drives' cap is `auth.users.storage_quota_bytes`. +-- 2. Resync `auth.users.storage_used_bytes` to the sum-of-personal-drives +-- formula. Prior deltas may have over-counted by including shared-drive +-- uploads in the user counter; this snaps every user back to the new +-- envelope. Same shape the periodic sweep uses going forward. +-- +-- Both statements `IS DISTINCT FROM`-guarded so reruns are cheap no-ops on +-- already-migrated databases. The order — NULL first, then resync — doesn't +-- matter for correctness but follows the doc's narrative. + +-- 1. Drop per-drive quotas for personal drives (no-op for already-NULL rows). +UPDATE storage.drives + SET quota_bytes = NULL + WHERE kind = 'personal' + AND quota_bytes IS NOT NULL; + +-- 2. Resync user-side cached counter to the new sum-of-personal-drives +-- semantics. Mirrors `update_all_users_storage_usage` in +-- `storage_usage_service.rs`. External users excluded (no storage). +UPDATE auth.users u + SET storage_used_bytes = COALESCE(t.total, 0) + FROM auth.users u2 + LEFT JOIN ( + SELECT g.subject_id AS user_id, + SUM(d.used_bytes)::bigint AS total + FROM storage.drives d + JOIN storage.role_grants g + ON g.resource_type = 'drive' + AND g.resource_id = d.id + AND g.role = 'owner' + AND g.subject_type = 'user' + WHERE d.kind = 'personal' + GROUP BY g.subject_id + ) t ON t.user_id = u2.id + WHERE u.id = u2.id + AND NOT u2.is_external + AND u.storage_used_bytes IS DISTINCT FROM COALESCE(t.total, 0); diff --git a/src/application/services/file_upload_service.rs b/src/application/services/file_upload_service.rs index 6f054f55..0ebab6f6 100644 --- a/src/application/services/file_upload_service.rs +++ b/src/application/services/file_upload_service.rs @@ -15,7 +15,7 @@ use crate::infrastructure::repositories::pg::FileBlobWriteRepository; use crate::infrastructure::services::dedup_service::DedupService; use crate::infrastructure::services::file_content_cache::FileContentCache; use crate::infrastructure::services::pg_acl_engine::PgAclEngine; -use tracing::{info, warn}; +use tracing::{Instrument, info, warn}; /// Service for file upload operations. /// @@ -326,21 +326,39 @@ impl FileUploadService { }; let delta = file.size as i64; - // Per-user delta — unchanged from `b5b80549` / `fbbae541`. - if let Some(owner) = file + let owner = file .owner_id .as_deref() - .and_then(|s| Uuid::parse_str(s).ok()) - { + .and_then(|s| Uuid::parse_str(s).ok()); + let folder = file + .folder_id + .as_deref() + .and_then(|s| Uuid::parse_str(s).ok()); + + // Per-user delta — only when the target drive is `kind='personal'`. + // The user envelope (`auth.users.storage_quota_bytes`) caps the SUM + // of `used_bytes` across the user's personal drives; shared-drive + // uploads do NOT count against any user. See + // `docs/plan/drive.md` §7. + // + // The discrimination happens in one SQL statement via an EXISTS + // subquery on the folder's drive kind — no extra round-trip vs + // the unconditional delta. Without a folder id (root-level + // upload — folder service refuses these) the user-side delta is + // simply skipped; the sweep reconciles regardless. + if let (Some(owner), Some(folder)) = (owner, folder) { let service_clone = Arc::clone(storage_service); - tokio::spawn(async move { - if let Err(e) = service_clone - .add_user_storage_usage_delta(owner, delta) - .await - { - warn!("Failed to bump storage usage for {owner}: {e}"); + tokio::spawn( + async move { + if let Err(e) = service_clone + .add_user_storage_usage_delta_if_personal(owner, folder, delta) + .await + { + warn!("Failed to bump user storage for {owner} (folder {folder}): {e}"); + } } - }); + .in_current_span(), + ); } // Per-drive delta (D4) — same fire-and-forget shape, resolves @@ -349,20 +367,19 @@ impl FileUploadService { // quota check and the picker quota bar read; drift from // deletes / trash is reconciled by the same sweep that handles // user-side drift. - if let Some(folder) = file - .folder_id - .as_deref() - .and_then(|s| Uuid::parse_str(s).ok()) - { + if let Some(folder) = folder { let service_clone = Arc::clone(storage_service); - tokio::spawn(async move { - if let Err(e) = service_clone - .add_drive_storage_usage_delta_by_folder(folder, delta) - .await - { - warn!("Failed to bump drive usage for folder {folder}: {e}"); + tokio::spawn( + async move { + if let Err(e) = service_clone + .add_drive_storage_usage_delta_by_folder(folder, delta) + .await + { + warn!("Failed to bump drive usage for folder {folder}: {e}"); + } } - }); + .in_current_span(), + ); } } } diff --git a/src/application/services/storage_usage_service.rs b/src/application/services/storage_usage_service.rs index 2d46f787..56ed625e 100644 --- a/src/application/services/storage_usage_service.rs +++ b/src/application/services/storage_usage_service.rs @@ -39,13 +39,22 @@ impl StorageUsageService { /// (was three: user lookup + SUM + UPDATE). NOT called on the request /// path — only by the per-upload background update and the sweep. pub async fn update_user_storage_usage(&self, user_id: Uuid) -> Result { + // User envelope = SUM of `drives.used_bytes` across personal + // drives owned by the user (see `docs/plan/drive.md` §7). Shared + // drives don't count. Ownership is canonical via `role_grants`. let total_usage: Option = sqlx::query_scalar( r#" UPDATE auth.users u SET storage_used_bytes = COALESCE(( - SELECT SUM(f.size)::bigint - FROM storage.files f - WHERE f.user_id = u.id AND NOT f.is_trashed), 0) + SELECT SUM(d.used_bytes)::bigint + FROM storage.drives d + JOIN storage.role_grants g + ON g.resource_type = 'drive' + AND g.resource_id = d.id + AND g.role = 'owner' + AND g.subject_type = 'user' + AND g.subject_id = u.id + WHERE d.kind = 'personal'), 0) WHERE u.id = $1 RETURNING u.storage_used_bytes "#, @@ -77,9 +86,15 @@ impl StorageUsageService { r#" UPDATE auth.users u SET storage_used_bytes = COALESCE(( - SELECT SUM(f.size)::bigint - FROM storage.files f - WHERE f.user_id = u.id AND NOT f.is_trashed), 0) + SELECT SUM(d.used_bytes)::bigint + FROM storage.drives d + JOIN storage.role_grants g + ON g.resource_type = 'drive' + AND g.resource_id = d.id + AND g.role = 'owner' + AND g.subject_type = 'user' + AND g.subject_id = u.id + WHERE d.kind = 'personal'), 0) WHERE u.username = $1 RETURNING u.storage_used_bytes "#, @@ -128,6 +143,50 @@ impl StorageUsageService { Ok(()) } + /// Conditional user-side delta: only fires when the target folder's + /// drive is `kind='personal'`. See `docs/plan/drive.md` §7. + /// + /// The new quota model: `auth.users.storage_quota_bytes` is the cap on + /// the SUM of `used_bytes` across the user's personal drives. Shared + /// drives never count against any user envelope. The upload hot path + /// reads `drives.kind` from the same JOIN that already runs for the + /// drive cap check; firing this conditional delta instead of the + /// unconditional [`Self::add_user_storage_usage_delta`] keeps the + /// counter aligned with that envelope semantics. Idempotent + clamped + /// at zero, same as the unconditional sibling. + /// + /// Implementation note: the EXISTS subquery is two indexed PK probes + /// (folder by id, drive by id) so the personal/shared discrimination + /// adds no real cost vs. the unconditional update. + pub async fn add_user_storage_usage_delta_if_personal( + &self, + user_id: Uuid, + folder_id: Uuid, + delta: i64, + ) -> Result<(), DomainError> { + sqlx::query( + "UPDATE auth.users u + SET storage_used_bytes = GREATEST(0, u.storage_used_bytes + $2) + WHERE u.id = $1 + AND EXISTS ( + SELECT 1 + FROM storage.folders f + JOIN storage.drives d ON d.id = f.drive_id + WHERE f.id = $3 + AND d.kind = 'personal' + )", + ) + .bind(user_id) + .bind(delta) + .bind(folder_id) + .execute(self.pool.as_ref()) + .await + .map_err(|e| { + DomainError::internal_error("StorageUsage", format!("usage delta if personal: {e}")) + })?; + Ok(()) + } + /// Incrementally adjust one drive's cached `storage.drives.used_bytes` /// by `delta` bytes — same shape as /// [`Self::add_user_storage_usage_delta`]: single statement, no @@ -312,15 +371,17 @@ impl StorageUsageService { loop { ticker.tick().await; debug!("Running scheduled storage-usage reconciliation"); - if let Err(e) = service.update_all_users_storage_usage().await { - error!("Scheduled user storage-usage reconciliation failed: {}", e); - } - // Drive sweep runs alongside the user sweep — same - // cadence, same maintenance pool. Failure is logged - // but doesn't skip the next tick. + // Drive sweep runs FIRST: the user-side sweep below + // reads `drives.used_bytes` (the per-drive sum) to + // compute its own counter, so the drive counter must + // be honest first. Failure of one is logged but + // doesn't skip the other or the next tick. if let Err(e) = service.update_all_drives_storage_usage().await { error!("Scheduled drive storage-usage reconciliation failed: {}", e); } + if let Err(e) = service.update_all_users_storage_usage().await { + error!("Scheduled user storage-usage reconciliation failed: {}", e); + } } }); } @@ -358,16 +419,33 @@ impl StorageUsagePort for StorageUsageService { async fn update_all_users_storage_usage(&self) -> Result<(), DomainError> { debug!("Starting storage-usage reconciliation sweep"); + // User envelope = SUM of `drives.used_bytes` across the user's + // personal drives. Shared drives don't count against any user + // (`docs/plan/drive.md` §7). The drive-side sweep runs FIRST + // (`start_reconciliation_job`) so `drives.used_bytes` is + // already honest by the time we read it here. + // + // Ownership lookup uses `role_grants` (canonical per §1) so + // both the user's default personal AND any secondary + // personals owned via Owner grants are summed. Secondaries + // aren't user-creatable today, but a backfill or admin path + // can produce them — covering that surface from day one. let result = sqlx::query( r#" UPDATE auth.users u SET storage_used_bytes = COALESCE(t.total, 0) FROM auth.users u2 LEFT JOIN ( - SELECT user_id, SUM(size)::bigint AS total - FROM storage.files - WHERE NOT is_trashed - GROUP BY user_id + SELECT g.subject_id AS user_id, + SUM(d.used_bytes)::bigint AS total + FROM storage.drives d + JOIN storage.role_grants g + ON g.resource_type = 'drive' + AND g.resource_id = d.id + AND g.role = 'owner' + AND g.subject_type = 'user' + WHERE d.kind = 'personal' + GROUP BY g.subject_id ) t ON t.user_id = u2.id WHERE u.id = u2.id AND NOT u2.is_external diff --git a/src/interfaces/api/handlers/admin_handler.rs b/src/interfaces/api/handlers/admin_handler.rs index 44ad42e3..b73e69ff 100644 --- a/src/interfaces/api/handlers/admin_handler.rs +++ b/src/interfaces/api/handlers/admin_handler.rs @@ -2072,15 +2072,21 @@ pub async fn internal_trigger_sweep( .into_response(); } }; - if let Err(e) = svc.update_all_users_storage_usage().await { - return AppError::internal_error(format!("user sweep failed: {e}")).into_response(); - } + // Order matches the periodic ticker (`start_reconciliation_job`): + // drive sweep first because the user sweep reads `drives.used_bytes` + // (sum-of-personal-drives — `docs/plan/drive.md` §7). Running them + // in the other order makes the user counter freeze on the previous + // tick's drive numbers — invisible in steady state but breaks any + // Hurl that trashes + sweeps within one call. if let Err(e) = svc.update_all_drives_storage_usage().await { return AppError::internal_error(format!("drive sweep failed: {e}")).into_response(); } + if let Err(e) = svc.update_all_users_storage_usage().await { + return AppError::internal_error(format!("user sweep failed: {e}")).into_response(); + } ( StatusCode::OK, - Json(serde_json::json!({ "ok": true, "ran": ["users", "drives"] })), + Json(serde_json::json!({ "ok": true, "ran": ["drives", "users"] })), ) .into_response() } diff --git a/src/interfaces/api/handlers/auth_handler.rs b/src/interfaces/api/handlers/auth_handler.rs index 0e760403..b28dab8b 100644 --- a/src/interfaces/api/handlers/auth_handler.rs +++ b/src/interfaces/api/handlers/auth_handler.rs @@ -471,11 +471,16 @@ pub async fn get_current_user( // Storage usage is served from the cached `storage_used_bytes` column — // it is NOT recomputed here. Recomputing on this hot endpoint meant an - // O(N) `SUM(size)` over all the user's files plus an `UPDATE` of - // `auth.users` on every single call (one of the most frequent endpoints). - // The cached value is kept current by the per-upload update and a periodic - // background reconciliation sweep + // O(N) `SUM(size)` plus an `UPDATE` of `auth.users` on every single call + // (one of the most frequent endpoints). The cached value is kept current + // by the per-upload update and a periodic background reconciliation sweep // (see `StorageUsageService::start_reconciliation_job`). + // + // Semantics (`docs/plan/drive.md` §7): `storage_used_bytes` is the SUM + // of `used_bytes` across the user's personal drives only. Shared drives + // never count against this envelope — collaborating in a team drive + // costs no personal bytes. The matching cap is + // `storage_quota_bytes` (admin-only mutation). let user = auth_service .auth_application_service .get_user_by_id(user_id) diff --git a/tests/api/run.sh b/tests/api/run.sh index 69d182ed..322f86b7 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -162,7 +162,8 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/nc_auth_failures.hurl" \ "$API_DIR/dedup_create.hurl" \ "$API_DIR/trash_per_drive.hurl" \ - "$API_DIR/drive_quota.hurl" + "$API_DIR/drive_quota.hurl" \ + "$API_DIR/user_envelope_quota.hurl" #bash "$API_DIR/dedup_bulk_upload.sh" diff --git a/tests/api/user_envelope_quota.hurl b/tests/api/user_envelope_quota.hurl new file mode 100644 index 00000000..47eb9775 --- /dev/null +++ b/tests/api/user_envelope_quota.hurl @@ -0,0 +1,254 @@ +# ============================================================= +# OxiCloud – User envelope quota (sum of personal drives) +# ============================================================= +# Run: +# hurl --variables-file tests/api/test.env --file-root tests \ +# --test tests/api/user_envelope_quota.hurl +# +# The model under test (`docs/plan/drive.md` §7): +# `auth.users.storage_quota_bytes` caps the SUM of `used_bytes` +# across the user's PERSONAL drives only. Shared drives never +# count against any user envelope. +# +# Cases: +# 1. Baseline — fresh user: `/me.storage_used_bytes == 0`. +# 2. Shared-drive upload does NOT touch the envelope — +# `/me.storage_used_bytes` stays 0 after upload + sweep. +# 3. Personal-drive upload DOES bump the envelope — +# `/me.storage_used_bytes == file_size` after upload + sweep. +# 4. Sweep self-heals — after trashing the personal file and +# `trigger-sweep`, `/me.storage_used_bytes` returns to 0. +# +# `trigger-sweep` is the deterministic synchronisation point: +# it runs the drive-side sweep then the user-side sweep +# (`StorageUsageService::start_reconciliation_job`), so both +# cached counters are authoritative ground-truth by the time +# the assertion fires. Gated by +# `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true` +# (set in `tests/common/server.env`). +# +# Self-contained: provisions `ue_owner` so it can run alongside +# the rest of the suite. +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 — Admin login. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ "username": "{{username}}", "password": "{{password}}" } + +HTTP 200 +[Captures] +admin_token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 — Provision `ue_owner` (user envelope under test). +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/users +Authorization: Bearer {{admin_token}} +Content-Type: application/json +{ + "username": "ue_owner", + "password": "UeOwnerPwd1!", + "email": "ue_owner@example.com", + "role": "user" +} + +HTTP 201 + +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ "username": "ue_owner", "password": "UeOwnerPwd1!" } + +HTTP 200 +[Captures] +owner_token: jsonpath "$.access_token" +owner_user_id: jsonpath "$.user.id" + + +# ───────────────────────────────────────────────────────────── +# Step 3 — Fetch the user's default Personal drive root folder. +# `GET /api/folders` returns root folders for the +# caller; for a fresh user that's a single entry — the +# Personal drive's root provisioned by +# `PersonalDriveLifecycleHook`. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/folders +Authorization: Bearer {{owner_token}} + +HTTP 200 +[Captures] +personal_root_id: jsonpath "$[0].id" +[Asserts] +jsonpath "$" count == 1 + + +# ───────────────────────────────────────────────────────────── +# Step 4 — Baseline. Fresh user's envelope is zero. +# ───────────────────────────────────────────────────────────── +GET {{base_url}}/api/auth/me +Authorization: Bearer {{owner_token}} + +HTTP 200 +[Asserts] +jsonpath "$.storage_used_bytes" == 0 + + +# ───────────────────────────────────────────────────────────── +# Step 5 — Admin creates a shared drive with `ue_owner` as +# direct user-Owner. No per-drive quota (NULL = unlim). +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/drives +Authorization: Bearer {{admin_token}} +Content-Type: application/json +{ + "kind": "shared", + "name": "ue-shared", + "owner": { "type": "user", "id": "{{owner_user_id}}" } +} + +HTTP 201 +[Captures] +shared_drive_id: jsonpath "$.id" +shared_root_id: jsonpath "$.root_folder_id" + + +# ───────────────────────────────────────────────────────────── +# Step 6 — Case 2: upload hello.txt (32 B) to the SHARED drive. +# The drive's `used_bytes` will move; the user envelope +# must NOT. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{owner_token}} +[MultipartFormData] +folder_id: {{shared_root_id}} +file: file,fixtures/hello.txt; text/plain + +HTTP 201 + + +# Wait for the drive-side fire-and-forget delta to settle. +# Acts as the synchronisation point: by the time `drives.used_bytes` +# reflects the upload, the sibling user-side delta task spawned in +# the same call has had its chance to run too. +GET {{base_url}}/api/drives +Authorization: Bearer {{owner_token}} +[Options] +retry: 10 +retry-interval: 200ms + +HTTP 200 +[Asserts] +jsonpath "$[?(@.id=='{{shared_drive_id}}')].used_bytes" == 32 + + +# Force the user-side sweep to run, authoritative ground-truth. +# If the delta path incorrectly fired the user counter, the sweep +# would still correct it back to 0 (the new SQL excludes shared +# drives) — this also validates the sweep formula. +POST {{base_url}}/api/admin/internal/trigger-sweep +Authorization: Bearer {{admin_token}} + +HTTP 200 + + +# Envelope untouched by the shared upload. Both delta path and +# sweep path agree on `0` for a user with no personal-drive +# content. +GET {{base_url}}/api/auth/me +Authorization: Bearer {{owner_token}} + +HTTP 200 +[Asserts] +jsonpath "$.storage_used_bytes" == 0 + + +# ───────────────────────────────────────────────────────────── +# Step 7 — Case 3: upload hello.txt (32 B) to the user's own +# default Personal drive. The envelope MUST move now. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/files/upload +Authorization: Bearer {{owner_token}} +[MultipartFormData] +folder_id: {{personal_root_id}} +file: file,fixtures/hello.txt; text/plain + +HTTP 201 +[Captures] +personal_file_id: jsonpath "$.id" + + +# Retry until the user-side delta lands. If the conditional-fire +# logic is broken (delta never fires for personal), retries time +# out at `0` and the test fails — this is the regression catch. +GET {{base_url}}/api/auth/me +Authorization: Bearer {{owner_token}} +[Options] +retry: 10 +retry-interval: 200ms + +HTTP 200 +[Asserts] +jsonpath "$.storage_used_bytes" == 32 + + +# Confirm the sweep agrees with the delta — both code paths must +# give the same number. +POST {{base_url}}/api/admin/internal/trigger-sweep +Authorization: Bearer {{admin_token}} + +HTTP 200 + +GET {{base_url}}/api/auth/me +Authorization: Bearer {{owner_token}} + +HTTP 200 +[Asserts] +jsonpath "$.storage_used_bytes" == 32 + + +# ───────────────────────────────────────────────────────────── +# Step 8 — Case 4: trash + empty the personal file, then sweep. +# Per-drive (and per-user) counters are NOT decremented +# on delete (same design as the per-drive quota model); +# the sweep is the correctness backstop. Asserts it +# actually closes the drift back to 0. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/files/{{personal_file_id}} +Authorization: Bearer {{owner_token}} + +HTTP 204 + +DELETE {{base_url}}/api/trash/empty +Authorization: Bearer {{owner_token}} + +HTTP 200 + +POST {{base_url}}/api/admin/internal/trigger-sweep +Authorization: Bearer {{admin_token}} + +HTTP 200 + +GET {{base_url}}/api/auth/me +Authorization: Bearer {{owner_token}} + +HTTP 200 +[Asserts] +jsonpath "$.storage_used_bytes" == 0 + + +# ───────────────────────────────────────────────────────────── +# Step 9 — Cleanup. Deleting `ue_owner` cascades through +# `default_for_user` (default Personal drive + root +# folder + files) and removes the `role_grants` rows +# tying them to the shared drive. The shared drive +# itself is owned by admin (the creator) and gets +# drained by `storage_cleanup_check.sh` later. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/admin/users/{{owner_user_id}} +Authorization: Bearer {{admin_token}} + +HTTP 200