cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(drive): personal drives have the user's quota in commun

Browse Source
This commit is contained in:
Edouard Vanbelle
2026-06-26 00:30:09 +02:00
parent 996ace8a1d
commit cfd783cbd3
8 changed files with 602 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plan/drive.md
+148 -54
View File
@@ -314,7 +314,9 @@ For reference, the equivalent (broken) one-CTE form looks like:
WITH new_drive AS (
INSERT INTO storage.drives
(kind, default_for_user, quota_bytes, policies)
VALUES ('personal', $user_id, $quota, '{}'::jsonb)
VALUES ('personal', $user_id, NULL, '{}'::jsonb) -- personal drives carry
-- NULL quota; the cap is
-- the user envelope, §7
RETURNING id
),
new_root AS (
@@ -366,10 +368,12 @@ The fix is the four-step transaction described above. Rust:
```rust
let mut tx = pool.begin().await?;
// Personal drives carry NULL quota_bytes — the cap is the user envelope
// (`auth.users.storage_quota_bytes`, §7), not the per-drive column.
let drive_id: Uuid = sqlx::query_scalar(
r#"INSERT INTO storage.drives (kind, default_for_user, quota_bytes)
VALUES ('personal', $1, $2) RETURNING id"#,
).bind(owner).bind(quota).fetch_one(&mut *tx).await?;
VALUES ('personal', $1, NULL) RETURNING id"#,
).bind(owner).fetch_one(&mut *tx).await?;
let folder_id: Uuid = sqlx::query_scalar(
r#"INSERT INTO storage.folders
@@ -450,7 +454,7 @@ that try to bypass it now hit a DB-level wall.
| Per-resource grant outward | yes (subject to drive policies) | yes | yes |
| Cross-drive move | yes (subject to `forbid_cross_drive_move`) | yes | yes |
| Kind conversion | no — always default-personal | yes → may be promoted to `kind='shared'` later (drops the single-user restriction, picks up members) | no |
| Change `quota_bytes` | **OxiCloud admin only** (not the drive owner — §7) | **OxiCloud admin only** | **OxiCloud admin only** |
| Change `quota_bytes` | N/A — `drives.quota_bytes` is NULL for personal drives. The envelope is `auth.users.storage_quota_bytes` (admin-only — §7) | N/A — same | **OxiCloud admin only** (not the drive owner — §7) |
### 4. Roles → permission bundles
@@ -493,7 +497,7 @@ against the same table.
| Event | Behaviour |
|---|---|
| New internal user registers | Auto-create a default personal drive (`kind='personal'`, `default_for_user=<new_user>`, `quota_bytes=<OXICLOUD_DEFAULT_QUOTA_BYTES>`) + its root folder (`name='Personal'`, `parent_id=NULL`, drive_id pinned) + the Owner role_grant (`role_grants(subject_type='user', subject_id=<user>, resource_type='drive', resource_id=<drive>, role='owner')`) — **all four writes in one CTE statement** (§3), atomic against server crash. |
| New internal user registers | Auto-create a default personal drive (`kind='personal'`, `default_for_user=<new_user>`, `quota_bytes=NULL` — the envelope lives on `auth.users.storage_quota_bytes`, see §7) + its root folder (`name='Personal'`, `parent_id=NULL`, drive_id pinned) + the Owner role_grant (`role_grants(subject_type='user', subject_id=<user>, resource_type='drive', resource_id=<drive>, role='owner')`) — **all four writes in one transaction** (§3), atomic against server crash. |
| External user invited (magic-link only) | **No personal drive created.** External users are grant-only recipients with no storage. |
| External user converts to internal (future flow) | Default personal drive created at conversion time. |
| User deleted | **Default** personal drive cascade-deletes via `ON DELETE CASCADE` on `default_for_user`. **Secondary** personal drives (`kind='personal' AND default_for_user IS NULL` and whose sole owner `role_grants` row points at the user) are deleted by an application-layer pass in the same transaction. `role_grants` rows referencing the deleted user are removed from all shared drives. If a removal would leave a shared drive with zero owners, deletion is refused — admin must transfer first. |
@@ -506,67 +510,157 @@ against the same table.
### 7. Quota model
The per-user `auth.users.storage_quota_bytes` field is **migrated to
the user's personal drive's `quota_bytes`** in one step, then the
column is deprecated (kept for one release cycle as a no-op, dropped
in a later migration).
Two ceilings, two different jobs:
After the cutover:
- Every drive owns its quota. Files inside a drive count against that
drive's `used_bytes` only.
- A user who collaborates in a 1 TB shared drive sees their personal
drive's quota as "their" quota; the shared drive's quota is owned
by the team.
- New drives default to a tenant-configured `OXICLOUD_DEFAULT_DRIVE_QUOTA_BYTES`
setting (separate env var, replacing today's per-user equivalent).
- **Per-user envelope** — `auth.users.storage_quota_bytes` stays
as the canonical "how much can this user store on this server."
It caps the sum of `used_bytes` across **every personal drive
the user owns** (default + any secondaries — see §2). Shared
drives never count against any user envelope.
- **Per-drive ceiling** — `drives.quota_bytes` is a per-drive cap
that applies **only to shared drives**. For personal drives
this column is `NULL` (unlimited at the drive layer); the
effective cap comes from the user envelope.
`used_bytes` is maintained incrementally on every file insert/delete
(plus a periodic reconciliation job to fix drift, similar to the
existing per-user accounting).
Why the asymmetry: a user's personal storage is a single budget
that the operator has agreed to provide; splitting it into
sub-quotas per personal drive is a sub-quota UX trap (users now
have to plan how to allocate "their" bytes between drives they
own). A shared drive's quota IS the team's resource budget, owned
by the operator, set independently.
#### Upload gate
Pre-upload, both checks run, in order:
1. **Drive cap** (`drives.quota_bytes`) — skipped when NULL.
Always skipped for personal drives by virtue of the NULL
convention; applies for shared drives.
2. **User envelope** (`auth.users.storage_quota_bytes`) — runs
only when the target drive is personal. The check sums
`used_bytes` across the caller's personal drives (or, fast
path while no secondaries exist on the UI, reads the cached
`auth.users.storage_used_bytes`).
For shared-drive uploads, only the per-drive check applies and the
user envelope is untouched — collaborating in a 1 TB shared drive
costs no personal bytes.
#### `used_bytes` accounting
Maintained incrementally on every file insert/delete in
`storage.drives.used_bytes`. The user-side cached counter
(`auth.users.storage_used_bytes`) is updated **only when the
target drive is personal** — the per-upload delta hook reads
`drives.kind` from the same query that already fetches
`drives.used_bytes` for the drive-cap check, so the hot path adds
zero round-trips.
A periodic reconciliation job rebuilds both counters from ground
truth:
```sql
-- Per-drive: unchanged from today.
UPDATE storage.drives SET used_bytes = (
SELECT COALESCE(SUM(size), 0) FROM storage.files
WHERE drive_id = d.id AND NOT is_trashed
) d;
-- Per-user: sum of personal-drive used_bytes owned by the user.
UPDATE auth.users u SET storage_used_bytes = COALESCE((
SELECT SUM(d.used_bytes)
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive' AND g.resource_id = d.id
AND g.role = 'owner'
AND g.subject_type = 'user' AND g.subject_id = u.id
WHERE d.kind = 'personal'
), 0);
```
Fast-path variant while only default personals are exposed:
```sql
UPDATE auth.users u SET storage_used_bytes = COALESCE((
SELECT used_bytes FROM storage.drives WHERE default_for_user = u.id
), 0);
```
Reconciliation runs on the maintenance pool — never blocks
uploads. Drift between deltas and the sweep is bounded by the
sweep interval (default 10 min).
#### Quota mutation is OxiCloud-admin only
Changing `drives.quota_bytes` is **not** in the drive `owner` role
bundle (§4). It requires the tenant-level OxiCloud admin role
(`auth.users.role = 'admin'`), checked at
`PATCH /api/admin/drives/{id}/quota` — the only callsite that
mutates the column. Drive owners can rename, edit policies, and
manage members; they cannot self-grant capacity.
Changing `drives.quota_bytes` (shared drives only) is **not** in
the drive `owner` role bundle (§4). It requires the tenant-level
OxiCloud admin role (`auth.users.role = 'admin'`), checked at
`PATCH /api/admin/drives/{id}/quota`. Drive owners can rename,
edit policies, and manage members; they cannot self-grant
capacity.
Changing `auth.users.storage_quota_bytes` (the personal envelope)
is likewise admin-only — same surface and audit pattern as today.
Why this seam matters:
- **Resource allocation is a tenant concern, not a drive
concern.** Storage bytes are a finite system resource the
operator pays for. The drive owner is empowered over the
drive's *use*; the admin is empowered over its *budget*. Same
separation that exists today between a user and the operator
who set `OXICLOUD_DEFAULT_QUOTA_BYTES`.
- **Privilege-escalation seam closed.** Without this carve-out,
any user with a personal drive (= every internal user) could
raise their own quota by virtue of being its sole owner —
trivially defeating the quota system.
- **Resource allocation is a tenant concern.** Storage bytes are
a finite system resource the operator pays for. The drive owner
is empowered over the drive's *use*; the admin is empowered
over its *budget*.
- **Privilege-escalation seam closed.** Without the per-drive
carve-out, an Owner of a shared drive could raise its quota.
Without the per-user carve-out, any internal user could raise
their own envelope by virtue of owning their personal drive.
- **Shared-drive coherence.** A shared drive's quota is set by
the operator at provisioning; subsequent capacity requests go
through the admin, not the drive's group owners. Keeps the
capacity decision auditable and out of intra-team politics.
through the admin, not the drive's group owners.
The admin endpoint is the same surface the operator uses today to
change `auth.users.storage_quota_bytes`; D4 simply re-targets the
write at `storage.drives.quota_bytes`. Audit log emits
`drive.quota_changed` with `granted_by=<admin_user_id>` and the
old/new values, mirroring the existing user-quota change event.
Audit log emits `drive.quota_changed` (shared drives) and
`user.quota_changed` (envelope) with `granted_by=<admin_user_id>`
and the old/new values.
**Chunk dedup vs per-drive quota.** With the CDC chunk store landed
in v0.7.0 (see `delta_upload_service`, `upload_ingest`, instant
upload by hash), a single chunk can be referenced by files in
multiple drives. The accounting decision: **each drive counts the
file's logical size in full against its own `used_bytes`** — dedup
savings are server-side only and never visible in the per-drive
quota number. This matches the existing per-user blob-dedup model
and avoids the alternative "pro-rated quota" trap (which makes
quota math depend on cross-drive content and breaks the user's
mental model of "I have 1 TB free"). Reconciliation job sums file
sizes per drive, not chunk allocations.
#### Multiple personal drives — schema-ready, no public surface
The schema and service layer treat personal drives as "any
personal drive owned by a user counts against the envelope," so
secondary personal drives (`kind='personal' AND
default_for_user IS NULL`) just work the day they ship. Today
there is **no public API surface to create them** — the only
`POST /api/drives` flow creates shared drives, and personal-drive
provisioning happens at user registration via the lifecycle hook
(§6). The capability matrix (§3) keeps the secondary column for
the migration backfill path and for the future, but it is not
user-reachable.
When secondary personals are eventually exposed (e.g. a "Vault"
end-to-end-encrypted drive kind, or a "Work" silo with a stricter
policy bag), the quota model needs no change — the sum-of-personal
formula already accounts for them.
#### Chunk dedup vs per-drive quota
With the CDC chunk store landed in v0.7.0 (see
`delta_upload_service`, `upload_ingest`, instant upload by hash),
a single chunk can be referenced by files in multiple drives. The
accounting decision: **each drive counts the file's logical size
in full against its own `used_bytes`** — dedup savings are
server-side only and never visible in the per-drive quota number.
This matches the existing per-user blob-dedup model and avoids
the "pro-rated quota" trap (which makes quota math depend on
cross-drive content and breaks the user's mental model of "I have
1 TB free"). Reconciliation sums file sizes per drive, not chunk
allocations.
#### Migration
One-shot at deploy: NULL out `drives.quota_bytes` for every
`kind='personal'` row (D4 backfilled them from
`auth.users.storage_quota_bytes` for the original "every drive
owns its quota" plan). Then run the new reconciliation sweep once
to resync `auth.users.storage_used_bytes` to "sum of personal
drives" (excludes any shared-drive bytes the old delta path may
have charged to it). Both steps idempotent.
### 8. Policies (JSONB, extensible)
migrations/20260806000000_personal_drive_quota_to_user_envelope.sql
+44
View File
@@ -0,0 +1,44 @@
-- Switch personal-drive quota semantics from "every drive owns its quota"
-- to "user envelope on the SUM of personal-drive `used_bytes`".
-- See docs/plan/drive.md §7.
--
-- Two idempotent steps:
-- 1. NULL `drives.quota_bytes` for every `kind='personal'` row. After this
-- migration the column is meaningful only for shared drives; personal
-- drives' cap is `auth.users.storage_quota_bytes`.
-- 2. Resync `auth.users.storage_used_bytes` to the sum-of-personal-drives
-- formula. Prior deltas may have over-counted by including shared-drive
-- uploads in the user counter; this snaps every user back to the new
-- envelope. Same shape the periodic sweep uses going forward.
--
-- Both statements `IS DISTINCT FROM`-guarded so reruns are cheap no-ops on
-- already-migrated databases. The order — NULL first, then resync — doesn't
-- matter for correctness but follows the doc's narrative.
-- 1. Drop per-drive quotas for personal drives (no-op for already-NULL rows).
UPDATE storage.drives
SET quota_bytes = NULL
WHERE kind = 'personal'
AND quota_bytes IS NOT NULL;
-- 2. Resync user-side cached counter to the new sum-of-personal-drives
-- semantics. Mirrors `update_all_users_storage_usage` in
-- `storage_usage_service.rs`. External users excluded (no storage).
UPDATE auth.users u
SET storage_used_bytes = COALESCE(t.total, 0)
FROM auth.users u2
LEFT JOIN (
SELECT g.subject_id AS user_id,
SUM(d.used_bytes)::bigint AS total
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
AND g.role = 'owner'
AND g.subject_type = 'user'
WHERE d.kind = 'personal'
GROUP BY g.subject_id
) t ON t.user_id = u2.id
WHERE u.id = u2.id
AND NOT u2.is_external
AND u.storage_used_bytes IS DISTINCT FROM COALESCE(t.total, 0);
src/application/services/file_upload_service.rs
+33 -16
View File
@@ -15,7 +15,7 @@ use crate::infrastructure::repositories::pg::FileBlobWriteRepository;
use crate::infrastructure::services::dedup_service::DedupService;
use crate::infrastructure::services::file_content_cache::FileContentCache;
use crate::infrastructure::services::pg_acl_engine::PgAclEngine;
use tracing::{info, warn};
use tracing::{Instrument, info, warn};
/// Service for file upload operations.
///
@@ -326,21 +326,39 @@ impl FileUploadService {
};
let delta = file.size as i64;
// Per-user delta — unchanged from `b5b80549` / `fbbae541`.
if let Some(owner) = file
let owner = file
.owner_id
.as_deref()
.and_then(|s| Uuid::parse_str(s).ok())
{
.and_then(|s| Uuid::parse_str(s).ok());
let folder = file
.folder_id
.as_deref()
.and_then(|s| Uuid::parse_str(s).ok());
// Per-user delta — only when the target drive is `kind='personal'`.
// The user envelope (`auth.users.storage_quota_bytes`) caps the SUM
// of `used_bytes` across the user's personal drives; shared-drive
// uploads do NOT count against any user. See
// `docs/plan/drive.md` §7.
//
// The discrimination happens in one SQL statement via an EXISTS
// subquery on the folder's drive kind — no extra round-trip vs
// the unconditional delta. Without a folder id (root-level
// upload — folder service refuses these) the user-side delta is
// simply skipped; the sweep reconciles regardless.
if let (Some(owner), Some(folder)) = (owner, folder) {
let service_clone = Arc::clone(storage_service);
tokio::spawn(async move {
tokio::spawn(
async move {
if let Err(e) = service_clone
.add_user_storage_usage_delta(owner, delta)
.add_user_storage_usage_delta_if_personal(owner, folder, delta)
.await
{
warn!("Failed to bump storage usage for {owner}: {e}");
warn!("Failed to bump user storage for {owner} (folder {folder}): {e}");
}
});
}
.in_current_span(),
);
}
// Per-drive delta (D4) — same fire-and-forget shape, resolves
@@ -349,20 +367,19 @@ impl FileUploadService {
// quota check and the picker quota bar read; drift from
// deletes / trash is reconciled by the same sweep that handles
// user-side drift.
if let Some(folder) = file
.folder_id
.as_deref()
.and_then(|s| Uuid::parse_str(s).ok())
{
if let Some(folder) = folder {
let service_clone = Arc::clone(storage_service);
tokio::spawn(async move {
tokio::spawn(
async move {
if let Err(e) = service_clone
.add_drive_storage_usage_delta_by_folder(folder, delta)
.await
{
warn!("Failed to bump drive usage for folder {folder}: {e}");
}
});
}
.in_current_span(),
);
}
}
}
src/application/services/storage_usage_service.rs
+94 -16
View File
@@ -39,13 +39,22 @@ impl StorageUsageService {
/// (was three: user lookup + SUM + UPDATE). NOT called on the request
/// path — only by the per-upload background update and the sweep.
pub async fn update_user_storage_usage(&self, user_id: Uuid) -> Result<i64, DomainError> {
// User envelope = SUM of `drives.used_bytes` across personal
// drives owned by the user (see `docs/plan/drive.md` §7). Shared
// drives don't count. Ownership is canonical via `role_grants`.
let total_usage: Option<i64> = sqlx::query_scalar(
r#"
UPDATE auth.users u
SET storage_used_bytes = COALESCE((
SELECT SUM(f.size)::bigint
FROM storage.files f
WHERE f.user_id = u.id AND NOT f.is_trashed), 0)
SELECT SUM(d.used_bytes)::bigint
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
AND g.role = 'owner'
AND g.subject_type = 'user'
AND g.subject_id = u.id
WHERE d.kind = 'personal'), 0)
WHERE u.id = $1
RETURNING u.storage_used_bytes
"#,
@@ -77,9 +86,15 @@ impl StorageUsageService {
r#"
UPDATE auth.users u
SET storage_used_bytes = COALESCE((
SELECT SUM(f.size)::bigint
FROM storage.files f
WHERE f.user_id = u.id AND NOT f.is_trashed), 0)
SELECT SUM(d.used_bytes)::bigint
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
AND g.role = 'owner'
AND g.subject_type = 'user'
AND g.subject_id = u.id
WHERE d.kind = 'personal'), 0)
WHERE u.username = $1
RETURNING u.storage_used_bytes
"#,
@@ -128,6 +143,50 @@ impl StorageUsageService {
Ok(())
}
/// Conditional user-side delta: only fires when the target folder's
/// drive is `kind='personal'`. See `docs/plan/drive.md` §7.
///
/// The new quota model: `auth.users.storage_quota_bytes` is the cap on
/// the SUM of `used_bytes` across the user's personal drives. Shared
/// drives never count against any user envelope. The upload hot path
/// reads `drives.kind` from the same JOIN that already runs for the
/// drive cap check; firing this conditional delta instead of the
/// unconditional [`Self::add_user_storage_usage_delta`] keeps the
/// counter aligned with that envelope semantics. Idempotent + clamped
/// at zero, same as the unconditional sibling.
///
/// Implementation note: the EXISTS subquery is two indexed PK probes
/// (folder by id, drive by id) so the personal/shared discrimination
/// adds no real cost vs. the unconditional update.
pub async fn add_user_storage_usage_delta_if_personal(
&self,
user_id: Uuid,
folder_id: Uuid,
delta: i64,
) -> Result<(), DomainError> {
sqlx::query(
"UPDATE auth.users u
SET storage_used_bytes = GREATEST(0, u.storage_used_bytes + $2)
WHERE u.id = $1
AND EXISTS (
SELECT 1
FROM storage.folders f
JOIN storage.drives d ON d.id = f.drive_id
WHERE f.id = $3
AND d.kind = 'personal'
)",
)
.bind(user_id)
.bind(delta)
.bind(folder_id)
.execute(self.pool.as_ref())
.await
.map_err(|e| {
DomainError::internal_error("StorageUsage", format!("usage delta if personal: {e}"))
})?;
Ok(())
}
/// Incrementally adjust one drive's cached `storage.drives.used_bytes`
/// by `delta` bytes — same shape as
/// [`Self::add_user_storage_usage_delta`]: single statement, no
@@ -312,15 +371,17 @@ impl StorageUsageService {
loop {
ticker.tick().await;
debug!("Running scheduled storage-usage reconciliation");
if let Err(e) = service.update_all_users_storage_usage().await {
error!("Scheduled user storage-usage reconciliation failed: {}", e);
}
// Drive sweep runs alongside the user sweep — same
// cadence, same maintenance pool. Failure is logged
// but doesn't skip the next tick.
// Drive sweep runs FIRST: the user-side sweep below
// reads `drives.used_bytes` (the per-drive sum) to
// compute its own counter, so the drive counter must
// be honest first. Failure of one is logged but
// doesn't skip the other or the next tick.
if let Err(e) = service.update_all_drives_storage_usage().await {
error!("Scheduled drive storage-usage reconciliation failed: {}", e);
}
if let Err(e) = service.update_all_users_storage_usage().await {
error!("Scheduled user storage-usage reconciliation failed: {}", e);
}
}
});
}
@@ -358,16 +419,33 @@ impl StorageUsagePort for StorageUsageService {
async fn update_all_users_storage_usage(&self) -> Result<(), DomainError> {
debug!("Starting storage-usage reconciliation sweep");
// User envelope = SUM of `drives.used_bytes` across the user's
// personal drives. Shared drives don't count against any user
// (`docs/plan/drive.md` §7). The drive-side sweep runs FIRST
// (`start_reconciliation_job`) so `drives.used_bytes` is
// already honest by the time we read it here.
//
// Ownership lookup uses `role_grants` (canonical per §1) so
// both the user's default personal AND any secondary
// personals owned via Owner grants are summed. Secondaries
// aren't user-creatable today, but a backfill or admin path
// can produce them — covering that surface from day one.
let result = sqlx::query(
r#"
UPDATE auth.users u
SET storage_used_bytes = COALESCE(t.total, 0)
FROM auth.users u2
LEFT JOIN (
SELECT user_id, SUM(size)::bigint AS total
FROM storage.files
WHERE NOT is_trashed
GROUP BY user_id
SELECT g.subject_id AS user_id,
SUM(d.used_bytes)::bigint AS total
FROM storage.drives d
JOIN storage.role_grants g
ON g.resource_type = 'drive'
AND g.resource_id = d.id
AND g.role = 'owner'
AND g.subject_type = 'user'
WHERE d.kind = 'personal'
GROUP BY g.subject_id
) t ON t.user_id = u2.id
WHERE u.id = u2.id
AND NOT u2.is_external
src/interfaces/api/handlers/admin_handler.rs
+10 -4
View File
@@ -2072,15 +2072,21 @@ pub async fn internal_trigger_sweep(
.into_response();
}
};
if let Err(e) = svc.update_all_users_storage_usage().await {
return AppError::internal_error(format!("user sweep failed: {e}")).into_response();
}
// Order matches the periodic ticker (`start_reconciliation_job`):
// drive sweep first because the user sweep reads `drives.used_bytes`
// (sum-of-personal-drives — `docs/plan/drive.md` §7). Running them
// in the other order makes the user counter freeze on the previous
// tick's drive numbers — invisible in steady state but breaks any
// Hurl that trashes + sweeps within one call.
if let Err(e) = svc.update_all_drives_storage_usage().await {
return AppError::internal_error(format!("drive sweep failed: {e}")).into_response();
}
if let Err(e) = svc.update_all_users_storage_usage().await {
return AppError::internal_error(format!("user sweep failed: {e}")).into_response();
}
(
StatusCode::OK,
Json(serde_json::json!({ "ok": true, "ran": ["users", "drives"] })),
Json(serde_json::json!({ "ok": true, "ran": ["drives", "users"] })),
)
.into_response()
}
src/interfaces/api/handlers/auth_handler.rs
+9 -4
View File
@@ -471,11 +471,16 @@ pub async fn get_current_user(
// Storage usage is served from the cached `storage_used_bytes` column —
// it is NOT recomputed here. Recomputing on this hot endpoint meant an
// O(N) `SUM(size)` over all the user's files plus an `UPDATE` of
// `auth.users` on every single call (one of the most frequent endpoints).
// The cached value is kept current by the per-upload update and a periodic
// background reconciliation sweep
// O(N) `SUM(size)` plus an `UPDATE` of `auth.users` on every single call
// (one of the most frequent endpoints). The cached value is kept current
// by the per-upload update and a periodic background reconciliation sweep
// (see `StorageUsageService::start_reconciliation_job`).
//
// Semantics (`docs/plan/drive.md` §7): `storage_used_bytes` is the SUM
// of `used_bytes` across the user's personal drives only. Shared drives
// never count against this envelope — collaborating in a team drive
// costs no personal bytes. The matching cap is
// `storage_quota_bytes` (admin-only mutation).
let user = auth_service
.auth_application_service
.get_user_by_id(user_id)
tests/api/run.sh
+2 -1
View File
@@ -162,7 +162,8 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
"$API_DIR/nc_auth_failures.hurl" \
"$API_DIR/dedup_create.hurl" \
"$API_DIR/trash_per_drive.hurl" \
"$API_DIR/drive_quota.hurl"
"$API_DIR/drive_quota.hurl" \
"$API_DIR/user_envelope_quota.hurl"
#bash "$API_DIR/dedup_bulk_upload.sh"
tests/api/user_envelope_quota.hurl
+254
View File
@@ -0,0 +1,254 @@
# =============================================================
# OxiCloud – User envelope quota (sum of personal drives)
# =============================================================
# Run:
# hurl --variables-file tests/api/test.env --file-root tests \
# --test tests/api/user_envelope_quota.hurl
#
# The model under test (`docs/plan/drive.md` §7):
# `auth.users.storage_quota_bytes` caps the SUM of `used_bytes`
# across the user's PERSONAL drives only. Shared drives never
# count against any user envelope.
#
# Cases:
# 1. Baseline — fresh user: `/me.storage_used_bytes == 0`.
# 2. Shared-drive upload does NOT touch the envelope —
# `/me.storage_used_bytes` stays 0 after upload + sweep.
# 3. Personal-drive upload DOES bump the envelope —
# `/me.storage_used_bytes == file_size` after upload + sweep.
# 4. Sweep self-heals — after trashing the personal file and
# `trigger-sweep`, `/me.storage_used_bytes` returns to 0.
#
# `trigger-sweep` is the deterministic synchronisation point:
# it runs the drive-side sweep then the user-side sweep
# (`StorageUsageService::start_reconciliation_job`), so both
# cached counters are authoritative ground-truth by the time
# the assertion fires. Gated by
# `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
# (set in `tests/common/server.env`).
#
# Self-contained: provisions `ue_owner` so it can run alongside
# the rest of the suite.
# =============================================================
# ─────────────────────────────────────────────────────────────
# Step 1 — Admin login.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "{{username}}", "password": "{{password}}" }
HTTP 200
[Captures]
admin_token: jsonpath "$.access_token"
# ─────────────────────────────────────────────────────────────
# Step 2 — Provision `ue_owner` (user envelope under test).
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/admin/users
Authorization: Bearer {{admin_token}}
Content-Type: application/json
{
"username": "ue_owner",
"password": "UeOwnerPwd1!",
"email": "ue_owner@example.com",
"role": "user"
}
HTTP 201
POST {{base_url}}/api/auth/login
Content-Type: application/json
{ "username": "ue_owner", "password": "UeOwnerPwd1!" }
HTTP 200
[Captures]
owner_token: jsonpath "$.access_token"
owner_user_id: jsonpath "$.user.id"
# ─────────────────────────────────────────────────────────────
# Step 3 — Fetch the user's default Personal drive root folder.
# `GET /api/folders` returns root folders for the
# caller; for a fresh user that's a single entry — the
# Personal drive's root provisioned by
# `PersonalDriveLifecycleHook`.
# ─────────────────────────────────────────────────────────────
GET {{base_url}}/api/folders
Authorization: Bearer {{owner_token}}
HTTP 200
[Captures]
personal_root_id: jsonpath "$[0].id"
[Asserts]
jsonpath "$" count == 1
# ─────────────────────────────────────────────────────────────
# Step 4 — Baseline. Fresh user's envelope is zero.
# ─────────────────────────────────────────────────────────────
GET {{base_url}}/api/auth/me
Authorization: Bearer {{owner_token}}
HTTP 200
[Asserts]
jsonpath "$.storage_used_bytes" == 0
# ─────────────────────────────────────────────────────────────
# Step 5 — Admin creates a shared drive with `ue_owner` as
# direct user-Owner. No per-drive quota (NULL = unlim).
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/drives
Authorization: Bearer {{admin_token}}
Content-Type: application/json
{
"kind": "shared",
"name": "ue-shared",
"owner": { "type": "user", "id": "{{owner_user_id}}" }
}
HTTP 201
[Captures]
shared_drive_id: jsonpath "$.id"
shared_root_id: jsonpath "$.root_folder_id"
# ─────────────────────────────────────────────────────────────
# Step 6 — Case 2: upload hello.txt (32 B) to the SHARED drive.
# The drive's `used_bytes` will move; the user envelope
# must NOT.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/files/upload
Authorization: Bearer {{owner_token}}
[MultipartFormData]
folder_id: {{shared_root_id}}
file: file,fixtures/hello.txt; text/plain
HTTP 201
# Wait for the drive-side fire-and-forget delta to settle.
# Acts as the synchronisation point: by the time `drives.used_bytes`
# reflects the upload, the sibling user-side delta task spawned in
# the same call has had its chance to run too.
GET {{base_url}}/api/drives
Authorization: Bearer {{owner_token}}
[Options]
retry: 10
retry-interval: 200ms
HTTP 200
[Asserts]
jsonpath "$[?(@.id=='{{shared_drive_id}}')].used_bytes" == 32
# Force the user-side sweep to run, authoritative ground-truth.
# If the delta path incorrectly fired the user counter, the sweep
# would still correct it back to 0 (the new SQL excludes shared
# drives) — this also validates the sweep formula.
POST {{base_url}}/api/admin/internal/trigger-sweep
Authorization: Bearer {{admin_token}}
HTTP 200
# Envelope untouched by the shared upload. Both delta path and
# sweep path agree on `0` for a user with no personal-drive
# content.
GET {{base_url}}/api/auth/me
Authorization: Bearer {{owner_token}}
HTTP 200
[Asserts]
jsonpath "$.storage_used_bytes" == 0
# ─────────────────────────────────────────────────────────────
# Step 7 — Case 3: upload hello.txt (32 B) to the user's own
# default Personal drive. The envelope MUST move now.
# ─────────────────────────────────────────────────────────────
POST {{base_url}}/api/files/upload
Authorization: Bearer {{owner_token}}
[MultipartFormData]
folder_id: {{personal_root_id}}
file: file,fixtures/hello.txt; text/plain
HTTP 201
[Captures]
personal_file_id: jsonpath "$.id"
# Retry until the user-side delta lands. If the conditional-fire
# logic is broken (delta never fires for personal), retries time
# out at `0` and the test fails — this is the regression catch.
GET {{base_url}}/api/auth/me
Authorization: Bearer {{owner_token}}
[Options]
retry: 10
retry-interval: 200ms
HTTP 200
[Asserts]
jsonpath "$.storage_used_bytes" == 32
# Confirm the sweep agrees with the delta — both code paths must
# give the same number.
POST {{base_url}}/api/admin/internal/trigger-sweep
Authorization: Bearer {{admin_token}}
HTTP 200
GET {{base_url}}/api/auth/me
Authorization: Bearer {{owner_token}}
HTTP 200
[Asserts]
jsonpath "$.storage_used_bytes" == 32
# ─────────────────────────────────────────────────────────────
# Step 8 — Case 4: trash + empty the personal file, then sweep.
# Per-drive (and per-user) counters are NOT decremented
# on delete (same design as the per-drive quota model);
# the sweep is the correctness backstop. Asserts it
# actually closes the drift back to 0.
# ─────────────────────────────────────────────────────────────
DELETE {{base_url}}/api/files/{{personal_file_id}}
Authorization: Bearer {{owner_token}}
HTTP 204
DELETE {{base_url}}/api/trash/empty
Authorization: Bearer {{owner_token}}
HTTP 200
POST {{base_url}}/api/admin/internal/trigger-sweep
Authorization: Bearer {{admin_token}}
HTTP 200
GET {{base_url}}/api/auth/me
Authorization: Bearer {{owner_token}}
HTTP 200
[Asserts]
jsonpath "$.storage_used_bytes" == 0
# ─────────────────────────────────────────────────────────────
# Step 9 — Cleanup. Deleting `ue_owner` cascades through
# `default_for_user` (default Personal drive + root
# folder + files) and removes the `role_grants` rows
# tying them to the shared drive. The shared drive
# itself is owned by admin (the creator) and gets
# drained by `storage_cleanup_check.sh` later.
# ─────────────────────────────────────────────────────────────
DELETE {{base_url}}/api/admin/users/{{owner_user_id}}
Authorization: Bearer {{admin_token}}
HTTP 200