perf: round 24 — download_zip per-item authz+metadata N+1 → batch (validated authorization pass)
The ROUND23-deferred download_zip N+1, given its own validated pass. The individually-selected files were authorized + fetched one at a time via get_file_with_perms (require + get = 2 serial round-trips/file) before any streaming — a 200-file selection was 400 serial round-trips. AFTER routes the whole multi-select through the new FileRetrievalService::get_files_by_ids_with_perms: one check_files_read_batch (the PgAclEngine resolves every file's drive in ONE query and primes the resource->drive cache) + one get_files_by_ids. 2N round-trips -> 2. Authorization is unchanged and still enforced BEFORE any ZIP entry is written: - add_file_entry_streamed writes the entry header (the filename) before it opens the authorized stream, so the pre-filter is load-bearing — a denied file must never reach it or its name leaks into the archive. AFTER a denied/missing id is absent from the authorized map and is skipped in the same input order, exactly as the old loop skipped a denied get_file_with_perms; it never reaches the entry write. The authz moved from a per-file require to one batch check EARLIER in the same function, not into or after the stream. - The stream open keeps its own per-file Read check (now a primed-cache hit) + Recents recording; check_files_read_batch is documented + gated as identical to looping require. Because the change is authorization-sensitive, the gate is the security property itself. bench_round24_zip_authz drives the real PgAclEngine over a seeded, interleaved mix of owned (granted drive) + denied (other drive) + missing ids and asserts: the batch inclusion set AND input order are identical to the per-file require loop; the included set is exactly the caller's owned files; no denied or missing id is ever included (the authz-regression tripwire); and the batch fetch returns exactly the owned files. Latency (cold, 600-item 1/3-owned selection): 559 -> 267 ms (2.10x; the realistic all-owned selection is O(1) -> a larger win). See benches/ROUND24.md. The folder selections are left as-is (root counts are small and there is no check_folders_read_batch primitive to batch through). Verified: cargo clippy --features bench --all-targets -D warnings clean, cargo fmt --all --check clean, cargo test --lib --features bench = 529 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKyQ4AnYtgp1JtjzweyMeo
This commit is contained in:
+15
@@ -354,6 +354,21 @@ name = "bench_micro_allocs"
|
|||||||
path = "examples/bench_micro_allocs.rs"
|
path = "examples/bench_micro_allocs.rs"
|
||||||
required-features = ["bench"]
|
required-features = ["bench"]
|
||||||
|
|
||||||
|
# Round-24 battery ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
# Round-24 download_zip authz+metadata N+1 → batch, VALIDATED. The per-file
|
||||||
|
# get_file_with_perms (require + get, 2 round-trips/file) becomes one
|
||||||
|
# check_files_read_batch + one get_files_by_ids. Because the change is
|
||||||
|
# authorization-sensitive, the gate is the security property: the batch check
|
||||||
|
# must make the identical per-file inclusion decision (same set AND input order)
|
||||||
|
# as the shipped-before require loop, over owned + denied + missing ids, and
|
||||||
|
# never include a denied/missing file. Drives the real PgAclEngine. Needs the
|
||||||
|
# dev Postgres up (reads DATABASE_URL from .env).
|
||||||
|
[[example]]
|
||||||
|
name = "bench_round24_zip_authz"
|
||||||
|
path = "examples/bench_round24_zip_authz.rs"
|
||||||
|
required-features = ["bench"]
|
||||||
|
|
||||||
# Round-23 battery ────────────────────────────────────────────────────────────
|
# Round-23 battery ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
# Round-23 CPU/alloc micro-pack (no Postgres) — deterministic alloc gates for
|
# Round-23 CPU/alloc micro-pack (no Postgres) — deterministic alloc gates for
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Round 24 — `download_zip` per-item authz+metadata N+1 → batch (validated authorization pass)
|
||||||
|
|
||||||
|
This is the ROUND23 "not shipped" item #2, given the dedicated validated pass it
|
||||||
|
needed. Unlike the other rounds it is **authorization-sensitive**, so the gate is
|
||||||
|
not an allocation count or a latency floor — it is the **security property
|
||||||
|
itself**: the batched authorization must make the *identical* per-file inclusion
|
||||||
|
decision as the shipped-before per-file `require` loop, and must never let a
|
||||||
|
denied or missing file into the archive.
|
||||||
|
|
||||||
|
Reproduce (needs the dev Postgres up; reads `DATABASE_URL` from `.env`):
|
||||||
|
|
||||||
|
```
|
||||||
|
cargo run --release --features bench --example bench_round24_zip_authz
|
||||||
|
```
|
||||||
|
|
||||||
|
## The change
|
||||||
|
|
||||||
|
`BatchOperations::download_zip` streamed a client's multi-selection into a ZIP.
|
||||||
|
For the **individually-selected files** it looped, per file:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
for file_id in &file_ids {
|
||||||
|
match self.file_retrieval.get_file_with_perms(file_id, user_id).await { // require + get = 2 round-trips
|
||||||
|
Ok(file_dto) => { self.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, &file_dto.mime_type, Some(user_id)).await … }
|
||||||
|
Err(_) => { /* skip + log */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`get_file_with_perms` is `require_file` (a `Read` authz round-trip) **plus**
|
||||||
|
`get_file` (a metadata round-trip) — so a selection of N files is **2N serial
|
||||||
|
round-trips** before a single byte is streamed. AFTER routes the whole selection
|
||||||
|
through one new service method:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let authorized = self.file_retrieval
|
||||||
|
.get_files_by_ids_with_perms(&file_ids, user_id).await?; // 1 batch check + 1 batch get
|
||||||
|
let by_id: HashMap<Uuid, FileDto> = authorized.into_iter()
|
||||||
|
.filter_map(|f| Uuid::parse_str(&f.id).ok().map(|u| (u, f))).collect();
|
||||||
|
for file_id in &file_ids { // same input order
|
||||||
|
let Some(file_dto) = Uuid::parse_str(file_id).ok().and_then(|u| by_id.get(&u)) else {
|
||||||
|
info!("Skipping file {file_id} (not accessible or missing)"); continue;
|
||||||
|
};
|
||||||
|
self.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, &file_dto.mime_type, Some(user_id)).await …
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`FileRetrievalService::get_files_by_ids_with_perms` authorizes every id in ONE
|
||||||
|
`AuthorizationEngine::check_files_read_batch` (the `PgAclEngine` override resolves
|
||||||
|
all files' drives in a single query and reuses the per-drive role cache) and
|
||||||
|
fetches only the authorized ids in ONE `get_files_by_ids`. **2N round-trips → 2.**
|
||||||
|
|
||||||
|
### Why this is authorization-safe (the part that made it a dedicated pass)
|
||||||
|
|
||||||
|
Three properties had to hold, all verified against the source before touching it:
|
||||||
|
|
||||||
|
1. **Authorization still happens before any ZIP entry is written.**
|
||||||
|
`add_file_entry_streamed` writes the entry header (the **filename**) *before*
|
||||||
|
it opens the authorized stream (`write_entry_stream` then
|
||||||
|
`get_file_stream_with_perms`). So the pre-filter is load-bearing: a denied
|
||||||
|
file must never reach `add_file_entry_streamed`, or its name would leak into
|
||||||
|
the archive (and leave a dangling entry). AFTER preserves this exactly — a
|
||||||
|
denied/missing id is absent from `by_id`, so it is `continue`-skipped and
|
||||||
|
never reaches the entry write. The authz simply moved from a per-file
|
||||||
|
`require` to one batch `check` **earlier** in the same function, not into or
|
||||||
|
after the stream.
|
||||||
|
|
||||||
|
2. **The per-file stream-open Read check + Recents recording are unchanged.**
|
||||||
|
`add_file_entry_streamed(Some(user_id))` still calls
|
||||||
|
`get_file_stream_with_perms`, which re-checks `Read` (now a primed-cache hit —
|
||||||
|
`check_files_read_batch` seeds the resource→drive cache) and records the
|
||||||
|
access in Recents. The old loop double-notified Recents (once in
|
||||||
|
`get_file_with_perms`, once in the stream open) and the throttle coalesced it
|
||||||
|
to one entry; AFTER notifies once (the stream open) — identical net effect.
|
||||||
|
|
||||||
|
3. **The batch authorization is identical to looping `require`.**
|
||||||
|
`check_files_read_batch` is documented and gated as "semantically identical to
|
||||||
|
looping `check`", and `require(Read)` succeeds iff `check(Read)` is true (a
|
||||||
|
denied `Read` is the 404 anti-enumeration shape). The §validation gate proves
|
||||||
|
this empirically on a mix of granted / denied / missing ids.
|
||||||
|
|
||||||
|
The **folder** selections (`get_folder_with_perms` per root, then the already-bulk
|
||||||
|
`add_folder_subtree_to_zip`) are left as-is: root counts are small and there is no
|
||||||
|
`check_folders_read_batch` primitive to batch through — see *Not shipped*.
|
||||||
|
|
||||||
|
## The validation
|
||||||
|
|
||||||
|
`bench_round24_zip_authz` drives the **real `PgAclEngine`** (the `fresh_engine`
|
||||||
|
shape from `bench_favorites_authz`) against a seeded fixture designed to exercise
|
||||||
|
every inclusion outcome:
|
||||||
|
|
||||||
|
- `owned` — N files on **drive A**, which the caller holds an `editor` grant on → **must be INCLUDED**
|
||||||
|
- `denied` — N files on **drive B**, which the caller has **no** grant on → **must be DENIED**
|
||||||
|
- `missing` — N random UUIDs that don't exist → **must be MISSING**
|
||||||
|
|
||||||
|
interleaved `owned, denied, missing, owned, …` so the **order** test is real. The
|
||||||
|
gate asserts, and `exit(1)`s on any failure:
|
||||||
|
|
||||||
|
- `before_included` (the per-file `require` filter, in input order) **==**
|
||||||
|
`after_included` (the batch `check_files_read_batch` filter, in input order) —
|
||||||
|
identical **set and order**;
|
||||||
|
- the included set is **exactly** the caller's `owned` files;
|
||||||
|
- **no** `denied` (other-drive) file is included — the authz-regression tripwire;
|
||||||
|
- **no** `missing` id is included;
|
||||||
|
- the batch `get_files_by_ids` of the authorized ids returns **exactly** the
|
||||||
|
`owned` files.
|
||||||
|
|
||||||
|
Latency (cold engine, empty caches — the first-download shape), `BENCH_FILES=200`
|
||||||
|
(600-item interleaved selection, ⅓ owned / ⅓ denied / ⅓ missing):
|
||||||
|
|
||||||
|
| arm | wall (600 items) | per file |
|
||||||
|
|---|---|---|
|
||||||
|
| per-file `require` loop | 559.47 ms | 932.45 µs |
|
||||||
|
| batch `check_files_read_batch` | 266.58 ms | 444.30 µs |
|
||||||
|
|
||||||
|
**2.10×** — and this is the *conservative* case: with ⅓ of the ids on a drive
|
||||||
|
the caller has no role on, `check_files_read_batch` still falls back to a per-file
|
||||||
|
`check_inner` for each un-readable-drive file. The realistic "download my own N
|
||||||
|
files" selection is **all** on drives the caller has a role on, where the batch
|
||||||
|
is genuinely O(1) (one drive-resolve query + cached role checks) against the
|
||||||
|
loop's 2N round-trips — a far larger win.
|
||||||
|
|
||||||
|
## Not shipped
|
||||||
|
|
||||||
|
- **Folder selections** (`download_zip`'s folder loop): `get_folder_with_perms`
|
||||||
|
per selected root. Root counts are typically 1–3, and there is no
|
||||||
|
`check_folders_read_batch` batch-authz primitive (only files have one), so
|
||||||
|
batching would still loop `check` per root — no round-trip win. Left as-is.
|
||||||
|
- **Dropping the stream-open re-check**: since the batch pre-check already
|
||||||
|
authorized (and primed the cache), `add_file_entry_streamed`'s
|
||||||
|
`get_file_stream_with_perms` re-check is now redundant (a cache hit). Replacing
|
||||||
|
it with the no-perms `get_file_stream` would save the cache lookups but would
|
||||||
|
also drop the Recents recording and the second authz barrier — not worth the
|
||||||
|
behavior change; kept as belt-and-suspenders.
|
||||||
|
|
||||||
|
## Environment / methodology
|
||||||
|
|
||||||
|
- Real `PgAclEngine` + `FileBlobReadRepository` against a local **PostgreSQL 16**
|
||||||
|
(schema from `migrations/`). The bench seeds its own two-drive fixture
|
||||||
|
(`bench_zipauthz_*` markers) and tears it down around the run.
|
||||||
|
- Built with `RUSTFLAGS="-C target-cpu=x86-64-v3"` (this session's host
|
||||||
|
intermittently `SIGILL`ed rustc under the repo's default `-C target-cpu=native`
|
||||||
|
AVX-512 after a host migration — see benches/ROUND23.md). Local build-flag
|
||||||
|
override only; the checked-in `.cargo/config.toml` is unchanged.
|
||||||
|
- The gate is the security equivalence (set + order + denied/missing exclusion),
|
||||||
|
not a perf threshold; the latency table is supporting evidence for the
|
||||||
|
round-trip collapse.
|
||||||
|
- Verified beyond the bench: `cargo clippy --features bench --all-targets
|
||||||
|
-D warnings` clean, `cargo fmt --all --check` clean, `cargo test --lib
|
||||||
|
--features bench` = 529 passed / 0 failed.
|
||||||
@@ -0,0 +1,387 @@
|
|||||||
|
//! Round-24 — `download_zip` per-item authz+metadata N+1 → batch, VALIDATED.
|
||||||
|
//!
|
||||||
|
//! `BatchOperations::download_zip` authorized + fetched each selected file with
|
||||||
|
//! a per-file `get_file_with_perms` (= `require_file` authz + `get_file`) — 2
|
||||||
|
//! serial round-trips per file, before any streaming. AFTER routes the whole
|
||||||
|
//! multi-select through `FileRetrievalService::get_files_by_ids_with_perms`,
|
||||||
|
//! which authorizes every id in ONE `check_files_read_batch` and fetches the
|
||||||
|
//! authorized ids in ONE `get_files_by_ids` (2 round-trips total). The
|
||||||
|
//! subsequent `add_file_entry_streamed` keeps its own per-file stream-open Read
|
||||||
|
//! check + Recents recording (now a primed-cache hit), so authorization still
|
||||||
|
//! happens BEFORE any ZIP entry is written — a denied file never leaks its name.
|
||||||
|
//!
|
||||||
|
//! Because this change is authorization-sensitive, the gate is the security
|
||||||
|
//! property itself: the batch `check_files_read_batch` must make the EXACT same
|
||||||
|
//! per-file inclusion decision as the shipped-before per-file `require` loop —
|
||||||
|
//! same **set** AND same **input order** — over a mix of
|
||||||
|
//! • files on a drive the caller is granted `editor` on (INCLUDED)
|
||||||
|
//! • files on a drive the caller has NO grant on (DENIED)
|
||||||
|
//! • ids that don't exist at all (MISSING)
|
||||||
|
//! and the batch fetch must return exactly the authorized, existing files.
|
||||||
|
//! Any divergence `std::process::exit(1)`s.
|
||||||
|
//!
|
||||||
|
//! Drives the REAL `PgAclEngine` + `FileBlobReadRepository` (the fresh_engine
|
||||||
|
//! shape from bench_favorites_authz).
|
||||||
|
//!
|
||||||
|
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||||
|
//! cargo run --release --features bench --example bench_round24_zip_authz
|
||||||
|
//! Tunables (env): BENCH_FILES (200), BENCH_POOL (20).
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::env;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use oxicloud::application::ports::authorization_ports::AuthorizationEngine;
|
||||||
|
use oxicloud::domain::services::authorization::{Permission, Resource, Subject};
|
||||||
|
use oxicloud::infrastructure::repositories::pg::{
|
||||||
|
FileBlobReadRepository, FolderDbRepository, SubjectGroupPgRepository,
|
||||||
|
};
|
||||||
|
use oxicloud::infrastructure::services::dedup_service::DedupService;
|
||||||
|
use oxicloud::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||||
|
use oxicloud::infrastructure::services::pg_acl_engine::PgAclEngine;
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use sqlx::postgres::PgPoolOptions;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||||
|
env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|v| v.parse().ok())
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Seeded {
|
||||||
|
caller: Uuid,
|
||||||
|
other: Uuid,
|
||||||
|
drive_a: Uuid,
|
||||||
|
drive_b: Uuid,
|
||||||
|
root_a: Uuid,
|
||||||
|
root_b: Uuid,
|
||||||
|
blob_hash: String,
|
||||||
|
/// The caller's accessible files (drive A) — the expected INCLUDED set.
|
||||||
|
owned: Vec<Uuid>,
|
||||||
|
/// Files on drive B (no grant to caller) — expected DENIED.
|
||||||
|
denied: Vec<Uuid>,
|
||||||
|
/// Non-existent ids — expected MISSING.
|
||||||
|
missing: Vec<Uuid>,
|
||||||
|
/// The full selection, interleaved owned/denied/missing (order matters).
|
||||||
|
selection: Vec<Uuid>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn seed(pool: &PgPool, n_files: usize) -> Seeded {
|
||||||
|
let mut tx = pool.begin().await.expect("begin");
|
||||||
|
let caller: Uuid = sqlx::query_scalar(
|
||||||
|
"INSERT INTO auth.users (username, email, role)
|
||||||
|
VALUES ('bench_zipauthz_a', 'bench_zipauthz_a@bench.invalid', 'user') RETURNING id",
|
||||||
|
)
|
||||||
|
.fetch_one(&mut *tx)
|
||||||
|
.await
|
||||||
|
.expect("seed caller");
|
||||||
|
let other: Uuid = sqlx::query_scalar(
|
||||||
|
"INSERT INTO auth.users (username, email, role)
|
||||||
|
VALUES ('bench_zipauthz_b', 'bench_zipauthz_b@bench.invalid', 'user') RETURNING id",
|
||||||
|
)
|
||||||
|
.fetch_one(&mut *tx)
|
||||||
|
.await
|
||||||
|
.expect("seed other");
|
||||||
|
|
||||||
|
let blob_hash = "benchzipauthz00000000000000000000000000000000000000000000000b24".to_string();
|
||||||
|
sqlx::query("INSERT INTO storage.blobs (hash, size, ref_count) VALUES ($1, 1, 1)")
|
||||||
|
.bind(&blob_hash)
|
||||||
|
.execute(&mut *tx)
|
||||||
|
.await
|
||||||
|
.expect("seed blob");
|
||||||
|
|
||||||
|
// Two shared drives; `caller` is granted editor on A only, `other` on B.
|
||||||
|
async fn drive_with_grant(
|
||||||
|
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||||
|
label: &str,
|
||||||
|
grantee: Uuid,
|
||||||
|
) -> (Uuid, Uuid) {
|
||||||
|
let drive: Uuid =
|
||||||
|
sqlx::query_scalar("INSERT INTO storage.drives (kind) VALUES ('shared') RETURNING id")
|
||||||
|
.fetch_one(&mut **tx)
|
||||||
|
.await
|
||||||
|
.expect("seed drive");
|
||||||
|
let root: Uuid = sqlx::query_scalar(
|
||||||
|
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||||
|
VALUES ($1, $2, 'x', $3) RETURNING id",
|
||||||
|
)
|
||||||
|
.bind(format!("Bench {label}"))
|
||||||
|
.bind(format!("/Bench {label}"))
|
||||||
|
.bind(drive)
|
||||||
|
.fetch_one(&mut **tx)
|
||||||
|
.await
|
||||||
|
.expect("seed folder");
|
||||||
|
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||||
|
.bind(root)
|
||||||
|
.bind(drive)
|
||||||
|
.execute(&mut **tx)
|
||||||
|
.await
|
||||||
|
.expect("stamp root");
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO storage.role_grants
|
||||||
|
(subject_type, subject_id, resource_type, resource_id, role, granted_by)
|
||||||
|
VALUES ('user', $1, 'drive', $2, 'editor'::storage.grant_role, $1)",
|
||||||
|
)
|
||||||
|
.bind(grantee)
|
||||||
|
.bind(drive)
|
||||||
|
.execute(&mut **tx)
|
||||||
|
.await
|
||||||
|
.expect("seed grant");
|
||||||
|
(drive, root)
|
||||||
|
}
|
||||||
|
|
||||||
|
let (drive_a, root_a) = drive_with_grant(&mut tx, "A", caller).await;
|
||||||
|
let (drive_b, root_b) = drive_with_grant(&mut tx, "B", other).await;
|
||||||
|
|
||||||
|
let mut owned = Vec::with_capacity(n_files);
|
||||||
|
let mut denied = Vec::with_capacity(n_files);
|
||||||
|
for i in 0..n_files {
|
||||||
|
for (drive, root, sink) in [
|
||||||
|
(drive_a, root_a, &mut owned),
|
||||||
|
(drive_b, root_b, &mut denied),
|
||||||
|
] {
|
||||||
|
let id: Uuid = sqlx::query_scalar(
|
||||||
|
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||||
|
VALUES ($1, $2, $3, 1, 'text/plain', $4) RETURNING id",
|
||||||
|
)
|
||||||
|
.bind(format!("bench-{i:04}.txt"))
|
||||||
|
.bind(root)
|
||||||
|
.bind(&blob_hash)
|
||||||
|
.bind(drive)
|
||||||
|
.fetch_one(&mut *tx)
|
||||||
|
.await
|
||||||
|
.expect("seed file");
|
||||||
|
sink.push(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tx.commit().await.expect("commit");
|
||||||
|
|
||||||
|
let missing: Vec<Uuid> = (0..n_files).map(|_| Uuid::new_v4()).collect();
|
||||||
|
|
||||||
|
// Interleave owned / denied / missing so the order test is meaningful.
|
||||||
|
let mut selection = Vec::with_capacity(n_files * 3);
|
||||||
|
for i in 0..n_files {
|
||||||
|
selection.push(owned[i]);
|
||||||
|
selection.push(denied[i]);
|
||||||
|
selection.push(missing[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
Seeded {
|
||||||
|
caller,
|
||||||
|
other,
|
||||||
|
drive_a,
|
||||||
|
drive_b,
|
||||||
|
root_a,
|
||||||
|
root_b,
|
||||||
|
blob_hash,
|
||||||
|
owned,
|
||||||
|
denied,
|
||||||
|
missing,
|
||||||
|
selection,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn cleanup(pool: &PgPool, s: &Seeded) {
|
||||||
|
for d in [s.drive_a, s.drive_b] {
|
||||||
|
let _ = sqlx::query("DELETE FROM storage.role_grants WHERE resource_id = $1")
|
||||||
|
.bind(d)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
let _ = sqlx::query("DELETE FROM storage.files WHERE drive_id = $1")
|
||||||
|
.bind(d)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||||
|
.bind(d)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
for f in [s.root_a, s.root_b] {
|
||||||
|
let _ = sqlx::query("DELETE FROM storage.folders WHERE id = $1")
|
||||||
|
.bind(f)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
let _ = sqlx::query("DELETE FROM storage.blobs WHERE hash = $1")
|
||||||
|
.bind(&s.blob_hash)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
let _ = sqlx::query("DELETE FROM auth.users WHERE id IN ($1, $2)")
|
||||||
|
.bind(s.caller)
|
||||||
|
.bind(s.other)
|
||||||
|
.execute(pool)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fresh_engine(pool: &Arc<PgPool>) -> (Arc<PgAclEngine>, Arc<FileBlobReadRepository>) {
|
||||||
|
let folder_repo = Arc::new(FolderDbRepository::new(pool.clone()));
|
||||||
|
let backend = Arc::new(LocalBlobBackend::new(std::path::Path::new(
|
||||||
|
"/tmp/bench-zipauthz-blobs",
|
||||||
|
)));
|
||||||
|
let dedup = Arc::new(DedupService::new(backend, pool.clone(), pool.clone()));
|
||||||
|
let file_repo = Arc::new(FileBlobReadRepository::new(
|
||||||
|
pool.clone(),
|
||||||
|
dedup,
|
||||||
|
folder_repo.clone(),
|
||||||
|
));
|
||||||
|
let group_repo = Arc::new(SubjectGroupPgRepository::new(pool.clone()));
|
||||||
|
let engine = Arc::new(PgAclEngine::new(
|
||||||
|
pool.clone(),
|
||||||
|
folder_repo,
|
||||||
|
file_repo.clone(),
|
||||||
|
group_repo,
|
||||||
|
));
|
||||||
|
(engine, file_repo)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// BEFORE, verbatim: the per-file `require` filter, preserving input order.
|
||||||
|
async fn before_included(engine: &PgAclEngine, user: Uuid, sel: &[Uuid]) -> Vec<Uuid> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
for id in sel {
|
||||||
|
if engine
|
||||||
|
.require(Subject::User(user), Permission::Read, Resource::File(*id))
|
||||||
|
.await
|
||||||
|
.is_ok()
|
||||||
|
{
|
||||||
|
out.push(*id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// AFTER: one batch check, then re-associate in input order (the download_zip
|
||||||
|
/// re-association).
|
||||||
|
async fn after_included(engine: &PgAclEngine, user: Uuid, sel: &[Uuid]) -> Vec<Uuid> {
|
||||||
|
let allowed: HashSet<Uuid> = engine
|
||||||
|
.check_files_read_batch(Subject::User(user), sel)
|
||||||
|
.await
|
||||||
|
.expect("batch check");
|
||||||
|
sel.iter()
|
||||||
|
.copied()
|
||||||
|
.filter(|id| allowed.contains(id))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main(flavor = "multi_thread")]
|
||||||
|
async fn main() {
|
||||||
|
dotenvy::dotenv().ok();
|
||||||
|
let url = env::var("DATABASE_URL")
|
||||||
|
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||||
|
.expect("set DATABASE_URL — the dev Postgres URL");
|
||||||
|
let n_files: usize = env_or("BENCH_FILES", 200);
|
||||||
|
let pool_size: u32 = env_or("BENCH_POOL", 20);
|
||||||
|
|
||||||
|
let pool = Arc::new(
|
||||||
|
PgPoolOptions::new()
|
||||||
|
.max_connections(pool_size)
|
||||||
|
.min_connections(pool_size)
|
||||||
|
.acquire_timeout(Duration::from_secs(10))
|
||||||
|
.connect(&url)
|
||||||
|
.await
|
||||||
|
.expect("connect Postgres"),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Clear any prior fixtures, then seed.
|
||||||
|
let _ = sqlx::query(
|
||||||
|
"DELETE FROM auth.users WHERE email IN ('bench_zipauthz_a@bench.invalid','bench_zipauthz_b@bench.invalid')",
|
||||||
|
)
|
||||||
|
.execute(pool.as_ref())
|
||||||
|
.await;
|
||||||
|
let seeded = seed(&pool, n_files).await;
|
||||||
|
|
||||||
|
// ── Equivalence gate (fresh engines so neither arm rides the other's cache) ──
|
||||||
|
let (eng_before, _) = fresh_engine(&pool);
|
||||||
|
let (eng_after, file_repo) = fresh_engine(&pool);
|
||||||
|
let before = before_included(&eng_before, seeded.caller, &seeded.selection).await;
|
||||||
|
let after = after_included(&eng_after, seeded.caller, &seeded.selection).await;
|
||||||
|
|
||||||
|
let owned_set: HashSet<Uuid> = seeded.owned.iter().copied().collect();
|
||||||
|
let denied_set: HashSet<Uuid> = seeded.denied.iter().copied().collect();
|
||||||
|
let missing_set: HashSet<Uuid> = seeded.missing.iter().copied().collect();
|
||||||
|
|
||||||
|
let mut fail = false;
|
||||||
|
if before != after {
|
||||||
|
eprintln!("GATE FAIL: batch inclusion set/order != per-file require loop");
|
||||||
|
fail = true;
|
||||||
|
}
|
||||||
|
// The included set must be EXACTLY the caller's owned files, in input order.
|
||||||
|
let expected: Vec<Uuid> = seeded
|
||||||
|
.selection
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.filter(|id| owned_set.contains(id))
|
||||||
|
.collect();
|
||||||
|
if after != expected {
|
||||||
|
eprintln!("GATE FAIL: included set is not exactly the caller's owned files (in order)");
|
||||||
|
fail = true;
|
||||||
|
}
|
||||||
|
if after.iter().any(|id| denied_set.contains(id)) {
|
||||||
|
eprintln!("GATE FAIL: a DENIED (other-drive) file was included — authz regression!");
|
||||||
|
fail = true;
|
||||||
|
}
|
||||||
|
if after.iter().any(|id| missing_set.contains(id)) {
|
||||||
|
eprintln!("GATE FAIL: a MISSING id was included");
|
||||||
|
fail = true;
|
||||||
|
}
|
||||||
|
// The batch fetch of the authorized ids must return exactly those files.
|
||||||
|
let allowed_ids: Vec<String> = after.iter().map(Uuid::to_string).collect();
|
||||||
|
let fetched = file_repo
|
||||||
|
.get_files_by_ids(&allowed_ids)
|
||||||
|
.await
|
||||||
|
.expect("batch fetch");
|
||||||
|
let fetched_ids: HashSet<Uuid> = fetched
|
||||||
|
.iter()
|
||||||
|
.filter_map(|f| Uuid::parse_str(f.id()).ok())
|
||||||
|
.collect();
|
||||||
|
if fetched_ids != owned_set {
|
||||||
|
eprintln!("GATE FAIL: batch fetch of authorized ids != owned files");
|
||||||
|
fail = true;
|
||||||
|
}
|
||||||
|
if fail {
|
||||||
|
cleanup(&pool, &seeded).await;
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("\n#################################################################");
|
||||||
|
println!("# download_zip authz+metadata: per-file require loop vs batch");
|
||||||
|
println!(
|
||||||
|
"# selection = {n} owned + {n} denied + {n} missing (interleaved)",
|
||||||
|
n = n_files
|
||||||
|
);
|
||||||
|
println!("# gate OK: identical inclusion set+order; denied+missing excluded;");
|
||||||
|
println!(
|
||||||
|
"# batch fetch returns exactly the {} owned files.",
|
||||||
|
seeded.owned.len()
|
||||||
|
);
|
||||||
|
println!("#################################################################\n");
|
||||||
|
println!("| {:<26} | {:>10} | {:>12} |", "arm", "wall ms", "µs/file");
|
||||||
|
|
||||||
|
// Latency: cold engine each run (empty caches — the first-download shape).
|
||||||
|
let total = seeded.selection.len();
|
||||||
|
for (label, batch) in [
|
||||||
|
("per-file require loop", false),
|
||||||
|
("batch check_files_read", true),
|
||||||
|
] {
|
||||||
|
let (engine, _) = fresh_engine(&pool);
|
||||||
|
let t = Instant::now();
|
||||||
|
let got = if batch {
|
||||||
|
after_included(&engine, seeded.caller, &seeded.selection).await
|
||||||
|
} else {
|
||||||
|
before_included(&engine, seeded.caller, &seeded.selection).await
|
||||||
|
};
|
||||||
|
let el = t.elapsed();
|
||||||
|
assert_eq!(got.len(), seeded.owned.len(), "arm {label} inclusion count");
|
||||||
|
println!(
|
||||||
|
"| {:<26} | {:>10.2} | {:>12.2} |",
|
||||||
|
label,
|
||||||
|
el.as_secs_f64() * 1e3,
|
||||||
|
el.as_secs_f64() * 1e6 / total as f64
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup(&pool, &seeded).await;
|
||||||
|
println!("\nAll Round-24 authz-equivalence gates passed.");
|
||||||
|
}
|
||||||
@@ -726,31 +726,46 @@ impl BatchOperationService {
|
|||||||
let mut items_added: usize = 0;
|
let mut items_added: usize = 0;
|
||||||
|
|
||||||
// ── Add individual files at the root of the ZIP ──────────────────
|
// ── Add individual files at the root of the ZIP ──────────────────
|
||||||
|
// Authorize + fetch metadata for the whole multi-select in 2 round-trips
|
||||||
|
// (one batch Read check + one batch get) instead of the per-file
|
||||||
|
// `get_file_with_perms` N+1 (2 round-trips/file). The batch check also
|
||||||
|
// primes the resource→drive cache, so `add_file_entry_streamed`'s
|
||||||
|
// per-file stream-open re-check lands on the cache. A denied / missing /
|
||||||
|
// unparseable id is absent from the map → skipped in the same input
|
||||||
|
// order, exactly as the old per-file loop skipped it. Authorization is
|
||||||
|
// UNCHANGED — still enforced (pre-check here + the stream open's own
|
||||||
|
// Read check + Recents recording) before any ZIP entry is written, so a
|
||||||
|
// denied file never leaks its name into the archive (benches/ROUND24.md).
|
||||||
|
let authorized = self
|
||||||
|
.file_retrieval
|
||||||
|
.get_files_by_ids_with_perms(&file_ids, user_id)
|
||||||
|
.await
|
||||||
|
.map_err(BatchOperationError::Domain)?;
|
||||||
|
let by_id: HashMap<Uuid, FileDto> = authorized
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|f| Uuid::parse_str(&f.id).ok().map(|u| (u, f)))
|
||||||
|
.collect();
|
||||||
for file_id in &file_ids {
|
for file_id in &file_ids {
|
||||||
|
let file_dto = match Uuid::parse_str(file_id).ok().and_then(|u| by_id.get(&u)) {
|
||||||
|
Some(f) => f,
|
||||||
|
None => {
|
||||||
|
info!("Skipping file {} (not accessible or missing)", file_id);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
};
|
||||||
match self
|
match self
|
||||||
.file_retrieval
|
.add_file_entry_streamed(
|
||||||
.get_file_with_perms(file_id, user_id)
|
&mut zip,
|
||||||
|
file_id,
|
||||||
|
&file_dto.name,
|
||||||
|
&file_dto.mime_type,
|
||||||
|
Some(user_id),
|
||||||
|
)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
Ok(file_dto) => {
|
Ok(_) => items_added += 1,
|
||||||
match self
|
|
||||||
.add_file_entry_streamed(
|
|
||||||
&mut zip,
|
|
||||||
file_id,
|
|
||||||
&file_dto.name,
|
|
||||||
&file_dto.mime_type,
|
|
||||||
Some(user_id),
|
|
||||||
)
|
|
||||||
.await
|
|
||||||
{
|
|
||||||
Ok(_) => items_added += 1,
|
|
||||||
Err(e) => {
|
|
||||||
info!("Could not add file {} to ZIP: {}", file_dto.name, e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
info!("Could not get file metadata {}: {}", file_id, e);
|
info!("Could not add file {} to ZIP: {}", file_dto.name, e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -287,6 +287,52 @@ impl FileRetrievalService {
|
|||||||
Ok(files.into_iter().map(FileDto::from).collect())
|
Ok(files.into_iter().map(FileDto::from).collect())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Batched, authorized multi-get for the ZIP-download multi-select — the
|
||||||
|
/// batch form of [`FileRetrievalUseCase::get_file_with_perms`] over an
|
||||||
|
/// explicit id list.
|
||||||
|
///
|
||||||
|
/// Authorizes `Read` on every id in ONE `check_files_read_batch`
|
||||||
|
/// round-trip (which resolves all drives in a single query AND primes the
|
||||||
|
/// resource→drive cache, so the per-file re-check the subsequent stream
|
||||||
|
/// open performs becomes a cache hit), then fetches only the authorized ids
|
||||||
|
/// in ONE `get_files_by_ids` query. Replaces `download_zip`'s per-file
|
||||||
|
/// `require_file` + `get_file` loop — 2 round-trips/file → 2 total.
|
||||||
|
///
|
||||||
|
/// Returns the authorized, existing files; a denied / missing / unparseable
|
||||||
|
/// id is simply **absent** from the result (the caller re-associates by id
|
||||||
|
/// and skips the rest, exactly as the per-file loop skipped a denied /
|
||||||
|
/// missing `get_file_with_perms`). Read-authorization is identical to the
|
||||||
|
/// per-file path (`check_files_read_batch` is documented and gated as
|
||||||
|
/// semantically identical to looping `require`). Recents recording is left
|
||||||
|
/// to the subsequent per-file stream open (`get_file_stream_with_perms`),
|
||||||
|
/// which records it (throttle-coalesced) — same net effect as the old
|
||||||
|
/// loop's `notify_file_accessed` + stream double-notify. Fail-closed if no
|
||||||
|
/// engine was injected, mirroring [`Self::require_file`].
|
||||||
|
pub async fn get_files_by_ids_with_perms(
|
||||||
|
&self,
|
||||||
|
ids: &[String],
|
||||||
|
caller_id: Uuid,
|
||||||
|
) -> Result<Vec<FileDto>, DomainError> {
|
||||||
|
let authz = self.authz.as_ref().ok_or_else(|| {
|
||||||
|
DomainError::internal_error("FileRetrieval", "Authorization engine unavailable")
|
||||||
|
})?;
|
||||||
|
// Unparseable ids can't be authorized (the per-file path 404s on them),
|
||||||
|
// so drop them here — they stay absent from the authorized set.
|
||||||
|
let uuids: Vec<Uuid> = ids.iter().filter_map(|s| Uuid::parse_str(s).ok()).collect();
|
||||||
|
if uuids.is_empty() {
|
||||||
|
return Ok(Vec::new());
|
||||||
|
}
|
||||||
|
let allowed = authz
|
||||||
|
.check_files_read_batch(Subject::User(caller_id), &uuids)
|
||||||
|
.await?;
|
||||||
|
if allowed.is_empty() {
|
||||||
|
return Ok(Vec::new());
|
||||||
|
}
|
||||||
|
let allowed_ids: Vec<String> = allowed.iter().map(Uuid::to_string).collect();
|
||||||
|
let files = self.file_read.get_files_by_ids(&allowed_ids).await?;
|
||||||
|
Ok(files.into_iter().map(FileDto::from).collect())
|
||||||
|
}
|
||||||
|
|
||||||
/// Range read for HTTP Range Requests, cache-aware.
|
/// Range read for HTTP Range Requests, cache-aware.
|
||||||
///
|
///
|
||||||
/// Media players and PDF viewers fetch these files *exclusively* through
|
/// Media players and PDF viewers fetch these files *exclusively* through
|
||||||
|
|||||||
Reference in New Issue
Block a user