perf: round 24 — download_zip per-item authz+metadata N+1 → batch (validated authorization pass)
The ROUND23-deferred download_zip N+1, given its own validated pass. The individually-selected files were authorized + fetched one at a time via get_file_with_perms (require + get = 2 serial round-trips/file) before any streaming — a 200-file selection was 400 serial round-trips. AFTER routes the whole multi-select through the new FileRetrievalService::get_files_by_ids_with_perms: one check_files_read_batch (the PgAclEngine resolves every file's drive in ONE query and primes the resource->drive cache) + one get_files_by_ids. 2N round-trips -> 2. Authorization is unchanged and still enforced BEFORE any ZIP entry is written: - add_file_entry_streamed writes the entry header (the filename) before it opens the authorized stream, so the pre-filter is load-bearing — a denied file must never reach it or its name leaks into the archive. AFTER a denied/missing id is absent from the authorized map and is skipped in the same input order, exactly as the old loop skipped a denied get_file_with_perms; it never reaches the entry write. The authz moved from a per-file require to one batch check EARLIER in the same function, not into or after the stream. - The stream open keeps its own per-file Read check (now a primed-cache hit) + Recents recording; check_files_read_batch is documented + gated as identical to looping require. Because the change is authorization-sensitive, the gate is the security property itself. bench_round24_zip_authz drives the real PgAclEngine over a seeded, interleaved mix of owned (granted drive) + denied (other drive) + missing ids and asserts: the batch inclusion set AND input order are identical to the per-file require loop; the included set is exactly the caller's owned files; no denied or missing id is ever included (the authz-regression tripwire); and the batch fetch returns exactly the owned files. Latency (cold, 600-item 1/3-owned selection): 559 -> 267 ms (2.10x; the realistic all-owned selection is O(1) -> a larger win). See benches/ROUND24.md. The folder selections are left as-is (root counts are small and there is no check_folders_read_batch primitive to batch through). Verified: cargo clippy --features bench --all-targets -D warnings clean, cargo fmt --all --check clean, cargo test --lib --features bench = 529 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKyQ4AnYtgp1JtjzweyMeo
This commit is contained in:
+15
@@ -354,6 +354,21 @@ name = "bench_micro_allocs"
|
||||
path = "examples/bench_micro_allocs.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-24 battery ────────────────────────────────────────────────────────────
|
||||
|
||||
# Round-24 download_zip authz+metadata N+1 → batch, VALIDATED. The per-file
|
||||
# get_file_with_perms (require + get, 2 round-trips/file) becomes one
|
||||
# check_files_read_batch + one get_files_by_ids. Because the change is
|
||||
# authorization-sensitive, the gate is the security property: the batch check
|
||||
# must make the identical per-file inclusion decision (same set AND input order)
|
||||
# as the shipped-before require loop, over owned + denied + missing ids, and
|
||||
# never include a denied/missing file. Drives the real PgAclEngine. Needs the
|
||||
# dev Postgres up (reads DATABASE_URL from .env).
|
||||
[[example]]
|
||||
name = "bench_round24_zip_authz"
|
||||
path = "examples/bench_round24_zip_authz.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# Round-23 battery ────────────────────────────────────────────────────────────
|
||||
|
||||
# Round-23 CPU/alloc micro-pack (no Postgres) — deterministic alloc gates for
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# Round 24 — `download_zip` per-item authz+metadata N+1 → batch (validated authorization pass)
|
||||
|
||||
This is the ROUND23 "not shipped" item #2, given the dedicated validated pass it
|
||||
needed. Unlike the other rounds it is **authorization-sensitive**, so the gate is
|
||||
not an allocation count or a latency floor — it is the **security property
|
||||
itself**: the batched authorization must make the *identical* per-file inclusion
|
||||
decision as the shipped-before per-file `require` loop, and must never let a
|
||||
denied or missing file into the archive.
|
||||
|
||||
Reproduce (needs the dev Postgres up; reads `DATABASE_URL` from `.env`):
|
||||
|
||||
```
|
||||
cargo run --release --features bench --example bench_round24_zip_authz
|
||||
```
|
||||
|
||||
## The change
|
||||
|
||||
`BatchOperations::download_zip` streamed a client's multi-selection into a ZIP.
|
||||
For the **individually-selected files** it looped, per file:
|
||||
|
||||
```rust
|
||||
for file_id in &file_ids {
|
||||
match self.file_retrieval.get_file_with_perms(file_id, user_id).await { // require + get = 2 round-trips
|
||||
Ok(file_dto) => { self.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, &file_dto.mime_type, Some(user_id)).await … }
|
||||
Err(_) => { /* skip + log */ }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`get_file_with_perms` is `require_file` (a `Read` authz round-trip) **plus**
|
||||
`get_file` (a metadata round-trip) — so a selection of N files is **2N serial
|
||||
round-trips** before a single byte is streamed. AFTER routes the whole selection
|
||||
through one new service method:
|
||||
|
||||
```rust
|
||||
let authorized = self.file_retrieval
|
||||
.get_files_by_ids_with_perms(&file_ids, user_id).await?; // 1 batch check + 1 batch get
|
||||
let by_id: HashMap<Uuid, FileDto> = authorized.into_iter()
|
||||
.filter_map(|f| Uuid::parse_str(&f.id).ok().map(|u| (u, f))).collect();
|
||||
for file_id in &file_ids { // same input order
|
||||
let Some(file_dto) = Uuid::parse_str(file_id).ok().and_then(|u| by_id.get(&u)) else {
|
||||
info!("Skipping file {file_id} (not accessible or missing)"); continue;
|
||||
};
|
||||
self.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, &file_dto.mime_type, Some(user_id)).await …
|
||||
}
|
||||
```
|
||||
|
||||
`FileRetrievalService::get_files_by_ids_with_perms` authorizes every id in ONE
|
||||
`AuthorizationEngine::check_files_read_batch` (the `PgAclEngine` override resolves
|
||||
all files' drives in a single query and reuses the per-drive role cache) and
|
||||
fetches only the authorized ids in ONE `get_files_by_ids`. **2N round-trips → 2.**
|
||||
|
||||
### Why this is authorization-safe (the part that made it a dedicated pass)
|
||||
|
||||
Three properties had to hold, all verified against the source before touching it:
|
||||
|
||||
1. **Authorization still happens before any ZIP entry is written.**
|
||||
`add_file_entry_streamed` writes the entry header (the **filename**) *before*
|
||||
it opens the authorized stream (`write_entry_stream` then
|
||||
`get_file_stream_with_perms`). So the pre-filter is load-bearing: a denied
|
||||
file must never reach `add_file_entry_streamed`, or its name would leak into
|
||||
the archive (and leave a dangling entry). AFTER preserves this exactly — a
|
||||
denied/missing id is absent from `by_id`, so it is `continue`-skipped and
|
||||
never reaches the entry write. The authz simply moved from a per-file
|
||||
`require` to one batch `check` **earlier** in the same function, not into or
|
||||
after the stream.
|
||||
|
||||
2. **The per-file stream-open Read check + Recents recording are unchanged.**
|
||||
`add_file_entry_streamed(Some(user_id))` still calls
|
||||
`get_file_stream_with_perms`, which re-checks `Read` (now a primed-cache hit —
|
||||
`check_files_read_batch` seeds the resource→drive cache) and records the
|
||||
access in Recents. The old loop double-notified Recents (once in
|
||||
`get_file_with_perms`, once in the stream open) and the throttle coalesced it
|
||||
to one entry; AFTER notifies once (the stream open) — identical net effect.
|
||||
|
||||
3. **The batch authorization is identical to looping `require`.**
|
||||
`check_files_read_batch` is documented and gated as "semantically identical to
|
||||
looping `check`", and `require(Read)` succeeds iff `check(Read)` is true (a
|
||||
denied `Read` is the 404 anti-enumeration shape). The §validation gate proves
|
||||
this empirically on a mix of granted / denied / missing ids.
|
||||
|
||||
The **folder** selections (`get_folder_with_perms` per root, then the already-bulk
|
||||
`add_folder_subtree_to_zip`) are left as-is: root counts are small and there is no
|
||||
`check_folders_read_batch` primitive to batch through — see *Not shipped*.
|
||||
|
||||
## The validation
|
||||
|
||||
`bench_round24_zip_authz` drives the **real `PgAclEngine`** (the `fresh_engine`
|
||||
shape from `bench_favorites_authz`) against a seeded fixture designed to exercise
|
||||
every inclusion outcome:
|
||||
|
||||
- `owned` — N files on **drive A**, which the caller holds an `editor` grant on → **must be INCLUDED**
|
||||
- `denied` — N files on **drive B**, which the caller has **no** grant on → **must be DENIED**
|
||||
- `missing` — N random UUIDs that don't exist → **must be MISSING**
|
||||
|
||||
interleaved `owned, denied, missing, owned, …` so the **order** test is real. The
|
||||
gate asserts, and `exit(1)`s on any failure:
|
||||
|
||||
- `before_included` (the per-file `require` filter, in input order) **==**
|
||||
`after_included` (the batch `check_files_read_batch` filter, in input order) —
|
||||
identical **set and order**;
|
||||
- the included set is **exactly** the caller's `owned` files;
|
||||
- **no** `denied` (other-drive) file is included — the authz-regression tripwire;
|
||||
- **no** `missing` id is included;
|
||||
- the batch `get_files_by_ids` of the authorized ids returns **exactly** the
|
||||
`owned` files.
|
||||
|
||||
Latency (cold engine, empty caches — the first-download shape), `BENCH_FILES=200`
|
||||
(600-item interleaved selection, ⅓ owned / ⅓ denied / ⅓ missing):
|
||||
|
||||
| arm | wall (600 items) | per file |
|
||||
|---|---|---|
|
||||
| per-file `require` loop | 559.47 ms | 932.45 µs |
|
||||
| batch `check_files_read_batch` | 266.58 ms | 444.30 µs |
|
||||
|
||||
**2.10×** — and this is the *conservative* case: with ⅓ of the ids on a drive
|
||||
the caller has no role on, `check_files_read_batch` still falls back to a per-file
|
||||
`check_inner` for each un-readable-drive file. The realistic "download my own N
|
||||
files" selection is **all** on drives the caller has a role on, where the batch
|
||||
is genuinely O(1) (one drive-resolve query + cached role checks) against the
|
||||
loop's 2N round-trips — a far larger win.
|
||||
|
||||
## Not shipped
|
||||
|
||||
- **Folder selections** (`download_zip`'s folder loop): `get_folder_with_perms`
|
||||
per selected root. Root counts are typically 1–3, and there is no
|
||||
`check_folders_read_batch` batch-authz primitive (only files have one), so
|
||||
batching would still loop `check` per root — no round-trip win. Left as-is.
|
||||
- **Dropping the stream-open re-check**: since the batch pre-check already
|
||||
authorized (and primed the cache), `add_file_entry_streamed`'s
|
||||
`get_file_stream_with_perms` re-check is now redundant (a cache hit). Replacing
|
||||
it with the no-perms `get_file_stream` would save the cache lookups but would
|
||||
also drop the Recents recording and the second authz barrier — not worth the
|
||||
behavior change; kept as belt-and-suspenders.
|
||||
|
||||
## Environment / methodology
|
||||
|
||||
- Real `PgAclEngine` + `FileBlobReadRepository` against a local **PostgreSQL 16**
|
||||
(schema from `migrations/`). The bench seeds its own two-drive fixture
|
||||
(`bench_zipauthz_*` markers) and tears it down around the run.
|
||||
- Built with `RUSTFLAGS="-C target-cpu=x86-64-v3"` (this session's host
|
||||
intermittently `SIGILL`ed rustc under the repo's default `-C target-cpu=native`
|
||||
AVX-512 after a host migration — see benches/ROUND23.md). Local build-flag
|
||||
override only; the checked-in `.cargo/config.toml` is unchanged.
|
||||
- The gate is the security equivalence (set + order + denied/missing exclusion),
|
||||
not a perf threshold; the latency table is supporting evidence for the
|
||||
round-trip collapse.
|
||||
- Verified beyond the bench: `cargo clippy --features bench --all-targets
|
||||
-D warnings` clean, `cargo fmt --all --check` clean, `cargo test --lib
|
||||
--features bench` = 529 passed / 0 failed.
|
||||
@@ -0,0 +1,387 @@
|
||||
//! Round-24 — `download_zip` per-item authz+metadata N+1 → batch, VALIDATED.
|
||||
//!
|
||||
//! `BatchOperations::download_zip` authorized + fetched each selected file with
|
||||
//! a per-file `get_file_with_perms` (= `require_file` authz + `get_file`) — 2
|
||||
//! serial round-trips per file, before any streaming. AFTER routes the whole
|
||||
//! multi-select through `FileRetrievalService::get_files_by_ids_with_perms`,
|
||||
//! which authorizes every id in ONE `check_files_read_batch` and fetches the
|
||||
//! authorized ids in ONE `get_files_by_ids` (2 round-trips total). The
|
||||
//! subsequent `add_file_entry_streamed` keeps its own per-file stream-open Read
|
||||
//! check + Recents recording (now a primed-cache hit), so authorization still
|
||||
//! happens BEFORE any ZIP entry is written — a denied file never leaks its name.
|
||||
//!
|
||||
//! Because this change is authorization-sensitive, the gate is the security
|
||||
//! property itself: the batch `check_files_read_batch` must make the EXACT same
|
||||
//! per-file inclusion decision as the shipped-before per-file `require` loop —
|
||||
//! same **set** AND same **input order** — over a mix of
|
||||
//! • files on a drive the caller is granted `editor` on (INCLUDED)
|
||||
//! • files on a drive the caller has NO grant on (DENIED)
|
||||
//! • ids that don't exist at all (MISSING)
|
||||
//! and the batch fetch must return exactly the authorized, existing files.
|
||||
//! Any divergence `std::process::exit(1)`s.
|
||||
//!
|
||||
//! Drives the REAL `PgAclEngine` + `FileBlobReadRepository` (the fresh_engine
|
||||
//! shape from bench_favorites_authz).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_round24_zip_authz
|
||||
//! Tunables (env): BENCH_FILES (200), BENCH_POOL (20).
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use oxicloud::application::ports::authorization_ports::AuthorizationEngine;
|
||||
use oxicloud::domain::services::authorization::{Permission, Resource, Subject};
|
||||
use oxicloud::infrastructure::repositories::pg::{
|
||||
FileBlobReadRepository, FolderDbRepository, SubjectGroupPgRepository,
|
||||
};
|
||||
use oxicloud::infrastructure::services::dedup_service::DedupService;
|
||||
use oxicloud::infrastructure::services::local_blob_backend::LocalBlobBackend;
|
||||
use oxicloud::infrastructure::services::pg_acl_engine::PgAclEngine;
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Seeded {
|
||||
caller: Uuid,
|
||||
other: Uuid,
|
||||
drive_a: Uuid,
|
||||
drive_b: Uuid,
|
||||
root_a: Uuid,
|
||||
root_b: Uuid,
|
||||
blob_hash: String,
|
||||
/// The caller's accessible files (drive A) — the expected INCLUDED set.
|
||||
owned: Vec<Uuid>,
|
||||
/// Files on drive B (no grant to caller) — expected DENIED.
|
||||
denied: Vec<Uuid>,
|
||||
/// Non-existent ids — expected MISSING.
|
||||
missing: Vec<Uuid>,
|
||||
/// The full selection, interleaved owned/denied/missing (order matters).
|
||||
selection: Vec<Uuid>,
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, n_files: usize) -> Seeded {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let caller: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role)
|
||||
VALUES ('bench_zipauthz_a', 'bench_zipauthz_a@bench.invalid', 'user') RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed caller");
|
||||
let other: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role)
|
||||
VALUES ('bench_zipauthz_b', 'bench_zipauthz_b@bench.invalid', 'user') RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed other");
|
||||
|
||||
let blob_hash = "benchzipauthz00000000000000000000000000000000000000000000000b24".to_string();
|
||||
sqlx::query("INSERT INTO storage.blobs (hash, size, ref_count) VALUES ($1, 1, 1)")
|
||||
.bind(&blob_hash)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("seed blob");
|
||||
|
||||
// Two shared drives; `caller` is granted editor on A only, `other` on B.
|
||||
async fn drive_with_grant(
|
||||
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||
label: &str,
|
||||
grantee: Uuid,
|
||||
) -> (Uuid, Uuid) {
|
||||
let drive: Uuid =
|
||||
sqlx::query_scalar("INSERT INTO storage.drives (kind) VALUES ('shared') RETURNING id")
|
||||
.fetch_one(&mut **tx)
|
||||
.await
|
||||
.expect("seed drive");
|
||||
let root: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ($1, $2, 'x', $3) RETURNING id",
|
||||
)
|
||||
.bind(format!("Bench {label}"))
|
||||
.bind(format!("/Bench {label}"))
|
||||
.bind(drive)
|
||||
.fetch_one(&mut **tx)
|
||||
.await
|
||||
.expect("seed folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(root)
|
||||
.bind(drive)
|
||||
.execute(&mut **tx)
|
||||
.await
|
||||
.expect("stamp root");
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.role_grants
|
||||
(subject_type, subject_id, resource_type, resource_id, role, granted_by)
|
||||
VALUES ('user', $1, 'drive', $2, 'editor'::storage.grant_role, $1)",
|
||||
)
|
||||
.bind(grantee)
|
||||
.bind(drive)
|
||||
.execute(&mut **tx)
|
||||
.await
|
||||
.expect("seed grant");
|
||||
(drive, root)
|
||||
}
|
||||
|
||||
let (drive_a, root_a) = drive_with_grant(&mut tx, "A", caller).await;
|
||||
let (drive_b, root_b) = drive_with_grant(&mut tx, "B", other).await;
|
||||
|
||||
let mut owned = Vec::with_capacity(n_files);
|
||||
let mut denied = Vec::with_capacity(n_files);
|
||||
for i in 0..n_files {
|
||||
for (drive, root, sink) in [
|
||||
(drive_a, root_a, &mut owned),
|
||||
(drive_b, root_b, &mut denied),
|
||||
] {
|
||||
let id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
VALUES ($1, $2, $3, 1, 'text/plain', $4) RETURNING id",
|
||||
)
|
||||
.bind(format!("bench-{i:04}.txt"))
|
||||
.bind(root)
|
||||
.bind(&blob_hash)
|
||||
.bind(drive)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed file");
|
||||
sink.push(id);
|
||||
}
|
||||
}
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
let missing: Vec<Uuid> = (0..n_files).map(|_| Uuid::new_v4()).collect();
|
||||
|
||||
// Interleave owned / denied / missing so the order test is meaningful.
|
||||
let mut selection = Vec::with_capacity(n_files * 3);
|
||||
for i in 0..n_files {
|
||||
selection.push(owned[i]);
|
||||
selection.push(denied[i]);
|
||||
selection.push(missing[i]);
|
||||
}
|
||||
|
||||
Seeded {
|
||||
caller,
|
||||
other,
|
||||
drive_a,
|
||||
drive_b,
|
||||
root_a,
|
||||
root_b,
|
||||
blob_hash,
|
||||
owned,
|
||||
denied,
|
||||
missing,
|
||||
selection,
|
||||
}
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, s: &Seeded) {
|
||||
for d in [s.drive_a, s.drive_b] {
|
||||
let _ = sqlx::query("DELETE FROM storage.role_grants WHERE resource_id = $1")
|
||||
.bind(d)
|
||||
.execute(pool)
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM storage.files WHERE drive_id = $1")
|
||||
.bind(d)
|
||||
.execute(pool)
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(d)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
for f in [s.root_a, s.root_b] {
|
||||
let _ = sqlx::query("DELETE FROM storage.folders WHERE id = $1")
|
||||
.bind(f)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
let _ = sqlx::query("DELETE FROM storage.blobs WHERE hash = $1")
|
||||
.bind(&s.blob_hash)
|
||||
.execute(pool)
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id IN ($1, $2)")
|
||||
.bind(s.caller)
|
||||
.bind(s.other)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
fn fresh_engine(pool: &Arc<PgPool>) -> (Arc<PgAclEngine>, Arc<FileBlobReadRepository>) {
|
||||
let folder_repo = Arc::new(FolderDbRepository::new(pool.clone()));
|
||||
let backend = Arc::new(LocalBlobBackend::new(std::path::Path::new(
|
||||
"/tmp/bench-zipauthz-blobs",
|
||||
)));
|
||||
let dedup = Arc::new(DedupService::new(backend, pool.clone(), pool.clone()));
|
||||
let file_repo = Arc::new(FileBlobReadRepository::new(
|
||||
pool.clone(),
|
||||
dedup,
|
||||
folder_repo.clone(),
|
||||
));
|
||||
let group_repo = Arc::new(SubjectGroupPgRepository::new(pool.clone()));
|
||||
let engine = Arc::new(PgAclEngine::new(
|
||||
pool.clone(),
|
||||
folder_repo,
|
||||
file_repo.clone(),
|
||||
group_repo,
|
||||
));
|
||||
(engine, file_repo)
|
||||
}
|
||||
|
||||
/// BEFORE, verbatim: the per-file `require` filter, preserving input order.
|
||||
async fn before_included(engine: &PgAclEngine, user: Uuid, sel: &[Uuid]) -> Vec<Uuid> {
|
||||
let mut out = Vec::new();
|
||||
for id in sel {
|
||||
if engine
|
||||
.require(Subject::User(user), Permission::Read, Resource::File(*id))
|
||||
.await
|
||||
.is_ok()
|
||||
{
|
||||
out.push(*id);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// AFTER: one batch check, then re-associate in input order (the download_zip
|
||||
/// re-association).
|
||||
async fn after_included(engine: &PgAclEngine, user: Uuid, sel: &[Uuid]) -> Vec<Uuid> {
|
||||
let allowed: HashSet<Uuid> = engine
|
||||
.check_files_read_batch(Subject::User(user), sel)
|
||||
.await
|
||||
.expect("batch check");
|
||||
sel.iter()
|
||||
.copied()
|
||||
.filter(|id| allowed.contains(id))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL — the dev Postgres URL");
|
||||
let n_files: usize = env_or("BENCH_FILES", 200);
|
||||
let pool_size: u32 = env_or("BENCH_POOL", 20);
|
||||
|
||||
let pool = Arc::new(
|
||||
PgPoolOptions::new()
|
||||
.max_connections(pool_size)
|
||||
.min_connections(pool_size)
|
||||
.acquire_timeout(Duration::from_secs(10))
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect Postgres"),
|
||||
);
|
||||
|
||||
// Clear any prior fixtures, then seed.
|
||||
let _ = sqlx::query(
|
||||
"DELETE FROM auth.users WHERE email IN ('bench_zipauthz_a@bench.invalid','bench_zipauthz_b@bench.invalid')",
|
||||
)
|
||||
.execute(pool.as_ref())
|
||||
.await;
|
||||
let seeded = seed(&pool, n_files).await;
|
||||
|
||||
// ── Equivalence gate (fresh engines so neither arm rides the other's cache) ──
|
||||
let (eng_before, _) = fresh_engine(&pool);
|
||||
let (eng_after, file_repo) = fresh_engine(&pool);
|
||||
let before = before_included(&eng_before, seeded.caller, &seeded.selection).await;
|
||||
let after = after_included(&eng_after, seeded.caller, &seeded.selection).await;
|
||||
|
||||
let owned_set: HashSet<Uuid> = seeded.owned.iter().copied().collect();
|
||||
let denied_set: HashSet<Uuid> = seeded.denied.iter().copied().collect();
|
||||
let missing_set: HashSet<Uuid> = seeded.missing.iter().copied().collect();
|
||||
|
||||
let mut fail = false;
|
||||
if before != after {
|
||||
eprintln!("GATE FAIL: batch inclusion set/order != per-file require loop");
|
||||
fail = true;
|
||||
}
|
||||
// The included set must be EXACTLY the caller's owned files, in input order.
|
||||
let expected: Vec<Uuid> = seeded
|
||||
.selection
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|id| owned_set.contains(id))
|
||||
.collect();
|
||||
if after != expected {
|
||||
eprintln!("GATE FAIL: included set is not exactly the caller's owned files (in order)");
|
||||
fail = true;
|
||||
}
|
||||
if after.iter().any(|id| denied_set.contains(id)) {
|
||||
eprintln!("GATE FAIL: a DENIED (other-drive) file was included — authz regression!");
|
||||
fail = true;
|
||||
}
|
||||
if after.iter().any(|id| missing_set.contains(id)) {
|
||||
eprintln!("GATE FAIL: a MISSING id was included");
|
||||
fail = true;
|
||||
}
|
||||
// The batch fetch of the authorized ids must return exactly those files.
|
||||
let allowed_ids: Vec<String> = after.iter().map(Uuid::to_string).collect();
|
||||
let fetched = file_repo
|
||||
.get_files_by_ids(&allowed_ids)
|
||||
.await
|
||||
.expect("batch fetch");
|
||||
let fetched_ids: HashSet<Uuid> = fetched
|
||||
.iter()
|
||||
.filter_map(|f| Uuid::parse_str(f.id()).ok())
|
||||
.collect();
|
||||
if fetched_ids != owned_set {
|
||||
eprintln!("GATE FAIL: batch fetch of authorized ids != owned files");
|
||||
fail = true;
|
||||
}
|
||||
if fail {
|
||||
cleanup(&pool, &seeded).await;
|
||||
std::process::exit(1);
|
||||
}
|
||||
|
||||
println!("\n#################################################################");
|
||||
println!("# download_zip authz+metadata: per-file require loop vs batch");
|
||||
println!(
|
||||
"# selection = {n} owned + {n} denied + {n} missing (interleaved)",
|
||||
n = n_files
|
||||
);
|
||||
println!("# gate OK: identical inclusion set+order; denied+missing excluded;");
|
||||
println!(
|
||||
"# batch fetch returns exactly the {} owned files.",
|
||||
seeded.owned.len()
|
||||
);
|
||||
println!("#################################################################\n");
|
||||
println!("| {:<26} | {:>10} | {:>12} |", "arm", "wall ms", "µs/file");
|
||||
|
||||
// Latency: cold engine each run (empty caches — the first-download shape).
|
||||
let total = seeded.selection.len();
|
||||
for (label, batch) in [
|
||||
("per-file require loop", false),
|
||||
("batch check_files_read", true),
|
||||
] {
|
||||
let (engine, _) = fresh_engine(&pool);
|
||||
let t = Instant::now();
|
||||
let got = if batch {
|
||||
after_included(&engine, seeded.caller, &seeded.selection).await
|
||||
} else {
|
||||
before_included(&engine, seeded.caller, &seeded.selection).await
|
||||
};
|
||||
let el = t.elapsed();
|
||||
assert_eq!(got.len(), seeded.owned.len(), "arm {label} inclusion count");
|
||||
println!(
|
||||
"| {:<26} | {:>10.2} | {:>12.2} |",
|
||||
label,
|
||||
el.as_secs_f64() * 1e3,
|
||||
el.as_secs_f64() * 1e6 / total as f64
|
||||
);
|
||||
}
|
||||
|
||||
cleanup(&pool, &seeded).await;
|
||||
println!("\nAll Round-24 authz-equivalence gates passed.");
|
||||
}
|
||||
@@ -726,31 +726,46 @@ impl BatchOperationService {
|
||||
let mut items_added: usize = 0;
|
||||
|
||||
// ── Add individual files at the root of the ZIP ──────────────────
|
||||
// Authorize + fetch metadata for the whole multi-select in 2 round-trips
|
||||
// (one batch Read check + one batch get) instead of the per-file
|
||||
// `get_file_with_perms` N+1 (2 round-trips/file). The batch check also
|
||||
// primes the resource→drive cache, so `add_file_entry_streamed`'s
|
||||
// per-file stream-open re-check lands on the cache. A denied / missing /
|
||||
// unparseable id is absent from the map → skipped in the same input
|
||||
// order, exactly as the old per-file loop skipped it. Authorization is
|
||||
// UNCHANGED — still enforced (pre-check here + the stream open's own
|
||||
// Read check + Recents recording) before any ZIP entry is written, so a
|
||||
// denied file never leaks its name into the archive (benches/ROUND24.md).
|
||||
let authorized = self
|
||||
.file_retrieval
|
||||
.get_files_by_ids_with_perms(&file_ids, user_id)
|
||||
.await
|
||||
.map_err(BatchOperationError::Domain)?;
|
||||
let by_id: HashMap<Uuid, FileDto> = authorized
|
||||
.into_iter()
|
||||
.filter_map(|f| Uuid::parse_str(&f.id).ok().map(|u| (u, f)))
|
||||
.collect();
|
||||
for file_id in &file_ids {
|
||||
let file_dto = match Uuid::parse_str(file_id).ok().and_then(|u| by_id.get(&u)) {
|
||||
Some(f) => f,
|
||||
None => {
|
||||
info!("Skipping file {} (not accessible or missing)", file_id);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
match self
|
||||
.file_retrieval
|
||||
.get_file_with_perms(file_id, user_id)
|
||||
.add_file_entry_streamed(
|
||||
&mut zip,
|
||||
file_id,
|
||||
&file_dto.name,
|
||||
&file_dto.mime_type,
|
||||
Some(user_id),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(file_dto) => {
|
||||
match self
|
||||
.add_file_entry_streamed(
|
||||
&mut zip,
|
||||
file_id,
|
||||
&file_dto.name,
|
||||
&file_dto.mime_type,
|
||||
Some(user_id),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => items_added += 1,
|
||||
Err(e) => {
|
||||
info!("Could not add file {} to ZIP: {}", file_dto.name, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(_) => items_added += 1,
|
||||
Err(e) => {
|
||||
info!("Could not get file metadata {}: {}", file_id, e);
|
||||
info!("Could not add file {} to ZIP: {}", file_dto.name, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -287,6 +287,52 @@ impl FileRetrievalService {
|
||||
Ok(files.into_iter().map(FileDto::from).collect())
|
||||
}
|
||||
|
||||
/// Batched, authorized multi-get for the ZIP-download multi-select — the
|
||||
/// batch form of [`FileRetrievalUseCase::get_file_with_perms`] over an
|
||||
/// explicit id list.
|
||||
///
|
||||
/// Authorizes `Read` on every id in ONE `check_files_read_batch`
|
||||
/// round-trip (which resolves all drives in a single query AND primes the
|
||||
/// resource→drive cache, so the per-file re-check the subsequent stream
|
||||
/// open performs becomes a cache hit), then fetches only the authorized ids
|
||||
/// in ONE `get_files_by_ids` query. Replaces `download_zip`'s per-file
|
||||
/// `require_file` + `get_file` loop — 2 round-trips/file → 2 total.
|
||||
///
|
||||
/// Returns the authorized, existing files; a denied / missing / unparseable
|
||||
/// id is simply **absent** from the result (the caller re-associates by id
|
||||
/// and skips the rest, exactly as the per-file loop skipped a denied /
|
||||
/// missing `get_file_with_perms`). Read-authorization is identical to the
|
||||
/// per-file path (`check_files_read_batch` is documented and gated as
|
||||
/// semantically identical to looping `require`). Recents recording is left
|
||||
/// to the subsequent per-file stream open (`get_file_stream_with_perms`),
|
||||
/// which records it (throttle-coalesced) — same net effect as the old
|
||||
/// loop's `notify_file_accessed` + stream double-notify. Fail-closed if no
|
||||
/// engine was injected, mirroring [`Self::require_file`].
|
||||
pub async fn get_files_by_ids_with_perms(
|
||||
&self,
|
||||
ids: &[String],
|
||||
caller_id: Uuid,
|
||||
) -> Result<Vec<FileDto>, DomainError> {
|
||||
let authz = self.authz.as_ref().ok_or_else(|| {
|
||||
DomainError::internal_error("FileRetrieval", "Authorization engine unavailable")
|
||||
})?;
|
||||
// Unparseable ids can't be authorized (the per-file path 404s on them),
|
||||
// so drop them here — they stay absent from the authorized set.
|
||||
let uuids: Vec<Uuid> = ids.iter().filter_map(|s| Uuid::parse_str(s).ok()).collect();
|
||||
if uuids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let allowed = authz
|
||||
.check_files_read_batch(Subject::User(caller_id), &uuids)
|
||||
.await?;
|
||||
if allowed.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let allowed_ids: Vec<String> = allowed.iter().map(Uuid::to_string).collect();
|
||||
let files = self.file_read.get_files_by_ids(&allowed_ids).await?;
|
||||
Ok(files.into_iter().map(FileDto::from).collect())
|
||||
}
|
||||
|
||||
/// Range read for HTTP Range Requests, cache-aware.
|
||||
///
|
||||
/// Media players and PDF viewers fetch these files *exclusively* through
|
||||
|
||||
Reference in New Issue
Block a user