Oxicloud/docs/plan/benchmake-and-performance-tracking.md

OxiCloud Load Test Scaffolding (K6) — Scenario A

Context

OxiCloud needs a repeatable way to detect performance regressions as features land. The current tests/load/ directory is empty except for a README asking K6-vs-drill. Scenario A focuses on a single user with many subfolders and cascading systems (folder ops, ReBAC permission groups, nested subject groups) — measuring p50/p95/p99 to identify inflections when new features are added. Scenario B (many concurrent users) is deferred.

Decisions already locked in via conversation:

• K6 (Go-based, JS scripting in goja) over Rust testers — client overhead is dwarfed by server latency for these scenarios; regression deltas only require client consistency, not raw speed.
• Two scenario tiers: a fast smoke for PRs (~1 min, no regression gate — just verifies the harness still runs) and long scenarios for nightly + manual (regression-gated).
• Hybrid seeding: bulk fixtures via direct SQL (cargo run --bin load-seed); the resources each scenario actively touches go through REST so the measured path is realistic.
• Result storage: baseline/baseline.json committed in repo (p50/p95/p99 + tolerance per metric); per-run raw JSON kept locally / CI artifact, gitignored. Baseline updates are deliberate PRs.
• No PR gating on long suite — nightly on main + manual workflow_dispatch only.

Approach

1. Directory layout

tests/load/
  README.md
  test.env                         # base_url=http://localhost:8088, admin creds
  run.sh                           # full suite: spawn-db → seed → server → k6 → compare → cleanup
  smoke.sh                         # smoke only: same shape, runs scenarios/smoke.js, no gate
  compare.mjs                      # diffs k6 summary.json vs baseline/baseline.json
  lib/
    auth.js                        # login() → bearer token (POST /api/auth/login)
    http.js                        # base URL + auth header helpers
    metrics.js                     # custom k6 Trends, naming convention <scenario>.<op>
  scenarios/
    smoke.js                       # 1 VU, 1 iter — login, create folder, upload, list, delete
    folder_cascade.js              # list/move/copy/trash on depth-8 fanout-5 tree
    share_cascade_rebac.js         # grant a folder, fetch N descendants as grantee
    subject_group_nested.js        # nested-group chain (depth 3), grant via group, fetch
  baseline/
    baseline.json                  # committed; { "<scenario>.<op>": {p50,p95,p99,tolerance_pct} }
  results/
    .gitkeep                       # raw runs, gitignored

Mirrors the tests/api/ shell pattern (run.sh, test.env, separate server port). Uses port 8088 to avoid colliding with api-test on 8087.

2. Rust bulk seeder — src/bin/load-seed.rs

New binary registered in Cargo.toml alongside generate-openapi and migrate-nfc-filenames.

CLI:

cargo run --bin load-seed -- \
  --depth 8 --fanout 5 --files-per-leaf 10 \
  --extra-users 20 --group-depth 3 --group-fanout 5

Inserts via sqlx (one transaction per phase):

• 1 admin + N extra users into auth.users (Argon2id with light params m=16384,t=1,p=1; same params as src/infrastructure/services/password_hasher.rs test path).
• One shared 0-byte blob row in storage.blobs with ref_count = total_files.
• Folder tree in storage.folders, inserted level-by-level so the trg_folders_path BEFORE-INSERT trigger can compute path/lpath from the parent chain.
• Files in storage.files, all referencing the shared blob hash for dedup.
• Subject groups in auth.subject_groups + membership rows in auth.subject_group_members (XOR member_user_id/member_group_id), forming a nested chain G_root → G_mid → G_leaf → users.
• ReBAC grants in storage.access_grants for a known set of test resources (used by share_cascade_rebac.js).

Reuses existing connection config: reads DATABASE_URL from env, same shape as tests/common/server.env.

Critical gotchas surfaced during exploration:

• Tree-ETag triggers (folders_bump_tree_etag_*) are STATEMENT-level (per commit 4200209d), so bulk inserts fire them once per statement — safe.
• ReBAC cascade is computed in the application layer (pg_acl_engine), not at DB level. Seeder just inserts grant rows; the cascade is the server behavior we want to measure.
• auth.subject_groups.name is CITEXT, max 64, RFC-5321 local-part shape — generated names use g_NNN pattern to stay valid.

3. K6 scenarios

All scenarios load baseline.json at startup and set thresholds dynamically from it (http_req_duration{op:x}: p(95)<baseline.x.p95 * (1 + tolerance_pct/100)). This makes K6 itself fail the run on regression, and compare.mjs produces the human-readable diff.

Endpoint contracts (confirmed by exploration):

• Login: POST /api/auth/login body {username, password} → {access_token, ...}.
• Folder ops: POST /api/folders, GET /api/folders/{id}/contents (or /contents/paginated at high depth), PUT /api/folders/{id}/move, POST /api/batch/folders/copy, DELETE /api/folders/{id}. Handlers in src/interfaces/api/handlers/folder_handler.rs and batch_handler.rs.
• Grants: POST /api/grants (subject {type, id|email}, resource {type, id}, role or permissions[]), GET /api/grants?resource_type=folder&resource_id=…. Handler src/interfaces/api/handlers/grant_handler.rs.
• Groups: POST /api/groups (admin), POST /api/groups/{id}/members body {user_id} or {group_id}. Handler src/interfaces/api/handlers/subject_group_handler.rs. Nesting limit is 8 — --group-depth 3 is well inside.
• Files: multipart POST /api/files/upload with fields folder_id + file.

Per-scenario metric naming (metrics.js enforces): <scenario>.<op> → e.g. folder_cascade.list_depth8, share_cascade_rebac.fetch_as_grantee, subject_group_nested.grant_via_chain3.

Cold/warm split: first iteration tagged cold (its metrics flow to <scenario>.<op>_cold), rest are warm. Single first iter avoids polluting warm metrics with one-shot connection setup.

4. Runner scripts

run.sh (mirrors tests/api/run.sh shape):

1. spawn-db.sh (reused from tests/common/)
2. init-test-schema.sh (reused)
3. cargo run --bin load-seed -- <args> against the test DB
4. Start target/<profile>/oxicloud on port 8088 with OXICLOUD_STORAGE_PATH=tests/load/storage
5. wait_for_http http://localhost:8088/ready
6. k6 run --summary-export=results/$(date +%s).json scenarios/folder_cascade.js scenarios/share_cascade_rebac.js scenarios/subject_group_nested.js
7. node compare.mjs results/<latest>.json baseline/baseline.json → exits non-zero on regression
8. trap cleanup EXIT tears down server + DB

smoke.sh: same shape but only runs scenarios/smoke.js, skips the seeder (smoke creates its own minimal data), skips compare.mjs. Goal is harness liveness, not regression.

5. compare.mjs

Plain Node script (no deps, just fs + process). Reads two JSONs, walks every metric in baseline, computes:

delta_pct = (current - baseline) / baseline * 100

Prints a human table:

folder_cascade.list_depth8.p95: 142ms → 198ms (+39%)  REGRESSION (tolerance 10%)
share_cascade_rebac.grant.p95:  60ms → 58ms  (-3%)    ok

Exit 1 if any metric exceeds tolerance. Exit 0 otherwise. Missing metrics in current = exit 1 (suite drift); new metrics in current = warn only.

6. Baseline format

{
  "folder_cascade.list_depth8": { "p50": 12.0, "p95": 45.0, "p99": 80.0, "tolerance_pct": 10 },
  "share_cascade_rebac.fetch_as_grantee": { "p50": 18.0, "p95": 55.0, "p99": 100.0, "tolerance_pct": 10 },
  ...
}

Values are placeholders — first run on a stable machine seeds real numbers. Updated only via the load-baseline recipe (which rewrites baseline.json from the latest run) and committed deliberately as chore(load): accept new baseline for <reason>.

7. Justfile recipes (added to existing justfile)

load:               # full suite — nightly + manual
    bash tests/load/run.sh

load-smoke:         # fast harness check
    bash tests/load/smoke.sh

load-baseline:      # rerun, overwrite baseline from latest result
    bash tests/load/run.sh
    cp tests/load/results/$(ls -t tests/load/results/*.json | head -1) tests/load/baseline/baseline.json

load-seed:          # standalone seeder for local poking
    cargo run --bin load-seed -- --depth 8 --fanout 5 --files-per-leaf 10

Matches existing recipe naming (test-*, front-*, api-test).

8. CI workflows (stubs)

• .github/workflows/load-nightly.yml — cron 0 3 * * * UTC + workflow_dispatch; runs just load on ubuntu-latest initially with a # TODO: replace with self-hosted runner — shared GH runners produce noisy results for regression detection comment; uploads tests/load/results/*.json as artifact; on compare.mjs non-zero, opens an issue using peter-evans/create-issue-from-file with the diff report.
• .github/workflows/load-smoke.yml — on every PR to main; runs just load-smoke; ~1 min budget; no regression gate, just "harness builds & runs."

9. Critical files to create / modify

Create:

• tests/load/README.md
• tests/load/test.env
• tests/load/run.sh, smoke.sh
• tests/load/compare.mjs
• tests/load/lib/auth.js, http.js, metrics.js
• tests/load/scenarios/smoke.js, folder_cascade.js, share_cascade_rebac.js, subject_group_nested.js
• tests/load/baseline/baseline.json (placeholder values)
• tests/load/results/.gitkeep
• src/bin/load-seed.rs
• .github/workflows/load-nightly.yml, load-smoke.yml

Modify:

• Cargo.toml — add [[bin]] name = "load-seed" path = "src/bin/load-seed.rs" after the migrate-nfc-filenames entry
• justfile — append four load* recipes
• .gitignore — add tests/load/results/*.json and tests/load/storage/

10. Reused existing utilities

• tests/common/spawn-db.sh, stop-db.sh, init-test-schema.sh, server.env — reused as-is by run.sh / smoke.sh.
• Argon2id parameters from src/infrastructure/services/password_hasher.rs test path — mirrored in load-seed.rs so test users can log in via the real auth flow.
• DB schema migrations (migrations/) — applied via the existing init-test-schema.sh; no schema changes needed.

Verification

1. Build: cargo build --bin load-seed succeeds; cargo clippy --all-features --all-targets -- -D warnings stays green.
2. Seeder works standalone: just db && just load-seed populates auth.users, storage.folders (depth-8 tree visible via SELECT count(*) FROM storage.folders WHERE user_id = …), auth.subject_groups, storage.access_grants.
3. Smoke runs locally: just load-smoke — exits 0 in under ~90 s, leaves no orphan postgres/oxicloud processes.
4. Full suite runs locally: just load — k6 prints per-scenario p50/p95/p99; compare.mjs prints the diff table; since baseline.json is empty placeholders, expect a "REGRESSION" stub-report that the user reviews before running just load-baseline to accept the first real baseline.
5. Baseline workflow: after first just load, just load-baseline rewrites baseline.json; subsequent just load exits 0 (within tolerance).
6. CI smoke: trigger load-smoke.yml on a draft PR; confirm it completes under the time budget and uploads no artifacts on success.
7. CI nightly: manually trigger load-nightly.yml via workflow_dispatch; confirm artifact upload and (since no real baseline yet) confirm issue creation works.

Out of scope

• Real baseline values (first run on user's hardware seeds them).
• Self-hosted runner setup (workflow stub uses ubuntu-latest with a TODO comment).
• Scenario B (many concurrent users) — deferred per user request.
• Goose / Rust-tester alternative — deferred until scenario B demonstrates K6 saturation issues.
• Additional scenarios (CalDAV / CardDAV / WebDAV / search depth) — the scaffolding makes them additive: drop a new file in scenarios/, add its metric names to baseline.json.