Edouard Vanbelle
0ef4c624c5
initial test from Ed's nuc:
metric pctl baseline current delta status
-----------------------------------------------------------------------------------------------
folder_cascade.list_depth1 p50 0.3ms 0.3ms -6.3% ok
folder_cascade.list_depth1 p95 2.3ms 0.5ms -75.7% ok
folder_cascade.list_depth1 p99 4.7ms 2.5ms -48.1% ok
folder_cascade.list_depth4 p50 0.4ms 0.3ms -10.0% ok
folder_cascade.list_depth4 p95 0.9ms 0.6ms -31.2% ok
folder_cascade.list_depth4 p99 2.4ms 1.0ms -56.7% ok
folder_cascade.list_depth8 p50 0.3ms 0.3ms -8.8% ok
folder_cascade.list_depth8 p95 0.6ms 0.5ms -22.2% ok
folder_cascade.list_depth8 p99 1.9ms 0.5ms -71.5% ok
folder_cascade.list_depth_deep p50 0.3ms 0.3ms -5.0% ok
folder_cascade.list_depth_deep p95 0.6ms 0.5ms -18.6% ok
folder_cascade.list_depth_deep p99 2.0ms 0.7ms -67.2% ok
share_cascade_rebac.list_grants p50 0.4ms 0.3ms -27.6% ok
share_cascade_rebac.list_grants p95 1.2ms 0.5ms -57.0% ok
share_cascade_rebac.list_grants p99 1.7ms 1.1ms -36.7% ok
share_cascade_rebac.fetch_as_grantee_depth1 p50 0.5ms 0.5ms -11.1% ok
share_cascade_rebac.fetch_as_grantee_depth1 p95 1.1ms 0.7ms -41.2% ok
share_cascade_rebac.fetch_as_grantee_depth1 p99 3.0ms 1.3ms -58.3% ok
share_cascade_rebac.fetch_as_grantee_depth4 p50 0.5ms 0.5ms -13.7% ok
share_cascade_rebac.fetch_as_grantee_depth4 p95 1.4ms 0.7ms -50.5% ok
share_cascade_rebac.fetch_as_grantee_depth4 p99 2.2ms 1.1ms -49.3% ok
share_cascade_rebac.fetch_as_grantee_depth8 p50 0.5ms 0.4ms -16.0% ok
share_cascade_rebac.fetch_as_grantee_depth8 p95 0.9ms 0.7ms -26.1% ok
share_cascade_rebac.fetch_as_grantee_depth8 p99 1.5ms 0.9ms -40.1% ok
share_cascade_rebac.fetch_as_grantee_depth_deep p50 0.5ms 0.4ms -17.3% ok
share_cascade_rebac.fetch_as_grantee_depth_deep p95 1.0ms 0.7ms -31.2% ok
share_cascade_rebac.fetch_as_grantee_depth_deep p99 1.6ms 0.8ms -49.4% ok
subject_group_nested.fetch_as_member_depth1 p50 0.5ms 0.4ms -8.4% ok
subject_group_nested.fetch_as_member_depth1 p95 0.6ms 0.6ms -10.2% ok
subject_group_nested.fetch_as_member_depth1 p99 1.4ms 0.6ms -55.2% ok
subject_group_nested.fetch_as_member_depth4 p50 0.5ms 0.5ms -7.9% ok
subject_group_nested.fetch_as_member_depth4 p95 0.6ms 0.6ms -4.0% ok
subject_group_nested.fetch_as_member_depth4 p99 0.7ms 0.6ms -2.2% ok
subject_group_nested.fetch_as_member_depth8 p50 0.5ms 0.4ms -8.9% ok
subject_group_nested.fetch_as_member_depth8 p95 0.5ms 0.6ms +7.1% ok
subject_group_nested.fetch_as_member_depth8 p99 0.6ms 0.7ms +10.3% ok
subject_group_nested.fetch_as_member_depth_deep p50 0.5ms 0.4ms -7.6% ok
subject_group_nested.fetch_as_member_depth_deep p95 0.6ms 0.5ms -11.9% ok
subject_group_nested.fetch_as_member_depth_deep p99 0.6ms 0.7ms +8.9% ok
95 lines
4.3 KiB
Markdown
95 lines
4.3 KiB
Markdown
# tests/load/
|
||
|
||
K6 load-test suite for OxiCloud. Detects performance regressions by comparing
|
||
each run's p50/p95/p99 against a committed baseline.
|
||
|
||
## Suites
|
||
|
||
- **smoke** — `just load-smoke`. Single VU, single iteration of one scenario.
|
||
Verifies the harness still builds and the server boots. ~1 minute. Run on
|
||
every PR. No regression gate.
|
||
- **full** — `just load`. Runs every scenario under `scenarios/` against a
|
||
seeded database. Compares results against `baseline/load.json`; exits
|
||
non-zero on regression beyond the per-metric tolerance. Run nightly on
|
||
`main` and manually.
|
||
|
||
## Scenarios
|
||
|
||
| File | What it measures |
|
||
| --------------------------------- | ------------------------------------------------------------------------------- |
|
||
| `scenarios/smoke.js` | Login + create folder + upload + list root + delete. Liveness only. |
|
||
| `scenarios/folder_cascade.js` | `GET /contents`, `PUT /move`, batch copy, `DELETE` on a depth-8 fanout-5 tree. |
|
||
| `scenarios/share_cascade_rebac.js`| `POST /grants` on a folder, then descendants fetched by the grantee. |
|
||
| `scenarios/subject_group_nested.js`| Grant via a 3-level nested group chain, then descendants fetched by a member. |
|
||
|
||
Add new scenarios as `scenarios/<name>.js`; register their metric names in
|
||
`baseline/load.json` (or `baseline/smoke.json` if you wire smoke gating).
|
||
|
||
## Seeding
|
||
|
||
`src/bin/load-seed.rs` (invoked by `run.sh`) bulk-inserts fixtures directly
|
||
via sqlx: users, deep folder tree, files (all sharing one dedup'd blob),
|
||
nested subject groups, ReBAC grants. Only the resources each scenario
|
||
actively touches (the grant being created, the move target, etc.) go
|
||
through the REST API at run time — that is the measured hot path.
|
||
|
||
## Baseline & regression detection
|
||
|
||
Baselines live under `baseline/`, split by which runner grades them:
|
||
|
||
| File | Used by | Regression-gated? |
|
||
| --------------------- | ------------------------ | ----------------- |
|
||
| `baseline/load.json` | `just load` (`run.sh`) | Yes |
|
||
| `baseline/smoke.json` | `just load-smoke` | Not yet (see below) |
|
||
|
||
Both have the same shape — one entry per `<scenario>.<op>`:
|
||
|
||
```json
|
||
{
|
||
"folder_cascade.list_depth1": { "p50": 0.97, "p95": 2.04, "p99": 4.82, "tolerance_pct": 10 }
|
||
}
|
||
```
|
||
|
||
K6 scenarios load the relevant file at startup and set `thresholds` from
|
||
it, so a regression fails the K6 run directly. `compare.mjs` also prints a
|
||
human-readable diff table after the run and exits non-zero if any metric
|
||
regresses beyond its tolerance.
|
||
|
||
The smoke scenario is currently **not regression-gated** — `smoke.sh` runs
|
||
the scenario but doesn't call `compare.mjs`. When you decide it should be,
|
||
mirror the `run.sh` pattern and point `compare.mjs` at
|
||
`baseline/smoke.json`.
|
||
|
||
**Updating a baseline is deliberate.** Run `just load-baseline` to rewrite
|
||
`baseline/load.json` from the latest run, then commit it as
|
||
`chore(load): accept new baseline for <reason>`. Never auto-update. For
|
||
`smoke.json`, pass explicit paths to `bake-baseline.mjs`.
|
||
|
||
## Local workflow
|
||
|
||
```bash
|
||
just db # start the test postgres (port 5433)
|
||
just load-seed # seed alone (poking around in psql)
|
||
just load-smoke # fast liveness check
|
||
just load # full suite + regression diff
|
||
just load-baseline # rerun, accept current numbers as the new bar
|
||
```
|
||
|
||
## CI
|
||
|
||
- `.github/workflows/load-smoke.yml` — every PR. ~1 min. No regression gate.
|
||
- `.github/workflows/load-nightly.yml` — cron daily + `workflow_dispatch`.
|
||
Runs the full suite, uploads results as artifact, opens an issue on
|
||
regression. Currently runs on `ubuntu-latest`; replace with a stable
|
||
self-hosted runner for trustworthy regression signal (shared GitHub
|
||
runners produce noisy timings).
|
||
|
||
## Why K6 and not Goose/drill
|
||
|
||
K6 is Go-based (JS scripting in `goja`), not Node. For scenario A, the
|
||
client adds ~50–200µs per request — negligible vs. multi-ms server work,
|
||
and regression deltas only need *consistency*. K6 also gives us
|
||
thresholds-as-DSL, native InfluxDB/Prometheus output, and faster
|
||
scenario iteration than a Rust tester would. Reassess when scenario B
|
||
(many concurrent users) demonstrates K6 saturation issues.
|