Stream uploads directly into the CDC chunk store (no spool, single write)

Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
+83 -216
View File
@@ -1,62 +1,56 @@
# Storage Fine Tuning
This page is for sysadmins who want to tune **where** OxiCloud spools
upload bodies and **why** the placement matters for throughput and
memory. The defaults work; the gains from a tuned layout are
This page is for sysadmins who want to tune **where** OxiCloud places
upload data on disk and **why** the placement matters for throughput
and memory. The defaults work; the gains from a tuned layout are
significant on busy instances or constrained containers.
## The upload lifecycle in 30 seconds
Every upload moves through two stages:
```
┌─── direct (single-PUT) upload ────────┐
client ─┤ ├──► OxiCloud accepts the
└─── multi-chunk upload │ bytes into a SPOOL on
(`/api/uploads` / │ local disk.
`/dav/uploads/...`) │
│ Direct upload → OXICLOUD_UPLOAD_TMPDIR
│ Chunked upload → OXICLOUD_CHUNK_DIR
│
▼
┌─── direct (single-PUT / multipart) ───┐
client ─┤ │ Streamed DIRECTLY into the
│ ├──► content-addressable store:
│ │ CDC chunking + BLAKE3 +
│ │ dedup checks happen while
│ │ the bytes arrive. No spool
│ │ file, no re-read; chunks
│ │ the store already has are
│ │ never written at all.
│ │
└─── multi-chunk upload ────────────────┤ Chunk PARTS accumulate on
(`/api/uploads` / │ disk under OXICLOUD_CHUNK_DIR
`/dav/uploads/...`) │ until /complete, which
│ streams them (in order)
▼ through the same CDC path.
┌─────────────────────────┐
│ Once the upload is │
│ complete (and verified │
│ if a checksum was │
│ supplied), OxiCloud │
│ MOVES the assembled │
│ blob into the configured│
│ STORAGE BACKEND: │
│ │
│ • local FS (.blobs/) │
│ • S3-compatible │
│ • Azure Blob │
│ STORAGE BACKEND │
│ • local FS (.blobs/) │
│ • S3-compatible │
│ • Azure Blob │
└─────────────────────────┘
```
Two practical consequences:
- **The spool/chunk directories see write-heavy churn** during uploads —
fast disk (NVMe) and sufficient free space matter more here than on
the final storage backend.
- **The promotion from spool → storage is a `rename(2)` whenever
source and destination share a filesystem** (i.e. when the backend
is `local` and the spool dir is on the same FS as `.blobs/`). On
remote backends (S3, Azure) the promotion is always a network
upload from the local spool; placement of the spool still matters
for intake throughput but the "same FS" rule doesn't apply.
- **Direct uploads no longer use a spool directory.** Each uploaded
byte is written at most once — straight into the blob backend as a
CDC chunk. Re-uploads and edited files write only the chunks the
store doesn't already have.
- **The chunk-session directory sees write-heavy churn** during
multi-chunk uploads — fast disk (NVMe) and sufficient free space
matter more here than on the final storage backend.
## Upload size caps — what each one bounds
Three independent caps control how large an upload OxiCloud will
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk
directories must be able to hold the worst case (cap × concurrent
uploads).
accept. Pick them with disk sizing in mind: the chunk directory must
be able to hold the worst case (cap × concurrent sessions).
| Variable | Default | What it caps | When it fires |
|---|---|---|---|
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `POST /api/files/upload`, `PUT /webdav/...`, and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `PUT /webdav/...` and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |
### Recommendation: prefer chunked uploads for large files
@@ -64,244 +58,117 @@ uploads).
The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
Files between those two caps can only succeed via the chunked
protocol. Three reasons to keep them that way:
protocol. The reason is **resilience**: a direct PUT at 95 % of 5 GB
that drops loses everything (the partially ingested chunks are
reclaimed by GC, but the client must restart from byte 0). The same
drop on a chunked upload loses one ~5 MB chunk; the client retries
that chunk and continues. NextCloud desktop and the OxiCloud web UI
already switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
- **Resilience.** A direct PUT at 95 % of 5 GB that drops loses
everything. The same drop on a chunked upload loses one ~5 MB
chunk; the client retries that chunk and continues.
- **Memory + disk pressure.** Direct PUT spools the full body to
disk per request. Ten concurrent 5 GB direct PUTs use up to 50 GB
of transient spool disk. Chunked spreads each upload across many
small PATCHes; per-request resource use stays bounded by
`OXICLOUD_CHUNK_MAX_BYTES`.
- **Convention.** NextCloud desktop and the OxiCloud web UI already
switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
### Disk sizing
### Why caps matter for tmpfs sizing
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded
(~10 MB per in-flight upload for the CDC ingest buffers) regardless
of the caps. **Disk space** scales with the caps:
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**,
however, scales with the caps:
- **Direct PUT / multipart**: no transient spool. Bytes land directly
in the blob backend as deduplicated chunks; worst-case extra disk
per upload is the file's own (deduplicated) size — the same space
the stored file occupies afterwards.
- **Chunked upload**: each in-flight session accumulates its chunk
parts under `OXICLOUD_CHUNK_DIR` until `/complete` streams them
into the blob store and the session is cleaned up. Worst case disk
per session = **file_size** (the parts); total =
`OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
- **Direct PUT**: each in-flight upload spools the full body to disk
under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst case disk =
`OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs`.
- **Chunked upload**: each in-flight session accumulates chunks
under `OXICLOUD_CHUNK_DIR`, then assembles them into a single temp
file before promotion. Worst case disk per session = **2 ×
file_size** (chunks + assembled file); total disk =
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
| Settings | Chunked worst case (5 sessions) | Safe on 4 GB volume? |
|---|---|---|
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB | 50 GB | ❌ overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB | 2.5 GB | ✅ fits |
The chunked formula uses `OXICLOUD_MAX_UPLOAD_SIZE` because that's
what bounds the declared `total_size` at session creation. The
direct-PUT formula uses the smaller `OXICLOUD_DIRECT_PUT_MAX_BYTES`
since that's what bounds each direct PUT body.
### Don't put `OXICLOUD_CHUNK_DIR` on tmpfs
### Sizing examples
A 4 GB tmpfs serving a small team (5 concurrent direct PUTs OR 5
concurrent chunked sessions):
| Settings | Direct-PUT worst case | Chunked worst case | Safe on 4 GB tmpfs? |
|---|---|---|---|
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=100 MB | 5 GiB (5 × 1 GiB) | 100 GB (5 × 2 × 10 GB) | ❌ chunked overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=100 MB, `OXICLOUD_CHUNK_MAX_BYTES`=20 MB | 500 MB | 5 GB | ⚠ direct PUT fits, chunked still overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=300 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=50 MB, `OXICLOUD_CHUNK_MAX_BYTES`=10 MB | 250 MB | 3 GB | ✅ both fit |
A real-disk volume (cheap, large):
| Settings | Direct-PUT worst case | Chunked worst case | Comment |
|---|---|---|---|
| Defaults (see row above) | 5 GiB | 100 GB | Fine on a 200+ GB volume; almost any real-disk setup |
| `OXICLOUD_MAX_UPLOAD_SIZE`=100 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=5 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=500 MB | 25 GiB | 1 TB | Plausible for video archives; needs a dedicated upload volume |
### Choosing tmpfs vs real disk
| Constraint | Choice |
|---|---|
| `OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs + 2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions ≤ free RAM × 0.5` | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
| Worst case exceeds half free RAM | **real disk** — same filesystem as `.blobs/` ideal |
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
The defaults (`OXICLOUD_MAX_UPLOAD_SIZE`=10 GB,
`OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB,
`OXICLOUD_CHUNK_MAX_BYTES`=100 MB) assume **real disk**. Don't run
the defaults against tmpfs unless you've sized it for the worst case.
In many container setups the OS temp dir is **tmpfs** — RAM-backed
storage that counts against the cgroup memory limit. A few concurrent
multi-GB chunked sessions on tmpfs will wake the OOMKiller long
before the uploads finish. Point `OXICLOUD_CHUNK_DIR` at a real-disk
directory in containers.
## TL;DR
| Variable | Default | Purpose |
|---|---|---|
| `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) |
| `OXICLOUD_UPLOAD_TMPDIR` | OS temp dir | Where non-chunked PUT bodies are spooled |
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate |
The two rules that matter most:
1. **Put all three on the same filesystem.** Blob promotion is an
atomic `rename(2)` when source and destination share an FS — cheap
and crash-safe. Across filesystems it becomes a full `read + write +
unlink`, multiplying the IO and widening the durability window.
2. **Don't leave the spool dir on tmpfs** (the default in many
containers). Spool bodies count against the cgroup memory limit
1. **Keep `OXICLOUD_CHUNK_DIR` off tmpfs** (the default in many
containers) — chunk parts count against the cgroup memory limit
and can trigger OOMKill on multi-GB uploads.
2. **NVMe for the chunk dir pays off** on deployments with heavy
large-file traffic: each chunk PUT writes a file and the progress
bitmap, and `/complete` reads them all back in order.
## Where each upload surface spools
OxiCloud has several entry points that accept request bodies. They
land in different places by default:
## Where each upload surface writes
| Surface | Default destination | Configurable via |
|---|---|---|
| REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` |
| REST chunked assemble (during `/complete`) | `{STORAGE_PATH}/.uploads/{upload_id}/assembled` | `OXICLOUD_CHUNK_DIR` |
| NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` |
| NextCloud chunked assemble (during `MOVE`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/.assembled` | `OXICLOUD_CHUNK_DIR` |
| Native WebDAV PUT (`PUT /webdav/{path}`) | OS temp dir (`/tmp`) | `OXICLOUD_UPLOAD_TMPDIR` |
| NextCloud single-file PUT (`PUT /dav/files/.../{path}`) | OS temp dir | `OXICLOUD_UPLOAD_TMPDIR` |
| REST multipart upload (`POST /api/files/upload`) | `{STORAGE_PATH}/.dedup_temp/upload-{uuid}` | `OXICLOUD_STORAGE_PATH` (subdir is hard-wired) |
| Final blob storage (after fsync + rename) | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
| Direct PUT / multipart / WOPI / chunked `/complete` | straight into the blob backend (CDC chunks) | `OXICLOUD_STORAGE_PATH` (local backend) |
| Final blob storage | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
## Why placement matters
### 1. Same filesystem ⇒ promotion is a rename
OxiCloud uses **content-addressable storage**: the final blob path is
derived from the file's BLAKE3 hash, which can only be known after the
last byte arrives. So every upload writes to a temp location first,
then **promotes** the temp file to `.blobs/{ab}/{abc…}.blob` by way of
a `rename(2)` call.
- **Same FS:** `rename` is atomic, O(1), no data copy. Total upload
cost = body bytes received + one rename syscall. Crash-safe — the
blob either exists at the final path or doesn't.
- **Cross-FS:** the kernel can't `rename(2)` across filesystems. The
blob backend falls back to `fs::copy + fs::remove_file` (visible in
`local_blob_backend.rs` as the EXDEV handler). Total cost = body
bytes received + one full file copy. Doubles the IO bandwidth used
per upload and widens the durability window.
### 2. Spool off tmpfs
`tempfile::NamedTempFile::new()` (used when `OXICLOUD_UPLOAD_TMPDIR`
is unset) honors `$TMPDIR`, which in many container setups points at
**tmpfs** — RAM-backed storage. A 2 GB upload spool then consumes 2 GB
of memory until the rename promotes it to disk.
In a Kubernetes pod with a 4 GB memory limit, the OOMKiller wakes up
long before the upload finishes. With `OXICLOUD_UPLOAD_TMPDIR` pointed
at a real-disk directory, the spool's memory footprint stays at ~one
HTTP frame regardless of file size.
### 3. NVMe for the hot path
The chunked-upload session directory sees a LOT of small writes —
each chunk PUT writes a file, the progress bitmap is rewritten after
each PUT, the assemble step reads them all back in order. Pointing
`OXICLOUD_CHUNK_DIR` at an NVMe device is a substantial win on
deployments that handle large file uploads, even if the final blob
storage is on slower disk.
The same applies to `OXICLOUD_UPLOAD_TMPDIR` (single-file PUTs).
A common high-throughput layout:
- **NVMe** (small, fast): `OXICLOUD_CHUNK_DIR`, `OXICLOUD_UPLOAD_TMPDIR`
- **HDD or NAS** (large, cheap): `OXICLOUD_STORAGE_PATH`/`.blobs/`
Trade-off: the rename optimization (rule 1) DOESN'T apply across
filesystems. If you split the hot path off the blob filesystem, every
upload pays a full file copy on promotion. You have to choose
between **fast intake** and **zero-copy promotion**.
| Goal | Layout | Cost per upload |
|---|---|---|
| Fastest possible intake | NVMe chunk dir + HDD blobs | 1× write to NVMe + 1× read NVMe + 1× write to HDD (copy) |
| Lowest IO + crash safety | NVMe everything OR HDD everything | 1× write to disk + 1 rename (~0 cost) |
| Default (do nothing) | Everything under `STORAGE_PATH` on whatever FS that is | Depends on `STORAGE_PATH` placement |
For most deployments **"same FS everywhere"** wins. The NVMe-split is
useful when intake latency dominates the user experience and you can
afford the doubled IO.
The local blob backend stages each chunk write under
`{STORAGE_PATH}/.dedup_temp/` and promotes it with an atomic
`rename(2)` — both directories live under `OXICLOUD_STORAGE_PATH`,
so same-filesystem placement (and therefore atomic promotion) is
automatic and not separately configurable.
## Recommended layouts
### Single-disk box (most common)
Defaults are fine. Optionally set `OXICLOUD_UPLOAD_TMPDIR` to keep
the PUT spool off `/tmp`:
Defaults are fine:
```bash
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/.spool
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
```
All three on the same filesystem → rename promotion → atomic and fast.
### Container with constrained memory
Critical: make sure neither spool sits on tmpfs.
Critical: make sure the chunk dir doesn't sit on tmpfs.
```bash
OXICLOUD_STORAGE_PATH=/data
OXICLOUD_UPLOAD_TMPDIR=/data/.spool
OXICLOUD_CHUNK_DIR=/data/.uploads
```
If you can't mount a writable `/data`, at minimum bind-mount a real
volume at the spool dirs.
### Split-disk (NVMe intake + HDD blobs)
```bash
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/
OXICLOUD_UPLOAD_TMPDIR=/mnt/nvme/oxi-spool
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
```
Faster intake; pays a copy on promotion. Worth it when uploads are
many small files (NVMe IOPS dominates) or when intake latency directly
hits user-visible UX.
Chunk parts land on NVMe (fast PUTs, fast `/complete` read-back);
the deduplicated chunks are written once to the HDD-backed blob
store as `/complete` streams through them.
## Sharing the spool and chunk directories
## Sharing the chunk directory
Pointing `OXICLOUD_UPLOAD_TMPDIR` and `OXICLOUD_CHUNK_DIR` at the
**same directory** is supported by design. Each writer tags its
output so the surfaces never interfere with each other:
The REST and NC chunked surfaces can share `OXICLOUD_CHUNK_DIR` by
design. Each writer tags its output so they never interfere:
| Writer | On-disk name pattern |
|---|---|
| PUT spool (single-file uploads) | `.tmpXXXXXXXX` — files (not directories), random suffix |
| REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix |
| NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir |
The 24-hour orphan-session cleanup loop filters strictly on the
`oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory
that happens to live alongside chunked sessions. The PUT spool's
`.tmpXXXX` files are files (not directories) and the NC subtree's
`nextcloud/` root has its own name — both are invisible to the
cleanup loop.
**Recommendation:** for new deployments, use separate directories
anyway (the defaults `.spool/` and `.uploads/` already do this) —
it makes disk-usage attribution clearer and keeps IOPS isolated when
both are busy. Shared directories are safe to use when disk layout
forces it.
## What's NOT yet configurable
- **REST multipart upload directory** (`POST /api/files/upload`) is
hard-wired to `{STORAGE_PATH}/.dedup_temp/`. It can't be moved
separately. Same-FS placement is automatic.
- **WOPI PutFile spool** (Office editor saves) uses the bare OS temp
dir without honoring `OXICLOUD_UPLOAD_TMPDIR`. This is a known
inconsistency and on the hardening backlog.
- **Per-user / per-drive spool directories** — all users share the
same `OXICLOUD_CHUNK_DIR` root today. Multi-tenant isolation
through separate spool dirs isn't supported.
that happens to live alongside chunked sessions.
## Quick verification