Claude
e3f04d58aa
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
187 lines
9.1 KiB
Markdown
187 lines
9.1 KiB
Markdown
# Storage Fine Tuning
|
||
|
||
This page is for sysadmins who want to tune **where** OxiCloud places
|
||
upload data on disk and **why** the placement matters for throughput
|
||
and memory. The defaults work; the gains from a tuned layout are
|
||
significant on busy instances or constrained containers.
|
||
|
||
## The upload lifecycle in 30 seconds
|
||
|
||
```
|
||
┌─── direct (single-PUT / multipart) ───┐
|
||
client ─┤ │ Streamed DIRECTLY into the
|
||
│ ├──► content-addressable store:
|
||
│ │ CDC chunking + BLAKE3 +
|
||
│ │ dedup checks happen while
|
||
│ │ the bytes arrive. No spool
|
||
│ │ file, no re-read; chunks
|
||
│ │ the store already has are
|
||
│ │ never written at all.
|
||
│ │
|
||
└─── multi-chunk upload ────────────────┤ Chunk PARTS accumulate on
|
||
(`/api/uploads` / │ disk under OXICLOUD_CHUNK_DIR
|
||
`/dav/uploads/...`) │ until /complete, which
|
||
│ streams them (in order)
|
||
▼ through the same CDC path.
|
||
┌─────────────────────────┐
|
||
│ STORAGE BACKEND │
|
||
│ • local FS (.blobs/) │
|
||
│ • S3-compatible │
|
||
│ • Azure Blob │
|
||
└─────────────────────────┘
|
||
```
|
||
|
||
Two practical consequences:
|
||
|
||
- **Direct uploads no longer use a spool directory.** Each uploaded
|
||
byte is written at most once — straight into the blob backend as a
|
||
CDC chunk. Re-uploads and edited files write only the chunks the
|
||
store doesn't already have.
|
||
- **The chunk-session directory sees write-heavy churn** during
|
||
multi-chunk uploads — fast disk (NVMe) and sufficient free space
|
||
matter more here than on the final storage backend.
|
||
|
||
## Upload size caps — what each one bounds
|
||
|
||
Three independent caps control how large an upload OxiCloud will
|
||
accept. Pick them with disk sizing in mind: the chunk directory must
|
||
be able to hold the worst case (cap × concurrent sessions).
|
||
|
||
| Variable | Default | What it caps | When it fires |
|
||
|---|---|---|---|
|
||
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
|
||
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `PUT /webdav/...` and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
|
||
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |
|
||
|
||
### Recommendation: prefer chunked uploads for large files
|
||
|
||
The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
|
||
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
|
||
Files between those two caps can only succeed via the chunked
|
||
protocol. The reason is **resilience**: a direct PUT at 95 % of 5 GB
|
||
that drops loses everything (the partially ingested chunks are
|
||
reclaimed by GC, but the client must restart from byte 0). The same
|
||
drop on a chunked upload loses one ~5 MB chunk; the client retries
|
||
that chunk and continues. NextCloud desktop and the OxiCloud web UI
|
||
already switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
|
||
|
||
### Disk sizing
|
||
|
||
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded
|
||
(~10 MB per in-flight upload for the CDC ingest buffers) regardless
|
||
of the caps. **Disk space** scales with the caps:
|
||
|
||
- **Direct PUT / multipart**: no transient spool. Bytes land directly
|
||
in the blob backend as deduplicated chunks; worst-case extra disk
|
||
per upload is the file's own (deduplicated) size — the same space
|
||
the stored file occupies afterwards.
|
||
- **Chunked upload**: each in-flight session accumulates its chunk
|
||
parts under `OXICLOUD_CHUNK_DIR` until `/complete` streams them
|
||
into the blob store and the session is cleaned up. Worst case disk
|
||
per session = **file_size** (the parts); total =
|
||
`OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
|
||
|
||
| Settings | Chunked worst case (5 sessions) | Safe on 4 GB volume? |
|
||
|---|---|---|
|
||
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB | 50 GB | ❌ overflows |
|
||
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB | 2.5 GB | ✅ fits |
|
||
|
||
### Don't put `OXICLOUD_CHUNK_DIR` on tmpfs
|
||
|
||
In many container setups the OS temp dir is **tmpfs** — RAM-backed
|
||
storage that counts against the cgroup memory limit. A few concurrent
|
||
multi-GB chunked sessions on tmpfs will wake the OOMKiller long
|
||
before the uploads finish. Point `OXICLOUD_CHUNK_DIR` at a real-disk
|
||
directory in containers.
|
||
|
||
## TL;DR
|
||
|
||
| Variable | Default | Purpose |
|
||
|---|---|---|
|
||
| `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) |
|
||
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate |
|
||
|
||
The two rules that matter most:
|
||
|
||
1. **Keep `OXICLOUD_CHUNK_DIR` off tmpfs** (the default in many
|
||
containers) — chunk parts count against the cgroup memory limit
|
||
and can trigger OOMKill on multi-GB uploads.
|
||
2. **NVMe for the chunk dir pays off** on deployments with heavy
|
||
large-file traffic: each chunk PUT writes a file and the progress
|
||
bitmap, and `/complete` reads them all back in order.
|
||
|
||
## Where each upload surface writes
|
||
|
||
| Surface | Default destination | Configurable via |
|
||
|---|---|---|
|
||
| REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` |
|
||
| NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` |
|
||
| Direct PUT / multipart / WOPI / chunked `/complete` | straight into the blob backend (CDC chunks) | `OXICLOUD_STORAGE_PATH` (local backend) |
|
||
| Final blob storage | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
|
||
|
||
The local blob backend stages each chunk write under
|
||
`{STORAGE_PATH}/.dedup_temp/` and promotes it with an atomic
|
||
`rename(2)` — both directories live under `OXICLOUD_STORAGE_PATH`,
|
||
so same-filesystem placement (and therefore atomic promotion) is
|
||
automatic and not separately configurable.
|
||
|
||
## Recommended layouts
|
||
|
||
### Single-disk box (most common)
|
||
|
||
Defaults are fine:
|
||
|
||
```bash
|
||
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
|
||
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
|
||
```
|
||
|
||
### Container with constrained memory
|
||
|
||
Critical: make sure the chunk dir doesn't sit on tmpfs.
|
||
|
||
```bash
|
||
OXICLOUD_STORAGE_PATH=/data
|
||
OXICLOUD_CHUNK_DIR=/data/.uploads
|
||
```
|
||
|
||
### Split-disk (NVMe intake + HDD blobs)
|
||
|
||
```bash
|
||
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/
|
||
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
|
||
```
|
||
|
||
Chunk parts land on NVMe (fast PUTs, fast `/complete` read-back);
|
||
the deduplicated chunks are written once to the HDD-backed blob
|
||
store as `/complete` streams through them.
|
||
|
||
## Sharing the chunk directory
|
||
|
||
The REST and NC chunked surfaces can share `OXICLOUD_CHUNK_DIR` by
|
||
design. Each writer tags its output so they never interfere:
|
||
|
||
| Writer | On-disk name pattern |
|
||
|---|---|
|
||
| REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix |
|
||
| NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir |
|
||
|
||
The 24-hour orphan-session cleanup loop filters strictly on the
|
||
`oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory
|
||
that happens to live alongside chunked sessions.
|
||
|
||
## Quick verification
|
||
|
||
Boot the server with `RUST_LOG=info` and the first lines after the
|
||
banner include:
|
||
|
||
```
|
||
oxicloud: Upload limits loaded from config max_upload_size_mb=10240 chunk_max_bytes_mb=100
|
||
```
|
||
|
||
That confirms the upload-cap env vars were read. To confirm
|
||
directory placement, watch for chunk file creation under your
|
||
`OXICLOUD_CHUNK_DIR` (or its default `{STORAGE_PATH}/.uploads/`)
|
||
during a chunked upload — `ls` while a sync is in progress shows the
|
||
`{uuid}/chunk_NNNNNN` files appearing in real time.
|