Claude
e3f04d58aa
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
9.1 KiB
Storage Fine Tuning
This page is for sysadmins who want to tune where OxiCloud places upload data on disk and why the placement matters for throughput and memory. The defaults work; the gains from a tuned layout are significant on busy instances or constrained containers.
The upload lifecycle in 30 seconds
┌─── direct (single-PUT / multipart) ───┐
client ─┤ │ Streamed DIRECTLY into the
│ ├──► content-addressable store:
│ │ CDC chunking + BLAKE3 +
│ │ dedup checks happen while
│ │ the bytes arrive. No spool
│ │ file, no re-read; chunks
│ │ the store already has are
│ │ never written at all.
│ │
└─── multi-chunk upload ────────────────┤ Chunk PARTS accumulate on
(`/api/uploads` / │ disk under OXICLOUD_CHUNK_DIR
`/dav/uploads/...`) │ until /complete, which
│ streams them (in order)
▼ through the same CDC path.
┌─────────────────────────┐
│ STORAGE BACKEND │
│ • local FS (.blobs/) │
│ • S3-compatible │
│ • Azure Blob │
└─────────────────────────┘
Two practical consequences:
- Direct uploads no longer use a spool directory. Each uploaded byte is written at most once — straight into the blob backend as a CDC chunk. Re-uploads and edited files write only the chunks the store doesn't already have.
- The chunk-session directory sees write-heavy churn during multi-chunk uploads — fast disk (NVMe) and sufficient free space matter more here than on the final storage backend.
Upload size caps — what each one bounds
Three independent caps control how large an upload OxiCloud will accept. Pick them with disk sizing in mind: the chunk directory must be able to hold the worst case (cap × concurrent sessions).
| Variable | Default | What it caps | When it fires |
|---|---|---|---|
OXICLOUD_MAX_UPLOAD_SIZE |
10 GB | Whole-file ceiling. Applies to both direct PUT (per-body) and chunked uploads (declared total_size). The absolute upper bound on any single file in OxiCloud. |
Chunked: at POST /api/uploads against the JSON-declared total_size, before any chunk is uploaded. Direct PUT: indirectly via OXICLOUD_DIRECT_PUT_MAX_BYTES, which is expected to be ≤ OXICLOUD_MAX_UPLOAD_SIZE. |
OXICLOUD_DIRECT_PUT_MAX_BYTES |
1 GiB | Non-chunked PUT body. Per-request cap for PUT /webdav/... and PUT /remote.php/dav/files/.../.... Set below OXICLOUD_MAX_UPLOAD_SIZE so larger files are pushed onto the chunked protocol — which is resumable on failure. |
During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
OXICLOUD_CHUNK_MAX_BYTES |
100 MB | Per-chunk body in a chunked-upload session (PATCH /api/uploads/{id} or PUT /remote.php/dav/uploads/.../chunk). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. |
During chunk-body streaming. Excess → 413. |
Recommendation: prefer chunked uploads for large files
The defaults (OXICLOUD_DIRECT_PUT_MAX_BYTES = 1 GiB, well below
OXICLOUD_MAX_UPLOAD_SIZE = 10 GB) are deliberately asymmetric.
Files between those two caps can only succeed via the chunked
protocol. The reason is resilience: a direct PUT at 95 % of 5 GB
that drops loses everything (the partially ingested chunks are
reclaimed by GC, but the client must restart from byte 0). The same
drop on a chunked upload loses one ~5 MB chunk; the client retries
that chunk and continues. NextCloud desktop and the OxiCloud web UI
already switch to chunked at ~10 MB (CHUNKED_UPLOAD_THRESHOLD).
Disk sizing
OxiCloud streams bodies frame-by-frame, so RAM is bounded (~10 MB per in-flight upload for the CDC ingest buffers) regardless of the caps. Disk space scales with the caps:
- Direct PUT / multipart: no transient spool. Bytes land directly in the blob backend as deduplicated chunks; worst-case extra disk per upload is the file's own (deduplicated) size — the same space the stored file occupies afterwards.
- Chunked upload: each in-flight session accumulates its chunk
parts under
OXICLOUD_CHUNK_DIRuntil/completestreams them into the blob store and the session is cleaned up. Worst case disk per session = file_size (the parts); total =OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions.
| Settings | Chunked worst case (5 sessions) | Safe on 4 GB volume? |
|---|---|---|
Defaults: OXICLOUD_MAX_UPLOAD_SIZE=10 GB |
50 GB | ❌ overflows |
OXICLOUD_MAX_UPLOAD_SIZE=500 MB |
2.5 GB | ✅ fits |
Don't put OXICLOUD_CHUNK_DIR on tmpfs
In many container setups the OS temp dir is tmpfs — RAM-backed
storage that counts against the cgroup memory limit. A few concurrent
multi-GB chunked sessions on tmpfs will wake the OOMKiller long
before the uploads finish. Point OXICLOUD_CHUNK_DIR at a real-disk
directory in containers.
TL;DR
| Variable | Default | Purpose |
|---|---|---|
OXICLOUD_STORAGE_PATH |
./storage |
Where .blobs/ lives (the canonical content store) |
OXICLOUD_CHUNK_DIR |
{STORAGE_PATH}/.uploads |
Where chunked-upload sessions accumulate |
The two rules that matter most:
- Keep
OXICLOUD_CHUNK_DIRoff tmpfs (the default in many containers) — chunk parts count against the cgroup memory limit and can trigger OOMKill on multi-GB uploads. - NVMe for the chunk dir pays off on deployments with heavy
large-file traffic: each chunk PUT writes a file and the progress
bitmap, and
/completereads them all back in order.
Where each upload surface writes
| Surface | Default destination | Configurable via |
|---|---|---|
REST chunked PUT (PATCH /api/uploads/{id}) |
{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN |
OXICLOUD_CHUNK_DIR |
NextCloud chunked PUT (PUT /dav/uploads/.../chunk) |
{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name} |
OXICLOUD_CHUNK_DIR |
Direct PUT / multipart / WOPI / chunked /complete |
straight into the blob backend (CDC chunks) | OXICLOUD_STORAGE_PATH (local backend) |
| Final blob storage | {STORAGE_PATH}/.blobs/{ab}/{abc…}.blob |
OXICLOUD_STORAGE_PATH |
The local blob backend stages each chunk write under
{STORAGE_PATH}/.dedup_temp/ and promotes it with an atomic
rename(2) — both directories live under OXICLOUD_STORAGE_PATH,
so same-filesystem placement (and therefore atomic promotion) is
automatic and not separately configurable.
Recommended layouts
Single-disk box (most common)
Defaults are fine:
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
Container with constrained memory
Critical: make sure the chunk dir doesn't sit on tmpfs.
OXICLOUD_STORAGE_PATH=/data
OXICLOUD_CHUNK_DIR=/data/.uploads
Split-disk (NVMe intake + HDD blobs)
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
Chunk parts land on NVMe (fast PUTs, fast /complete read-back);
the deduplicated chunks are written once to the HDD-backed blob
store as /complete streams through them.
Sharing the chunk directory
The REST and NC chunked surfaces can share OXICLOUD_CHUNK_DIR by
design. Each writer tags its output so they never interfere:
| Writer | On-disk name pattern |
|---|---|
| REST chunked sessions | oxi-chunk-{uuid}/ — directories with a well-known prefix |
| NC chunked subtree | nextcloud/{user}/{uuid}/ — under its own root subdir |
The 24-hour orphan-session cleanup loop filters strictly on the
oxi-chunk- prefix, so it can NEVER delete a non-OxiCloud directory
that happens to live alongside chunked sessions.
Quick verification
Boot the server with RUST_LOG=info and the first lines after the
banner include:
oxicloud: Upload limits loaded from config max_upload_size_mb=10240 chunk_max_bytes_mb=100
That confirms the upload-cap env vars were read. To confirm
directory placement, watch for chunk file creation under your
OXICLOUD_CHUNK_DIR (or its default {STORAGE_PATH}/.uploads/)
during a chunked upload — ls while a sync is in progress shows the
{uuid}/chunk_NNNNNN files appearing in real time.