Oxicloud/docs/config/storage-fine-tuning.md

# Storage Fine Tuning

This page is for sysadmins who want to tune **where** OxiCloud places
upload data on disk and **why** the placement matters for throughput
and memory. The defaults work; the gains from a tuned layout are
significant on busy instances or constrained containers.

## The upload lifecycle in 30 seconds

```
        ┌─── direct (single-PUT / multipart) ───┐
client ─┤                                       │    Streamed DIRECTLY into the
        │                                       ├──► content-addressable store:
        │                                       │    CDC chunking + BLAKE3 +
        │                                       │    dedup checks happen while
        │                                       │    the bytes arrive. No spool
        │                                       │    file, no re-read; chunks
        │                                       │    the store already has are
        │                                       │    never written at all.
        │                                       │
        └─── multi-chunk upload ────────────────┤    Chunk PARTS accumulate on
             (`/api/uploads` /                  │    disk under OXICLOUD_CHUNK_DIR
              `/dav/uploads/...`)               │    until /complete, which
                                                │    streams them (in order)
                                                ▼    through the same CDC path.
                                  ┌─────────────────────────┐
                                  │ STORAGE BACKEND          │
                                  │  • local FS (.blobs/)    │
                                  │  • S3-compatible         │
                                  │  • Azure Blob            │
                                  └─────────────────────────┘
```

Two practical consequences:

- **Direct uploads no longer use a spool directory.** Each uploaded
  byte is written at most once — straight into the blob backend as a
  CDC chunk. Re-uploads and edited files write only the chunks the
  store doesn't already have.
- **The chunk-session directory sees write-heavy churn** during
  multi-chunk uploads — fast disk (NVMe) and sufficient free space
  matter more here than on the final storage backend.

## Upload size caps — what each one bounds

Three independent caps control how large an upload OxiCloud will
accept. Pick them with disk sizing in mind: the chunk directory must
be able to hold the worst case (cap × concurrent sessions).

| Variable | Default | What it caps | When it fires |
|---|---|---|---|
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `PUT /webdav/...` and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |

### Recommendation: prefer chunked uploads for large files

The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
Files between those two caps can only succeed via the chunked
protocol. The reason is **resilience**: a direct PUT at 95 % of 5 GB
that drops loses everything (the partially ingested chunks are
reclaimed by GC, but the client must restart from byte 0). The same
drop on a chunked upload loses one ~5 MB chunk; the client retries
that chunk and continues. NextCloud desktop and the OxiCloud web UI
already switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).

### Disk sizing

OxiCloud streams bodies frame-by-frame, so **RAM** is bounded
(~10 MB per in-flight upload for the CDC ingest buffers) regardless
of the caps. **Disk space** scales with the caps:

- **Direct PUT / multipart**: no transient spool. Bytes land directly
  in the blob backend as deduplicated chunks; worst-case extra disk
  per upload is the file's own (deduplicated) size — the same space
  the stored file occupies afterwards.
- **Chunked upload**: each in-flight session accumulates its chunk
  parts under `OXICLOUD_CHUNK_DIR` until `/complete` streams them
  into the blob store and the session is cleaned up. Worst case disk
  per session = **file_size** (the parts); total =
  `OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.

| Settings | Chunked worst case (5 sessions) | Safe on 4 GB volume? |
|---|---|---|
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB | 50 GB | ❌ overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB | 2.5 GB | ✅ fits |

### Don't put `OXICLOUD_CHUNK_DIR` on tmpfs

In many container setups the OS temp dir is **tmpfs** — RAM-backed
storage that counts against the cgroup memory limit. A few concurrent
multi-GB chunked sessions on tmpfs will wake the OOMKiller long
before the uploads finish. Point `OXICLOUD_CHUNK_DIR` at a real-disk
directory in containers.

## TL;DR

| Variable | Default | Purpose |
|---|---|---|
| `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) |
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate |

The two rules that matter most:

1. **Keep `OXICLOUD_CHUNK_DIR` off tmpfs** (the default in many
   containers) — chunk parts count against the cgroup memory limit
   and can trigger OOMKill on multi-GB uploads.
2. **NVMe for the chunk dir pays off** on deployments with heavy
   large-file traffic: each chunk PUT writes a file and the progress
   bitmap, and `/complete` reads them all back in order.

## Where each upload surface writes

| Surface | Default destination | Configurable via |
|---|---|---|
| REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` |
| NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` |
| Direct PUT / multipart / WOPI / chunked `/complete` | straight into the blob backend (CDC chunks) | `OXICLOUD_STORAGE_PATH` (local backend) |
| Final blob storage | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |

The local blob backend stages each chunk write under
`{STORAGE_PATH}/.dedup_temp/` and promotes it with an atomic
`rename(2)` — both directories live under `OXICLOUD_STORAGE_PATH`,
so same-filesystem placement (and therefore atomic promotion) is
automatic and not separately configurable.

## Recommended layouts

### Single-disk box (most common)

Defaults are fine:

```bash
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
```

### Container with constrained memory

Critical: make sure the chunk dir doesn't sit on tmpfs.

```bash
OXICLOUD_STORAGE_PATH=/data
OXICLOUD_CHUNK_DIR=/data/.uploads
```

### Split-disk (NVMe intake + HDD blobs)

```bash
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud      # .blobs/ + .dedup_temp/
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
```

Chunk parts land on NVMe (fast PUTs, fast `/complete` read-back);
the deduplicated chunks are written once to the HDD-backed blob
store as `/complete` streams through them.

## Sharing the chunk directory

The REST and NC chunked surfaces can share `OXICLOUD_CHUNK_DIR` by
design. Each writer tags its output so they never interfere:

| Writer | On-disk name pattern |
|---|---|
| REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix |
| NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir |

The 24-hour orphan-session cleanup loop filters strictly on the
`oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory
that happens to live alongside chunked sessions.

## Quick verification

Boot the server with `RUST_LOG=info` and the first lines after the
banner include:

```
oxicloud: Upload limits loaded from config max_upload_size_mb=10240 chunk_max_bytes_mb=100
```

That confirms the upload-cap env vars were read. To confirm
directory placement, watch for chunk file creation under your
`OXICLOUD_CHUNK_DIR` (or its default `{STORAGE_PATH}/.uploads/`)
during a chunked upload — `ls` while a sync is in progress shows the
`{uuid}/chunk_NNNNNN` files appearing in real time.