feat(uploads): add OXICLOUD_CHUNK_DIR + documentation for admins
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
This commit is contained in:
@@ -12,6 +12,9 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
|
|||||||
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
||||||
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
||||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Maximum upload size in bytes (10 GB on 64-bit, 1 GB on 32-bit) |
|
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Maximum upload size in bytes (10 GB on 64-bit, 1 GB on 32-bit) |
|
||||||
|
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Distinct from `OXICLOUD_MAX_UPLOAD_SIZE`, which is the whole-file cap. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
|
| `OXICLOUD_UPLOAD_TMPDIR` | (OS temp dir) | Spool directory for non-chunked PUT bodies (`/api/files/upload`, WebDAV PUT). Point at a real-disk path on the same FS as `.blobs/` to avoid tmpfs OOMKill and make blob promotion an atomic rename. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
|
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Same-FS / NVMe placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
||||||
|
|
||||||
## Database
|
## Database
|
||||||
|
|||||||
@@ -0,0 +1,294 @@
|
|||||||
|
# Storage Fine Tuning
|
||||||
|
|
||||||
|
This page is for sysadmins who want to tune **where** OxiCloud spools
|
||||||
|
upload bodies and **why** the placement matters for throughput and
|
||||||
|
memory. The defaults work; the gains from a tuned layout are
|
||||||
|
significant on busy instances or constrained containers.
|
||||||
|
|
||||||
|
## The upload lifecycle in 30 seconds
|
||||||
|
|
||||||
|
Every upload moves through two stages:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─── direct (single-PUT) upload ────────┐
|
||||||
|
client ─┤ ├──► OxiCloud accepts the
|
||||||
|
└─── multi-chunk upload │ bytes into a SPOOL on
|
||||||
|
(`/api/uploads` / │ local disk.
|
||||||
|
`/dav/uploads/...`) │
|
||||||
|
│ Direct upload → OXICLOUD_UPLOAD_TMPDIR
|
||||||
|
│ Chunked upload → OXICLOUD_CHUNK_DIR
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────┐
|
||||||
|
│ Once the upload is │
|
||||||
|
│ complete (and verified │
|
||||||
|
│ if a checksum was │
|
||||||
|
│ supplied), OxiCloud │
|
||||||
|
│ MOVES the assembled │
|
||||||
|
│ blob into the configured│
|
||||||
|
│ STORAGE BACKEND: │
|
||||||
|
│ │
|
||||||
|
│ • local FS (.blobs/) │
|
||||||
|
│ • S3-compatible │
|
||||||
|
│ • Azure Blob │
|
||||||
|
└─────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
Two practical consequences:
|
||||||
|
|
||||||
|
- **The spool/chunk directories see write-heavy churn** during uploads —
|
||||||
|
fast disk (NVMe) and sufficient free space matter more here than on
|
||||||
|
the final storage backend.
|
||||||
|
- **The promotion from spool → storage is a `rename(2)` whenever
|
||||||
|
source and destination share a filesystem** (i.e. when the backend
|
||||||
|
is `local` and the spool dir is on the same FS as `.blobs/`). On
|
||||||
|
remote backends (S3, Azure) the promotion is always a network
|
||||||
|
upload from the local spool; placement of the spool still matters
|
||||||
|
for intake throughput but the "same FS" rule doesn't apply.
|
||||||
|
|
||||||
|
## Upload size caps — what each one bounds
|
||||||
|
|
||||||
|
Three independent caps control how large an upload OxiCloud will
|
||||||
|
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk
|
||||||
|
directories must be able to hold the worst case (max cap × concurrent
|
||||||
|
uploads).
|
||||||
|
|
||||||
|
| Variable | Default | What it caps |
|
||||||
|
|---|---|---|
|
||||||
|
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | Total file size — whether uploaded as one `PUT` or assembled from many chunks. The hard ceiling on any single file. |
|
||||||
|
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | A single chunked-PUT request body (`PATCH /api/uploads/{id}` or `PUT /dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file uploaded in 100 MB chunks needs ~50 PUTs, each bounded by this. |
|
||||||
|
| `OXICLOUD_MAX_UPLOAD_SIZE` (again, for the PUT path) | 10 GB | A single non-chunked PUT body. Same env var as the whole-file cap because for non-chunked uploads they're equivalent. |
|
||||||
|
|
||||||
|
### Why these matter for tmpfs sizing
|
||||||
|
|
||||||
|
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one
|
||||||
|
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**,
|
||||||
|
however, scales with the caps:
|
||||||
|
|
||||||
|
- **Direct PUT** (single-file): each in-flight upload spools the full
|
||||||
|
body to disk under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst
|
||||||
|
case disk = `OXICLOUD_MAX_UPLOAD_SIZE × concurrent_PUTs`.
|
||||||
|
- **Chunked upload**: each in-flight session accumulates chunks under
|
||||||
|
`OXICLOUD_CHUNK_DIR`, then assembles them into a single temp file
|
||||||
|
before promotion. Worst case disk per session = **2 × file_size**
|
||||||
|
(chunks + assembled file); total disk =
|
||||||
|
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_sessions`.
|
||||||
|
|
||||||
|
### Sizing examples
|
||||||
|
|
||||||
|
A 4 GB tmpfs serving a small team (~5 concurrent uploads):
|
||||||
|
|
||||||
|
| Setting | Disk worst case | Safe on 4 GB tmpfs? |
|
||||||
|
|---|---|---|
|
||||||
|
| `MAX_UPLOAD=10 GB`, no `CHUNK_MAX` tweak | 50 GB direct, 100 GB chunked | ❌ no — single upload OOMs the tmpfs |
|
||||||
|
| `MAX_UPLOAD=500 MB`, `CHUNK_MAX=50 MB` | 2.5 GB direct, 5 GB chunked | ⚠ direct fits, chunked overflows |
|
||||||
|
| `MAX_UPLOAD=300 MB`, `CHUNK_MAX=30 MB` | 1.5 GB direct, 3 GB chunked | ✅ both fit |
|
||||||
|
|
||||||
|
A real-disk volume (cheap, large):
|
||||||
|
|
||||||
|
| Setting | Disk worst case | Comment |
|
||||||
|
|---|---|---|
|
||||||
|
| `MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB` (defaults) | 100 GB chunked worst case | Fine on a 200+ GB volume; almost any real-disk setup |
|
||||||
|
| `MAX_UPLOAD=100 GB`, `CHUNK_MAX=500 MB` | 1 TB chunked worst case | Plausible for video archives; needs a dedicated upload volume |
|
||||||
|
|
||||||
|
### Choosing tmpfs vs real disk
|
||||||
|
|
||||||
|
| Constraint | Choice |
|
||||||
|
|---|---|
|
||||||
|
| Upload caps × concurrent users ≤ free RAM × 0.5 | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
|
||||||
|
| Upload caps × concurrent users > free RAM × 0.5 | **real disk** — same FS as `.blobs/` ideal |
|
||||||
|
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
|
||||||
|
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
|
||||||
|
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
|
||||||
|
|
||||||
|
The defaults (`MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB`) assume **real
|
||||||
|
disk**. Don't run the defaults against tmpfs unless you've sized it
|
||||||
|
for the worst case.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
| Variable | Default | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) |
|
||||||
|
| `OXICLOUD_UPLOAD_TMPDIR` | OS temp dir | Where non-chunked PUT bodies are spooled |
|
||||||
|
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate |
|
||||||
|
|
||||||
|
The two rules that matter most:
|
||||||
|
|
||||||
|
1. **Put all three on the same filesystem.** Blob promotion is an
|
||||||
|
atomic `rename(2)` when source and destination share an FS — cheap
|
||||||
|
and crash-safe. Across filesystems it becomes a full `read + write +
|
||||||
|
unlink`, multiplying the IO and widening the durability window.
|
||||||
|
2. **Don't leave the spool dir on tmpfs** (the default in many
|
||||||
|
containers). Spool bodies count against the cgroup memory limit
|
||||||
|
and can trigger OOMKill on multi-GB uploads.
|
||||||
|
|
||||||
|
## Where each upload surface spools
|
||||||
|
|
||||||
|
OxiCloud has several entry points that accept request bodies. They
|
||||||
|
land in different places by default:
|
||||||
|
|
||||||
|
| Surface | Default destination | Configurable via |
|
||||||
|
|---|---|---|
|
||||||
|
| REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` |
|
||||||
|
| REST chunked assemble (during `/complete`) | `{STORAGE_PATH}/.uploads/{upload_id}/assembled` | `OXICLOUD_CHUNK_DIR` |
|
||||||
|
| NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` |
|
||||||
|
| NextCloud chunked assemble (during `MOVE`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/.assembled` | `OXICLOUD_CHUNK_DIR` |
|
||||||
|
| Native WebDAV PUT (`PUT /webdav/{path}`) | OS temp dir (`/tmp`) | `OXICLOUD_UPLOAD_TMPDIR` |
|
||||||
|
| NextCloud single-file PUT (`PUT /dav/files/.../{path}`) | OS temp dir | `OXICLOUD_UPLOAD_TMPDIR` |
|
||||||
|
| REST multipart upload (`POST /api/files/upload`) | `{STORAGE_PATH}/.dedup_temp/upload-{uuid}` | `OXICLOUD_STORAGE_PATH` (subdir is hard-wired) |
|
||||||
|
| Final blob storage (after fsync + rename) | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
|
||||||
|
|
||||||
|
## Why placement matters
|
||||||
|
|
||||||
|
### 1. Same filesystem ⇒ promotion is a rename
|
||||||
|
|
||||||
|
OxiCloud uses **content-addressable storage**: the final blob path is
|
||||||
|
derived from the file's BLAKE3 hash, which can only be known after the
|
||||||
|
last byte arrives. So every upload writes to a temp location first,
|
||||||
|
then **promotes** the temp file to `.blobs/{ab}/{abc…}.blob` by way of
|
||||||
|
a `rename(2)` call.
|
||||||
|
|
||||||
|
- **Same FS:** `rename` is atomic, O(1), no data copy. Total upload
|
||||||
|
cost = body bytes received + one rename syscall. Crash-safe — the
|
||||||
|
blob either exists at the final path or doesn't.
|
||||||
|
- **Cross-FS:** the kernel can't `rename(2)` across filesystems. The
|
||||||
|
blob backend falls back to `fs::copy + fs::remove_file` (visible in
|
||||||
|
`local_blob_backend.rs` as the EXDEV handler). Total cost = body
|
||||||
|
bytes received + one full file copy. Doubles the IO bandwidth used
|
||||||
|
per upload and widens the durability window.
|
||||||
|
|
||||||
|
### 2. Spool off tmpfs
|
||||||
|
|
||||||
|
`tempfile::NamedTempFile::new()` (used when `OXICLOUD_UPLOAD_TMPDIR`
|
||||||
|
is unset) honors `$TMPDIR`, which in many container setups points at
|
||||||
|
**tmpfs** — RAM-backed storage. A 2 GB upload spool then consumes 2 GB
|
||||||
|
of memory until the rename promotes it to disk.
|
||||||
|
|
||||||
|
In a Kubernetes pod with a 4 GB memory limit, the OOMKiller wakes up
|
||||||
|
long before the upload finishes. With `OXICLOUD_UPLOAD_TMPDIR` pointed
|
||||||
|
at a real-disk directory, the spool's memory footprint stays at ~one
|
||||||
|
HTTP frame regardless of file size.
|
||||||
|
|
||||||
|
### 3. NVMe for the hot path
|
||||||
|
|
||||||
|
The chunked-upload session directory sees a LOT of small writes —
|
||||||
|
each chunk PUT writes a file, the progress bitmap is rewritten after
|
||||||
|
each PUT, the assemble step reads them all back in order. Pointing
|
||||||
|
`OXICLOUD_CHUNK_DIR` at an NVMe device is a substantial win on
|
||||||
|
deployments that handle large file uploads, even if the final blob
|
||||||
|
storage is on slower disk.
|
||||||
|
|
||||||
|
The same applies to `OXICLOUD_UPLOAD_TMPDIR` (single-file PUTs).
|
||||||
|
|
||||||
|
A common high-throughput layout:
|
||||||
|
|
||||||
|
- **NVMe** (small, fast): `OXICLOUD_CHUNK_DIR`, `OXICLOUD_UPLOAD_TMPDIR`
|
||||||
|
- **HDD or NAS** (large, cheap): `OXICLOUD_STORAGE_PATH`/`.blobs/`
|
||||||
|
|
||||||
|
Trade-off: the rename optimization (rule 1) DOESN'T apply across
|
||||||
|
filesystems. If you split the hot path off the blob filesystem, every
|
||||||
|
upload pays a full file copy on promotion. You have to choose
|
||||||
|
between **fast intake** and **zero-copy promotion**.
|
||||||
|
|
||||||
|
| Goal | Layout | Cost per upload |
|
||||||
|
|---|---|---|
|
||||||
|
| Fastest possible intake | NVMe chunk dir + HDD blobs | 1× write to NVMe + 1× read NVMe + 1× write to HDD (copy) |
|
||||||
|
| Lowest IO + crash safety | NVMe everything OR HDD everything | 1× write to disk + 1 rename (~0 cost) |
|
||||||
|
| Default (do nothing) | Everything under `STORAGE_PATH` on whatever FS that is | Depends on `STORAGE_PATH` placement |
|
||||||
|
|
||||||
|
For most deployments **"same FS everywhere"** wins. The NVMe-split is
|
||||||
|
useful when intake latency dominates the user experience and you can
|
||||||
|
afford the doubled IO.
|
||||||
|
|
||||||
|
## Recommended layouts
|
||||||
|
|
||||||
|
### Single-disk box (most common)
|
||||||
|
|
||||||
|
Defaults are fine. Optionally set `OXICLOUD_UPLOAD_TMPDIR` to keep
|
||||||
|
the PUT spool off `/tmp`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
|
||||||
|
OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/.spool
|
||||||
|
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
|
||||||
|
```
|
||||||
|
|
||||||
|
All three on the same filesystem → rename promotion → atomic and fast.
|
||||||
|
|
||||||
|
### Container with constrained memory
|
||||||
|
|
||||||
|
Critical: make sure neither spool sits on tmpfs.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OXICLOUD_STORAGE_PATH=/data
|
||||||
|
OXICLOUD_UPLOAD_TMPDIR=/data/.spool
|
||||||
|
OXICLOUD_CHUNK_DIR=/data/.uploads
|
||||||
|
```
|
||||||
|
|
||||||
|
If you can't mount a writable `/data`, at minimum bind-mount a real
|
||||||
|
volume at the spool dirs.
|
||||||
|
|
||||||
|
### Split-disk (NVMe intake + HDD blobs)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/
|
||||||
|
OXICLOUD_UPLOAD_TMPDIR=/mnt/nvme/oxi-spool
|
||||||
|
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
|
||||||
|
```
|
||||||
|
|
||||||
|
Faster intake; pays a copy on promotion. Worth it when uploads are
|
||||||
|
many small files (NVMe IOPS dominates) or when intake latency directly
|
||||||
|
hits user-visible UX.
|
||||||
|
|
||||||
|
## Sharing the spool and chunk directories
|
||||||
|
|
||||||
|
Pointing `OXICLOUD_UPLOAD_TMPDIR` and `OXICLOUD_CHUNK_DIR` at the
|
||||||
|
**same directory** is supported by design. Each writer tags its
|
||||||
|
output so the surfaces never interfere with each other:
|
||||||
|
|
||||||
|
| Writer | On-disk name pattern |
|
||||||
|
|---|---|
|
||||||
|
| PUT spool (single-file uploads) | `.tmpXXXXXXXX` — files (not directories), random suffix |
|
||||||
|
| REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix |
|
||||||
|
| NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir |
|
||||||
|
|
||||||
|
The 24-hour orphan-session cleanup loop filters strictly on the
|
||||||
|
`oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory
|
||||||
|
that happens to live alongside chunked sessions. The PUT spool's
|
||||||
|
`.tmpXXXX` files are files (not directories) and the NC subtree's
|
||||||
|
`nextcloud/` root has its own name — both are invisible to the
|
||||||
|
cleanup loop.
|
||||||
|
|
||||||
|
**Recommendation:** for new deployments, use separate directories
|
||||||
|
anyway (the defaults `.spool/` and `.uploads/` already do this) —
|
||||||
|
it makes disk-usage attribution clearer and keeps IOPS isolated when
|
||||||
|
both are busy. Shared directories are safe to use when disk layout
|
||||||
|
forces it.
|
||||||
|
|
||||||
|
## What's NOT yet configurable
|
||||||
|
|
||||||
|
- **REST multipart upload directory** (`POST /api/files/upload`) is
|
||||||
|
hard-wired to `{STORAGE_PATH}/.dedup_temp/`. It can't be moved
|
||||||
|
separately. Same-FS placement is automatic.
|
||||||
|
- **WOPI PutFile spool** (Office editor saves) uses the bare OS temp
|
||||||
|
dir without honoring `OXICLOUD_UPLOAD_TMPDIR`. This is a known
|
||||||
|
inconsistency and on the hardening backlog.
|
||||||
|
- **Per-user / per-drive spool directories** — all users share the
|
||||||
|
same `OXICLOUD_CHUNK_DIR` root today. Multi-tenant isolation
|
||||||
|
through separate spool dirs isn't supported.
|
||||||
|
|
||||||
|
## Quick verification
|
||||||
|
|
||||||
|
Boot the server with `RUST_LOG=info` and the first lines after the
|
||||||
|
banner include:
|
||||||
|
|
||||||
|
```
|
||||||
|
oxicloud: Upload limits loaded from config max_upload_size_mb=10240 chunk_max_bytes_mb=100
|
||||||
|
```
|
||||||
|
|
||||||
|
That confirms the upload-cap env vars were read. To confirm
|
||||||
|
directory placement, watch for chunk file creation under your
|
||||||
|
`OXICLOUD_CHUNK_DIR` (or its default `{STORAGE_PATH}/.uploads/`)
|
||||||
|
during a chunked upload — `ls` while a sync is in progress shows the
|
||||||
|
`{uuid}/chunk_NNNNNN` files appearing in real time.
|
||||||
@@ -224,6 +224,15 @@ pub struct StorageConfig {
|
|||||||
/// memory limit and can trigger OOMKill on large files). Env:
|
/// memory limit and can trigger OOMKill on large files). Env:
|
||||||
/// `OXICLOUD_UPLOAD_TMPDIR`.
|
/// `OXICLOUD_UPLOAD_TMPDIR`.
|
||||||
pub upload_temp_dir: Option<PathBuf>,
|
pub upload_temp_dir: Option<PathBuf>,
|
||||||
|
/// Root directory for chunked-upload sessions. When `Some`, chunks land
|
||||||
|
/// under `{chunk_dir}/{upload_id}/` (REST) and
|
||||||
|
/// `{chunk_dir}/nextcloud/{user}/{upload_id}/` (NC). When `None`, falls
|
||||||
|
/// back to `{root_dir}/.uploads/`. Pointing this at the **same
|
||||||
|
/// filesystem** as `.blobs/` keeps the final assembled-to-blob promotion
|
||||||
|
/// an atomic `rename(2)` rather than a full cross-FS copy; pointing it
|
||||||
|
/// at fast storage (NVMe) accelerates the chunk-write + assembly loop
|
||||||
|
/// independently of where final blobs live. Env: `OXICLOUD_CHUNK_DIR`.
|
||||||
|
pub chunk_dir: Option<PathBuf>,
|
||||||
/// Interval (seconds) of the background sweep that reconciles every user's
|
/// Interval (seconds) of the background sweep that reconciles every user's
|
||||||
/// cached `storage_used_bytes` with the real sum of their files. Keeps the
|
/// cached `storage_used_bytes` with the real sum of their files. Keeps the
|
||||||
/// quota fresh for all mutations without recomputing on the request path.
|
/// quota fresh for all mutations without recomputing on the request path.
|
||||||
@@ -368,6 +377,7 @@ impl Default for StorageConfig {
|
|||||||
max_upload_size: MAX_UPLOAD_SIZE,
|
max_upload_size: MAX_UPLOAD_SIZE,
|
||||||
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
|
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
|
||||||
upload_temp_dir: None,
|
upload_temp_dir: None,
|
||||||
|
chunk_dir: None,
|
||||||
usage_reconcile_secs: 600, // 10 minutes
|
usage_reconcile_secs: 600, // 10 minutes
|
||||||
backend: StorageBackendType::Local,
|
backend: StorageBackendType::Local,
|
||||||
s3: None,
|
s3: None,
|
||||||
@@ -1245,6 +1255,16 @@ impl AppConfig {
|
|||||||
{
|
{
|
||||||
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
|
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
|
||||||
}
|
}
|
||||||
|
// Chunked-upload session root — separate from the PUT spool because
|
||||||
|
// chunked sessions accumulate disk on long uploads (multi-chunk
|
||||||
|
// resumable transfers) while PUT spool is short-lived. Sysadmins
|
||||||
|
// commonly want one of them on fast/local storage (NVMe) and the
|
||||||
|
// other on bulk storage; this knob lets that be expressed.
|
||||||
|
if let Ok(dir) = env::var("OXICLOUD_CHUNK_DIR")
|
||||||
|
&& !dir.trim().is_empty()
|
||||||
|
{
|
||||||
|
config.storage.chunk_dir = Some(PathBuf::from(dir.trim()));
|
||||||
|
}
|
||||||
|
|
||||||
// Background storage-usage reconciliation interval
|
// Background storage-usage reconciliation interval
|
||||||
if let Ok(secs) =
|
if let Ok(secs) =
|
||||||
|
|||||||
+27
-4
@@ -171,11 +171,23 @@ impl AppServiceFactory {
|
|||||||
// Initialize thumbnail directories
|
// Initialize thumbnail directories
|
||||||
thumbnail_service.initialize().await?;
|
thumbnail_service.initialize().await?;
|
||||||
|
|
||||||
// Chunked upload service for large files (>10MB)
|
// Chunked upload service for large files (>10MB).
|
||||||
let chunked_temp_dir = std::path::PathBuf::from(&self.storage_path).join(".uploads");
|
// Root for both REST (`/api/uploads/...`) and NC (`/dav/uploads/...`)
|
||||||
|
// chunked sessions: honour `OXICLOUD_CHUNK_DIR` when set so sysadmins
|
||||||
|
// can put session directories on fast storage (NVMe) or on the same
|
||||||
|
// filesystem as `.blobs/` (turns the final blob promotion into an
|
||||||
|
// atomic rename instead of a cross-FS copy). Falls back to
|
||||||
|
// `{storage_path}/.uploads/` when unset — backwards-compatible with
|
||||||
|
// every existing deployment.
|
||||||
|
let chunk_root = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.chunk_dir
|
||||||
|
.clone()
|
||||||
|
.unwrap_or_else(|| std::path::PathBuf::from(&self.storage_path).join(".uploads"));
|
||||||
let chunked_upload_service = Arc::new(
|
let chunked_upload_service = Arc::new(
|
||||||
crate::infrastructure::services::chunked_upload_service::ChunkedUploadService::new(
|
crate::infrastructure::services::chunked_upload_service::ChunkedUploadService::new(
|
||||||
chunked_temp_dir,
|
chunk_root.clone(),
|
||||||
)
|
)
|
||||||
.await,
|
.await,
|
||||||
);
|
);
|
||||||
@@ -870,7 +882,18 @@ impl AppServiceFactory {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
let chunk_base = self.storage_path.join(".uploads/nextcloud");
|
// NC chunked-upload sessions root. Honour `OXICLOUD_CHUNK_DIR`
|
||||||
|
// (same env var that the REST chunked service uses) so a single
|
||||||
|
// value covers both surfaces and they stay co-located on one
|
||||||
|
// filesystem; fall back to `{storage_path}/.uploads/` to match
|
||||||
|
// the legacy layout.
|
||||||
|
let chunk_root = self
|
||||||
|
.config
|
||||||
|
.storage
|
||||||
|
.chunk_dir
|
||||||
|
.clone()
|
||||||
|
.unwrap_or_else(|| self.storage_path.join(".uploads"));
|
||||||
|
let chunk_base = chunk_root.join("nextcloud");
|
||||||
let chunked_uploads = Arc::new(NextcloudChunkedUploadService::new(chunk_base));
|
let chunked_uploads = Arc::new(NextcloudChunkedUploadService::new(chunk_base));
|
||||||
|
|
||||||
let file_id_repo = Arc::new(
|
let file_id_repo = Arc::new(
|
||||||
|
|||||||
@@ -44,10 +44,32 @@ pub const MAX_PARALLEL_CHUNKS: usize = 6;
|
|||||||
/// Upload session expiration time (24 h)
|
/// Upload session expiration time (24 h)
|
||||||
const SESSION_EXPIRATION: Duration = Duration::from_secs(24 * 60 * 60);
|
const SESSION_EXPIRATION: Duration = Duration::from_secs(24 * 60 * 60);
|
||||||
|
|
||||||
|
/// Prefix every session directory name with this string so the cleanup
|
||||||
|
/// loop can be safely co-located with unrelated writers (PUT spool
|
||||||
|
/// tempfiles, the NC chunked subtree, anything else a sysadmin places
|
||||||
|
/// under the same `OXICLOUD_CHUNK_DIR`). The orphan-cleanup scan
|
||||||
|
/// filters by this prefix, so non-OxiCloud directories sharing the
|
||||||
|
/// root are never touched.
|
||||||
|
const SESSION_DIR_PREFIX: &str = "oxi-chunk-";
|
||||||
|
|
||||||
/// Sentinel file names inside each session directory
|
/// Sentinel file names inside each session directory
|
||||||
const SESSION_META_FILE: &str = "session.json";
|
const SESSION_META_FILE: &str = "session.json";
|
||||||
const PROGRESS_FILE: &str = "progress.bin";
|
const PROGRESS_FILE: &str = "progress.bin";
|
||||||
|
|
||||||
|
/// Build a session directory name from an upload_id by attaching the
|
||||||
|
/// well-known prefix. Symmetric with [`strip_session_prefix`].
|
||||||
|
fn session_dir_name(upload_id: &str) -> String {
|
||||||
|
format!("{}{}", SESSION_DIR_PREFIX, upload_id)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract the upload_id from a session directory name. Returns
|
||||||
|
/// `None` when the directory wasn't created by this service (no
|
||||||
|
/// `oxi-chunk-` prefix) — the recovery and cleanup paths use this to
|
||||||
|
/// skip foreign directories cohabiting under `OXICLOUD_CHUNK_DIR`.
|
||||||
|
fn strip_session_prefix(dir_name: &str) -> Option<&str> {
|
||||||
|
dir_name.strip_prefix(SESSION_DIR_PREFIX)
|
||||||
|
}
|
||||||
|
|
||||||
// ─── Serialisable types ──────────────────────────────────────────────────────
|
// ─── Serialisable types ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
/// Chunk status
|
/// Chunk status
|
||||||
@@ -252,6 +274,17 @@ impl ChunkedUploadService {
|
|||||||
if !dir.is_dir() {
|
if !dir.is_dir() {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// Only consider directories WE created — anything without the
|
||||||
|
// `oxi-chunk-` prefix belongs to a sibling writer (NC subtree,
|
||||||
|
// PUT spool tempfiles, sysadmin-placed dirs) and must be left
|
||||||
|
// strictly alone. See `SESSION_DIR_PREFIX`.
|
||||||
|
let dir_name = match dir.file_name().and_then(|n| n.to_str()) {
|
||||||
|
Some(n) => n,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
if strip_session_prefix(dir_name).is_none() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
let meta_path = dir.join(SESSION_META_FILE);
|
let meta_path = dir.join(SESSION_META_FILE);
|
||||||
let meta_bytes = match fs::read(&meta_path).await {
|
let meta_bytes = match fs::read(&meta_path).await {
|
||||||
@@ -346,21 +379,32 @@ impl ChunkedUploadService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Also clean orphaned temp directories (no session.json or very old)
|
// Also clean orphaned temp directories (no session.json or very old).
|
||||||
|
// Filter strictly on the `oxi-chunk-` prefix so we never touch
|
||||||
|
// sibling directories sharing `OXICLOUD_CHUNK_DIR` (NC subtree
|
||||||
|
// `nextcloud/`, PUT spool tempfiles which are files anyway,
|
||||||
|
// operator-placed dirs). Without the prefix filter this loop
|
||||||
|
// would silently delete anything older than 24 h sitting at the
|
||||||
|
// root of the chunked-upload dir.
|
||||||
if let Ok(mut entries) = fs::read_dir(&temp_base_dir).await {
|
if let Ok(mut entries) = fs::read_dir(&temp_base_dir).await {
|
||||||
while let Ok(Some(entry)) = entries.next_entry().await {
|
while let Ok(Some(entry)) = entries.next_entry().await {
|
||||||
let path = entry.path();
|
let path = entry.path();
|
||||||
if path.is_dir() {
|
if !path.is_dir() {
|
||||||
let dir_name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
|
continue;
|
||||||
|
}
|
||||||
|
let dir_name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
|
||||||
|
let upload_id = match strip_session_prefix(dir_name) {
|
||||||
|
Some(id) => id,
|
||||||
|
None => continue, // not ours — never touch
|
||||||
|
};
|
||||||
|
|
||||||
if !sessions.contains_key(dir_name)
|
if !sessions.contains_key(upload_id)
|
||||||
&& let Ok(metadata) = fs::metadata(&path).await
|
&& let Ok(metadata) = fs::metadata(&path).await
|
||||||
&& let Ok(modified) = metadata.modified()
|
&& let Ok(modified) = metadata.modified()
|
||||||
&& modified.elapsed().unwrap_or_default() > SESSION_EXPIRATION
|
&& modified.elapsed().unwrap_or_default() > SESSION_EXPIRATION
|
||||||
{
|
{
|
||||||
let _ = fs::remove_dir_all(&path).await;
|
let _ = fs::remove_dir_all(&path).await;
|
||||||
tracing::info!("🧹 Cleaned orphaned upload dir: {:?}", path);
|
tracing::info!("🧹 Cleaned orphaned upload dir: {:?}", path);
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -396,8 +440,11 @@ impl ChunkedUploadService {
|
|||||||
let chunk_size = chunk_size.unwrap_or(DEFAULT_CHUNK_SIZE);
|
let chunk_size = chunk_size.unwrap_or(DEFAULT_CHUNK_SIZE);
|
||||||
let chunk_count = UploadSession::calculate_chunk_count(total_size, chunk_size);
|
let chunk_count = UploadSession::calculate_chunk_count(total_size, chunk_size);
|
||||||
|
|
||||||
// Create temp directory for chunks
|
// Create temp directory for chunks. The `oxi-chunk-` prefix
|
||||||
let temp_dir = self.temp_base_dir.join(&upload_id);
|
// tags the directory as belonging to this service so the
|
||||||
|
// shared-`OXICLOUD_CHUNK_DIR` story holds — see
|
||||||
|
// `SESSION_DIR_PREFIX` for the full rationale.
|
||||||
|
let temp_dir = self.temp_base_dir.join(session_dir_name(&upload_id));
|
||||||
fs::create_dir_all(&temp_dir)
|
fs::create_dir_all(&temp_dir)
|
||||||
.await
|
.await
|
||||||
.map_err(|e| format!("Failed to create temp directory: {e}"))?;
|
.map_err(|e| format!("Failed to create temp directory: {e}"))?;
|
||||||
@@ -1254,7 +1301,7 @@ mod tests {
|
|||||||
.expect("upload_chunk 0");
|
.expect("upload_chunk 0");
|
||||||
|
|
||||||
// Verify files exist on disk
|
// Verify files exist on disk
|
||||||
let session_dir = base.join(&upload_id);
|
let session_dir = base.join(session_dir_name(&upload_id));
|
||||||
assert!(session_dir.join(SESSION_META_FILE).exists());
|
assert!(session_dir.join(SESSION_META_FILE).exists());
|
||||||
assert!(session_dir.join(PROGRESS_FILE).exists());
|
assert!(session_dir.join(PROGRESS_FILE).exists());
|
||||||
assert!(session_dir.join("chunk_000000").exists());
|
assert!(session_dir.join("chunk_000000").exists());
|
||||||
@@ -1366,7 +1413,7 @@ mod tests {
|
|||||||
.await
|
.await
|
||||||
.expect("create");
|
.expect("create");
|
||||||
|
|
||||||
let session_dir = base.join(&resp.upload_id);
|
let session_dir = base.join(session_dir_name(&resp.upload_id));
|
||||||
assert!(session_dir.exists());
|
assert!(session_dir.exists());
|
||||||
|
|
||||||
service
|
service
|
||||||
@@ -1385,8 +1432,10 @@ mod tests {
|
|||||||
let base = std::env::temp_dir().join(format!("oxicloud_test_{}", Uuid::new_v4()));
|
let base = std::env::temp_dir().join(format!("oxicloud_test_{}", Uuid::new_v4()));
|
||||||
let _ = fs::create_dir_all(&base).await;
|
let _ = fs::create_dir_all(&base).await;
|
||||||
|
|
||||||
// Manually create an expired session on disk
|
// Manually create an expired session on disk. The dir name MUST
|
||||||
let session_dir = base.join("expired-session");
|
// carry the `oxi-chunk-` prefix or recovery will (correctly) skip
|
||||||
|
// it as belonging to another writer co-located in chunk_dir.
|
||||||
|
let session_dir = base.join(session_dir_name("expired-session"));
|
||||||
let _ = fs::create_dir_all(&session_dir).await;
|
let _ = fs::create_dir_all(&session_dir).await;
|
||||||
|
|
||||||
let expired_session = UploadSession {
|
let expired_session = UploadSession {
|
||||||
@@ -1429,7 +1478,7 @@ mod tests {
|
|||||||
let base = std::env::temp_dir().join(format!("oxicloud_test_{}", Uuid::new_v4()));
|
let base = std::env::temp_dir().join(format!("oxicloud_test_{}", Uuid::new_v4()));
|
||||||
let _ = fs::create_dir_all(&base).await;
|
let _ = fs::create_dir_all(&base).await;
|
||||||
|
|
||||||
let session_dir = base.join("partial-session");
|
let session_dir = base.join(session_dir_name("partial-session"));
|
||||||
let _ = fs::create_dir_all(&session_dir).await;
|
let _ = fs::create_dir_all(&session_dir).await;
|
||||||
|
|
||||||
let session = UploadSession {
|
let session = UploadSession {
|
||||||
|
|||||||
Reference in New Issue
Block a user