feat(OXICLOUD_DIRECT_PUT_MAX_BYTES): add a security limit on direct PUT
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
This commit is contained in:
+3
-2
@@ -11,8 +11,9 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
|
|||||||
| `OXICLOUD_SERVER_PORT` | `8086` | Server port |
|
| `OXICLOUD_SERVER_PORT` | `8086` | Server port |
|
||||||
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
||||||
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
||||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Maximum upload size in bytes (10 GB on 64-bit, 1 GB on 32-bit) |
|
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Whole-file size ceiling, in bytes (10 GB on 64-bit, 1 GB on 32-bit). Applies to BOTH direct PUTs (per-request body) and chunked uploads (declared `total_size`, checked upfront at session creation). |
|
||||||
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Distinct from `OXICLOUD_MAX_UPLOAD_SIZE`, which is the whole-file cap. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | `1073741824` | Per-request cap for non-chunked PUT bodies, in bytes (1 GiB). Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol (resumable on failure). See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
|
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Per-chunk cap, independent of `OXICLOUD_MAX_UPLOAD_SIZE` (whole-file cap). See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
| `OXICLOUD_UPLOAD_TMPDIR` | (OS temp dir) | Spool directory for non-chunked PUT bodies (`/api/files/upload`, WebDAV PUT). Point at a real-disk path on the same FS as `.blobs/` to avoid tmpfs OOMKill and make blob promotion an atomic rename. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
| `OXICLOUD_UPLOAD_TMPDIR` | (OS temp dir) | Spool directory for non-chunked PUT bodies (`/api/files/upload`, WebDAV PUT). Point at a real-disk path on the same FS as `.blobs/` to avoid tmpfs OOMKill and make blob promotion an atomic rename. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Same-FS / NVMe placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
|
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Same-FS / NVMe placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||||
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
||||||
|
|||||||
@@ -50,60 +50,85 @@ Two practical consequences:
|
|||||||
|
|
||||||
Three independent caps control how large an upload OxiCloud will
|
Three independent caps control how large an upload OxiCloud will
|
||||||
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk
|
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk
|
||||||
directories must be able to hold the worst case (max cap × concurrent
|
directories must be able to hold the worst case (cap × concurrent
|
||||||
uploads).
|
uploads).
|
||||||
|
|
||||||
| Variable | Default | What it caps |
|
| Variable | Default | What it caps | When it fires |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | Total file size — whether uploaded as one `PUT` or assembled from many chunks. The hard ceiling on any single file. |
|
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
|
||||||
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | A single chunked-PUT request body (`PATCH /api/uploads/{id}` or `PUT /dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file uploaded in 100 MB chunks needs ~50 PUTs, each bounded by this. |
|
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `POST /api/files/upload`, `PUT /webdav/...`, and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
|
||||||
| `OXICLOUD_MAX_UPLOAD_SIZE` (again, for the PUT path) | 10 GB | A single non-chunked PUT body. Same env var as the whole-file cap because for non-chunked uploads they're equivalent. |
|
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |
|
||||||
|
|
||||||
### Why these matter for tmpfs sizing
|
### Recommendation: prefer chunked uploads for large files
|
||||||
|
|
||||||
|
The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
|
||||||
|
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
|
||||||
|
Files between those two caps can only succeed via the chunked
|
||||||
|
protocol. Three reasons to keep them that way:
|
||||||
|
|
||||||
|
- **Resilience.** A direct PUT at 95 % of 5 GB that drops loses
|
||||||
|
everything. The same drop on a chunked upload loses one ~5 MB
|
||||||
|
chunk; the client retries that chunk and continues.
|
||||||
|
- **Memory + disk pressure.** Direct PUT spools the full body to
|
||||||
|
disk per request. Ten concurrent 5 GB direct PUTs use up to 50 GB
|
||||||
|
of transient spool disk. Chunked spreads each upload across many
|
||||||
|
small PATCHes; per-request resource use stays bounded by
|
||||||
|
`OXICLOUD_CHUNK_MAX_BYTES`.
|
||||||
|
- **Convention.** NextCloud desktop and the OxiCloud web UI already
|
||||||
|
switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
|
||||||
|
|
||||||
|
### Why caps matter for tmpfs sizing
|
||||||
|
|
||||||
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one
|
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one
|
||||||
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**,
|
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**,
|
||||||
however, scales with the caps:
|
however, scales with the caps:
|
||||||
|
|
||||||
- **Direct PUT** (single-file): each in-flight upload spools the full
|
- **Direct PUT**: each in-flight upload spools the full body to disk
|
||||||
body to disk under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst
|
under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst case disk =
|
||||||
case disk = `OXICLOUD_MAX_UPLOAD_SIZE × concurrent_PUTs`.
|
`OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs`.
|
||||||
- **Chunked upload**: each in-flight session accumulates chunks under
|
- **Chunked upload**: each in-flight session accumulates chunks
|
||||||
`OXICLOUD_CHUNK_DIR`, then assembles them into a single temp file
|
under `OXICLOUD_CHUNK_DIR`, then assembles them into a single temp
|
||||||
before promotion. Worst case disk per session = **2 × file_size**
|
file before promotion. Worst case disk per session = **2 ×
|
||||||
(chunks + assembled file); total disk =
|
file_size** (chunks + assembled file); total disk =
|
||||||
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_sessions`.
|
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
|
||||||
|
|
||||||
|
The chunked formula uses `OXICLOUD_MAX_UPLOAD_SIZE` because that's
|
||||||
|
what bounds the declared `total_size` at session creation. The
|
||||||
|
direct-PUT formula uses the smaller `OXICLOUD_DIRECT_PUT_MAX_BYTES`
|
||||||
|
since that's what bounds each direct PUT body.
|
||||||
|
|
||||||
### Sizing examples
|
### Sizing examples
|
||||||
|
|
||||||
A 4 GB tmpfs serving a small team (~5 concurrent uploads):
|
A 4 GB tmpfs serving a small team (5 concurrent direct PUTs OR 5
|
||||||
|
concurrent chunked sessions):
|
||||||
|
|
||||||
| Setting | Disk worst case | Safe on 4 GB tmpfs? |
|
| Settings | Direct-PUT worst case | Chunked worst case | Safe on 4 GB tmpfs? |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| `MAX_UPLOAD=10 GB`, no `CHUNK_MAX` tweak | 50 GB direct, 100 GB chunked | ❌ no — single upload OOMs the tmpfs |
|
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=100 MB | 5 GiB (5 × 1 GiB) | 100 GB (5 × 2 × 10 GB) | ❌ chunked overflows |
|
||||||
| `MAX_UPLOAD=500 MB`, `CHUNK_MAX=50 MB` | 2.5 GB direct, 5 GB chunked | ⚠ direct fits, chunked overflows |
|
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=100 MB, `OXICLOUD_CHUNK_MAX_BYTES`=20 MB | 500 MB | 5 GB | ⚠ direct PUT fits, chunked still overflows |
|
||||||
| `MAX_UPLOAD=300 MB`, `CHUNK_MAX=30 MB` | 1.5 GB direct, 3 GB chunked | ✅ both fit |
|
| `OXICLOUD_MAX_UPLOAD_SIZE`=300 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=50 MB, `OXICLOUD_CHUNK_MAX_BYTES`=10 MB | 250 MB | 3 GB | ✅ both fit |
|
||||||
|
|
||||||
A real-disk volume (cheap, large):
|
A real-disk volume (cheap, large):
|
||||||
|
|
||||||
| Setting | Disk worst case | Comment |
|
| Settings | Direct-PUT worst case | Chunked worst case | Comment |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| `MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB` (defaults) | 100 GB chunked worst case | Fine on a 200+ GB volume; almost any real-disk setup |
|
| Defaults (see row above) | 5 GiB | 100 GB | Fine on a 200+ GB volume; almost any real-disk setup |
|
||||||
| `MAX_UPLOAD=100 GB`, `CHUNK_MAX=500 MB` | 1 TB chunked worst case | Plausible for video archives; needs a dedicated upload volume |
|
| `OXICLOUD_MAX_UPLOAD_SIZE`=100 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=5 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=500 MB | 25 GiB | 1 TB | Plausible for video archives; needs a dedicated upload volume |
|
||||||
|
|
||||||
### Choosing tmpfs vs real disk
|
### Choosing tmpfs vs real disk
|
||||||
|
|
||||||
| Constraint | Choice |
|
| Constraint | Choice |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Upload caps × concurrent users ≤ free RAM × 0.5 | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
|
| `OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs + 2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions ≤ free RAM × 0.5` | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
|
||||||
| Upload caps × concurrent users > free RAM × 0.5 | **real disk** — same FS as `.blobs/` ideal |
|
| Worst case exceeds half free RAM | **real disk** — same filesystem as `.blobs/` ideal |
|
||||||
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
|
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
|
||||||
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
|
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
|
||||||
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
|
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
|
||||||
|
|
||||||
The defaults (`MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB`) assume **real
|
The defaults (`OXICLOUD_MAX_UPLOAD_SIZE`=10 GB,
|
||||||
disk**. Don't run the defaults against tmpfs unless you've sized it
|
`OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB,
|
||||||
for the worst case.
|
`OXICLOUD_CHUNK_MAX_BYTES`=100 MB) assume **real disk**. Don't run
|
||||||
|
the defaults against tmpfs unless you've sized it for the worst case.
|
||||||
|
|
||||||
## TL;DR
|
## TL;DR
|
||||||
|
|
||||||
|
|||||||
+38
-8
@@ -31,18 +31,48 @@ OXICLOUD_SERVER_HOST=127.0.0.1
|
|||||||
# Example: https://cloud.example.com
|
# Example: https://cloud.example.com
|
||||||
#OXICLOUD_BASE_URL=https://cloud.example.com
|
#OXICLOUD_BASE_URL=https://cloud.example.com
|
||||||
|
|
||||||
# Maximum upload size in bytes (default: 10 GB on 64-bit)
|
# ── Upload size caps ──────────────────────────────────────────────────
|
||||||
|
# See docs/config/storage-fine-tuning.md for sizing guidance.
|
||||||
|
|
||||||
|
# Whole-file size ceiling. Applies to BOTH direct PUTs (per-request body)
|
||||||
|
# and chunked uploads (declared `total_size` at session creation — checked
|
||||||
|
# upfront so oversized requests never accumulate chunks on disk).
|
||||||
|
# Default: 10 GB on 64-bit, 1 GB on 32-bit.
|
||||||
#OXICLOUD_MAX_UPLOAD_SIZE=10737418240
|
#OXICLOUD_MAX_UPLOAD_SIZE=10737418240
|
||||||
|
|
||||||
# Directory for upload spool temp files. Uploads are streamed to a temp file
|
# Per-request cap for non-chunked PUT bodies (`POST /api/files/upload`,
|
||||||
# before deduplication. By default this uses the OS temp dir ($TMPDIR / /tmp),
|
# `PUT /webdav/...`, `PUT /remote.php/dav/files/...`). Set below
|
||||||
# which in many containers is tmpfs (RAM) — writing a large upload there fills
|
# MAX_UPLOAD_SIZE so files larger than this are pushed onto the chunked
|
||||||
# page-cache that counts against the cgroup memory limit and can OOMKill the
|
# protocol (resumable on failure, bounded per-request by CHUNK_MAX_BYTES).
|
||||||
# process. Point this at a real-disk path (same filesystem as the storage
|
# Default: 1 GiB.
|
||||||
# backend is ideal) to keep the upload footprint off RAM. Leave unset to use
|
#OXICLOUD_DIRECT_PUT_MAX_BYTES=1073741824
|
||||||
# the OS default.
|
|
||||||
|
# Per-chunk cap for a single chunked-upload PUT (PATCH /api/uploads/{id}
|
||||||
|
# or PUT /dav/uploads/.../chunk). NC desktop and the OxiCloud frontend
|
||||||
|
# split large files into chunks of this size or smaller, so this knob
|
||||||
|
# tightly bounds the worst-case per-request memory/disk footprint
|
||||||
|
# independently of the whole-file cap. Default: 100 MB.
|
||||||
|
#OXICLOUD_CHUNK_MAX_BYTES=104857600
|
||||||
|
|
||||||
|
# ── Upload spool directories ─────────────────────────────────────────
|
||||||
|
# Where in-flight uploads land BEFORE being promoted into final blob
|
||||||
|
# storage. Same-filesystem placement (with `.blobs/`) makes the
|
||||||
|
# promotion an atomic rename(2); NVMe placement speeds up intake.
|
||||||
|
# See docs/config/storage-fine-tuning.md for layout examples.
|
||||||
|
|
||||||
|
# Directory for non-chunked PUT spool tempfiles. Default: OS temp dir
|
||||||
|
# ($TMPDIR / /tmp), often tmpfs (RAM) in containers — writing a large
|
||||||
|
# upload there fills page-cache that counts against the cgroup memory
|
||||||
|
# limit and can OOMKill the process. Point this at a real-disk path
|
||||||
|
# (same FS as the storage backend is ideal).
|
||||||
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
|
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
|
||||||
|
|
||||||
|
# Root directory for chunked-upload sessions (REST + NextCloud chunked
|
||||||
|
# share this root). Default: {STORAGE_PATH}/.uploads. Pointing this at
|
||||||
|
# NVMe accelerates the chunk-write + assembly loop; pointing it at the
|
||||||
|
# same FS as `.blobs/` makes blob promotion atomic.
|
||||||
|
#OXICLOUD_CHUNK_DIR=/var/lib/oxicloud/.uploads
|
||||||
|
|
||||||
# How often (seconds) the background sweep reconciles each user's cached
|
# How often (seconds) the background sweep reconciles each user's cached
|
||||||
# storage usage with the real sum of their files (default: 600 = 10 min).
|
# storage usage with the real sum of their files (default: 600 = 10 min).
|
||||||
# GET /api/auth/me serves the cached value instead of recomputing per request;
|
# GET /api/auth/me serves the cached value instead of recomputing per request;
|
||||||
|
|||||||
@@ -218,6 +218,16 @@ pub struct StorageConfig {
|
|||||||
/// be far tighter than the whole-file cap and prevents one HTTP request
|
/// be far tighter than the whole-file cap and prevents one HTTP request
|
||||||
/// from monopolising server memory or disk. Env: `OXICLOUD_CHUNK_MAX_BYTES`.
|
/// from monopolising server memory or disk. Env: `OXICLOUD_CHUNK_MAX_BYTES`.
|
||||||
pub chunk_max_bytes: usize,
|
pub chunk_max_bytes: usize,
|
||||||
|
/// Maximum size of a single non-chunked PUT body, in bytes (default:
|
||||||
|
/// 1 GiB). Set below `max_upload_size` so files larger than this are
|
||||||
|
/// pushed onto the chunked-upload protocol (`/api/uploads/…` or
|
||||||
|
/// `/dav/uploads/…`) — which is resilient to mid-transfer failures,
|
||||||
|
/// resumable, and bounded per-request by `chunk_max_bytes`. Without
|
||||||
|
/// this cap a 10 GB direct PUT spools 10 GB to disk in a single
|
||||||
|
/// request; a connection drop at 95 % loses everything. The server
|
||||||
|
/// returns 413 with a "use chunked upload" hint when a direct PUT
|
||||||
|
/// exceeds this cap. Env: `OXICLOUD_DIRECT_PUT_MAX_BYTES`.
|
||||||
|
pub direct_put_max_bytes: usize,
|
||||||
/// Directory for upload spool temp files. When `Some`, large uploads are
|
/// Directory for upload spool temp files. When `Some`, large uploads are
|
||||||
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
|
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
|
||||||
/// containers, where the spool's page-cache counts against the cgroup
|
/// containers, where the spool's page-cache counts against the cgroup
|
||||||
@@ -376,6 +386,7 @@ impl Default for StorageConfig {
|
|||||||
trash_retention_days: 30, // 30 days
|
trash_retention_days: 30, // 30 days
|
||||||
max_upload_size: MAX_UPLOAD_SIZE,
|
max_upload_size: MAX_UPLOAD_SIZE,
|
||||||
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
|
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
|
||||||
|
direct_put_max_bytes: 1024 * 1024 * 1024, // 1 GiB — pushes larger uploads onto the chunked protocol
|
||||||
upload_temp_dir: None,
|
upload_temp_dir: None,
|
||||||
chunk_dir: None,
|
chunk_dir: None,
|
||||||
usage_reconcile_secs: 600, // 10 minutes
|
usage_reconcile_secs: 600, // 10 minutes
|
||||||
@@ -1247,6 +1258,12 @@ impl AppConfig {
|
|||||||
{
|
{
|
||||||
config.storage.chunk_max_bytes = val;
|
config.storage.chunk_max_bytes = val;
|
||||||
}
|
}
|
||||||
|
if let Ok(direct_max) =
|
||||||
|
env::var("OXICLOUD_DIRECT_PUT_MAX_BYTES").map(|v| v.parse::<usize>())
|
||||||
|
&& let Ok(val) = direct_max
|
||||||
|
{
|
||||||
|
config.storage.direct_put_max_bytes = val;
|
||||||
|
}
|
||||||
|
|
||||||
// Upload spool directory — keep large upload temp files off tmpfs/RAM
|
// Upload spool directory — keep large upload temp files off tmpfs/RAM
|
||||||
// (otherwise their page-cache counts against the cgroup memory limit).
|
// (otherwise their page-cache counts against the cgroup memory limit).
|
||||||
|
|||||||
@@ -162,6 +162,33 @@ impl ChunkedUploadHandler {
|
|||||||
.into_response();
|
.into_response();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Whole-file cap ──────────────────────────────────────────
|
||||||
|
// Reject upfront, before any chunk is uploaded — wasting
|
||||||
|
// bandwidth + server disk on an upload that's going to be
|
||||||
|
// rejected at /complete is the worst-of-both-worlds outcome.
|
||||||
|
// `max_upload_size` is the same ceiling that bounds direct
|
||||||
|
// PUTs (per-byte during streaming there; declared per-session
|
||||||
|
// here). When quotas are disabled, this is the only whole-file
|
||||||
|
// limit for chunked uploads — without it a hostile client
|
||||||
|
// could declare `total_size: 1 TB` and accumulate chunks
|
||||||
|
// until disk fills.
|
||||||
|
let max_upload = state.core.config.storage.max_upload_size as u64;
|
||||||
|
if request.total_size > max_upload {
|
||||||
|
tracing::warn!(
|
||||||
|
"⛔ CHUNKED UPLOAD REJECTED (total_size cap): user={}, file={}, declared={}, max={}",
|
||||||
|
auth_user.username,
|
||||||
|
request.filename,
|
||||||
|
request.total_size,
|
||||||
|
max_upload
|
||||||
|
);
|
||||||
|
return AppError::payload_too_large(format!(
|
||||||
|
"Declared total_size {} exceeds the server's `max_upload_size` cap ({} bytes). \
|
||||||
|
Raise OXICLOUD_MAX_UPLOAD_SIZE on the server if larger uploads are expected.",
|
||||||
|
request.total_size, max_upload
|
||||||
|
))
|
||||||
|
.into_response();
|
||||||
|
}
|
||||||
|
|
||||||
// ── Permission pre-check: caller must have Create on the target
|
// ── Permission pre-check: caller must have Create on the target
|
||||||
// folder BEFORE we allocate a session and accept chunks. The
|
// folder BEFORE we allocate a session and accept chunks. The
|
||||||
// upload service re-checks at finalize time, but failing here
|
// upload service re-checks at finalize time, but failing here
|
||||||
|
|||||||
@@ -927,8 +927,10 @@ async fn handle_put(
|
|||||||
// another user — acceptable risk since PathResolver should always
|
// another user — acceptable risk since PathResolver should always
|
||||||
// be enabled in production)
|
// be enabled in production)
|
||||||
|
|
||||||
// Hard upload size limit from config
|
// Direct PUT cap — see `nextcloud/webdav_handler::handle_put` for
|
||||||
let max_upload = state.core.config.storage.max_upload_size;
|
// the reasoning. Files above `direct_put_max_bytes` must go through
|
||||||
|
// the chunked-upload protocol (`/api/uploads/…`) which is resumable.
|
||||||
|
let max_upload = state.core.config.storage.direct_put_max_bytes;
|
||||||
|
|
||||||
// Extract content type before consuming the request
|
// Extract content type before consuming the request
|
||||||
let content_type = req
|
let content_type = req
|
||||||
|
|||||||
@@ -596,7 +596,14 @@ async fn handle_put(
|
|||||||
.and_then(|v| v.to_str().ok())
|
.and_then(|v| v.to_str().ok())
|
||||||
.and_then(|v| v.parse::<i64>().ok());
|
.and_then(|v| v.parse::<i64>().ok());
|
||||||
|
|
||||||
let max_upload = state.core.config.storage.max_upload_size;
|
// ── Direct PUT cap ───────────────────────────────────────────────
|
||||||
|
// We use `direct_put_max_bytes` (default 1 GiB), not `max_upload_size`
|
||||||
|
// (default 10 GB). Larger files must come through the chunked upload
|
||||||
|
// protocol (`/dav/uploads/...`) which is resumable on failure and
|
||||||
|
// bounded per-request by `chunk_max_bytes`. Trying to stream a
|
||||||
|
// multi-GB body through a single PUT is a footgun: a connection drop
|
||||||
|
// at 95 % loses everything.
|
||||||
|
let max_upload = state.core.config.storage.direct_put_max_bytes;
|
||||||
|
|
||||||
// Stream the body to a temp file + incremental hash — never buffer the
|
// Stream the body to a temp file + incremental hash — never buffer the
|
||||||
// full upload in RAM. The old `body::to_bytes` path loaded the entire
|
// full upload in RAM. The old `body::to_bytes` path loaded the entire
|
||||||
|
|||||||
@@ -70,7 +70,10 @@ pub async fn spool_body_to_temp(
|
|||||||
drop(file);
|
drop(file);
|
||||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||||
return Err(AppError::payload_too_large(format!(
|
return Err(AppError::payload_too_large(format!(
|
||||||
"Upload exceeds maximum size of {max_upload} bytes"
|
"Upload body exceeds the direct-PUT cap ({max_upload} bytes). \
|
||||||
|
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
|
||||||
|
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
|
||||||
|
Chunked uploads are resumable on transient failure."
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
hasher.update(chunk);
|
hasher.update(chunk);
|
||||||
|
|||||||
@@ -150,6 +150,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|||||||
// hardest to spot from chunked-upload tests.
|
// hardest to spot from chunked-upload tests.
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
max_upload_size_mb = config.storage.max_upload_size / (1024 * 1024),
|
max_upload_size_mb = config.storage.max_upload_size / (1024 * 1024),
|
||||||
|
direct_put_max_bytes_mb = config.storage.direct_put_max_bytes / (1024 * 1024),
|
||||||
chunk_max_bytes_mb = config.storage.chunk_max_bytes / (1024 * 1024),
|
chunk_max_bytes_mb = config.storage.chunk_max_bytes / (1024 * 1024),
|
||||||
"Upload limits loaded from config"
|
"Upload limits loaded from config"
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -563,3 +563,120 @@ POST {{base_url}}/api/uploads/{{upload_id_nobody}}/complete
|
|||||||
Authorization: Bearer {{token}}
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
HTTP 201
|
HTTP 201
|
||||||
|
|
||||||
|
|
||||||
|
# ═════════════════════════════════════════════════════════════
|
||||||
|
# Whole-file size cap (OXICLOUD_MAX_UPLOAD_SIZE)
|
||||||
|
# ═════════════════════════════════════════════════════════════
|
||||||
|
# The whole-file cap is checked at session creation against the
|
||||||
|
# JSON-declared `total_size` — no upload body required to trip
|
||||||
|
# it. Test default `OXICLOUD_MAX_UPLOAD_SIZE` is 10 GiB; we
|
||||||
|
# declare 100 GiB to be safely above without bothering with a
|
||||||
|
# Hurl `--variables` override.
|
||||||
|
#
|
||||||
|
# This guards the case where chunks would otherwise accumulate
|
||||||
|
# disk space against an oversized declared upload — the reject
|
||||||
|
# fires BEFORE any chunk is PATCHed.
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 21 — POST /api/uploads with `total_size` above
|
||||||
|
# OXICLOUD_MAX_UPLOAD_SIZE → 413 Payload Too Large.
|
||||||
|
# No body bytes are sent; the check is purely
|
||||||
|
# against the declared JSON field.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/uploads
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"filename": "way-too-big.bin",
|
||||||
|
"folder_id": "{{home_folder_id}}",
|
||||||
|
"content_type": "application/octet-stream",
|
||||||
|
"total_size": 107374182400,
|
||||||
|
"chunk_size": 5242880
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 413
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 22 — Sanity check: a session JUST BELOW the cap is
|
||||||
|
# accepted. Uses a sub-cap value (32 bytes — same
|
||||||
|
# pattern as earlier steps so no extra fixture is
|
||||||
|
# needed). Confirms that the reject in step 21 was
|
||||||
|
# the size cap, not an unrelated regression.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
POST {{base_url}}/api/uploads
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/json
|
||||||
|
{
|
||||||
|
"filename": "small-and-fine.txt",
|
||||||
|
"folder_id": "{{home_folder_id}}",
|
||||||
|
"content_type": "text/plain",
|
||||||
|
"total_size": 32,
|
||||||
|
"chunk_size": 1048576
|
||||||
|
}
|
||||||
|
|
||||||
|
HTTP 201
|
||||||
|
[Captures]
|
||||||
|
upload_id_sanity: jsonpath "$.upload_id"
|
||||||
|
|
||||||
|
# Cleanup — cancel without uploading.
|
||||||
|
DELETE {{base_url}}/api/uploads/{{upload_id_sanity}}
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP 204
|
||||||
|
|
||||||
|
|
||||||
|
# ═════════════════════════════════════════════════════════════
|
||||||
|
# Direct-PUT body cap (OXICLOUD_DIRECT_PUT_MAX_BYTES)
|
||||||
|
# ═════════════════════════════════════════════════════════════
|
||||||
|
# `OXICLOUD_DIRECT_PUT_MAX_BYTES` (4 MiB in the test env) bounds
|
||||||
|
# a single non-chunked PUT body. Larger uploads must come through
|
||||||
|
# the chunked protocol — the server returns 413 with a hint
|
||||||
|
# pointing at `/api/uploads/...` / `/dav/uploads/...`. The cap
|
||||||
|
# fires mid-stream via the same accumulator that bounds chunks.
|
||||||
|
#
|
||||||
|
# Tested via the native REST WebDAV PUT endpoint (`/webdav/...`)
|
||||||
|
# because it's JWT-authed (no app-password mint dance) and
|
||||||
|
# straightforwardly path-mapped. The cap is wired into the same
|
||||||
|
# `spool_body_to_temp` helper from the NextCloud single-file PUT
|
||||||
|
# path, so this exercise covers both wirings.
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 23 — Direct PUT of the 5 MiB fixture → 413. The same
|
||||||
|
# fixture used in step 8 for the chunked-PATCH cap;
|
||||||
|
# here it lands against the direct-PUT cap.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
PUT {{base_url}}/webdav/test-direct-put-over-cap.bin
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: application/octet-stream
|
||||||
|
file,fixtures/chunk-over-cap-5mb.bin;
|
||||||
|
|
||||||
|
HTTP 413
|
||||||
|
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
# Step 24 — Sanity: direct PUT WELL under the cap → 201 or 204.
|
||||||
|
# Confirms step 23's reject was the cap, not a route /
|
||||||
|
# auth / handler regression.
|
||||||
|
# ─────────────────────────────────────────────────────────────
|
||||||
|
PUT {{base_url}}/webdav/test-direct-put-under-cap.txt
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
Content-Type: text/plain
|
||||||
|
file,fixtures/hello.txt;
|
||||||
|
|
||||||
|
# WebDAV PUT returns 201 on new file, 204 on overwrite. Either
|
||||||
|
# value confirms a successful body acceptance.
|
||||||
|
HTTP *
|
||||||
|
[Asserts]
|
||||||
|
status >= 200
|
||||||
|
status < 300
|
||||||
|
|
||||||
|
# Cleanup so a re-run finds a clean slate (DELETE is idempotent
|
||||||
|
# enough for this — 204 on success, 404 if nothing left).
|
||||||
|
DELETE {{base_url}}/webdav/test-direct-put-under-cap.txt
|
||||||
|
Authorization: Bearer {{token}}
|
||||||
|
|
||||||
|
HTTP *
|
||||||
|
|||||||
@@ -27,6 +27,14 @@ RUST_LOG="warn,audit=info"
|
|||||||
# under the cap, while the cap test sends a 5 MiB fixture to trigger 413.
|
# under the cap, while the cap test sends a 5 MiB fixture to trigger 413.
|
||||||
OXICLOUD_CHUNK_MAX_BYTES=4194304
|
OXICLOUD_CHUNK_MAX_BYTES=4194304
|
||||||
|
|
||||||
|
# Direct-PUT (non-chunked) cap, exercised by chunked_upload_cap.hurl.
|
||||||
|
# 4 MiB: same threshold as the chunked cap so the existing 5 MiB
|
||||||
|
# fixture (chunk-over-cap-5mb.bin) can prove BOTH caps with one
|
||||||
|
# generated file. All existing direct-PUT tests
|
||||||
|
# (test_dedup_webdav_multichunk.sh = 2.76 MB, _ref_count = ~66 KB,
|
||||||
|
# _nextcloud_put_blake3 = 32 B) stay safely under this cap.
|
||||||
|
OXICLOUD_DIRECT_PUT_MAX_BYTES=4194304
|
||||||
|
|
||||||
# grow up limits for tests
|
# grow up limits for tests
|
||||||
OXICLOUD_RATE_LIMIT_REFRESH_MAX=360
|
OXICLOUD_RATE_LIMIT_REFRESH_MAX=360
|
||||||
OXICLOUD_RATE_LIMIT_LOGIN_MAX=360
|
OXICLOUD_RATE_LIMIT_LOGIN_MAX=360
|
||||||
|
|||||||
Reference in New Issue
Block a user