security(upload): cap upload size to prevent memody/disk consumption
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
(differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
hurl test validate the change
Streams the request body straight to the chunk file with peak heap of
~one HTTP frame, regardless of chunk size or the configured cap. The
`storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
default 100 MB) bounds a single PUT — separate from `max_upload_size`
which governs whole-file uploads. Without this separation, a client
could submit a chunk up to the whole-file cap (10 GB default) and
monopolise server memory.
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# =============================================================
|
||||
# OxiCloud — Chunked upload size cap + streaming verification
|
||||
# =============================================================
|
||||
# Validates `storage.chunk_max_bytes` (env `OXICLOUD_CHUNK_MAX_BYTES`,
|
||||
# set to 4 MiB by `tests/common/server.env`) on the REST chunked upload
|
||||
# surface `/api/uploads/{id}`.
|
||||
#
|
||||
# Two scenarios:
|
||||
# 1. SUCCESS path — uploads `hello.txt` (32 bytes) in one chunk and
|
||||
# asserts the stored file's BLAKE3 matches the known hash of the
|
||||
# fixture. Confirms streaming + assembly + dedup integrity.
|
||||
# 2. CAP REJECTION — uploads `chunk-over-cap-5mb.bin` (5 MiB) which
|
||||
# exceeds the 4 MiB cap, asserting the server returns 413 Payload
|
||||
# Too Large instead of OOMing or silently truncating.
|
||||
#
|
||||
# Why REST chunked, not NC chunked: both surfaces share the same
|
||||
# `chunk_max_bytes` config; the REST path is JWT-authed (already set
|
||||
# up in this Hurl run), the NC path would require minting an app
|
||||
# password mid-test.
|
||||
# =============================================================
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 1 — Login and capture the JWT token
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/auth/login
|
||||
Content-Type: application/json
|
||||
{
|
||||
"username": "{{username}}",
|
||||
"password": "{{password}}"
|
||||
}
|
||||
|
||||
HTTP 200
|
||||
[Captures]
|
||||
token: jsonpath "$.access_token"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 2 — Resolve home folder id (single root folder for admin)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
GET {{base_url}}/api/folders
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 200
|
||||
[Captures]
|
||||
home_folder_id: jsonpath "$[0].id"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 3 — Create chunked upload session for hello.txt (32 bytes)
|
||||
# chunk_size = 1 MiB (server minimum is 1 MiB). With
|
||||
# total_size < chunk_size the server computes a single
|
||||
# chunk of `total_size` bytes for index 0.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/uploads
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/json
|
||||
{
|
||||
"filename": "chunked-cap-hello.txt",
|
||||
"folder_id": "{{home_folder_id}}",
|
||||
"content_type": "text/plain",
|
||||
"total_size": 32,
|
||||
"chunk_size": 1048576
|
||||
}
|
||||
|
||||
HTTP 201
|
||||
[Captures]
|
||||
upload_id_ok: jsonpath "$.upload_id"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 4 — Upload the single 32-byte chunk → 200
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PATCH {{base_url}}/api/uploads/{{upload_id_ok}}?chunk_index=0
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/octet-stream
|
||||
file,fixtures/hello.txt;
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
header "Upload-Complete" == "true"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 5 — Complete the upload → 201, capture file id
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/uploads/{{upload_id_ok}}/complete
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 201
|
||||
[Captures]
|
||||
file_id: jsonpath "$.file_id"
|
||||
[Asserts]
|
||||
jsonpath "$.filename" == "chunked-cap-hello.txt"
|
||||
jsonpath "$.size" == 32
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 6 — Verify the stored file's BLAKE3 matches the known
|
||||
# hash of `tests/fixtures/hello.txt`. This proves the
|
||||
# streaming path wrote the exact bytes — no truncation,
|
||||
# no buffering corruption, no off-by-one.
|
||||
#
|
||||
# Expected hash:
|
||||
# b2208c5dc33ff951227bd0c139f5eccb04105d6da6a7519ee23f7bc00a17bb5a
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
GET {{base_url}}/api/files?folder_id={{home_folder_id}}
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 200
|
||||
[Asserts]
|
||||
jsonpath "$[?(@.id == '{{file_id}}')].content_hash" includes "b2208c5dc33ff951227bd0c139f5eccb04105d6da6a7519ee23f7bc00a17bb5a"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 7 — CAP REJECTION: create a session declaring a 5 MiB
|
||||
# chunk and attempt to upload exactly 5 MiB → 413.
|
||||
#
|
||||
# The cap fires in `axum::body::to_bytes(req, max_chunk)`
|
||||
# BEFORE the inner chunk-size check, so the server never
|
||||
# materialises the full 5 MiB body in memory.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/uploads
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/json
|
||||
{
|
||||
"filename": "chunked-cap-over.bin",
|
||||
"folder_id": "{{home_folder_id}}",
|
||||
"content_type": "application/octet-stream",
|
||||
"total_size": 5242880,
|
||||
"chunk_size": 5242880
|
||||
}
|
||||
|
||||
HTTP 201
|
||||
[Captures]
|
||||
upload_id_big: jsonpath "$.upload_id"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 8 — Send a 5 MiB chunk → 413 (cap is 4 MiB).
|
||||
# Pre-fix this PATCH would either OOM the server or be
|
||||
# silently treated as an empty body (via the old
|
||||
# `to_bytes(.., usize::MAX).unwrap_or_default()` path).
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PATCH {{base_url}}/api/uploads/{{upload_id_big}}?chunk_index=0
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/octet-stream
|
||||
file,fixtures/chunk-over-cap-5mb.bin;
|
||||
|
||||
HTTP 413
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 9 — Clean up the abandoned over-cap session.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
DELETE {{base_url}}/api/uploads/{{upload_id_big}}
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 204
|
||||
+14
-1
@@ -85,6 +85,18 @@ log "Waiting for server at $base_url..."
|
||||
wait_for_http "$base_url/ready" 120
|
||||
log "Server is ready."
|
||||
|
||||
# ── 3.5. Generate ephemeral test fixtures ─────────────────────────────────────
|
||||
# chunked_upload_cap.hurl needs a body > OXICLOUD_CHUNK_MAX_BYTES (4 MiB) to
|
||||
# trigger the 413. Committing a 5 MiB binary to the repo would bloat git for
|
||||
# every clone; generating it at run time is reproducible and the file is in
|
||||
# `.gitignore`.
|
||||
|
||||
OVER_CAP_FIXTURE="$REPO_ROOT/tests/fixtures/chunk-over-cap-5mb.bin"
|
||||
if [[ ! -s "$OVER_CAP_FIXTURE" ]]; then
|
||||
log "Generating 5 MiB fixture for chunk-cap test → $OVER_CAP_FIXTURE"
|
||||
dd if=/dev/zero of="$OVER_CAP_FIXTURE" bs=1024 count=5120 status=none
|
||||
fi
|
||||
|
||||
# ── 4. Run Hurl tests ─────────────────────────────────────────────────────────
|
||||
|
||||
log "Running Hurl tests..."
|
||||
@@ -104,7 +116,8 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test
|
||||
"$API_DIR/grants.hurl" \
|
||||
"$API_DIR/subject_groups.hurl" \
|
||||
"$API_DIR/grants_nested_groups.hurl" \
|
||||
"$API_DIR/external_users.hurl"
|
||||
"$API_DIR/external_users.hurl" \
|
||||
"$API_DIR/chunked_upload_cap.hurl"
|
||||
|
||||
#bash "$API_DIR/dedup_bulk_upload.sh"
|
||||
|
||||
|
||||
@@ -22,6 +22,11 @@ RUST_LOG="warn,audit=info"
|
||||
#RUST_LOG=debug
|
||||
#RUST_LOG=info
|
||||
|
||||
# Per-chunk upload cap, exercised by chunked_upload_cap.hurl.
|
||||
# 4 MiB: lets the existing grants.hurl single-chunk test (2.76 MB) pass
|
||||
# under the cap, while the cap test sends a 5 MiB fixture to trigger 413.
|
||||
OXICLOUD_CHUNK_MAX_BYTES=4194304
|
||||
|
||||
# grow up limits for tests
|
||||
OXICLOUD_RATE_LIMIT_REFRESH_MAX=360
|
||||
OXICLOUD_RATE_LIMIT_LOGIN_MAX=360
|
||||
|
||||
Reference in New Issue
Block a user