fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on dedup hits, due to three separate full-file-in-memory paths: - NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via body::to_bytes before any dedup logic, then re-wrote and re-hashed it. Now streams the body to a temp file with incremental BLAKE3 and goes through update_file_streaming (shared spool helper with the native WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size. - DedupService::store_chunks materialized every new chunk's data in a Vec before uploading. Now reads each new chunk by positioned I/O (read_exact_at, off the runtime via spawn_blocking) just before its upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK. - The upload spool used the OS temp dir, often tmpfs/RAM in containers where its page-cache counts against the cgroup memory limit. Add OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk. Also collapse a pre-existing clippy collapsible_else_if in carddav_handler. Refs #404 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -34,6 +34,15 @@ OXICLOUD_SERVER_HOST=127.0.0.1
|
||||
# Maximum upload size in bytes (default: 10 GB on 64-bit)
|
||||
#OXICLOUD_MAX_UPLOAD_SIZE=10737418240
|
||||
|
||||
# Directory for upload spool temp files. Uploads are streamed to a temp file
|
||||
# before deduplication. By default this uses the OS temp dir ($TMPDIR / /tmp),
|
||||
# which in many containers is tmpfs (RAM) — writing a large upload there fills
|
||||
# page-cache that counts against the cgroup memory limit and can OOMKill the
|
||||
# process. Point this at a real-disk path (same filesystem as the storage
|
||||
# backend is ideal) to keep the upload footprint off RAM. Leave unset to use
|
||||
# the OS default.
|
||||
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
|
||||
|
||||
# Allow multiple processes to bind to the same port (SO_REUSEPORT).
|
||||
# DISABLED by default — leaving this off means a second accidental instance
|
||||
# will fail immediately with "address already in use", which is the safe behaviour.
|
||||
|
||||
Reference in New Issue
Block a user