fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files

Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:

- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
  body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
  Now streams the body to a temp file with incremental BLAKE3 and goes
  through update_file_streaming (shared spool helper with the native
  WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.

- DedupService::store_chunks materialized every new chunk's data in a Vec
  before uploading. Now reads each new chunk by positioned I/O
  (read_exact_at, off the runtime via spawn_blocking) just before its
  upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.

- The upload spool used the OS temp dir, often tmpfs/RAM in containers
  where its page-cache counts against the cgroup memory limit. Add
  OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.

Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.

Refs #404

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DioCrafts
2026-06-07 01:03:52 +02:00
parent c5e0800336
commit 061306cc84
12 changed files with 315 additions and 153 deletions
+9
View File
@@ -34,6 +34,15 @@ OXICLOUD_SERVER_HOST=127.0.0.1
# Maximum upload size in bytes (default: 10 GB on 64-bit)
#OXICLOUD_MAX_UPLOAD_SIZE=10737418240
# Directory for upload spool temp files. Uploads are streamed to a temp file
# before deduplication. By default this uses the OS temp dir ($TMPDIR / /tmp),
# which in many containers is tmpfs (RAM) — writing a large upload there fills
# page-cache that counts against the cgroup memory limit and can OOMKill the
# process. Point this at a real-disk path (same filesystem as the storage
# backend is ideal) to keep the upload footprint off RAM. Leave unset to use
# the OS default.
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
# Allow multiple processes to bind to the same port (SO_REUSEPORT).
# DISABLED by default — leaving this off means a second accidental instance
# will fail immediately with "address already in use", which is the safe behaviour.