fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files

Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:

- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
  body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
  Now streams the body to a temp file with incremental BLAKE3 and goes
  through update_file_streaming (shared spool helper with the native
  WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.

- DedupService::store_chunks materialized every new chunk's data in a Vec
  before uploading. Now reads each new chunk by positioned I/O
  (read_exact_at, off the runtime via spawn_blocking) just before its
  upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.

- The upload spool used the OS temp dir, often tmpfs/RAM in containers
  where its page-cache counts against the cgroup memory limit. Add
  OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.

Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.

Refs #404

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DioCrafts
2026-06-07 01:03:52 +02:00
parent c5e0800336
commit 061306cc84
12 changed files with 315 additions and 153 deletions
+15
View File
@@ -211,6 +211,12 @@ pub struct StorageConfig {
/// Maximum upload file size in bytes (default: 10 GB).
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
pub max_upload_size: usize,
/// Directory for upload spool temp files. When `Some`, large uploads are
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
/// containers, where the spool's page-cache counts against the cgroup
/// memory limit and can trigger OOMKill on large files). Env:
/// `OXICLOUD_UPLOAD_TMPDIR`.
pub upload_temp_dir: Option<PathBuf>,
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
pub backend: StorageBackendType,
/// S3-compatible backend configuration (used when `backend == S3`).
@@ -348,6 +354,7 @@ impl Default for StorageConfig {
parallel_threshold: 100 * 1024 * 1024, // 100 MB
trash_retention_days: 30, // 30 days
max_upload_size: MAX_UPLOAD_SIZE,
upload_temp_dir: None,
backend: StorageBackendType::Local,
s3: None,
azure: None,
@@ -1212,6 +1219,14 @@ impl AppConfig {
config.storage.max_upload_size = val;
}
// Upload spool directory — keep large upload temp files off tmpfs/RAM
// (otherwise their page-cache counts against the cgroup memory limit).
if let Ok(dir) = env::var("OXICLOUD_UPLOAD_TMPDIR")
&& !dir.trim().is_empty()
{
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
}
// Storage backend selection
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
match backend.to_lowercase().as_str() {
+2 -1
View File
@@ -425,7 +425,8 @@ impl AppServiceFactory {
repos.file_read_repository.clone(),
)
.with_content_cache(core.file_content_cache.clone())
.with_file_lifecycle_hook(core.file_lifecycle.clone()),
.with_file_lifecycle_hook(core.file_lifecycle.clone())
.with_upload_temp_dir(self.config.storage.upload_temp_dir.clone()),
);
let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache(
+1
View File
@@ -4,3 +4,4 @@ pub mod errors;
pub mod locale;
pub mod mime_detect;
pub mod stubs;
pub mod temp;
+26
View File
@@ -0,0 +1,26 @@
//! Shared helper for creating upload spool temp files.
//!
//! Upload paths spool the request body to a temp file before deduplication.
//! By default `tempfile` uses the OS temp dir (`std::env::temp_dir()`, i.e.
//! `$TMPDIR` / `/tmp`), which in many container setups is **tmpfs (RAM)**.
//! Writing a multi-hundred-MB upload there fills page-cache that counts
//! against the cgroup memory limit and can OOMKill the process. Pointing the
//! spool at a real-disk directory (`OXICLOUD_UPLOAD_TMPDIR`) keeps the upload
//! footprint proportional to the streaming buffer, not the file size.
use std::path::Path;
use tempfile::NamedTempFile;
/// Create a [`NamedTempFile`], honoring an optional configured spool directory.
///
/// When `dir` is `Some`, the temp file is created there (the directory is
/// created if missing); otherwise the OS default temp dir is used.
pub fn new_spool_temp_file(dir: Option<&Path>) -> std::io::Result<NamedTempFile> {
match dir {
Some(d) => {
std::fs::create_dir_all(d)?;
NamedTempFile::new_in(d)
}
None => NamedTempFile::new(),
}
}