fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on dedup hits, due to three separate full-file-in-memory paths: - NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via body::to_bytes before any dedup logic, then re-wrote and re-hashed it. Now streams the body to a temp file with incremental BLAKE3 and goes through update_file_streaming (shared spool helper with the native WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size. - DedupService::store_chunks materialized every new chunk's data in a Vec before uploading. Now reads each new chunk by positioned I/O (read_exact_at, off the runtime via spawn_blocking) just before its upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK. - The upload spool used the OS temp dir, often tmpfs/RAM in containers where its page-cache counts against the cgroup memory limit. Add OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk. Also collapse a pre-existing clippy collapsible_else_if in carddav_handler. Refs #404 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -211,6 +211,12 @@ pub struct StorageConfig {
|
||||
/// Maximum upload file size in bytes (default: 10 GB).
|
||||
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
|
||||
pub max_upload_size: usize,
|
||||
/// Directory for upload spool temp files. When `Some`, large uploads are
|
||||
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
|
||||
/// containers, where the spool's page-cache counts against the cgroup
|
||||
/// memory limit and can trigger OOMKill on large files). Env:
|
||||
/// `OXICLOUD_UPLOAD_TMPDIR`.
|
||||
pub upload_temp_dir: Option<PathBuf>,
|
||||
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
|
||||
pub backend: StorageBackendType,
|
||||
/// S3-compatible backend configuration (used when `backend == S3`).
|
||||
@@ -348,6 +354,7 @@ impl Default for StorageConfig {
|
||||
parallel_threshold: 100 * 1024 * 1024, // 100 MB
|
||||
trash_retention_days: 30, // 30 days
|
||||
max_upload_size: MAX_UPLOAD_SIZE,
|
||||
upload_temp_dir: None,
|
||||
backend: StorageBackendType::Local,
|
||||
s3: None,
|
||||
azure: None,
|
||||
@@ -1212,6 +1219,14 @@ impl AppConfig {
|
||||
config.storage.max_upload_size = val;
|
||||
}
|
||||
|
||||
// Upload spool directory — keep large upload temp files off tmpfs/RAM
|
||||
// (otherwise their page-cache counts against the cgroup memory limit).
|
||||
if let Ok(dir) = env::var("OXICLOUD_UPLOAD_TMPDIR")
|
||||
&& !dir.trim().is_empty()
|
||||
{
|
||||
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
|
||||
}
|
||||
|
||||
// Storage backend selection
|
||||
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
|
||||
match backend.to_lowercase().as_str() {
|
||||
|
||||
+2
-1
@@ -425,7 +425,8 @@ impl AppServiceFactory {
|
||||
repos.file_read_repository.clone(),
|
||||
)
|
||||
.with_content_cache(core.file_content_cache.clone())
|
||||
.with_file_lifecycle_hook(core.file_lifecycle.clone()),
|
||||
.with_file_lifecycle_hook(core.file_lifecycle.clone())
|
||||
.with_upload_temp_dir(self.config.storage.upload_temp_dir.clone()),
|
||||
);
|
||||
|
||||
let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache(
|
||||
|
||||
@@ -4,3 +4,4 @@ pub mod errors;
|
||||
pub mod locale;
|
||||
pub mod mime_detect;
|
||||
pub mod stubs;
|
||||
pub mod temp;
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
//! Shared helper for creating upload spool temp files.
|
||||
//!
|
||||
//! Upload paths spool the request body to a temp file before deduplication.
|
||||
//! By default `tempfile` uses the OS temp dir (`std::env::temp_dir()`, i.e.
|
||||
//! `$TMPDIR` / `/tmp`), which in many container setups is **tmpfs (RAM)**.
|
||||
//! Writing a multi-hundred-MB upload there fills page-cache that counts
|
||||
//! against the cgroup memory limit and can OOMKill the process. Pointing the
|
||||
//! spool at a real-disk directory (`OXICLOUD_UPLOAD_TMPDIR`) keeps the upload
|
||||
//! footprint proportional to the streaming buffer, not the file size.
|
||||
|
||||
use std::path::Path;
|
||||
use tempfile::NamedTempFile;
|
||||
|
||||
/// Create a [`NamedTempFile`], honoring an optional configured spool directory.
|
||||
///
|
||||
/// When `dir` is `Some`, the temp file is created there (the directory is
|
||||
/// created if missing); otherwise the OS default temp dir is used.
|
||||
pub fn new_spool_temp_file(dir: Option<&Path>) -> std::io::Result<NamedTempFile> {
|
||||
match dir {
|
||||
Some(d) => {
|
||||
std::fs::create_dir_all(d)?;
|
||||
NamedTempFile::new_in(d)
|
||||
}
|
||||
None => NamedTempFile::new(),
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user