fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on dedup hits, due to three separate full-file-in-memory paths: - NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via body::to_bytes before any dedup logic, then re-wrote and re-hashed it. Now streams the body to a temp file with incremental BLAKE3 and goes through update_file_streaming (shared spool helper with the native WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size. - DedupService::store_chunks materialized every new chunk's data in a Vec before uploading. Now reads each new chunk by positioned I/O (read_exact_at, off the runtime via spawn_blocking) just before its upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK. - The upload spool used the OS temp dir, often tmpfs/RAM in containers where its page-cache counts against the cgroup memory limit. Add OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk. Also collapse a pre-existing clippy collapsible_else_if in carddav_handler. Refs #404 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -211,6 +211,12 @@ pub struct StorageConfig {
|
||||
/// Maximum upload file size in bytes (default: 10 GB).
|
||||
/// Applied as a hard limit to WebDAV PUT and streaming uploads.
|
||||
pub max_upload_size: usize,
|
||||
/// Directory for upload spool temp files. When `Some`, large uploads are
|
||||
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
|
||||
/// containers, where the spool's page-cache counts against the cgroup
|
||||
/// memory limit and can trigger OOMKill on large files). Env:
|
||||
/// `OXICLOUD_UPLOAD_TMPDIR`.
|
||||
pub upload_temp_dir: Option<PathBuf>,
|
||||
/// Which blob storage backend to use (`local`, `s3`, or `azure`).
|
||||
pub backend: StorageBackendType,
|
||||
/// S3-compatible backend configuration (used when `backend == S3`).
|
||||
@@ -348,6 +354,7 @@ impl Default for StorageConfig {
|
||||
parallel_threshold: 100 * 1024 * 1024, // 100 MB
|
||||
trash_retention_days: 30, // 30 days
|
||||
max_upload_size: MAX_UPLOAD_SIZE,
|
||||
upload_temp_dir: None,
|
||||
backend: StorageBackendType::Local,
|
||||
s3: None,
|
||||
azure: None,
|
||||
@@ -1212,6 +1219,14 @@ impl AppConfig {
|
||||
config.storage.max_upload_size = val;
|
||||
}
|
||||
|
||||
// Upload spool directory — keep large upload temp files off tmpfs/RAM
|
||||
// (otherwise their page-cache counts against the cgroup memory limit).
|
||||
if let Ok(dir) = env::var("OXICLOUD_UPLOAD_TMPDIR")
|
||||
&& !dir.trim().is_empty()
|
||||
{
|
||||
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
|
||||
}
|
||||
|
||||
// Storage backend selection
|
||||
if let Ok(backend) = env::var("OXICLOUD_STORAGE_BACKEND") {
|
||||
match backend.to_lowercase().as_str() {
|
||||
|
||||
Reference in New Issue
Block a user