cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files

Browse Source 
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:

- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
  body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
  Now streams the body to a temp file with incremental BLAKE3 and goes
  through update_file_streaming (shared spool helper with the native
  WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.

- DedupService::store_chunks materialized every new chunk's data in a Vec
  before uploading. Now reads each new chunk by positioned I/O
  (read_exact_at, off the runtime via spawn_blocking) just before its
  upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.

- The upload spool used the OS temp dir, often tmpfs/RAM in containers
  where its page-cache counts against the cgroup memory limit. Add
  OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.

Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.

Refs #404

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DioCrafts
2026-06-07 01:03:52 +02:00
parent c5e0800336
commit 061306cc84
12 changed files with 315 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/interfaces/api/handlers/carddav_handler.rs
+3 -5
View File
@@ -141,12 +141,10 @@ fn strip_username_prefix(path: &str) -> &str {
} else {
&path[pos + 1..]
}
} else if uuid::Uuid::parse_str(path).is_ok() {
path
} else {
if uuid::Uuid::parse_str(path).is_ok() {
path
} else {
""
}
""
}
}
src/interfaces/api/handlers/webdav_handler.rs
+13 -41
View File
@@ -822,9 +822,7 @@ async fn handle_put(
req: Request<Body>,
path: String,
) -> Result<Response<Body>, AppError> {
use http_body_util::BodyStream;
use tokio::io::AsyncWriteExt;
use tokio_stream::StreamExt;
use crate::interfaces::upload_spool::spool_body_to_temp;
let user = extract_user(&req)?;
@@ -882,44 +880,18 @@ async fn handle_put(
.to_string();
// ── Streaming spool: body → temp file + incremental hash ──
let temp_file = tempfile::NamedTempFile::new()
.map_err(|e| AppError::internal_error(format!("Failed to create temp file: {}", e)))?;
let temp_path = temp_file.path().to_path_buf();
let mut file = tokio::fs::File::create(&temp_path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open temp file: {}", e)))?;
let mut hasher = blake3::Hasher::new();
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(req.into_body());
while let Some(frame_result) = stream.next().await {
let frame = frame_result
.map_err(|e| AppError::bad_request(format!("Failed to read request body: {}", e)))?;
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_upload {
// Abort early — stop reading, delete temp file
drop(file);
let _ = tokio::fs::remove_file(&temp_path).await;
return Err(AppError::payload_too_large(format!(
"Upload exceeds maximum size of {} bytes",
max_upload
)));
}
hasher.update(chunk);
file.write_all(chunk).await.map_err(|e| {
AppError::internal_error(format!("Failed to write to temp file: {}", e))
})?;
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush temp file: {}", e)))?;
drop(file);
let hash = hasher.finalize().to_hex().to_string();
// Shared with the NextCloud-compat PUT handler; peak heap ~one frame
// regardless of file size. Honors `upload_temp_dir` to keep the spool
// off tmpfs/RAM.
let spooled = spool_body_to_temp(
req.into_body(),
max_upload,
state.core.config.storage.upload_temp_dir.clone(),
)
.await?;
let temp_path = spooled.temp.path().to_path_buf();
let total_bytes = spooled.size as usize;
let hash = spooled.hash;
// ── Quota enforcement ────────────────────────────────────
if let Some(storage_svc) = state.storage_usage_service.as_ref()