41aad26702
feat(upload): cover chunk upload + add support of different digest hash
...
Prefer stream storage rather using buffered (in memory)
note: on many unix like tmpfs are in-memory, sungle PUT are sized limited
Storage map (NC stands for Nextcloud gateway)
┌───────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
│ Streaming surface │ Destination │ Configurable via │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ REST chunked PUT /api/uploads/{id} chunk │ {storage_path}/.uploads/{upload_id}/chunk_{NNNNNN} │ OXICLOUD_STORAGE_PATH (the .uploads subdir is │
│ │ │ hard-wired) │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ REST chunked assemble (during /complete) │ {storage_path}/.uploads/{upload_id}/assembled │ same │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ NC chunked PUT /dav/uploads/.../{chunk} │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name} │ same │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ NC chunked assemble (during MOVE) │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/.assembled │ same │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ NC single-file PUT /dav/files/.../{path} (via │ OXICLOUD_UPLOAD_TMPDIR if set, else OS default temp (/tmp on │ OXICLOUD_UPLOAD_TMPDIR │
│ spool_body_to_temp) │ Linux) │ │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ REST WebDAV PUT /webdav/{path} (via │ same as above │ OXICLOUD_UPLOAD_TMPDIR │
│ spool_body_to_temp) │ │ │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ REST multipart upload /api/files/upload │ {storage_path}/.dedup_temp/upload-{uuid} │ OXICLOUD_STORAGE_PATH (hard-wired subdir) │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ WOPI PutFile │ OS default temp via NamedTempFile::new() (no override) │ (none — bug worth tracking) │
├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ Final blob storage (after fsync + rename) │ {storage_path}/.blobs/{ab}/{abc…}.blob │ OXICLOUD_STORAGE_PATH │
└───────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┘
one caveat: a malicious user can create many chunked upload and saturate local storage
2026-06-09 09:52:09 +02:00
5e638691ad
security(upload): cap upload size to prevent memody/disk consumption
...
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
(differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
hurl test validate the change
Streams the request body straight to the chunk file with peak heap of
~one HTTP frame, regardless of chunk size or the configured cap. The
`storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
default 100 MB) bounds a single PUT — separate from `max_upload_size`
which governs whole-file uploads. Without this separation, a client
could submit a chunk up to the whole-file cap (10 GB default) and
monopolise server memory.
2026-06-09 09:26:33 +02:00
Edouard Vanbelle